by Glenn Hoyle | Nov 8, 2023 | Accreditation_Actions, Assessment, Training
How ARTS uses its Standards and Criteria for Excellence
ARTS-COA created the ARTS Standards and Criteria for Excellence which integrate the COA standards with a listing of typical documentation and suggested evaluative questions and criteria. In developing this document ARTS-COA reviewed similar documents published by TRACS, ABHE, PEQAB (Ontario Canada), and ATS. This is a living document, reviewed at the 2023 annual meeting, and will be reviewed annually.
These standards of excellence are rubrics that are used by site visit teams, and the COA, as the basis for evaluating self-studies, during accreditation visits, and the ARTS annual reports. Our criteria contain the following scale values:
- Standard is met
- Standard is partially met
- Standard is not met
However, the COA has had discussions (at the recent annual meeting and during some COA meetings) regarding adding additional gradations to these three scale values. Would it be better to expand our criteria values from 3 to a greater number (such as 5-7)? I thought it would be worth summarizing our discussions and the pros and cons of expanding our scale.
Pros of Expanding the Scale:
- Enhanced Precision: A larger scale provides more granularity, allowing for a more precise assessment of performance. This can help in distinguishing between various degrees of achievement, especially in cases where the standard is not clearly met or not met.
- Improved Feedback: A larger scale can offer more detailed feedback to institutions, enabling them to identify specific areas where they excel or need improvement. This can lead to more targeted and actionable recommendations.
- Fairness and Flexibility: A larger scale allows for a more nuanced evaluation, accommodating situations where an institution might partially meet a standard but excel in some aspects. This added flexibility can be fairer to institutions that don’t neatly fit into a binary “met” or “not met” category.
- Encouraging Continuous Improvement: With a larger scale, institutions may be more motivated to strive for higher levels of performance, as there are more gradations to aim for. This can foster a culture of continuous improvement.
- Reflecting Complexity: In situations where standards are complex and multifaceted, a larger scale can better capture the complexity of the evaluation process, providing a more accurate picture of an institution’s performance.
Cons of Expanding the Scale:
- Increased Subjectivity: A larger scale may introduce more subjectivity in the evaluation process. It can be challenging to reach a consensus when there are numerous gradations, and different evaluators may interpret the criteria differently.
- Longer Evaluation Processes: Expanding the scale may require more time and resources for evaluations, as each additional gradation necessitates more detailed analysis and discussion.
- Difficulty in Maintaining Consistency: With more gradations, it can be more challenging to maintain consistency in evaluations across different evaluators, leading to potential discrepancies in assessments.
- Complexity in Reporting: A larger scale can make reporting and communication of evaluation results more complex, potentially making it harder for stakeholders to understand and act upon the feedback.
- Potential for Ambiguity: As the scale becomes more granular, there’s a risk of introducing ambiguity about what each gradation means, making it harder for evaluators to apply the scale consistently.
The current thinking of the COA is that ultimately a school is either meeting or not meeting the standard. During a site visit, or in discussions of an Annual Report submission, participants have the opportunity to explore the nuances of what has been presented, ask questions and seek clarification. It’s an inductive process which goes from the qualitative to a quantitative conclusion. Our current thinking is that we will not change this scale, but continue to think about our rubric and how best to evaluate. We invite further discussion on this topic. Please contact the executive director of ARTS if you would like to make your opinion known.
by Glenn Hoyle | Nov 4, 2023 | ARTS Standard 6, Training
Our ongoing list of ARTS education and training resources
Since we have many education and training resources at ARTS I decided to create this page to collect these resources in one place. This page will be updated as needed.
Resources for the Annual Reports.
Between March and April of 2023, the COA planned and implemented 5 training sessions on standards 2-6. These Zoom sessions were recorded. These sessions discussed the COA standards and how to use them to improve academic quality at member schools. Starting with Standard 6 we explained quantitative and qualitative methods of evaluation and assessment and then progressed to other standards. The recordings can be found here: Session 1: Standard 6; Session 2: Standard 2, Session 3: Standard 3, Session 4: Standard 5, Session 5: Standard 4.
Links for Training Videos for ARTS-COA Site Visit Teams – 2023 Edition. The following are training videos for the ARTS-COA Commissioners. Each site visit member is required to complete this training.
Annual Meeting Trainings (2023)
by Glenn Hoyle | Nov 2, 2023 | Accreditation_Actions, News
Annual Meeting News and Events.
At the ARTS-COA Annual Meeting, October 5-6, 2023, the following official actions were taken:
Official Actions – Congratulations to Metro Atlanta Seminary, Metro Baltimore Seminary and Grace Bible Theological Seminary. In official actions of the COA:
- Metro Atlanta Seminary (MAS) was presented an official certificate documenting the letter of Commendation and Continuation after satisfying the requirements of their reaffirmation study.
- Metro Baltimore Seminary (MBS) appeared before the COA and successfully answered all final questions regarding their candidacy. The COA voted to award membership and accreditation to MBS.
- Grace Bible Theological Seminary (GBTS) also appeared before the COA. The COA voted to extend full accreditation to GBTS with a letter of notation. Items in the letter of notation must be addressed by the next annual meeting.
Official Actions with Affiliates — At the COA annual meeting on Thursday October 5th, the COA approved all the Annual Reports from Affiliate members. Approval for several affiliates had been delayed because the COA had been waiting for additional information. The application for Affiliate Membership for Forge Theological Seminary was also approved.
New COA Members: Dr. Thad James of Birmingham Theological Seminary was confirmed by the ARTS Board of Directors to fill the open COA position created by the resignation of Dr. Joey Pipa of Greenville Seminary. The Board also approved Dr. Sidney Dyer of Greenville Seminary as an alternate to the COA. Dr. Dyer will participate in COA activities, go through the Commissioner training, but will not have a vote. Welcome to both Thad and Sid!
You-Tube Channel: Please check out the ARTS YouTube Channel: https://www.youtube.com/channel/UCcBVS_mFw34_9DW3B0aIvPg Special thanks to Brice Bigham at CBTS for creating the videos on this channel.
COA Commissioner Training Program: In early 2023 ARTS-COA created a Commissioners training program for site-visitors. During the COA meeting the effectiveness of the training was evaluated and areas of improvement were noted. The revised training course is located here: q5vv74r
Review of COA Documentation: Another important activity we took at the annual meeting was the annual review of COA documentation and training. Each year we look at our Policies and Procedures, our website, our procedures for evaluation of site-visits and much more. Our goal is continuous improvement.
Finally, a copy of the COA’s report which was given on October 6th to the BOD is found here.
by Glenn Hoyle | Sep 30, 2023 | Accreditation_Actions, Announcements, News
Metro Atlanta Seminary re-accredited for 10 years.
The COA voted on September 7th to approve a letter of Commendation and Continuance to Metro Atlanta Seminary. The accreditation period will be good for 10 years. Thad James, chair of the site-visit committee, reported that members of the site-visit team were satisfied in MAS’s responses in resolving the 11 notations issued to them on April 29th, 2023.
Background on this decision
At its regularly stated meeting on February 10th, 2023, the COA reviewed a letter of notation to Metro Atlanta Seminary (MAS) in response to their 5-year reaccreditation visit, subsequent follow up report, and an MAS response to the report. (More information about site-visit reports and process is found in this blog post.) The reaccreditation site-visit was conducted in July of 2022.
The Letter of Notation documenting discrepancies is based on a two-thirds decision of the ARTS-COA. In the case of MAS, the newly revised ARTS-COA rubric for assessing compliance with standards was employed. Our rubric specifies that institutions must fully meet all the criteria for sub-standards in order to be in compliance with the full standard. Institutions receiving a letter of notation remain full members, but are expected to respond to the letter within 60 days, and resolve all issues noted within 12 months after the date of the notation. (Policies and Procedures, Section 1.d.iii)
Accordingly the COA is seeking from MAS further details regarding a curriculum overview showing a plan for resolution for standards 2.4, 3.5, 5.2, 5.3, 6 and 10.5. We mention these items to be in compliance with ARTS-COA Policies and Procedures, Section 4.a.ii.
While further clarification is being sought on the above items, the COA also commended MAS for the many other improvements which MAS has implemented as a result of the site-visit report.
by Glenn Hoyle | Sep 26, 2023 | ARTS Standard 6
Standard 6.4 and Course Evaluations.
How to “evaluate” the tools you use to evaluate student learning.
ARTS Standard 6-4 asks the question, how are you evaluating the instruments which you are using to assess student learning? One of the most common responses is that institutions are using course evaluation form. One issue that I noted from a review of the ARTS Annual Reports is that schools often don’t understand how to apply this standard. So here’s an example.
One of the questions I took from a school’s course evaluation was: “This course made important contributions of my understanding of the Bible” — The options for response were “Yes” or “No”. To apply Standard 6-4 would be to evaluate the effectiveness of your student course evaluation forms, and to evaluate the individual questions you ask in your evaluations. As we will see below, this question can be significantly improved.
The question, “This course made important contributions to my understanding of the Bible,” while straightforward, has some problems that can be improved to gather more meaningful feedback:
- Lack of Nuance: The question only provides two response options, “yes” and “no,” which doesn’t allow for a nuanced evaluation of the course’s impact. It doesn’t capture the extent or degree of the contributions made to the student’s understanding.
- Limited Insight: It doesn’t provide specific details or examples of how the course contributed to the understanding of the Bible. This limits the instructor’s ability to identify what aspects of the course were particularly effective or areas that might need improvement.
- No Room for Comments: The question doesn’t include an open-ended comment section where students can provide qualitative feedback, suggestions, or additional context for their responses. Such comments can offer valuable insights into the student experience.
To improve this question, you could consider the following:
Revised Question: “To what extent did this course contribute to your understanding of the Bible?”
Response Options:
- Significantly
- Moderately
- Slightly
- Not at all
Additional Comment Section: Include an open-ended comment section where students can provide specific examples, suggestions for improvement, or elaborate on their responses.
This revised question allows for a more nuanced evaluation of the course’s impact and provides valuable qualitative feedback that can help instructors and course developers make targeted improvements to the curriculum and teaching methods. As you can see, the application of a little common sense in question design will help to improve the feedback your school receives from it’s evaluation and assessment instruments.
