Measuring Standards of Excellence

by Glenn Hoyle | Nov 8, 2023 | Accreditation_Actions, Assessment, Training

How ARTS uses its Standards and Criteria for Excellence

ARTS-COA created the ARTS Standards and Criteria for Excellence which integrate the COA standards with a listing of typical documentation and suggested evaluative questions and criteria. In developing this document ARTS-COA reviewed similar documents published by TRACS, ABHE, PEQAB (Ontario Canada), and ATS. This is a living document, reviewed at the 2023 annual meeting, and will be reviewed annually.

These standards of excellence are rubrics that are used by site visit teams, and the COA, as the basis for evaluating self-studies, during accreditation visits, and the ARTS annual reports. Our criteria contain the following scale values:

• Standard is met
• Standard is partially met
• Standard is not met

However, the COA has had discussions (at the recent annual meeting and during some COA meetings) regarding adding additional gradations to these three scale values. Would it be better to expand our criteria values from 3 to a greater number (such as 5-7)? I thought it would be worth summarizing our discussions and the pros and cons of expanding our scale.

Pros of Expanding the Scale:

1. Enhanced Precision: A larger scale provides more granularity, allowing for a more precise assessment of performance. This can help in distinguishing between various degrees of achievement, especially in cases where the standard is not clearly met or not met.
2. Improved Feedback: A larger scale can offer more detailed feedback to institutions, enabling them to identify specific areas where they excel or need improvement. This can lead to more targeted and actionable recommendations.
3. Fairness and Flexibility: A larger scale allows for a more nuanced evaluation, accommodating situations where an institution might partially meet a standard but excel in some aspects. This added flexibility can be fairer to institutions that don’t neatly fit into a binary “met” or “not met” category.
4. Encouraging Continuous Improvement: With a larger scale, institutions may be more motivated to strive for higher levels of performance, as there are more gradations to aim for. This can foster a culture of continuous improvement.
5. Reflecting Complexity: In situations where standards are complex and multifaceted, a larger scale can better capture the complexity of the evaluation process, providing a more accurate picture of an institution’s performance.

Cons of Expanding the Scale:

1. Increased Subjectivity: A larger scale may introduce more subjectivity in the evaluation process. It can be challenging to reach a consensus when there are numerous gradations, and different evaluators may interpret the criteria differently.
2. Longer Evaluation Processes: Expanding the scale may require more time and resources for evaluations, as each additional gradation necessitates more detailed analysis and discussion.
3. Difficulty in Maintaining Consistency: With more gradations, it can be more challenging to maintain consistency in evaluations across different evaluators, leading to potential discrepancies in assessments.
4. Complexity in Reporting: A larger scale can make reporting and communication of evaluation results more complex, potentially making it harder for stakeholders to understand and act upon the feedback.
5. Potential for Ambiguity: As the scale becomes more granular, there’s a risk of introducing ambiguity about what each gradation means, making it harder for evaluators to apply the scale consistently.

The current thinking of the COA is that ultimately a school is either meeting or not meeting the standard. During a site visit, or in discussions of an Annual Report submission, participants have the opportunity to explore the nuances of what has been presented, ask questions and seek clarification. It’s an inductive process which goes from the qualitative to a quantitative conclusion. Our current thinking is that we will not change this scale, but continue to think about our rubric and how best to evaluate. We invite further discussion on this topic. Please contact the executive director of ARTS if you would like to make your opinion known.