by Glenn Hoyle | Nov 4, 2023 | ARTS Standard 6, Training
Our ongoing list of ARTS education and training resources
Since we have many education and training resources at ARTS I decided to create this page to collect these resources in one place. This page will be updated as needed.
Resources for the Annual Reports.
Between March and April of 2023, the COA planned and implemented 5 training sessions on standards 2-6. These Zoom sessions were recorded. These sessions discussed the COA standards and how to use them to improve academic quality at member schools. Starting with Standard 6 we explained quantitative and qualitative methods of evaluation and assessment and then progressed to other standards. The recordings can be found here: Session 1: Standard 6; Session 2: Standard 2, Session 3: Standard 3, Session 4: Standard 5, Session 5: Standard 4.
Links for Training Videos for ARTS-COA Site Visit Teams – 2023 Edition. The following are training videos for the ARTS-COA Commissioners. Each site visit member is required to complete this training.
Annual Meeting Trainings (2023)
by Glenn Hoyle | Sep 26, 2023 | ARTS Standard 6
Standard 6.4 and Course Evaluations.
How to “evaluate” the tools you use to evaluate student learning.
ARTS Standard 6-4 asks the question, how are you evaluating the instruments which you are using to assess student learning? One of the most common responses is that institutions are using course evaluation form. One issue that I noted from a review of the ARTS Annual Reports is that schools often don’t understand how to apply this standard. So here’s an example.
One of the questions I took from a school’s course evaluation was: “This course made important contributions of my understanding of the Bible” — The options for response were “Yes” or “No”. To apply Standard 6-4 would be to evaluate the effectiveness of your student course evaluation forms, and to evaluate the individual questions you ask in your evaluations. As we will see below, this question can be significantly improved.
The question, “This course made important contributions to my understanding of the Bible,” while straightforward, has some problems that can be improved to gather more meaningful feedback:
- Lack of Nuance: The question only provides two response options, “yes” and “no,” which doesn’t allow for a nuanced evaluation of the course’s impact. It doesn’t capture the extent or degree of the contributions made to the student’s understanding.
- Limited Insight: It doesn’t provide specific details or examples of how the course contributed to the understanding of the Bible. This limits the instructor’s ability to identify what aspects of the course were particularly effective or areas that might need improvement.
- No Room for Comments: The question doesn’t include an open-ended comment section where students can provide qualitative feedback, suggestions, or additional context for their responses. Such comments can offer valuable insights into the student experience.
To improve this question, you could consider the following:
Revised Question: “To what extent did this course contribute to your understanding of the Bible?”
Response Options:
- Significantly
- Moderately
- Slightly
- Not at all
Additional Comment Section: Include an open-ended comment section where students can provide specific examples, suggestions for improvement, or elaborate on their responses.
This revised question allows for a more nuanced evaluation of the course’s impact and provides valuable qualitative feedback that can help instructors and course developers make targeted improvements to the curriculum and teaching methods. As you can see, the application of a little common sense in question design will help to improve the feedback your school receives from it’s evaluation and assessment instruments.
by Glenn Hoyle | Mar 4, 2023 | ARTS Standard 6, Assessment
ARTS requires schools to provide an annual assessment plan highlighting student learning outcomes. This can sound like a daunting task, but it’s actually common sense. Let me break this down into 7 simple steps.
- Who is responsible? The first thing your assessment plan should have is a statement of the person or persons at your institution who get together to discuss how well your students are meeting your learning goals. Your plan should list those people, how they are organized, and how often they meet and where meeting minutes are kept .
- Do you have clearly stated learning outcomes? All schools have program goals and learning outcomes, but when was the last time your reviewed and updated them? Understand the difference between general goals, and specific learning outcomes. Goals are big vision statements. E.g. “Our goal is to equip men to be pastors”. But there are many dimension in “equipping”. If I were to ask you, show me an “equipped man” you should be able to list specific things that this person would be able to know, value or do – those specific items are your learning outcomes.
- What are the tools you use to measure your learning outcomes? The one tool that always comes to mind is the student’s course evaluation form. But there are other evaluation methods. Measuring student completion and drop out rates is a form of assessment. Peer reviews can be helpful. Asking other faculty members to review completed assignments is another tool. Reviewing rubrics used to evaluate student sermons, papers or portfolios is yet another. There are several types of tools you can use to assess student learning.
- How do you analyze your assessment data? While all seminaries collect course evaluations, not everyone takes the time to thoughtfully analyze the results. A quick glance at what students say is not sufficient for an assessment plan. Take the time to analyze, summarize, distribute and discuss the data.
- Do you conduct an annual review your assessment tools? When was the last time you reviewed your assessment processes and tools? Are you still using the same assignments, rubrics, surveys and course evaluation forms that you were 5 years ago? Things change. Have you changed to keep up?
- Are you improving based on your assessment results? In assessment lingo, this is called “closing the loop”. Your plan should explain how the data you’ve collected in your assessments is helping you to improve your teaching and learning results. Each year you should list specific things you are doing better because of your assessment results.
- Are you telling the world about your learning results? Where on your school’s web page is a link that says something like: “Educational quality”, or “Measuring student learning”? This page should have the data you collect annually regarding graduation rates, results of student survey, testimonials, etc.
When you answer the above seven questions you will have created an ARTS annual learning assessment plan. The first time you create a plan, it will be relatively simple. As assessment becomes part of your regular academic cycle you will find that the process becomes much easier and the results more valuable.
In a future posting I’ll explore details of how to create meaningful learning outcomes.
