Academic Program Assessment Report: Artificial Intelligence Graduate Certificate, 2024–2026
Biennial: 2024-2025/2025-2026
Academic Year(s) Assessed: 2024 - 2026
College: Norm Asbjornson College of Engineering
Department: Gianforte School of Computing
Department Head: John Paxton
Submitted by: John W. Sheppard
PROGRAM(S) ASSESSED
| Majors | Minors, Options, etc. |
|---|---|
| Artificial Intelligence Graduate Certificate | N/A |
NOTE:
Given the low number of enrollments in this program, the GSoC faculty intend to sunset this program. This decision will be reflected in the assessment that follows.
Section 1. Past Assessment Summary.
Summarize findings from last time this PLO(s) was assessed. Include findings from the last assessment of this PLO that influenced this cycle’s assessment approach. (Note: this is NOT a summary of what was done during last year’s assessment cycle unless you are assessing the same PLOs again this year; this is a summary of the last time you assessed the PLO(s) that you are re-assessing this cycle. You will connect the summary to the reflection portion in Section 7: Closing the Loop. Table has been provided to support the intention of this question.)
| Which PLO(s) are you assessing this cycle? Please include the description. | When was the last time you assessed this PLO(s)? | Action items from last assessment of this PLO(s). | Were past action items addressed in this reporting cycle? This will be discussed in depth in Section 4. |
|---|---|---|---|
| 1 | N/A | N/A | N/A |
| 2 | N/A | N/A | N/A |
| 3 | N/A | N/A | N/A |
| 4 | N/A | N/A | N/A |
| 5 | N/A | N/A | N/A |
Section 2. Actionable Research Question for Your Assessment.
How effective was the AI Graduate Certificate Program in meeting the needs of students enrolled in the program? Note that a deeper research question is not relevant, given the intent to sunset the program.
Section 3. Assessment Plan, Schedule, and Data Sources.
- Did you change the previously established Assessment Plan Schedule. If yes, how was it changed?
No
- Please provide a multi-year assessment schedule that will show when all program learning outcomes will be assessed, and by what criteria (data). List all of your PLOs in full for reference - not just the ones being assessed in this cycle. Add rows as necessary.
ASSESSMENT PLANNING SCHEDULE CHART
| PLO# | PROGRAM LEARNING OUTCOME | 2024-2025 | 2025-2026 | 2026-2027 | 2027-2028 | Data Source |
|---|---|---|---|---|---|---|
| 1 | Define the breadth and depth of AI, and the potential impacts of AI on society. | N/A | 9/30/26 | N/A | N/A | DegreeWorks |
| 2 | Compare the strengths and weaknesses of current and emerging methods and technologies in AI. | N/A | 9/30/26 | N/A | N/A | DegreeWorks |
| 3 | Analyze the suitability of AI methods to current application areas. | NA | 9/30/26 | NA | NA | DegreeWorks |
| 4 | Evaluate the behavior of AI methods on real-world problems. | N/A | 9/30/26 | N/A | N/A | DegreeWorks |
| 5 | Create AI solutions to traditionally unsolvable problems. | N/A | 9/30/26 | N/A | N/A | DegreeWorks |
- What are the threshold values for which your program demonstrates student achievement? Provide a rationale for your threshold values.
| PROGRAM LEARNING OUTCOME | Threshold Value | Data Source |
|---|---|---|
| Define the breadth and depth of AI, and the potential impacts of AI on society. | At least 75% of students will pass CSCI 446 with a grade of B or better. At least 33% of students will pass CSCI 446 with a grade of A or better. This course is focused on defining the depth and breadth of AI as a field. | All students who completed the certificate |
| Compare the strengths and weaknesses of current and emerging methods and technologies in AI. | At least 75% of students will have passed one of CSCI 546 or CSCI 547, and at least 33% of students will have passed both CSCI 546 and CSCI 547, both of which are focused on current and emerging methods in AI and ML. | All students who completed the certificate |
| Analyze the suitability of AI methods to current application areas. | At least 50% of students will have passed one of CSCI 547 or M 508, and at least 33% of students will have passed both CSCI 547 and M 508. These courses involve critical analysis and application of machine learning methods | All students who completed the certificate |
| Evaluate the behavior of AI methods on real-world problems. | At least 50% of students will have passed one of CSCI 547, CSCI 550, STAT 511, STAT 512, or M 507. At least 33% of students will have passed at least two of these courses. These courses include projects focused on real-world problems. | All students who completed the certificate |
| Create AI solutions to traditionally unsolvable problems. | At least 50% of students will have passed one of CSCI 546, CSCI 547, or CSCI 550. At least 33% of students will have passed at least two of these courses. The problems studied in these courses are not solvable by traditional means. | All students who completed the certificate |
Section 4 What Was Done?
-
Was the completed assessment consistent with the program’s assessment plan? If not, please explain the adjustments that were made.
Self-reporting Metric (required answer):
☒ Yes ☐ No
- How was the data collected and analyzed and by whom? Please include method of collection and sample size.
All students who completed the certificate were included in the assessment. There were only three students to complete the certificate during the assessment period.
- Please provide a rubric for each PLO being assessed that demonstrates how your data was evaluated. (See Instructions document for an example of what this could look like.)
PLO-1: UG evaluated via projects (direct) and active learning activities (indirect). G evaluated via in-class discussions (indirect), written summaries (direct), and project reports (direct).
PLO-2: All evaluations are via in-class discussions (indirect) and written assignments (direct).
PLO-3: All evaluations are via in-class discussions (indirect) and written assignments (direct).
PLO-4: All evaluations are via in-class discussions (indirect) and written assignments (direct).
PLO-5: All evaluations are via in-class discussions (indirect) and written assignments (direct).
Section 5. What Was Learned.
- Based on the analysis of the data, and compared to the threshold values established, what was learned from the assessment? (Relate this to your research question in Section 3.)
PLO1: Only one student took one of the undergraduate courses, and that student received an A in the course. The learning objectives of the course address the depth and breadth of the field of AI, so this student satisfied this objective.
PLO2: All three students completed either CSCI 546 or CSCI 547 with a grade of B or better, thus meeting the threshold value for this objective. The learning outcome was addressed through the design and learning objective of these two courses, which are both focused on comparing a variety of emerging methods in artificial intelligence and machine learning.
PLO3: All three students passed either M 508 or CSCI 547 with a grade of B or better. None of the students took both M 508 and CSCI 547 as this is not required for the program. The first sub-objective has been satisfied. The second has not, simply by the fact none of the students met the criterion of having taken both courses. The learning outcome was addressed given the analytical nature of both courses. M 508 considers the mathematical foundations of machine learning and their relationship to machine learning applications, while CSCI 547 focuses on an in-depth research project that includes an assessment of the results vis-à-vis application areas in machine learning.
PLO4: Of the three students who completed the certificate, two took and passed CSCI 547, two took and passed CSCI 550, two took and passed STAT 511, one took and passed STAT 512, and none of the students took M 507. As such, the threshold has been met for this PLO. All of the above courses are designed to consider the application of the methods to real-world problems.
PLO5: Of the three students who completed the certificate, two took and passed the equivalent of CSCI 546 (note that CSCI 548 is an early version of this course), two took and passed CSCI 547, and two took and passed CSCI 550. The methods addressed in these courses do not follow “standard means”, and all students are required to complete projects utilizing these methods.
- What areas of strength in the program were identified from this assessment process?
The courses in the certificate that were chosen were relatively broad in scope. None of the students completed exactly the same courses, and all of the students performed well. All three also either completed or are in the process of completing a PhD, two in computer science and one in physics. Thus a strength of the program is that it provides students with a grounding in AI that can be utilized in a variety of disciplines and research interests. c) What areas were identified that either need improvement or could be improved in a different way from this assessment process?
The primary issue is with respect to enrollments. The numbers of students pursuing this program are quite low, thus a decision has been made to sunset the program. The belief is that the certificate does not support the broader MSU community, given the way the enrollment process has been designed.
Therefore, to improve service to the university community, a new MS in AI will be proposed to replace this program.
Section 6. How We Responded.
- Describe how “What Was Learned” was communicated to the department, or program faculty. How did faculty discussions re-imagine new ways program assessment might contribute to program growth/improvement/innovation beyond the bare minimum of achieving program learning objectives through assessment activities conducted at the course level?
The results of this report were distributed to the faculty via email. Multiple discussions about the program have been held at annual retreats and faculty meetings. The consensus of the faculty is that the program should be sunset in favor of proposing a new MS in Artificial Intelligence.
- How are the results of this assessment informing changes to enhance student learning in the program?
No changes are being made at this time in the courses that make up this program. Note, however, that one of the byproducts of creating the program was the creation of CSCI 546, which is a topics-based course that students are permitted to take twice. Given the increase in faculty in the department with interest in AI and ML, more topics are being offered to the students, which increases their ability to encounter, explore, and apply a broader range of methods in artificial intelligence.
- If information outside of this assessment is informing programmatic changes, please describe that.
The primary information informing the recommendation to sunset the program is the low enrollments.
- What support and resources (e.g., workshops, training, etc.) might you need to make these adjustments?
N/A
Section 7. Closing the Loop(s).
Closing the Loop refers to continuous improvement and planning for future action. Reflect on the program learning outcomes and how they were assessed in the previous cycle (refer to #1 of the report). What was learned in this cycle and what actions will the program take in planning for future assessment of the PLO(s).
- Self-Reporting Metric (required answer): Based on the findings and/or faculty input, will there be any changes made (such as plans for measurable improvements, realignment of learning outcomes, curricular changes, etc.) in preparation for upcoming assessments?
☐ Yes ☒ No
(This is not applicable given decision to sunset)
- In reviewing the last report that assessed the PLO(s) in this assessment cycle, what changes proposed were implemented and will be measured in future assessment reports? What action will be taken to improve student learning objectives going forward?
N/A. This was the first assessment.
- Have you seen a change in student learning based on other program adjustments made in the past? Please describe the adjustments made and subsequent changes in student learning.
N/A. This was the first assessment.
- If the program sees anything emerging from this assessment cycle that it anticipates would be a factor or an item of discussion in its 7-year program review cycle, please use this space to document that for future reference.
N/A.
OTHER COMMENTS
Use this area to documented anything not covered in previous sections that department wants to remember for future reference.
Between 2024 and 2026, only three students completed the Graduate AI Certificate. The way the certificate was designed limited enrollment to those students already enrolled in an MS or PhD program at MSU and did not support people applying to the program just for purposes of receiving the certificate. This model does not work, and enrollments have not met the minimum required by the Board of Regents. The faculty met to discuss this and have decided that this certificate is going to be sunset in favor of creating a new MS in AI.
A review of the Year 0 plan resulted in several recommendations with respect to assessing the PLOs. Because of the decision to sunset this program, while the recommendations were considered, the evaluation was not updated to reflect those recommendations.
Final Steps:
- Submit report to programassessment@montana.edu
- Update Department program assessment report website and send link to deborahblanchard@montana.edu or programassessment@montana.edu
- Update PLO language in CIM if needed so catalog will be up-to-date.
Do you want to know more about how to use Canvas Outcomes to conduct Program Assessment and Insights for Canvas Outcomes to collect data? If yes, Deb Blanchard will reach out to schedule a time to discuss it further to see if it is the right fit for your program(s).
☐ Yes ☒ No
