Annual: 2025-2026 or Biennial: 2024-2025/2025-2026

Academic Year(s) Assessed: 2024-2025/2025-2026
College: Engineering
Department: School of Computing
Department Head: John Paxton
Submitted by: Sean Yaw

PROGRAM(S) ASSESSED

Majors Minors, Options, etc.
MS in Computer Science Thesis Track and Courses-only Track

Section 1. Past Assessment Summary.

The MS Computer Science program last assessed all Program Learning Outcomes during the 2022-2023 and 2023-2024 academic years. Students met the established achievement thresholds for all PLOs. The assessment found strong performance in graduate coursework and thesis-track research activities, while assessment of scientific communication through presentations in CSCI 532 also indicated satisfactory student performance.

The previous assessment and subsequent Assessment and Outcomes Committee feedback identified opportunities to improve the assessment methods, particularly for PLOs 1 and 2, which had been assessed using overall graduate GPA. The Committee recommended measures more directly aligned with the individual PLOs. In response, the current assessment uses breadth of successfully completed graduate-level CSCI coursework at degree completion for PLO 1 and direct evidence from student projects in required MS courses for PLO 2. PLO 3 continues to be assessed through presentations from CSCI 532. PLOs 4 and 5 continue to be assessed through the thesis defense and thesis, respectively.

Which PLO(s) are you assessing this cycle? Please include the description. When was the last time you assessed this PLO(s)? Action items from last assessment of this PLO(s). Were past action items addressed in this reporting cycle? This will be discussed in depth in Section 4.
1 - Demonstrate technical expertise in the fundamental areas of computer science. (Thesis and courses-only tracks) 2022-2023/2023-2024 Improve assessment beyond overall graduate GPA by using a measure more closely aligned with the PLO. Yes. Assessment now considers breadth of CSCI coursework completed by students at degree completion.
2 - Integrate their knowledge of disparate computer science subjects. (Thesis and courses-only tracks) 2022-2023/2023-2024 Improve assessment beyond overall graduate GPA by using direct evidence of student learning. Yes. Student projects in required MS coursework (CSCI 532) are used as direct evidence of integration of CS knowledge.
3 - Effectively communicate knowledge to a scientific audience. (Thesis and courses-only tracks) 2022-2023/2023-2024 No changes identified as necessary. N/A. Assessment continues through CSCI 532 presentation performance.
4 - Communicate research effectively to a scientific audience. (Thesis track) 2022-2023/2023-2024 No changes identified as necessary. N/A. Assessment continues through the thesis defense.
5 - Perform original research. (Thesis track) 2022-2023/2023-2024 No changes identified as necessary. N/A. Assessment continues through the thesis defense.

Section 2. Actionable Research Question for Your Assessment.

To what extent do MS Computer Science students demonstrate achievement of the program learning outcomes using the revised assessment measures, and are the established performance thresholds being met?

Section 3. Assessment Plan, Schedule, and Data Sources.

  1. Did you change the previously established Assessment Plan Schedule. If yes, how was it changed?

No.

  1. Please provide a multi-year assessment schedule that will show when all program learning outcomes will be assessed, and by what criteria (data). List all of your PLOs in full for reference - not just the ones being assessed in this cycle. Add rows as necessary.

ASSESSMENT PLANNING SCHEDULE CHART

PLO# PROGRAM LEARNING OUTCOME 2024-2025 2025-2026 2026-2027 2027-2028 Data Source
1 Demonstrate technical expertise in the fundamental areas of computer science. (Thesis and courses-only tracks) X X X X Completed coursework records for graduating MS students
2 Integrate their knowledge of disparate computer science subjects. (Thesis and courses-only tracks) X X X X CSCI 532* student projects
3 Effectively communicate knowledge to a scientific audience. (Thesis and courses-only tracks) X X X X CSCI 532* student presentations
4 Communicate research effectively to a scientific audience. (Thesis track) X X X X Thesis Defense
5 Perform original research. (Thesis track) X X X X Thesis

*CSCI 532 was selected as assessment data sources for these PLOs because it is a required course for all CS MS students and data is available.

  1. What are the threshold values for which your program demonstrates student achievement? Provide a rationale for your threshold values.
Threshold Values
PROGRAM LEARNING OUTCOME Threshold Value Data Source
1 - Demonstrate technical expertise in the fundamental areas of computer science. (Thesis and courses-only tracks) 80% of graduating MS students will have successfully completed more than 50% of their traditional coursework in CSCI courses. For this measure, traditional coursework excludes thesis, independent study, and seminar credits. Completing the majority of traditional coursework within CSCI provides evidence of broad formal study in computer science while allowing students flexibility to pursue relevant coursework in related disciplines. Completed coursework records for graduating MS students
2 - Integrate their knowledge of disparate computer science subjects. (Thesis and courses-only tracks) 80% of students will successfully complete the projects in CSCI 532/538. These projects require students to apply computer science knowledge to substantial technical problems and provide opportunities to integrate knowledge from multiple computer science subjects. The 80% threshold represents the expectation that a substantial majority of MS students demonstrate successful performance on this work. CSCI 532 student projects
3 - Effectively communicate knowledge to a scientific audience. (Thesis and courses-only tracks) 80% of students pass courses in-class presentation. In-class presentations provide direct evidence of students' ability to communicate technical knowledge to a scientific audience. The 80% threshold is retained from the previous assessment cycle. CSCI 532 presentations
4 - Communicate research effectively to a scientific audience. (Thesis track) 80% of thesis students pass their defense. The thesis defense provides direct evidence of students' ability to communicate their research to a scientific audience. The 80% threshold is retained from the previous assessment cycle. Thesis defense
5 - Perform original research. (Thesis track) 80% of thesis students have their thesis accepted by their graduate committee. Completion and committee acceptance of the thesis provides direct evidence of students' ability to perform original research. The 80% threshold is retained from the previous assessment cycle. Thesis

Section 4 What Was Done?

  1. Self-reporting Metric (required answer):

    Was the completed assessment consistent with the program’s assessment plan? If not, please explain the adjustments that were made.

☒ Yes
☐ No

  1. How was the data collected and analyzed and by whom? Please include method of collection and sample size.
  • Data was collected and analyzed by Sean Yaw (MS Program Coordinator).
  • Course enrollment data was collected from DegreeWorks for all MS students that graduated in the assessment period.
  • Grades on presentations and course projects in CSCI 532 were collected from the Fall 2025 (20 MS students) and Spring 2026 (7 MS students) offerings,
  • There were 14 MS graduates in 2024-2025 and 13 MS graduates in 2025-2026.
  • There were 3 MS thesis defenses in 2024-2025 and 3 in 2025-2026.
  • Analysis was done by binning students according to the rubric and calculating percentages.
  1. Please provide a rubric for each PLO being assessed that demonstrates how your data was evaluated. (See Instructions document for an example of what this could look like.)
PLO Beginning - 1 Developing- 2 Competent- 3 Accomplished- 4
1 - Demonstrate technical expertise in the fundamental areas of computer science. (Thesis and courses-only tracks) <40% of courses were CS courses ≥40% and ≤50% of courses were CS course >50% and ≤75% of courses were CS courses >75% of courses were CS courses
2 - Integrate their knowledge of disparate computer science subjects. (Thesis and courses-only tracks) Project grades between 70% and 80% Project grades between 80% and 90% Project grades between 90% and 95% Project grades between 95% and 100%
3 - Effectively communicate knowledge to a scientific audience. (Thesis and courses-only tracks) Applicable assignment grades between 70% and 80% Applicable assignment grades between 80% and 90% Applicable assignment grades between 90% and 95% Applicable assignment grades between 95% and 100%
4 - Communicate research effectively to a scientific audience. (Thesis track) Successfully Defends thesis
5 - Perform original research. (Thesis track) Successfully Defends thesis

Section 5. What Was Learned.

  1. Based on the analysis of the data, and compared to the threshold values established, what was learned from the assessment? (Relate this to your research question in Section 3.)

Presented below are the assessment values applied to the rubric for each Indicator.
Indicator 1 - Demonstrate technical expertise in the fundamental areas of computer science.

Beginning
<40% of courses were CS courses
Developing
≥40% and ≤50% of courses were CS course
Competent
>50% and ≤75% of courses were CS courses
Accomplished
>75% of courses were CS courses
Number (percentage) of students in range (2024-2025) 0 (0%) 0 (0%) 4 (29%) 10 (71%)
Number (percentage) of students in range (2025-2026) 0 (0%) 0 (0%) 7 (54%) 6 (46%)

Indicator 2 - Integrate their knowledge of disparate computer science subjects. (Thesis and courses-only tracks for both)

Deficient
Project grades below 70%
Beginning
Project grades between 70% and 80%
Developing
Project grades between 80% and 90%
Competent
Project grades between 90% and 95%
Accomplished
Project grades between 95% and 100%
Number (percentage) of students in range (2024-2025) 0 (0%) 0 (0%) 3 (15%) 2 (10%) 15 (75%)
Number (percentage) of students in range (2025-2026) 0 (0%) 0 (0%) 0 (0%) 1 (14%) 6 (86%)

Indicator 3 - Effectively communicate knowledge to a scientific audience. (Thesis and courses-only tracks)

Deficient
Presentation grades below 70%
Beginning
Presentation grades between 70% and 80%
Developing
Presentation grades between 80% and 90%
Competent
Presentation grades between 90% and 95%
Accomplished
Presentation grades between 95% and 100%
Number (percentage) of students in range (2024-2025) 0 (0%) 1 (5%) 1 (5%) 5 (25%) 13 (65%)
Number (percentage) of students in range (2025-2026) 0 (0%) 0 (0%) 0 (0%) 2 (29%) 5 (71%)

Indicator 4 - Communicate research effectively to a scientific audience.
Indicator 5 - Perform original research. (Thesis track for both)

Deficient
Unsuccessfully Defends thesis
Beginning Developing Competent Accomplished
Successfully Defends thesis
Number (percentage) of students in range (2024-2025) 0 (0%) - - - 3 (100%)
Number (percentage) of students in range (2025-2026) 0 (0%) - - - 3 (100%)

Assessment takeaways:

Overall, student performance was strong across all five indicators. All 27 graduating students met the criterion for PLO 1, with the majority classified as Competent or higher, based on the proportion of traditional coursework completed in CS. All students assessed for PLO 2 achieved at least the Developing level on the CSCI 532 project, with most classified as Accomplished. For PLO 3, 26 of 27 students achieved at least the Developing level on the CSCI 532 presentation, with most students classified as Competent or Accomplished. All six thesis-track students successfully defended their theses and completed the thesis requirements associated with PLOs 4 and 5. Overall, the results indicate consistently strong student performance, relative to the PLOs, across both academic years.

  1. What areas of strength in the program were identified from this assessment process?

Student performance was strong across all five PLOs. In particular, students demonstrated strong breadth of computer science coursework and strong performance on the project and presentation assessments in CSCI 532. Thesis-track students also demonstrated strong research and research communication outcomes, with all assessed students successfully completing their thesis and thesis defense requirements.

  1. What areas were identified that either need improvement or could be improved in a different way from this assessment process?

No areas requiring programmatic improvement were identified through this assessment. All five PLOs exceeded the established achievement thresholds in both academic years. The program will continue to monitor student performance in future assessment cycles using the revised assessment measures.

Section 6. How We Responded.

  1. Describe how “What Was Learned” was communicated to the department, or program faculty. How did faculty discussions re-imagine new ways program assessment might contribute to program growth/improvement/innovation beyond the bare minimum of achieving program learning objectives through assessment activities conducted at the course level? TBD
  2. How are the results of this assessment informing changes to enhance student learning in the program? No curricular changes are planned based on the assessment results, as all PLOs exceeded the established achievement thresholds. The program will continue to emphasize technical presentation and scientific communication skills in graduate coursework.
  3. If information outside of this assessment is informing programmatic changes, please describe that. N/A
  4. What support and resources (e.g., workshops, training, etc.) might you need to make these adjustments?

N/A

Section 7. Closing the Loop(s).

Closing the Loop refers to continuous improvement and planning for future action. Reflect on the program learning outcomes and how they were assessed in the previous cycle (refer to #1 of the report). What was learned in this cycle and what actions will the program take in planning for future assessment of the PLO(s).

  1. Self-Reporting Metric (required answer): Based on the findings and/or faculty input, will there be any changes made (such as plans for measurable improvements, realignment of learning outcomes, curricular changes, etc.) in preparation for upcoming assessments?

☐ Yes
☒ No

  1. In reviewing the last report that assessed the PLO(s) in this assessment cycle, what changes proposed were implemented and will be measured in future assessment reports? What action will be taken to improve student learning objectives going forward? The assessment methods for PLOs 1 and 2 were revised in response to feedback from the previous assessment cycle. PLO 1 was assessed using the proportion of traditional coursework completed in CS courses at degree completion rather than overall graduate GPA. PLO 2 was assessed using student performance on the CSCI 532 project rather than overall graduate GPA. These revised measures were implemented during the current assessment cycle and provide evidence more specifically aligned with the respective learning outcomes.
  1. Have you seen a change in student learning based on other program adjustments made in the past? Please describe the adjustments made and subsequent changes in student learning. No. The changes implemented from the previous assessment cycle concerned assessment methods rather than curriculum or instruction.
  2. If the program sees anything emerging from this assessment cycle that it anticipates would be a factor or an item of discussion in its 7-year program review cycle, please use this space to document that for future reference. N/A

OTHER COMMENTS

N/A


Final Steps:

  1. Submit report to programassessment@montana.edu
  2. Update Department program assessment report website and send link to deborahblanchard@montana.edu or programassessment@montana.edu
  3. Update PLO language in CIM if needed so catalog will be up-to-date.

Do you want to know more about how to use Canvas Outcomes to conduct Program Assessment and Insights for Canvas Outcomes to collect data? If yes, Deb Blanchard will reach out to schedule a time to discuss it further to see if it is the right fit for your program(s).

☐ Yes
☒ No