Academic Year(s) Assessed: AY 2024-2025
College: NACOE
Department: Gianforte School of Computing
Department Head: John Paxton
Submitted by: Daniel DeFrance, John Paxton

Program(s) Assessed

List all majors (including each option), minors, and certificates that are included in this assessment – add or subtract rows as needed – please use official titles:

Majors Minors, Options, etc.
Computer Science BA

Section 1. Past Assessment Summary.

In AY 2023–2024 (first report for the CS-BA), we learned: (1) capstone portfolios showed noweaknesses, and (2) the custom exam showed that students needed more practiceevaluating computing solutions—especially time-complexity reasoning on the custom exam – and need moreengagement in out-of-classroom activities.

Strengths of the capstone portfolios included strong team projects and effective client engagement. These findings shaped this year’s emphasis on evaluation (PLO 2C) and brought new attention to continuous learning.

Section 2. Institutional Assessment Data Request.

Based on the rationale on the Instructions page, please review your program learning outcomes (PLOs) and identify whether you have PLOs that address the Core Qualities. There are no right or wronganswers.

Yes. Two courses in the program have been identified as overlapping with MSU Core Quality learning outcomes at the “Developing” and “Proficient/Mastery” levels

Identify 1-2 major-required courses that might have student assignments designed to meet these objectives at least at a surface level. If you cannot identify a course in your program that aligns with this request, please check the appropriate box. At this juncture, this is for information gathering as we plan future institutional assessment endeavors.

Core Quality LOs are Institutional Learning Outcome (ILO) PLO overlaps with MSU Core Quality
Mark X if program has at least one PLO that overlaps with an ILO
Beginning Level
e.g. CORE Courses (US, W, Q, IN, CS, IA, IH, IS, D)
Developing Level
e.g. list one 200- or 300-level course
Proficient Level
e.g. list one 300- or 400-level courses, Capstone, Research (R) Core courses
Not Applicable (N/A)
No course exists in our program that addresses this Core Quality / ILO
Thinkers & Problem Solvers X Core classes are designed to address an introductory, foundational level of Core Qualities. Some may overlap into the developing level, but most intermediate-to-developing or proficient/mastery level courses will exist within the majors. CSCI 232 (Data Structures) ESOF 423 (Capstone)
Effective Communicators X ESOF 423 (Capstone: written/oral artifacts)
Local & Global Citizen X ESOF 423 (client/community projects)

Section 3. Actionable Research Question for Your Assessment.

How effectively are students demonstrating..?

  1. evaluation of computing-based solutions in context (PLO 2C), and

  2. engagement in continuous learning (PEO 5) as part of professional preparation

Section 4. Assessment Plan, Schedule, and Data Sources.

  1. Did you change the previously established Assessment Plan Schedule. If yes, how was it changed?

No

  1. Please provide a multi-year assessment schedule that will show when all program learning outcomes will be assessed, and by what criteria (data). List your PLOs in full for reference. Add rows as necessary.
ASSESSMENT PLANNING SCHEDULE CHART
PLO# PROGRAM LEARNING OUTCOME 2024-2025 2025-2026 2026-2027 2027-2028 Data Source*
1 Analyze a complex computing problem and apply principles of computing and other relevant disciplines to identify solutions. X X Capstone Portfolio
2 Design, implement, and evaluate a computing-based solution to meet a given set of computing requirements in the context of the program’s discipline. X X Custom Exam
3 Communicate effectively in a variety of professional contexts. X X Capstone Portfolio, Custom Exam
4 Recognize professional responsibilities and make informed judgments in computing practice based on legal and ethical principles. X X Custom Exam
5 Function effectively as a member or leader of a team engaged in activities appropriate to the program’s discipline. X X Capstone Portfolio
6 Apply computer science theory and software development fundamentals to produce computing-based solutions. x X Capstone Portfolio
  1. What are the threshold values for which your program demonstrates student achievement? Provide a rationale for your threshold values.
Threshold Values; Rationale: At least half of graduating seniors must demonstrate core competencies on rigorous learning outcomes expected of high performing computer scientists; especially those with interdisciplinary training and experiences.
PROGRAM LEARNING OUTCOME Threshold Value Data Source
1) Analyze a complex computing problem and apply principles of computing and other relevant disciplines to identify solutions. 50% of assessed students score above 2 on a 1-4 scoring rubric. Capstone Portfolio
2) Design, implement, and evaluate a computing-based solution to meet a given set of computing requirements in the context of the program’s discipline. 50% of assessed students score above 2 on a 1-4 scoring rubric. Custom Exam
3) Communicate effectively in a variety of professional contexts. 50% of assessed students score above 2 on a 1-4 scoring rubric. Capstone Portfolio, Custom Exam
4) Recognize professional responsibilities and make informed judgments in computing practice based on legal and ethical principles. 50% of assessed students score above 2 on a 1-4 scoring rubric. Custom Exam
5) Function effectively as a member or leader of a team engaged in activities appropriate to the program’s discipline. 50% of assessed students score above 2 on a 1-4 scoring rubric. Capstone Portfolio
6) Apply computer science theory and software development fundamentals to produce computing-based solutions. 50% of assessed students score above 2 on a 1-4 scoring rubric. Capstone Portfolio

Section 5. What Was Done?

  1. Self-reporting Metric (required answer): Was the completed assessment consistent with the program’s assessment plan? If not, please explain the adjustments that were made.

Yes

  1. How was the data collected and analyzed and by whom? Please include method of collection and sample size.

All graduating BA seniors completed the 80-minute custom exam. Responses were evaluated for data-structure/algorithmic fit, logic/correctness, and time-complexity reasoning (Q3); and for evidence of continuous learning (Q12). Exams are reviewed bya faculty assessment team in the Gianforte School of Computing. In AY 24-25, capstone portfolios were assessed by BrendanMumey and John Paxton.

All Computer Science B.A. students take ESOF 423, where they work with a group to complete the capstone project. Each group submits a Capstone Portfolio, which is graded as part of the coursework, and later evaluated by at least two faculty in the Gianforte School of Computing. In AY 24-25, Capstone portfolios were assessed by Hunter Lloyd and Binhai Zhu.

  1. Rubric that demonstrates how data was evaluated:
Indicators Beginning - 1 Developing- 2 Competent- 3 Accomplished- 4
Capstone portfolio indicator 3 (for PLO 1) No design pattern information in portfolio. A design pattern was used but wasn’t justified as the best approach. A design pattern was used, but with incomplete justification. A fully justified design pattern was used.
Capstone portfolio indicator 5 (for PLO 1) No UML information in portfolio. Diagrams and code don’t match. Diagrams and code match, at most two types of UML diagrams used in the project. Diagrams and code match, more than two types of UML diagrams used in the project.
Capstone portfolio indicator 6 (for PLO 1) No design trade-off information in portfolio, or the example given is not explained as a design trade-off. A design trade-off is described, but no justification is given. A design trade-off is described, but the decision made was not justified correctly. A design trade-off is described, with correct analysis.
Custom exam indicator 1 (for PLO 2) Incorrect data structure. Correct data structure, no implementations. Correct data structure, one correct implementation. Correct data structure, two correct implementations.
Custom exam indicator 2 (for PLO 2) 3 or more logic errors in solution. 2 logic errors in solution. 1 logic error in solution. No logic errors in solution.
Custom exam indicator 3 (for PLO 2) Neither case has the correct time complexity. One case has the correct time complexity but the wrong explanation. One case is fully correct. Both cases are fully correct.
Capstone portfolio indicator 4 (for PLO 3) No technical documentation example in portfolio. Documentation contained ten or more grammatical and/or spelling errors per page or was poorly formatted. Documentation had less than ten grammatical or spelling errors per page but did not accurately describe the project. Documentation fully described the project.
Custom exam indicator 4 (for PLO 3) No involvement Less than 25 hours Less than 50 hours 50 hours or more
Custom exam indicator 5 (for PLO 4) Incorrect response. Accurate description of algorithmic bias. Accurate description of algorithmic bias plus partially correct example. Accurate description of algorithmic bias plus correct example.
Custom exam indicator 6 (for PLO 4) No answer. One impact explained. Two impacts explained. Three impacts explained.
Custom exam indicator 7 (for PLO 4) No answer. One correct type of harm. Two correct types of harm. Three correct types of harm.
Capstone portfolio indicator 2 (for PLO 5) No team project information in portfolio. One or more team members did not impact the success of the project. Some team members only completed a specific component of the project, without regard to the rest of the project. Demonstrated genuine teamwork, where the team worked together to develop the project.
Capstone portfolio indicator 7 (for PLO 5) No life cycle information in portfolio. Development did not follow the life cycle described. Development followed the life cycle model described. Development followed the life cycle model described, and benefits and/or problems were described.
Capstone portfolio indicator 1 (for PLO 6) No program in portfolio. Program submitted with no, or incomplete, specifications. Program did not meet specifications. Specifications and a matching program were both submitted.

Section 6. What Was Learned.

  1. Based on the analysis of the data, and compared to the threshold values established, what was learned from the assessment?

Preparedness self-ratings were strongest in teamwork and solution design.

Students desire more industry tools and project-based learning opportunities. Students noted that receiving communication over varying platforms from one course to another is an area to streamline.

  1. What areas of strength in the program were identified from this assessment process?

No weaknesses were notified in the capstone portfolios during this cycle; teams delivered solid technical solutions with effective collaboration and acceptable documentation per rubric.

In a graduating senior survey, students frequently cited the following as being the most valuablecourses: CSCI 232, ESOF 423, ESOF 322, CSCI 366. Students also praised the practical, team-based, well-structured capstone experience.

  1. What areas were identified that either need improvement or could be improved in a different way from this assessment process?

Weakness 1: Custom Exam Question 3 revealed difficulty articulating best/worst-case time complexity for list insertion when k ≪ n and explaining the reasoning clearly. This indicates a weakness in Learning Outcome 2: a student should be able to evaluate a computing-based solution to meet a given set of computing requirements in the context of the program’s discipline.

Weakness 2: Exam Question 12 documentation varied in specificity and depth of non-classroom upskilling activities. This indicates a weakness in Program Educational Objective 5: a student should engage in continuous learning.

Least Valuable Course mentions included ESOF 322 and CSCI 127 due to perceived redundancy (when students had significant prior experience and still started with CSCI 127) or relevance of material.

Section 7. How We Responded.

  1. Describe how “What Was Learned” was communicated to the department, or program faculty. How did faculty discussions re-imagine new ways program assessment might contribute to program growth/improvement/innovation beyond the bare minimum of achieving program learning objectives through assessment activities conducted at the course level?

Results were shared at the faculty retreat in August 2025. Faculty discussed strategies to address evaluation and continuous learning weaknesses.

  1. How are the results of this assessment informing changes to enhance student learning in the program?

Two main responses were agreed upon:

  1. Exam Prompts:

    When the custom exam is given, the instructor will provide a brief overview of each question. Specific examples of continuous learning will be provided to make sure that students fully understand the types of responses that are valid.
  2. Custom Exam Support:

    A new help session will be offered to graduating seniors prior to the exam to refresh them on core computing concepts and provide structured review. This will help address identified weaknesses in evaluation and continuous learning.
  1. If information outside of this assessment is informing programmatic changes, please describe that.

Starting in Spring 2026, the nature of the B.A. Capstone will be modified due to an initiative that the GSoC is undertaking with Northeastern’s CIC. Over the past decade, the CIC has studied nearly 100 computing programs in the U.S. and has developed research-based practices to making computing as appealing as possible to a broad audience. One of these research-based practices is to make sure that interdisciplinary capstones enable students to combine computing with their other interdisciplinary focus. The instructor of the B.A. Capstone, ESOF 423: Software Engineering Applications, has been tasked to revamp the course to ensure it will provide enrolled students with the ability to undertake a meaningful capstone project that involves their non-CS concentration area, individual background and interests.

  1. What support and resources (e.g., workshops, training, etc.) might you need to make these adjustments?

Funding for the work of redesigning the B.A. capstone is being provided by a 3-year grant from the CIC.

Section 8. Closing the Loop(s).

Reflect on the program learning outcomes, how they were assessed in the previous cycle (refer to #1 of the report), and what was learned in this cycle about any actions stemming from the previous cycle.

  1. Self-Reporting Metric (required answer): Based on the findings and/or faculty input, will there be any changes made (such as plans for measurable improvements, realignment of learning outcomes, curricular changes, etc.) in preparation for upcoming assessments?

Yes – the B.A. capstone is being modified, and adjustments are being made to help students better prepare for the custom exam.

  1. In reviewing the last report that assessed the PLO(s) in this assessment cycle, what changes proposed were implemented and will be measured in future assessment reports? What action will be taken to improve student learning objectives going forward?

We implemented the changes described in last year’s report to CSCI 132, CSCI 232, CSCI 246 and ESOF 423. The changes we made last year to address a weakness with the technical writing in the capstone course were effective, resulting in technical writing not being a weakness this year.

Based on this year’s findings, the program will implement measurable changes to improve student learning:

  • Capstone Redesign: Launching in Spring 2026, emphasizing interdisciplinary projects, inclusivity, and continuous learning.
  • Curricular Adjustment: Increased attention to evaluation of computing-based solutions (time complexity analysis) in CSCI 132, 232, and 246.
  • Custom Exam Help Session: Starting in 2025, a structured review session will support seniors in preparing for assessment.
  • Future Monitoring: Continued use of custom exam and portfolios to track improvement on PLO 2C and PEO 5.

These actions represent a progression from last year’s findings and demonstrate program growth. Future assessment cycles will measure the effectiveness of the redesigned capstone, and custom exam support measures.

  1. Have you seen a change in student learning based on other program adjustments made in the past? Please describe the adjustments made and subsequent changes in student learning.

Yes: There was an improvement in the quality of the Capstone Portfolios. Students were shown exemplary prior portfolios and were subsequently better at meeting the requirements for themselves.

  1. If the program sees anything emerging from this assessment cycle that it anticipates would be a factor or an item of discussion in its 7-year program review cycle, please use this space to document that for future reference.

It is possible we will see higher enrollments in the CS B.A. in the future due to:

  1. The work we are doing with the CIC to improve and more effectively brand our degrees.
  2. The ongoing work of our outreach coordinator, who reached more than 7,200 K-12 Montana students this past year.
  1. The opening of Gianforte Hall in July 2026. Gianforte Hall will include a presence of the digital arts, and this might raise interest in cross-disciplinary study between computing and the digital arts.

Submit report to programassessment@montana.edu
Update Department program assessment report website.
Update PLO language in CIM if needed (Map PLOs to Course LOs)