Annual assessment:

Academic Year 2025–2026

College:

Norm Asbjornson College of Engineering

Department:

Gianforte School of Computing

Program:

B.A. in Computer Science

Department Head:

John Paxton

Submitted by:

Daniel DeFrance

> Assessment findings are drawn from the 2026 BA custom-exam and capstone-portfolio grading workbooks. Indicator scores are reported separately by reviewer. Portfolio scores are team-level assessments, not individual student scores.

1. Past Assessment Summary

The BA program assesses its learning outcomes annually using a custom examination and faculty-scored capstone portfolios. In AY 2024–2025, the program reported strengths in teamwork, design, and portfolio documentation. Its custom examination identified a need for stronger reasoning about time complexity, particularly best- and worst-case analysis, and more substantive descriptions of continuous learning.

The previous report proposed: redesigning BA capstone work to better connect with students' secondary fields; offering a custom-exam review session; clarifying the continuous-learning prompt; emphasizing evaluation and time complexity in CSCI 132, 232, and 246; and continuing to share successful portfolio examples. The sections below document which steps were taken and which remain under discussion.

The Assessment and Outcomes Committee's spring 2026 review commended the clarity of the program learning outcomes, the use of direct evidence and multiple reviewers, and the program's reflective use of results. The AOC recommended reviewing the 50% achievement benchmark, identifying all data sources in advance, making rubric indicators clearer, and considering alumni or advisory feedback. This report explicitly addresses those recommendations.

Program learning outcomes (PLOs):

  1. Analyze a complex computing problem and apply principles of computing and other relevant disciplines to identify solutions.
  2. Design, implement, and evaluate a computing-based solution to meet a given set of computing requirements in the context of the program’s discipline.
  3. Communicate effectively in a variety of professional contexts.
  4. Recognize professional responsibilities and make informed judgments in computing practice based on legal and ethical principles.
  5. Function effectively as a member or leader of a team engaged in activities appropriate to the program’s discipline.
  6. Apply computer science theory and software development fundamentals to produce computing-based solutions.

All six PLOs have been evaluated with the program's annual instruments. The prior report's schedule omitted an X for AY 2025–2026 even though assessments are conducted annually; the schedule is corrected below.

2. Actionable Research Question

To what extent do graduating BA students demonstrate the ability to evaluate computing solutions,particularly through time-complexity reasoning, while maintaining the program's strengths in design,teamwork, technical documentation, and implementation?

This question responds to the preceding cycle's findings and permits direct follow-up using the custom examination and capstone portfolio rubrics.

3. Assessment Plan, Schedule, and Data Sources

(a) Schedule change: No substantive change to annual assessment practice.

The previous report’s schedule omitted AY 2025–2026, although both instruments are used annually. The table below corrects that documentation discrepancy.

(b) Multiyear schedule

PLO 2024–25 2025–26 2026–27 2027–28 Direct data source
1 X X X X BA capstone portfolio: design pattern, UML, trade-offs
2 X X X X BA custom exam: data structure, logic, time complexity
3 X X X X Portfolio: technical writing; custom exam: continuous learning
4 X X X X Custom exam: algorithmic bias, impacts, harms
5 X X X X Portfolio: teamwork, lifecycle model
6 X X X X Portfolio: program and specifications

The 2026–2027 and 2027–2028 entries reflect the intended continuation of annual assessment practice, rather than completed assessments.

(c) Thresholds:

The prior plan uses a threshold of at least 50% of assessed students scoring 3 or 4 on a four-point rubric for each PLO. This benchmark is retained here to permit continuity with earlier reporting; the AOC recommended revisiting its rigor. In response to the AOC feedback, the program proposes raising the threshold to 60% beginning with the next assessment cycle. The existing 50% threshold is retained for evaluation of 2025–2026 results; the proposed change is not applied retroactively. Since the portfolio workbook scores three portfolio columns rather than explicitly identifying student-level individual results, indicator-level counts below are not represented as verified student-level PLO attainment.

PLO Existing threshold Assessment evidence
1–6 ≥50% scoring 3 or 4 on relevant 4-point rubric indicators See sources and indicator mappings above

Data sources identified at the outset:

(1) BA custom-exam grading workbook for Fall 2025/Spring 2026, with independent evaluations by John Paxton and Brendan Mumey; (2) 2026 BA ESOF 423 portfolio-grading worksheet, with evaluations by Binhai and Hunter. The prior report also discusses a graduating-senior survey and student self-ratings, but 2025–2026 survey data have not been provided or analyzed here and are not used to support this year's findings. Alumni and advisory board input likewise have not been provided.

4. What Was Done?

(a) Consistent with plan? Yes, with a schedule-documentation discrepancy.

Both annual direct-assessment instruments were used; the prior printed multiyear table did not reflect annual assessment in AY 2025–2026.

(b) Collection and analysis:

The BA custom-exam workbook contains five BA exams from Fall 2025 and Spring 2026, each scored on 12 questions by John Paxton and Brendan Mumey. The BA portfolio worksheet contains three scored portfolio columns for ESOF 423, each evaluated on seven indicators by Binhai and Hunter. The three portfolio columns should not be assumed to represent three individual students without confirmation of the portfolio unit and any associated team sizes. The analysis counts rubric scores of 3 or 4 separately for each evaluator; it does not average ratings to hide differences between reviewers.

  1. Rubric: The four-point scale is 1 Beginning, 2 Developing, 3 Competent, 4 Accomplished. The prior rubric describes portfolio indicators as: program/specification match, teamwork, design patterns, technical writing, UML, design trade-offs, and lifecycle model. It describes custom-exam indicators for data-structure selection, implementation logic, time complexity, continuous learning, algorithmic bias, impacts, and harms. These map to PLOs as indicated in Section 3. The prior assessment rubric maps custom-exam indicators 1–3 to PLO 2, indicator 4 to PLO 3, and indicators 5–7 to PLO 4. This mapping is retained. The 2026 workbook also contains Q8–Q12; their PLO assignments are not established by the prior rubric, so they are reported descriptively without assigning them to a PLO. Portfolio indicators follow the prior mapping: 3, 5, and 6 to PLO 1; 4 to PLO 3; 2 and 7 to PLO 5; and 1 to PLO 6. These are indicator-level and team-level results, not aggregated individual PLO scores.

5. What Was Learned?

Direct assessment: BA custom exam (five exams, two evaluators)

Question Paxton: 3–4 / 5 Mumey: 3–4 / 5 Interpretation
Q1 5 (100%) 5 (100%) Consistently strong
Q2 4 (80%) 3 (60%) Above 50% benchmark
Q3 3 (60%) 2 (40%) Reviewer disagreement; Mumey identified a weakness
Q4 3 (60%) 3 (60%) Above benchmark
Q5 4 (80%) 3 (60%) Above benchmark
Q6 5 (100%) 5 (100%) Consistently strong
Q7 5 (100%) 5 (100%) Consistently strong
Q8 5 (100%) 5 (100%) Consistently strong
Q9 3 (60%) 3 (60%) Above benchmark
Q10 5 (100%) 5 (100%) Consistently strong
Q11 5 (100%) 5 (100%) Consistently strong
Q12 3 (60%) 3 (60%) Above benchmark

For Q3, Paxton recorded three of five scores at 3 or 4, while Mumey recorded two of five. The workbook's own weakness annotations are 'None!' for Paxton and 'Q3' for Mumey. Under the existing 50% benchmark, Q3 meets the threshold under Paxton's ratings but not Mumey's. This is a meaningful follow-up to the previous year's concern about evaluating time complexity. With only five exams, each student's result changes the percentage by 20 points; the result should be interpreted cautiously.

Direct assessment: BA capstone portfolios (three columns, two evaluators)

Indicator Binhai: 3–4 / 3 Hunter: 3–4 / 3 Interpretation
Program/specifications 3 (100%) 3 (100%) Consistently strong
Teamwork 3 (100%) 3 (100%) Consistently strong
Design pattern 3 (100%) 3 (100%) Consistently strong
Technical writing 3 (100%) 3 (100%) Consistently strong
UML 2 (67%) 2 (67%) Lowest portfolio indicator; one portfolio scored 1 by both reviewers
Design trade-offs 3 (100%) 3 (100%) Consistently strong
Lifecycle model 3 (100%) 3 (100%) Consistently strong, with some reviewer variation in score level

The portfolio results support continued strengths in teamwork, implementation evidence, technical writing, and design reasoning. Both reviewers gave one UML portfolio a score of 1, making UML documentation a worthwhile area for targeted improvement despite its indicator-level rate exceeding the prior 50% benchmark.

Interpretation and limitations: These are indicator-specific direct-assessment findings. Portfolio counts represent teams, not students, and should not be converted into PLO-wide student attainment percentages. The results do not establish causal effects of the capstone redesign or exam preparation without comparable earlier evidence.

6. How We Responded

Faculty discussion.

We discussed the BA assessment findings at our August 21, 2026 annual faculty retreat. Time-complexity reasoning remained a concern. The instructors of CSCI 132, 232, and 246 were present for the discussion about reinforcing algorithm evaluation and complexity in those courses. This was a discussion of instructional priorities; we do not yet have evidence that specific course changes have been implemented.

Capstone redesign.

We redesigned CSCI 423 to make students' required secondary fields of study—minors or second majors—an integral part of their projects. With a smaller cohort, we could organize students into teams of approximately two. During the first week, students learned about each other's secondary fields and project interests so we could form complementary partnerships. Rather than assigning clients, as in previous semesters, we required students to identify and contact professionals in the relevant fields who had a genuine interest in the proposed software. We continued to share examples of successful portfolios to make expectations for technical documentation clear.

Fall 2026 is the second offering following the redesign. We have added more class-wide meetings so teams are better informed about each other's work. The new space in Gianforte Hall has also contributed to enthusiasm for the course. I invited a representative from Blackstone LaunchPad to discuss financial and legal support available to students interested in continuing their projects beyond the classroom. These Fall 2026 developments are subsequent improvements, not 2025–2026 assessment results.

Custom-exam preparation.

In Spring 2026, I introduced Canvas review materials, including videos and documents students could use asynchronously. The custom exam does not have a regular class meeting, so finding a common time for a live review session is impractical. I reposted the materials for Fall 2026. The exam is administered as a live, written examination; the Fall 2026 sitting is scheduled for October 14 at 6 p.m. in Barnard Hall 103. The Spring 2026 assessment still identified time-complexity reasoning as an area to improve.

AOC feedback and planned responses.

The AOC recommended reconsidering our 50% achievement threshold. We propose 60% for future cycles, while retaining 50% when evaluating the 2025–2026 results. We also present data sources before results, identify evaluators, and separate indicator scores for clarity. We will consider alumni or advisory feedback as an additional indirect source, but have not collected such evidence for this report. The 2026 portfolio results also suggest that clearer UML examples or checkpoints could be useful; this is a proposed response, not a change already implemented.

Resources.

The CIC partnership informs our interdisciplinary capstone and program planning. No specific new resource request has been established through this report.

7. Closing the Loop(s)

(a) Changes for upcoming assessments? Yes, proposed.

We propose increasing the achievement threshold from 50% to 60% beginning with the next cycle, retaining the same direct assessment instruments, and continuing to address time-complexity reasoning. We will also consider clearer UML guidance and more consistent reporting of assessment data sources. These are planned changes, not claims that faculty have formally approved a revised assessment policy.

(b) Follow-up on previous actions

Previous action What happened Evidence and next step
Redesign BA capstone around secondary fields Implemented: small teams, first-week partnership development, student-selected professional clients, projects tied to secondary fields 2026 portfolios were strong on most indicators; continue documenting how projects connect to students' other disciplines
Offer custom-exam review Implemented differently: asynchronous Canvas videos and documents were introduced for Spring 2026 and reposted for Fall 2026 Five BA exams were scored; time-complexity reasoning remains a concern
Add examples to the continuous-learning question No revision was needed; the existing question already contained examples of activities students could discuss Continue reviewing response quality; do not equate this item with PLO 5 (teamwork)
Emphasize algorithm evaluation and time complexity in CSCI 132, 232, 246 Discussed at the August 21, 2026 retreat with the relevant instructors present Specific instructional changes and their effects remain to be documented
Share successful portfolio examples Continued in CSCI 423 Strong technical-writing scores; no causal claim is made

(c) Evidence of change in learning.

The 2026 portfolios were strong on teamwork, technical writing, design patterns, and several other indicators, although UML documentation was weak in one portfolio. The exam results show continuing difficulty with time-complexity reasoning, with different conclusions from the two reviewers on Question 3. These results provide a baseline for follow-up, but do not establish that the capstone redesign or Canvas materials caused a change in learning. A five-exam sample also limits generalization.

(d) Seven-year program review and program planning.

We are discussing the branding of our computing degree options both at the August 21 retreat and with the Council of Independent Colleges. The label “Professional” for one BS option may unintentionally suggest that the BA and Interdisciplinary options are less professional. We are considering “CS+” as a shared identity for the BA and Interdisciplinary options, emphasizing the combination of computing with another field. CIC has identified similar language as successful elsewhere. No replacement name for the Professional option has been selected, and no formal naming changes have been approved. We will retain longitudinal assessment results, capstone project examples, and evidence of instructional changes for the seven-year review.

Other Comments

We appreciate the AOC's detailed feedback. In this report we identify all direct data sources before discussing findings, name the reviewers, and separate the indicators so that results are easier to interpret. We also propose increasing the threshold to 60% prospectively and will consider broader indirect evidence in future cycles.