Annual Undergraduate Report: 2025-2026

Academic Year Assessed: 2025-2026
College: Norm Asbjornson College of Engineering
Department: Gianforte School of Computing
Department Head: John Paxton
Submitted by: Neda Nazemi and Ann Marie Reinhold

PROGRAM(S) ASSESSED

Majors Minors, Options, etc.
Bachelor of Science in Data Science Minor and B.S.

Section 1. Past Assessment Summary.

Summarize findings from last time this PLO(s) was assessed. Include findings from the last assessment of this PLO that influenced this cycle’s assessment approach. (Note: this is NOT a summary of what was done during last year’s assessment cycle unless you are assessing the same PLOs again this year; this is a summary of the last time you assessed the PLO(s) that you are re-assessing this cycle. You will connect the summary to the reflection portion in Section 7: Closing the Loop. Table has been provided to support the intention of this question.)

This is the first direct assessment cycle for the Bachelor of Science in Data Science. Therefore, there are no prior PLO results against which to compare student achievement. The Year 0 plan established the PLOs, data source, analytic rubric, and desired performance level. This report provides the baseline for future comparisons and identifies refinements needed in both student learning and the assessment process.

Which PLOs are being assessed? Previous assessment Prior action items Status in this cycle
PLO 1: Explain and integrate the data science lifecycle.
PLO 2: Design, implement, test, and document computational solutions.
PLO 3: Design and interpret accurate, accessible, audience appropriate visualizations.
PLO 4: Select, apply, justify, and evaluate mathematical and statistical methods.
Not previously assessed. The 2025-2026 Year 0 report established the first assessment plan for this new program. No prior learning findings or improvement actions existed. The Year 0 plan called for collection of capstone artifacts and use of a program level analytic rubric. Two spring 2026 capstone project portfolios were collected and assessed. The available evidence represents seven students and all four PLOs.

Section 2. Actionable Research Question for Your Assessment.

Primary research question: To what extent do the BA Data Science students demonstrate the four program learning outcomes through integrated, authentic project work?

Diagnostic question: Across the rubric dimensions, which capabilities are consistently demonstrated, and which need stronger curricular or assignment level support?

Section 3. Assessment Plan, Schedule, and Data Sources.

  1. Did you change the previously established Assessment Plan Schedule. If yes, how was it changed?

The Year 0 plan originally scheduled assessment biennially, but the schedule was revised to require annual assessment.
Additionally, the learning outcomes were modified to address the feedback received on year 0 plan from AOC.

  1. Please provide a multi-year assessment schedule that will show when all program learning outcomes will be assessed, and by what criteria (data). List all of your PLOs in full for reference -not just the ones being assessed in this cycle. Add rows as necessary.

ASSESSMENT PLANNING SCHEDULE CHART

Program Learning Outcomes Course Alignments: Include rubric, number, and course title Identification of Assessment Artifact Year to be assessed Year to be assessed Year to be assessed
PLO 1 – Explain and integrate the data science lifecycle. CSCI 252 Intro to Data Science
CSCI 347 Data Mining
CSCI 487 Data Science in Practice
Capstone portfolio indicators 1 & 2 2025-2026
Assessed
2026-2027
Planned
2027-2028
Planned
PLO 2 – Design, implement, test, and document computational solutions. CSCI 127 Joy & Beauty of Data
CSCI 232 Data Structures and Algorithms
CSCI 347 Data Mining
Capstone portfolio indicators 3 2025-2026
Assessed
2026-2027
Planned
2027-2028
Planned
PLO 3 – Design and interpret accurate, accessible, audience appropriate visualizations. CSCI 444 Data Visualization
CSCI 487 Data Science in Practice
CSCI 252 Intro to Data Science
Capstone portfolio indicators 4 2025-2026
Assessed
2026-2027
Planned
2027-2028
Planned
PLO 4 – Select, apply, justify, and evaluate mathematical and statistical methods. STAT 337 Intermediate Statistics with Introduction to Statistical Computing
M 441 Numerical Linear Algebra and Optimization
CSCI 347 Data Mining
Capstone portfolio indicators 5 and 6 2025-2026
Assessed
2026-2027
Planned
2027-2028
Planned
  1. What are the threshold values for which your program demonstrates student achievement? Provide a rationale for your threshold values.

For each rubric criterion, the desired performance level is met when more than 50% of assessed students earn a score of 3 (Competent) or 4 (Accomplished) on the four-level program rubric. Because each submitted artifact was jointly produced, every student on a team was assigned the team artifact score for aggregate reporting. A PLO with more than one criterion is considered achieved only when each associated criterion meets the desired performance level. This threshold is retained from the Year 0 plan and is appropriate for an initial baseline; its rigor will be reviewed after additional cohorts are available.

Threshold Values
Program learning outcome Threshold value Data source
PLO 1: Data science lifecycle More than 50% score 3 or 4 on each associated criterion. Two capstone portfolios: technical documentation, poster, and presentation
PLO 2: Computational solutions More than 50% score 3 or 4. Technical documentation and presentation
PLO 3: Data visualization More than 50% score 3 or 4. Poster, presentation, and dashboard visualizations
PLO 4: Mathematical and statistical methods More than 50% score 3 or 4 on each associated criterion. Technical documentation, reported analyses, and presentation

Section 4 What Was Done?

  1. Self-reporting Metric (required answer):

    Was the completed assessment consistent with the program’s assessment plan? If not, please explain the adjustments that were made.

Yes [ X] No [] Explanation provided below.

  1. How was the data collected and analyzed and by whom? Please include method of collection and sample size.

The spring 2026 capstone instructor (Dr. Reinhold) collected artifacts for two projects.

Project 1, Unified Schema Intelligence Engine, included a poster, an 18-slide presentation, and 24 pages of technical documentation.

Project 2, FACT-D Healthcare Data Quality Dashboard, included a poster, a 32-slide presentation, and 4 pages of technical transfer documentation.

The projects included three and four students, respectively, for an aggregate sample of seven students.

Each portfolio was reviewed by Dr. Nazemi and Dr. Reinhold against six analytic criteria aligned with the four PLOs. Only evidence present in the submitted artifacts was scored. A team score was applied to each team member solely to calculate the student level percentage specified in the Year 0 threshold. Scores of 3 or 4 were classified as meeting the desired performance level.

Please note that this is the program’s first assessment plan. In the next assessment cycle, we will carefully evaluatewhether the capstone portfolios provide sufficient evidence to assess all learning outcomes effectively. For anyoutcomes that are difficult to assess using these portfolios, we will identify appropriate artifacts from other courses forfuture evaluations, as recommended by the AOC.

  1. Please provide a rubric for each PLO being assessed that demonstrates how your data was evaluated. (See Instructions document for an example of what this could look like.)

The program rubric below operationalizes the Year 0 criteria. Indicators were evaluated using a four-level scale, with a score of 3 indicating competent achievement and a score of 4 indicating accomplished achievement. A more detailed rubric is provided in Appendix A. This rubric is used specifically for program assessment; the instructor uses a separate rubric tograde the portfolios.

Criterion 1 Beginning 2 Developing 3 Competent 4 Accomplished
Pipeline implementation (PLO 1) No pipeline or code evidence Incomplete or not reproducible Functional, with some transparency or reproducibility limitations Complete, reproducible, clearly documented pipeline
Data lifecycle and ethics (PLO 1) No lifecycle or ethics discussion Superficial mention without justification Most lifecycle stages addressed, but limited depth Comprehensive lifecycle, reproducibility, and ethics discussion
Computational design (PLO 2) Unclear or disconnected methods Partly logical, with limited rationale or detail Logical methods linked to objectives Comprehensive, well justified design aligned with objectives
Visualization (PLO 3) No visualization Poor, misleading, or unclear Accurate and interpretable, with some limits in depth, accessibility, or design Elegant, accurate, accessible communication for technical and nontechnical audiences
Modeling (PLO 4) No model Misapplied or unjustified model Appropriate model with partial evaluation or limited reasoning Well justified model with rigorous evaluation and interpretation
Mathematical and statistical reasoning (PLO 4) No quantitative reasoning Basic analysis without conceptual understanding Correct but limited application and interpretation Comprehensive application with clear understanding and interpretation

Section 5. What Was Learned.

  1. Based on the analysis of the data, and compared to the threshold values established, what was learned from the assessment? (Relate this to your research question in Section 3.)

Four of the four PLOs met the desired performance level. PLO 2, PLO 3, and PLO 4 each had 100% of assessed students represented by artifacts scoring at Level 3 or higher on every associated criterion.

For PLO 1, even though neither team explicitly addressed broader data lifecycle and reproducibility, both teams demonstrated strong reproducible pipeline implementation.

The first project documented a reproducible hybrid classification workflow using a fine-tuned LLaMA model, engineered features, principal component analysis, kernel density estimation, and ensemble probabilities. It clearly linked implementation to the schema mapping problem. Its model evaluation evidence was incomplete: the artifacts reported average confidence of approximately 65% and defined accuracy, precision, and F1 metrics in code, but did not present held out metric values, uncertainty analysis, or a comparative validation of the ensemble.

The second project documented an operational dashboard and multiple health rules for assessing issues in the customer’s overarching data pipelines; they developed a series of rules for assessing critical metrics related to data and it’s processing. The final metrics that they assessed were co-developed with the customer and included freshness, accuracy, completeness, transformation, and duration—hence, they coined these metrics FACT-D. The artifacts described architecture, file organization, execution order, rule logic, statistical thresholds, dashboard interpretation, and future improvements. The quantitative rules were appropriate for a capstone implementation, but the supplied evidence did not validate alert thresholds, quantify false alarms, or evaluate dashboard accessibility and usability. The latter issue was an emergent property of the students not being able to access “real” healthcare data; the students were forced to work with synthetic data due to ethical concerns and privacy and confidentiality constraints.

Importantly, this class is designed around solving a “non-trivial data-science problem”. The students did just this—fully tackling their data-science problems by developing solutions that enabled them to achieve all 4 learning outcomes. They did not surmount every challenge that they faced, but they met all learning outcomes.

Because the portfolio is the unit of evidence, percentages below indicate the number of students represented by qualifying team artifacts. They should not be interpreted as independent individual scores.

  1. What areas of strength in the program were identified from this assessment process?

Students consistently translated authentic healthcare data engineering problems into functional computational systems. Both teams provided substantial technical documentation, made their system architecture understandable, and produced artifacts intended for transfer to an external partner. The projects also demonstrated appropriate use of statistical and computational methods and communicated limitations and future work rather than presenting the systems as complete production solutions.

Criterion PLO Project 1 Project 2 Students at Level 3 or 4 Threshold
Pipeline implementation PLO 1 4 4 7 of 7 (100%) Met
Data lifecycle and ethics PLO 1 3 3 7 of 7 (100%) Met
Computational design PLO 2 4 3 7 of 7 (100%) Met
Visualization PLO 3 3 3 7 of 7 (100%) Met
Modeling PLO 4 3 3 7 of 7 (100%) Met
Mathematical and statistical reasoning PLO 4 3 3 7 of 7 (100%) Met
  1. What areas were identified that either need improvement or could be improved in a different way from this assessment process?

This assessment identified no immediate need for curricular changes. All four PLOs exceeded the established achievement thresholds for the academic year assessed. The program will continue to monitor student performance and consider whether artifacts from additional courses could provide a more comprehensive assessment of the PLOs, including evidence of individual achievement alongside team capstone work.

Section 6. How We Responded.

  1. Describe how “What Was Learned” was communicated to the department, or program faculty. How did faculty discussions re-imagine new ways program assessment might contribute to program growth/improvement/innovation beyond the bare minimum of achieving program learning objectives through assessment activities conducted at the course level?

This report is shared for discussion by the Computer Science program faculty. Consistent with the Year 0 communication plan, the findings should be distributed and discussed by the program faculty. The feedback from program has beenincorporated before the report is submitted and shared with the program advisory council [TBD].

  1. How are the results of this assessment informing changes to enhance student learning in the program?

No curricular changes are planned based on the assessment results, as all PLOs exceeded the established achievement thresholds. However, the program will consider whether artifacts from additional courses could provide a more comprehensive assessment of the PLOs.

  1. If information outside of this assessment is informing programmatic changes, please describe that.

No additional evidence was supplied for this report.

  1. What support and resources (e.g., workshops, training, etc.) might you need to make these adjustments?

N/A

Section 7. Closing the Loop(s).

Closing the Loop refers to continuous improvement and planning for future action. Reflect on the program learning outcomes and how they were assessed in the previous cycle (refer to #1 of the report). What was learned in this cycle and what actions will the program take in planning for future assessment of the PLO(s).

  1. Self-Reporting Metric (required answer): Based on the findings and/or faculty input, will there be any changes made (such as plans for measurable improvements, realignment of learning outcomes, curricular changes, etc.) in preparation for upcoming assessments?

Yes [X] No [ X]

  1. In reviewing the last report that assessed the PLO(s) in this assessment cycle, what changes proposed were implemented and will be measured in future assessment reports? What action will be taken to improve student learning objectives going forward?

The previous report established the Year 0 assessment plan rather than evaluating student achievement. Its principal proposed action, collecting and evaluating capstone artifacts, was implemented. Going forward, the program will retain capstone portfolios as a central component of assessment while considering whether artifacts from other courses could provide complementary evidence to assess PLO 1 more fully.

  1. Have you seen a change in student learning based on other program adjustments made in the past? Please describe the adjustments made and subsequent changes in student learning.

Not yet. This is the first direct assessment and establishes the baseline. No causal claim about changes in student learning can be made until a later cohort is assessed using the same or a documented comparable measure after the planned improvements are implemented.

  1. If the program sees anything emerging from this assessment cycle that it anticipates would be a factor or an item of discussion in its 7-year program review cycle, please use this space to document that for future reference.

For the seven-year program review, the program should monitor whether capstone work alone provides sufficient coverage of all PLOs and sufficient evidence of individual achievement.

OTHER COMMENTS

Use this area to documented anything not covered in previous sections that department wants to remember for future reference.

Interpretation caveat: the cohort is small, the projects were completed in teams, and both projects involved the same external partner and healthcare data context. These features make the work authentic but limit generalizability across domains and prevent estimation of individual performance. Future reports should preserve team level results while adding individual evidence and, when possible, artifacts from a broader range of project contexts.

Appendix A- Data Science Portfolio Criteria and Evaluation Rubric

Section I. Grading Rubrics

Indicator 1: Data Science Pipeline Implementation

Include link to code repository (GitHub), data processing scripts, and documentation of the full data science pipeline (acquisition → cleaning → analysis → visualization → modeling → communication).

Evaluation:

  1. No pipeline documentation or code submitted.
  2. Incomplete or non-reproducible pipeline; missing steps or unclear documentation.
  3. Pipeline functional but lacks transparency or reproducibility (e.g., undocumented cleaning or transformations).
  4. Full, reproducible pipeline clearly documented and implemented

Indicator 2 – Integration of Data Lifecycle Concepts

Explain how you addressed data ethics, storage, documentation, and management practices throughout your project. Include any steps you took to make your data findable, accessible, interoperable, and reusable.

Evaluation:

  1. No discussion of data lifecycle or ethics.
  2. Mentions steps superficially without justification.
  3. Addresses most lifecycle stages but lacks depth.
  4. Comprehensive discussion of lifecycle, reproducibility, and ethical considerations.

Indicator 3 – Computational Design and Implementation

Describe how your team designed and implemented the computational aspects of your project. Explain the overall structure, logic, and rationale behind your analysis methods, including how your approach addresses the project’s objectives.

Evaluation:

  1. Analysis methods are unclear, unjustified, or disconnected from project goals.
  2. Analysis methods are partially logical or justified but lack comprehensive explanation or implementation detail.
  3. Analysis methods are logical and mostly justified, with a clear link between rationale and implementation.
  4. Analysis methods are fully logical, well-justified, comprehensive, and clearly aligned with project objectives, demonstrating sound computational design.

Indicator 4 – Visualization Design and Interpretation

Include at least two key figures from your project that best represent your work. For each figure, provide a short interpretation explaining what it shows, why it was chosen, and how it communicates the complexity of the problem, and your key findings.

Evaluation:

  1. No visualization included.
  1. Visuals poorly designed or misleading; unclear labeling or interpretation.
  2. Visuals convey information but lack depth or design quality.
  3. Visuals are elegant, accurate, and effectively communicate key insights to technical and non-technical audiences.

Indicator 5 – Statistical and Computational Modeling

Describe the model(s) you used, why you selected them, and how you evaluated their performance. Explain how your modeling supported your project’s goals.

Evaluation:

  1. No model described.
  2. Model misapplied or unjustified.
  3. Appropriate model with partial evaluation or limited reasoning.
  4. Well-justified model with rigorous evaluation and clear interpretation.

Indicator 6 – Mathematical and Statistical Reasoning

Explain how you used mathematical or statistical reasoning to understand relationships in your data (e.g., hypothesis testing, correlations, linear algebraic transformations, uncertainty analysis).

Evaluation:

  1. No quantitative reasoning described.
  2. Basic analysis without conceptual understanding.
  3. Correct but limited application of statistical or mathematical methods.
  4. Comprehensive application with clear understanding and interpretation.

Section II. Desired Performance Level

The desired performance level on each indicator is achieved if more than 50% of students earn a score of 3 or higher.

Section III. Evaluation Methodology

Two assessment committee members will independently review each portfolio using this rubric. Indicators not meeting the desired performance level will be identified. If significant discrepancies occur between graders, a third reviewer will arbitrate.