Program Assessment Report: BA in Computer Science, 2023–2024
Academic Year(s) Assessed: AY 2023-2024
College: NACOE
Department: Gianforte School of Computing
Department Head: John Paxton
Submitted by: Dan DeFrance, John Paxton
Program(s) Assessed
List all majors (including each option), minors, and certificates that are included in this assessment – addor subtract rows as needed – please use official titles:
| Majors | Minors, Options, etc. |
|---|---|
| Computer Science BA | |
1. Past Assessment Summary.
Not applicable. This is our first report.
2. Action Research Question.
Do we need to make any adjustments to the assessment process that was described in the Year 0 plan that we submitted last year?
Yes – Due to the small number of Computer Science BA students (5 last year), we needed to evaluate each of their custom exams rather than a randomly selected subset. Had we followed the described project, only 2 custom exams would have been evaluated and that would not have given us enough information to feed into our assessment process.
3. Assessment Plan, Schedule, and Data Sources.
- Please provide a multi-year assessment schedule that will show when all program learning outcomes will be assessed, and by what criteria (data).
| PROGRAM LEARNING OUTCOME | 2024-2025 | 2025-2026 | 2026-2027 | 2027-2028 | Data Source* |
|---|---|---|---|---|---|
| 1 - Analyze a complex computing problem and apply principles of computing and other relevant disciplines to identify solutions. | x | x | Capstone Portfolio | ||
| 2 - Design, implement, and evaluate a computing-based solution to meet a given set of computing requirements in the context of the program’s discipline. | x | x | Custom Exam | ||
| 3 - Communicate effectively in a variety of professional contexts. | x | x | Capstone Portfolio, Custom Exam | ||
| 4 - Recognize professional responsibilities and make informed judgments in computing practice based on legal and ethical principles. | x | x | Custom Exam | ||
| 5 - Function effectively as a member or leader of a team engaged in activities appropriate to the program’s discipline. | x | x | Capstone Portfolio | ||
| 6 - Apply computer science theory and software development fundamentals to produce computing-based solutions. | x | x | Capstone Portfolio |
- What are the threshold values for which your program demonstrates student achievement?
| PROGRAM LEARNING OUTCOME | Threshold Value | Data Source |
|---|---|---|
| 1 - Analyze a complex computing problem and apply principles of computing and other relevant disciplines to identify solutions. | The threshold value for this outcome is for 50% of assessed students to score above 2 on a 1-4 scoring rubric. | Randomly selected student essays |
| 2 - Design, implement, and evaluate a computing-based solution to meet a given set of computing requirements in the context of the program’s discipline. | Same. | Same. |
| 3 - Communicate effectively in a variety of professional contexts. | Same. | Same. |
| 4 - Recognize professional responsibilities and make informed judgments in computing practice based on legal and ethical principles. | Same. | Same. |
| 5 - Function effectively as a member or leader of a team engaged in activities appropriate to the program’s discipline. | Same. | Same. |
| 6 - Apply computer science theory and software development fundamentals to produce computing-based solutions. | Same. | Same. |
4. What Was Done.
- Self-reporting Metric: Was the completed assessment consistent with the program’s assessment plan? If not, please explain the adjustments that were made.
Yes
- How were data collected and analyzed and by whom? Please include method of collection and sample size.
All students take the custom exam in their graduating semester. In AY 2024, 5 Computer Science B.A. students graduated and took the custom exam. All 5 exams were assessed by John Paxton. In the future, they will be assessed by at least 2 people. All Computer Science B.A. students take ESOF 423, where they work with a group to complete the capstone project. In AY 2024, there were 4 groups who completed ESOF 423. All 4 capstone portfolios were assessed by Hunter Lloyd and Binhai Zhu.
- Please provide a rubric that demonstrates how your data were evaluated. (Delete example below and replace with program’s assessment-specific rubric.)
| Indicators | Beginning - 1 | Developing- 2 | Competent- 3 | Accomplished- 4 |
|---|---|---|---|---|
| Capstone portfolio indicator 3 (for PLO 1) | No design pattern information in portfolio. | A design pattern was used, but wasn’t justified as the best approach. | A design pattern was used, but with incomplete justification. | A fully justified design pattern was used. |
| Capstone portfolio indicator 5 (for PLO 1) | No UML information in portfolio. | Diagrams and code don’t match. | Diagrams and code match, at most two types of UML diagrams used in the project. | Diagrams and code match, more than two types of UML diagrams used in the project. |
| Capstone portfolio indicator 6 (for PLO 1) | No design trade-off information in portfolio, or the example given is not explained as a design trade-off. | A design trade-off is described, but no justification is given. | A design trade-off is described, but the decision made was not justified correctly. | A design trade-off is described, with correct analysis. |
| Custom exam indicator 1 (for PLO 2) | Incorrect data structure. | Correct data structure, no implementations. | Correct data structure, one correct implementation. | Correct data structure, two correct implementations. |
| Custom exam indicator 2 (for PLO 2) | 3 or more logic errors in solution. | 2 logic errors in solution. | 1 logic error in solution. | No logic errors in solution. |
| Custom exam indicator 3 (for PLO 2) | Neither case has the correct time complexity. | One case has the correct time complexity but the wrong explanation. | One case is fully correct. | Both cases are fully correct. |
| Capstone portfolio indicator 4 (for PLO 3) | No technical documentation example in portfolio. | Documentation contained ten or more grammatical and/or spelling errors per page, or was poorly formatted. | Documentation had less than ten grammatical or spelling errors per page, but did not accurately describe the project. | Documentation fully described the project. |
| Custom exam indicator 4 (for PLO 3) | No involvement | Less than 25 hours | Less than 50 hours | 50 hours or more |
| Custom exam indicator 5 (for PLO 4) | Incorrect response. | Accurate description of algorithmic bias. | Accurate description of algorithmic bias plus partially correct example. | Accurate description of algorithmic bias plus correct example. |
| Custom exam indicator 6 (for PLO 4) | No answer. | One impact explained. | Two impacts explained. | Three impacts explained. |
| Custom exam indicator 7 (for PLO 4) | No answer. | One correct type of harm. | Two correct types of harm. | Three correct types of harm. |
| Capstone portfolio indicator 2 (for PLO 5) | No team project information in portfolio. | One or more team members did not affect the success of the project. | Some team members only completed a specific component of the project, without regard to the rest of the project. | Demonstrated genuine teamwork, where the team worked together to develop the project. |
| Capstone portfolio indicator 7 (for PLO 5) | No life cycle information in portfolio. | Development did not follow the life cycle described. | Development followed the life cycle model described. | Development followed the life cycle model described, and benefits and/or problems were described. |
| Capstone portfolio indicator 1 (for PLO 6) | No program in portfolio. | Program submitted with no, or incomplete, specifications. | Program did not meet specifications. | Specifications and a matching program both submitted. |
5. What Was Learned.
Based on the analysis of the data, and compared to the threshold values established, what waslearned from the assessment?
Item 1) Capstone Portfolio Assessment – Technical writing was identified as a weakness, and we learned that students need to focus effort on applying skills learned in their writing classes to the portfolios.
Item 2) Custom Exam – Indicator 3 showed that there is a weakness with subset of PLO 2 - Evaluating (as opposed to implementing or designing) a computing-based solution to meet a given set of computing requirements.
Item 3) Senior Survey (Indirect Evidence) – Although not related to our PLOs, we learned that Computer Science B.A. students found CSCI 366, ESOF 322 and ESOF 423 most valuable while finding CSCI 246 and CSCI 305 least valuable. We also learned that students would enjoy a 100-level course on industry tools and internships. Finally students mentioned that assignments outside of client projects in ESOF 423 take away from the capstone.
Item 4) ESOF 423 Instructor Observations based on course feedback and experience (Indirect Evidence): Students were able to interact effectively with the nonprofit organization HRDC to fill real software needs with quality solutions to a client organization.
What areas of strength in the program were identified from this assessment process?
- Capstone Portfolio Assessment – The only weakness of the capstone portfolios is that students did not provide effective technical write-ups. The semester-long projects turned out well and some will be used by the HRDC.
- Graduating Custom Exam – Except for not being able to evaluate a proposed solution for time complexity, Computer Science B.A. students performed well on the custom exam, showing that they are generally attaining our PLOs.
- Senior Survey – Students identified several valuable courses that include CSCI 366, ESOF 322 and ESOF 423.
- ESOF 423 Instructor Observations – Working with a non-profit, community focused organization like the HRDC was a positive experience for the students and the client.
What areas were identified that either need improvement or could be improved in a differentway from this assessment process?
- CSCI 132, CSCI 232 and CSCI 246 need to provide students with more experience using time complexity analysis to evaluate computing-based solutions.
- The Computer Science B.A. capstone course, ESOF 423, needs to show students examples of effective technical writing so that they better understand the type of technical writing that is required to be included with their capstone portfolios.
- ESOF 423 should consider requiring fewer assignments that are not directly related to the semester-long capstone project.
3) How We Responded.
- Describe how “What Was Learned” was communicated to the department, or program faculty. How did faculty discussions re-imagine new ways program assessment might contribute to program growth/improvement/innovation beyond the bare minimum of achieving program learning objectives through assessment activities conducted at the course level?
Results were shared and discussed during our annual faculty retreat on August 16, 2024. Ideas were brainstormed and shared that impact CSCI 132, CSCI 232, CSCI 246 and ESOF 423 (see above subsection).
- How are the results of this assessment informing changes to enhance student learning in the program?
The B.A. capstone class, ESOF 423, will share examples of appropriate technical writing that past students have submitted. In ESOF 423, more focus will be directed towards the client projects, and less on outside assignments with learning objectives. CSCI 132, CSCI 232 and CSCI 246 instructors know that time complexity is a difficult concept and will consider their pedagogy with respect to this concept.
- If information outside of this assessment is informing programmatic change, please describe that.
Not applicable.
- What support and resources (e.g. workshops, training, etc.) might you need to make these adjustments?
Not applicable.
7. Closing the Loop(s).
Reflect on the program learning outcomes, how they were assessed in the previous cycle (refer to #1 of the report), and what was learned in this cycle. What action will be taken to improve student learning objectives going forward?
- Self-Reporting Metric (required answer): Based on the findings and/or faculty input, will there be any curricular or assessment changes (such as plans for measurable improvements, or realignment of learning outcomes)?
No. This is our first report for the Computer Science B.A. and we need to use our process more than once to see what is working and what is not.
- In reviewing the last report that assessed the PLO(s) in this assessment cycle, what changes proposed were implemented and will be measured in future assessment reports?
Not applicable – this is our first report, last year we submitted the Year 0 planning document
- Have you seen a change in student learning based on other program adjustments made in the past? Please describe the adjustments made and subsequent changes in student learning.
Not applicable – this is our first report
