Academic Year(s) Assessed: 2022-2023, 2023-2024

College: NACOE

Department: Gianforte School of Computing

Department Head: John Paxton

Submitted by: Binhai Zhu

Program(s) Assessed

List all majors (including each option), minors, and certificates that are included in this assessment – addor subtract rows as needed – please use official titles:

Majors Minors, Options, etc.
PhD in Computer Science

1. Past Assessment Summary.

The main findings of our previous report (AY 2021 / AY 2022) were

  • Student milestone completion and GPA were positive indicators.
  • Total number of student publications dropped in comparison to previous report; possibly due to the departure of four tenure-track faculty who departed during COVID years.

2. Action Research Question.

Are students performing quality original research (as indicated by refereed publications)?

3. Assessment Plan, Schedule, and Data Sources.

  1. Please provide a multi-year assessment schedule that will show when all program learning outcomes will be assessed, and by what criteria (data).
ASSESSMENT PLANNING SCHEDULE CHART
PROGRAM LEARNING OUTCOME 2024-2025 2025-2026 2026-2027 2027-2028 Data Source*
Demonstrate technical expertise in an emphasis area. X X X X Course grades, Pass rate on qualifying exam, comprehensive exam and dissertation defense.
Effectively communicate research results to a scientific audience. X X X X Accepted papers at technical conferences.
Independently perform quality original research. X X X X Publication of research results, publication and defense of dissertation.
  1. What are the threshold values for which your program demonstrates student achievement?
Threshold Values
PROGRAM LEARNING OUTCOME Threshold Value Data Source
(1) Demonstrate technical expertise in an emphasis area. 90% of students maintain a 3.0 GPA throughout their degree program. 75% of students pass the following exams when they take them: qualifying exam, comprehensive exam, dissertation defense Course grades, Pass rate on qualifying exam, comprehensive exam and dissertation defense.
(2) Effectively communicate research results to a scientific audience. 50% of students will author or co-author a refereed conference or journal paper that is accepted during the 2-year assessment period. Accepted papers at technical conferences.
(3) Independently perform quality original research. 90% of students will have at least three conference or journal publications at the time they graduate. 50% of students will have at least 5 publications at the time they graduate. Accepted papers at technical conferences.

4. What Was Done.

  1. Self-reporting Metric (required answer): Was the completed assessment consistent with the program’s assessment plan? If not, please explain the adjustments that were made.

YES – Although specific percentages were added to the threshold values based on feedback from the previous assessment report and clarifications were made to the data sources.

  1. How were data collected and analyzed and by whom? Please include method of collection and sample size.
  • Our administrative associate collected GPA data for all Ph.D. students during the assessed period. This same person summarized information for all Ph.D. students who took the qualifying exam, the comprehensive exam or the dissertation defense.
  • Dr. Zhu, our Ph.D. coordinator, collected publication data for all Ph.D. students during the assessed period from GSoC faculty.
  • Dr. Zhu analyzed the data.
  • The number of Ph.D. student ranged from 18 – 20.
  1. Please provide a rubric that demonstrates how your data were evaluated. (Delete example below and replace with program’s assessment-specific rubric.)
Indicators Expectations not met Meets Expectations Exceeds Expectations
Grade Point Average Student fails to maintain a 3.0 GPA over foundational courses and courses on the program of study Student maintains a 3.0 GPA over foundational courses and courses on the program of study Student maintains a 3.5 GPA over all courses on the program of study
Qualifying examination Student reviews five computer science research papers (both written and orally) but fails to adequately explain the technical problems, the mechanisms behind the technical solution, or the relevant open research questions. Student reviews five computer science research papers (both written and orally) and summarizes paper motivation, the technical problem, the technical solution, and any open research questions. Student reviews five computer science research papers (both written and orally) and clearly summarizes paper motivation, the technical problem, the technical solution, open research questions, the broader impact of the solution in computer science, or the broader society.
Comprehensive examination Student presents a research proposal (written and oral) but fails to motivate the significance of the research, the approach to completing the research, or any preliminary results demonstrating feasibility of the research. Student presents a research proposal (written and oral) and motivates the significance of the research and an approach to completing the research. Students also present preliminary results demonstrating feasibility of the research. Student presents a research proposal (written and oral) and motivates the significance of the research and an approach to completing the research. Students also present preliminary results, together with one or two publications, demonstrating feasibility of the research.
Dissertation defense Student fails to motivate the work, explain their technical contribution, demonstrate any novelty in the research, or communicate the results of their research to a technical but non-expert audience. Student motivates their work, explains their technical contribution, and evaluates its performance with data. The solution has some novelty. The student is also able to communicate the results of their research to a technical but nonexpert audience. Student motivates their work, explains their technical contribution, and evaluates its performance with data. The solution is novel. The student is also able to communicate the results of their research, clearly, to a technical but non-expert audience, as evidence by insightful questions or comments from the audience.
Dissertation Student fails to motivate the work, explain their technical contribution, or demonstrate any novelty in the research. Student motivates their work, explains their technical contribution, and evaluates its performance with data. The solution has some novelty. Student motivates their work, explains their technical contribution, and evaluates its performance with data. The solution is novel.

5. What Was Learned.

The summary of statistics for the past two years are listed in the following table.

2022-2023 2023-2024
Qualifying Examination 1 1
Comprehensive Examination 4 1
Doctoral Dissertation 1 6
Dissertation Defense 1 6

Some other statistics collected in this study are listed below by semester.

Fall 2022 Spring 2023 Fall 2023 Spring 2024
New PhDs Admitted 3 1 5 1
Total PhD Students 19 18 19 20
Average Semester GPA 3.65 3.55 3.64 3.79
Students with Semester GPA <3.0 0 1 1 0
Students with Semester GPA in 3.0-3.5 2 2 1 2
Students with Semester GPA >= 3.5 5 5 9 10
Students with no Semester GPA 12 10 8 8
Average Cumulative GPA 3.69 3.67 3.62 3.46
Students with Cumulative GPA < 3.0 0 0 1 0
Students with Cumulative GPA in 3.0-3.5 3 3 3 6
Students with Cumulative GPA >= 3.5 16 15 15 14
Total Number of Student Conference Publications 15 13 14 10
Total Number of Student Journal Publications 6 7 7 6
Total Number of Student Publications (refereed) 21 20 21 16
  1. Based on the analysis of the data, and compared to the threshold values established, what was learned from the assessment?
  • The total number of PhD students (19, 18, 19, 20) is slightly lower than in the previous report (26, 24, 23, 23). There were some faculty departures, and it takes time for newly hired faculty to recruit PhD students and build research programs.
  • GPA assessment and milestone completion are accurate metrics for student abilities across a range of computer science areas.
  • PhD publications in this report (21, 20, 21, 16) are significantly higher than in the previous report (12, 11, 11, 10), representing an improvement of 60% to 90%. With three recent hires and a search in progress, these numbers might continue to grow.
  1. What areas of strength in the program were identified from this assessment process?
  • Performance on coursework remains strong, as indicated by the high term and cumulative GPAs.
  • PhD productivity in artificial intelligence and machine learning remain strong.
  • PhD productivity in cybersecurity and software quality is growing.
  • PhD productivity in algorithms remains stable at the rate of one per year.
  • Overall publication rates are strong. Note that computer science tends to emphasize refereed conference publications over journal publications due to the rapid pace of change in the field. This emphasis is reflected in the publication statistics above.
  1. What areas were identified that either need improvement or could be improved in a different way from this assessment process?
  • Some new research initiatives were stifled due to the loss of faculty working in those areas. For example, we no longer have research expertise in systems or networks.
  • There is a need to encourage students to share their research in public forums in the university, beyond lab meetings. This can be accomplished by encouraging interim presentations, dry runs for conference presentations, and using the school’s seminar series for milestone presentations.
  • Some graduate students terminate their degrees early in favor of accepting industry positions. More time spent with career planning might be helpful.
  • We need to initiate a process to better track individual publications.

6. How We Responded.

  1. Describe how “What Was Learned” was communicated to the department, or program faculty. How did faculty discussions re-imagine new ways program assessment might contribute to program growth/improvement/innovation beyond the bare minimum of achieving program learning objectives through assessment activities conducted at the course level?
  • This document was discussed at a faculty meeting on October 10th, 2024. The feedback was then incorporated into the report.
  • Given the refined threshold values, we will adjust our data collection process during the next two years to track the number of student publications per student.
  1. How are the results of this assessment informing changes to enhance student learning in the program?
  • As part of continuing discussions on improving the PhD program, results of this assessment as well as feedback from student and faculty experiences inform discussions during school meetings and retreats.
  • We will revisit requirements for qualifying examinations and comprehensive examinations with the goal of better assessing student progress and readiness for the next milestone.
  1. If information outside of this assessment is informing programmatic change, please describe that.
  • The school’s industry advisory board provides periodic high-level feedback.
  • It might be helpful to form a PhD program advisory committee that includes external members. The committee could help us refine our publication thresholds and standards.
  1. What support and resources (e.g. workshops, training, etc.) might you need to make these adjustments?
  • If we form a PhD advisory committee, having resources to invite an external person to campus for a workshop could be helpful.

7. Closing the Loop(s).

Reflect on the program learning outcomes, how they were assessed in the previous cycle (refer to #1 of the report), and what was learned in this cycle. What action will be taken to improve student learning objectives going forward?

  1. Self-Reporting Metric (required answer): Based on the findings and/or faculty input, will there be any curricular or assessment changes (such as plans for measurable improvements, or realignment of learning outcomes)?

NO – However, since we have added more specific thresholds, we will collect slightly different information in the coming two years and then adjust our indicators in section 4.

  1. In reviewing the last report that assessed the PLO(s) in this assessment cycle, what changes proposed were implemented and will be measured in future assessment reports?
  • Students have been informed about publication expectations.
  • The last report recommended that students consider meeting with their committee more than once per year as this could potentially decrease the time to degree completion. This has been accomplished with some of the faculty labs but has not been implemented as a school-wide practice. We continue to discuss the pros and cons of requiring students to meet with their committees more than once per year.
  • The last report mentioned that students should be encouraged to present their research more often in public forums. Given that all milestones include a public presentation, this does occur to some extent. In addition, a few students present their work in school-wide seminars, and many of the labs (e.g., Applied Algorithms, CompTaG and NISL) include regular presentations by students. Furthermore, grants have started to include seminar programs (e.g., the USDA DIFM grant) whereby students are able to present their work related to the grant to a focused research audience.
  1. Have you seen a change in student learning based on other program adjustments made in the past? Please describe the adjustments made and subsequent changes in student learning.
  • Based on the increased number of student presentations, students seem better prepared to communicate the results of their research.
  • More students are prepared to achieve key degree milestones.