Academic Years Assessed: 2020-2021/2021-2022
College: Engineering
Department: Gianforte School of Computing
Submitted by: John W. Sheppard

Undergraduate Assessment reports are to be submitted annually by program/s. The report deadline is October 15th.

Graduate Assessment reports are to be submitted annually by program/s. The report deadline is October 15th.

Program(s) Assessed:

List all majors (including each option), minors, and certificates that are included in this assessment:

  • PhD in Computer Science

☑ Have you reviewed the most recent Annual Program Assessment Reportsubmitted and Assessment and Outcomes Committee feedback? (please contactAssistant Provost Martha Peters if you need a copy of either one).


The Assessment Report should contain the following elements, which are outlined in this template:

  1. Assessment Plan, Schedule, and Sources
  2. What was done this assessment cycle – including rubrics, how data was collected, and who analyzed it
  3. What was learned – including areas of strength and areas for improvement
  4. How we responded
  5. Closing the loop

Sample reports and guidance can be found at: https://www.montana.edu/provost/assessment/program_assessment.html

1. Assessment Plan, Schedule and Data Source.

  1. Please provide a multi-year assessment schedule that will show when all programlearning outcomes will be assessed, and by what criteria (data). (You may use thetable provided, or you may delete and use a different format).
ASSESSMENT PLANNING CHART
PROGRAM LEARNING OUTCOME 2021-2022 2022-2023 2023-2024 2024-2025 Data Source*
Demonstrate technical expertise in an emphasis area. ✓ ✓ ✓ ✓ Course grades, qualifying examination, comprehensive examination, dissertation.
Effectively communicate research results to a scientific audience. ✓ ✓ ✓ ✓ Seminars associated with qualifier, comprehensive, and defense; presentations at technical conferences.
Independently perform quality original research. ✓ ✓ ✓ ✓ Publication of research results, publication and defense of dissertation.

*Data sources can be items such as randomly selected student essays or projects, specificallydesigned exam questions, student presentations or performances, or a final paper. Do not usecourse evaluations or surveys as primary sources for data collection.

  1. What are the threshold values for which you demonstrate student achievement?(Example provided in the table should be deleted before submission)
Threshold Values
PROGRAM LEARNING OUTCOME Threshold Value Data Source
1. Demonstrate technical expertise in an emphasis area. Students must maintain a 3.0 GPA throughout their degree program and must pass all three examinations to the satisfaction of their graduate committee. Course grades, qualifying examination, comprehensive examination, dissertation.
2. Effectively communicate research results to a scientific audience. 90% of students must pass the three examinations, each of which include formal presentations. Several courses also include project presentations. Conference presentations demonstrate oral communication. Acceptance of papers in conferences and journals indicate writing communication. Seminars associated with qualifier, comprehensive, and defense; presentations at technical conferences.
3. Independently perform quality original research. Three-to-five publications submitted and published by the time student completes degree requirements. Note that, in addition to the dissertation, each advisor usually sets their own standard for an expected number of papers submitted and published. Publication of research results, publication and defense of dissertation.

2. What Was Done

  1. Was the completed assessment consistent with the program’s assessment plan?

    ☑ Yes ☐ No

  2. If no, please explain.

  3. How were data collected and analyzed? (Please include method of collection andsample size).

  • GPA as reported in Degree Works has been used to approve students for graduation. Students need a 3.0 to graduate, which lets us know that our graduating students meet learning outcome 1.
  • Presentations and/or written report assignments are assigned in all our graduate level courses. They are a significant portion of the grade, so indirectly, students who pass these courses do well on these assignments. In addition, all students are required to make public presentations to the school as part of their qualifier, comprehensive, and dissertation defense examinations.
  • All doctoral students must submit a dissertation manuscript and defend it.
  1. Please provide a rubric that demonstrates how your data was evaluated. (Example provided below should be deleted before submission – your rubric may be verydifferent, it just needs to explain the criteria used for evaluating student achievement).
Component Expectations not met Meets Expectations Exceeds Expectations
Grade Point Average Student fails to maintain a 3.0 GPA over foundational courses and courses on the program of study Student maintains a 3.0 GPA over foundational courses and courses on the program of study Student maintains a 3.5 GPA over all courses on the program of study
Qualifying examination Student reviews five computer science research papers (both written and orally) but fails to adequately explain the technical problems, the mechanisms behind the technical solution, or the relevant open research questions. Student reviews five computer science research papers (both written and orally) and summarizes paper motivation, the technical problem, the technical solution, and any open research questions. Student reviews five computer science research papers (both written and orally) and clearly summarizes paper motivation, the technical problem, the technical solution, open research questions, the broader impact of the solution in computer science, or the broader society.
Comprehensive examination Student presents a research proposal (written and oral) but fails to motivate the significance of the research, the approach to completing the research, or any preliminary results demonstrating feasibility of the research. Student presents a research proposal (written and oral) and motivates the significance of the research and an approach to completing the research. Students also present preliminary results demonstrating feasibility of the research. Student presents a research proposal (written and oral) and motivates the significance of the research and an approach to completing the research. Students also present preliminary results, together with one or two publications, demonstrating feasibility of the research.
Dissertation defense Student fails to motivate the work, explain their technical contribution, demonstrate any novelty in the research, or communicate the results of their research to a technical but non-expert audience. Student motivates their work, explains their technical contribution, and evaluates its performance with data. The solution has some novelty. The student is also able to communicate the results of their research to a technical but non-expert audience. Student motivates their work, explains their technical contribution, and evaluates its performance with data. The solution is novel. The student is also able to communicate the results of their research, clearly, to a technical but non-expert audience, as evidence by insightful questions or comments from the audience.
Dissertation Student fails to motivate the work, explain their technical contribution, or demonstrate any novelty in the research. Student motivates their work, explains their technical contribution, and evaluates its performance with data. The solution has some novelty. Student motivates their work, explains their technical contribution, and evaluates its performance with data. The solution is novel.

This type of rubric can be used for all levels of assessment (the anticipated evaluation score may varyaccording to the course level). Some rubrics/assessments may be more tailored for courses (e.g.designed to assess outcomes in upper division courses or for lower division) and therefore the scoresmight be similar across course levels. Or, if you are assessing more basic learning outcomes, you mightexpect outcomes to be established earlier in the academic career.

NOTE: Student names must not be included in data collection. Dialog on successful completions, mannerof assessment (publications, thesis/dissertation, or qualifying exam) may be presented in table format ifthey apply to learning outcomes. In programs where numbers are very small and individual identificationcan be made, focus should be on programmatic improvements rather than student success. Data shouldbe collected through the year on an annual basis.

3. What Was Learned

The notification of students passing each of the milestones in the PhD program has been passed to the Graduate School each semester as the milestones are completed. The only exception is the qualifying examination, which is not tracked by the graduate school. Summary statistics for the past two years are listed here.

2020-2021 2021-2022
Qualifying examination 5 2
Comprehensive examination 0 8
Doctoral dissertation 2 3
Dissertation defense 2 3

Other statistics collected in this study are listed here by semester.

Fall 2020 Spring 2021 Fall 2021 Spring 2022
New PhD’s Admitted 6 1 7 1
Total PhD Students 26 24 23 23
Average Semester GPA 3.83 3.89 3.73 3.71
Students with Semester GPA < 3.0 0 0 0 0
Students with Semester GPA in [3.0,3.5) 1 0 1 2
Students with Semester GPA >= 3.5 16 13 11 9
Students with no Semester GPA (e.g., research only) 9 11 11 12
Average Cumulative GPA 3.72 3.74 3.73 3.71
Students with Cumulative GPA < 3.0 0 0 0 0
Students with Cumulative GPA in [3.0,3.5) 5 2 2 3
Students with Cumulative GPA >= 3.5 21 22 21 20
Total Number of Student Conference Publications 8 10 8 7
Total Number of Student Journal Publications 4 1 3 3
Total Number of Student Publications (refereed) 12 11 11 10
  1. Based on the analysis of the data, and compared to the threshold values established,what was learned from the assessment?
  • GPA assessment and milestone completion are accurate metrics for student abilities across a range of computer science areas.
  • Publishing appears to be somewhat lower than the previous report; however, this may be reflected in the reduced number of faculty working with PhD students, compared to two years ago (four faculty left during this time; new faculty are still building programs).
  1. What areas of strength in the program were identified?

  • Performance on coursework remains strong, as indicated by the high term and cumulative GPAs.
  • PhD productivity in artificial intelligence and machine learning remain strong.
  • PhD productivity in cybersecurity and software quality is growing.
  • Overall publication rates are strong, even with the above-identified reduction in raw numbers. Note that computer science tends to emphasize refereed conference publications over journal publications due to the rapid pace of change in the field. This emphasis is reflected in the publication statistics above.
  1. What areas were identified that need improvement?

  • Some new research initiatives were stifled due to the loss of faculty working in those areas. For example, this was evident in networking, software engineering, and cybersecurity with the recent loss of one networking faculty (Wittie), one software engineering faculty (Kanewala) and one security faculty (Peters).
  • With some students approaching the 10-year limit, time between milestones needs to be shortened to improve graduation rates, especially with respect to completing the qualifier and the time between qualifier and comprehensive.
  • There continues to be a need to improve interactions between students and their committees (either as a whole or with individual committee members). Currently, students meet with their committees once per year for annual evaluations. It might work better to meet at least once per semester to better inform and guide the research being done.
  • There continues to be a need to encourage students to share their research in public fora in the university, beyond lab meetings. This can be accomplished by encouraging interim presentations, dry runs for conference presentations, and using the school’s seminar series for milestone presentations.
  • We are starting to see more graduate students terminate their degrees early in favor of accepting industry positions. Recent trends to improve student stipends is likely to help, but it will still be difficult (if not impossible) to compete with industry salaries. Helping students to be more strategic in setting career paths is needed.
  • A process needs to be put in place to track specific publications by students to track the new threshold.

4. How we responded

  1. Describe how “What Was Learned” was communicated to the department, or programfaculty.
  • This document will be discussed in school meetings following submission.
  • Given updated thresholds, new statistics to be collected in future analyses will include total number of student publications at graduation by student.
  1. How are the results of this assessment informing changes to enhance studentlearning in the program?
  • As part of continuing discussions on improving the PhD program, results of this assessment as well as feedback from student and faculty experiences inform discussions during school meetings and retreats.
  • There will be a revisiting of the requirements for qualifier examinations and comprehensive examinations in an upcoming school meeting with an eye towards better assessment of student progress and readiness to move to the next milestone.
  1. If information outside of this assessment is informing programmatic change, pleasedescribe that.
  • We consider feedback on our PhD program from the school’s industry advisory board; however the advisory board tends to be more focused on undergraduates. Therefore, feedback tends to be sparse. Furthermore, coming out of the COVID pandemic, interaction with the advisory board was greatly reduced.

5. Closing the Loop

  1. In reviewing last year’s report, what changes proposed were implemented and will bemeasured in future assessment reports?
  • The last report mentioned that the assessment methods and thresholds should be communicated to students through the school’s website. This was not done; however, students have been made aware of the need for increasing publications and decreasing time to degree through advisor and committee meetings.
  • The last report mentioned that more regular meetings of students with their full committees should be encouraged beyond the one meeting required per year. This has been accomplished with some of the faculty labs, but it is has not been implemented as a school-wide practice. This continues to be a point of discussion among the faculty in school meetings and retreats.
  • The last report mentioned that students should be encouraged to present their research more often in public forums within the school. With the requirement that all milestones include public presentations, this is occurring to some extent. In addition, a few students have started to present their work in school seminars, and some labs (e.g., CompTaG and NISL) include regular presentations by students. Furthermore, grants have started to include seminar programs (e.g., the USDA DIFM grant) whereby students are able to present their work related to the grant to a focused research audience.
  1. Have you seen a change in student learning based on other program adjustmentsmade in the past? Please describe the adjustments made and subsequent changes instudent learning.
  • Based on the increased involvement in student presentations, we are seeing students as better prepared in communicating the results of their research. We are also finding better preparation leading into the key degree milestones.

Submit report to programassessment@montana.edu