Year 0 Assessment Plan: Artificial Intelligence Certificate, 2024–2025
Academic Year of Year 0 Plan: 2024-2025
College: Norm Asbjornson College of Engineering
Department: Gianforte School of Computing
Submitted by: Dr. John W. Sheppard, Program Coordinator, AI
Certificate
Date of Submission: October 15, 2024
Program(s) to be Assessed.
List all majors, minors, certificates and/or options that are included in this new Assessment Plan
| Majors/Minors/Certificate | Options |
|---|---|
| Artificial Intelligence Certificate | N/A |
Is this a new program? Yes_X__ No___
Are you keeping existing outcomes? Yes _X__ No___
If no, please identify all that apply:
Consolidating PLOs ____
Rewriting PLOs to be more assessable ____
Rewriting PLOs to be more aligned with program objectives ____
Other:
Part 1: Program Learning Outcomes (PLOs).
List the Program Learning Outcomes.
| PLO# | PLO Description |
|---|---|
| 1 | Formally define the breadth and depth of AI, and the potential impacts of AI on society. |
| 2 | Assess the strengths and weaknesses of current and emerging methods and technologies in AI. |
| 3 | Critically analyze and assess the suitability of AI methods to current application areas. |
| 4 | Determine and evaluate the behavior of AI methods on real-world problems. |
| 5 | Apply methods in AI to problems not normally solvable by traditional means. |
| 6 | |
| 7 |
Part 2: Development of Assessment Plan.
-
Discuss your threshold values and how you will determine them for your courses and PLOs.
Threshold Values.
PLO-1: At least 75% of students will pass CSCI 446 with a grade of B or better. At least 33% of students will pass CSCI 446 with a grade of A or better. This course is focused on defining the depth and breadth of AI as a field.
PLO-2: At least 75% of students will have passed one of CSCI 546 or CSCI 547, and at least 33% of students will have passed both CSCI 546 and CSCI 547, both of which are focused on current and emerging methods in AI and ML.
PLO-3: At least 50% of students will have passed one of CSCI 547 or M 508, and at least 33% of students will have passed both CSCI 547 and M 508. These courses involve critical analysis and application of machine learning methods.
PLO-4: At least 50% of students will have passed one of CSCI 547, CSCI 550, STAT 511, STAT 512, or M 507. At least 33% of students will have passed at least two of these courses. These courses include projects focused on real-world problems.
PLO-5: At least 50% of students will have passed one of CSCI 546, CSCI 547, or CSCI 550. At least 33% of students will have passed at least two of these courses. The problems studied in these courses are not solvable by traditional means.
-
Discuss methods and potential data sources of student work.
Methods of Assessment & Data Source.
Undergraduate courses (CSCI 446, CSCI 447) are evaluated via multiple programming projects that require students to implement and test AI and ML methods from scratch. These projects involve creating design documents, implementing code-based solutions, and conducting experiments on multiple data/problem sets based on the implemented code.
Graduate courses are project-based, involving personalized research. Products such as research papers, paper summaries and critiques, and public presentations are evaluated.
PLO-1: UG evaluated via projects (direct) and active learning activities (indirect). G evaluated via in-class discussions (indirect), written summaries (direct), and project reports (direct).
PLO-2: All evaluations are via in-class discussions (indirect) and written assignments (direct).
PLO-3: All evaluations are via in-class discussions (indirect) and written assignments (direct).
PLO-4: All evaluations are via in-class discussions (indirect) and written assignments (direct).
PLO-5: All evaluations are via in-class discussions (indirect) and written assignments (direct).
-
Develop a multi-year assessment schedule that will show when all program learning outcomes will be assessed.
Timeframe for Collecting and Analyzing Data.
The faculty will review the AI Certificate program at a faculty meeting every other year. Data used for the evaluation will be collected based on the threshold values indicated above.
-
Using the chart below, fill in the map. This table can be recreated to make more room for PLOs and/or change the layout. Mapping should also occur in the Courseleaf CIM system.
Curriculum Map & Assessment Planning Chart.
| Program Learning Outcomes | Course Alignments: Include rubric, number, and course title | Identification of Assessment Artifact | 2024-2025 | 2025-2026 | 2026-2027 | 2027-2028 | 2028-2029 |
|---|---|---|---|---|---|---|---|
| PLO-1 | CSCI 446: Artificial Intelligence | Sample projects (assessed by CS) | N/A | X | X | ||
| PLO-2 | CSCI 546: Advanced Machine Learning | Sample reports (assessed by CS) | N/A | X | X | ||
| CSCI 547: Machine Learning | Sample reports (assessed by CS) | N/A | X | X | |||
| PLO-3 | CSCI 547: Machine Learning | Sample reports (assessed by CS) | N/A | X | X | ||
| M 508: Mathematical Foundations of Machine Learning | Sample assignments (assessed by Math) | N/A | X | X | |||
| PLO-4 | CSCI 547: Machine Learning | Sample reports (assessed by CS) | N/A | X | X | ||
| CSCI 550: Advanced Data Mining | Sample reports (assessed by CS) | N/A | X | X | |||
| STAT 511: Methods of Data Analysis I | Sample assignments (assessed by Stats) | N/A | X | X | |||
| STAT 512: Methods of Data Analysis II | Sample assignments (assessed by Stats) | N/A | X | X | |||
| M 507: Mathematical Optimization | Sample assignments (assessed by N/A Math) | N/A | X | X | |||
| PLO-5 | CSCI 546: Advanced Artificial Intelligence | Sample reports (assessed by CS) | N/A | X | X | ||
| CSCI 547: Machine Learning | Sample reports (assessed by CS) | N/A | X | X | |||
| CSCI 550: Advanced Data Mining | Sample reports (assessed by CS) | N/A | X | X |
Part 3: What Will be Done.
Explain how assessment will be conducted, who receives the analyzed assessment data, and how it will be used by program faculty for program improvement(s).
- How will assessment artifacts be identified?
Course instructors determine appropriate artifacts that align with PLOs in their respective courses.
- How will they be collected (and by whom)?
Course artifacts will be collected by course instructors via the university Learning Management System (e.g., Brightspace or Canvas)
- Who will be assessing the artifacts?
Course instructors will assess the artifacts and will report the results of their assessments during regular CS faculty meetings the number of students who completed the indicated courses in the previous two years and provide summary statistics on graded artifacts and course grades. These statistics will also be summarized for the biennial evaluation.
Part 4: Assessment-Specific Rubrics.
All plans must include at least one program-specific assessment rubric (the methodology of how student artifacts are to be assessed). This is different than course-specific rubrics.
| Indicators or Criteria | Level 1 | Level2 | Level 3 | Level 4 | Threshold Values |
|---|---|---|---|---|---|
| Analysis of Information, Ideas, or Concepts | Identifies problem types suitable for AI | Understands complexity for AI problem types | Devises approaches to solve AI problem types | Demonstrates AI problem solutions | 80% of students will exceed Level 3; 60% Level 4. |
| Indicators or Criteria | Level 1 | Level2 | Level 3 | Level 4 | Threshold Values |
|---|---|---|---|---|---|
| Synthesis | Identifies key issues in AI methods | Analyzes underlying theories enabling AI solutions | Relates AI methods to diverse problem types | Demonstrates appropriate association of methods to problems | 80% of students will meet or exceed Level 3; 60% Level 4. |
| Evaluation | Verifies functioning of implemented methods | Develops strategies for quantifying performance | Assesses performance of implemented methods | Analyzes reasons for observed performance | 80% of students will meet or exceed Level 3; 60% Level 4. |
| Indicators or Criteria | Level 1 | Level2 | Level 3 | Level 4 | Threshold Values |
|---|---|---|---|---|---|
| Application of Information, Ideas, or Concepts | Determines how to apply standard solution methods | Assesses effectiveness of standard solution methods | Proposes extended solution methods | Assesses proposed extensions against standard methods | 80% of students will meet or exceed Level 3; 60% Level 4. |
| Indicators or Criteria | Level 1 | Level2 | Level 3 | Level 4 | Threshold Values |
|---|---|---|---|---|---|
| Synthesis | Identifies key issues in AI methods | Analyzes underlying theories enabling AI solutions | Relates AI methods to diverse problem types | Demonstrates appropriate association of methods to problems | 80% of students will meet or exceed Level 3; 60% Level 4. |
| Evaluation | Verifies functioning of implemented methods | Develops strategies for quantifying performance | Assesses performance of implemented methods | Analyzes reasons for observed performance | 80% of students will meet or exceed Level 3; 60% Level 4. |
| Indicators or Criteria | Level 1 | Level2 | Level 3 | Level 4 | Threshold Values |
|---|---|---|---|---|---|
| Analysis of Information, Ideas, or Concepts | Identifies problem types suitable for AI | Understands complexity for AI problem types | Devises approaches to solve AI problem types | Demonstrates AI problem solutions | 80% of students will meet or exceed Level 3; 60% Level 4. |
| Application of Information, Ideas, or Concepts | Determines how to apply standard solution methods | Assesses effectiveness of standard solution methods | Proposes extended solution methods | Assesses proposed extensions against standard methods | 80% of students will meet or exceed Level 3; 60% Level 4. |
| Synthesis | Identifies key issues in AI methods | Analyzes underlying theories enabling AI solutions | Relates AI methods to diverse problem types | Demonstrates appropriate association of methods to problems | 80% of students will meet or exceed Level 3; 60% Level 4. |
| Evaluation | Verifies functioning of implemented methods | Develops strategies for quantifying performance | Assesses performance of implemented methods | Analyzes reasons for observed performance | 80% of students will meet or exceed Level 3; 60% Level 4. |
Part 5: Program Assessment Planning & Report Communication
- How will annual assessment be communicated to faculty within the department? How will faculty participating in the collecting of assessment data (student work/artifacts) be notified?
A regular faculty meeting will be held every other year to assess the program. Faculty will be notified by the program coordinator two weeks before the scheduled meeting to collect artifacts and generate summary statistics.
- When will the data be collected and reviewed, and by whom?
Data will be collected by relevant faculty and provided to the program coordinator. The program coordinator will summarize the assessments and program data for presentation to at the faculty meeting.
- Who will be responsible for the writing of the report?
The program coordinator will be responsible for preparing the biennial report. The school administrative assistant will be responsible for recording minutes of the faculty meeting reviewing the material.
- How, when, and by whom, will the report be shared?
The biennial report will be shared with the faculty and discussed by the faculty at a regularly scheduled faculty meeting prior to submission. Faculty meeting minutes are available upon request from the school administrative assistant.
Part 6: Closing the Loop(s).
“Closing the Loop” is the self-reflective portion of the assessment where faculty have an opportunity to evaluate how a PLO(s) was assessed previously compared to the findings in the current report. The goal of program assessment is continual student learning improvement even if thresholds have been met. Please explain plans for how Closing the Loop will be documented going forward?
Given that this is a Year 0 report, PLOs have not yet been assessed for this certificate program. After data is collected, shared and discussed at a faculty meeting, the results will be documented in the biennial report for the Artificial Intelligence certificate.
Other Comments: N/A
Submit report to programassessment@montana.edu Upload Assessment Plan to department website for future reference.
