Comprehensive Standard #3.3.1 (Institutional Effectiveness)

The institution identifies expected outcomes for its educational programs and its administrative and educational support services; assesses whether it achieves these outcomes; and provides evidence of improvement based on analysis of those results.

Compliance Certification

The College of William and Mary is in compliance with this comprehensive standard.

Explanation

The College recently adopted a new process of institutional effectiveness or PIE (see Core Requirement 2.5). In the new process, educational programs and support services (1) describe purposes that link to the College's mission; (2) specify learning expectations and operational objectives, (3) identify experiences and activities that relate directly to the expectations and objectives, and (4) evaluate the extent to which expectations and objectives are met. The evaluations are used to (5) maintain and work toward excellence in student learning and support services. This process is aligned with a university-wide process of institutional effectiveness that ensures intentional and transparent budgeting and decision-making in a continual effort to enhance student learning.*

The College has been engaged in rigorous reviews of general education since the implementation of the new curriculum in 1996 and of Departments and Programs for well over a decade. As described below, the new PIE builds on the earlier assessment processes.

General Education

The Educational Policy Committee, assisted by the Assessment Steering Committee (see Faculty Committees of the Faculty of Arts and Sciences and 2004-2005 College-wide Committee List has assessed the general education curriculum since its implementation in 1996. In the current cycle of reviews, the College is implementing a general education PIE. The work began in earnest in 2004 when faculty members drafted purposes and expectations for the seven General Education Requirements (GERs) and the Freshman Seminar Program. The purposes and expectations are based on the faculty-approved definitions and elaborations of each requirement. Faculty members, working with members of the Assessment Steering Committee and Planning and Assessment staff, modified the portfolio methodology used in the first round of assessments to align with the PIE framework. In the modified methodology, portfolios include the course syllabus, a narrative linking course experiences to learning expectations, and a sample of student work that demonstrates student learning relevant to the expectations. In addition, the Office of Planning and Assessment provides a summary of survey responses from students enrolled in the course. A pilot study of the portfolio evaluation process was conducted in Spring 2005.

For the pilot study, 26 faculty members constructed portfolios and 16 faculty members served on working groups to evaluate the portfolios. The Dean of Undergraduate Studies appointed the working group members. Participants rated the extent to which each portfolio demonstrated that course experiences resulted in the types of student learning defined in the learning expectations. After achieving consensus, working group members posted the ratings in the individual course portfolios. Working groups also noted ways to improve the rating guide, the rating process, and the construction of course portfolios, and they provided feedback to instructors about the structure and content of their individual portfolios.

While the overwhelming majority of portfolios included the information necessary for assessment, in a very few cases instructors provided no material. Based on the pilot study, the working groups endorsed a strategy to involve departments and programs more fully in the review, recognizing that at William and Mary, courses originate at the department, program, or school level, and accountability is greatest within that context. The working groups also proposed assessment at the course-section level so that the Educational Policy Committee can monitor the effectiveness of the general education curriculum at the individual courses level.

The first round of GER/Freshman Seminar assessments indicated that most courses and seminars are meeting the criteria for each requirement. The Educational Policy Committee proposed several changes based on the assessments. The only substantive recommendations were to the Natural Sciences requirements (GER 2; see Minutes of A&S meeting; GER 2 revision). Other recommendations included modest word changes, for example, for the requirement in GER 5 (Literature and History of the Arts; see Minutes of A&S Meeting; GER 5 revision). The working groups proposed a number of improvements to the assessment process. Working groups expressed frustration with not being able to identify individual course sections that only marginally met the criteria of the GER. Similar to the recommendations of the pilot working groups, the groups in the first round wanted to monitor the curriculum at both the curricular and course section levels. The PIE framework encourages this level of oversight by the Educational Policy Committee and each department and program in which GERs are taught.

In the first round of assessments, course portfolios were not electronic and many were returned to the instructor at the end of the review. Portfolios from more recent reviews (e.g., GER 4: World Cultures and History, GER 2: The Natural Sciences) are on file in the Office of Planning and Assessment. In the next round, most portfolios will be electronic. This makes the construction and review of the portfolios easier. As evidenced in the portfolios collected for the pilot study, some faculty members prefer to submit paper copies of student work, but all other material will be loaded into a designated Blackboard ("Blackboard Learning System") site.

The table below shows the schedules for the first two rounds of general education reviews, with hyperlinks to the working group reports from the first round and a list of courses included in the Spring 2005 pilot study. Course portfolios are presented in Comprehensive Standard 3.5.1 (see Table 3.5.1A and Table 3.5.1B). You may also visit the GER Blackboard site .

Table 3.3.1A
1st Round of Assessments of GERs and Pilot Study of 2nd Round
REQUIREMENT	1st round of assessments		Pilot Study 2nd round of assessments
	Working Group Report	Year	Pilot Study Portfolios	Year of next review
GER 1 - Mathematics and Quantitative Reasoning	GER 1 report GER 1 tables	1999-2000	ECON 307 MATH 112	2007-2008
GER 2 - The Natural Sciences	GER 2 report	2000-2001	GER 2 Courses A: PHYS 176 B: BIOL 203 GER 2 Labs A: PHYS 108 B: KINE 305	2007-2008
GER 3 - The Social Sciences	GER 3 report	1997-1998	ECON 102 SOCL 312	2006-2007
GER 4 - World Cultures and History	GER 4 report	2001-2002	A: AMST 201 A: HIST 112 B: HIST 131 B: RELG 203 C: ANTH 202 C: WMST205	2008-2009
GER 5 - Literature and History of the Arts	GER 5 report	1998-1999	ARTH 251 CLCV 218 ENGL 210 MUSC 213	2005-2006
GER 6 - Creative and Performing Arts	GER 6 report	1998-1999	ARTH 212 DANC 112 MUSC E04	2006-2007
GER 7 - Philosophical, Religious, and Social Thought	GER 7 report	1999-2000	KINE 493 PHIL 201	2005-2006
Freshman Seminars	FS report	1997-1998	CHEM 150W ENGL 150W JAPN 150W	2005-2006

Departments and Programs

Most departments and programs have developed a set of student learning expectations and mapped them to curricular experiences. Most also have designed methods for evaluating the extent to which learning expectations are met and a developed plans for using results to improve student learning (see PIE Implementation Status Worksheet and PIE Database). The chair of the Assessment Steering Committee reviewed each PIE plan and provided detailed feedback to the departments and programs. This Fall, the Assessment Steering Committee will review the PIE plans, monitor implementation, and integrate the old program review process into the PIE framework.

The College has included student outcomes in program reviews for nearly two decades. The PIE framework introduces annual progress reports into the College's rich history of program reviews. The new annual process shifts the focus from a more traditional and static self-study model to a dynamic and progressive planning and evaluation model. The university-wide PIE integrates information from academic departments, programs, and schools with information from Vice Presidents and other administrative officers to provide annual progress reports. The Provost reviews these reports, along with members of the Faculty Committee on University Priorities and the Budget Advisory Group. In turn, the Provost reports on annual dispersed budgets and discretionary decisions. The five-year comprehensive reviews of academic units will look at trends in the program and the discipline, and will focus strategically on results from the annual PIE reviews . Internal and external evaluators will continue to be central to five-year reviews. The new comprehensive review process will be phased in beginning in 2006-2007. The tentative five-year review schedule is aligned with the GER PIE schedule.

This new process builds on the previous two rounds of program reviews. These reviews included student outcomes components. The initial round began in 1988 with a review of undergraduate majors. In that first round, each department and program provided a statement of educational goals, a description of the curriculum, background information on the program, student exit-time assessment of achievements, student and alumni surveys, and overall evaluations by outside evaluators. Department and program faculty members presented plans of action based on the review, and the Dean of the Faculty of Arts and Sciences wrote detailed responses. Follow-up reports on the plans were part of the second round of program reviews (see Examples of Program Review Results).

The most recent round of reviews focused on four primary areas: actions taken since the first assessment, writing and computing proficiencies, advising in the major, and library holdings. In addition, each review packet included descriptions of the program and department facilities, enrollment and graduation trends, post-graduation pursuits, and student and alumni survey results. To help reduce the workload on departments and programs, the Office of Assessment (now the Office of Planning and Assessment) conducted surveys, collected information, and packaged the review material. The Office of Assessment also coordinated visits by external evaluators.

During the second round (in 1997), the Assessment Steering Committee and the Committee on Graduate Studies developed a joint undergraduate and graduate review process to consolidate the two independent review efforts. Initially, these reviews were concurrent but they were not integrated. The Committees revised the guidelines to make the processes parallel, and a pilot study of the new review process was conducted in 2001 (see excerpts from Minutes from Assessment Steering Committee, October 3. 2001).

In 2002, Provost Gillian Cell suspended program reviews when a state budget crisis led to serious budget reductions and the State Council of Higher Education for Virginia restricted use of consultants and travel funds. By that time, the task force that had been appointed by Provost Cell to study existing practices of planning and evaluation had recommended the development of a fully integrated process of institutional effectiveness. This recommendation was consistent with issues raised about the program review process. In addition, the Assessment Steering Committee expressed concern about the effectiveness of program review to inform decision-making (see excerpts from Minutes from Assessment Steering Committee, September 11, 2002). The new process addresses those concerns.

The College was moving to electronic reviews when the second cycle was suspended. The following schedule of first and second rounds of reviews includes links to the most recent review materials. These examples come from a variety of disciplines and they illustrate the structure of the program reviews. Materials from the first and second rounds of program reviews are on file in the Office of Planning and Assessment. Note that the initial cycle was five years followed by an extended nine-year cycle. The State Council of Higher Education in Virginia agreed to the extended cycle so that we could devote more resources to implementing and assessing the new general education curriculum. The upcoming round of program reviews (including general education) will be on the original five-year schedule.

Table 3.3.1B
1st and 2nd Rounds of Department and Program Reviews
1st Round of Program Reviews		2nd Round of Program Reviews
Year	Department/Program	Year	Department/Program
1988-1989	Chemistry Philosophy Sociology	1995-1996	Business Philosophy Physics (joint) Psychology (joint) Public Policy (graduate)
1989-1990	Accounting Education Geology Mathematics Music Physics Psychology	1996-1997	AppliedScience(graduate) Chemistry Geology GER/Freshman Seminar Pilot
1990-1991	Business Classical Studies English Religion	1997-1998	Computer Science (joint) History (joint) Public Policy (graduate) GER 3 Freshman Seminars
1991-1992	Art/Art History Biology Government History Kinesiology	1998-1999	Anthropology (joint) American Studies (graduate) Mathematics Religion GER 5 GER 6
1992-1993	Anthropology Computer Science Economics Modern Languages/Literatures Theatre/Speech	1999-2000	Art/Art History Chemistry (graduate) GER 1 GER 7
1993-1994	American Studies Environmental Science International Studies/Relations Linguistics Public Policy Women's Studies	2000-2001	Classical Studies Kinesiology Music Theatre/Speech/Dance Physics (graduate) Psychology (graduate) GER 2
1994-1995	Evaluation: Program Reviews	2001-2002	Biology Economics English Government Public Policy (graduate) Sociology GER 4

Administrative and Educational Support Services

As reported in Core Requirement 2.5, a process was implemented in January 1996 whereby administrative offices and educational support services set goals and objectives and reported on their progress in annual reports to the Budget and Priorities Advisory Committee. In the face of serious budget cuts, the reports focused on budget reductions rather than goals and objectives. Under the leadership of Provost Geoffrey Feiss, a new Process of Institutional Effectiveness (PIE) was designed in 2004 to inform decision making in times of restricted and expanded budgets. The PIE ensures that decision makers will use assessment results to inform planning and resource allocations, and that they will document the decisions and their relationship to efforts towards continual improvement PIE Database.

The following examples show how administrative and educational support services reinforce the College's mission and commitment to student learning. Each program is reviewed regularly and adjusted based on those reviews.

Roy R. Charles Center:

Among its many functions, the Charles Center promotes faculty development and student research. Three programs highlight the types of educational support offered through the Charles Center : the Monroe Scholars program, May Seminars, and the Teaching Project.

Monroe Scholars:

Each year, the most academically distinguished undergraduates are designated as James Monroe Scholars. Students are selected based on standardized test scores and class rank with a special emphasis on "a concern for community, intellectual depth, curiosity and demonstrated devotion to learning for learning's sake." Monroe Scholars are guaranteed $3,000 to pursue a summer research project after their sophomore or junior year. Students work with faculty advisors to design the research project and submit a final "product" to their advisors. The advisors review the work and evaluate the extent to which intended objectives for the research were achieved. The evaluations and student work are reviewed further by Joel Schwartz, Dean of Honors and Interdisciplinary Studies and Director of the Charles Center, and Lisa Grimes, Associate Director and Coordinator of the Office of Undergraduate Research and National Scholarships. Reviews have been used to make changes to project requirements and the Monroe Scholars program.

May Seminar Program and University Teaching Project:

The Charles Center coordinates several opportunities for faculty development. The May Seminar Program awards grants to faculty members to support teaching and curriculum development projects. The program supports projects for 2 to about 10 faculty members for:

specific curriculum development or teaching improvement objectives, or
faculty enrichment objectives, more broadly understood.

The Committee on Honors and Interdisciplinary Studies monitors the May Seminar Program through written reports submitted by each faculty member who is awarded a May Seminar fellowship. The University Teaching Project, co-sponsored by the Provost and the Charles Center , is an opportunity for faculty from across the university "to learn about successful teaching approaches and to engage in a sustained dialogue on teaching with colleagues from across the campus." Participants construct "course portfolios" to show how they are applying what they learn in in the Teaching Project to their new or enhanced courses. The portfolios are presented to other faculty members to promote discussions about teaching and to share good practices.

Library Services:

As noted in the discussion of Compliance Standard 3.8.1, Swem Library staff regularly conduct and participate in assessment activities. For example, they conduct an annual satisfaction survey of William & Mary seniors (see Senior Survey (Library) 2003; data from the 2004 survey have been examined in a preliminary way but are not yet available for release). In addition, the Arts and Sciences faculty conducts a periodic survey of faculty members, with one major section focused on library services (see, for example, library section of the faculty survey). Department and program faculty members and external review teams also review library resources as part of program reviews. To get a feel for the library component of the reviews, see relevant sections in the review packets for Sociology Department and Biology Department external reviewer comments for Government Department (page 5), and the response of the Classical Studies Department faculty (page 8). Beyond these, Library staff members have recently completed a Strategic Plan (Strategic Initiatives), identifying regular and systematic assessment as one of four major goals.

Volunteer Services:

The Office of Student Volunteer Services was established in 1994 and supports the College's emphasis on public and community service. As noted on its website, the Office has conducted extensive assessment on the impact of its programs and volunteers. The findings reveal that community agencies have come to rely heavily on student involvement, that students who volunteer regularly develop a wide variety of competencies, and that students are volunteering more now than ever in the history of the College of William and Mary.

Footnotes

* Academic leaders at William and Mary have engaged in assessment for nearly two decades. Throughout that time, faculty members, department chairs, program directors, deans, the provost, and others have thought seriously about the goals of assessment and about developing directions in academic accountability. We have developed and pursued scholarly research, written and published papers and articles, presented at national academic conferences, and otherwise reflected on the value of assessment, and more particularly, the evaluation of "student outcomes."

We are fortunate to attract uniformly bright, highly motivated students and faculty members who are committed to the highest standards of teaching and scholarship. In the course of many years of external program reviews and extensive assessment of general education, we have reached two conclusions that we hold with some confidence. First, we have in place effective mechanisms for testing the quality of our curricula against external standards. Second, committed teachers working within the context of carefully organized academic courses do the most authentic evaluation of student academic work (skills, abilities, knowledge, and critical thinking abilities, in particular).

It does not follow from these observations that we need nothing further by way of academic assessment. Indeed, given (1) motivated, prepared students; (2) courses conceived as parts of a well-designed curricula; and (3) committed, well-trained, and motivated teachers, there remains a critical question about linkage: Do the courses deliver the specific content and skills required by and for the curricula? We believe that this fundamentally is a question about learning experiences. We have designed a process that focuses specifically on that question. The assessment process asks faculty members to identify the specific learning expectations for their courses. Faculty working groups check these against the learning expectations for the curriculum of which the courses are a part. We ask instructors to describe the learning experiences that they provide for students to learn what is expected. Faculty working groups evaluate those using their own experiences and common understandings within the profession. We use samples of student work, identified by instructors as adequate, to examine (1) the effectiveness of the learning experiences and (2) the standards being applied by the teacher. Through survey data, students describe learning experiences, and these descriptions provide a context for evaluating learning through samples of student work. In addition, faculty members may explain student survey responses, and these explanations can clarify the work of faculty working groups.

We have looked carefully at assessment strategies in other universities, especially those that purport to focus on student "outcomes." We believe that our current strategy provides the data most useful for assessing the quality of student learning. Further, the assessment strategy provides direct information about how to improve student learning when we identify that need. [return]

The College of William and Mary

SACS Compliance Certification

August 2005