University of Utah Graduate Council Program Reviews…

University of Utah
Graduate Council Program Reviews
__________________________________________________________




Description
and
Procedures




The Graduate School
302 Park Building
201 South Presidents Circle
University of Utah
Salt Lake City, Utah 84112-9016
                Persons Involved with Graduate Council Program Reviews



David S. Chapman
Dean of the Graduate School
phone: (801) 581-6925
e-mail: dchapman@admin.utah.edu


Frederick Rhodewalt
Associate Dean of the Graduate School
phone: (801) 585-7199
e-mail: fred.rhodewalt@admin.utah.edu


Denise E. Haynie
Executive Secretary
phone: (801) 581-6925
e-mail: dhaynie@admin.utah.edu




University of Utah
Graduate Council Program Reviews

Description
and
Procedures


Produced Winter Quarter 1996
Revised Spring Semester 2006

The Graduate School
302 Park Building
201 South Presidents Circle
University of Utah
Salt Lake City, Utah 84112-9016




                                                TABLE OF CONTENTS


                                                                                                                         Page

     Introduction ................................................................................................        1

     Purpose ......................................................................................................       1

     Elements of an effective program review ..................................................                           2

     Participants in program reviews and their responsibilities ........................                                  3

     Steps in the Graduate Council program review .....................................                                   5

     Nominating potential reviewers ................................................................                      7

     Program self-study ....................................................................................              8

     Sample Department Review Sheet                          .........................................................    9
.
     Degree completion/attrition data form ...........................................                                    13

     Scheduling external and internal reviewer site visits ................................                               14

     Sample itinerary for external reviewers ......................................................                       16

     Guidelines for external and internal reviewers ..........................................                            18

     Report format for external and internal reviews .........................................                             21
                                          Introduction

   All undergraduate and graduate degree granting programs at the University of Utah are
subject to regular review. Reviews for programs with both graduate and undergraduate
components or that award only graduate degrees are under the jurisdiction of the Graduate
Council and are administered by the Dean and Associate Dean of the Graduate School. The
Undergraduate Council reviews programs that are solely undergraduate in nature.

    These reviews are characterized by a general approach: they are collegial in the broadest
sense of the term and are based on the concept of peer review; they are scholarly in that they
seek to define questions whose answers will increase understanding of the programs; they are
comprehensive in that they view the programs under review as being connected both to other
programs within the university and to the intellectual issues of the discipline at large; and
finally, they are dynamic in that they result in actions that will improve undergraduate and
graduate education.1

                                            Purpose

   Educational units within universities require regular scrutiny and self-examination to
improve, and the systematic review of academic programs is an integral part of this process of
improvement. The purpose of program review is to improve quality education in the State of
Utah by strengthening established programs and eliminating or upgrading those, which fail to
meet acceptable standards

    Program review has several associated objectives or goals: (1) For the university,
program review helps in long-range planning by providing information about the size and
stability or vitality of a program, its faculty resources and student demand, its equipment and
space needs, its strengths and weaknesses, and its contribution to the mission of the
institution. It helps set goals for the future and ensures that overall academic plans and budget
decisions are based on real information and agreed-upon priorities, not vague impressions. (2)
For the educational unit, program review provides a mechanism for change and improvement
by creating a structured, scheduled opportunity for a program to be examined. The
mechanism should be well-reasoned, far-seeing, and as apolitical as possible. (3) From an
external point of view, program review provides a mechanism for universities to be
accountable to society (state government, funding agencies, donors, taxpayers, and tuition-
paying students) for their activities and for the quality of their programs.




1 Sections in "Introduction", "Purpose", and "Elements of an Effective Program Review" are
adapted from Academic Review of Graduate Programs - A Policy Statement, Council of
Graduate Schools, Washington, D.C., 1990, pp.26.
                          Elements of an Effective Program Review


1.   Program review is initiated and administered within the institution.

2.   Program review is evaluative, not just descriptive. More than the compilation of data on
     a particular program, it requires academic judgments about the quality of the program
     and the adequacy of its resources. It goes beyond an assessment of the minimum
     standards adequate for licensure or accreditation to evaluations of quality by peers and
     recognized experts in the field.

3.   Review of programs is forward-looking; it is directed toward improvement of the
     program, not simply assessment of its current status.

4.   Departments engaged in program review are evaluated using academic criteria.

5.   To the extent possible, program review is an objective process. It asks departments to
     engage in self-studies, which assess, as objectively as possible, their own programs. It
     brings faculty members from other departments and from outside the institution to
     review the self-studies and to make their own evaluations, using independent judgments.
     It is part of an established, public process in which all programs are similarly reviewed.

6.   Program review is an independent process, separate from other reviews. Reviews
     conducted by regional or professional accrediting associations, licensing agencies or
     budget committees are separate and distinct. Data collection and parts of the department
     self-study may often serve a number of review purposes and thus program review will
     often be scheduled to coincide with an accreditation or other external review. But to be
     effective, program review must be a unique, identifiable process, which stands on its
     own, draws its own set of conclusions, and directs its recommendations to the only
     individuals who have the power to improve programs: the faculty and administrators of
     the institution.

7.   Most important of all, program review results in action. Growing out of the reviewers'
     comments and recommendations, the institution develops a plan to implement the
     desired changes on a specific, agreed-upon timetable. This plan is linked to the
     institution's budget and planning process, to help ensure that recommended changes
     actually get made, that necessary resources are set aside, and that the program's goals fit
     into the institution's overall academic plans.

8.   Successful program review, then, is a process of evaluation, which has all of the above
     characteristics. It provides answers to the following kinds of questions:
          - Is the teaching and training of students effective and useful?
          - Is the department advancing the state of the discipline or profession?
          - Does the program meet the institution's goals?
          - Does it respond to the profession's needs?
          - How is the program assessed by experts in the field?
        Participants in University of Utah Program Reviews and their Responsibilities

   Program review involves the participation of two groups of consultants: (a) an internal
committee consisting of faculty members drawn from other departments of the University of
Utah; and (b) external consultants who are professors of national reputation in the discipline
under review.

    Internal committee members are appointed by the Graduate School but are selected in part
based on names suggested by the department under review. As a general rule there are three
faculty members on each internal review committee. The internal committee has two major
functions: (a) to provide judgment on departmental programs from the viewpoint of
colleagues at the University of Utah; and (b) to review in detail the academic program of
students and to assess the achievement of students and the quality of work accepted toward an
undergraduate or graduate degree. Internal committees are encouraged to focus their attention
on questions that require considerable familiarity with departments and their relationship to
other departments and to the activities of the total university.

    External reviewers are faculty members from other universities who are nationally
recognized educators and scholars in their respective subject fields. External reviewers are
appointed by the Graduate School but are selected in part based on names suggested by the
department under review. Three external reviewers are selected to visit the University of
Utah. The task of the external reviewers is to formulate objective judgments of quality and
effectiveness of undergraduate and graduate programs. This evaluation is concerned
primarily with the quality of education actually achieved by students and includes, but is not
restricted to, an assessment of the quality of faculty, the adequacy of curriculum offerings and
program options, the existence of policies and practices in support of students, adequacy of
the departmental budget, and the adequacy of staff support, physical facilities, library
resources, equipment, and research facilities. In addition, the review considers the
justification of the program in terms of such factors as employment demand, potential student
population, and service functions performed by the department.

     As an aid to the external and internal review teams, packets of documentary material (the
program self-study) are prepared by the departments under review and made available through
the Graduate School to reviewers in advance of the review. These materials follow a format
described later in this document and include such information as (1) departmental goals and
mission plan, (2) faculty vitae, (3) course listings and program options, (4) admission policies
and degree requirements, (5) statistical data on enrollment, degrees granted, class size, etc. (6)
financial data, and (7) a description of research facilities, equipment, space, library holdings,
etc.

    The external and internal review teams each prepare a written report based on interviews,
on-site studies, and the appropriate documentary materials are prepared by the internal review
committee and the external reviewers and submitted to the Associate Dean of the Graduate
School. The dean then transmits copies of the reports to the department chair and dean of the
college and requests written responses after they have had the opportunity to discuss the
reports with all department faculty. The chairman is expected to share all reports with all
members of the regular faculty.
     All documents are then submitted to an ad hoc committee of the Graduate Council
charged with synthesizing the reports and producing a summary report for the program
review. The ad hoc committee meets with external consultants when they are on campus and
with the internal committee when it submits its report to the Dean of the Graduate School.
The ad hoc committee may, but is not required to, meet with the college dean or the
department chair to clarify issues raised in the various reports. The ad hoc committee submits
its report with commendations and recommendations, along with all materials on which they
are based, to the Graduate Council, which debates and takes appropriate action on the
reviews.

    All Graduate Council actions on reviews are reported to the cognizant Senior Vice
President for Academic Affairs or the Senior Vice President for Health Sciences. The vice
president, who has not been directly involved in the program review until this stage, then
meets with the department/program chair, college dean, and dean and associate dean of the
graduate school, to discuss program review recommendations. A memorandum of
understanding of this wrap-up meeting for the program review and the Graduate Council
report of the review are transmitted in full to the University Senate and to the State Board of
Regents by the vice president.

    Staff support and expenses for preparing the program self-study are borne by individual
departments or units. The Graduate School provides honoraria, travel expenses, housing, and
meals for external reviewers invited to the campus. Internal committee members do not
receive honoraria for their work.
        Steps in the Graduate Council Program Review and Approximate Timelines.


1.   A program is informed formally of a scheduled Graduate Council program review
     during the academic year prior to the year of the actual review. Note that a seven-year
     plan for reviews is maintained in the Graduate School and in the office of each academic
     dean.


2.   The Associate Dean of the Graduate School and staff meets with the program chair and
     staff to review procedures and set timelines for the review. This meeting should occur
     no later than spring semester in the academic year preceding the review.

3.   The program is requested to:
     (a) prepare a self-study (see later section for format and description),
     (b) supply the Graduate School with names and contact information of at least 5
potential
          external reviewers,
     (c) supply the Graduate School with names and contact information of at least 5
potential
          internal reviewers.

4.   Following meetings with programs, the Graduate School contacts and appoints external
     reviewers and internal reviewers. The Graduate School informs the program chair and
     the college dean of the names of the respective review teams no later than spring
     semester in the year preceding the review.

5.   The program, in coordination with the Graduate School, arranges detailed itineraries for
     external and internal reviewers. Visits of external visitors should end with an exit
     interview with the Dean and Associate Dean of the Graduate School and with the ad hoc
     committee of the Graduate Council assigned to that particular program review. External
     reviewers are requested to submit their reports within one month of visiting the
     University.

6.   The internal review committee, upon completion of its work, delivers its report to, and
     discusses the report in person with the Dean and Associate Dean of the Graduate School
     and with the ad hoc committee of the Graduate Council assigned to that particular
     program review. The internal review team is requested to submit its report within one
     month of
     visiting the department.

7.   The Graduate School receives written reports from the external and internal review
     teams, which are then sent to the department/program chair and college dean for a
     written response. A period of one month is provided for replies.

8.   Both reports, plus responses from the department chair and college dean and the
     department's self-study are given to a Graduate Council ad hoc committee which is
      responsible for writing the synthesis report on the program review. The ad hoc
      committee includes one person assigned from the Undergraduate Council. In advance of
      writing the formal report the ad hoc committee meets with the Dean and Associate Dean
      of the Graduate School to discuss main findings of the review. At this meeting the
      report is scheduled for discussion at a specific Graduate Council meeting, leaving
      generally one month for ad hoc committee deliberation and writing.

9.    All Graduate Council members are given the program review synthesis report together
      with all supporting documents. The Graduate Council discusses, amends, and approves
      the report.

10.   The Graduate Council report is distributed to the department chair and to the college
      dean for correction of factual information. The program is requested to inform the
      Graduate School of significant actions taken since the inception of the review and not
      contained in other supporting documents.

11.   Wrap-up meeting. After the Graduate Council report is submitted to the Senior Vice
      President for Academic Affairs or Senior Vice President for Health Sciences, the
      department chair, college dean, and Dean and Associate Dean of the Graduate School
      meet with the cognizant vice president to discuss program review recommendations.
      For each recommendation that is deemed achievable within the resources of the
      institution, responsibilities and timelines are established for implementing the
      recommendation.

12.   The Graduate School Dean and Associate Dean are available to attend a program faculty
      meeting to discuss the review process in general and the recommendations and wrap-up
      memorandum in particular.

13.   The summary Graduate Council report and the wrap-up recommendation are sent to the
      Senior Vice President for Academic Affairs or the Senior Vice President for Health
      Sciences for forwarding to the Academic Senate, Board of Trustees and the State Board
      of Regents as information items.
                               Nominating Potential Reviewers

External Reviewers

    The program should supply the Graduate School with a list of five potential external
reviewers. They should be faculty members from other universities who are recognized
scholars and teachers in the fields under review. Please consider gender, ethnic, and age
diversity when suggesting potential reviewers. Also, please report any prior connections with
potential reviewers that may raise concerns about the reviewers' partiality. Note that the
Graduate School, not the department, contacts reviewers. The Graduate School also may
select reviewers not on the department list.

    For each person nominated, briefly describe the qualifications (i.e., relevant academic and
professional experience) that make this person an appropriate site visitor for your unit. In
order to avoid overlap and ensure appropriate coverage of the department, specify the person's
principal area of scholarly activity in terms of the areas represented by the department being
reviewed (e.g., history of the American West, organic chemistry, environmental geology).


Internal Reviewers

    The program should supply the Graduate School with a list of five potential internal
reviewers. So as to avoid conflict of interest, internal reviewers should not be based in the
same college as the program being reviewed. Please consider gender, ethnic, and age
diversity when suggesting names.

   For each person nominated, briefly describe the qualifications and relationship to your
department that make this person an appropriate internal committee member for your review.


Format for Listing Potential Reviewers

    To aid the Graduate School in contacting potential reviewers, please supply the following
information for each of the suggested external and internal reviewers, and each of the
thesis/dissertation reviewers or reviewing departments.

       Name:                                                 Phone:
       Title/Rank:                                           Fax:
       Dept./Academic Unit:                                  e-mail
       University:
       Address:
                                       Program Self-Study

     The self-study is prepared by the faculty of the department, college, or program and is
both descriptive and evaluative; it provides basic information on the nature of the program
and gives the faculty's assessment of the program's strengths and weaknesses. A program
self-study is the faculty's opportunity to scrutinize itself, to publicize its accomplishments and
examine its flaws. It is also a chance to explain itself and to demonstrate how it is viewed by
its peers. A self-study should lead reviewers through the following four questions:
        What do you do?
        Why do you do it?
        How well do you do it, and what is the measure of your success?
        What difference does it make whether you do it or not?

    Eight copies of the program self-study must be provided to the Graduate School. One
copy is sent by the Graduate School to each of the three external reviewers and each of the
three internal reviewers. Two copies are retained in the Graduate School. The program bears
the cost of producing the self-studies.

    The following list is a suggested organization for the self-study. It is not an exhaustive list
and individual programs may depart from the suggested format and/or include additional
information where appropriate. Wherever possible, data should be provided for the period
since the last Graduate Council review (normally seven years).


1. Program overview

1.1   Program mission and organization.
      This is an introduction to the program, its purpose, organization and history.

1.2   Program planning (centerpiece of the self study!)
      Include goals of the program, strategic or long-range plans. Include areas of teaching,
      research, or public service in which the program regards itself as especially outstanding
      and areas of teaching, research, or public service in which the program would like to
      improve. Give an outline of intended changes in the scope and/or direction of
      undergraduate or graduate education (new degrees, shifts in organization, new areas of
      specialization), and the resources at your command to make those changes.

1.3   Previous review and actions
      Include the Graduate Council report to the Academic Senate from your last review.
      Provide narrative on actions taken since the review.

1.4  Department review sheet
     Include Department Review Sheet with information on student enrollment, majors,
     degrees granted, faculty headcount, and salary averages by rank. This document is
     prepared by the Institutional Analysis Office and distributed to you by the Graduate
School
     (see sample, pg. 9). Provide narrative interpretation of the review sheet where
appropriate.
2. Faculty

2.1   Faculty profile/diversity
      Summary of faculty profile providing information on the number of faculty (tenure/non-
      tenure rank; regular and research/clinical; full-time and part-time); faculty hired or
retired    during the past five years, or no longer with your program; average age; age range
and distribution. Address separately the diversity of your faculty (gender, ethnicity) and
      departmental efforts to achieve appropriate diversity by hiring strategies and procedures.

2.2   Faculty teaching.
      Summary of recognition, awards, indicators of successful teaching. Include policies and
      practices in place to encourage and recognize good teaching. Describe participation in
      General Education, Honors, and other university-wide programs.

2.3   Faculty scholarship.
      Summary of faculty research, scholarship, or creative activity; individual productivity;
      sources and distribution of external and internal funding; quality measures, peer
      assessments. Provide narrative about the funding levels in your discipline necessary to
      keep a productive scholar functioning. Include policies and practices in place to create
      opportunities for scholarly growth.

2.4   Faculty service.
      Summary of faculty involvement in university, professional, and community service.
      Include policies in place to recognize service. Provide narrative about how the program
is
      impacted (positively and/or negatively) by its service components.

2.5   Retention, Promotion and Tenure.
      Include a copy of the college or department RPT guidelines with date adopted. Provide
a
      table showing all RPT cases considered since the last Graduate Council review with
      outcomes. Describe any faculty mentoring procedures you may have in place.

2.6   Faculty Vitae
      Include separate short-form (4 page maximum) vitae for all faculty. Vitae should
      include education and summary of training, honors and awards, other academic
      achievements that indicate a faculty member's academic stature, courses taught, current
      research, and selected publications and/or artistic presentations. You may wish to limit
      the length of faculty vitae to include activities and publications in this review cycle (last
      seven years) plus particularly noteworthy achievements from previous years.

3. Students

3.1   Student recruitment/diversity
      A statement on methods employed in recruiting, evaluating, and admitting both
      undergraduate
      and graduate students. What practices do you employ to retain students of particular merit in
      your undergraduate and graduate student body? Address separately department efforts to
      recruit minority students and to achieve appropriate diversity among your student body.
3.2  Graduate admissions
     Information indicating the quality of graduate students admitted to the program. A
useful    table would include undergraduate school, major, GPA, GRE or similar exam
scores.

3.3   Student support
      Methods of supporting and levels of support for graduate students (distinguish between
      Teaching Assistant, Research Assistant, Fellowships, Graduate Assistants). List
      scholarships and other financial support for undergraduates.

3.4   Student advising
      Description of academic advising practices for undergraduate majors and minors and
      prospective graduate students in the program. Assess the efficiency and effectiveness of
      your academic advising. Include written policies for handling student appeals.

3.5   Teaching assistant (TA) training
      Description of the program to prepare and train TAs in the art of effective teaching.
      Include TA orientation, on-going support and supervision, and TA evaluation. Describe
      additional measures, if any, taken to assist international TAs with communication skills.

3.6   Employment
      Provide statistical information and data, where available, on the present and projected
job
      market for degree recipients and for further graduate or professional study.

4. Curriculum and programs of study

4.1   Degree and certificate requirements
      List of all degrees, degree requirements, certificates, and program specialties in the
      program (may be copied from the current General Catalog if that list is up to date).

4.2   Courses offered
      Listing of all the courses offered in the program (from the current General Catalog).

4.3 Programs of study
    Give typical programs of study for (a) the Bachelor's degree, (b) the Master's degree,
and (c) the Ph.D. degree. For the Bachelor's degree include typical course sequences for
    the various program specialties offered in the program, where possible by semester and
    year. For the Master's and Ph.D. degree programs, copies of representative candidacy
    and program of study forms could be used.

4.4  Professional development
     Describe program efforts to provide training in professional development and
professional
     ethics and standards.

4.5   Outreach education
      Describe the unit's efforts to deliver education programs at sites remote from the central
      campus. What technologies are available to assist in your outreach programs? What is
the
      relationship between outreach offerings and programs and the unit's overall instructional
      program, goals, and mission? What credits are accepted from outside providers; what is
      the contractual and oversight relationship to faculty, curriculum, and credit?

4.6  Qualifying Exams
     Give the program policy for qualifying exams for master's and doctoral students.
Provide
     copies of questions for the last five qualifying exams. How do students perform on your
     qualifying exams? Give numbers of passes, fails, and retakes. Student responses should
     be kept in the program as exhibits for possible examination.

4.7   Theses and dissertations
      Tabulate all Master's theses and Ph.D. dissertations completed since the previous
      Graduate School review (normally the last seven years). Include the following: name of
      student, master's or doctorate, year of completion, name of principal faculty supervisor,
      title of thesis or dissertation. Also include abstracts of five recent dissertations and
      five recent theses.
5. Program effectiveness ­ Outcomes assessment

    Each educational unit has an obligation to plan carefully its courses of instruction in
response
    to student needs, to evaluate the effectiveness of that educational program in terms of the
    change it brings about in students, and to make improvements in the program dictated by
the
    evaluative process.

5.1   Outcomes assessment procedures
      List and describe the unit's processes for assessing its educational programs. The list
      may include, but is not restricted to, the use of outcomes measures in the following
      areas: (a) student information (recruitment, quality of students, retention, graduation
      rates, gender and ethnicity blend) and trends over time, (b) mid-program assessments,
      (c) end of program assessment (standard exam, capstone experience), (d) alumni
      satisfaction and loyalty, and (e) employment and/or employer satisfaction measures.

5.2   Outcomes assessment feedback
      Provide specific examples of how the assessment activities have been used to improve
      teaching and learning in the unit. Of particular interest would be descriptions of the
entire
      assessment feedback loop: identification and publication of expected learning outcomes,
      assessment measures, analysis, and interventions arising from the analysis that lead to an
      improvement in the program.

5.3   Degree completion data
      Using the form on pg. 13, provide data indicating graduate degree completion/attrition
      data.

6. Facilities and Resources

6.1   Operating budget issues
      Assess the budget adequacy with respect to the program's mission.

6.2  Physical facilities
     Describe the ways in which physical facilities in the unit encourage or limit the
educational objectives of the program. In what ways do they fail to meet the unit's needs?

6.3   Libraries
      Describe the program's general and special requirements for library resources in order to
      meet its educational and research objectives. Indicate ways in which the present library
      resources satisfy and do not satisfy these needs.

6.4   Centers, institutes or bureaus associated with the program.
      List any centers that are associated with the academic program and explain briefly the
      relationships (funding, faculty appointments, student supervision, etc.) between the
      centers and the academic program.
6.5   Computers
      Provide a general description of computing, word processing, networking, and e-mail
      facilities in the college or department/program. Give an outline of what facilities you
      would hope to have in place in five years' time.

6.6 Staff support.
    Describe the existing staff support for your educational and research missions.
Sample Department Academic Profile - Statistical Summary
                       Scheduling External and Internal Reviewer Site Visits


Coordination

    The Graduate School coordinates travel arrangements for the site visits, which are usually
about a day and a half long for external reviewers, with the visitors arriving the evening
before the visit and departing the late afternoon or early evening of the second day of the site
visit. The internal team visit is generally conducted in one day It is the unit's responsibility
to schedule the meetings described below, with the exception of the final exit meeting for
external reviewers, which is scheduled by the Graduate School.


Visits with Students

    Some of the most helpful meetings are those with students. Because students often bring
up questions for which the site visitors will want to seek answers, these meetings should be
set up fairly early in the schedule. Some units have found a brown bag lunch on the first day
or a coffee/refreshment hour early on the second day to be good. Separate meetings with
undergraduates and graduate students are desirable. After the visitors are introduced and their
purpose explained, program faculty members should leave so that students feel free to discuss
issues that either they or the site visitors bring up.


Visits with Faculty Members

    Depending on the size of the faculty, two or three meetings might be desirable so that
most faculty will have a chance to express their opinions. In addition to the general faculty
meetings, there could be small meetings with the faculty who prepared the self-study, the
graduate advisory committee, the undergraduate advisory committee, or other committees
whose work relates to the program review. Individual faculty may also request time with the
external reviewers. The head of the academic unit should not attend the meetings with
faculty.


Visits with Departmental Staff

    The external and internal review teams should have opportunities to meet with departmental
staff.


Visits with Program Chair and College Dean

    At least an hour should be scheduled for the site visitors to meet with the head of the
academic unit. Because site visitors will usually have questions from their conversations with
students and faculty, some time for this visit with administrators should be saved rather late in
the schedule. The program also should schedule a meeting between the external reviewer and
the dean of the supervising college.
Visits with Graduate School Dean, Associate Dean, and Members of Graduate Council

   The site visits for external reviewers should end with an exit interview in the Graduate
School. The exit interview will be attended by the Dean and Associate Dean of the Graduate
School and by the Graduate Council ad hoc committee charged with writing the final summary
report. Please note that an exit meeting is not held for internal review teams.


An Extra Note on Hospitality for Reviewers

    Please ask a faculty member to serve as a local host who will pick up external reviewers at
their hotel, escort them to their first meetings each day, arrange return transportation, and lend
general assistance over the two days. Lavish entertaining of the site visitors is not expected or
encouraged. Faculty members often go out to dinner with external reviewers; however,
reviewers may also appreciate the opportunity to have dinner alone as a review team in order
to discuss review business. The internal review teams generally require only lunch on the
day of the site visit. The Graduate School will reimburse meal expenses for the reviewers
only (contact the Graduate School for information about processing reimbursements). If
faculty members wish to go out to dinner or lunch with the visitors, the individuals or
department are responsible for their own expenses. The Graduate School cannot make
reimbursement for alcoholic beverages.
                         Sample Itinerary for External Reviewers

Wednesday, November 1

Evening      Reviewers arrive and take complimentary shuttle to Little America if
             convenient, or, if not, take taxi.


Thursday, November 2

7:15 a.m.    Professor Brown meets review team at the Little America coffee shop for
             breakfast and brings them to campus.

8:30 a.m.    Meet with Professor Smith, Chair of the program, in room 321, Building XYZ.

9:30 a.m.    Meet with the department executive committee in room 322, Building XYZ.

10.30 a.m.   Meet with undergraduate students in room 323, Building XYZ. Student group
             organized by SAC chairperson J.W. Powell.

12:00 p.m.   Lunch at the Crimson View Room, University Union, hosted by Professor
Jones,
             and with Professors Green, White, and Black.

1:30 p.m.    Meet with graduate students. Group organized by Ph.D. student P. King.

2:30 p.m.    Meet with faculty group 1. Conference room, Building XYZ.

3:30 p.m.    Meet with faculty group 2. Conference room, Building XYZ.

4.30 p.m.    Tour of facilities. Professor Smith will conduct the tour. Professor Smith will
             escort review team back to Little America Hotel after the tour.

7:00 p.m.    Dinner at Al Forno's. Professor Higgins will host. Also in attendance: Professors
             Miller, Gray, and Wright. After dinner, Professor Higgins will escort review team
             back to Little America.


Friday, November 3

7:15 a.m.    Professors Smith (Chair) and Hutton (Assoc. Chair) meet review team at the
             Little America coffee shop for breakfast and bring them to campus.

8:30 a.m.    Meet with faculty group 3. Conference room, Building XYZ.

9:30 a.m.    Meet with faculty group 4. Conference room, Building XYZ.

10:30 a.m.   Time to review documents, exhibits, course materials, or to meet with individual
             faculty who have asked for private meetings.

11:30 a.m.   Box lunch with director of Graduate Studies and Director of Undergraduate
             Studies.

1:00 p.m.    Meet with College Dean, Professor Harrison, in room 456, Building MPH.

2:00 p.m.    Exit interview with the Dean and Associate Dean of the Graduate School and
the          Graduate Council ad hoc committee in room 302 of the Park Building. Dean
             Harrison will arrange for someone to escort review team to the meeting.

3:15 p.m.    Professor Marshall will pick up reviewers in the Park Building and drive them
to
             the airport for their late afternoon flights.
                         Guidelines for External and Internal Reviewers

     The task of the external reviewers is to formulate objective judgments of quality and
effectiveness of undergraduate and graduate programs, and to determine where the program
fits in the discipline regionally, nationally, or internationally. This evaluation is concerned
primarily with the quality of education actually achieved by students and includes, but is not
restricted to: overall quality and direction of the program; an assessment of the quality of
faculty; students and the existence of policies and practices in support of them; curriculum
offerings and program options; and the adequacy of staff support, physical facilities, library
resources, equipment, and research facilities and of the program budget.

    The internal committee has two major functions: (a) to provide judgment on program
quality and governance from the viewpoint of colleagues at the University of Utah; and (b) to
review the academic program of students and the quality of work accepted toward an
undergraduate or graduate degree. Internal committees also are encouraged to focus their
attention on questions concerning the relationships of programs to the goals of the total
university.

    The global request of both external and internal reviewers is to single out those features of
the program that merit special commendation, and to make recommendations about situations
where there is room for improvement. Put simply: "What is the program doing very well?" and,
"What could the program do better?" Evaluations of these questions should be included in the
sections of the reviewers' reports titled II Commendations and III Recommendations (see report
format below).

    Reviewers' investigations and subsequent reports should address issues pertinent to the
following topics listed in the program self study:

1. Program Overview
     1.1 Program mission and organization
     1.2 Program planning
     1.3 Previous Graduate Council review and actions
     1.4 Department review sheet

Issues to be addressed include: the program's mission statement and organization and its
suitability for the 1990's and beyond; the scale of the program in terms of the number and
quality of the faculty, students, staff, facilities, and other resources; the extent of well-defined
departmental policies supported by concrete goals and intermediate objectives and methods of
assessing progress toward those goals and objectives; balance between teaching, research, and
service; adequacy of salaries and fringe benefits to attract and retain outstanding faculty and
staff; and program response to recommendations made in the previous Graduate Council
review.

2. Faculty
     2.1 Faculty profile/diversity
     2.2 Faculty teaching
     2.3 Faculty scholarship
     2.4 Faculty service
     2.5   Retention, promotion and tenure (RPT)
     2.6   Faculty vitae

Issues to be addressed include: the numbers of faculty and diversity of faculty interest for the
undergraduate and graduate programs offered (address separately the gender and ethnic
diversity of the faculty and departmental efforts to achieve appropriate diversity by hiring
strategies and procedures); policies and efforts in the recruitment, retention, and promotion of
minority and
women faculty; the concern for, and performance in, teaching at all levels of the program; the
general scholarly quality of the faculty compared with that of other leading colleges and

universities in the country; publication records of faculty in monographs and refereed journals;
record of external funding where appropriate; effectiveness of faculty performance evaluation,
including tenured faculty review; suitability of RPT guidelines; faculty mentoring; faculty
morale.

3. Students
     3.1 Student recruitment/diversity
     3.2 Graduate admissions
     3.3 Student support
     3.4 Student advising
     3.5 Teaching assistant (TA) training
     3.6 Employment

Issues to be addressed include: quality of undergraduate and graduate students (address separately
department efforts to recruit minority students and to achieve appropriate diversity among the
student body); admission standards (too low? too demanding?); financial support for students;
undergraduate and graduate student academic advising practices; appropriate training of teaching
assistants; admissibility of graduating seniors into graduate programs of their choice; student input
into the decision-making process in the program; completion rate of programs within normal time
limits; opportunities for student-faculty exchange; student morale; and need for the program as
indicated by (a) employers who hire graduates, (b) prospective students of high ability who apply
for admission into the program, (c) knowledgeable persons who urge that well-prepared
practitioners or researchers and new research findings and/or improved professional practice in the
field are needed by society

4. Curriculum and programs of study
     4.1 Degree and certificate requirements
     4.2 Courses offered
     4.3 Programs of study
     4.4 Professional development
     4.5 Outreach education
     4.6 Qualifying Exams
     4.7 Theses and dissertations

Issues to be addressed include: diversity of curriculum offerings to allow for a broad range of
educational experiences and for specialization in the major sub-divisions of the discipline;
program requirements (courses, graduation requirements, graduate qualifying examinations)
compared with other leading universities in the country; instructional methods and
innovations; training of teaching assistants; training in professional development and ethics;
guidelines for thesis and dissertation completion; supervisory committee guidelines and
operation.

5. Program effectiveness - Outcomes assessment
     5.1 Outcomes assessment procedures
     5.2 Outcomes assessment feedback
     5.3 Degree completion data

Issues to be addressed include: processes for assessing educational programs, including but
not restricted to the use of outcomes measures in the following areas: (a) student information
(recruitment, quality of students, retention, graduation rates, gender and ethnicity blend) and
trends over time, (b) mid-program assessments, (c) end of program assessment (standard
exam, capstone experience), (d) degree completion rates and time to degree, (e) alumni
satisfaction and loyalty, and (f) employment and/or employer satisfaction measures. How
have the department's assessment activities been used to improve teaching and learning in the
unit? Of particular interest would be descriptions of the entire assessment feedback loop:
identification and publication of expected learning outcomes, assessment measures, analysis,
and interventions arising from the analysis that lead to an improvement in the program.
6. Facilities and Resources
      6.1 Operating budget issues
      6.2 Physical facilities
      6.3 Libraries
      6.4 Centers, institutes, and bureaus
      6.5 Computers
      6.6 Staff support

Issues to be addressed include: budget adequacy with respect to the program's mission; How
appropriate are physical facilities (classrooms, office space, laboratories, study and lounge
space), libraries, and computer facilities in terms of instructional, research, and service goals
of the program? How do any centers, institutes, and bureaus associated with the program
affect the academic and research goals and operation of the program? In what ways is staff
support adequate or inadequate to support the educational mission of the program?
                       Report Format for External and Internal Reviews

       The most useful reports for the Graduate Council, and for program and administrative
heads, are those containing specific suggestions for improving the program. External
reviewers should concentrate on remarks that relate the unit being reviewed to national norms
and peer groups elsewhere. Internal reviewers should concentrate on the functioning of the
program in comparison with practice within the University of Utah. It is particularly useful to
receive descriptions of "good practice" external to, and within, the University of Utah that
would alleviate problematic practices that are uncovered in the review process.

       The length of reports is not prescribed; many reports received in the past vary in
length between three and ten pages.


Part I: General

        Part I should contain detailed observations, comments and discussion, organized into
the five topics listed above:
           Program overview
           Faculty
           Students
           Curriculum and programs of study
           Program effectiveness - outcomes assessment
           Facilities and resources
As the Graduate Council ad-hoc committee charged with preparing a synthesis of the multiple
reviewer reports will assemble most of the background information concerning the program
from the self study document, reviewers need not write extensively in this section. But it is
important to provide a context for understanding commendations and recommendations.


Part II: Commendations

       Part II should enumerate commendations. What is the program doing very well? And
in what areas has the program made significant recent progress that also deserves
commendation?


Part III: Recommendations

        Part III is reserved for recommendations that would improve any aspect of the
program. These recommendations may be addressed to the program participants (students,
faculty, staff), to program administrators, to the Graduate School, or to the University
administration. Documentation on specific recommendations that are known to have been
effective elsewhere are especially welcomed. Note: External and internal reports are
distributed widely among faculty and administration. As such, it is generally inappropriate to
name specific individuals in a critical manner.