Instructor:  Dr. Laurie M. Joyner (ljoyner@loyno.edu)
Office:   Monroe Hall – Room 537E
Phone:   865-2571
Office Hours:  TR 2:00 – 5:00 p.m. or by appointment

Class Meetings

• September 7th (6 – 10 p.m.)    and  September 8th   (9 a.m. – 5 p.m.)
• October 5th (6 – 10 p.m.)        and  October 6th   (9 a.m. – 5 p.m.)
• October 26th (6 – 10 p.m.)      and  October 27th (9 a.m. – 5 p.m.)

Course Description:  This course will introduce key concepts, methods, and approaches in the interdisciplinary field of evaluation research. Students will also be exposed to the theoretical and methodological diversity inherent in current evaluation practice across a number of substantive areas (e.g., social services, education, public health, and criminal justice). The comprehensive range of activities involved in designing, implementing, and assessing the utility of social programs, especially human service programs, will occupy our attention throughout the course. The course will be divided into the following six sections:

Introduction to Evaluation Research

Key Concepts in Program Planning and Evaluation

Designing and Implementing Comprehensive Program Evaluations

Evaluation Results in Selected Substantive Areas

Current Issues and Debates in the Field of Evaluation

The Future of Evaluation Research

• Critically Explore Social Issues
• Identify Need for Services
• Determine the Evaluability of Interventions
• Conceptualize Programmatic Interventions
• Develop Different Evaluation Designs
• Choose and Apply Appropriate Research/Evaluation Methods
• Collect Relevant Data and Apply Appropriate Data Analysis Techniques
• Conduct a Formative Evaluation
• Conduct a Summative Evaluation
• Assess Program/Policy Options
• Effectively Communicate the Results of an Evaluation
• Discuss Common Ethical Issues Facing Evaluators

(1) Joint Committee on Standards for Educational Evaluation. 1994. The Program Evaluation Standards. (2nd ed.) Thousand Oaks: SAGE Publications.

(2) Rossi, Peter, Howard E. Freeman, and Mark W. Lipsey. 1999. Evaluation: A Systematic Approach. (6th ed.) Thousand Oaks: SAGE Publications.

(3) Readings on reserve in the Monroe Library (see attached reserve reading list and recommended additional readings)

(1) A comprehensive final exam on Saturday, October 27th from 9:00 a.m. – 12:00 noon (100 points);

(2) Exercise # 1 - Summary and oral presentation of five articles related to a specific evaluation topic or technique due on Saturday, October 6th (100 points); and

(3) Exercise # 2 - Design of a program or policy evaluation due on Monday, November 26th (100 points)

A  =   95  –  100
B+ = 90  –    94
B  =   85  –   89
C+ = 80  –   84
C  =   75  –   79

(1) Class attendance is a requirement of this course. Absence from each class will result in a letter grade reduction in one’s final grade for the course.

(2) Preparation for class by completing the assigned readings and participating in class discussions is required.

(3) Students are required to meet all deadlines associated with exams and assignments in this course.

(4) Students should keep a copy of all work submitted to the Instructor since I keep all tests, papers, and exercises on file until the semester following the completion of the course.

(5) Students are responsible for all course materials. The final exam may include information from readings, lectures, class presentations/discussions, guest speakers, etc.

(6) Any student needing special consideration for a physical disability and/or learning difficulty should meet with me before the first class and I will do what I can to assist you.

(7) All assignments for this class must be typed.

Each student will select and summarize five journal articles related to a specific evaluation model, technical issue, or substantive area that has appeared in the literature over the past decade. Prior approval of your topic is strongly recommended. A one-page summary of each article should be completed along with an overall statement of key points, similarities, and/or differences emerging from all of the selected materials. Students will report their major findings in a 10-15 minute oral presentation to the class on Saturday, October 6, 2001.

Exercise # 2 - Evaluation Term Project*

Your task is to choose a social intervention program or policy area and develop an appropriate evaluation design. This assignment involves planning an evaluation and writing a paper describing how you would go about conducting the evaluation. It may be helpful to consider the following questions as you begin organizing your project:

• What is the purpose of the evaluation?
• What is the structure and context of the program/policy to be evaluated?
• What are the major goals and objectives of the program/policy?
• Why is the program/policy being evaluated?
• What are the major issues and questions with which the evaluation is to deal?
• What methods will be used to answer evaluation questions?
• What are the resources available for the evaluation?
• What relationship will be developed with the stakeholders during the evaluation?
• How is program implementation being assessed?
• Is sampling required? Where and how?
• What are the primary data sources?
• How will the data be collected?
• How will data quality issues be addressed?
• What statistical approaches will be used to analyze evaluation data?
• What are the strengths and weaknesses of the evaluation methods?
• What will be the reporting procedures?
• What recommendations are likely to be included in the evaluation?
• How will the report recommendations be implemented?

*The following is a partial list of journals devoted to program and policy analysis:
 

• Evaluation Review: A Journal of Applied Social Research (Sage Publications)
• American Journal of Evaluation (JAI Press)
• New Directions for Evaluation (Jossey-Bass)
• Evaluation and Program Planning (Pergamon)
• Evaluation & the Health Professions (Sage Publications)
• Educational Evaluation and Policy Analysis (American Educational Research Association)
• Assessment and Evaluation in Higher Education

First Weekend - September 7th and 8th

Readings:  Rossi, Freeman, and Lipsey Ch. 1-3 and Ch. 12; The Program Evaluation Standards 1994; Carter 1994; Chelimsky 1997; Cook 1997

Topics:    Introduction to Evaluation Research
                History of the Field
                Applied versus Basic Research
                Disciplinary Backgrounds of Evaluators
                Tailoring Evaluations
                Frameworks of Evaluation
                Evaluation Utilization
                Ethical Issues in Evaluation

Second Weekend - October 5th and 6th  (Student presentations Saturday, October 6th of Exercise #1)

Readings:  Rossi, Freeman, and Lipsey Ch. 4-6; Caudle 1994; Dean 1994; Hatry 1994; Scheirer 1994; Weiss 1997

Topics:    Needs Assessment
                Evaluability Assessment
                The Role of Theory in Program Design and Evaluation
                Program Monitoring/Implementation
                Data Collection Strategies
Third Weekend - October 26th and 27th  (Student final exam on Saturday, October 27th, 9:00 a.m. – 12:00 noon)

Readings: Rossi, Freeman, and Lipsey Ch. 7-11; Miller 1994; Newcomer 1994; Hendricks 1994; Lipsey 1997; Levin 2001)

Topics:    Program Impact and Outcomes Assessment
                Experimental and Quasi-Experimental Designs
                Data Collection Strategies
                Approaches to Data Analysis
                Meta-Analysis in Evaluation
                Cost-Benefit Analysis
                Reporting and Communicating Evaluation Findings

Books and Articles on Reserve:

Carter, Reginald. 1994. “Maximizing the Use of Evaluation Results” Pp. 576-589 in Handbook of Practical Program Evaluation, edited by Joseph H. Wholey, Harry P. Hatry, and Kathryn E. Newcomer. San Fransisco: Jossey-Bass Publishers.

Caudle, Sharon L. 1994. “Using Qualitative Approaches” Pp. 69-95 in Handbook of Practical Program Evaluation, edited by Joseph H. Wholey, Harry P. Hatry, and Kathryn E. Newcomer. San Fransisco: Jossey-Bass Publishers.

Chalk, Rosemary and Joel H. Garner. 2001. “Evaluating Arrest for Intimate Partner Violence: Two Decades of Research and Reform.” New Directions for Evaluation 90:9-23.

Chelimsky, Eleanor. 1997. “The Political Environment of Evaluation and What it Means for the Development of the Field” Pp. 53-68 in Evaluation for the 21st Century: A Handbook, edited by Eleanor Chelimsky and William R. Shadish. Thousand Oaks: SAGE Publications.

Cook, Thomas D. 1997. “Lessons Learned in Evaluation Over the Past 25 Years” Pp. 30-52 in Evaluation for the 21st Century: A Handbook, edited by Eleanor Chelimsky and William R. Shadish. Thousand Oaks: SAGE Publications.

Dean, Debra L. 1994. “How to Use Focus Groups” Pp. 338-349 in Handbook of Practical Program Evaluation, edited by Joseph H. Wholey, Harry P. Hatry, and Kathryn E. Newcomer. San Fransisco: Jossey-Bass Publishers.

Fetterman, David M. 1997. “Empowerment Evaluation and Accreditation in Higher Education” Pp. 381-395 in Evaluation for the 21st Century: A Handbook, edited by Eleanor Chelimsky and William R. Shadish. Thousand Oaks: SAGE Publications.

Gueron, Judith M. 1997. “Learning about Welfare Reform: Lessons from State-Based Evaluations.” New Directions for Evaluation 76:79-94.

Hatry, Harry P. 1994. “Collecting Data from Agency Records” Pp. 374-385 in Handbook of Practical Program Evaluation, edited by Joseph H. Wholey, Harry P. Hatry, and Kathryn E. Newcomer. San Fransisco: Jossey-Bass Publishers.

Hendricks, Michael. 1994. “Making a Splash: Reporting Evaluation Results Effectively” Pp. 549-575 in Handbook of Practical Program Evaluation, edited by Joseph H. Wholey, Harry P. Hatry, and Kathryn E. Newcomer. San Fransisco: Jossey-Bass Publishers.

Kessler, D.A. and S. Duncan. 1996. “The Impact of Community Policing in Four Houston Neighborhoods.” Evaluation Review 6 (20):627-669.

Levin, Henry M. 2001. “Waiting for Godot: Cost-Effectiveness Analysis in Education.” New Directions for Evaluation 90:55-68.

Light, Jennifer S. 2001. “The Effects of Privatization on Public Services: A Historical Evaluation Approach.” New Directions for Evaluation 90:25-39.

Lipsey, Mark W. 1997. “What Can You Build with Thousands of Bricks? Musings on the Cumulation of Knowledge in Program Evaluation.” New Directions for Evaluation 76:7-23.

Miech, Edward, Bill Nave, and Frederick Mosteller. 2001. “Large-Scale Professional Development for Schoolteachers: Cases from Pittsburgh, New York City, and the National School Reform Faculty.” New Directions for Evaluation 90:83-99.

Miller, Thomas I. 1994. “Designing and Conducting Surveys” Pp. 271-292 in Handbook of Practical Program Evaluation, edited by Joseph H. Wholey, Harry P. Hatry, and Kathryn E. Newcomer. San Fransisco: Jossey-Bass Publishers.

Newcomer, Kathryn E. 1994. “Using Statistics Appropriately” Pp. 389-416 in Handbook of Practical Program Evaluation, edited by Joseph H. Wholey, Harry P. Hatry, and Kathryn E. Newcomer. San Fransisco: Jossey-Bass Publishers.

Scheirer, Mary Ann. 1994. “Designing and Using Process Evaluation” Pp. 40-68 in .) Handbook of Practical Program Evaluation, edited by Joseph H. Wholey, Harry P. Hatry, and Kathryn E. Newcomer. San Fransisco: Jossey-Bass Publishers.

Stoto, Michael A. 2001. “Preventing Perinatal Transmission of HIV: Target Programs, Not People.” New Directions for Evaluation 90:41-53.

Weiss, Carol H. 1997. “Theory-Based Evaluation: Past, Present, and Future.” New Directions for Evaluation 76:41-55.

Alkin, Marvin C. 1990. Debates on Evaluation. Newbury Park: SAGE Publications.

Berk, Richard A. and Peter H. Rossi. 1999. Thinking About Program Evaluation. (2nd Ed.) Thousand Oaks: SAGE Publications.

Boruch, Robert. 1987. “Conducting Social Experiments.” New Directions for Program Evaluation 34:45-66.

Campbell, D.T. and J.C. Stanley. 1966. Experimental and Quasi-Experimental Designs for Research. Chicago: Rand McNally.

Cook, T. and D. Campbell. 1979. Quasi-Experimentation. Chicago: Rand McNally.

Cronbach, L.J. 1982. Designing Evaluations of Educational and Social Programs. Menlo Park, CA: Jossey-Bass.

Devine, Joel A., James D. Wright, and Laurie M. Joyner. 1994. “Issues in Implementing Randomized Experiments in Field Settings: The New Orleans Homeless Substance Abuse Program.” New Directions for Program Evaluation 63:27-40.

Lincoln, Yvonna S. and Egon G. Guba. 1986. “But Is It Rigorous? Trustworthiness and Authenticity in Naturalistic Evaluation.” New Directions for Program Evaluation 30:73-83.

Patton, M.Q. 1997. Utilization Focused Evaluation. 3rd Ed. Thousand Oaks, CA: SAGE Publications.

Rossi, Peter H. 1987. “The Iron Law of Evaluation and Other Metallic Rules.” Research in Social Problems and Public Policy 4:3-20.

Rossi, Peter H. 1994. “The War Between the Quals and the Quants: Is Lasting Peace Possible?” New Directions for Program Evaluation 61:23-36.

Scriven, Michael. 1991. Evaluation Thesaurus (4th Ed.) Newbury Park: SAGE Publications.

Stufflebeam, Daniel L. 2001. “Evaluation Models.” New Directions for Evaluation 89:1-98.

Weiss, C. 1972. Evaluation Research.  Englewood Cliffs, NJ: Prentice Hall.

Wholey, Joseph S., Harry P. Hatry, and Kathryn E. Newcomer. 1994. Handbook of Practical Program Evaluation. San Fransisco: Jossey-Bass Publishers.

---

CITY COLLEGE HOME PAGE  (http://www.loyno.edu/citycollege/)                     LOYOLA UNIVERSITY HOME PAGE (http://www.loyno.edu)

"The contents of this communication are the sole responsibility of Dr. Laurie Joyner and do not necessarily represent the opinions or policies of Loyola University New Orleans."

Copyright © 1996-2001 Loyola University New Orleans

Last updated on August 15, 2001