PROGRAM PLANNING, IMPLEMENTATION, & EVALUATION

Instructor:  George E. Capowich, Ph.D. 

Voice:  865-2574 

FAX 865-3229 

Course Overview.This seminar focuses on the implementation and evaluation of programs and policy initiatives.(For purpose of clarity I will use the words program and policy interchangeably.)Whether we think of a program that is designed to increase public involvement in the criminal justice system (e.g., Neighborhood Watch), implement a change the way police focus their daily activities (e.g., problem-solving), or a pass new sentencing law (e.g., mandatory minimum sentences), we can analyze it by thinking of it as comprised of three factors:

1.  An underlying theoretical and logical rationale that is the foundation for the program’s design.

2.  A process of implementation that makes the program operational. 

3.  An expected impact.

This course examines each of these three areas in detail by devoting one of our weekend meetings to each topic.The emphasis is on understanding each topic and recognizing how they link together to form a guide for designing, implementing, and evaluating a specific program.

Course Objectives.1.  Understand and apply the theory and logic of evaluation research.

2. Review analytic methods and apply them to assessing implementation.

3.  Extend familiarity with using social science methods to assess impact.

4.  Expand skills of critical thinking and analysis.

Required Books:Berk, Richard A. & Peter H. Rossi (1999).Thinking About Program Evaluation (2^nd Edition).Newbury Park:Sage.

Patton, Michael Quinn (1997).Utilization-Focused Evaluation (3^rd Edition).Thousand Oaks:Sage.

Rossi, Peter H., Howard E. Freeman, Mark W. Lipsey (1999). Evaluation: A Systematic Approach (6^th Edition). Thousand Oaks: Sage.

Grades will be assessed according to the following scale (established MCJ scale):

A = 95-100C+ = 80-84

B+ = 90-94C = 75-79

B = 85-89D+ = 70-74

F = 69 and below

Course Requirements:

Students will be graded on the following assignments:

Class preparation and participation……………15%

Evaluation Exercises…………………………..10%

Journal article review (Separate Handout)…….20%

Evaluation Plan (Separate Handout)…………..35%

Class Presentation……………………………..20%

Attendance and participation.These are essential aspects of any graduate seminar, with attendance being particularly important in the compressed format being used for this class.Attendance at every class is required and every one is expected to be an active participant in the seminar discussions.

Evaluation exercises.At the end of each Saturday session I give you a set of questions and problems that cover that weekend’s topics.These exercises will be due in writing approximately one week later.These assignments may be turned in one of three ways:fax, as an email attachment (please let me know if you use a word processing program other than Word or WordPerfect), or in hard copy delivered to my office.

There will be separate handouts explaining the article review (this handout also covers the presentation) and evaluation plan that will be discussed in detail during the first class meeting.Nevertheless, let me explain briefly what each entails.

Journal Article Summary.This assignment requires that you find an article published in a peer reviewed journal that presents program evaluation results.You will then write a 4-5 page essay that reviews and critiques the article by using the information covered in the class.

Evaluation Plan.This is the major seminar paper for the class.You will need to find a description of a specific program and write an evaluation plan that explains how you would design an evaluation of the program.{Please note that you will NOT actually carry out the evaluation.Rather you will describe how you would evaluate the program if you had the opportunity.}

Late Assignments.Late assignments will be accepted only if prior arrangements are agreed upon or the lateness resulted from an emergency beyond your control.Computer problems such as, among other things, accidentally erased computer files, damaged disks, malfunctioning printers, careless roommates who delete files, etc. are not considered emergencies.The computer age we all live in requires planning and caution, so take steps to protect your work by making back-up files, keeping computer disks away from harsh elements, and not waiting until you are on your way to class to print an assignment. 

Conduct.Please refer to your Student Handbook for Loyola University’s rules of conduct.Academic study rests on principles of respect for people, their ideas, and their right to learn.Behavior that is rude, disrespectful, or disruptive not only violates school rules, but also violates individual dignity and interferes with the educational process.I encourage disagreement with and the questioning of ideas, but these must be done respectfully in the spirit of furthering everyone’s learning and promoting open discussion.This is the case whether the idea expressed comes from the reading, the professor, a guest speaker, or a student in the class.

In addition, please turn off cell phones and pagers before class begins.If you work at a profession that requires you to keep your beeper or phone on, please sit near the door, and if you get called, please take and complete the call in the hallway.

Since universities rely on individual honesty misdeeds such as cheating, plagiarism, or other misrepresentations and fabrications cannot be tolerated.Dishonesty in any form is unfair to others, embarrasses the individual who behaves in such a manner, and disrespects the educational process.Please do not do something that will sully the semester for you and for me.Violations of Loyola’s honesty and behavior code will result in an F for the course and will be handled according to procedures explained in the Handbook.

Disabilities.I recognize that some students have disabilities that do not reflect on their intelligence or capacity to learn, but present special challenges to learning.I am very willing to work with students who need some kind of accommodation.Please understand that if you confront a situation that requires accommodation because of a disability, it is your responsibility to notify Loyola’s Disability Resource Center (Monroe Hall, 4^th floor)) and me to make suitable arrangements at the beginning of the semester.

Teaching Philosophy & Approach.I incorporate several principles into my teaching.Each is briefly explained below:

1.Multiple methods facilitate learning.The class combines lecture, discussion, reading, writing, and independent research to enhance learning.Individuals learn differently, but whatever our strengths and learning styles, multiple learning methods reinforce each other to amplify learning.

2.Collaboration.I encourage students to study together if they wish and use each other as resources.Also, I encourage students to talk with me in class as well as during my office hours (8 hours are set aside each week) about the course and their work outside of class.If necessary, appointments also are available.Furthermore, I suggest students use email and the phone to communicate with me.Problems and difficulties in learning do not go away if left unattended, so please address any confusion or other difficulty you experience as soon as it arises.

3.Understanding & Critical Analysis.Lectures, discussions, and writing exercises are intended to promote understanding, link of ideas together, and integrate wide-ranging knowledge.The class structure and assignments are designed to assess knowledge, facilitate understanding, improve analysis, and promote critical thinking. Used in this way, the term critical thinking refers to using sound analytical processes (e.g., logic, theory, and evidence) to examine ideas.

The seminar topics are divided into three sections, one for each weekend that the class meets.The readings for each section are organized around a broad topic for each class session.There are three or four review questions listed for each section that are intended to guide your reading and study of the material.

Any program, regardless of focus, has some underlying rationale.The rationale may be explicit or implicit, but there is always some way to think about the program’s theoretical justification.This section of the course focuses on specific questions that include:

1.  How do social science and evaluation research combine?

2.  How do we define theory?

3.  How do we apply theory to specific programs and policies?

4.  How do we understand and analyze the logic that underlies a program?

 Date Topic Reading 

Review handout on evaluation plan

Setting the Stage

Evaluation and Social Science

The Nature and Role of TheoryBerk & Rossi (1999)

Evaluation Theory and ConceptsRossi et al.(1999) Ch. 1

Saturday A.M.(2/23)Program Design and Process EvaluationRossi, et al. Ch.2-4

Constructing a Logic ModelClassroomExercise 

Saturday P.M (2/23)  Program Theory Rossi, et al., Ch. 5

Program Theory and Evaluation Patton (1997), Ch. 1-5

*  Written answers to Section I exercises due by 5:00 P.M. on March 4^th via fax, email 

attachment, or hard copy in my office.

*  Full citation (see handout for details on how to cite) for journal article due by 5:00 P.M.

on March 8^th via fax, email, or hard copy in my office.

Once a program is designed with its goals, objectives, and components defined then it is time to think about implementation.This is a crucial phase in the program’s operation and an evaluation.Since the program has a specified design and purpose it will not be possible to analyze the program’s success or failure unless we know how closely the program’s routine activities are linked to the program’s design. If a program is implemented as designed and fails, then we can conclude it was a failure of the program’s theory.On the other hand if the well-implemented program succeeds, then we can conclude that it was a success of the program’s theory.If the program is not implemented as designed, however, then neither a success nor a failure can be attributed to the program’s theoretical foundation.A process evaluation is meant to monitor the program’s implementation in order to help the program’s operation and establish a firm footing for an impact evaluation.The following questions guide this section and the study of implementation and process evaluation:

1.  How does social and political context affect evaluation?

2.  How do we combine our knowledge of the program’s theory and structure to 

analyze implementation?

3.  What is the basic structure of a process evaluation, including measures?

4.  How do we link the process evaluation with the impact evaluation?

 Date Topic Reading 

Patton, Ch. 14

Focusing an EvaluationPatton, Ch. 6-8

Copy of program description for Evaluation Plan due in class.

Saturday A.M. (3/16)Establishing a Process EvaluationPatton, Ch. 9 &10

Review Evaluation Plan assignment

Review Article Review assignment

Saturday P.M. (3/16)Implementing a Process EvaluationRossi, et al., Ch. 6

Bridge between Process and Impact 

Process evaluation exercises given out in class.

_______________________________________________________

*  Written answers to Section II exercises due by 5:00 P.M. on March 22^nd via fax, email 

attachment, or hard copy in my office.

*  Article reviews due by 5:00 P.M. on March 29^th via fax, email attachment, or hard copy in my 

office.

*  Outline of Evaluation Plan due by 5:00 P.M. on April 5^th via fax, email attachment, or hard copy 

in my office.

After analyzing the implementation and operation of the program we are ready to turn our attention to effectiveness.In this section we are concerned with the results of the program; that is, whether the program had the desired effect on the problem or issue it was designed to address.Several questions guide our discussion of the issues related to assessing impact, including:

1.  How can we be sure that the evaluation results are attributable to the program?

2.  How differentiate between short- and long-term results?

3.  How do we develop sound impact measures?

 Date Topic Reading 

Friday (4/19)Return & Review Exercises

Regression Model in EvaluationLecture

Designing an Impact Assessment

Design and Quantitative Analysis Rossi & Freeman, Ch. 7-10

Saturday A.M. (4/20)Design and Qualitative AnalysisPatton, Ch 11-13; 15

Impact evaluation exercises given out in class.

Saturday P.M) (4/20)STUDENT PRESENTATIONS

_______________________________________________________

_______________________________________________________

*  Written answers to Section III exercises due by 5:00 P.M. on April 29^th via fax, email attachment, 

or hardcopy in my office.

* Evaluation Plan due by 5:00 P.M. on May 15^th via fax, email attachment, or hardcopy in 

my office.