Volume:  19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1



A peer-reviewed electronic journal. ISSN 1531-7714 
Search:
Copyright 1999, PAREonline.net.

Copyright is retained by the first or sole author, who grants right of first publication to Practical Assessment, Research & Evaluation. Permission is granted to distribute this article for nonprofit, educational purposes if it is copied in its entirety and the journal is credited. PARE has the right to authorize third party reproduction of this article in print, electronic and database forms.


Scriven, Michael (1999). The nature of evaluation part i: relation to psychology. Practical Assessment, Research & Evaluation, 6(11). Retrieved September 2, 2014 from http://PAREonline.net/getvn.asp?v=6&n=11 . This paper has been viewed 51,082 times since 11/13/1999.

The Nature of Evaluation Part I: Relation to psychology

Michael Scriven,
Claremont Graduate University

The discipline of evaluation is devoted to the systematic determination of merit, worth, or significance. It is divided into fields according to the type of entity evaluated--for example, program evaluation, or personnel evaluation--and there are more than twenty of these recognized fields of evaluation. Some specific aspects of evaluation methodology have been developed to solve problems of evaluation in only one or a few of these fields (e.g., bias control in panel selection, systematic side-effect identification in program evaluation, road-testing techniques in product evaluation). However, the underlying logic of the process of evaluation--for example, the difference between merit and worth, or between grading and ranking--and a substantial portion of its general methodology (e.g., techniques of measurement, causality determination, applying the requirement of informed consent) are shared across all or many of these fields. Many of these general techniques (the 'general methodology') come from the applied social sciences, and are learnt by students in the normal course of education in those fields. But the logic of evaluation has been developed for and applies only to evaluation; and the field-specific methodologies of evaluation must also be mastered in order to deal with evaluation in the fields to which they apply. Teaching evaluation therefore focuses on these evaluation-specific topics, the general logic and the special methodologies.

This article addresses the role of evaluation, the basic logic, and a description of how the field is structured. A separate article describes some of the basic logic of evaluation skills and methodological skills that need to be mastered..

Evaluation in applied psychology

Just as it was previously found that a good grasp of probability and statistics had become an essential tool for a great deal of work in applied psychology, so we now find that a knowledge of the logic of evaluation and of some of its specialized methodologies are increasingly crucial for much investigation in applied social science. Funding agencies letting evaluation contracts or assigning staff to evaluation increasingly want to know not only what is happening when an intervention is supported, and exactly what causes the results - familiar questions for applied social science - but also 

  1. whether the intervention is worth what it cost, 
  2. whether there are unintended bad results as well as planned good ones, 
  3. whether the methods used in the intervention were proper by current professional and ethical standards, and 
  4. whether there are better ways to do the same thing. 

This is the domain of evaluation, and none of those questions can be answered reliably without some use of its special logic and its special methodologies.

The Basic Logic of Evaluation

In order to reach evaluative conclusions it is usually necessary to establish or identify two kinds of premises: 

factual premises (e.g., about nature, performance or impact) and 
value premises (e.g., about the relevant legal or scientific principles). 

There are usually many of these premises in the evaluation of complex entities (or entities with complex functions), and there may be several hundred of them. To obtain the required kind of overall evaluative conclusion, it is typically necessary to combine all of them by means of what is called 'the internal synthesis process.' This synthesis step is one of the key logical processes in evaluation and is a long way from the simple deduction and statistical inference that are more common elements in scientific inference.

Factual premises in a field like program evaluation are commonly established using the standard procedures of social science, sometimes with the assistance of other disciplines such as history and jurisprudence. Value premises usually come from one or more of eight sources: legal principles; scientific and mathematical standards of truth (especially relevant when the program disseminates information or is based on scientific theories or common assumptions); professional, cultural, or organizational standards of proper conduct (e.g., the APA testing standards); needs assessments; definitions (which provide linguistic standards of propriety); market research; logic; and ethics. Again, the social sciences are a common source for several of these types of values e.g., the scientific standards of truth used in judging the quality of the assertions--or assumptions--built into or propagated by the program. From psychology, we frequently encounter premises about maturational rates, cognitive processes, or leadership research. The logic of evaluation comes in with the frequent need to balance these value considerations when they conflict: that logic originates in jurisprudence and moral reasoning, but has been expanded to cover other fields of evaluation besides these, e.g., evaluation of alternatives in high-stakes decision-making.

Evaluation Fields

The better-known fields of applied evaluation vary considerably in quality as well as in their relevance to and dependence on the social sciences. While most of program and personnel evaluation is heavily dependent on the social sciences and capable of a high degree of objectivity and utility, others vary independently on these dimensions. Some come close to being pseudo-evaluative (e.g., wine tasting, art criticism), some are partially valid (architectural criticism, portfolio management, literary criticism), and some support highly valid evaluations but are not dependent on the social sciences (e.g., the reviews done by appellate courts, the evaluations of claimed proofs of Fermat's Theorem in mathematical journals). Eight of these applied fields are of particular importance, for practical or logical reasons: they fall into two groups. The 'Big Six' are the fields of program, personnel, performance, policy, proposal, and product evaluation (the latter including technology assessment). The 'Super Two' are: 

  1. intradisciplinary evaluation--the evaluation of the entities that are the currency of a discipline's economy such as theories, hypotheses, classifications, data, research designs and results, practitioners, contributions, and journals--and of the discipline itself: and 
  2. meta-evaluation, the evaluation of evaluations themselves, a practice that demonstrates the reflexive nature of evaluation and the reflective integrity of its practitioners. 

The first is the backbone of all disciplines--it is what makes them disciplines. The second is the backbone of evaluation--it is what makes it consistent by making it practice what it preaches.

Of the Big Six fields--the conventional fields of evaluation--program evaluation is the one with the largest associated job market at the moment, with personnel evaluation (an Human Resources staple) and performance evaluation (especially in the educational area, where its academic fountain is often referred to as 'tests and measurement') coming next.

Additional Reading

Chelimsky, E and Shadish W.R (eds) (1997) Evaluation for the 21st Century : A Handbook. Sage Publications. [amazon]

Joint Committee on Standards for Educational Evaluation (1998). Program Evaluation Standards : How to Assess Evaluations of Educational Programs. Corwin Press. [amazon]

Scriven, M. (1991) Evaluation Thesaurus 4th edition. Sage Publications. [amazon]

Shadish W. R. (Chair) (1998) Guiding Principles for Evaluators. A Report from the Americian Evaluation Association Task Force on Guiding Principles for Evaluators. [available online http://www.eval.org/EvaluationDocuments/aeaprin6.html].

Shadish, William (1998). Some Evaluation Questions. Practical Assessment, Research & Evaluation, 6(3). [Available online: http://pareonline.net/getvn.asp?v=6&n=3].

Descriptors: *Evaluation Methods; Evaluation Problems; Evaluation Utilization; Interdisciplinary Approach; *Meta Analysis; Performance Based Assessment; *Personnel Evaluation; *Program Evaluation; *Psychology; *Synthesis