An adversary evaluation approach in policy analysis is one which reflects a valuing orientation. [1] This approach developed in response to the dominant objectifying approaches in policy evaluation [2] and is based on the notions that: 1) no evaluator can be truly objective, and, 2) no evaluation can be value-free. [3] To this end, the approach makes use of teams of evaluators who present two opposing views (these teams are commonly referred to as adversaries and advocates). These two sides then agree on issues to address, collect data or evidence which forms a common database, and present their arguments. A neutral party is assigned to referee the hearing, and is expected to arrive at a fair verdict after consideration of all the evidence presented. [4]
There are many different models for adversary evaluations, including judicial, congressional hearing and debate models. However, models which subscribe to a legal-framework are most prominent in the literature. [5]
The judicial evaluation model is an adaptation of legal procedures for an evaluative framework. Unlike legal adversary hearings, the objective of this approach is not to win, but rather to provide a comprehensive understanding of the program in question. [2] [4] [5] This model assumes that it is impossible for an evaluator not to have a biasing impact. Therefore, the focus of these evaluations shifts from scientific justification to public accountability. [2] Multiple stakeholders are involved, and this approach aims at informing both the public, and those involved in the evaluation about the object of evaluation. While the model is flexible, it usually incorporates a hearing, prosecution, defence, a jury, charges and rebuttals. [3] Dependent upon the evaluation in question, this model may also incorporate pre-trial conferences, direct questioning and redirected questions, and summaries by prosecution and defence (Owens, 1973). [1] Proponents of this model, however, stress the importance of carefully adapting the model to the environment in which it is deployed, and the policy which it intends to address.
While flexibility is encouraged when implementing an adversary evaluation, some theorists have attempted to identify the stages of specific adversary models.
Wolf (1979) [2] and Thurston, [6] propose the following four stages for a judicial evaluation:
Owens (1973) [2] provides a more detailed description of the hearing stage in an advocate-adversary setting. He attributes the following characteristics to this aspect of the model (list adapted from Crabbe & Leroy, p. 129):
The following are identified as benefits of using an adversarial approach:
According to Smith (1985), [4] many of the limitations of this approach relate to its competitive nature, the complexity of the process, and the need for skilled individuals willing to perform the various roles needed for a hearing. Listed are the main limitations of the adversary evaluation:
Although currently out of favour, this approach has been used quite extensively in the field of educational evaluation (Owens, 1973). [4] It has also been applied to ethnographic research (Schensul, 1985) [4] and the evaluation of state employment agencies (Braithwaite & Thompson, 1981). [4]
Crabbe and Leroy [2] contend that an adversary approach to evaluation should be beneficial when:
Popham and Carlson [7] proposed that adversary evaluation was flawed based on the following six points:
Popham and Carlson, [7] however, were in turn criticised by others in the field. Gregg Jackson [8] argues that these criticisms do a "gross injustice" (p. 2) to adversary evaluation. He proposes that the only valid criticism amongst those listed is "difficulty in framing issues" (p. 2), stating that the other points are unfair, untrue or exaggerated. He further noted that Popham and Carlson [7] seemed to hold adversary evaluation to a higher or different standard to other forms of evaluation. Thurston [6] argues in line with Jackson, [8] but proposes two alternative criticisms of adversary evaluation. He states that issue definition and the use of the jury pose major problems for this approach.
Finally, Worthen [5] notes that at present there is little more than personal preference which determines which type of evaluation will best suite a program. Crabbe and Leroy [2] caution that all evaluations should be approached with regard to their unique needs and goals, and adjusted and implemented accordingly; there is unlikely to be one approach which satisfies the needs of all programs.
An adversary evaluation approach in policy analysis is one which reflects a valuing orientation. [1] This approach developed in response to the dominant objectifying approaches in policy evaluation [2] and is based on the notions that: 1) no evaluator can be truly objective, and, 2) no evaluation can be value-free. [3] To this end, the approach makes use of teams of evaluators who present two opposing views (these teams are commonly referred to as adversaries and advocates). These two sides then agree on issues to address, collect data or evidence which forms a common database, and present their arguments. A neutral party is assigned to referee the hearing, and is expected to arrive at a fair verdict after consideration of all the evidence presented. [4]
There are many different models for adversary evaluations, including judicial, congressional hearing and debate models. However, models which subscribe to a legal-framework are most prominent in the literature. [5]
The judicial evaluation model is an adaptation of legal procedures for an evaluative framework. Unlike legal adversary hearings, the objective of this approach is not to win, but rather to provide a comprehensive understanding of the program in question. [2] [4] [5] This model assumes that it is impossible for an evaluator not to have a biasing impact. Therefore, the focus of these evaluations shifts from scientific justification to public accountability. [2] Multiple stakeholders are involved, and this approach aims at informing both the public, and those involved in the evaluation about the object of evaluation. While the model is flexible, it usually incorporates a hearing, prosecution, defence, a jury, charges and rebuttals. [3] Dependent upon the evaluation in question, this model may also incorporate pre-trial conferences, direct questioning and redirected questions, and summaries by prosecution and defence (Owens, 1973). [1] Proponents of this model, however, stress the importance of carefully adapting the model to the environment in which it is deployed, and the policy which it intends to address.
While flexibility is encouraged when implementing an adversary evaluation, some theorists have attempted to identify the stages of specific adversary models.
Wolf (1979) [2] and Thurston, [6] propose the following four stages for a judicial evaluation:
Owens (1973) [2] provides a more detailed description of the hearing stage in an advocate-adversary setting. He attributes the following characteristics to this aspect of the model (list adapted from Crabbe & Leroy, p. 129):
The following are identified as benefits of using an adversarial approach:
According to Smith (1985), [4] many of the limitations of this approach relate to its competitive nature, the complexity of the process, and the need for skilled individuals willing to perform the various roles needed for a hearing. Listed are the main limitations of the adversary evaluation:
Although currently out of favour, this approach has been used quite extensively in the field of educational evaluation (Owens, 1973). [4] It has also been applied to ethnographic research (Schensul, 1985) [4] and the evaluation of state employment agencies (Braithwaite & Thompson, 1981). [4]
Crabbe and Leroy [2] contend that an adversary approach to evaluation should be beneficial when:
Popham and Carlson [7] proposed that adversary evaluation was flawed based on the following six points:
Popham and Carlson, [7] however, were in turn criticised by others in the field. Gregg Jackson [8] argues that these criticisms do a "gross injustice" (p. 2) to adversary evaluation. He proposes that the only valid criticism amongst those listed is "difficulty in framing issues" (p. 2), stating that the other points are unfair, untrue or exaggerated. He further noted that Popham and Carlson [7] seemed to hold adversary evaluation to a higher or different standard to other forms of evaluation. Thurston [6] argues in line with Jackson, [8] but proposes two alternative criticisms of adversary evaluation. He states that issue definition and the use of the jury pose major problems for this approach.
Finally, Worthen [5] notes that at present there is little more than personal preference which determines which type of evaluation will best suite a program. Crabbe and Leroy [2] caution that all evaluations should be approached with regard to their unique needs and goals, and adjusted and implemented accordingly; there is unlikely to be one approach which satisfies the needs of all programs.