Research Design and the Use of
Experiments in Political Science
5.-8. Juni, 2000
Bergen, Norge
Kursledere:
Professor Richard E. Matland
LOS-Senteret
Universitetet i Bergen
Professor Rick K. Wilson
Department of Political
Science
Rice University
Course Objective:
This course has as its primary
mission the consideration of experiments in political science. We are unabashed advocates of
methodological pluralism and of the use of experiments. Through this course we hope to introduce
the next generation of Nordic political
scientists to the use of experiments in political science.
Among the topics we wish to
discuss are the strengths (and weaknesses) of experiments, the role of
experiments in a multiple methods arsenal, how experiments are presently being
used in different areas in political science, and how one could bring an
experimental approach to bear to the study of political science issues students
are interested in. In our discussion of
experimental design issues, we will deal with questions of sample selection,
the strengths and weaknesses of field research versus laboratory research, the
manner in which treatment variables can be introduced, and questions of
experimental research ethics. In
addition to questions of how to do experimental research we will also review a
variety of fields in political science where experimental research has advanced
our knowledge. Among the specific
areas of political science in which experiments have made a noticeable contribution
are collective action theory, theories of distributive justice, the effect of
institutional rules on policy outcomes, and evaluation of women as political
candidates. We will review each of
these. We will also look at experiments
as a source of information on survey bias and how experiments can be used in
applied public policy settings.
We will set off time for
students to be able to discuss possible experiments that they are considering
in their own research. Questions of
design, sample selection, reasonable independent and dependent measures will
all be raised. The course has a total
of 24 hours of class time and is a 3 credit course for Nordic doctoral students
(3 vekttallskurs). Students desiring to
receive full credit for the course are expected to be present for the seminars,
actively participate, and write a 15-25 page paper which will be due on October
15th. The papers may be written in a
Scandinavian language or English. We
would urge students to consider writing in English so that after revisions the
papers can be sent out to international journals for publication consideration.
The course will be taught in English, men kursleder er ganske stø i Norsk, så
det vil ikkje by på noko som helst problem å ta opp spørsmål på Norsk.
Required Textbooks
Donald R. Kinder and Thomas R.
Palfrey (1993) Experimental Foundations of Political Science University
of Michigan Press.
Donald T. Campbell and Julian
C. Stanley (1963) Experimental and
Quasi-Experimental Designs for Research.
Chicago: Rand McNally College
Publishing Co.
Daniel Friedman and Shyam
Sunder. (1994) Experimental Methods
London: Cambridge University Press.
COURSE OUTLINE: REQUIRED
READINGS
Note: All readings on this syllabus are
"required". We have starred
(**) one reading in each section as the reading deserving the most care in
reading. There is a Recommended Reading
List attached. The readings on the
Recommended Reading list are supplemental and are only for those who have an
interest in greater reading in a specific area. We may, however, ask each participant to choose one of the
recommended readings from the list and present to the class.
MONDAY
10:00-12 am Introduction to Course.
OBJECTIVES: Introduce ourselves and the course. Discuss the variety of experiments used in the social
sciences. The range includes examples
taken from rational choice to social psychology to attitude formation in
developing survey instruments to applied field experiments. Have students participate in a series of
experiments.
**Donald R. Kinder and Thomas
R. Palfrey "On behalf of an
experimental political science."
In Donald R. Kinder and Thomas R. Palfrey (eds.) Experimental Foundations of Political Science
University of Michigan Press, pp. 1-39.
1:00-3:00 pm Basics of Experiments and Multiple
Methods
OBJECTIVES: What is the goal of social science? We argue it is the ability to specify causal relationships. The importance of multiple methods to
reaching that goal will be discussed.
Time will be spent discussing the concept of causality. Special emphasis will be laid on the
essential elements in experimental
design: randomization and unobtrusive measures.
**Friedman and Sunder,
1994. Experimental Methods. pp. 1-37.
Campbell and Stanley.
1963. Experimental and
Quasi-Experimental Designs for Research.
pp. 1-34.
3:00 - 5:00 pm Collective
Action
OBJECTIVES: Introduce students to the problem of collective action. Common Pool Goods and Public Goods dilemmas
will be discussed. Some initial
experimental results will be presented.
We will discuss predictions of how a Nordic population might react if
placed in similar experiments. We will start designing our own collective
action experiment within a Nordic context.
We will urge students to start thinking about how to design an
experiment that might tie in with their own interests.
**Elinor Ostrom 1990. Governing the Commons: The
Evolution of Institutions for Collective Action Cambridge: Cambridge University Press, Chapter 1 "Reflections on the
Commons", pp. 1-28.
Ostrom, V. and
E. Ostrom. 1977. "Public
Goods and Public Choices," Pp. 7-49 in Alternatives for Delivering
Public Services: Toward Improved
Performance. (E. S. Savas,
ed.) Westview Press.
Palfrey, Thomas R. 1993.
"The Conflict Between Private Interests and the Common
Good." In Donald R. Kinder and
Thomas R. Palfrey (eds.) Experimental Foundations of Political Science University of
Michigan Press, pp. 211-219.
TUESDAY
10:00-12:00 Experimental
Research Design: Internal Validity
Considerations.
OBJECTIVES: We shall discuss how internal validity concerns relate to
experimental design. While experiments
have a big advantage over other forms of research in terms of internal
validity, there are still a set of internal validity concerns that must be
faced when developing experiments.
**Friedman and Sunder,
1994. Experimental Methods. pp. 38-73.
Messick, D. M. and C. G.
McClintock. 1968. "Motivational Bases of Choice in
Experimental Games." Journal of
Experimental Social Psychology
4:1-25.
Komorita, S. S. and J. M.
Barth. 1985. "Components of Reward in Social Dilemmas." Journal of Personality and Social Psychology 48:364-373.
1:00 - 3:00 pm Collective Action -- Operational Design
Considerations
OBJECTIVES: In this section we want
students to seriously think about how to operationalize their question. What are the appropriate variables? What controls might we want to introduce? How will we operationalize and manipulate
our key INDEPENDENT VARIABLE?
**Isaac, R. Mark, James M.
Walker, and Susan H. Thomas.
"Divergent Evidence on Free Riding: An Experimental Examination of Possible Explanations." In Donald R. Kinder and Thomas R. Palfrey
(eds.) Experimental Foundations of
Political Science University of Michigan Press, pp. 265-302.
Dawes, Robyn M., John M.
Orbell, Randy T. Simmons, and Alphons J. C. van de Kragt. "Organizing Groups for Collective
Action." In Donald R. Kinder and
Thomas R. Palfrey (eds.) Experimental Foundations of Political Science University of
Michigan Press, pp. 245-263.
van Dijk, Eric and Henk
Wilke. (1995) "Coordination Rules in Asymmetric Social Dilemmas: A Comparison between Public Good Dilemmas
and Resource Dilemmas." Journal
of Experimental Social Psychology
31: 1-27.
3:00 - 5:00 pm Case Studies: Experimental Studies of
Distributive Justice
OBJECTIVES: This section should force students to think about the importance
of specifying a theoretical set of linkages.
We will emphasize the ways in which these papers take a "big"
idea and then step back to test out specific linkages. Emphasis will be placed on identifying how
broader theories of distributive justice are operationalized into an
experimental context.
**Richard E. Matland, John T. Scott, Toril Aalberg, Brian Bornstein, Philip A. Michelbach and Ola
Listhaug. (1999) An Experimental Study of Distributive Justice Norms Across Cultures.
Norman Frohlich and Joe A.
Oppenheimer. (1990) Choosing Justice in
Experimental Democracies with Production
American Political Science Review 84(2): 461-477.
Mitchell, George, Philip E. Tetlock, Barbara A.
Mellers, and Lisa Ordunez. 1993. “Judgments of Social Justice: Compromises
Between Equality and Efficiency”, Journal of Personality and Social
Psychology 65: 629-639.
9:30-11:00 Experimental Research Design: Design Issues and Ethics
OBJECTIVES: To clearly enunciate the ethical code that experimental
researchers must adhere to. Clearly
stipulate what is acceptable and unacceptable behavior by a researcher. Where the limits go and why they are placed
there. Discuss the possible ethical dilemmas involved in our collective action
experiment.
**Milgram, Stanley 1974.
Obedience to Authority: An
Experimental View. New York: Harper and Row, Chapters 1-2 and Appendix I
(pp. 1-26; 193-202).
Friedman and Sunder,
1994. Experimental Methods: A Primer for Economists. pp. 74-109.
Cooper, Joel. 1976.
"Deception and Role Playing:
On Telling the Good Guys from the Bad Guys." American Psychologist 31: 605-610.
Bonetti, Shane. 1998.
"Experimental Economics and Deception." Journal of Economic
Psychology 19(3): 377-95.
11:00 – 12:30 pm Collective
Action Choices -- Construct Validity, the case of Communication
OBJECTIVES: In this section we want students to think more about
operationalization. Discuss threats to
construct validity. We will spend a
great deal of time thinking about how to create and measure theoretical
constructs within the context of an experiment.
**Wilson, Rick K.
and Jane Sell. 1997. “‘Liar, Liar ...’ Cheap Talk and Reputation in Repeated Public Goods
Settings.” Journal of Conflict
Resolution 41 (5): 695-717.
Orbell, John M., Alphons J. C.
van de Kragt, and Robyn M. Dawes.
1988. “Explaining
Discussion-Induced Cooperation” Journal
of Personality and Social Psychology
54: 811-819.
Ostrom, E.,
J. M. Walker, and R. Gardner.
1992. "Covenants with and
without a sword: Self-governance is
possible." American Political
Science Review 86: 404-417.
1:30 - 3:15 pm Case Studies: Experimental Evidence on Candidate Evaluation and Bias in Survey
Research Instruments
OBJECTIVES: This section provides a series of studies that use experiments to
look at questions either in survey applications or questions of how women are
evaluated as candidates. We will
emphasize the unobtrusive nature of experiments that allow us to tap
information that would be impossible to get at otherwise.
**Richard E. Matland (1995)
"Kjønnstereotype forestillinger om politikere: en
eksperimentell studie av likestillingen i Norge."
i Kjønn og Politikk ed. Nina Raaum.
D.
King and R. Matland (1999). “Partisanship and the Impact of Candidate Gender in
Congressional Elections: Results of an
Experiment”. (manuscript, R&R at
JOP)
Howard Schuman and Lawrence
Bobo. "Survey Based Experiments on White Racial Attitudes toward
Residential Integration." In
Donald R. Kinder and Thomas R. Palfrey (eds.) Experimental Foundations of Political Science University of
Michigan Press, pp. 53-78
Sullivan, John L., James
Pierson and George E. Marcus.
"Ideological Constraint in the Mass Public: A Methodological Critique and Some New
Findings." In Donald R. Kinder
and Thomas R. Palfrey (eds.) Experimental Foundations of Political Science University of
Michigan Press, pp. 79-96.
3:15 – 5:00 Case Studies: Experimental
Evidence on the Effect of Political Institutions
OBJECTIVES: This section provides a series of cases which use experiments to look at questions of the effects of political institutions. We emphasize the control experiments provide so that the relevant factors can be carefully studied and analyzed.
**Fiorina, Morris P. and
Charles R. Plott. "Committee
Decisions under Majority Rule: An
Experimental Study." In Donald
R. Kinder and Thomas R. Palfrey (eds.) Experimental Foundations of Political Science University of
Michigan Press, pp. 399-433.
Palfrey, Thomas R. "Agendas and Decisions in
Government." In Donald R. Kinder
and Thomas R. Palfrey (eds.) Experimental Foundations of Political Science University of
Michigan Press, pp.389-397.
10:00 - 12:00 am Experimental Research Design: External Validity
Objectives: This section will talk about the use of
field experiments and the tradeoffs between impact and control. Guidance on how to adapt experiments to
external environments will be emphasized.
**Campbell and Stanley.
1963. Experimental and
Quasi-Experimental Designs for Research.
pp. 34-71.
1:00-3:00 pm Case Studies: Applying The Experimental Method to Public Policy Questions.
OBJECTIVES: This section provides a
series of cases which use experiments to look at questions of public policy. We
will emphasize the relevance of experiments in solving applied public policy
questions.
**Weiss, Janet A. "Coping with Complexity: An Experimental Study of Public Policy
Decision-Making." In Donald R.
Kinder and Thomas R. Palfrey (eds.) Experimental Foundations of Political Science University of
Michigan Press, pp. 185-208.
Dyer, J. S.
and R. E. Miles. 1976. "An Actual Application of Collective
Choice Theory to the Selection of Trajectories for the Mariner Jupiter/Saturn
1977 Project." Operations
Research 24: 220-44.
Iyengar, Shanto, Mark D.
Peters, and Donald R. Kinder.
"Experimental Demonstrations of the 'Not-So-Minimal' Consequences
of Television News Programs." In
Donald R. Kinder and Thomas R. Palfrey (eds.) Experimental Foundations of Political Science University of
Michigan Press, pp. 313-331.
3:00-5:00 pm Collective Action -- Laboratory Procedures
OBJECTIVES: In this segment we will spend
time talking about procedures for conducting an experiment. Hopefully by this point we'll have a
paper/pencil design that we can run. We
will go through the steps of running such an experiment and if possible will
run the experiment on a sample of available subjects.
Listed below are the
recommended readings for each section.
These readings supplement and fill out the required readings. We may ask students to read ONE of the
articles listed on this recommended readings list and report on it in class.
COURSE OUTLINE
10:00-12 am Introduction to Course
1:00-3:00 pm Basics of Experiments and Multiple Methods
Kaplan, Abraham. 1964. The Conduct of Inquiry: Methodology for Behavioral Science. San Francisco: Chandler Publishing Co.
Chapter 4 (Experiment), pp. 126-170.
Plott, C. R. 1979.
"The Application of Laboratory Experimental Methods to Public
Choice." In Collective Decision
Making: Applications from Public Choice
Theory Clifford Russell (ed.). Washington, D.C.: Resources for the Future, pp. 137-160.
King, Gary, Robert Keohane,
and Sidney Verba. 1994. Designing
Social Inquiry: Scientific Inference in
Qualitative Research. Princeton,
NJ: Princeton University Press, pp. 75-114.
Roth, Alvin E. 1994.
"Lets Keep the Con out of Experimental Econ.: A Methodological Note."
Empirical Economics 19(2): 279-89.
3:00 - 5:00 pm Collective Action
Hardin, Garrett 1968.
"The Tragedy of the Commons" Science 162:1243-1248.
Olsen, Mancur 1965. The Logic of Collective Action. Cambridge, MA: Harvard University Press.
Parks, Craig D. and Anh D.
Vu. 1994. "Social Dilemma Behavior of Individuals from Highly
Individualist and Collectivist Cultures."
Journal of Conflict Resolution.
38: 708-718.
TUESDAY
10:00-12:00 Experimental
Research Design: Internal Validity
Considerations.
Smith, Vernon. 1982.
"Microeconomic Systems as an Experimental Science" American
Economic Review 72: 923-955.
Marwell , G. and R. E.
Ames. 1979. "Experiments on the Provision of Public Goods, I: Resources, Interest, Group Size and the Free-Rider
Problem." American Journal of
Sociology 84:1335-1360.
1:00 - 3:00 pm Collective Action -- Operational Design
Considerations
Ledyard, John. 1995.
"Public Goods: A Survey of Experimental Research." In John H.
Kagel and Alvin Roth (eds.) The Handbook of Experimental Economics.
Princeton,
NJ: Princeton
University Press. pp. 111-194.
Messick, David M. and Marilyn
B. Brewer. 1983. "Solving Social Dilemmas: A Review." Review of Personality and Social Psychology 4:11-44.
Dawes, R. M. 1980. "Social Dilemmas." Annual Review of Psychology . 31: 169-193.
3:00 - 5:00 pm Case Studies: Experimental Studies of
Distributive Justice
Norman Frohlich and Joe A.
Oppenheimer. (1994) Choosing
Justice: An Experimental Approach to
Ethical Theory Berkeley, CA:
University of California Press.
Norman Frohlich, Joe A. Oppenheimer, and Cheryl Eavey (1987) " Choices of Principles of
Distributive Justice in Experimental Groups" American Journal of Political Science 31(3): 606-636.
Kjell Törnblom (1992) “The Social Psychology of Distributive Justice,”
in Justice:
Interdisciplinary Perspectives, ed. Klaus R. Scherer (pp.
177-236).
Kjell Törnblom and Uriel G. Foa (1983) “Choice of a Distribution
Principle: Crosscultural
Evidence on the Effects of Resources,” Acta
Sociologica 26:161-173
Virginia Murphy-Berman and John J. Berman (1984) “Factors Affecting
Allocation to Needy
and Meritorious Recipients: A Cross-Cultural
Comparison.” Journal of Personality
and Social Psychology 46(6):1267-1272.
Lissowski, Grzegorz and Piotr
Swistak (1995) "Choosing the Best Social Order: New Principles of Justice and Normative Dimensions of
Choice". American
Political Science Review 89(1):74-98.
WEDNESDAY:
9:30-11:00 Experimental Research Design: Design Issues and Ethics
American Psychological
Association. 2000. “Ethical prinipcles
of psychologists and code of conduct.”
Section 6, Teaching, Training Supervision, Research and Publishing.
Forward, John, Rachelle Canter
and Ned Kirsch. 1976. "Role-Enactment and Deception
Methodologies." American
Psychologist 31: 595-604.
Farr, James L. and W. Burleigh
Seaver. 1975. "Stress and Discomfort in Psychological Research: Subject Perceptions of Experimental
Procedures." American
Psychologist 30: 770-773.
11:00 – 12:30 pm Collective Action Choices -- Construct
Validity, the case of Communication
Kerr, Norbert L. and Cynthia
M. Kaufman-Gilliland. 1994.
“Communication, Commitment and Cooperation in Social Dilemmas” Journal of Personality and Social
Psychology 66: 513-529.
Sell, Jane, W. I. Griffith and
Rick K. Wilson. 1993. "Are Women
More Cooperative than Men in Social Dilemmas?
Evidence from Two Experiments."
Social Psychology Quarterly,
(September 1993) V. 56: 211-222.
1:30 - 3:15 pm Case Studies: Experimental Evidence on Candidate Evaluation and Bias in Survey
Research Instruments
Zaller, John (1992). The Nature and Origins of Mass Opinion. Cambridge:
Cambridge University Press.
Chapter 5, pp. 76-96.
Seth Thompson and Janie
Steckenrider (1997) “The Relative
Irrelevance of Candidate Sex”
Woman and
Politics 17(4): 71-92.
Virginia Sapiro (1982)
"If U.S. Senator Baker Were a Woman:
An Experimental Study of Candidate Images" Political Psychology 2:61-83.
Lee Sigelman and Carol K.
Sigelman (1982) "Sexism, Racism, and Ageism in Voting Behavior: An
Experimental Analysis" Social
Psychology Quarterly 45: 263-269.
R. Matland (1994) "Putting Scandinavian Equality to the
Test: An Experimental Evaluation
of Gender
Stereotyping of Political Candidates in a Sample of Norwegian Voters".
British
Journal of Political Science 24:
273-292.
Laurie E. Ekstrand and William
A. Eckert (1981) "The Impact of Candidate's Sex on Voter Choice” Western
Political Quarterly 34:78-87.
3:15 – 5:00 Experimental Evidence on the Effect of
Political Institutions
Levine, Michael E. and Charles
R. Plott. "Agenda Influence and
Its Implications." In Donald R.
Kinder and Thomas R. Palfrey (eds.) Experimental Foundations of Political Science University of Michigan Press, pp. 461-496.
Wilson, R. K. 1986.
"Forward and Backward Agenda Procedures: Committee Experiments on
Structurally Induced Equilibrium." Journal of Politics 48: 390-409.
Haney, Patrick, Roberta
Herzberg and Rick K. Wilson. 1992. "Advice and Consent: Unitary Actors, Advisory Models and
Experimental Tests." Journal of
Conflict Resolution 36:603-633.
THURSDAY
10:00 - 12:00 am Experimental Research Design: External Validity
Ellsworth, Phoebe (1977)
"From Abstract Ideas to Concrete Instances: Some Guidelines for Choosing Natural Research Settings. American Psychologist 32:604-615.
Erev, Ido, Gary Bornstein and
Rachely Galili. 1993. "Constructive Intergroup Competition as
a Solution to the Free Rider Problem: A
Field Experiment." Journal of
Experimental Social Psychology 29:
463-478.
Greenber, David H. and Philip K. Robins 1986. "The Changing Role of Social
Experiments in Policy Analysis, " Journal
of Policy Analysis and Management Vol. 5(2): 340-362.
1:00-3:00 pm Applying The Experimental Method to Public
Policy Questions.
W. Steven Barnett (1985) Benefit/Cost Analysis of the Perry Preschool
Program and its Policy Implications".
Educational Evaluation and Policy Analysis 7(4): 333-342.
Wesley A. Magat, John W.
Payne, and Peter F. Brucatio (1986)
"How Important is Information Format? An Experimental Study of Home Energy Audit Programs" Journal of Policy Analysis and Management
Vol. 6(1): 20-34.
3:00-5:00 pm Collective Action -- Laboratory Procedures