Graziano and Raulin (8th ed)Graziano & Raulin
Research Methods (8th edition)

Research Design Checklist Tutorial

Many students feel overwhelmed by all the decisions and preparations that must be addressed to design and carry out quality research. With practice, these elements become second nature, but initially it just feels like a million different things to remember. To deal with this feeling of being overwhelmed, we added two features to the text. The first uses a flowchart system to help students make decisions about the appropriate statistical procedures to use with their research (Appendix D). We have included a tutorial on using the statistical flowchart elsewhere on this website. This unit provides a tutorial with examples on how to use the Research Design Checklist to verify that all key decisions have been made during the design and preparation phases of research.

Purpose of the Research Design Checklist

The Research Design Checklist organizes the complex task of pulling together all of the details behind a well designed research study. By itself, it will not design good research. Good research starts with good ideas, which are based on strong theories and a familiarity with the research done by others. But once you have a good idea for a research project, you have to get all the details right or the study will be a waste of time. That is what the research design checklist is all about--getting the details right, or at least not forgetting any critical details.

The research design checklist runs you through a number of critical components of a good research study. Not every component is critical to every study, but if you consider each of the elements in the checklist before you run your study, you are unlikely to forget any important elements. In time, the checklist will become less necessary as the design principles emphasized in this text become more second nature. By that time, research will have become an exciting enterprise aimed at finding answers to the mysteries around us.

Elements of the Research Design Checklist

A copy of the research design checklist is included on this website. You may find it useful to print it out or to follow along using the research design checklist in the text (Table E.1) as we quickly cover the elements here.

Initial Problem Definition

It is amazing how many students skip this first step, jumping into designing the research study without giving sufficient attention to the underlying question that they are attempting to answer. You actually can design a study this way, but it usually ends up being virtually worthless, because it was never designed to answer a specific, meaningful question about nature. 

To appreciate this fact, randomly select several studies from the research literature. Then randomly select one and use whatever sampling procedure it used to obtain participants. Randomly select another to determine how participants will be assigned to groups. Randomly select a third to determine what dependent measures will be used and a fourth to determine what statistical analyses will be used. You will have most of the elements of a research study, but it would be shocking if you had a study worth running. 

Starting the design of a study before you have thoroughly reviewed the research literature, clearly identified the problem you want to study, and identified and operationally defined the critical variables is only one step above the random design just described. We cannot stress enough how important these early problem definition steps are because ALL of the remaining elements will follow from them.

Research Hypothesis

If you have carefully defined the problem and identified and operationally defined the variables, it is a relatively easy task to state your research hypothesis. Remember that the research hypothesis puts all of these elements together, as well as suggests the appropriate research design to test the hypothesis. 

The rule of thumb is to word the research hypothesis to imply the use of the strongest research design that is practical and ethical. You may have to word the research hypothesis in several ways to identify the best approach. 

Whenever possible, an experimental design should be implied by the wording of the research hypothesis. Experiments are by far the most powerful research procedures in psychology. However, sometimes you simply cannot assign participants to be in the theoretically relevant groups defined in your problem statement. If you are interested in gender differences, for example, you cannot randomly assign participants to be male and female. At other times assigning to groups is unethical, such as when you might want to explore the results of head injuries. Assigning participants to be head injured is clearly unethical, but studying those who have experienced head injuries and comparing them with participants who have not is both ethical and practical. Remember that most problem statements can be translated into several different research hypotheses. See Chapter 8 for more details on this process.

Statistical Analysis

Even though the statistical analyses will not be performed until after the data have been collected, the specific analyses that will be conducted should be planned prior to data collection. The descriptive and inferential statistics can be planned in detail. General guidelines can usually be specified for the post hoc and secondary analyses, but the actual post hoc and secondary analyses will be determined by what is found in the research study. These secondary analyses often help to interpret the findings from the primary analyses or help to point to additional questions that should be pursued in later studies. The descriptive statistics should not only provide a description of the dependent measures, broken down by group, but should also include a description of the demographic characteristics of the sample. These latter descriptive statistics help to define how representative the sample is and often allow other researchers to compare the characteristics of their samples with the sample from your study. If you are unsure about what demographic variables are most important, check out what other researchers investigating the same or similar questions report in describing their sample.

The descriptive statistics will follow from the nature of the variables measured, and the inferential statistics will follow from the research design and hypotheses. The statistical flowchart, which was introduced in Appendix D, will help with these decisions. We also have included a functional version of the statistical flowchart, which walks you through the process and link you to instructions on how to compute the various statistics. 

A mistake made by many students is to conduct the inferential statistics, but not compute the critical descriptive statistics that would show the pattern of scores obtained. The inferential statistics only show if the groups are different. They do not by themselves show how the groups are different.

Theoretical Basis

The theoretical basis of the research should be a part of every design decision. This is a good point to stop and look at your basic design to verify that the theory on which it is based is clear and that the design is adequate to test the theoretically-derived research hypotheses.

Independent Variable Manipulation (experimental research)

If your study is an experiment, you will be manipulating the independent variable. You will need to operationally define how that manipulation will occur. In a sense, you already operationally defined the independent variable manipulation when you formulated the research hypothesis. However, at that level you usually do not get down to the nuts and bolts level of defining it. Here, you want to clarify in detail every step that you will take in the study to produce the desired manipulation of the independent variable. 

Whenever possible, it is best to rely on manipulations that have been used in past research, because then you will know that the manipulation is possible. If that is not the case, make sure that you have piloted your manipulation to make sure that it actually works. It is best to include a manipulation check to verify that the independent variable really was varied by the manipulation you used in your study.

Dependent Measures

As with the independent variable, you have already operationally defined the dependent variable at a general level when you developed the research hypothesis. At this stage, you are defining it right down to the smallest detail of the measurement process. If possible, it is best to use existing dependent measures with established reliability and validity. If that is not possible, the dependent measure in your study should be pilot tested at a minimum, and if possible, some initial reliability and validity data should be collected either before or during the study. We recommend that you routinely arrange to gather your data in such a way that you can compute the critical reliability indices for all of your dependent variables.

Controls

There are a dozen or more control procedures that can be applied to a given study. Some of the controls are a function of the design. For example, an experimental design with random assignment of participants to groups and a control group will control for many sources of confounding, thus protecting internal validity. But beyond the research design, there are a number of other controls that should be routinely considered for each study. Most of these are outlined in Chapter 9 of the text.

The set of control procedures that we outlined under the heading of general control procedures in Chapter 9 are relevant in virtually all research studies. The research setting should be carefully considered. It should be functional, as realistic as possible, and free of distraction that could add error variance to your measurements. For some studies, something as simple as a quiet room with a table and chairs may be all that is needed. For other studies, considerable effort is required to create a laboratory situation that is sufficiently life-like to be a valid test of real-world behavior. 

Keeping the research setting free from distractions and other elements that would add error variance will increase the statistical validity and increase the sensitivity of your study. Making it life-like and natural will increase the generalizability of your study. 

We have already discussed the importance of carefully developed operational definitions for all the variables measured or manipulated in the study. This is another of the general control procedures. Replication, the last of the general control procedures, is not a variable that we need to consider within an individual study. Nevertheless, it is useful to recognize that replication demonstrates the robustness of a finding.

There is a large class of potential confounding known as subject and experimenter effects. These effects are based on the knowledge of the researcher, the participant, or both about the research and the expected results. This knowledge, coupled with the desire by both the research and the participants to perform their roles as expected, can often distort the results in favor of the research hypothesis. Controlling these effects usually involve (1) keeping this critical knowledge from those involved in the research, (2) minimizing the effect that such knowledge might have, or (3) including procedures that would allow us to check on the extent of bias if it existed.

Blind procedures--either single-blind or double-blind--reduce the critical knowledge of what condition participants are in. In a drug study, for example, the single-blind procedure would prevent the participants from knowing whether they were receiving the active medication or a placebo. The pills would look the same. Usually they would be capsules so that there would be no taste to cue participants about whether they have the drug or the placebo. In a double-blind procedure, the doctor administering the medications and evaluating their impact on the patient would also be unaware of which patients got the drug and which ones got placebos. 

Using deception is another way of reducing critical knowledge that might produce subject effects. Making the participant think you are doing one thing, when in fact you are doing something quite different, will prevent them from distorting their actions to favor the research hypothesis.

Automating the study will prevent some of the influence that a researcher might have on the participants, because the participants are interacting with a machine instead of the researcher. Simply using objective, easy to quantify measures will often reduce experimenter bias dramatically. Including multiple observers will not only allow you to measure the interrater reliability of your measures, but will also tend to decrease the tendency to distort data to favor the hypothesis, because the observers know that their accuracy is being monitored and, therefore, will strive to achieve the highest accuracy possible.

The remaining general control procedures have to do with the selection and assignment of participants, and therefore will be covered in the next section.

Participants

The participants are a critical factor in psychological research. We want our sample of participants to represent the intended target population as closely as possible. This is a sampling issue, best handled with random sampling or stratified random sampling from the target population, assuming that such procedures are possible. 

Most often, we simply do not have access to the target population, and therefore we end up selecting an ad hoc sample from the accessible population. It is critical that we have procedures in place for measuring the key demographic variables from this ad hoc sample, so that we can adequately describe our sample for other researchers. We also have to decide how large a sample one should select. This decision is based on the analysis of the anticipated power of the study. The details for this procedure are beyond the scope of this text, but are explained in virtually all graduate statistics texts in psychology. The careful sampling of participants will enhance external validity.

The careful assignment of participants to groups will enhance internal validity. This is especially true in experiments, in which the participants are assigned to groups or conditions randomly. Even in lower-constraint research, such as differential research, careful assignment to groups will enhance the internal validity of the study. In differential research, the assignment is based on preexisting characteristics, which must be measured to make the assignment. For example, the groups may be formed based on the diagnosis of participants. The more carefully the diagnosis is made, the more precisely defined the groups will be. If you are using a matched-subjects design, you must not only do the matching and the random assignment to conditions with care, but you must choose carefully what variables to match on. You must also remember to preserve the information on which participants are matched with which other participants for the data analysis.

We must insure the well being of the participants in our study. For both animal and human studies, approval for the study is required from the appropriate board. This approval will be based on how effectively ethical concerns have been dealt with in our research proposal. With human research, informed consent is a requirement. 

If the participant is under the age of consent, the consent must be obtained from the most appropriate adult acting on the behalf of the participant. Usually, this will be the participant's parents. Even with parental consent, one must also get the assent of children who are participating in the study. 

Finally, in most human research, one has the obligation to provide an appropriate level of feedback at the end of the study. Procedures for all of these ethical safeguards must be in place before the first participant is tested, and the procedures must be approved by the appropriate IRB.

Preparation of Setting

We talked earlier about how the preparation of the setting was one of the general control procedures in a research study. At that point, we were interested in creating a setting that would minimize problems and maximize external validity. At this point, we are interested in getting all of the nuts and bolts of the setting and the procedures worked out so that the research study will run smoothly. 

The adequacy of the space should be ascertained. Is it large enough, and does it have the resources that you will require for your study. In the complex environment of today's psychological research, that may include more than just enough space. It may include adequate lighting, ventilation, sound dampening, access to computer facilities, and sometimes even electrical and magnetic shielding (if complex psychophysiological measures are planned). 

The equipment selected must be dependable and adequate to the task. It must be thoroughly checked for operational precision, and those checks should be repeated throughout the study to assure that the equipment continues to operate correctly. This is especially important with more complex equipment, in which subtle problems can easily develop without the researcher's awareness. 

The staff involved in running the study are every bit as critical to the success of the study as the equipment and space. They must be carefully trained in all of the procedures of the study, including how to handle unusual or potentially dangerous situations. When appropriate and possible, it is best to have the staff running the study blind to the hypotheses and condition that each participant is tested under.

Adequacy of Participant Preparation, Instruction, and Procedures

The final issue is the adequacy of the instructions to participants. Not only must great care be taken to make sure that they are clear and unambiguous, but the instructions should be piloted to make sure that they are working as intended.

Examples Using the Research Design Checklist

The best way to illustrate the many steps of the research design checklist described above is to walk you through some examples. We have included a couple of examples to illustrate the key decisions. Click on the heading of this section to access the examples.

Exercises Using the Research Design Checklist

The best way to learn any complex skill is to practice it in real life situations. We have included several research studies that you can use to practice the skills of creating the details of the study. To access those exercises, click on the heading of this section.