Business Research Methods:Data Collection Design: Experiments

Business Research Methods:

Data Collection Design: Experiments

中国经济管理大学

CHAPTER LEARNING OBJECTIVES

After reading this chapter, students should understand…

1. How an experiment is different than other primary data collection methods.

2. The seven steps of a well-planned experiment.

3. Internal and external validity with experimental research designs.

4. The three types of experimental designs and the variations of each.

5. The advantages and disadvantages of the experimental method.

6. The ethical issues in experiments and their solutions.

In this chapter we define causation and experiments, and the advantages and disadvantages. 

The questions of internal and external validity are also examined: Does the experimental treatment determine the observed difference, or was some extraneous variable responsible? And how can one generalize the results of the study across times, settings, and persons?  The chapter concludes with a review of the most widely accepted designs and a CloseUp example.

KEY TERMS

Key terms are shown in bold, as they appear in the text, throughout the lecture notes.

POWERPOINT

• A complete PowerPoint slide set comes with this chapter. 

o Slides are ordered as the chapter is organized.  

o Each, at minimum, contains the following:

o Learning Objectives slide

o Pullquote slide, key thought that opens this chapter

o Exhibit slides, one or more per exhibit

o Additional slides that cover critical concepts not covered by exhibits 

o Key Terms slide(s)

o Additional Discussion Opportunities slides: You can arrange these slides within the slide set as desired. This slide section contains several types of slides; suggestions for using these slides are in the Discussion and Project Ideas section of this manual.  The slides include all or some of the following:

 Snapshot slides, one for each Snapshot; contains an image or graphic to serve as a visual anchor for the discussion. 

 PicProfile slides, one for each PicProfile; contains the image.

 CloseUp slides, at least one per CloseUp; contains the images or graphs.

 Additional Pullquote from thought leaders, at least one per chapter

 PulsePoint: a statistic drawn from a research project that relates to some chapter concept.

TEST BANK

The test bank for each chapter contains the following:

• Multiple-choice or true-false objective questions of one or more types, with answer noted in RED:

o Definition-based questions on key terms and concepts

o Application-based questions posing decision scenarios

o Application-based questions asking for justification or explanation

• Essay Questions, with one possible answer noted in RED.

CONNECT

Connect is the location for several resources:

• Quiz questions

o You select from this additional set of multiple-choice and true-false questions for each chapter to create a self-assessment quiz for that chapter.  Each question provides a pop-up learning note for the correct answer, that you may opt to show (or not).

• Connect Library for Instructors

o PowerPoint Slide Sets

• Instructors often modify these sets to reflect their own teaching style and pedagogy for a chapter’s material; you may opt to share these sets (or not) with your students, as presented or modified.

• Each slide sets contains the graphical exhibits contained in the text.

o Instructor’s Manual for each chapter

o Test Bank for each chapter

o Written and video cases

o Additional Materials Related to Cases (e.g., case discussion notes, data sets, video material, etc.)

o Additional Materials Related to Chapters

o Supplemental appendices on topics you may want to assign related to a chapter.

• APP_Complex Experimental Designs-BRM13e.pdf

• BRM 13e_IM_Appendix_Complex Experimental Designs.doc

• APP_Test Markets-BRM13e.pdf

• BRM 13e_IM_Appendix_Test Markets.doc

o Supplemental chapter-related materials

o Sample Student Project

o Excel Chart Templates  

• Connect Library for Students

o Written and video cases

o Additional Materials Related to Cases (e.g., data sets, video material, etc.)

o Additional Materials Related to Chapters

o Supplemental appendices on topics.

o Supplemental chapter-related materials

o Sample Student Project

o Excel Chart Templates  

SMARTBOOK

This is a, digital version of Business Research Methods, which can be accessed online via laptop.   It is linked to Business Research Methods’s Connect features. The content of Business Research Methods SmartBook is the same as the printed version of Business Research Methods but the digital features help focus a student’s learning on particular book content. Students pay for a subscription to Business Research Methods SmartBook for the duration of your term or semester.  

• As the instructor, you may assign Business Research Methods SmartBook or  students may choose to subscribe to SmartBook on their own. 

• If you want your students to have access to Business Research Methods SmartBook and its learning features, you will need to set up your Business Research Methods Connect course.  

DISCUSSION AND PROJECT IDEAS

• Snapshots

o Experiments in Improving Employee Health…discusses an experiment to determine whether or not financial incentives delivered through premium adjustments work to help people lose weight; one group got the incentrive immediately, one got it at the end of the year.

o Robotic Experiments…discusses the latest Japanese experiment with robots, a fully automated low-cost hotel near the city of Nagasaki.

o Zeotap Experiments with Mercedes Benz…discusses an experiment completed for Mercedes-Benz by Zeotap, a German startup that interfaces multiple telecom companies’ data with advertisers wishing to target specific groups.

o MIT SENSEable City Lab…discusses experiments done by SENSEable on the design of work spaces.

• PicProfiles

o None in this Chapter

• CloseUp

o A Job Enrichment Quasi-Experiment…describes a job enrichment study at British Chemical

• PulsePoint:  Published research reveals many ways that businesses use research. You might use such research findings to discuss a current phase of the research process or a current issue. 

• This statistic relates to Americans and ‘healthy lifestyle.”  Ask students to determine what should be included in ‘healthy lifestyle’ including not only what type of food and the quantity consumeed, but exercise, body mass index, etc.  The authors of the study at May Clinic included moderate or vigorous exercise at least 150 minutes each week, diet score in the Healthy Eating Index, body fat percentage under 20% (men) or 30% (women), and not smoking.

 2.7… The percent of Americans living a healthy lifestyle.

• PullQuote:  Use each chapter’s pull quote to discuss a current issue related to the chapter.  

• This quote deals with a critical element of uncertainty in the hypothesis being tested with an experimental design.

“It's not an experiment if you know it's going to work.”

Jeff Bezos, CEO, 

Amazon

• This quote deals with the importance of learning through failure.  You can ask students to look at their use of e-book (Kindle, Nook) and tablets. What failures in this arena would be critical to device success. Or you can encourage students to search for web sites to find lists of the top ten apps for these devices and discuss how apps are evaluated via beta tests before they ever get permission to be part of a device. 

“This is a time of [e-book app] experimentation.  I’d be really disappointed if we weren’t seeing both successes and failures. I do think that everyone seems to be looking at these differently.”

Barbara Marcus, consultant and adviser,

Open Road Integrated Media

• This quote deals with the hypothesis that guides an experiment and the importance of keeping an open mind.  You can use this quote to discuss being open to the results of any research, and avoiding perceiving only the results you expected to find.

o ‘There is no such thing as a failed experiment, only experiments with unexpected

outcomes.”

Richard Buckminster Fuller, engineer and architect

• WWW Exercise

 The search engine BING has been running an experiment against its rival search engines in the hope of demonstrating superiority. Have the student visit the web site below and participate in the experiment, then describe the experiment and make a recommendation to make it better.  How might BING use this experiential information?

 http://www.bingiton.com/

 Use a search engine to find an experiment described on the Web. Remember that experiments sometimes go by other names, like test market or taste test in consumer food products or beta test in software products. Also, use terms introduced in this chapter. What experiment could you do that would use the same methodology as the one you discovered?

• Class discussion or research project:

 One class period should be allocated data collection design via experiment. In part, this is possible because all the information you need is in this chapter. The Conducting an Experiment section serves as a good review to begin the discussion or lecture, allowing you to concentrate on the threats to validity (internal and external). 

 Consider dividing your class into teams and have them design an experiment on an issue you specify or they choose from a range of options.  They must then present their design, and address the concepts in the chapter in its defense.

 To expand the coverage of experimentation, the analysis of variance material from Chapters 14 and  Chapter 15 may be included. 

 This chapter affords numerous opportunities to discuss ethical responsibilities.

CHAPTER EXHIBITS

Exhibit Number Exhibit title

8-1 Mills' Method of Agreement

8-2 Mill's Method of Difference

8-3 Four Types of Asymmetrical Causal Relationships

8-4 Experiments in the Research Process

8-5 Experiment of Benefits Module Placement of within a Sales Presentation

8-6 Quota Matrix Example

8-7 Key to Design Symbols

CU-1 Assessment of EOs’ Monthly Reports

Notes an exhibit in the Research Process Series

CHAPTER LECTURE NOTES

EXPERIMENTS AND CAUSATION

• Why do events occur under some conditions and not under others? 

• Experiments are research studies designed to explain why something happens.

• The researcher manipulates some variable (IV) and then observes whether the hypothesized dependent variable (DV) is affected.

Causation Evidence

• The essential element of causation is that A “produces” B or A “forces” B to occur.

• There is at least one independent variable (IV) and one dependent variable (DV) recin a causal relationship.

• Causation needs evidence to support an indicative conclusion; you can’t demonstrate a causal relationship deductively.

• We seek three types of evidence in testing a causal hypothesis:

• Covariation between A and B.

o Do we find that A and B occur together in the way hypothesized?

o When A does not occur, is there also an absence of B?

o When there is more or less of A, does one also find more of less of B?

• Time order of events moving in the hypothesized direction.

o Does A occur before B?

• No other possible causes of B.

o No one can determine that C, D, and E do not covary with B in a way that suggests possible causal connections?

• In addition to the three conditions above, successful inference making from experimental designs must meet two additional requirements:

• Control: all factors, with the exception of the independent variable, must be held constant and not confound with another variable that is not part of the study.

• Random assignment: each person must have an equal chance for exposure to each level of the independent variable.

Causal Conclusion

• Meeting the ideal standard of causation requires that one variable always causes another variable, and no other variable has the same causal effect.

• TIP: The method of agreement states that “When two or more cases of a given phenomenon have one and only one condition in common, then that condition may be regarded as the cause (or effect) of the phenomenon.”

• The method of agreement helps rule out some variables as irrelevant. A, B, D, and E are unlikely to be causes of Z. However, there is an implicit assumption that there are no variables to consider. 

• If we can find Z and only Z in every case where we find C, and no others are found with Z, then we can conclude that C and Z are causally related 

(see Exhibit 8-1).

• Exhibit 8-1 Mill’s Method of Agreement

• The negative canon of agreement states that where the absence of C is associated with the absence of Z, there is evidence of a causal relationship between C and Z. (Exhibit 8-2) 

o Together with the method of agreement, this forms the basis for the method of difference: “If there are two or more cases, and in one of them observation Z can be made, while in the other it cannot; and if variable C occurs when observation Z can be made, while in the other it cannot; and if variable C occurs when observation Z is made, and does not occur when observation Z is not made; then it can be asserted that there is a causal relationship between C and Z.

• Exhibit 8-2 Mill’s Method of Difference

• No one can ever be certain that variable A causes variable B to occur, but one can gather evidence that increases the belief that A leads to B.

• There are three possible relationships between two variables: Symmetrical, Reciprocal, and Asymmetrical

• A symmetrical relationship is one in which two variables fluctuate together, but we assume the changes in neither variable are due to changes in the other.

o Symmetrical conditions are most often found when two variables are alternate indicators of another cause or independent variable.

o We might conclude that a correlation between low work attendance and active participation in a camping club is the result of (dependent on) another factor, such as a lifestyle preference.

• A reciprocal relationship exists when two variables mutually influence or reinforce each other.

o This could occur if reading an advertisement leads to the use of a product. The usage, in turn, sensitizes the person to notice and read more of the advertising for that product.

• When trying to prove causation a researcher looks for an asymmetrical relationships.

o With these, we postulate that changes in one independent variable (IV) are responsible for changes in a dependent variable (DV). 

o The identification of the IV and DV is often obvious, but sometimes the choice is not clear. In these cases, dependence and independence should be evaluated on the basis of:

 The degree of which each variable may be altered.

 The time order between the variables.

• Exhibit 8-3 describes the four types of asymmetrical causal relationships:

• Stimulus-response (a type of asymmetrical causal relationship where an event or change in the IV results in a response in the DV)

• Property-disposition (a type of asymmetrical causal relationship where an existing property causes a disposition; e.g., age and attitude about saving).

• Disposition-behavior (a type of asymmetrical causal relationship where a disposition causes a specific behavior; e.g., attitude about a brand causes its purchase or rejection).

• Property-behavior (a type of asymmetrical causal relationship where an existing property causes a specific behavior; e.g., age and sports participation)

• Exhibit 8-3 Four Types of Asymmetrical Causal Relationships

• Experiments in Business

• Business experiments usually involve stimulus-response relationships.

• Property-disposition relationships are often studied in business in social science research.

• Much ex post facto research involves relationships between properties, dispositions, and behaviors.

• Although experimental methods neither ensure discovery of all relevant variables nor provide certain proof of causation, they help advance our understanding of causality by eliminating inadequate causal arguments

• Example: Bystanders and Thieves

• Ex post facto research designs, in which a researcher interviews respondents or observes what is or what has been, also have the potential for discovering causality. 

Snapshot Experiments in Improving Employee Health

CONDUCTING AN EXPERIMENT 

• In a well-executed experiment, researchers must complete a series of activities to carry out their craft successfully. 

• There are seven activities the researcher must accomplish to make the endeavor successful (see Exhibit 8-4): 

• Select relevant variables. 

• Specify the level(s) of the treatment. 

• Control the experimental environment. 

• Choose the experimental design. 

• Select and assign the participants. 

• Pilot-test, revise, and test. 

• Analyze the data. 

• Exhibit 8-4 Experiments in the Research Process

Snapshot Robotic Experiments

Select Relevant Variables 

• The researcher must translate an amorphous problem into the research question or hypothesis that best states the objectives of the research. 

• Example: Placement of a benefits module within a sales presentation.

• Does a sales presentation that describes product benefits in the introduction of the message lead to improved retention of product knowledge? 

• The researchers would need to select variables that best operationalize the concepts sales presentation, product benefits, retention, and product knowledge (transformed into variables to make them measurable and subject to testing)..  

• Because a hypothesis is a speculation about the outcome of the study, it might take this form: 

o Sales presentations in which the benefits module is placed in the introduction of a 12-minute message produce better retention of product knowledge than those where the benefits module is placed in the conclusion. 

• The product’s classification and the nature of the intended audience should also be defined.

• The term better could be operationalized statistically by means of a significance test.

• The number of variables in an experiment is constrained by:

o The project budget

o The time allocated

o The availability of appropriate controls

o The number of participants being tested

o For statistical reasons, there must be more participants than variables.

• The selection of measures for testing requires a thorough review of the available literature and instruments.

• In addition, measures must be adapted to the unique needs of the research situation without compromising their intended purpose or original meaning.

Specifying Treatment Levels

• In an experiment, the researcher manipulates the independent variable, called the experimental treatment.

• The treatment levels of the independent variable are the arbitrary or natural groups the researcher chooses within the independent variable of an experiment.

o Example: If salary is hypothesize to have an effect on employees’ exercising stock purchase options, salary might be divided into high, middle, and low ranges to represent three treatment levels of the independent variable.

• The levels assigned to an independent variable should be based on simplicity and common sense.

o In the sales presentation example, the experimenter should not select 8 minutes and 10 minutes as the starting points to represent the two treatment levels if the average message about the product is 12 minutes long.

o if the benefits module is placed in the first and second minutes of the presentation, observable differences may not occur because the levels are too close together. 

o In a trial run, position the midpoint of the benefits module the same interval from the end of the introduction as from the end of the conclusion (see Exhibit 8-5).

• Exhibit 8-5 Experiment of Benefits Module Placement of within a Sales Presentation

• Under a different hypothesis, several levels of the independent variable may be needed to test order-of-presentation effects.

o Alternatively, a control group could provide a base level for comparison.

o The control group is composed of participants who are not exposed to the independent variable(s).

Snapshot Zeotap Experiments with Mercedes Benz

Control the Experimental Environment 

• At this stage, however, we are principally concerned with environmental control. 

• The introduction of the experiment to the participants and the instructions would likely be videotaped for consistency. 

• The arrangement of the room, the time of administration, the experimenter’s contact with the participants, and so forth, must be consistent across each experiment. 

• Other forms of control involve participants and experimenters. 

• When participants do not know if they are receiving the experimental treatment, they are said to be blind. 

• When the experimenters do not know if they are giving the treatment to the experimental group or to the control group, the experiment is said to be double blind. 

• Both approaches control unwanted complications, such as participants’ reactions to expected conditions or experimenter influence. 

• Example: In our sales presentation experiment, extraneous variables can appear as differences in age, gender, race, dress, communications competence, and other characteristics of the presenter, the message, or the situation. 

• These have the potential for distorting the effect of the treatment on the dependent variable and must be controlled or eliminated. 

Choose the Experimental Design 

• Experimental designs are unique to the experimental method. 

• They designate relationships between experimental treatments and the experimenter’s observations or measurement points in the scheme of the study. 

• The researchers select one design that is best suited to the goals of the research. 

• Judicious selection of the design improves the probability that the observed change in the dependent variable was caused by the manipulation of the independent variable, not by another factor. 

o It simultaneously strengthens the generalizability of results beyond the experimental setting. 

Select and Assigning Cases

• The participants or subjects selected for the experiment should be representative of the target population to which the researcher wishes to generalize the study’s results. 

• In the sales presentation example, corporate buyers or purchasing managers would provide better generalizing power than undergraduate college students 

if the product in question was targeted for industrial use. 

• The procedure for random sampling of experimental participants is similar to the selection of respondents for a survey. 

• The researcher prepares a sampling frame and then assigns the participants for the experiment to groups, using a randomization technique. 

• Random assignment to the groups is required to make the groups as comparable as possible with the dependent variable. 

• Randomization does not guarantee that if a pretest of the groups was conducted, the groups would be pronounced identical.

• However, it is an assurance that those differences remaining are randomly distributed. 

• Example: In our example, we would need three randomly assigned groups—one for each of the two treatments and one for the control group. 

• When it is not possible to randomly assign participants to groups, matching may be used. 

• Matching employs a nonprobability quota sampling approach. 

• The object of matching is to have each experimental and control case matched on every characteristic used in the research.

o This becomes cumbersome as the number of variables and groups in the study increases.

• Because the characteristics of concern are only those that are correlated with the treatment condition or the dependent variable, they are easier to identify, control, and match.

o Example: In the sales presentation experiment, if a large part of the sample was composed of businesswomen who had recently completed communications training, we would not want the characteristics of gender, business experience, and communication training to be disproportionately assigned to one group.

• Some authorities suggest a quota matrix as the most efficient means of visualizing the matching process.

• In Exhibit 8-6, one-third of the participants from each cell would be assigned to each of the three groups.

• Exhibit 8-6 Quota Matrix Example

• If matching does not alleviate the assignment problem, a combination of matching, randomization, and increasing the sample size would be used.

Pilot Test, Revising, and Pretest

• Pilot testing is intended to reveal errors in the design and improper control of extraneous or environmental conditions.

• Pretesting the measurement instruments permits refinement before the final test.

• This is the researcher’s best opportunity to revise scripts, look for control problems with laboratory conditions, and scan the environment for factors that might confound the results.

• In field experiments, researchers are sometimes caught off guard by events that have a dramatic effect on cases:

o The test marketing of a competitor’s product

o A reduction in force, reorganization, or merger

• The experiment should be timed to that participants are not sensitized to the independent variable by factors in the environment.

Snapshot: MIT Senseable City Lab

Analyzing the Data

• If adequate planning and pretesting have occurred, the experimental data will take an order and structure uncommon to surveys and unstructured observational studies.

• It’s not that data from experiments are easy to analyze, they are simply more conveniently arranged because of the:

• Levels of the treatment condition

• Pretests and post-tests

• Group structure

• If adequate planning and pretesting have occurred, the choice of statistical techniques is simplified.

• Researchers have several measurement and instrument options with experiments. Among them are: 

• Observational checklists and coding schemes. 

• Self-report instruments with open-ended or closed questions. 

• Scaling techniques (Likert scales, semantic differentials, Q-sort). 

• Physiological measures (galvanic skin response, EKG, voice pitch analysis, eye dilation). 

VALIDITY IN EXPERIMENTATION

• Even when an experiment is the ideal research design, there is always a question about whether the results are true. 

• Validity means that a measure accomplishes its claims. 

• There are several types of validity, here only the two major varieties are considered: 

• Internal validity—Do the conclusions we draw about a demonstrated experimental relationship truly imply cause?

• External validity—Does an observed causal relationship generalize across persons, set-tings, and times?

o Each type of validity has specific threats we must guard against. 

Internal Validity

• Among the many threats to internal validity, we consider seven:

• History 

• Maturation 

• Testing 

• Instrumentation 

• Selection 

• Regression Toward the Mean

• Experimental mortality 

History

Note: The key to design symbols is in Exhibit 8-7. 

• During the time that an experiment is taking place, some events may occur that confuse the relationship being studied. 

• In many experimental designs, we take a control measurement (O1) of the dependent variable before introducing the manipulation (X). 

• After the manipulation, we take an after-measurement (O2) of the dependent variable. 

• The difference between O1 and O2 is the change that the manipulation caused. 

• Management may wish to find the best way to educate its workers about the financial condition of the company before labor negotiations. 

• To assess the value of such an effort, managers test employees on their knowledge of the company’s finances (O1). 

• Then, they present the educational campaign (X) to these employees.

• Afterward, they again measure their knowledge level (O2). 

• This design, known as a pre-experiment because it is not a very strong design, can be diagrammed as follows: 

O1        X O2

Pretest Manipulation Posttest

• Between O1 and O2, many events could occur to confound the effects of the education effort, such as the topic appearing in a newspaper article or being discussed at a union meeting.

Maturation 

• Changes also may occur within the participant that are a function of the passage of time and are not specific to any particular event. 

• These are of special concern when the study covers a long time, but they may also be factors in tests that are as short as an hour or two. 

• A participant can become hungry, bored, or tired in a short time, and this condition can affect response results. 

Testing 

• The process of taking a test can have a learning effect that influences the results of a second test. 

Instrumentation 

• This threat to internal validity results from changes between observations in either the measuring instrument or the observer. 

• Using different questions at each measurement is an obvious source of potential trouble, but using different observers or interviewers also threatens validity. 

• There can even be an instrumentation problem if the same observer is used for all measurements. 

• Observer experience, boredom, fatigue, and anticipation of results can all distort the results of separate observations. 

Selection 

• An important threat to internal validity is the differential selection of participants for experimental and control groups. 

• Validity considerations require that the groups be equivalent in every respect. 

• If participants are randomly assigned to experimental and control groups, this selection problem can be largely overcome. 

• Matching the members of the groups on key factors can also enhance the equivalence of the groups. 

Regression Toward the Mean

• This factor operates when groups have been selected by their extreme scores. 

• Example: Measure the output of all workers in a department for a few days before an experiment and then conduct the experiment with only those workers whose productivity scores are in the top 25 percent and bottom 25 percent. 

o No matter what is done between O1 and O2, there is a strong tendency for the average of the high scores at O1 to decline at O2 and for the low scores at O1 to increase. 

o This tendency results from imperfect measurement that, in effect, records some persons abnormally high and abnormally low at O1. 

o In the second measurement, members of both groups score more closely to their long-run mean scores. 

Experiment Mortality

• This occurs when the composition of the study groups changes during the test.

• Attrition is especially likely in the experimental group, and with each dropout the group changes. 

• Because members of the control group are not affected by the testing situation, they are less likely to withdraw. 

• In a compensation incentive study, some employees might not like the change in compensation method and may withdraw from the test group. 

o This could distort the comparison with the control group that has continued working under the established system, perhaps without knowing a test is under way. 

• All the threats mentioned to this point are generally dealt with adequately in experiments by random assignment. However, five additional threats to internal validity are independent of whether or not one randomizes.

• The first three have the effect of equalizing experimental and control groups. 

o Diffusion or imitation of treatment

o Compensatory equalization

o Compensatory rivalry

o Resentful demoralization of the disadvantaged

o Local history

External Validity

• Internal validity factors cause confusion about whether the experimental treatment (X) or extraneous factors are the source of observation differences. 

• External validity is concerned with the interaction of the experimental treatment with other factors and the resulting impact on the ability to generalize to (and across) times, settings, or persons. 

• Among the major threats to external validity: 

• Reactivity of testing on X 

• Interaction of selection and X

• Other reactive factors

The Reactivity of Testing on X 

• The reactive effect refers to sensitizing participants via a pretest so they respond to the experimental stimulus (X) in a different way. 

• This before-measurement effect can be particularly significant in experiments where the IV is a change in attitude. 

Interaction of Selection and X 

• The process by which test participants are selected for an experiment may be a threat to external validity. 

• The population from which one selects participants may not be the same as the population to which one wishes to generalize results. 

• Example: You use a selected group of workers in one department for a test of the piecework incentive system. Can you extrapolate those results to all production workers?

• Example: You ask a cross section of a population to participate in an experiment, but a substantial number refuse. If you conduct the experiment only with those who agree to participate (self-selection), can the results be generalized to the total population? 

Other Reactive Factors 

• Experimental settings themselves may have a biasing effect on a participant’s response 

to X. 

• An artificial setting can produce results not representative of larger populations. 

o Suppose workers who are given incentive pay are moved to a different work area to separate them from the control group. These new conditions alone could create a strong reactive condition. 

• If participants know they are participating in an experiment, there may be a tendency to role-play in a way that distorts the effects of X. 

• Another reactive effect is the possible interaction between X and participant characteristics. 

o An incentive pay proposal may be more effective with persons in one type of job, with a certain skill level, or with a certain personality trait. 

• Problems of internal validity can be solved by the careful design of experiments, but this is less true for problems of external validity. 

• External validity is largely a matter of generalization, which is an inductive process of extrapolating beyond the data collected. 

• Generalizing means estimating the factors that can be ignored and that will interact with the experimental variable. 

o The closer two events are in time, space, and measurement, the more likely they are to follow the same laws. 

• Seek internal validity first.

o Try to secure as much external validity as is feasible by making experimental conditions as similar to the conditions under which the results will apply as possible.

EXPERIMENTAL RESEARCH DESIGNS

• Experimental designs vary widely in their power to control contamination of the relationship between independent and dependent variables. 

• The most widely accepted designs are based on this characteristic of control (see Exhibit 8-7): 

• Preexperiments

• True experiments

• Field experiments 

• Exhibit 8-7  Key to Design Symbols

Preexperimental Designs 

• All three preexperimental designs are weak in their scientific measurement power—that is, they fail to control adequately the various threats to internal validity. 

• After-only Design

• One-Group Pretest-Posttest Design

• Static Group Comparison

After-Only Design

• This may be diagrammed as follows: 

X O (1)

Treatment or manipulation of independent variable Observation or measurement of dependent variable

• Example: An employee education campaign about the company’s financial condition without a prior measurement of employee knowledge. Results would reveal only how much the employees know after the education campaign, but there is no way to judge the effectiveness of the campaign. 

• The lack of a pretest and control group makes this design inadequate for establishing causality. 

One-Group Pretest-Posttest Design

• This is the design used earlier in the educational example. It meets the various threats to internal validity better than the after-only study, but it is still a weak design. 

• How well does it control for history? Maturation? Testing effect? The others? 

O X O (2)

Pretest Manipulation Post-test

Static Group Comparison

• This design provides for two groups, one of which receives the experimental stimulus while the other serves as a control. 

• In a field setting, imagine this scenario. A forest fire or other natural disaster is the experimental treatment, and psychological trauma (or property loss) suffered by the residents is the measured outcome. 

• A pretest before the forest fire would be possible, but not on a large scale (as in the California fires). 

• Timing of the pretest would be problematic. 

• The control group, receiving the posttest, would consist of residents whose property was spared.

X

_ _ _ _ _ _ _ _ _ _ O1

_ _ _ _ _ _ _ _ _ _ (3)

O2

• The addition of a comparison group creates a substantial improvement over the other two designs. 

• Its chief weakness is that there is no way to be certain that the two groups are equivalent. 

True Experimental Designs

• The major deficiency of the preexperimental designs is that they fail to provide comparison groups that are truly equivalent. 

• Equivalence is achieved through matching and random assignment. 

• True experimental designs include

• Pretest-Posttest Control Group Design

• Posttest-Only Control Group Design

• It is common to show an X for the test stimulus and a blank for the existence of a control situation. 

• This is an oversimplification of what really occurs. More precisely, there is an X1 and an X2, and sometimes more. 

• The X1 identifies one specific independent variable, while X2 is another independent variable that has been chosen, often arbitrarily, as the control case. 

• Different levels of the same independent variable may also be used, with one level serving as the control. 

Pretest-Posttest Control Group Design 

• This design consists of adding a control group to the one-group pretest-posttest design and assigning the participants to either of the groups by a random procedure (R). 

• The diagram is: 

R O1 X O2 (4)

R O3 O4

• The effect of the experimental variable is 

E = (O2 – O1) – (O4 – O3)

• In this design, the seven major internal validity problems are dealt with fairly well, although there are still some difficulties. 

• Local history may occur in one group and not the other. 

• Also, if communication exists between people in test and control groups, there can be rivalry and other internal validity problems. 

• Maturation, testing, and regression are handled well because one would expect them to be felt equally in experimental and control groups. 

• Mortality can be a problem if there are different dropout rates in the study groups. 

• Selection is adequately dealt with by random assignment. 

• The record of this design is not as good on external validity, however. 

• There is a chance for a reactive effect from testing. This might be a substantial influence in attitude change studies where pretests introduce unusual topics and content. 

• Nor does this design ensure against reaction between selection and the experimental variable. 

• Even random selection may be defeated by a high decline rate by participants. 

o This would result in using a disproportionate share of people who are essentially volunteers and who may not be typical of the population. 

o If this occurs, the experiment must be replicated several times with other groups, under other conditions, before we can be confident of external validity. 

Posttest-Only Control Group Design

• In this design, the pretest measurements are omitted. 

• Pretests are well established in classical research design but are not really necessary when it is possible to randomize. 

• The design is: 

R X O1 (5)

R O2

• The experimental effect is measured by the difference between O1 and O2: E = (O2  – O1). 

• The simplicity of this design makes it more attractive than the pretest-posttest control group design. 

• Internal validity threats from history, maturation, selection, and statistical regression are adequately controlled by random assignment. 

• Because participants are measured only once, the threats of testing and instrumentation are reduced, but different mortality rates between experimental and control groups continue to be a potential problem. 

• The design reduces the external validity problem of testing interaction effect. 

Field Experiments: Quasi- or Semi-Experiments

• Under field conditions, we often cannot control enough of the extraneous variables or the experimental treatment to use a true experimental design. 

• Because the stimulus condition occurs in a natural environment, a field experiment is required. 

• A modern version of the bystander and thief field experiment involves the use of electronic article surveillance to prevent shrinkage due to shoplifting. 

• Proprietary study: A shopper came to the optical counter of an up-scale mall store and asked to be shown designer frames. The salesperson, a confederate of the experimenter, replied that she would get them from a case in the adjoining department and disappeared. The “thief” selected two pairs of sunglasses from an open display, deactivated the security tags at the counter, and walked out of the store. Thirty-five percent of the participants (store customers) reported the theft upon the return of the salesperson. Sixty-three percent reported it when the salesperson asked about the shopper. Unlike previous studies, the presence of a second customer did not reduce willingness to report a theft.

o This study was not possible with a control group, a pretest, or randomization of customers, but the information gained was essential and justified a compromise of true experimental designs.

Closeup A Job Enrichment Quasi-Experiment (includes Exhibit CU-1 Assessment of EO’s Monthly Reports)

• We use preexperimental designs or quasi-experiments to deal with such conditions. 

• In a quasi-experiment, we often cannot know when or to whom to expose the experimental treatment, but we can decide when and whom to measure. 

• A quasi-experiment is inferior to a true experimental design, but is usually superior to preexperimental designs. 

• Preexperimental designs or quasi-experiments include:

• Nonequivalent Control Group Design 

o Intact equivalent design

o Self-selected experimental group design

• Separate sample Tretest-Posttest design

• Group Time Series Design

Nonequivalent Control Group Design

• This is a strong and widely used quasi-experimental design. 

• It differs from the pretest-posttest control group design, because the control groups are randomly assigned. 

• The design is diagrammed as follows: 

O1

_ _ _ _ _ _ _ _ _ _ X

_ _ _ _ _ _ _ _ _ _ O2

_ _ _ _ _ _ _ _ _ _ (6)

O3 O4

• There are two varieties. 

• Intact equivalent design: The membership of the experimental and control groups is naturally assembled. 

o Ideally, the two groups are as alike as possible. 

o This design is especially useful when any type of individual selection process would be reactive. 

• Self-selected experimental group design: A weaker design because volunteers are recruited to form the experimental group, while nonvolunteer participants are used for control. 

o Such a design is likely when participants believe it would be in their interest to be a participant in an experiment. 

• Comparison of pretest results (O1 – O3) is one indicator of the degree of equivalence between test and control groups. 

• If the pretest results are significantly different, there is a real question about the groups’ comparability. 

• If pretest observations are similar between groups, there is reason to believe internal validity of the experiment is good. 

Separate Sample Pretest-Posttest Design

• This design is most applicable when we cannot know when and to whom to introduce the treatment, but we can decide when and whom to measure. 

• The basic design is: 

R     O1 (X)

(7)

R O2

• The bracketed treatment (X) is irrelevant to the purpose of the study but is shown to suggest that the experimenter cannot control the treatment. 

• This is not a strong design; several threats to internal validity are not handled adequately. 

• History can confound the results but can be overcome by repeating the study at other times, in other settings. 

• It is considered superior to true experiments in external validity. 

• Its strength results from being a field experiment in which the samples are usually drawn from the population to which we wish to generalize our findings. 

• This design would be more appropriate if:

• The population were large.

• A before-measurement were reactive.

• If there were no way to restrict the application of the treatment. 

o Example: A company is planning an intense campaign to change its employees’ attitudes toward energy conservation. It might draw two random samples of employees, one of which is interviewed about energy use attitudes before the information campaign. After the campaign, the other group is interviewed. 

Group Time Series Design

• A time series design introduces repeated observations before and after the treatment and allows participants to act as their own controls. 

• The single treatment group design has before-after measurements as the only controls. 

• A multiple design has two or more comparison groups, as well as the repeated measurements in each treatment group. 

• The time series format is especially useful where regularly kept records are a natural part of the environment and are unlikely to be reactive. 

• The time series approach is also a good way to study unplanned events in an ex post facto manner. 

• Example: If the federal government were to suddenly begin price controls, we could study the effects of this action later if we had regularly collected records for the period before and after the advent of price control. 

• The internal validity problem for this design is history. 

• To reduce this risk, keep a record of possible extraneous factors during the experiment and attempt to adjust the results to reflect their influence. 

AN EVALUATION OF EXPERIMENTS 

Advantages 

• Causality cannot be proved with certainty, but the probability of one variable being linked to another can be established convincingly. 

• The experiment comes closer than any primary data collection method to accomplishing this goal. 

• The foremost advantage is the researcher’s ability to manipulate the independent variable. 

• Consequently, there is an increased probability that changes in the dependent variable are a function of that manipulation. 

• A control group serves as a comparison to assess the existence and potency of the manipulation. 

• The second advantage of the experiment is that contamination from extraneous variables can be controlled more effectively than in other designs. 

• This helps the researcher isolate experimental variables and evaluate their impact over time. 

• Third, the convenience and cost of experimentation are superior to other methods. 

• These benefits allow opportunistic scheduling of data collection and the flexibility to adjust variables and conditions that evoke extremes not observed under routine circumstances. 

• In addition, the experimenter can assemble combinations of variables for testing, rather than searching for their fortuitous appearance in the study environment. 

• Fourth, replication—repeating an experiment with different participant groups and conditions—leads to the discovery of an average effect of the independent variable across people, situations, and times. 

• Fifth, researchers can use naturally occurring events and field experiments to reduce participants’ perceptions of the researcher as a source of intervention or deviation in their everyday lives. 

Disadvantages 

• The artificiality of the laboratory is the primary disadvantage of the experimental method. 

• Many participants’ perceptions of a contrived environment can be improved by investment in the facility. 

• Second, generalization from nonprobability samples can pose problems, despite random assignment. 

• The extent to which a study can be generalized from college students to managers or executives is open to question. 

• When an experiment is unsuccessfully disguised, volunteer participants are often those with the most interest in the topic. 

• Third, the cost of experimentation can far outrun the budgets for other primary data collection methods. 

• Fourth, experimentation is most effectively targeted at problems of the present or immediate future. 

• Experimental studies of the past are not feasible, and studies about intentions or predictions are difficult. 

• Finally, management research is often concerned with the study of people. 

• There are limits to the types of manipulation and controls that are ethical. 

ETHICAL ISSUES AND THEIR SOLUTIONS

• The major ethical issues involved with experiments are participant welfare, participant privacy, research quality and sponsor and result non-disclosure.

• Privacy

• Is not an issue when participants are recruited for laboratory experiments as informed consent is a normal protocol.

• Can be an issue in field experiments where participants don’t know they are part of a research study.

• Researchers deal with this by using debriefing—sharing what details they can after the experiment, without damaging the continuing experiment.

• Participant Welfare

• Researchers can’t know what emotional harm might come to participants in an experiment.

• The researcher should always judge the value of the information and insights against any possible harm to a participant.

• If the researcher has significant concerns, they can arrange for trained personnel (e.g., psychologists, counselors, etc.) to deal with expected or likely repercussions.

• Research Quality

• A researcher guarantees quality by carefully planning the design and execution of the experiment.

o This includes thorough training of any data collectors or parties executing the manipulation of variables.

• Non-disclosure

• It may not be possible to provide sponsor non-disclosure.

• A researcher can provide findings non-disclosure, by protecting results.

o Even if debriefing a participant is required in the protocol, it is inappropriate for the researcher or data collector to discuss results or research objective.

ANSWERS TO DISCUSSION QUESTIONS

Terms in Review 

1. Distinguish between the following: 

a) Internal validity and external validity. 

Internal validity was called simply "validity" in Chapter 7. It involves the question of whether we are measuring what we think we are, i.e., is the experimental treatment the real cause of the result we find in the experimental group? External validity concerns the degree to which the experiment can be generalized across persons, times, or settings. That is, can the experiment be viewed as an accurate sample of some more general conditions?

b) Preexperimental design and quasi-experimental design. 

Pre-experiment designs are the crudest forms of "experimentation" because they fail to control extraneous variables and they often omit the basic process of comparison. History, maturation, and instrumentation problems often plague these designs. Quasi-experiment designs are more sophisticated than pre-experiment designs, but they don’t qualify as true experiments either. These designs are used when the researcher can control only some of the variables. In the quasi-experiment the researcher cannot establish equivalent experimental and control groups through random assignment, and often he/she cannot determine when or to whom to expose the experimental variable.  On the other hand, researchers can often determine when and whom to measure.

c) History and maturation. 

Both are problems of internal validity. History effects represent specific events that occur during a study that can influence the IV-DV relationship. Maturity effects occur purely as a function of time passage and are not specific to a given event or condition.

d) Random sampling, randomization, and matching. 

Random Sampling (Chapter 14) is the special case of the probability sample where each population element has an equal chance of selection. Randomization and matching are both useful for improving the equivalency of control and experimental groups. Neither method is perfect, but randomization is the basic method because it is the primary means of assuring compatibility within some known error interval. Participants are randomly assigned to groups by probability sampling, the type depending on the nature of the experimental design. Matching, which employs a nonprobability quota sampling approach, is a way to supplement random assignment and can improve the equivalence of test and control groups.

e) Environmental variables and extraneous variables. 

Active factors are those variables that an experimenter can manipulate by causing various participants to receive more or less of the factor. Blocking factors are those that a participant has in some degree and can not be changed by the experimenter. The experimenter can only identify and classify participants on these blocking factors.

2. Compare the advantages of experiments with the advantages of survey and observational methods.

The observational method is a more useful method for collecting data from children, illiterate, and functionally illiterate persons. The intrusion of observation is often better accepted than questioning. Disguised and unobtrusive measures are often easier to carry out than disguised questioning. It is generally a slow, expensive process with limited opportunity to learn about the past. Further comparisons are found in the accompanying table. 

Survey Observation Experiment

• Advantageous for discovering a person's opinions, attitudes, motivations.

• Interviewer may observe nonverbal behavior (though not with mail, and limited to voice inflections with telephone).

• Structure of schedule or guide focuses attention on study purpose.

• Instruments may restrict study to previously chosen questions.

• Relationship between interviewer and participant is of short duration and promotes comparative objectivity.

• Results may be product of method rather than objective reality.

• Good quantification potential; data are organized by the instrument and easily systematized.

• Anonymity/Confidentiality safeguards.

• Largest sample sizes. • Observation and recording (notes, videotapes) is superior to depending on someone's recollections.

• Primary method for non-verbal behavior analysis; has wide application for behavioral and non-behavioral analysis. First hand observation of phenomenon.

• Ethical questions about consent, anonymity, etc.

• Observation instruments often lack structure (but unstructured observation offers flexibility).

• Observation may be used as exploratory front-end stage to survey or experiment.

• Relationship between observer and participant is often extended and provides more detail (but may reduce objectivity).

• Natural environmental setting lessens reactivity of participants; results are more realistic because method is less restrictive (but there is loss of control over variables).

• Difficulties with quantification: large amounts of data, coding problems, and lower power in statistical analysis.

• Smaller sample sizes than survey. • Superior method for establishing causality; surveys (except panels and other longitudinal designs) do not have experimental pre-post or multiple measurement advantage; observation may be longitudinal but there is little environmental control or ability to measure change in dependent variable.

• Ultimate method for control of variables: smaller samples and management of extraneous factors.

• Better opportunity to study change than cross-sectional surveys.

• Reactive effects may be caused by experimenter (also caused by survey interviewer)

• Laboratory settings are required for high levels of control: natural behavior is often altered or disappears. Change to natural settings reduces control options.

• Ethical considerations heavily influence manipulation and control of variables.

• Smallest sample sizes.

3. Why would a noted business researcher say, “It is essential that we always keep in mind the model of the controlled experiment, even if in practice we have to deviate from an ideal model”?

The statement essentially says that the model of experimental design is the most powerful basis we have for determining causation.  Therefore research efforts should seek to approach this ideal model as closely as possible.

4. What ethical problems do you see in conducting experiments with human participants? 

Student answers will vary. Sample answer:

In the past there has often been too little concern among researchers regarding this problem.  Clearly participants in experiments have rights that can be violated easily, particularly in research involving students who may not feel free to refuse participation.  The federal government has promulgated regulations concerning the use of humans as participants in research and many universities and colleges have formed committees to monitor faculty research projects in this respect.  Student-run projects have generally not been monitored but there is no compelling reason why they should not be regulated. 

The discussion of this point should consider the degree to which the following ten items are pertinent to the experimental method:

• Privacy

• Involving people in research without their knowledge or consent.

• Coercing people to participate.

• Invading the privacy of the participant.

• Participant Welfare

• Withholding from the participant the true nature of the research.

• Deceiving the participant.

• Leading the participants to commit acts that diminish their self-respect.

• Violating the right to self-determination: research on behavior control and character change.

• Exposing the participant to physical or mental stress.

• Withholding benefits from participants in control groups.

• Failing to treat participants fairly and to show them consideration and respect.

5. What essential characteristics distinguish a true experiment from other research designs?

The major characteristic of the true experiment is the achievement of equivalency between experimental and comparison groups through the use of random assignment.

Making Research Decisions 

6. A lighting company seeks to study the percentage of defective glass shells being manufactured. Theoretically, the percentage of defectives is dependent on temperature, humidity, and the level of artisan expertise. Complete historical data are available for the following variables on a daily basis for a year: 

a Temperature (high, normal, low).

b Humidity (high, normal, low).

c Artisan expertise level (expert, average, mediocre).

Some experts feel that defectives also depend on production supervisors. However, data on supervisors in charge are available for only 242 of the 365 days. How should this study be conducted?

There is complete data on three variables temperature (T), humidity (H) and artisan experience (E) available for a year (365 days). Whereas T, H, and E are the DVs, the corresponding data on the percentage of defective glass shells being manufactured (the DV) is also available. This would permit the use of the "factorial design approach," which allows us to test for both main and interaction effects. While not an experiment, there is time series field data available for each of the 27 (3 X 3 X 3) cells in the factorial design approach. At this point it is suggested that the student consider the concept of interaction, and how the effects of temperature and humidity may not be just additive main effects, but there can be an "interaction." Analogously, a more experienced artisan may be better able to combat the negative effects of adverse temperature and humidity conditions, as compared to less experienced artisans, explaining a possible interaction between E and T or E and H.

The information says that while "supervisors" may be a factor impacting the number of defectives the data for this is available for only 242 out of the 365 days. The instructor may discuss two possibilities. Suppose there are four supervisors. In that case the study may be conducted using only the data of the 242 days for which data on T, H, E and supervisor identity is complete, to come to conclusions, using a (T X H X E X Supervisor Identity) factorial framework, with 3 X 3 X 3 X 4 cells. However this implies a loss of data for 365-242= 123 days. 

The alternative would be to: 

(1) Use the 3 X 3 X 3 X 4 framework to assign to various cells the observations for the 242 days, for which data is complete. 

(2) For the remaining 123 days classification data in terms of T, H, and E is available, but the supervisor identity is unknown. 

In such cases while T, H, and E classifications would be used as earlier, the cases can be randomly assigned between the different supervisors.

7. Much Internet advertising is priced based on click-through activity. A prospect is shown an ad on a host website based on search words he or she might have entered in a search engine such as Google or Bing.  If the prospect clicks directly on the ad he or she sees on the host website to visit the advertiser’s site the ad is considered effective and the advertiser must pay the host website for the ad. But research in 2009 revealed that while all prospects do not click on the ad they are shown on a host website, many do visit the advertiser’s site. They simply key in the advertiser’s URL directly into their browser or search engine. How would you design an experiment to determine if non-click-through ads displayed on your host website were actually effective in getting a prospect to an advertiser’s website?

In this experiment, the motivation is that the company hosting the advertising only gets paid if they can demonstrate that increased traffic of the advertiser’s website resulted from the ad placement. But, the pay structure, currently doesn’t measure the behavior described. The experiment must measure whether the person seeing the ad 1) uses a search engine, keys in the appropriate advertiser or its product name, and visits the advertiser’s website via the provided link, or 2) keys in the advertiser’s provided URL into their browser window. The first thing the student should realize is that the host of the ad currently does not have the ability to track behavior on a search engine like Google, Yahoo!, or Ask.com without their compliance. The second realization is that they cannot track behavior, and timing of behavior, of a browser like Internet Explorer or Mozilla Firefox without their assistance. This compliance will make the experiment more complex.

Assuming the partnerships can be forged, the students must determine if the above behaviors are the only one of importance. Some additional behaviors associated with what advertisers call ‘conversion’ might prove the host site’s value in a persuasive way. When you click on many internet ads, you are taken to a specific ad-related landing page that in itself has certain action objectives. For example, does the person who sees the ad, then exhibits action (1) or (2) above, actually exhibit an action such as requesting information or designing their ideal car or purchasing the product? If so, 

then the experiment should measure those actions as well. Tools are available to track where a mouse moves and where a mouse clicks on a website. Therefore, the advertiser must participate in the experiment as well.

Students should be asked whether everyone who sees the ad should be followed, or only a sample of people who see the ad. Since the ad that is shown to the prospect can often be specific to the prospect’s original search behavior (the keywords they use to see the ad in the first place), the people chosen to sample can be specific to their original behavior.

8. A pharmaceuticals manufacturer is testing a drug developed to treat cancer. During the final stages of development, the drug’s effectiveness is being tested on individuals for different (1) dosage conditions and (2) age groups. One of the problems is patient mortality during experimentation. Justify your design recommendations through a comparison of alternatives and in terms of external and internal validity. 

a) Recommend the appropriate design for the experiment. 

b) Explain the use of control groups, blinds and double blind studies, if you recommend them. 

a. For the testing of drugs we need to study/test both main effects and interaction effects. The effects of the same dosages on different age groups are not expected to be equal. To take the more obvious case, infants require lower dosages than adults. Analogously, the elderly may be more sensitive to certain drugs. Such effects refer to the interaction between age and dosage. The factorial design allows us to vary all factors simultaneously and test for main and interaction effects. A completely randomized factorial design is suggested with each of the dosage levels being randomly assigned to each of the age groups. Alternatively the randomized block design serves the same purpose, and age may be used as the blocking factor. Whether the design improves the precision of the experimental measurement depends on how successfully the design reduces the within block variance and maximizes between block variance.

b. In drug testing, control is made more stringent through the use of double blinds. When participants do not know that they are receiving the experimental treatment then they are "blind." When the experimenters also do not know whether they are giving the treatment to the experimental or the control group it is referred to as a double blind. The issue of human mortality during experiments is a complicated one. Attrition in an experimental group is expected and this is normally handled through random assignment of such cases. However, in the case of medical experiments even the few deaths have high "real" significance. The age/dosage and their interaction may in some cases, however few have main or interaction effects that are related to death. One historical solution has been to attempt to, and in experimental replications, collect enough such observations to test the age/dosage main and interaction effects in such cases alone, taking into account other possible extraneous factors impacting these cases through techniques such as covariance analysis.

9. You are asked to develop an experiment for a study of the effect that compensation has on the response rates secured from personal interview participants. This study will involve 300 people who will be assigned to one of the following conditions: (1) No compensation, (2) $10 compensation, and (3) $30 compensation.

A number of sensitive issues will be explored concerning various social problems, and the 300 people will be drawn from the adult population. Describe your design. You may find Appendix: Complex Experimental Designs valuable for this question.

A. Completely randomized design 

A = no incentive        R      A      01

B = $10 Incentive        R      B      02

C = $30 incentive        R      C      03

B. Randomized block design 

Assume that there are strong reasons to believe that the experiment should be blocked on political affiliation and that the three political classifications are:

       Democrat     Independent     Republican

No Incentives R A A A

$10                      R B B B

$30                      R C C C

C. Latin square 

Assume that a second extraneous factor, age, is believed to have an important effect. We divide all participants into young, middle age, and old groups.

Age                    Democrat        Independent         Republican

Young A C B

Middle B A C

Old C B A

D. Factorial design 

Assume that we wish to test the effect of the sex of the interviewer at the same time we test the incentives.  The factorial design might be:

      No incentive $10 $30    

Interviewer (A) (B) (C)

Male (M) MA MB MC

Female (F) FA FB FC

With the information given it is not possible to determine which design to use. If there is no apparent reason to use a more complex design, one might use the completely randomized design. If there is a useful basis on which to block, this would normally increase the precision of results as this type of stratified sampling is typically more statistically efficient than simple random sampling. With a total sample of 300 one might prefer a design with only a few cells.

10. What type of experimental design would you recommend in each of the following cases? Suggest in some detail how you would design each study:

a) A test of three methods of compensation of factory workers. The methods are hourly wage, incentive pay, and weekly salary. The department variable is direct labor cost per unit of output.

Probably the most appropriate design is a quasi-experiment called the nonequivalent control group. One might assume that there are three different factories (e.g. assembly plants) and each one will use a different compensation method. While the specific method assigned to each plant could be done randomly, this limited randomization does not give much equivalency assurance. The use of the same compensation system within each plant would at least partially guard against the contamination effect that might be found if one were to try three experimental patterns within the same plant.

b) A study of the effects of various levels of advertising effort and price reduction on the sale of specific branded grocery products by a retail grocery chain.

This case calls for a factorial design since there are two variables that are being tested simultaneously. This project might call for setting up the experiment in several cities so as to achieve different levels of advertising.

c) A study to determine whether it is true that the use of fast-paced music played over a store’s public address system will speed the shopping rate of customers without an adverse effect on the amount spent per customer.

Several different designs might be suggested here. Some will suggest a time series quasi-experiment design in which various time periods will be control periods (i.e., no music) while others will be experimental periods. In this case each time period is a unit of observation.

Another approach might be to use a randomized block design in which blocking are done on time periods with different traffic levels since this may affect shopping speed. Individuals entering the store during either control or experimental study periods would be timed.

From Concept to Practice 

12. Using Exhibit 8-7, diagram an experiment described in one of the Snapshots in this chapter using research design symbols.

In Nose for Problem Odors, there are several treatments and several measures. In symbols, the quasi-experiment might look like this:

R X1        X2 O1 O2

where X1 = cleaning of armpit and application of deodorant 

where X2 = specified time in the hot room at specific temperature

where O1 = wetness measures taken from the cotton pad

where O2 = offensive odor measure taken by human odor detective

or different formula could be applied to the different armpits of specified participants

R1 R1 X1 X2 O1 O2

R2 R2 X1 X2 O3 O4

where R1 = is the participant's right armpit 

where R2 = is the participant's left armpit 

where X1 = cleaning of armpit and application of deodorant formula 1 or 2

where X2 = specified time in the hot room at specific temperature

where O1 = wetness measures taken from the cotton pad

where O2 = offensive odor measure taken by human odor detective

where O3 = wetness measures taken from the cotton pad

where O4 = offensive odor measure taken by human odor detective

In Ramping Up Skate Sales, the experiment might look like this:

R1 O1 O2 X1 O3 O4

R2 O5 O6 O7 O8

where R1 = is a group of stores matched on sales

where R2 = is a group of stores matched on sales, control group

where X1 = is K2Sports sales ambassador

where O1-8 = is a measure of monthly K2Sports sales in the test and control stores

From the Headlines

13. Dolby, the company that provides advanced sound and light systems in movie theaters, is operating in an environment where theater ticket sales are dropping 5-10% per year due to a better variety of entertainment options, shorter attention spans, and better in-home and device streaming. Dolby Cinema now does for the eyes what it once did for the ears. In its LA operations, Dolby conducts lab experiments measuring brain waves, galvanic skin response, and heart rate, as volunteer participants experience a Dolby Cinema enhanced movie. Diagram this experiment and discuss its strengths and weaknesses.

This question is based on a CBS news story by Jeff Glor (www.cbsnews.com/videos/future-of-the-movie-theater-experience/). 

To diagram this experiment, students must first decide if the participants will be shown both enhanced and unenhanced movie clips.  Students must also determine if a control group will be used. We know participants will be hooked up to at least the three machines noted in the question. Student designs might look like this.

Without a control group, exposed to the enhanced movie clip only

O1 O2 O3 X1 O4 O5 O6

Where O1, O4 are measures of heart rate (O4 - O1)

Where O2, O5 are measures of galvanic skin response (O5 – O2)

Where O3, O6 are measures of brain wave activity (O6 – O3)

With a control group

R O1 O2 O3 X1 O4 O5 O6

RC O1 O2 O3 X2 O4 O5 O6

Where X1 is Dolby-enhanced movie clip

Where X2 is standard clip

An argument can also be made to design an experiment using the same participant with multiple measurements with two different movie clips, one enhanced and one not enhanced.

O1 O2 O3 X1 O4 O5 O6 X2 O7 O8 O9

Where X1 is Dolby-enhanced movie clip

Where X2 is standard clip

Where O1, O4, O7  are measures of heart rate (O4 - O1)

Where O2, O5, O8 are measures of galvanic skin response (O5 – O2)

Where O3, O6, O9 are measures of brain wave activity (O6 – O3)

The best part of the discussion is hearing student arguments for their design. This question works effectively for a classroom discussion or a testing excise.

ADDITIONAL DISCUSSION OPPORTUNITIES

Video or Written Case Discussion

You will find a description of each case in the Case Abstracts section of the textbook. Cases and case supplement, including discussion guides, are downloadable from www.mhhe.com/schindler13e. Cases appropriate for discussion of concepts in this chapter include the following. 

• McDonald’s Tests Catfish Sandwich 

• NetConversions Influences Kelley Blue Book

Discussion Using Prior Snapshots, PicProfiles, Pull Quotes, or Exhibits

Referenced URLs may have changed as some companies have merged and/or are renamed.

Additional Discussion Questions

One of the hardest aspects of a merger is making the IT systems of the merging companies talk with each other. That problem grows in magnitude when you are talking about merging airlines, Continental and United Airlines. While the ticketing conversion worked well, the distinctness of the air traffic tracking systems made the conversion problematic. So when the company decided that United’s system was more suitable for the merged airline, it needed to test it.  How would you design an experiment to see if your tracking system could know where flights were, what their arrival times and departure times were, what the flight numbers were, and whether they deviated from their flight plan?

The merged company actually planned an exhaustive set of tests, but the final dress rehearsal before the merging of the systems included the following: "...the transition team had an empty Continental 737 fly from Houston to El Paso and back just to make sure the ops center could track it. The team had the pilots pretend to have a mechanical problem and return to the gate. That showed up in the system. Then it had the pilots change the flight number and reroute the plane to Austin to see if that showed up. It did."  See what other mid-air or on the ground scenarios the students can generate to test.

Describe how you would operationalize variables for experimental testing in the following research question: What are the performance differences between 10 microcomputers connected in a local-area network (LAN) and one minicomputer with 10 terminals?

Students should operationally define the following variables: performance differences, microcomputers, local area network, minicomputer, terminals.  There are permissible differences in the definitions.  Below, some alternatives are covered.

Performance Differences - Define performance as the speed at which the system completes a task. Several tasks could be measured and combined in a weighted average to come up with a scalar number that represents the performance of each of the two computer setups, in a particular customer environment. For example, the following tasks could be included: time to retrieve client information from a database; time to update a database, time to complete a calculation, time to check the spelling of a document, time to print a document. Since the research question involves more than one "workstation," the tasks should be measured with varying numbers of people doing the tasks (i.e., 1, 5, and 10). The performance differences are represented by the difference between the time it takes the first system to complete the tasks and the time it takes the second system to complete the tasks.

Microcomputers - A microcomputer/PC should be defined as a personal computer containing a microprocessor with a specific speed and hard disk of specific size and speed.  It may include a monitor and specific audio options. For example, it may be an IBM compatible personal computer with a Pentium processor running at 877 MHz and a 15-gigabyte hard disk with a 10-msec access time. Or it may be an Apple with a PowerPC processor running at 500 MHz and a15-gigabyte hard disk having a 13-msec access time. 

Local area network (LAN) - A local area network is defined as a set of microcomputers connected together by cabling, able to share data, programs, printers, scanners, and etc. Several types of LANs are available and the type of LAN will affect the performance. Therefore, LANs should be further operationally defined to be token ring or Ethernet and the speed at which it runs should be defined (4 or 16 Mb/sec for token ring, 10Mb/sec for Ethernet).

Minicomputer - Any of a number of midrange computer systems, like Digital's VAX machine or IBM’s AS400. These systems operate only through terminals or microcomputers that are attached directly or via a modem. They have more complex software, faster processors, and larger hard disk storage than microcomputers. They support anywhere from several to more than one hundred terminals.

Terminals - Includes a keyboard and a screen that connects to a computer. A terminal has no "intelligence" (processor) to do computing on its own and depends on the computer to which it is linked for data storage as well.

Establishing causality is difficult, whether conclusions have been derived inductively or deductively.

(a) Explain and elaborate on the implications of this statement.

(b) Why is ascribing causality more difficult when conclusions have been reached through induction?

(c) Correlation does not imply causation. Illustrate this point with examples from business.

a. A relationship between variables is latent, but what is manifest are only the possible effects. A relationship itself can only be theoretically postulated. For instance, a higher income level may induce the purchase of higher priced cars, and this can be theoretically postulated. Yet the data focuses on a manifest variable (purchase) rather than the latent psychological processes.

b. Inductive conclusions, unlike deductive conclusions, have no “necessary” connections between facts and conclusions. Thus the conclusion of an induction may be simply one explanation for an observed fact whereas the conclusion of a deduction is the explanation, if the deduction’s requirements are met. This means that when dealing with causal relationships we require other more rigorous devices to assure ourselves that our probabilistic statements contain the least possible margin for error. Methods such as experimentation and statistical tests help to improve our confidence in ascribing cause to inductive conclusions.

c. There may be a correlation in the following variable pairs found statistically at a point of time, however there is no causal relationship between the variables; such correlations are said to be spurious.

Increases in productivity: Increases in stock offerings to the public

Decreases in job satisfaction: Decreases in the consumer price index

A classic example of spurious correlation is exemplified in the fallacious argument:

All alcoholic beverages contain water; hence an excessive consumption of water leads to the cirrhosis of the liver. 

Using yourself as the subject, give an example of each of the following asymmetrical relationships:

1. Stimulus-response

2. Property-disposition

3. Disposition-behavior

4. Property-behavior

a. Stimulus-response: When you are challenged to justify your position during a management meeting your pulse rate increases rapidly, and you speak out strongly in defense of your position.

b. Property-disposition: You are a member of a minority ethnic group and this makes you very sensitive to ethnic type comments by others.

c. Disposition-behavior: You have strong opinions about the degradation of our physical environment by some industries; as a result you are highly selective in choosing the companies with whom you interview for career opportunities.

d. Property-behavior: You have grown up as a member of the upper-lower social class and now follow the typical consumption practices of that class.

Why not use more control variables, rather than depend on randomization as the means of controlling extraneous variables?

There are virtually an infinite number of possible extraneous variables that may confound a causal relationship. Many are unanticipated and unidentified. We also have a limited ability to control more than a few variables. By randomization we can, within specific limits of variance, expect to equalize out the influence or potential influence of these many extraneous variables. We can, however, control for a few variables that are expected to be most important. By so doing we can assume that they do not confound our study results.

Researchers seek causal relationships by either experimental or ex post facto research designs.

(a) In what ways are these two approaches similar?

(b) In what ways are they different?

a. These two approaches are similar in their objective of trying to show IV-DV, or causal relationships, basically by means of:

1. Studying co-variation patterns between variables. 

2. Determining time order relationships.

3. Attempting to eliminate the confounding effects of other variables on the IV-DV relationship.

They often use the same data collection and data manipulation methods. For example, either may use interviews or observation, use certain statistical methods, and the like.

b. They differ in their ability to measure causal effects. In experimental design we can set up situations, manipulate variables, assign participants to exposure or control groups, and control other variables. With ex post facto research, we must accept what is, or what has been, uncover comparative groups that have been exposed and others who have not been exposed to the “causal” factor, attempt to learn the time order effect after the fact, and attempt to “control” other variables by various after-the-event statistical or classification procedures. Given these problems it is easily apparent why experimental design is the more powerful of the two methods for causal analysis.

中国经济管理大学

中国经济管理大学MBA公益课堂：

中国经济管理大学|中国经济管理大学|中国经济管理大学|中国经济管理大学培训|MBA实战|中国经济管理大学|MBA培训|硕士研究生|职业资格|管理培训