中国经济管理大学 《心理学与人力资源管理》 （MBA研究生课程班）

Chapter 4 – Criteria Concepts, Measurement, and Evaluation

      韦恩.F.卡西欧（Wayne F. Cascio）美国科罗拉多大学（丹佛校区）商学院Robert H. Reynolds全球领导与管理讲座教授，拥有罗切斯特大学工业与组织心理学博士学位。担任美国心理学会工业／组织心理学分会、美国管理学会、全美人力资源学会等多个组织会士。
　　赫尔曼·阿吉斯（Herman Aguinis）美国科罗拉多大学（丹佛校区）商学院Mehalchin管理讲座教授。曾担任美国管理学会研究方法分会会长。

Overview 
 Chapter four discusses the challenges for human resource managers with regard to the definition, measurement, and evaluation of work related behaviors.  An operational definition of criterion as performance standards (both predictive and evaluative) provides the foundation for comprehending additional criteria concepts.  These conceptual challenges include the static, dynamic, individual, and group criterion differences.  Challenges include performance and situational criterion characteristics.  Additional issues surround criterion deficiencies, contaminations, and composites.  For each challenge and issue, the authors present research findings which address those topics.  Finally, the authors present an effective research design model for ongoing study of criterion. 
Annotated Outline
I. At a Glance
1. Criterion challenges for HRM include
a. Adequacy
b. Accuracy
2. Also challenging are
a. Dynamics (changes) & multi-dimensionality
b. Unreliability of performance
c. Biases
d. Relevance, sensitivity, practicality
3. Criterion development objectives
a. Utility of procedures (costs, benefits)
b. Understanding successful performance of jobs 
4. Important focus is on composite criterion development
II. Definition
1. Yardstick for performance (future success)  or evaluative standards (observed success)
2. Predictions
a. Selection
b. Placement
c. Performance management
3. Evaluations
a. Performance
b. Attitude
c. Motivation
4. Possibilities
a. Output measures
b. Quality measures
c. Lost time
d. Trainability and promotability
e. Performance ratings
f. Counterproductive behaviors
III. Job Performance as a Criterion
1. Performance – observed behaviors relevant to the jobs 
a. Doing the jobs and
b. Outcomes from doing the job 
2. Ultimate criterion – full domain of performance, doing and succeeding   
IV. Dimensionality of Criteria
1. Static
a. Highs and lows for multiple aspects
b. Maximum and typical performances
i. Maximum – best that person can do willingly, selectively
ii. Typical – the average that a person does do, habitually
c. Contextual performance
i. Pro-social behaviors or organizational citizenship
1. Making the environment a pleasant place to work 
2. Enthusiasm, volunteerism
3. Following rules
4. Endorsing objectives
ii. Dark side or antisocial behaviors, workplace deviance 
1. Padding hours worked
2. Negative rumors
3. Gossiping
4. Hiding mistakes
d. Task performance
i. Transforming raw materials into product
ii. Providing services
iii. Distribution of finished products
iv. Planning, coordinating, supervising
2. Dynamic or temporal dimensionality
a. When should criterion measures be taken? 
b. Situations will impact measurements
c. Interests of ratee and rater will impact measurements
d. Personalities of ratee and rater will impact measurements
e. Performances may change in time over
i. Group efforts
ii. Validity of coefficients (predictive accuracies for performances) 
iii. Rank orderings of the criterions
f. Researching changes in dynamics may follow
i. Changing tasks model (jobs redefined with time)
ii. Changing subjects model (people who do work change over time) 
V. Challenges in Criterion Development
1. Job Performance Unreliability
a. Refers to the consistency of job performance over time
b. Intrinsic unreliability
i. Personal inconsistencies
ii. Lifespace pressures
iii. Environmental and situational pressures
c. Extrinsic
i. Pressures outside of job demands
ii. Extreme weather conditions
iii. Natural catastrophes
iv. Machine failures
d. Aggregate behaviors
i. No short cuts for near term or far term
ii. Not a panacea, there will always be some biases or contaminations
e. Conclusion –
i. Rank ordering of individuals will change over time 
ii. These changes do not reflect badly on individuals involved
2. Job Performance Observations
a. Differing observation methods lead to different conclusions
b. Assessment Center observations
c. Mentor observations
d. Experienced professionals in the field
e. Independent raters
f. Trainers
g. Conclusions
i. Observation methods may yield different results
ii. Observation methods must be consistent
iii. Methods must be related to the jobs observed 
3. Dimensionality of Job Performance
a. Most job performance measures use global criterion 
b. Many individual measures taken and added into global score 
c. Global scores even for lower-level jobs tend to be too simplistic
d. More attention needed for the most important predictors 
e. Conclusions
i. For specific aspects of job, single criterion (global) use works well
ii. For jobs with more aspects, multiple criteria need consideration 
VI. Performance and Situational Characteristics
1. Environmental and Organizational Characteristics
a. Organization
i. Pay
ii. Promotion policies
b. Interpersonal factors
i. Group cohesiveness
ii. Friendship 
iii. Satisfaction with supervision
c. Job-related
i. Role clarity   
ii. Task repetition
d. Personal factors
i. Moods
ii. Age
iii. Tenure
iv. Family size
2. Environmental Safety
a. Previous injuries
b. Management commitment
c. Equipment maintenance
3. Lifespace variables
a. Interactions with organization expectations and family demands
b. Task demands on & off the job
i. On call
ii. Global teams
iii. Virtual participation
c. Task challenge on first job
d. Life stability
e. Supervisor-subordinate personality match
f. Immediate supervisor’s success
4. Job and location
a. Origin of the organization
b. Type of organization
c. Number of employees
d. Number of supervisors
e. Tenure of the management
f. Other variables account for more variance than the previous 5 variables
g. Mush more study is needed in this area
5. Extraindividual Differences and Sales Performance
a. Includes influences beyond an individual’s control
b. Territory workload
c. Market potential
d. Company market share
e. Advertising effort
f. Sales experience
g. Sales effort
6. Leadership
a. Effectiveness influenced by personal characteristics
i. Age
ii. Gender
iii. Job experience
b. Effectiveness influenced by groups and organization characteristics
i. Size
ii. Structure
iii. Management behavior
VII. Steps in Criterion Development
1. All efforts must consider utility and understanding.
2. Sequential development steps
a. Job analysis and organizational needs
b. Measurements of actual behaviors relative to expected behaviors
c. Criterion dimension identifications
d. Reliable measures development
e. Predictive validity determination 
VIII. Evaluating Criteria
1. Relevance
a. Must be logically connected to the considered performance domain
b. Will reflect ranked standing of employees with respect to
i. work behaviors 
ii. outcome measures 
c. Per APA Task Force on Employment Testing of Minorities 
i. Tests must show direct measure of employees’ job proficiency after    specific length of time 
ii. These tests are not developed quickly and may change over time
d. May include both objective and subjective measures
e. Objective and subjective measures have small correlation (approx +.39)
2. Sensitivity
a. Must determine differences between effective and ineffective employees
b. May or may not be correlated to the relevance of criteria
c. May change over time
d. May not reflect actual cost or risk to organization if incorrect
3. Practicality
a. Must include whether organization has criterion skills and knowledge 
b. Must balance necessary record keeping and data collection
c. Must clarify personal responsibilities (input, analysis)  
IX. Criterion Deficiency
1. Effective criteria cover entire domain of performance
2. Including multiple criteria increases probability accurate evaluation criteria
3. Example – Domain of University Professor
a. Typical criteria
i. Teaching
ii. Service
iii. Research
b. Better criteria
i. Teaching
ii. Service
iii. Research
iv. Personal development
v. Administration
vi. Publication
X. Criterion Contamination
a. Occurs when no research or training accompanies criterion decisions
b. Example
i. Marine sergeants use of aptitude test results without understanding statistics
ii. Better to use Marines with statistical and interpretive training, too
c. Occurs when actual criterion includes variance not related to ultimate criterion
i. This effect may increase, decrease, or have no effect
ii. Operational (actual) criterion definitions must minimize bias through research 
d. Bias types
i. Prior knowledge of predictor information
1. May be innocent bias
2. May be intentional bias
3. Best to obtain actual criteria data prior to releasing predictor criteria
ii. Group Membership
1. Based on demographics
2. Similarities to raters
iii. Ratings
1. Inadequate observations
2. Poor rater training
3. Intentional rater manipulation 
4. Unintentional rater characteristic – Haloing
a. Rating all factors the same by knowing rating on one factor
XI. Criterion Equivalence
1. Meaning
a. Two variables that correlate highly without being hypothesized to relate 
i. Analyze the reliability or unreliability
ii. If correlation remains, equivalence exists 
2. Example of equivalence
a. Relationship between training and performance
3. Must consider type of criterion
4. Must consider timing of criterion measurement
5. Must consider level of criterion performance for success
6. High correlations may result in
a. Elimination of one or more criteria (fewer are more practical) 
b. Keep all criteria (more is better approach)
c. Collect, keep, convert to average score
XII. Composite Criterion Versus Multiple Criteria
1. Composite criterion
a. Composites are best “yardsticks” for overall performance
b. Must use relevant weighting for individual criterion
c. Should represent economic impacts to organization
d. Validation necessary for true economic impact
2. Multiple criteria
a. Measures should be considered independently as behavior cannot be averaged
b. Examples
i. Selling differs from
ii. Human relations differs from
iii. Organizing 
c. Should represent behavioral &/or psychological effects for organization
d. Validation necessary for increased understanding and economic impact
3. Differing assumptions
a. Nature of the constructs (criteria) considered relevant & predictive
b. Primary purpose of actual validation process
4. Resolving the dilemma
a. When research is goal, use multiple criteria, and increase understanding
b. When management is goal, use composite criterion, and improve economics
XIII. Research Design and Criterion Theory
1. Operationally define predictors and outcome criteria
2. Acknowledge the behavior and outcome domain components of performance domain
3. Recognize two way relationships between predictors and outcome criteria
a. Criterion related
4. Recognize two way relationships between predictors and psychological constructs 
a. Construct validity
5. Recognize two way relationships between psychological constructs and performance domain
a. Construct validity
b. Results in derived job specifications
6. Recognize two way relationships between performance domain and criterion measures (outcomes)
7. Recognize two way relationships between performance domain and predictor measures 
8. Recognize two way relationships between performance domain and actual job 
a. Results in job descriptions
 

XIV. Summary
1. Criterion considerations are necessary for effective human resource management
2. Conclusion qualities dependent upon
a. Specific criterion measures
b. Timing of measures
c. Conditions outside individual control
d. Situational factors
i. Potential distortions
ii. Potential biases
Discussion Questions
1. Why do objective measures of performance often tell an incomplete story about performance?   Pages 61 - 63
 Objective measures of performance are those based on factual data such as units produced, hours worked, clients seen, and number of calls made.  While easy to understand and not subject to personal biases, these types of performance measurements often tell an incomplete story because they leave out contextual and contingent performance variables.  For example, family dynamics may influence employee attendance. An employee with one or more children may be required to miss work more often than an employee with no children. However, the employee who is a parent may perform as well or better than employees with no children.  Quality of raw materials is another contingency performance variable.  Employees who have to work with materials that turn out to be inferior may be misjudged if quality of product produced is evaluated without considering the raw materials provided to the employees.  Leadership effectiveness also may influence employee performance criterion evaluation. Leaders who are more effective at communications may lead to greater performance evaluations for employees than leaders with less effective communication skills but with equivalent employees.  Performance and organizational characteristics may influence criteria. Lifespace variables, jobs, locations, extraindividual differences, and sales performance are additional variables that may be more meaningful than objective data alone. 

2. Develop some examples of immediate, intermediate, and summary criteria for (a) a student, (b) a judge, and (c) a professional golfer?  Based on information from pages 54 - 63
Student
 Immediate Criteria might include grades on tests, project grades, and attendance per course per semester. Intermediate criteria could include the ratio of credit hours attempted to credit hours earned per academic year and class ranking. Summary criteria could include cumulative GPA, class ranking within graduating class, and class ranking within major. 
Judge
 Immediate criteria might include number of cases presided over per year per category of law suit and the number of cases with decisions per year per category. Intermediate criteria might include the ratio of cases decided versus appeals filed with decisions reversed and the number of judges in training sponsored. Summary criteria might include the ratio of decisions reached relative to the citations for decision precedence and the ratio of judgments to the recurrence of recidivism. 
Professional Golfer
 Immediate criteria might include the number of tournaments completed and average score per game per tournament. Intermediate criteria might include the number of clinics conducted to promote the sport and the ratio of salary earned per tournament. Summary criteria might include the ratio of lifetime earnings to number of tournaments completed, the number of additional young golfers mentored, and the total monetary value community development efforts based on lifetime earnings. 

3. Discuss the problems that dynamic criteria pose for employment decisions.  Pages 57, 58
 After operational definitions are determined for the performance criteria, the next question to be answered is at what time in the performance cycle should the criteria be measured? The best times for each criterion will vary based on the situational, environmental, and job characteristics. The purposes for each measurement also need to be considered. For example, when making an initial performance prediction the criterion performance margins may be wider than when actual performances after training and practice have occurred.  It is also important to consider whether individual performance criteria or group performance criteria are appropriate.  Criterion performance as a function of experience is important as well.  Variations in criteria as determined by changes in the raters and the raters’ experiences must be consider in addition to the changes in the importance of the criterion over time. To summarize, criteria may be dynamic with regard to changes over time in average levels of group performance, validity coefficients, and in the rank ordering of criterion over time. The changes may be studied from three perspectives, that of the changing task model and that of the changing subjects model, and the changing raters model. 
4. What are the implications of the typical versus maximum performance distinction for personnel selection? Pages 54 - 59
 If personnel selection were absolutely accurate, the only employees who would be hired would be those employees who always perform at the maximum level possible for the position in which they are placed. However, personnel selection is not absolutely accurate and employees do not always perform at the highest level possible for the positions in which they are placed.  Individual performances will vary per employee. Personnel selection will vary in effectiveness as well.  Therefore, typical performance refers to an employee’s average level of performance and maximal performance refers to an employee’s peak level of performance. Research data indicates that employees strive for peak performance when evaluations are imminent or in progress. The challenge for personnel selection becomes the placement of individuals who will sustain predictable high levels of performance regardless of whether they are being evaluated.  Additionally, personnel selection must consider the impacts of environmental and situational factors on employees’ performance regardless of where and when performance measurements are taken during evaluation cycles. 
5. What are the implications for theory and practice of the concept of in situ performance?  Pages 61 – 63
 In situ performance refers to all of the outside influences (situational, contextual, strategic, and environmental) that may affect individual, group, team, or organizational performance.  In situ performance also considers the relationships that may exist between these influences.  Theoretical understanding of these influences exists but has been challenging to measure empirically. Therefore, some HR practices surrounding in situ performance have developed more from tactical considerations than from empirical evidence recommendations. Some in situ performance influences include environmental and organizational characteristics such as pay, promotions, group cohesiveness, tasks, and role clarity.  Environmental safety is another category of influences which considers injury, safety practices, and hazardous materials.  Lifespace variables include personal orientation, career confidence, and job stress. Job and location, territories, market potential, company market share, advertising, experience, effort, and leadership are influences included in the categories of extraindividual differences, sales performance, and leadership.
6. How can the reliability of job performance observation be improved? Pages 59, 60
 Job performance observation reliability can be improved when personal performance inconsistencies are understood and when variations external to job demands or individual behaviors are managed. Understanding personal performance inconsistencies will increase the intrinsic reliabilities of criteria. For example, understanding the lifespace variables for an employee will increase the intrinsic reliability for this person’s performance evaluation.  Managing external variations will increase extrinsic reliabilities of criteria.  For example, performance evaluations should consider machine performance, weather related challenges, and supply delays.  One approach to dealing with factors influencing the reliability of ratings is use aggregate or average behavior over situations, over time, and over raters. While neither simple nor inexpensive, taking an aggregate approach for criterion measurements may lead to fewer challenges and greater acceptance and therefore improve the quality and the effects of this process. 
7. What are the factors that should be considered in assigning differential weights when creating a composite measure of performance? Pages 67, 68
 When creating composite measures of performance, the factors that should be considered include the weighting of the criteria, which criteria, and the context of the criteria.  For example, when the evaluation criterion is total sales volume, the composite evaluation criterion should include the number of customers contacted, the length of time since to the last evaluation, the average sale per customer, and the variety of products sold relative to the total sales volume.  Each of the criteria could be weighted differently.  Sales volume for some sales people will need to make more travel related trips to further distances to complete customer contacts. Traveling further more often may take more time than a sales person making contacts in a heavily populated area. Some customer areas may be more open to products with lower product costs leading to smaller volumes for greater expended effort.  Organizations may stress that some products be sold over existing products which will influence for total sales volume, as well.  Documenting the composite criterion strategy such that both the raters and the ratees are aware will improve the overall effectiveness for the composite criterion performance. 
8. Describe the performance domain of a university professor. Then propose a criterion measure to be used making promotion decisions. How would you rate this criterion regarding relevance, sensitivity, and practicality?   Pages 69 - 71, Figure 4-3 from page 70
 The performance domain of a university professor includes teaching, administration, service, research, publication, and continued development within their specialty area. Criterion measurement to influence promotion decisions could include the following.
Teaching – Teaching could be evaluated based on student opinions, student performance on departmental and standardized tests, observations by peers, and self ratings.  Each criterion score could be added and then an average calculated to form a teaching composite criterion. While very relevant and sensitive, the practicality of this composite would be very difficult to achieve. 
Administration – A composite criterion for administration could be developed by determining adherence to university standards for policies regarding when syllabi, grades, and forms are completed.   This criterion is not as relevant to classroom performance but could be both sensitive and practical and data documenting administrative efficiency already exists within most universities. 
Service – Service to the community by way of seminars, volunteering, and outreach would vary by discipline and would present particular challenge for composite criterion development.  The measure would be relevant but would need to very flexible and therefore not too sensitive with regard to overall performance. The practicality of this measure presents challenges, particularly, in the development of fair standards and oversight. 
Research – Research needs to be evaluated by varying standards, too. For example, some research efforts lead to relatively quick results and are considered publication worthy while other efforts may take years to complete and not result in publications but do result in service to the body of knowledge for the involved disciplines.  Therefore research is relevant but should not be weighted too heavily or be too sensitive relative to the other critical performance criterion. 
Publication – Publications need to be considered by category, frequency, and diversification.  For example, some disciplines will only consider empirical journals as performance worthy but equivalent efforts and influence may result from publications in other media and other journal types. For example, textbook publications, course materials and supplements, and public speaking are all worthy of publication consideration.  Additionally, all of the effort prior to publication attempts needs to be considered regardless of actual publications. This measure is relevant and depending on the discipline and university could be considered highly sensitive. Given clear expectations and relatively simple record keeping for efforts, attempts, journals receiving submissions and the outcomes, this measure offers some practicality.  
On-going development – This criterion could be developed through the accumulation of continuing education credits, additional courses completed, degrees earned.  Additional consideration should be given to seminars, conferences, and training programs attended.  This criterion is relevant, sensitive, and practical. However, the sensitivity should be not be set too highly as the ability to complete will be determined by university budgets and by the demands of the other criteria.