This report describes the validation evidence for the Judgment at Work Survey, a tool for screening job applicants.

This report describes the validation evidence for the Judgment at Work Survey, a tool for screening job applicants. Based on a primary validity study, the survey is a valid predictor of job performance correlating .33 with job performance, .20 with attendance, and .18 with job longevity. There is also validity generalization evidence for situational judgment tests (see Criterion-Related Validity below) yielding a validity of .34. The parallel form reliability is .89. The survey shows no meaningful score difference between Whites and Blacks. Additional analyses show that the survey measures conscientiousness, agreeableness, and openness to experience.

This technical documentation is designed to meet various professional and legal guidelines. As such it contains language unfamiliar to many human resource and other professionals. We would be happy to answer any questions you have concerning this documentation. Questions on this report may be directed to McDaniel@workskillsfirst.com  or by calling 804.301.3036.

The Judgment at Work Survey consists of a series of situational judgment items. Each item describes a situation that one might encounter at work. Each situation is followed by several possible actions and the applicant is asked to evaluate the effectiveness of each action. The situations were generated by individuals working in a variety of different jobs and are applicable for screening applicants for a wide range of jobs. Employers seeking to determine the usefulness of the survey for a specific job may review the test content to determine the appropriateness of the test for the job. Employers may also consider the reliability and validity evidence presented in this report.

Recent research (Nguyen, Biderman & McDaniel, 2005; International Journal of Selection and Assessment) has shown that situational judgment tests with knowledge instructions (such as the Judgment at Work Survey for Customer Service) are substantially resistant to applicant faking. In contrast, personality tests are easily faked. Thus, employers who are concerned with applicant miss-representation on personality-based customer service tests may find this test particularly attractive.

Situational judgment tests tend to measure multiple constructs because the determinants of behavior in a given situation are typically a function of multiple variables including general cognitive ability, personality, and job experience. Parallel forms reliability is an appropriate reliability for heterogeneous tests. The parallel forms reliability for the Judgment at Work Survey for Customer Service is .89.

Our criterion-related validity evidence rests on a study reported in a Journal of Applied Psychology article. The research was a meta-analysis of 102 validity coefficients with a total sample size of 10,640 individuals. The mean validity, corrected for measurement error in the criterion, was .34. Almost all criteria used in the study were supervisor performance appraisals. The 90% confidence interval for the validity distribution was .16 indicating that situational judgment tests have validity in almost all applications.

The survey score is reported as a stanine score. A distribution of stanine scores has a mean of 5.5 and a standard deviation of 2.  This stanine score standardization places most applicant scores on a scale from 1 to 9. We offer the following hiring recommendations as reasonable. Applicants with scores of 7, 8, or 9 receive our highest recommendation for hire. Applicants with scores of 4, 5, or 6 receive our next best recommendation for hire. Applicants with scores of 3 and below are not recommended for hire.  Very low scores, particularly negative scores, likely indicate that the applicant responded to the items without reading the questions, read the questions but lacked the reading comprehension skills to understand the questions, or was purposely trying to look bad.

Users of the survey may wish to make their own hire recommendations based on the availability of applicants in the local applicant pool, the performance records of employees who were previously tested, and the adverse impact of the survey on their applicants. We recommend that users of the survey consult with an industrial organizational psychologist. Work Skills First, Inc. invites users of the the survey  to contact its staff with doctoral degrees in industrial/organizational psychology if the user has questions about the appropriate use of the survey

References cited in this technical documentation are listed below and can be obtained by e-mail  at no charge:

Nguyen, N. T., Biderman, M.D. & McDaniel, M.A. (2005). Effects of response instruction on faking a situational judgment test.  International Journal of Selection and Assessment., 13, 250-260.

McDaniel, M.A., Morgeson, F.P., Finnegan, E.B., Campion, M.A., & Braverman, E.P. (2001). Use of situational judgment tests to predict job performance: A clarification of the literature. Journal of Applied Psychology, 86, 730-740..

Copyright © 2006 by Work Skills First, Inc.