An implicit assumption underlying the use of personality tests in organisations (and most surveys for that matter) is that individuals will respond to items as they see themselves generally in life, across all situations – in other words, that their frame of reference is the same.
Questions in these types of tests are typically non-contextualised, for example asking respondents to rate how much they agree with the statement ‘I pay attention to details’ in the case of a personality test, or ‘I have the opportunity to contribute to issues that affect me’ in the case of an employee engagement survey.
Although questions like these tick the right boxes for best-practice survey design thanks to their objectivity and simplicity, they may in fact be harming the validity of the tests which comprise them.
Criterion validity is a measure of how well a variable predicts a particular criteria, and is usually considered in terms of concurrent (the degree of relatedness between two variables designed to measure the same thing) and predictive (the degree to which a variable can predict an outcome) validity.
In terms of predictive validity, the challenge of using non-contextualised survey items is that individuals will often adopt a particular contextual perspective (or frame-of-reference) in providing an answer, which means the measure used to predict our outcome of interest is not necessarily measuring the same content for every person.
“Non contextualized personality items are open to interpretation by respondents… As a result, when answering test items, one respondent may consider the way he or she behaves at work, and another respondent may consider the way he or she behaves in social situations… thus these respondents are not responding to the same item when taking into account their differences in item interpretation” –Mark Bing and colleagues
Intuitively, these different perspectives could certainly impact the predictive validity of questionnaire based tests for outcomes like future job performance, and a great deal of research suggests this is indeed the case.
In one study, Mark Schmit and colleagues found that adding an “at work” suffix to existing personality inventory items- for example, “I am courteous to everyone I meet at work” rather than “I am courteous to everyone I meet”- resulted in very different patterns of responding for individuals. In particular, when administered in an “at work” test environment, work-specific questions had significantly more positive responses than non-contextualised questions for all of the Big-Five traits personality traits. In a second study by the same authors, the validity of a personality measure was also found to be higher when the context of the test items (e.g. ‘at university’) matched the context of the test environment (e.g. ‘university admissions office’).
From these results, it is clear that a person’s life experiences play an important role in determining their frame of reference, and hence how they respond to questionnaires. Another important source of contextual influence on respondents are the social and cultural roles to which they adhere.
In a detailed body of research on contextualised personality (for another example see here), Daniel Heller and colleagues have highlighted the importance of social and context specific cues to a person’s frame of reference. In one study the authors found that compared to a baseline non-contextualised personality measure, participants who were asked to respond to items while recalling acting in a “friend” role scored higher in extraversion, agreeableness and openness to experience, while conscientiousness was rated more highly in a recalled “student” role. Elsewhere, the authors found significant divergence in individuals’ perceptions of home and work personalities, which independently, were significantly associated with contextually related outcomes such as job satisfaction and marital satisfaction.
Such results as these clearly indicate the importance of context and situational factors to how a person responds to test items, which has clear implications for the use of questionnaire based tests in the workplace. In a recruitment and selection context, for example, the difference between a candidate responding in a social frame of reference and a candidate responding with a work frame of reference could mean missing out on the better talent. To minimise the prediction error introduced by different frames of reference, therefore, it is important to create a common frame of reference by ensuring there is alignment between the situational context and survey items
Latest posts by Andrew Comensoli (see all)
- Behavioural Economics in Action: ScootBiz Case Study - September 16, 2015
- How Passive Leadership is Destroying Your Culture - July 3, 2015
- The People Dimension of Strategy Execution - December 9, 2014