Frame of Reference: Why Perspective Matters

An implicit assumption underlying the use of personality tests in organisations (and most surveys for that matter) is that individuals will respond to items as they see themselves generally in life, across all situations – in other words, that their frame of reference is the same.

Questions in these types of tests are typically non-contextualised, for example asking respondents to rate how much they agree with the statement ‘I pay attention to details’ in the case of a personality test, or ‘I have the opportunity to contribute to issues that affect me’ in the case of an employee engagement survey.

Although questions like these tick the right boxes for best-practice survey design thanks to their objectivity and simplicity, they may in fact be harming the validity of the tests which comprise them.

Criterion validity is a measure of how well a variable predicts a particular criteria, and is usually considered in terms of concurrent (the degree of relatedness between two variables designed to measure the same thing) and predictive (the degree to which a variable can predict an outcome) validity.

In terms of predictive validity, the challenge of using non-contextualised survey items is that individuals will often adopt a particular contextual perspective (or frame-of-reference) in providing an answer, which means the measure used to predict our outcome of interest is not necessarily measuring the same content for every person.

“Non contextualized personality items are open to interpretation by respondents… As a result, when answering test items, one respondent may consider the way he or she behaves at work, and another respondent may consider the way he or she behaves in social situations… thus these respondents are not responding to the same item when taking into account their differences in item interpretation” –Mark  Bing and colleagues

Intuitively, these different perspectives could certainly impact the predictive validity of questionnaire based tests for outcomes like future job performance, and a great deal of research suggests this is indeed the case.

READ MORE  How to Create Habits That Really Stick

In one study, Mark Schmit and colleagues found that adding an “at work” suffix to existing personality inventory items- for example, “I am courteous to everyone I meet at work” rather than “I am courteous to everyone I meet”- resulted in very different patterns of responding for individuals. In particular, when administered in an “at work” test environment, work-specific questions had significantly more positive responses than non-contextualised questions for all of the Big-Five traits personality traits. In a second study by the same authors, the validity of a personality measure was also found to be higher when the context of the test items (e.g. ‘at university’) matched the context of the test environment (e.g.  ‘university admissions office’).

From these results, it is clear that a person’s life experiences play an important role in determining their frame of reference, and hence how they respond to questionnaires. Another important source of contextual influence on respondents are the social and cultural roles to which they adhere.

In a detailed body of research on contextualised personality (for another example see here), Daniel Heller and colleagues have highlighted the importance of social and context specific cues to a person’s frame of reference. In one study the authors found that compared to a baseline non-contextualised personality measure, participants who were asked to respond to items while recalling acting in a “friend” role scored higher in extraversion, agreeableness and openness to experience, while conscientiousness was rated more highly in a recalled “student” role. Elsewhere, the authors found significant divergence in individuals’ perceptions of home and work personalities, which independently, were significantly associated with contextually related outcomes such as job satisfaction and marital satisfaction.

READ MORE  Motivation and the Burning Ambition for Change

Such results as these clearly indicate the importance of context and situational factors to how a person responds to test items, which has clear implications for the use of questionnaire based tests in the workplace. In a recruitment and selection context, for example, the difference between a candidate responding in a social frame of reference and a candidate responding with a work frame of reference could mean missing out on the better talent. To minimise the prediction error introduced by different frames of reference, therefore, it is important to create a common frame of reference by ensuring there is alignment between the situational context and survey items

The following two tabs change content below.
Andrew is a full-time Management Consultant and part-time blogger who loves getting at the heart what makes businesses successful and customers happy. Read more about Andrew at his website andrewcomensoli.com

Tags:

  1. Excellent insight Andrew. Can contextualisation can also be impacted by the way leaders and managers encourage completion in staff engagement or other workplace surveys?

    1. Thanks for the comment Grant! Yes I would think that frame of reference could be influenced by manager communications in the same way that survey instructions and item contextualisation do. It’s all fundamentally about trying to keep everybody on the same page, so every piece of reinforcement helps!

Leave a Reply