Practitioner Guidelines for Psychometric Testing

JOB REQUIREMENT EXERCISE (JRE)

When using psychometric questionnaires, such as SHL’s OPQ or MQ it is advisable to translate the requirements of the job into the ‘language’ of the questionnaire. This can then lend confidence that the constructs being explored by the questionnaire hold some relevance to job performance. This is especially if the JRE is completed by ‘job experts’* .

The purpose of the JRE process is to identify the critical make-or-break job requirements that experience suggests are linked to job success. These can then be used to help identify potential high performers. The JRE also establishes those characteristics that are less important or neutral to job performance. It can also help determine those qualities that might undermine job performance.

One important benefit of the JRE process is that it helps the decision making panel reach a necessary consensus about the person specification for the job. It makes overt areas of ambiguity, conflicting perceptions and confusion that might cause bias and adverse impact, creating a common language and perspective about the job from which a more discerning decision can be made.

The following example instructions are based upon SHL’s Occupational Personality Questionnaire (OPQ32) and Motivation Questionnaire (MQ), but can be adapted to suit other instruments.

Instructions for Geographically Diverse Job Experts**

JRE for SHL’s OCCUPATIONAL PERSONALITY QUESTIONNAIRE

PURPOSE - To identify the critical make-or-break job requirements. These will be used to identify potential high performers.

JOB TITLE OF JOB BEING ASSESSED:

NAME OF JOB ASSESSOR:

DATE:

PLEASE NOTE: The validity of the exercise is enhanced if the Assessor is directly responsible for the job concerned. Remember, the JRE relates to the job demands, not how the current job holder performs in the job.

1. Please study the right hand descriptions of the 32 competencies listed on the accompanying chart .
2. Rate each one according to its general importance for the job†. Place a cross as follows:

    

   9 or 10 - CRITICAL to first rate job performance

   7 or 8 - IMPORTANT to good job performance

   5 or 6 - USEFUL in a general sense, although does
   not contribute greatly to job performance

   3 or 4 - IRRELEVANT for it offers no advantage to
   job performance

   1 or 2 - UNHELPFUL for it could adversely affect
   job performance

3. Of the 32 competencies being rated, decide which are the first five most important for job success (these will be chosen from those competencies you have rated 8, 9 or 10). Having made your ranking, write it in the box alongside the relevant competency, ‘1’ for the most important, ‘2’ for the next, and so on.
4. Now study your pattern of ratings and rankings. What issues or concerns would you like to raise in any subsequent discussion? Please consider factors rated 1 or 2 as well as those with very high scores.

The task of the HR or external facilitator would then be to collate these JREs and initiate a debate around those factors where there are disparate points of view. However, it is not always essential that there is total Panel consensus about those factors; it is often enough that these differing views have been aired and that they are now known by all. This can inform the debate and ensure that minority views are heard.

JRE for SHL’s MOTIVATION QUESTIONNAIRE

PURPOSE - To identify the critical make-or-break job requirements. These will be used to identify potential high performers.

JOB TITLE OF JOB BEING ASSESSED:

NAME OF JOB ASSESSOR:

DATE:

PLEASE NOTE: The validity of the exercise is enhanced if the Assessor is directly responsible for the job concerned. Remember, the JRE relates to the job demands, not how the current job holder performs in the job.

1. Please study the right hand descriptions of the 18 competencies listed on the accompanying chart.
2. Rate each one according to its general importance for the job†. Place a cross as follows:

   9 or 10 - CRITICAL to first rate job performance

   7 or 8 - IMPORTANT to good job performance

   5 or 6 - USEFUL in a general sense, although does
   not contribute greatly to job performance

   3 or 4 - IRRELEVANT for it offers no advantage to
   job performance

   1 or 2 - UNHELPFUL for it could adversely affect
   job performance

3. Of the 18 competencies being rated, decide which are the first five most important for job success (these will be chosen from those competencies you have rated 8, 9 or 10). Having made your ranking, write it in the box alongside the relevant competency, ‘1’ for the most important, ‘2’ for the next, and so on.
4. Now study your pattern of ratings and rankings. What issues or concerns would you like to raise in any subsequent discussion? Please consider factors rated 1 or 2 as well as those with very high scores.

An important variant on this theme is when a selection or other decision making Panel can be gathered together prior to the assessment process. The following represents a process for allowing the Panel to explore the issues and gain greater understanding of differing perspectives around the table. Again, this lessens the danger of undue bias and subjectivity intruding on decision making.

GUIDELINES FOR PANEL-BASED JRE COMPLETION

PURPOSE

1. To create a person specification for the job in the language of the psychometric measures used.
2. To familiarise the selection Panel with the Occupational Personality Questionnaire (OPQ) and Motivational Questionnaire (MQ) factors.
3. To create a consensus among the Panel about critical factors for job success.

STEP 1

a. Issue JRE to all Panel members.

b. Panel members completed JRE individually.

c. One Panel member collates results and seeks to create a Panel consensus (the consensus is NOT the mean) regarding (1) the profile itself and (2) the rank order of the critical factors.

d. In situations of particular job significance, seniority or where scenario-based questions are to be employed by the Panel, a 'card-sort'*† approach is recommended. This should involve:

The most widely employed method for assessing people is the interview. The frailties of unstructured interviews are well documented and may well lead to adverse impact. A very real and measurable improvement in interviewing can be obtained if more structure is introduced. This structure usually involves including four types of question:

• questions that ask the person how they would respond to or previously have responded to different types of challenge;
• knowledge questions about the type of work the job entails;
• specific questions about the skills of the job, using simulation or a job sample, often employing the language of the job;
• questions about the person’s preparedness to meet the various requirements of the job (i.e. to complete certain tasks, in certain ways, within specific time frames).

The superiority of the structured interview using such an approach is well documented. Because the structured interview is based upon a more thorough understanding of the requirements of the job, more pertinent and telling issues are explored with the interviewee. This places all candidates on a ‘level playing field’, bringing more objectivity into HR decision-making.

The JRE process, therefore, can prompt questions that explore critical job requirements.

1. Listing all factors noted by Panel members that have been awarded a top five ranking.
2. Write factor names on separate cards. Rank cards according to job importance.
3. Collectively gain consensus and narrow down to those three critical factors on which scenario-based questions will be constructed.

STEP 2

a. Completed JRE returned to HR.

b. HR compares JRE with whatever population statistics they possess for that job family.

c. HR discusses the JRE with the Panel representative, challenging their thinking where necessary.

STEP 3

a. After assessment of candidates plot each candidate on a separate JRE on which a population mean for that job family has also been indicated.

b. With highlighter pen indicate those factors where three point discrepancies exist.

c. Undertake PMI Analysis (see separate Appendix) ready for feedback to Panel.

A very useful variant to this approach, which can be a powerful means of professional development, is for the employee to undertake research into the perceptions about their job from a range of stakeholders using the JRE format. This gives the person considerable ownership over their development and, if they complete the JRE themselves, allows a negotiated appreciation of the person specification for their job to emerge. This is made all the more powerful if the job assessors are held in high esteem by the job encumbent.

This type of enquiry might be of two main types:

• how the person has demonstrated such a characteristic in the past (they would describe the Situation or Task, the Action they took and the Result achieved – a STAR based answer);
• how they would respond if faced with a specific scenario (here the Panel would need to undertake some prior research which displays how high performers differ from average and low performers in the way they respond – thoughts/ feelings/ behaviour - to some critical incident).

In fact, both of these types of question appear to have similar utility, and do correlate positively with each other. This is intuitively clear when it is understood that the latter approach creates responses that are built upon a person’s experience of the former. Thus, it is a very reasonable premise that a person’s stated judgement about how they would react is strongly based upon past experience in similar situations: past behaviour is likely to determine future behaviour, as will previous training and immersion in similar contexts where others have been observed to respond to parallel challenges.

GUIDELINES FOR USE OF JRE IN COACHING AND DEVELOPMENT SITUATIONS
PURPOSE

1. To clarify the job demands for the job incumbent.
2. To reveal the inevitably disparate expectations of the various job ‘customers’.
3. To provide a vehicle for reconciling conflicting job demands.
4. To accommodate the jobholder to the inevitable tensions that are inherent in most managerial and professional jobs.

RECCOMMENDED STEPS

a. Jobholder and coach select a representative sample of ‘job experts’ to undertake a 360° job analysis using the JRE format. These experts would be the jobholder and all those who interface closely with the job from above, below and across the organisation.

b. Collate findings and profile on one sheet noting areas of congruence and disparity, both in the profile itself, but also in the rankings of critical factors. Note particularly any discrepancies between the jobholder’s perception of the position and that of others, especially boss and subordinates. (It is of special importance here, if the person has two people to whom they report, that these JREs are compared and contrasted).

c. Jobholder takes responsibility for sharing findings with others in one-to-one meetings and/ or in a group setting. Varying perceptions are discussed and debated, noting areas of inevitable tension and those where movement is possible.

d. Map jobholder’s OPQ and MQ profiles onto the emerging ‘template’ in order to devise an action plan which focuses on behavioural adaptation to job-customer expectations.