Managerial Assessment of Proficiency - Validating The MAP Assessment

Validating The MAP Assessment

Before making the assessment available to clients, the developers of MAP validated the methodology in 11 organisations. Managers were selected to cover the full range of proficiency at work from ‘excellent’ to ‘below average’. Working independently, three senior managers assigned ratings on a 5-point scale to each manager being assessed, thereby establishing a rank order.

A total of 253 managers from 11 organisations went through the video-based assessment. Their overall proficiency percentiles (average of the 12 competency scores) were compared with their senior managers’ ratings of their performance at work, using the Spearman's rank order correlation analysis.

Correlations were positive, ranging from .71 to .92. This supported the following assumptions:-

1. the competencies that were assessed enabled the developers to discriminate between high performing managers and their less effective counterparts,
2. the assessment instrument could therefore be used as a predictive index of one’s performance on the job,
3. the process constituted the basis for a ‘needs analysis’ for identifying training and development opportunities.

Cliff Lansley, Director and owner of Development Processes Group plc, the lead organisation for MAP across the UK, is undertaking an extended research and validation exercise and the results will be referenced on completion.

The construct validity can be measured by the degree to which the developers’ assessment data agrees with the actual performance of managers at work. But participants are also concerned with face validity, the degree to which they can identify with the video episodes and accept their proficiency profile as accurate. On the post-assessment evaluation sheet, 92% said they had no difficulty relating to the episodes, and 86% said that the scores they received were probably accurate.

A further examination of the list of 12 competencies will support the assumption that they can all be improved via training. This is in contrast to the characteristics, qualities, and attributes that appear in some organisations’ lists of competencies¼ self-confidence, initiative, flexibility, ambition and so on. These are best defined as personality traits, not competencies, according to Dr. Scott Parry. They are typically formed early in life and, short of clinical intervention, are not subject to significant change through participation in a training programme. The developers of MAP restricted the assessment to competencies that can be developed through training.