How To Develop a Test: Blueprints

Test development begins with the identification of critical tasks performed by competent people working in a given profession. This is typically done through job analysis. Job analysis involves consultations with job experts as well as the review of documents that describe tasks, knowledge and skills required for the occupation (e.g., National Occupational Standards).

According to Knapp’s Certification Industry Scan (2007):
- 90% of certifying bodies utilize a formal study to identify/validate the content of the assessments; and,
- 72% of certifying bodies perform validation study updates every 5 years or less.

The results of the job analysis dictate the type of the assessment tool(s) to be developed. For example, if the analysis reveals that successful job performance is highly dependent on specialized and technical knowledge (e.g., specific IT programming language), then a tool should focus on assessing knowledge rather than skills or abilities. Alternately, if, for example, the job analysis indicates that customer focus is a crucial competency, candidates’ skills may be best assessed through a performance evaluation tool, such as observation or an on-the-job simulation. The importance of determining the competencies to be assessed is paramount. Often, a wide range of competencies and skills are identified, and in some cases, the competencies cannot be assessed by a single assessment tool. It is essential to carefully identify the competencies and skills to be assessed and select suitable assessment tools.

A test blueprint specifies the characteristics that an assessment tool must meet. It links the specific task areas from job analysis to the tool (i.e., it specifies how much weight should be given to each task area or category from occupational standards). In establishing the blueprint, content validity is essential; that is, the tool must assess what it is designed to assess.

Test blueprints typically include:
A list of competencies, skills and/or knowledge areas
The type of assessment tool (e.g., written test, structured interview)
Format of questions (e.g., multiple-choice, short-answer)
Number of questions
Proportion of questions within each category
Characteristics of questions (e.g., cognitive level, context, domain)
Scoring procedures
Format of assessment tool (e.g., paper-based, computer-based)
Target population

A blueprint is, in a sense, like a recipe. It provides instructions on how to assemble a test. This procedure ensures consistency and equivalency across different versions of the test. As such, the tool must be developed and/or revised by job experts.

This post is based on content from 'Assessment Tools Certification' by Human Resource Systems Group, Ltd.