Test Development: Assessments Tools and Establishing a Standard

Regardless of the tool selected, the assessment for certification must be based on the blueprint requirements. Moreover, the development process should be implemented by testing and measurement experts and informed by job experts. Job experts should be trained to develop or review test questions or set the passmark. They should also be representative of the candidate population (e.g., geographic, roles, specialties, work environment, protected groups).

The tool should be reviewed by a group of content experts, often an advisory committee. When possible, it is recommended to pilot test the assessment tool. If the certification program is offered in more than one language, there must be an equivalent translation(s) of the assessment tool. On this point, the Association of Test Publishers provides standards that should be adhered to for proper translation and adaptation of the certification test content.

When developing tests for professional certification programs, the passmark must be linked to expected on-the-job performance and consistent with the nature and intended use of the assessment. As such, it is a formal, standardized process that usually includes a criterion-referenced method. The criterion-referenced method generally fixes the passing score to a certain percentage (e.g., in school 50% or 60%) of the subject matter the test is designed to assess. With this method, hypothetically every candidate could pass or fail but only those who pass have acquired the specified level of the subject matter. The criterion-referenced approach stands in contrast to an approach in which the pass mark is based upon the distribution of scores. Using this approach, approximately 15% of candidates fail for every test administration, regardless of the difficulty or the exam or candidates’ competence level.

In Knapp’s Certification Industry Scan (2007):
76% use criterion-referenced method
6% use normative method
18% use score-selected method based on professional consensus or academic standards

When establishing the standard, one must consider the target candidates (e.g., entry level, fully working, advanced), the consequences of certification (low stakes, such as a hotel attendant or a website designer, or high stakes, such as a physician or a pilot), as well as the level of difficulty of the assessment.

The passmark cannot be an arbitrary number (e.g., 75%). Careful consideration is required: if the pass mark is set too high, so that only the best candidates pass, this may discourage candidates from obtaining the certification. On the other hand, if the assessment is too lenient, it may not be perceived as relevant and no added value is gained by obtaining the credential.

This post is based on content from 'Assessment Tools Certification' by Human Resource Systems Group, Ltd.