Oh, and by the way, we also assessed your ELC
December 17, 2024, Greg Pope, MSc
Sometimes high-stakes exams designed to assess a specific set of competencies get used to assess other competencies for which the exam was not designed to assess (much to the disappointment of psychometricians). The most common example in Canada I’ve seen is when a high-stakes exam required for licensure (e.g., an entry-to-practice knowledge exam) is also used as an assessment of candidate’s English Language Competence (ELC). The rationale for using an exam as a proxy or indirect measure for ELC is that if a candidate passes an exam that was composed of questions that are written in English, it is inferred that the candidate must also have sufficient English Language Competence to practice in their profession competently and safely. Proxy indicators are used in some fields of science in certain cases under certain conditions, and so in some ways this may make sense in an exam context – there’s no doubt that it takes a certain level of ELC to pass an exam where all the questions are written in English.
But, there are also some significant questions about this approach. How do we know what the components and level of ELC being assessed are for an exam that isn’t designed to assess ELC? And wouldn’t that level of competence in the English language vary depending on what the exam is assessing? For example, one would think that there would be a big difference in the minimum level of ELC required for a candidate to pass a highly technical entry to practice exam in a scientific area where most of the questions have math calculations, versus an entry to practice jurisprudence exam where the questions are mostly English words. In this case the technical scientific exam, which may be mostly full of Greek letters and numbers, would generally require a lower ELC level to pass compared to a jurisprudence exam that uses specialized words and complex sentence structure, grammar, and so on. Because neither the technical exam nor the jurisprudence exam were designed to measure ELC we have no idea what the level of English Language Competence being measured indirectly by the exams actually are.
While we might be able to determine a rough estimate of the reading level of the exams (e.g., using a readability score that Microsoft Word calculates) this is a far cry from assessing ELC via an assessment that is designed specifically for that purpose, such as the International English Language Testing System (IELTS). The IELTS assesses four components of English language competence, Listening, Reading, Writing, and Speaking the results of which are converted to the IELTS 9-band scale (i.e., 9 = expert with full operational command of the language to 1 = no ability to use the language except for isolated words) for each ELC area assessed. This English language assessment provides detailed information regarding all four components of ELC, whereas using another exam as a proxy to assess ELC would only provide a rough non-standardized estimate of a candidate’s competence in Reading with no assessment of Listening, Speaking or Writing competence. It becomes clear that using an exam not designed to assess ELC to assess ELC is imprecise at best and therefore risky to an organization making decisions regarding a candidate’s English language competence.
The temptation of using proxy measures can extend to other areas. For example, in order to take a computer-based or paper-based exam a person generally requires some degree of physical coordination and dexterity, such as operating a mouse to indicate responses to items or manipulating a pencil to mark a response on a bubble sheet. If a person can pass a multiple-choice computer-based or paper-based exam that requires physical coordination to do so, does that mean that those candidates have the minimum physical coordination requirements necessary to practice safely in their profession? If a candidate passes a national dental hygiene knowledge exam, can we also reasonably say that the candidate has the dexterity required to perform dental hygiene procedures in a client’s mouth? We cannot reasonably say that this is the case as there would be little to no evidence to support this decision based on the knowledge exam results. To assess hands-on skills, an Observed Structured Clinical Examination (OSCE) would be a defensible approach where an exam is specifically designed to assess competence in practical areas. In similar manner, to assess ELC we would want to use an assessment that is specifically designed to assess English language competence in a standardized way, such as IELTS, the Canadian Academic English Language (CAEL) Test, or the Canadian English Language Proficiency Index Program (CELPIP) Test.
If we have little to no information on the extent to which an exam is indirectly measuring ELC (or hand-eye coordination or anything else that the exam is not directly designed to measure), it is not defensible to assume that if a candidate can pass an exam that is written in English that they have sufficient minimal competence in the use of the English language (listening, reading, writing, and speaking) to practice independently and safely within their profession. Further, if we put in the time and significant resources to develop high-stakes exams that are defensible (where we gather significant evidence that the exam is measuring what it should be measuring in a reliable way) why would we abandon this approach to make decisions on whether candidates have minimal ELC based on them passing an exam that isn’t designed to determine ELC. In fact, this practice is rather risky for regulators as any challenges that are launched by candidates who are not licensed due to their perceived insufficient minimal competence in ELC, would find the regulator caught in a situation where there is no evidence to support the high-stakes decisions about ELC that are being made (not licensing people who are competent to be licensed). The risks also extend to protection of the public, for example a situation where an applicant is licensed because they passed an exam not designed to assess ELC, deemed to be English language competent, but in actuality are not minimally English language competent resulting in a workplace communication issue that leads to an accident that impacts public safety (licensing people who are not yet minimally competent to practice safely).
The risks of using exams to ‘sort of, maybe, roughly, indirectly’ assess other areas of competence that those exams are not designed to assess vastly outweigh any perceived benefits or conveniences of this approach. There are existing assessments available that are specifically designed to assess ELC and have significant research evidence to support what they are measuring. Let’s use those ELC assessments instead and use the exams that regulators painstakingly create only for the purposes that they are designed for.
