Sessions

Visualization in test item writing

When writing grammar test items, the item writer is often reliant on intuition and professional judgement in selecting distractors and structuring samples. However, one criticism of this is that it is open to bias and misjudgments that can reduce the reliability and validity of a test. The English Language Program at International University of Japan was recently tasked with improving the grammar portion of a placement test. The purpose of the placement test is to determine which incoming students need further language support for their English-only Masters degrees. To improve the test, we took a visual approach that helped us to make decisions about test constructs, specific items and distractors. Through analysis of a learner corpus and an academic corpus, we created visual plots, including mosaic plots, residual plots, word clouds and bar charts which work alongside standard concordancing software in helping us to improve the placement test. This poster focuses on how we put visualization into practice in order to construct test items by explaining how we interpreted the various plots. The poster will also outline how the plots were generated and offer further suggestions for automating this process in the future.