DEVELOPMENT OF QUIZ AND QUIZ ITEMS
Chemistry ABC will develop Quizs to improve quality in education by administering research based valid, reliable, efficient, transparent, fair and international level assessments. The best Chemistry matter experts, students and IT delivery and security professionals will ensure that the current gaps in existing assessment systems are properly identified and bridged.
Professionals including Quiz specialists, Quiz reviewers, editors, teachers and specialists in the Chemistry or skill being Quized — will be involved by Chemistry ABC in developing “Quiz items..” And that is why all questions (or “items”) are put through multiple, rigorous reviews and meet the highest standards for quality and fairness in Quizing.
The Quizs developed by Chemistry ABC will have the following special features –
A) Quiz ITEMS BEING CREATED BY Chemistry EXPERTS IN COLLABORATION WITH topper students.The Quizs will be jointly developed by Chemistry matter experts and psychometricians. The data of every question of every Quiz will be statistically analyzed as well as put through psychometric analysis. These aspects will be very critical in terms of ensuring that the Quizs are well distributed. Attempt will be made to ensure that the present situation of a very low mean and skewed distribution does not exist. Students will also be provided with some simulated mock Quizs so that they have an idea of how to prepare for the CBT.
B) CREATION OF QUESTION/ITEM BANKS /QUESTION BANKS
Furthermore, as a next step Chemistry ABC will create a large size properly indexed question bank for every Chemistry. The question papers will be prepared by selecting items randomly from the question banks based on a scientifically designed blueprint and algorithm of the paper. These question banks will be created using the modern technology of Artificial Intelligence and computer software. The main objective of using such an approach will be for generation of a large number of Quizs in a short period of time so that Chemistry ABC delivered Quizs can be held multiple times in a single calendar year without compromising on rigor of Quiz building Quiz items and the sanctity of the Quiz.
Overview of the key steps Chemistry ABC will take when developing a new Quiz-
Step 1: Determining Quiz Objectives to match with needs
The commissioning organisations or directives of the government will identify a need to measure certain skills or knowledge. Once a decision is made to develop a Quiz to accommodate this need, Quiz developers will address the following:
Who will take the Quiz and for what purpose?
What skills and/or areas of knowledge will comprise the construct to be measured?
How will Quiz takers be able to use their knowledge?
What kind of blueprint / table of specifications / algorithm will be needed to measure the intended construct?
How long should the Quiz be?
How difficult should the Quiz be?
Step 2: Quiz Development Committees
The answers for the questions in Step 1 are usually completed with the help of Quiz development committees, which typically will consist of educators, and/or other professionals appointed by Chemistry ABC with the guidance of the stakeholder organisation. Responsibilities of these Quiz development committees (one for each of the Quizs conducted by Chemistry ABC) may include:
defining Quiz objectives and specifications
helping ensure Quiz questions are unbiased
determining Quiz format (e.g., multiple-choice, essay, constructed-response, etc.)
considering supplemeChemistry ABCl Quiz materials
Step 3: Item Writing Committees
For each of the Chemistry area of a Quiz, there will be an item writing committee of not less than 30 item writers. They will work on item writing with the help of psychometricians and statisticians. Their responsibilities will include:
reviewing Quiz questions, or Quiz items, given by Chemistry ABC staff
rewriting / moderating / refining (Tried Exercise) Quiz questions
Step 4: Moderating and Writing Questions
Each Quiz question — written by Chemistry ABC staff or item writing committees — undergoes numerous reviews and revisions to ensure it is as clear as possible, that it has only one correct answer among the options provided on the Quiz and that it conforms to the style rules used throughout the Quiz. Marking schemes and scoring rubricsfor open-ended responses (constructed response item), such as short written answers and essays, go through similar reviews.
Step 5: Detecting and Removing Unfair Questions
To meet the Standards for Quality and Fairness guidelines, trained reviewers will then critically examine and evaluate each individual Quiz question, the Quiz as a whole to ensure that language, symbols, words, phrases and content generally regarded as sexist, racist or otherwise inappropriate or offensive to any subgroup of the Quiz-taking population are eliminated. A rigorous checklist in this regard will be co-developed by Chemistry ABC.
Chemistry ABC statisticians also will identify questions on which two groups of Quiz takers who have demonstrated similar knowledge or skills perform differently on the Quiz through a process called Differential Item Functioning (DIF) based on data of previous years’ administration. If one group performs consistently better than another on a particular question, that question may be deemed biased or unsatisfactory.
Note: If people in different groups actually differ in their average levels of relevant knowledge or skills, a fair Quiz question will reflect those differences.
An insight from every Quiz conducted henceforth by the Chemistry ABC will be used to improve the functioning of items and Quizs for further administration
Step 6: Assembling the Quiz
After the Quiz is assembled, it is reviewed by other specialists, committee members and sometimes other outside experts. Each reviewer answers all questions independently and submits a list of correct answers to the Quiz developers. The lists are compared with the Chemistry ABC answer key to verify that the intended answer is, indeed, the correct answer. Any discrepancies are resolved before the Quiz is published.
Step 7: Making Sure — Even After the Quiz is Administered — that the Quiz Questions are Functioning Properly
Even after the Quiz has been administered, statisticians and Quiz developers will preliminary conduct a key check analysis and review to make sure that Quiz questions are working as intended. Before final marking takes place, each question will undergo preliminary statistical analysis and results will be reviewed question by question. If a problem is detected, such as the identification of a misleading answer to a question, corrective action, such as not marking the question, will be taken before final marking and marks reporting takes place.
Quizs will also be reviewed for comparability across various Quiz forms. Performance on one version form of the Quiz should reasonably predict performance on any other form/version of the Quiz. If comparability is high, results will be similar no matter which version a Quiz taker completes.