I am part of the team creating content for the IJP exams and have been wondering about how questions are selected when a user generates an exam.
Currently, each question belongs to an exam type (like the L1 exam) and has one of two different question types (“rules question” or “policy question”); whenever a user generates an exam, that exam pulls a certain number of questions from its rules pool and a certain number of questions from its policy pool (18 rules questions and seven policy questions for that L1 exam).
Could these pools be made more granular? A simple example would be one pool for “rules: combat phase,” one for “rules: other,” and one for “policy.” (I did see the “topics” field to categorize the question, but this seems to be purely descriptive) If there was an imbalance between combat questions and other questions in the overall question set, this feature could guarantee that each L1 exam still has a good split between combat-oriented questions and other rules questions.
Another use case would be a balanced but very large pool of rules questions: in that case, the exam might randomly pull too many combat questions from the pool and not enough questions on other topics.
Very granular pools would also enable a greater number of variants for the same question. Let’s say I had a pool for “rules: double-block combat math.” I could quickly write 20 questions for this pool, each with the same base scenario, but with different creatures of different sizes, and therefore different answers. One of these questions would show up in each L1 exam, but since that slot is drawn from a large pool, a user is less likely to see the exact same question twice.
The “types” of questions aren’t the right way to do this, as they are very limited, and tightly coupled into parts of the database and interface.
I do have a proposed feature where the “topics” on a question are treated as a comma-separated list of tags, and when building an exam, all else being equal, questions with at least one topic that isn’t already represented on the exam, are preferred over questions with only topics that are already covered on the exam.
However, note that this wouldn’t guarantee topic coverage on an exam, and in a situation like the one you describe with 20 near-identical questions, it is possible that many of those questions could appear on the same exam if the the candidate had already exhausted other parts of the pool that didn’t have as much depth. Other factors, like the availability of translations, may also take a higher precedence than the topics field.
Note that your exam type would have to carefully decide how to use “topic” tags, as every tag on a question would be counted towards deciding whether a question might cover new topics.
If you wish to use this, you can begin entering questions with topics with this in mind, and I can enable this feature once it becomes available.
Just to be sure, would I choose to use this feature on a per-exam basis, or would it automatically affect all exams across Judge Foundry and International Judge Program?
So, bad timing, but I just resigned from the IJP exam content team. As such, I won’t be using this feature anytime soon. I informed the rest of the team, so someone might pick it up in the future. I am sorry for any inconvenience I caused.