When we started building the sequencing engine for Adaptcourse in late 2024, we had a clear sense of what we needed it to do and almost no clarity on how to build it. That is a fairly normal starting position for a system that sits at the core of your product and has to be both correct and fast enough to make decisions during an active student session.
This is a technical account of the decisions we made, the ones we got wrong, and where we ended up. We are writing it because there is not much practical documentation out there from teams building this kind of system at the early product stage, and the academic literature on adaptive sequencing tends to either describe very large-scale systems (with resources and data we do not have) or theoretical models that do not account for the operational constraints of a real deployment.
What the Engine Actually Needs to Do
Before getting to architecture decisions, it is worth being specific about the requirements. A lesson sequencing engine in our context needs to answer one question with low latency: given this student's current knowledge state and the content graph, what should this student work on next?
That sounds simple. The complexity comes from the constraints on the answer:
- The next item must be appropriate to the student's current mastery level on prerequisites, not just on the current concept.
- The next item must serve a sequencing goal: either advancing on the current concept, addressing a prerequisite gap, or delivering a spaced review of a previously mastered concept.
- The sequencing must be explainable to teachers and curriculum directors in plain terms. A black-box ranker that produces good outcomes but cannot be explained will not get adopted in education contexts.
- The system must handle new content (items that have not yet accumulated response data) without defaulting to random selection.
- Latency must be under 200ms in the 95th percentile. Students notice delays between questions.
These constraints ruled out several attractive approaches before we got started. Pure ML ranking requires substantial historical response data per item, which we do not have for new content. Pure random sampling within a difficulty band fails the explainability requirement and performs poorly for prerequisite gap resolution. Simple rule-based systems based only on the content graph can be built fast, but they tend to be brittle when student state diverges from the graph's assumptions.
The Architecture We Started With: Weighted DAG Traversal
Our first production version was a directed acyclic graph traversal with weighted edges. The content graph defines nodes (concepts) and edges (prerequisite relationships). Edge weights encode both the strength of the prerequisite dependency and the relative importance of the prerequisite for the downstream concept.
The traversal logic was straightforward: given the student's current target concept, walk backward through the prerequisite graph to find concepts that are below threshold. Sort those by weight. Surface the highest-weight below-threshold concept as the next practice target. Within that concept, select a question from the item bank at a difficulty level calibrated to the student's current performance on that concept.
This worked well for the primary use case: a student who needs to address a prerequisite gap before proceeding. It was fast, explainable, and deterministic. We could tell a teacher exactly why a student was working on a specific concept in plain terms.
The first problem we hit: the traversal assumed prerequisite chains were trees. They are not. In a realistic math curriculum, many concepts have multiple parents. Proportional reasoning requires both ratio understanding and fraction-decimal equivalence. The DAG traversal would sometimes resolve prerequisite gaps in an order that created unnecessary work, addressing a weak prerequisite of a prerequisite before the more directly relevant prerequisite gap.
We addressed this by adding a "criticality score" to each prerequisite edge: how directly does mastery of this prerequisite predict performance on the target concept, based on response data. Edges with high criticality got prioritized in the traversal. This improved routing but created a new problem: criticality scores computed from response data were noisy for less commonly reached concept transitions, and the system would sometimes de-prioritize genuinely important prerequisites because they had low response counts.
The Mistake That Took Longest to Diagnose
The hardest bug we had in the early engine was not a traversal error. It was a student state model error that corrupted the traversal inputs.
We were modeling student mastery per concept as a single float between 0 and 1, updated via a simplified Bayesian model each time a student answered a question on that concept. The update rule was reasonably standard: correct answers pushed the estimate up, incorrect answers pushed it down, the magnitude of the update scaled with item difficulty and current estimate uncertainty.
The problem: our item difficulty calibration was wrong in the early question bank. Some items tagged as "medium" were effectively "easy" for students who had the prerequisite knowledge, and some items tagged as "hard" were actually catching a different prerequisite gap that was not reflected in the concept's prerequisite edges in the graph.
This meant a student could answer several "medium" items correctly and reach a mastery threshold on a concept without actually having demonstrated solid understanding, because the "medium" items were not actually probing the concept at medium difficulty. The student state model would update the concept's mastery estimate upward based on correct responses, but the responses were not providing real evidence for the concept's mastery.
Diagnosing this took a while because the symptom was not errors on those specific concept nodes. The symptom was anomalously high error rates on downstream concepts that depended on those nodes. Students were being advanced past concepts they had not really mastered because the evidence we were using to estimate mastery was unreliable.
The fix had two parts: better item calibration (which is ongoing and never fully finished) and a "mastery stability check" before accepting a mastery threshold update. The stability check requires that mastery estimates reach threshold across a minimum number of item variations, not just a minimum score. A student who answered five "medium" fraction addition items correctly passes the stability check. A student who answered three items of very similar type correctly does not, regardless of the score-based threshold.
Moving to a Hybrid: Rule-Based Routing, ML-Assisted Item Selection
After several months of operating the graph traversal engine, we concluded that the routing layer (which concept to target next) was well-served by the rule-based DAG approach. It is explainable, fast, and handles the prerequisite gap resolution case cleanly. The cases where it made poor decisions were almost always cases where the graph itself was wrong, not where the traversal logic was wrong. Fixing the graph fixed the routing.
The item selection layer (which specific question to present, given the target concept and the student's current state within that concept) was a different story. Rule-based item selection based on difficulty tag matching worked acceptably but produced noticeable variance in session quality. Two students at the same estimated mastery level on the same concept would get different sequences of items due to random variation in what was available at the target difficulty, and some of those sequences were genuinely better at probing understanding than others.
We added a lightweight ML ranking model at the item selection layer. The input features are: student's current mastery estimate for the concept, student's recent error patterns (which question types produced errors in the last N items for this concept), item's historical performance data (how predictive has this item been of future performance for students at this mastery level), and item's coverage of question subtypes within the concept.
The model is small by current ML standards, a gradient-boosted ranker with a few dozen features trained on our historical response logs. It does not replace the rule-based routing. It operates within the routing's output: given that the routing has decided "this student should practice fraction addition," the ML ranker selects which fraction addition items to present in which order.
The explainability concern is addressed by keeping the routing layer fully rule-based and transparent. If a teacher asks "why is this student working on fraction addition," we can answer with the traversal path. If they ask "why this specific question," we have a less precise answer, but in practice teachers are interested in the concept-level routing, not the item-level selection.
What We Would Change
Two things, if we were starting over.
First, we would invest more time in the content graph before shipping the first version of the engine. The graph's quality is the hard constraint on everything downstream. An excellent traversal algorithm with a mediocre graph produces mediocre routing. We spent more engineering time early on the traversal logic and item selection than on the graph itself, and that was the wrong priority ordering.
Second, we would separate the student state model from the mastery gate logic more clearly from the start. In our first version, the mastery gate was just a threshold on the student state estimate. This conflated two things: "what does our model estimate about this student's mastery" and "has this student demonstrated mastery at a level sufficient to advance." Those are related but distinct questions. The first is a probabilistic estimate that is always uncertain. The second is a policy decision with curriculum consequences. Having them entangled in the same threshold made it harder to tune both independently.
We have since separated them. The student state model produces a confidence distribution over mastery levels. The mastery gate operates on that distribution using a policy that can be tuned by curriculum directors (stricter for high-stakes prerequisite concepts, more lenient for concepts where spending more time is unlikely to help and forward progress is more valuable). That separation has been useful both for correctness and for the product conversation with districts who want to understand and customize how the system makes advancement decisions.
The sequencing engine is never finished. The graph gets revised as we add new content and observe routing patterns. The item ranker gets retrained as response logs accumulate. The mastery gate policies get adjusted based on retention data. That ongoing maintenance is the real operational cost of this kind of system, and it is worth planning for from the start rather than treating initial build as the finish line.