Early-Warning Classifier for Student Outcomes

Problem

The institution ran an early-warning system from an external vendor that flagged students likely to struggle in a module. I built an internal replacement. It predicts a three-way outcome per student per module: high (final mark 80 or above), middle (60 to 80), or at-risk (below 60). Two versions run per course, one at the start of the semester using only prior history and one mid-semester that adds live engagement signals, matching how the vendor segmented its own predictions so the two could be compared directly.

Class imbalance set the rules

The majority class is 60 to 65 percent of rows, which makes plain accuracy misleading: a sensible model can score below the majority-class baseline on plain accuracy while doing genuinely more useful work across the other classes. I discarded plain accuracy early and used balanced accuracy as the headline metric throughout.

Validation built to prevent leakage

The problem is time-ordered, so a random train/test split would leak future-term information into training. I used walk-forward validation only: train on earlier term cohorts, validate on the next term, pool metrics across folds. Some folds were small, a few thousand rows, which drove later regularization choices.

I also ran a transferability check as a second-order test: train on one course, evaluate on a completely separate course with zero shared training rows, to confirm a pooled model would actually generalize across curricula before committing to that design. It did, at 0.62 balanced accuracy against a 0.33 majority-class baseline.

Comparing twelve model families

I ran the same comparison across twelve model families on identical features, targets, and folds: gradient boosting (LightGBM, XGBoost, CatBoost, HistGradientBoosting), tree ensembles (Random Forest, Extra Trees), linear and kernel models, KNN, Naive Bayes, an MLP, and a regression-then-bucket baseline. Models without native categorical or missing-value support ran through a one-hot, impute, and scale pipeline fit per training fold to avoid leakage.

I then repeated the whole sweep on a second, structurally different population. Both converged on Extra Trees, which I took as evidence the result was a property of the algorithm, its extra split-level randomness self-regularizing on small noisy folds, rather than a quirk of one dataset. I also diagnosed why rejected models failed rather than just ranking scores: XGBoost and SVM, for instance, showed high recall but very low precision on the minority class.

Diagnosing overfitting

Checking the train/validation gap was a mandatory step before accepting any result. It caught a 31-point gap on the first unregularized baseline, effectively memorization given the fold sizes and a high-cardinality categorical feature, plus two smaller cases later that a scoreboard-only view would have missed. I fixed these with explicit capacity limits (max depth, minimum leaf size, tree count) tuned by grid search rather than defaults. On the original baseline the fix raised validation accuracy and shrank the gap at the same time, 0.61 to 0.64 with the gap going from 31 to 12.5 points, a real joint improvement rather than a bias-variance trade-off. Final models land at a 0.6 to 1.7 point gap.

Imbalance and threshold levers, tested not assumed

I compared class weighting against categorical-aware SMOTE oversampling on the final model: statistically indistinguishable, but SMOTE carried a larger overfitting gap, so I rejected it and documented the result rather than inferring it from the literature. A weight-dampening sweep between unweighted and fully balanced showed a smooth monotonic curve with no interior optimum, so fully balanced was genuinely optimal, not a convenient default.

Separately I tested probability-threshold overrides, the technique the vendor system uses in production. They helped under-regularized models by up to 5 points of balanced accuracy but did nothing once a model was already well-regularized, since thresholding and regularization were often correcting the same miscalibration from different angles. That became a consistent rule across all four production variants, with one exception on the pooled model that I flagged and investigated rather than assumed away.

Feature engineering

Prior-history features (rolling GPA, prior pass and fail counts) are all time-safe. The mid-semester models add live signals engineered from event-level logs: quiz time-on-task and quiz-score percentile computed against same-module, same-term peers rather than raw scores, so per-module difficulty is controlled for, plus a student's concurrent cross-module average as a load proxy.

Several feature hypotheses were dropped on evidence rather than intuition: a GPA-trend slope after one validation fold turned out to be entirely missing it, and an admissions categorical set after it measurably diluted performance by fragmenting tree splits across low-signal categories. Fixing a latent pipeline bug in the process, a timestamp column that had never been cast to a date, recovered a feature's real signal.

One model or several

Extending to a second course pair, I measured module-code overlap first as a gate on whether pooling was even valid, roughly 28 to 49 percent here against under 2 percent for a candidate I ruled out. I compared a fully shared model, a shared model with a course-identity feature, and separate per-course models, and made the choice a rule rather than a judgment call: a course gets its own model only if it measurably beats the shared one. The shared multi-course model came within about a point of the dedicated models at a quarter of the models to maintain.

Results

Balanced accuracy 0.651 at the start of semester and 0.657 mid-semester, up from a 0.564 first pass. At-risk recall of 0.78 to 0.81 across the final models, at or above the top of the vendor benchmark's own 0.54 to 0.84 range on a population-matched comparison. Train/validation gap of 0.6 to 1.7 points against 31 on the initial baseline. The shared multi-course model stays within about a point of dedicated per-course models at roughly a quarter of the model count to maintain.

What's next

The models are validated but not yet wired into a monitored production loop. Drift tracking and a scheduled retrain against each new term's data are the next step.