The failure mode
Teaching programming to a cohort creates a bottleneck that no amount of lecturing removes. Every student submits code that has to be compiled, run against several inputs, and compared to expected output. Doing that by hand, for a class, every week, is the largest time sink an instructor has — and it is precisely the work that delays feedback, which is when feedback is worth the most.
Automate the grading and a second problem surfaces immediately. A beginner blocked at 11pm has three options: guess, give up, or paste the problem into a general chatbot that hands back a complete, working solution.
That third option looks like help and is the most expensive of the three. The exercise existed to make the student produce the reasoning. A correct answer delivered from outside does not produce a programmer; it produces a submission.
Why the obvious fix breaks
"Add an AI assistant to the challenge page" is the right instinct and the wrong implementation, for three separate reasons.
A general-purpose model given the problem statement will solve the problem. That is what it is for. Asking it in the prompt to be encouraging and guide the student is a request, and requests get satisfied approximately — three exchanges in, the student says they are still stuck, and the model does the kind thing.
Worse, if the hidden test cases are anywhere in the assistant's context, they will eventually come out. Not through an attack — through a student asking what the tests check, and a helpful model answering. Once hidden tests are visible, the grader is measuring whether the student can special-case four inputs.
And a tutor that cannot see the actual failure has nothing specific to say. It can restate the topic, suggest checking types, recommend printing intermediate values. That is horoscope advice. The student already knows they are wrong; they do not know where.
The framing that worked: the sandbox decides correctness, and the assistant is only ever allowed to talk about the gap between the student's code and their own failing results.
The mechanism
Four things carry the design.
Execution is a commodity, so it is replaceable. Code runs in an external sandbox — Piston as the primary engine, Judge0 retained as a fallback for legacy challenges — behind an internal service interface. Neither engine is permitted to leak into the domain logic, because the grader is the part of this system most likely to be swapped.
The comparison is tolerant on purpose. Raw string equality is a poor grader for beginners, so a challenge declares what "correct output" means rather than inheriting one universal rule:
// The grader's leniency is a property of the challenge, not a global
// setting — a formatting exercise needs `exact`, an arithmetic one does not.
type OutputFormat =
| "exact" // byte-for-byte
| "trimmed" // ignore leading/trailing whitespace
| "lenient" // normalise line endings, trailing space, blank-line runs
| "numeric_only"; // compare the numbers, ignore everything around them
interface Challenge {
outputFormat: OutputFormat;
allowExtraOutput: boolean; // the learner who prints "Enter a number:" first
}Without this, a beginner who solved the problem gets told they failed because they printed a prompt alongside their answer — and learns that the system is arbitrary.
The tutor is given the failure, and only the failure. When a challenge enables it, the assistant receives the challenge description, the student's current code, the hint list, and their latest failing test results. Its system prompt forces a diagnose-first, four-level scaffold: a nudge, then a conceptual hint, then a two-to-five line partial pattern, then guided pseudocode. It is explicitly forbidden from producing a complete solution, from writing snippets longer than five lines, and from revealing hidden test cases. It answers in Turkish or English to match the learner.
The hidden tests are excluded structurally, not textually. The prompt says not to reveal them; the context does not contain them. Those are different strengths of guarantee, and only the second one holds when the prompt is talked around.
The edge cases that shaped it
A crash is not a diagnosis. A C program killed by SIGKILL after producing no output is a specific, common beginner failure — an unflushed stdout buffer — and reporting it as an opaque termination teaches nothing. The runner detects that shape and returns a plain-language explanation instead.
Access is decided at submission time, not at render time. An instructor targets a challenge at individual students or a whole class tag, with a start and end time in Europe/Istanbul. The service expands the tag into an explicit recipient list and notifies each student over Server-Sent Events — but the check that matters runs in the API: outside the window or outside the recipient list, the submission is refused. A later syncRecipients call folds in students who joined the class after the assignment was created, so a late enrolment is not silently locked out.
Snapshotting recipients is what makes the report trustworthy. Because the assignment records who it was sent to, completion is a fact about a fixed population rather than a query against today's class list. On time, late, or not completed — for the seven people who were actually assigned it.
The AI endpoint is a cost surface before it is a feature. Per-user rate limiting at 20 requests a minute, conversation truncation before each call, and a request-log table for auditing. Six providers sit behind one interface with an automatic failover provider, and responses stream token by token — a tutor that stalls for eight seconds is a tutor the student abandons.
Motivation has to be wired to the work or it is decoration. XP scales with difficulty, decays with repeated attempts at 0.9^(attempts-1), and pays a speed bonus under five minutes. Levels follow floor(sqrt(xp/100)) + 1. Streaks are computed on UTC calendar days, with unit tests covering the break-and-continue edge cases — because a streak that resets wrongly at midnight is worse than no streak at all.
Limits, and what I would do differently
A system prompt is a default, not a boundary. I would not describe the four-level scaffold as a guarantee. A determined student will find phrasing that walks the model further down the ladder than intended. The parts I would defend as boundaries are the ones outside the prompt: hidden tests absent from context, submission access enforced server-side, rate limits, truncation.
The scaffold is a fixed ladder. Every student climbs the same four steps at the same pace regardless of how close they already are. The honest version adapts the entry level to the diagnosis — someone with an off-by-one in an otherwise correct loop does not need a conceptual hint about loops.
No learning outcome is claimed. The platform reports its own public counters, and product analytics and in-app satisfaction surveys are live in the production build. What has not been measured is whether students taught with a constrained tutor learn more than students taught with an unconstrained one. That is the only number that would actually settle the argument this post is making, and I do not have it.
If you are putting a model next to work that a person is supposed to do themselves, the design question is not how helpful to make it. It is what it is structurally incapable of handing over.