Two-Agent LLM Architecture Tries to Make Oral Exams Scale — Early Results Show Promise and the Obvious Gaps

A two-agent LLM pipeline splits interviewer from evaluator to run interactive oral-style exams at scale. Tested with simulated students and a small real course. Honest about what broke.

Imagine you're a restaurant inspector, but you're not allowed to see the menu. Someone behind a wall reads you descriptions of each dish, and you decide what to taste next based only on what you've heard so far. That's the core mechanism here: a two-agent architecture where the interviewer (front-of-house) never sees the rubric or the answers, and a hidden evaluator (behind the wall) tracks progress and decides what to probe next. The split exists to prevent the interviewer from accidentally leaking correct answers to students — a real and well-documented failure mode of single-agent LLM assessment systems. The committed claim: a two-agent LLM pipeline can conduct interactive, text-based oral exams that assess students' ability to make and support arguments, at scale, without leaking answers. The architecture separates the student-facing interviewer from a hidden evaluator that holds the rubric and guides the conversation. This is not the first LLM-for-assessment paper, but it is the first to use this specific dual-agent separation-of-concerns design explicitly to prevent answer leakage during conversational assessment. The evaluation approach is where this paper gets genuinely interesting and genuinely limited at the same time. The authors built a collection of simulated students — LLM personas calibrated to particular ability levels or adversarial tactics like prompt injection, answer extraction, and confident fabrication. Crucially, they used a fictional assessment topic to prevent personas from accessing real knowledge baked into the LLM's pretraining. That's a smart methodological move: if your simulated weak student can just retrieve the answer from pretraining, your test is useless. The fictional-topic control is the most replicable contribution in the paper. The real-world deployment was small: a single elective AI literacy course, used in both formative and summative contexts. Manual and post-hoc AI review of transcripts identified two failure modes — follow-up question generation sometimes missed the mark, and grading occasionally credited ideas the interviewer contributed rather than the student. Both are the exact problems you'd predict from this architecture, and the authors are transparent about them. But the sample size is too small to draw statistical conclusions about grading reliability or fairness. The ladder against prior work is thin. There's no named baseline with numbers — no comparison against, say, ProctorU's AI proctoring metrics, or structured rubric-based auto-grading systems like Codio or Gradescope's rubric tools. The paper doesn't claim to beat anything specific; it demonstrates viability of a new architecture. That's honest, but it means we're in proof-of-concept territory, not performance territory. The architecture itself — two-agent, separation-of-concerns, one agent blind to answers — is clean and well-motivated from a security standpoint. It belongs to the multi-agent LLM pipeline family, using standard LLM inference (likely GPT-class models, though the specific model isn't named in the abstract). The key structural choice is that information asymmetry IS the security mechanism. This is an elegant idea, but it's also fragile: if the interviewer model can infer the rubric from the evaluator's guidance patterns over many conversations, the wall leaks. The paper was submitted to SIGCSE TS 2027 and not accepted. The authors note a revision is in preparation. The honest read: the contribution is architecturally interesting but empirically undercooked. A larger deployment with inter-rater reliability metrics, comparison to human oral examiners, and adversarial red-teaming at scale would transform this from a promising sketch into a publishable result.