What we learned shipping an AI interviewer to production | Brain Quest
{{ post.deck }}
{{ sec.h }}
{{ para }}
- {{ li }}
{{ sec.quote }}
The market need
Screening is the most expensive unpaid work in hiring. A mid-sized company running forty open roles will put three hundred candidates through a twenty-minute first call, and almost all of that time is spent confirming things a structured conversation could have established. Recruiters know it. They do it anyway, because the alternative — a form — tells them nothing about how someone thinks.
Every team we spoke to described the same trade: they either interview too few people carefully, or too many people badly. An agent that can hold a real conversation at three in the morning, in the candidate’s own timezone, changes which of those they have to choose.
Latency is the whole product
A written assistant can take two seconds to answer and nobody minds. In a spoken interview, two seconds is a silence, and a silence reads as judgement. Candidates start explaining themselves into the gap.
We spent more engineering time on the response pipeline than on the reasoning behind it: streaming partial transcripts, starting speech synthesis on the first clause rather than the finished answer, and keeping a short spoken acknowledgement ready for the moments when the model genuinely needs time. The target was under 700ms to first audio, and everything that failed to fit that budget got moved out of the critical path.
The follow-up is where the signal is
A first answer tells you what someone has rehearsed. The second tells you whether they understand it. Our early version asked good questions and accepted whatever came back, which produced transcripts that looked impressive and ranked nobody usefully.
The fix was a small planner that tracks which criteria are still unevidenced and spends its remaining turns there. If a candidate claims ownership of a migration, the agent asks what broke. If the answer is generic, it asks once more, then moves on and records that the claim is unsupported rather than pretending it was proven.
One criterion per question. Compound questions get compound, unscoreable answers.
Two follow-ups maximum before moving on. Three feels like an interrogation.
Record unevidenced claims explicitly. A gap is information, not a failure.
Telling people, and what happens next
We assumed disclosure would hurt completion rates. It did the opposite. Candidates told at the start that they are speaking to an AI, and told exactly what happens to the recording, complete at a higher rate than the ones who work it out midway through and spend the rest of the call performing for a machine they do not trust.
The other thing they want is a way out. Every session offers a human alternative, and roughly one in twelve takes it. That is a feature: the people who opt out are usually the ones whose situation does not fit the questions, which is exactly when a recruiter should be involved.
Disclosure is not a compliance checkbox. It is the thing that makes the next twenty minutes honest.
What we would do differently
We built scoring before we built the review interface, and it cost us a month. Hiring managers do not want a number; they want the sentence in the transcript that produced the number, and the ability to disagree with it in one click. Once we shipped that, trust in the scores went up even though the scores themselves had not changed.