The take-home was a small service: read a CSV of transactions, apply some rules, expose two endpoints, write tests. It had been the first stage of our hiring process for four years and it was good at its job. The submissions spread out. Some were rough and fast, some were careful and slow, some over-engineered, some missed a rule. The spread was information.

By last autumn the spread was gone. Twenty submissions in a row were clean, idiomatic, fully tested, with a README, a Dockerfile and a CI config we had not asked for. They were also nearly interchangeable, down to the structure of the test file. The candidates were not cheating in any sense we had defined. They had used the tools that every working engineer uses now, and the tools produce a good answer to a well-specified problem in an hour.

We had a test that measured whether a candidate could operate a coding agent on an easy task. Everyone can. It measured nothing.

What we were actually hiring for#

Before changing the process we had to answer a question we had been avoiding: what does an engineer do on this team that the agent does not.

The answer, after some honest discussion, was in four parts. They decide what to build, at the level of a ticket, when the ticket is vague. They notice when something is wrong, in code they did not write, before it ships. They know when the agent's output is right and when it is plausible but wrong, and the second one is the whole skill. And they carry the context of the system across weeks, so that the thing built this week fits the thing built last month.

None of those show up in a take-home with a clear spec. All of them show up when you watch someone work with an agent on a task that is a bit wrong.

The exercise#

The new first stage is a ninety minute session, on a video call, with the candidate sharing their screen and using whatever agent and editor they normally use. We say that explicitly: use your tools, we want to see how you work, not how you work with one hand tied.

The task is a small existing codebase, about 2,000 lines, that we give them at the start. It has a feature request attached, written the way a real ticket is written, which is to say incompletely. And it has three things wrong with it that we do not mention.

One is a bug in the existing code that the feature request will make visible. One is a place where the obvious implementation of the feature is wrong for a reason that is only clear from reading a different part of the codebase. And one is a test that passes and should not, because it asserts the buggy behaviour.

The candidate is asked to implement the feature. That is the whole brief.

What we watch#

We are not watching whether they finish. Most do, roughly, because the agent will produce an implementation of the feature in a few minutes. We are watching what happens around that.

Do they read the codebase before prompting, or prompt first and read the output. Both can work. The ones who prompt first and then read the output carefully are fine. The ones who prompt first and accept the output are not, and the exercise is built so that accepting the output ships the bug.

When the agent's implementation is plausible but wrong, because of the thing in the other part of the codebase, do they notice. This is the centre of the exercise. The strongest candidates notice within a few minutes, usually because they read the module the feature touches and see the constraint. The middle notice when a test fails, if they wrote a test that exercises it, which the agent will not do unprompted because it does not know the constraint exists. The rest ship it.

When they find the existing bug, what do they do. Fix it silently, fix it and mention it, note it and leave it, or ask. Any of the last three is fine. The first one is a small flag, because silently changing behaviour outside the ticket is the habit that causes incidents.

When the agent's test suite is green, do they believe it. The passing test that should fail is there to see whether anyone reads the tests as claims or as proof. The candidates who read the test and say "this is asserting the wrong thing" are the ones we want.

How do they talk to the agent. This one surprised us. There is a large difference between candidates who give the agent context, "this module has a constraint that X, implement the feature so that it holds", and candidates who give it the ticket verbatim and iterate on the output. The first group finishes faster and ships fewer bugs, and the difference is almost entirely whether they had read the code before asking.

What the exercise found#

We ran it for about forty candidates over the winter and spring. A few things came out that changed how I think about the job.

Years of experience predicted almost nothing. Some of the best sessions were from people with three years, and some of the worst from people with fifteen, and the difference was whether they had adapted their working habits to an agent or were using it as a faster autocomplete while working the way they did in 2019.

The skill that predicted the most was reading. Not writing. The candidates who read the codebase for the first ten minutes, before touching the agent, found all three problems at a much higher rate. Reading code has always been the underrated skill and the agent made it the main one.

Comfort with saying "I don't know if this is right" was the second predictor. The exercise has a moment where the agent's output looks right and is not, and the candidates who paused there and said "I want to check this against the other module" did well, and the ones who said "looks good" did not. That is a temperament as much as a skill and it is the one I would now hire for over almost anything.

The rest of the loop#

After the session there is one more technical stage, a conversation about a system the candidate has built, where we ask about the decisions and the things that went wrong. That stage did not change, because it never measured typing.

The take-home is gone. We have not missed it. The ninety minutes is more work for us per candidate than reviewing a submission was, and it tells us ten times more.

We also changed what the offer says about tools. It used to be silent. It now says that the agent is part of the job, that we expect people to use one, and that we expect them to read what it produces. The second sentence is there because the failure mode we saw in the exercise is the failure mode we see in the team, and naming it on day one is cheaper than discovering it in a post-mortem.

The exercise, in enough detail to steal#

People ask for the codebase. I will not share it, because it is the exercise, but the recipe is more useful than the artefact and here it is.

Start with a real, small service. Ours is a cut down version of a scheduling API we once ran: a Postgres schema with five tables, an HTTP layer with eight endpoints, a job that sends reminders, and a test suite with about 60 tests. Two thousand lines is the right size. Smaller and there is nowhere to hide the problems. Larger and ninety minutes is not enough to read it.

Write the feature request the way your product manager writes them. Ours is four sentences and it asks for recurring events. It does not say what happens to a recurring event's reminders, does not say whether editing one occurrence edits the series, and does not say what the API should return for a series. A candidate who asks about those things in the first ten minutes has told you a great deal.

Then plant the three problems. The existing bug is in the reminder job: it uses the event's start time in UTC and the user's timezone offset from when the event was created, so an event created before a daylight saving change fires its reminder an hour off after it. Recurring events make this visible because a series spans the change. The constraint in the other module is that the reminder job assumes one row per event, and the obvious implementation of recurring events, one row per occurrence, would send one reminder per occurrence, which for a daily event for a year is 365 reminders on the day the series is created. The wrong test asserts that a reminder is sent at the stored offset, so it passes against the bug.

None of the three are tricks. They are the kinds of problems that exist in every real codebase, and a strong engineer who reads the reminder job before implementing the feature meets all of them.

Give the candidate the repository fifteen minutes before the call, so that the first ten minutes of the session are not spent on setup. Tell them the session is recorded for the panel and that we will be watching their screen and their agent's conversation, and that both are fine to show us.

During the session, the interviewer says almost nothing. Two prompts are allowed: at the halfway mark, "how is it going", and at seventy-five minutes, "what would you want to check before merging this". The second one is where the candidates who noticed something but did not say it get their chance to say it.

Score on four lines, each yes or no, written down before any discussion with the panel: did they read before prompting, did they catch the constraint, did they question the passing test, did they handle the existing bug in a way that was not silent. Two yeses is a pass to the next stage. Four is rare and every one of them has been a strong hire.

What changed for the team#

The exercise turned out to be a mirror. The first four times we ran it, the panel disagreed on the scoring, and when we dug into the disagreement it was because the panel members themselves worked differently with agents. Two of us read first and prompt second. Two prompt first and review after. Each group was scoring the candidates who worked like them higher.

That was a useful thing to find out about ourselves, and it led to a team session where we did the exercise on each other. Everyone on the panel now does it once a year, with a fresh planted bug, and the scoring calibration meeting happens after. It is the only training we do that people ask to repeat.

The other change is that the ninety minutes became a template for onboarding. A new engineer's first task is a small feature on a real service with a real, unannounced bug adjacent to it, and their onboarding buddy watches the same four things the interview panel watched. The transition from interview to first week became continuous in a way it never was when the interview measured typing and the job measured reading.

The candidates' side#

Several candidates told us afterward that it was the first interview in a year where they felt the process matched the job. A couple said it was the first where they had been allowed to use their tools at all, which I found remarkable, because the alternative is an interview that measures a way of working nobody does any more.

One candidate, who we hired, said the thing that stayed with me. She said the exercise was the first time an interviewer had seemed to care whether she could tell when the machine was wrong. That is the job now. It was always part of the job, checking your own work and other people's, but the volume of plausible work that needs checking went up by an order of magnitude in two years, and the hiring process had to move to where the job went.