How to Interview an ML Engineer When Nobody on Your Team Is One

Eleven interviews in, and the Founder still could not tell us which of the Candidates was any good. He has hired Backend, Frontend and Platform Engineers for a decade. Put a Machine Learning Engineer in front of him and his usual instincts stopped working.

If you are making your first ML hire without an ML person in the building, you are in the same position, and you can still run a credible interview loop.

You just have to test different things.

Stop trying to assess the maths

The instinct is to find someone who can quiz the Candidate on gradient descent, or transformer internals, or whatever the current shibboleth is. Resist it.

For one thing, you cannot judge the answers. For another, it is the wrong test. Very few applied ML roles fail because the person did not understand the maths. They fail because the person could not frame a problem, could not tell when a model was quietly wrong, or could not work with the rest of the engineering team.

Those are all things a non-specialist panel can assess properly.

Ask for a real artefact

Whiteboard exercises are close to useless here. Ask instead for something the candidate actually built and shipped. A model in production. A pipeline. An evaluation harness. Something with consequences attached.

Then have them walk you through it for forty minutes, and treat it like a code review of the decisions rather than the code.

You want the story: what the problem was, what they tried first, what broke, what they changed, what it cost, and what it is doing now. If they cannot tell that story clearly, that is information. Strong Engineers of any flavour can explain their own work to a smart person outside their specialism.

Five questions that do most of the work

How did you choose your success metric, and who agreed to it? Weak Candidates describe accuracy. Strong ones describe a negotiation with a product owner about what kind of mistake was acceptable.

What happened when production performance did not match your evaluation? Everyone who has shipped a model has this story. If they do not, they have not shipped a model. Listen for how they found out, not just how they fixed it.

What did you decide not to build? This separates the Engineers from the enthusiasts faster than anything else on the list. Good ML people talk you out of models regularly.

Where did your training data come from, and what was wrong with it? There is always something wrong with it. A Candidate who describes clean data has either been very lucky or has not looked.

How did the rest of the engineering team interact with your work? You are listening for whether they were a colleague or an island. Islands are expensive.

What a weak answer sounds like

Fluency without specifics. Lots of tool names, few numbers. Model choices explained by what is popular rather than what the constraints were. A production story with no failure in it. Metrics quoted without any sense of what the baseline was or what the business did with the result.

You do not need ML expertise to spot any of that. You need to keep asking “and then what happened” until you either hit bedrock or you do not.

Should you set a take-home?

Usually not, and almost never a modelling one. A take-home that asks a senior ML engineer to train a model on a toy dataset tests patience rather than ability, and the strongest candidates decline it. If you want written work, ask for something short and judgement-shaped instead: a one-page note on how they would approach your actual problem, with the assumptions they would need to check first.

That is a two-hour task, it respects their time, and it produces something your whole panel can read and argue about. It also shows you how they think about a problem they have not seen before, which is the thing you are buying.

Who should be in the room

Three people is plenty. A senior Engineer who will work alongside the hire, a product person who owns the problem the model is meant to solve, and whoever is going to manage them. The product person matters more than teams expect. A machine learning engineer who cannot hold a sensible conversation about trade-offs with product will spend a year building something nobody asked for.

Give each of them one thing to assess and write it down beforehand. Panels drift when everyone is quietly assessing everything.

Where an Outside Specialist genuinely helps

One session, not five. Bring someone in for a single deep technical conversation, ideally after your own loop has already told you the person can reason and communicate. Ask that specialist for a written view on depth, not a thumbs up or down. The hiring decision stays with you, because they will not be living with the consequences.

An Advisor, a friendly CTO at a portfolio company, or your Investor’s Technical Partner will usually do this for an hour. So will we.

Know which ML role you are actually hiring

Half of the interviews that go badly were mis-scoped before anyone sat down. Applied ML, LLM and AI engineering, ML platform and MLOps, and research engineering are four different jobs with four different markets, and a brief that mixes them will attract candidates you cannot compare to each other.

We have set out how those four roles differ, and how we run searches for each, on our machine learning and AI engineer recruitment page. Worth ten minutes before you write the job description rather than after.

The Honest Summary

You are not trying to work out whether this person is a better Machine Learning Engineer than the last one. You are trying to work out whether they can frame a problem, tell you the truth about what went wrong, and work with your team while doing it.

That is a judgement you are already qualified to make. The eleventh interview was not the problem. The first one was.