Why ML Engineers Live or Die on Data Infrastructure

Most machine learning hires that disappoint were not doomed by the calibre of the engineer. They were doomed by the state of the data infrastructure the engineer inherited on day one.

A brilliant ML engineer dropped into a company with no reliable pipelines, no feature store, no clean training data and no way to monitor a model in production will spend months doing plumbing before they build anything, and will often leave before they get the chance.

The unglamorous foundation – how data moves, where it lives, and whether you can trust it – is the single biggest determinant of whether an ML hire pays off.

The job is mostly data, not modelling

There is a persistent myth that machine learning engineers spend their days designing novel architectures and tuning models. In practice, the modelling is often the smallest part of the work. The bulk of the job is getting data into a usable state: sourcing it, cleaning it, joining it across systems, handling missing and malformed records, and building repeatable pipelines that deliver fresh data reliably. Industry surveys have long put the share of time spent on data preparation at well over half, and for teams with immature infrastructure it can be far higher.

This matters for hiring because it reframes what “good” looks like. The strongest applied ML engineers are not the ones with the most exotic modelling knowledge; they are the ones who are pragmatic about data, comfortable with messiness, and able to build the pipelines that make everything downstream possible. If your infrastructure is early, you need someone who is energised rather than demoralised by that reality – and you need to screen for it directly rather than assuming a strong modeller will happily do the groundwork.

Feature stores and pipelines are the real product

When people imagine an ML system, they picture the model. The parts that actually determine whether it works are the pipelines that feed it and the feature store that serves consistent inputs to both training and production. A feature computed one way during training and a slightly different way in production is one of the most common and most insidious causes of models that look excellent offline and quietly fail once deployed. This is training-serving skew, and it is an infrastructure problem, not a modelling one.

Investing in this foundation is what lets a team move quickly later. Once clean, well-defined features are available and reusable, building a new model becomes a matter of days rather than months. Without it, every project starts from scratch, every engineer reinvents the same data-wrangling, and the team’s output stays stubbornly low no matter how talented the individuals are. When you hire an ML engineer into an environment like that, you are not really hiring a modeller – you are hiring someone to build the factory before they can make anything in it, and you should be honest with yourself and with candidates about that.

Monitoring is the difference between shipping and hoping

A model that ships without monitoring is not really in production; it is in production on a hope. Models degrade. The world shifts underneath them – customer behaviour changes, upstream data formats drift, a seasonal pattern that held for a year suddenly breaks. Without infrastructure to watch input distributions, track prediction quality and alert when something moves, a model can quietly get worse for months while everyone assumes it is fine. The damage is invisible precisely because nobody is measuring it.

Good ML engineers know this in their bones, and they will ask about it in interviews. If a candidate wants to understand how you monitor models, how you detect drift, and how you roll back a bad deployment, that is a strong signal – they have been burned before and learned the lesson. If your honest answer is that you have none of that yet, it becomes part of the role, and part of the sell: the chance to build the monitoring and evaluation discipline that a maturing ML practice depends on.

What this means for Hiring

The practical implication is that you cannot separate the ML hire from the infrastructure decision. Before you write the job description, take an honest inventory. How clean and accessible is your data? Do reliable pipelines exist, or will they need to be built? Is there a feature store, or does every project wrangle raw data independently? Can you monitor a model once it is live? Your answers determine both who you should hire and how you should describe the role.

If the foundation is missing, target engineers who lean towards data and platform work and who see building that foundation as interesting rather than beneath them. Be transparent in the process that the first months will be about infrastructure, not headline models — the wrong candidate will be put off, which is exactly what you want. If the foundation already exists, you can hire for modelling depth and expect faster returns, because the new engineer can build on solid ground rather than pouring it themselves.

Build the Foundation, then hire on top of it

The teams that get the most from machine learning are rarely the ones with the most impressive individual hires. They are the ones that treated data infrastructure as a first-class investment, so that every engineer who joins can build quickly and ship reliably. If you are about to make your first serious ML hire, the most valuable thing you can do is look hard at the ground they will be standing on. Fix what you can before they arrive, be honest about what remains, and hire someone who understands that in machine learning the data infrastructure is not a supporting act – it is the thing the whole practice lives or dies on.