Data / AI Expert Interview Questions

I would check the boring causes first, because they are the common ones: training and serving computing a feature differently, leakage that let the offline set see something the live path cannot, a distribution that has moved since the training window, or an evaluation split that was not actually held out. Then I would compare the live feature values against the training distribution feature by feature, which usually finds it. A strong answer says how they would have caught it before launch, normally a shadow deployment against live traffic.

I would monitor the inputs, not just the outputs: distribution of each significant feature against the training baseline, the rate of missing or default values, and the distribution of the model’s own predictions, which moves before any label arrives. Where labels come back late, prediction drift is the early warning. Then I would decide in advance what triggers what — an alert, a retrain, or a rollback — because a drift dashboard nobody has agreed thresholds for gets ignored. A strong answer distinguishes drift that matters from noise.

When the accuracy gain does not survive contact with the constraints: latency, cost, the need to explain a decision to a regulator or a customer, or the maintenance burden of a pipeline only one person understands. Also when the extra accuracy sits in a region the business does not act on — a better score that changes no decision is not worth anything. I would rather ship the simple model, establish the baseline and the feedback loop, and earn the complexity later.

I would randomise at the user rather than the request, so a person gets a consistent experience and sessions are not correlated across arms. The primary metric should be the downstream behaviour we actually want rather than click-through, which recommendations can inflate while making the product worse, plus a guardrail for diversity or long-term engagement. I would size the test before running it and set the duration to cover a full weekly cycle. A strong answer mentions the interference risk when users share a catalogue or inventory.

First I would stop looking at accuracy, since a model predicting the majority class always looks excellent. I would pick a metric that reflects the cost of each error type — precision and recall at an operating point, or area under the precision-recall curve. Then handle the imbalance with class weights or threshold tuning before reaching for resampling, and never resample the validation set. A strong answer connects the threshold to the business decision, because that choice matters more than the sampling technique.

I would build the rule and treat it as the baseline, because it gives an answer this week and sets the bar the model has to clear. Often the rule is good enough and the real request underneath was for the problem to be solved, not for it to be solved with machine learning. If a model is warranted, the rule becomes the fallback and the comparison. A strong answer avoids both extremes — refusing the work, and building a pipeline the team cannot maintain to beat an if-statement by a point.

I would label a sample myself against the written guideline and see whether I agree with it, then measure agreement between annotators, since low agreement means the guideline is ambiguous rather than that people are careless. I would look for systematic bias by annotator, by time period, and by data source, and check how edge cases were resolved. A strong answer says what they would do about labels that turn out to be wrong — usually relabel a stratified sample and quantify the ceiling it puts on measured accuracy.

I would connect the prediction to the decision it drives and the value of that decision being right — fraud caught net of the cost of false positives, support tickets deflected, revenue from recommendations that would not have happened otherwise. Then measure it against a holdout rather than against the previous period, so seasonality does not get credited to the model. A strong answer includes the cost side: inference cost, the operational load of false positives, and the people who have to handle them.

I would quantify the damage before touching anything: retrain without the leaking feature and see how much of the reported performance was real, because that number determines whether this is a correction or a rollback. Then check whether decisions already made on the model need revisiting. I would fix the cause rather than the symptom — usually a feature computed after the fact, or a split that let the same entity appear on both sides. A strong answer includes telling the stakeholders whose numbers just changed.

Offline performance against a properly held-out set is the entry requirement, not the decision. I would want it to beat the current baseline by enough to matter, to hold up on the segments we care about rather than only in aggregate, to run within the latency and cost budget, and to have monitoring and a rollback path in place before it takes traffic. Then a shadow run against live data. A strong answer states the criteria before seeing the results, so the bar is not moved to fit them.

I would explain what the model is deciding and on what evidence, not how the algorithm works. For an individual case that means the few factors that pushed the decision and what would have changed it, which is what a reviewer or an affected customer actually wants. For the model overall, the main drivers and where it is least reliable. I would avoid presenting an attribution method as if it were ground truth. A strong answer says what the reviewer can do when they disagree with an output.

Input distributions against the training baseline, the prediction distribution, missing and default feature rates, latency and error rates on the serving path, and the business metric the model exists to move. Accuracy comes later and only where labels arrive, so the leading indicators have to be input and output drift. For rollback triggers I would define them in advance: a guardrail metric crossing a threshold, prediction distribution shifting beyond a bound, or serving errors — anything that does not require a meeting to interpret at two in the morning.

In my last Data / AI Expert project, I found a repeated bottleneck in modeling, data quality, experimentation, deployment, and business impact and turned it into a focused improvement plan. I defined the baseline, aligned the team on the metric, shipped the change in two iterations, and reviewed the result after launch. The strongest version of this answer names the exact metric you moved, the decision you owned, and the tradeoff you made to get the result without hiding team contributions.

I would describe a situation where I needed buy-in from a peer team or senior stakeholder, then show how I earned trust with evidence rather than pressure. A strong answer explains the other person’s concern, the evidence I used to reduce their risk — in Data / AI Expert work that usually means something concrete drawn from modeling, data quality, experimentation, deployment, and business impact — and the compromise that moved the work forward. The outcome should show influence through clarity, not title power.

I would pick feedback that changed a real work habit, not a harmless weakness. For example, I once received feedback that my updates were too technical for business stakeholders, so I started sending one-line context, impact, risk, and next step summaries before deeper detail. For a Data / AI Expert the feedback worth choosing is one that changed how you handle modeling, data quality, experimentation, deployment, and business impact. The answer should end with evidence that the change stuck, such as faster decisions, fewer clarification meetings, or better stakeholder confidence.