Data Science

Why “missing” isn’t one thing

• Missing completely at random (MCAR): the missingness has nothing to do with any variable — e.g., a GPS/onboarding sync glitch that dropped distance_km for a random slice of commuters. Safe to impute without introducing bias.
• Missing at random (MAR): missingness relates to other observed variables — e.g., commuters who onboarded through a particular employer partner are less likely to have distance_km captured, but conditional on employer it’s random. Imputation using related features can work well.
• Missing not at random (MNAR): the missingness itself carries information — e.g., subsidy_code_used is missing because the commuter simply never redeemed one. Here, “missing” is the signal, and imputing a fake code destroys it.

Handling the distance_km example

With only 8% missing and no obvious reason to think it’s tied to the outcome, reasonable options include:

• Median/mean imputation (simple, but slightly biases variance downward and ignores relationships with other features like home_city).
• Model-based imputation — predicting distance_km from other features (home_city, employer_subsidy_enrolled, typical trip patterns) — usually more accurate than a single constant.
• Adding a distance_km_was_imputed binary flag alongside the imputed value, so the model can learn if imputed rows behave differently.

Handling the subsidy_code_used example

Here, missing doesn’t mean “unknown” — it means “no subsidy code was ever used,” which is itself informative (a commuter who never touches the subsidy program looks very different from one who does). The right move is usually to encode it explicitly rather than impute:

df["subsidy_code_used_flag"] = df["subsidy_code_used"].notna().astype(int)
df["subsidy_code_used"] = df["subsidy_code_used"].fillna("none")

Risks of naive imputation

• Mean/median imputation shrinks variance and can weaken real relationships between features.
• Imputing MNAR data (like subsidy_code_used) as if it were MCAR can inject a systematic bias that actively hurts model performance and interpretability.
• Dropping all rows with any missing value can silently shrink your dataset and bias it toward “easy,” fully-observed commuters, especially if missingness correlates with the outcome (e.g., newer commuters who haven’t had time to redeem a subsidy code yet).

Why one dataset isn’t enough

A model can achieve near-perfect accuracy on data it was trained on simply by memorizing it, especially if it’s flexible enough (deep trees, high-degree polynomials, large neural nets). That number tells you nothing about how the model will perform on a new commuter, a new partner city, or next month’s trips — which is the only thing GreenMile actually cares about before spending subsidy budget on outreach.

The three-way split and what each part is for

• Training set: the commuters the model’s parameters are actually fit on.
• Validation set: used to make decisions about the model — which hyperparameters, which features, which algorithm — by checking performance on commuters not used for fitting.
• Test set: touched exactly once, at the very end, to report an unbiased estimate of real-world performance.

[ ------ 60% train ------ | -- 20% validation -- | -- 20% test -- ]
        fit model               tune choices         final check only

Why the test set has to stay untouched

If you peek at test performance while iterating — trying a new feature, adjusting a threshold, swapping models — and keep the change whenever test performance improves, you’re implicitly fitting to the test set through your own decisions. The reported “test” number stops being an honest estimate of generalization and starts reflecting how many times you got to guess. This is the same mechanism as p-hacking in statistics: repeated peeking inflates apparent performance.

Practical notes

• With a smaller dataset (e.g., a program that’s only launched in a few cities), k-fold cross-validation is often used in place of a single validation split to make better use of limited data, while the test set is still held out separately.
• The split should respect the data’s structure — split by commuter_id (not by row), since the same commuter appears across many trips, to avoid leaking information about a commuter from train into test.

Why ensembles typically beat a single model

A single model’s errors come from its own particular blind spots and sensitivities to the training sample — in this case, a specific mix of commuters, cities, and trip histories. Combining multiple models whose errors aren’t perfectly correlated averages out a lot of that noise — the ensemble’s mistakes tend to cancel out more than any individual member’s mistakes do, so the combined prediction of “will this commuter switch to bike” is more accurate and stable than any single tree’s guess.

Bagging (Bootstrap Aggregating) — e.g., Random Forest

for i in 1..N:
    sample a random subset of commuters (with replacement) and a random subset of features
      (distance_km, employer_subsidy_enrolled, historical mode mix, city, ...)
    train an independent tree on that subset
final prediction = average (probability) or majority vote across all trees

Each tree is trained independently and in parallel, on a different random slice of the commuters/trips-derived features. Because the trees are decorrelated from each other, averaging their predictions reduces variance substantially without increasing bias much — this is exactly the fix for an overfit single tree.

Boosting — e.g., Gradient Boosting, XGBoost

model = weak initial predictor
for i in 1..N:
    compute residual errors of current ensemble on the switch-to-bike label
    train a new (usually shallow) tree to predict those residuals
    add it to the ensemble, scaled by a learning rate

Trees are trained sequentially, each one specifically targeting the commuters the ensemble is currently getting wrong — say, commuters with a short distance_km who the model keeps under-predicting as switch-likely. This reduces bias and often achieves higher accuracy than bagging, but it’s more prone to overfitting if run for too many rounds or with too high a learning rate, and it’s inherently harder to parallelize since each tree depends on the previous ones.

Choosing between them in practice

Random forests are a robust, low-fuss default that’s hard to badly misconfigure and resistant to overfitting — a reasonable first model for GreenMile’s switch-prediction problem. Gradient boosting (XGBoost, LightGBM, CatBoost) usually squeezes out more accuracy but needs more careful tuning (learning rate, number of trees, early stopping) to avoid overfitting — start with a random forest baseline, then reach for boosting once you’re ready to invest tuning effort for extra performance.

How to recognize it

The clearest signal is a large, persistent gap between training and validation performance: the model keeps improving on training commuters while validation performance plateaus or gets worse. Plotting both curves against training epochs or model complexity makes this visible directly — it’s often called a “learning curve” or “validation curve.”

Why it happens

Overfitting occurs when a model has enough capacity (parameters, tree depth, degrees of freedom) to fit noise and idiosyncrasies specific to the training sample of commuters — a handful of odd one-off trips, a quirky mix of cities — rather than the underlying signal that generalizes. This is more likely with small datasets (e.g., a newly onboarded partner city), very flexible models, or too many engineered features relative to the number of commuters.

Concrete mitigations, roughly in order of how often they’re reached for first

• Regularization — add an L1/L2 penalty (or max depth / min samples per leaf for trees) that discourages the model from fitting overly complex patterns in commuter behavior.
• Early stopping — stop training once validation loss stops improving, rather than training until training loss is minimized.
• Simpler model — fewer features, shallower trees, fewer layers.
• More training data — often the single most effective fix; more commuters and trip history across more cities makes memorization harder.
• Dropout / data augmentation (for neural nets) — randomly perturbing the training process forces the model to learn more robust patterns instead of specific commuters.
• Feature selection — dropping noisy or irrelevant features (like a poorly-tracked GPS-derived field) reduces the model’s opportunity to fit spurious correlations.

What to tell the stakeholder

Frame it as expected and fixable: “the model is currently too tuned to quirks in our training commuters; here are three levers we’ll pull and re-validate against, and we’ll report the honest validation number, not the training number, going forward.”

Why raw values don’t work

Most models expect numeric input and can’t directly use a raw timestamp string or a text category — and even models that can technically ingest them won’t extract the underlying patterns (e.g., “car-to-bike switches spike right after a subsidy launch, or on weekdays during rush hour”) without help.

Decomposing a timestamp

df["hour"] = df["trip_start_ts"].dt.hour
df["day_of_week"] = df["trip_start_ts"].dt.dayofweek
df["is_weekend"] = df["day_of_week"].isin([5, 6]).astype(int)
df["is_rush_hour"] = df["hour"].isin([7, 8, 9, 16, 17, 18]).astype(int)
df["days_since_program_launch"] = (df["trip_start_ts"] - CITY_LAUNCH_DATE).dt.days

Which components matter depends on the business question: a model predicting whether a trip is mislabeled (e.g., a car trip that’s actually a bike trip logged wrong) might care about is_rush_hour, while a bike-adoption model might care more about day-of-week and season (winter vs. summer biking rates).

Encoding a categorical column

• One-hot encoding: creates a binary column per category (is_city_portland, is_city_denver, …). Works well for low-to-moderate cardinality — GreenMile’s mode column (car/bike/bus/walk) is a good example, since one-hot on 4 categories is trivial — but blows up dimensionality fast with something like 42 cities.
• Target encoding: replace each city with the mean of the target (e.g., switch-to-bike rate) for that city, computed carefully on training folds only to avoid leakage. Handles high cardinality well but risks overfitting on smaller cities unless smoothed.
• Frequency encoding: replace each city with how often it appears in the data — simple and leakage-free, useful when a city’s trip volume itself is informative (e.g., newer, smaller partner cities behave differently).
• Native categorical support: some tree-based libraries (e.g., LightGBM, CatBoost) can handle categorical columns like city or mode directly without manual encoding.

The judgment call

With 42 cities, one-hot encoding is borderline — it’s usable but bloats the feature space for a linear model; for a tree-based model, native categorical handling or target encoding usually scales better and preserves more signal about which cities behave similarly (e.g., dense, bike-lane-rich cities vs. sprawling car-dependent ones).

Why “yes or no” is the wrong ask, and how to bridge it anyway

A predicted 12% reduction with a wide interval already is the honest answer — collapsing it into a guaranteed “yes, 15%” discards real uncertainty about how many commuters will actually change behavior, and can set the city up to publicly commit to a number the program may not hit. But refusing to give a straight answer at all isn’t useful either — the fix is to make the uncertainty itself explicit and decision-relevant: “our best estimate is a 12% reduction, and based on similar programs in comparable cities, the true result will likely land somewhere between 8% and 16% — here’s what would need to be true to land at the high end.”

A concrete way to frame it for a non-technical audience

• “This isn’t us guessing blindly — it’s built from how commuters in similar cities responded to comparable subsidy and messaging programs, and that historical spread is exactly why we give a range instead of one number.”
• “If we want to push toward 15%, the levers that move the estimate up are things like a richer e-bike subsidy or targeting commuters with the shortest distance_km, who are cheapest to convert — we can model those scenarios directly.”
• Anchor the number to a decision: “at our current funding level, 15% is optimistic but not guaranteed — if hitting it matters for a grant commitment, we should either increase the subsidy budget or set the public target at 10-12% and treat anything above that as upside.”

Naming limitations without undermining trust

• Be specific about what the model does and doesn’t know: “this model is trained on past program rollouts; it doesn’t know about a competing transit fare cut the city is about to announce, since that wasn’t in the training data.”
• Give a sense of typical error in plain terms: “our past predictions for similar programs have landed within about 3-4 percentage points of the actual result — which is why we’re comfortable committing to a range, not a single number.”
• Avoid false precision: presenting “12%” as if it were exact invites over-trust; framing it as “an estimated 12%, most likely between 8-16%” is more honest to the model’s actual reliability.

The underlying principle

The goal is to give the stakeholder a decision-ready answer without pretending the model has certainty it doesn’t have — an estimate plus context (the range, what drives it, a recommended target) is more useful and more honest than a bare guarantee.

Why one split can mislead you

Any single train/test split is a sample of a sample — the specific commuters and trips that end up in the test fold might happen to be easier or harder than average, or might not represent the full range of cities and seasons in the data. A model can look artificially strong or weak purely due to which rows landed where, especially when the dataset is small or a particular city is over- or under-represented.

How k-fold cross-validation works

Split commuters into k equal folds (e.g., k=5)
for i in 1..k:
    hold out fold i as the validation set
    train on the remaining k-1 folds
    record validation score
report mean and std. dev. of the k scores

Every commuter gets used for both training and validation across different folds, and averaging the k scores smooths out the luck of any one particular split. Reporting the standard deviation alongside the mean is just as important — a mean of 90% with folds ranging from 80% to 98% tells a very different story than a mean of 90% with folds tightly clustered around it.

When k-fold is the wrong tool

• Seasonal trip data: standard k-fold shuffles trips randomly across folds, which lets the model train on summer trips (when biking rates are high) to predict winter trips (when they’re much lower) — a form of leakage that hides how much biking rates swing by season. Use time-series/rolling-origin cross-validation instead, where each fold trains only on trips before the validation window, so a fold validating on January trips never trained on data from that same winter.
• Grouped data (multiple trips per commuter): random folds can put some of a commuter’s trips in train and others in test, leaking commuter-specific information (their typical distance, cost, home city). Use group k-fold, keeping each commuter entirely within one fold.
• Very large datasets: a single, well-sized validation split is often “good enough” and far cheaper computationally than retraining k times.

Practical takeaway

Cross-validation is about getting a reliable estimate, not a magic accuracy boost — if the chosen splitting strategy doesn’t match how the model will actually be used in production (e.g., predicting a commuter’s likely mode next winter using only data through last summer), the cross-validated number will be just as misleading as the single lucky split, only more confidently wrong.

Global vs. local importance

Global feature importance (e.g., built-in tree-based importance, or averaged SHAP values across all commuters) tells you which features matter most overall for predicting a switch to bike — useful for understanding the model in general, but useless for explaining one specific commuter’s score, since the features driving their individual prediction might differ from the global average.

Local/per-prediction explanations (SHAP, LIME) answer “why did the model say this about this commuter” — which is what the program coordinator actually needs.

How SHAP works, intuitively

SHAP (SHapley Additive exPlanations) is grounded in cooperative game theory: it treats each feature as a “player” contributing to the final prediction and fairly distributes credit for the prediction across features based on their marginal contribution across many possible feature combinations.

base_rate (average predicted switch likelihood)   = 9%
+ short distance_km (< 4km) contribution          = +21%
+ rising car cost_usd trend contribution          = +14%
+ employer_subsidy_enrolled contribution          = +8%
- long tenure as primary_mode=car contribution    = -6%
= final predicted switch likelihood               = 46%

This gives a clean, additive breakdown: base rate plus each feature’s push up or down, summing to the actual prediction.

Why a stakeholder needs this

Program coordinators, account managers, and city officials all need to act on a prediction, not just trust it — a bare score of “46%” gives no guidance on what to actually say or offer, while “short commute distance and rising gas costs” gives the coordinator a concrete pitch to make (“here’s an e-bike subsidy that could cut your $180/month car cost”). It also builds trust: stakeholders are far more willing to act on a model when they can sanity-check that it’s using sensible signals rather than something spurious.

Caveats worth mentioning

SHAP values are an approximation of feature contribution under specific assumptions (e.g., feature independence, for the faster approximate methods) — they explain the model’s reasoning, not necessarily the true real-world causal driver of a commuter’s decision, and correlated features (like distance_km and cost_usd, which move together) can split credit in ways that are technically correct but confusing to explain simply.

Why accuracy fails here

If 99% of commuters don’t fully switch to bike in a quarter, a model that always predicts “won’t switch” scores 99% accuracy while providing zero value to the program — it never identifies the commuters actually worth targeting with a subsidy nudge. This is the single most common trap when a model looks great on a dashboard but is useless in practice.

Fixing the evaluation first

• Look at precision, recall, and F1 for the “switched” class specifically, not just overall accuracy.
• Use a confusion matrix to see the actual counts, not just a summary percentage.
• Consider PR-AUC over ROC-AUC, since ROC-AUC can also look artificially strong under imbalance (see the ROC-AUC vs. precision-recall discussion for GreenMile’s switch-prediction model).

                    Predicted: No Switch   Predicted: Switch
Actual: No Switch          9,801                 18
Actual: Switch                99                  2      <- only 2 of 101 real switchers caught

Fixing the model / data

• Class weights: penalize misclassifying a real switcher more heavily in the loss function (most libraries support a class_weight="balanced" or equivalent argument) — often the simplest, lowest-risk fix.
• Oversampling the minority (switched) class (random duplication, or synthetic methods like SMOTE that interpolate between similar switchers) to give the model more signal to learn from.
• Undersampling the majority (non-switcher) class — simpler, but throws away data and can hurt if non-switching commuters are genuinely diverse.
• Combine with threshold tuning: even a well-trained model needs its decision threshold moved away from the default 0.5 to reflect the true switch rate and the cost of missing a likely switcher vs. wasting a subsidy offer on someone who wouldn’t have switched anyway.

The order that usually works best

Start with class weights (cheap, no data manipulation, easy to reverse), evaluate with the right metrics, and only reach for oversampling/undersampling or more exotic techniques if class weights alone aren’t sufficient — piling on multiple imbalance-correction techniques at once makes it hard to tell what’s actually helping.

The definitions, quickly

Precision = TP / (TP + FP)   -> "Of the claims I flagged as fraud, how many actually were?"
Recall    = TP / (TP + FN)   -> "Of all the actually fraudulent claims, how many did I catch?"
F1        = 2 * (Precision * Recall) / (Precision + Recall)

Why finance leans toward recall

Missing a fraudulent claim (a false negative) means GreenMile pays out real money for a car trip disguised as a bike trip — direct, repeatable financial loss, since a commuter who gets away with it once will likely try again. A false positive (flagging a legitimate commuter) is a recoverable cost — a manual review or a request for an additional receipt. Given that asymmetry, finance typically wants to accept more false positives to catch more true fraud, i.e., optimize for recall (often subject to a precision floor so the review queue doesn’t drown).

Why the commuter-experience team leans toward precision

Wrongly flagging a legitimate commuter’s reimbursement (a false positive) delays their subsidy payout and can feel like being accused of lying about a receipt they submitted honestly — a trust-eroding experience that can make a commuter quit the program entirely. Missing an occasional fraudulent claim (false negative) is a smaller, more diffuse cost to any one commuter’s experience. So the commuter-experience team is naturally biased toward keeping precision high, even if a bit more fraud slips through.

How to decide in general

1. Ask: what does a false positive cost the business, in dollars, trust, or commuter experience? What does a false negative cost?
2. Whichever error type is more expensive should be minimized more aggressively — meaning you push the classification threshold to favor the corresponding metric.
3. F1 is a reasonable default when both error types matter roughly equally and you want a single number, but it shouldn’t replace an explicit cost-based conversation between finance and the commuter-experience team — GreenMile rarely values a delayed reimbursement and an undetected fraud claim equally.
4. In practice, a workable compromise is often to pick a threshold that hits a target recall (e.g., “catch at least 90% of fraudulent claims”) and then see what precision — and how many legitimate commuters get delayed — that implies, rather than optimizing each team’s metric in isolation.

Two different kinds of error

Bias is the error from a model being too simple to capture the true relationship — a straight (or barely-curved) line trying to fit how co2_kg actually scales with distance_km and mode will systematically miss in predictable ways no matter how much trip data you feed it. Variance is the error from a model being so flexible that it fits the idiosyncrasies (GPS noise, one-off traffic detours) of the specific batch of trips it saw — retrain it on a different month of trips and the predictions swing wildly.

Why training error is the wrong signal

A degree-10 polynomial has enough flexibility to snake through nearly every trip’s co2_kg value in the training sample, driving training error toward zero. That’s not skill — it’s memorization of quirks like a handful of unusually long detours or mis-logged distances. The model has learned the noise, not just the signal, so it will perform much worse on next month’s trips than the training error suggests.

What actually decides the tradeoff

total_error ≈ bias² + variance + irreducible_error

You want the model complexity that minimizes the sum, not either term alone. In practice you find this by comparing performance on a held-out set of trips (or via cross-validation) across a range of complexities and picking the point where validation error is lowest — often well before training error has bottomed out.

What this looks like in practice

• Plotting training vs. validation error against polynomial degree: training error keeps falling, validation error dips then rises — the rising part is variance taking over.
• Regularization, capping the polynomial degree, or simply choosing degree-2 or degree-3 are all ways of deliberately trading a bit of bias for a large reduction in variance.
• With a smaller sample of trips (e.g., a newly onboarded partner city), the variance problem is worse, so simpler models often win by default until more trip history accumulates.

What PCA actually does

PCA finds new axes (principal components) that are linear combinations of the original features, ordered by how much variance in the data they capture. The first component captures the most variance, the second captures the most remaining variance while being orthogonal (uncorrelated) to the first, and so on — so a small number of components can often capture the bulk of the information in a much larger set of correlated commuter-behavior features.

300 correlated commuter-behavior features (trip counts, cost, distance, time-of-day)
        -> PCA
20 components explaining, say, 95% of total variance

What you gain

• Speed: training and inference on 20 features instead of 300 is dramatically faster.
• Reduced overfitting risk: fewer dimensions relative to the number of commuters means less room for the model to fit noise.
• Removes multicollinearity: components are orthogonal by construction, which helps algorithms (like logistic regression) that get unstable coefficient estimates when input features — like weekday trip count and weekend trip count — are highly correlated.
• Noise reduction: lower-variance components often correspond to noise (a handful of odd one-off trips) rather than signal, so dropping them can clean up the data.

What you give up

• Interpretability: component 3 might be 0.4 * weekday_morning_trip_count - 0.3 * avg_bus_cost_usd + ... — not something you can explain to a city partner as “this behavior matters.” If stakeholders need to know which original behavior drove a prediction, PCA gets in the way.
• Some information loss: unless you keep all components, you’re discarding some variance — usually low-variance noise, but not always safely so if a rare-but-important signal (like a small cluster of commuters with an unusual weekend biking pattern) happens to live in a low-variance direction.
• Sensitivity to scale: PCA requires standardizing features first, since it’s variance-driven — an unscaled feature like raw cost_usd (dollars) would dominate the components over a feature like is_rush_hour (0/1) for no meaningful reason.

When it’s the right call

Good fit when features are highly correlated, dimensionality is genuinely a bottleneck, and the downstream use case doesn’t require explaining individual original behaviors. Poor fit when stakeholders need transparent, feature-level explanations for why a commuter was targeted, or when the original features are already few and mostly independent.

The core tradeoff

Simple, interpretable models (logistic regression, single decision trees, rule lists) let you say exactly why a given prediction came out the way it did — “short commute distance and employer subsidy enrollment drove this commuter’s high predicted uptake” — in a way a city official can verify and put in a public report. Complex models (gradient boosting, random forests, neural nets) often perform better because they capture nonlinear interactions, but their reasoning is diffuse across hundreds of trees, making a single prediction harder to explain faithfully.

Why context changes the answer

• Regulated, public-facing, individual-impact decisions (a city-facing grant report, credit, hiring): explainability is often required or at least expected — a 4-point AUC gain rarely outweighs the risk of a report the city can’t defend to its council or constituents.
• Low-stakes, internal-only decisions (e.g., ranking commuters for an internal email A/B test with no public reporting): a performance gain matters more since there’s no external party entitled to an explanation.
• In between (targeting subsidy nudges internally, fraud triage): often the practical answer is a complex model with a post-hoc explanation layer (like SHAP) that approximates why it made a given call, even if it’s not a perfect account of the underlying computation.

What “pairing” looks like in practice

Using SHAP on top of a gradient boosting model can produce a per-commuter breakdown of which features pushed the predicted subsidy uptake up or down — this can inform the narrative in a report even if the headline model driving the public-facing numbers is the simpler logistic regression, and city partners often still prefer a genuinely interpretable model as the primary reported decision-maker with the complex model used for internal targeting only.

Bottom line for an interview answer

Frame it as “performance is one input to a decision, not the whole decision” — the right model depends on who needs to understand the prediction, what it costs GreenMile if the city can’t defend how the program targets people, and whether the reporting requirement forces the choice regardless of the accuracy numbers.

Two distinct failure modes

• Data drift (covariate shift): the distribution of input features changes over time — e.g., a new partner city brings in commuters with a very different distance_km and mode mix than the training data. The model’s learned relationships may no longer apply well to the new population.
• Concept drift: the relationship between inputs and the target itself changes — e.g., gas prices spike or e-bike prices drop, so the actual cost_usd a commuter saves by switching from car to bike shifts even though the commuter mix looks the same.

Both cause the same symptom (predictions get worse) but call for different fixes — drift in inputs might be handled by monitoring and retraining on recent trips, while concept drift (like a real change in the cost of gas or e-bikes) may require refreshing the vehicle_cost_reference table and retraining on the new cost structure.

What should have been monitored from day one

• Input distribution monitoring: track summary statistics (mean, variance, missingness rate, category frequencies) of key features — distance_km, mode mix, cost_usd — over time, and alert when they shift meaningfully from the training distribution.
• Prediction distribution monitoring: track the distribution of the model’s own outputs — a sudden shift (e.g., average predicted savings jumping 30%) is often visible before ground truth is even available.
• Outcome/performance monitoring: once actual cost_usd and co2_kg are logged for a trip, continuously recompute the real evaluation metric (MAE, RMSE) on a rolling window, not just at launch.
• Automated alerting on thresholds for the above, rather than relying on a human to notice during a periodic review.

production pipeline:
  new predictions -> log inputs (distance_km, mode, city) + predicted cost/CO2
  actual trip cost/CO2 logged (lagged) -> compute rolling error metrics
  compare current input/prediction distributions vs. training baseline
  alert if drift or error exceeds threshold -> trigger investigation/retraining

The retraining question

Monitoring tells you that something changed; it doesn’t automatically mean “retrain immediately” — a spike in gas prices might reflect a genuine, lasting shift worth retraining the cost model for, while a one-week anomaly (a local fuel shortage) might just add noise to a retrain. Good monitoring surfaces the signal quickly enough that a human can make that call before it costs a quarter’s worth of bad savings estimates in program reports.

Why this specific feature leaks

If “switched to bike” is defined as “primary_mode is bike for the most recent 90 days,” then pct_trips_bike_90d computed at the time the dataset was built is almost a direct restatement of the label — commuters who already switched will, by construction, have a huge value for that feature, and still-driving commuters a small one. The model isn’t learning to predict an upcoming switch; it’s learning to detect a switch that has already happened, using trip history that wouldn’t exist yet if you were scoring a currently car-commuting member today.

Why offline metrics didn’t catch it

Offline validation (even with a proper train/test split) can still look great under leakage, because the leaked information is present in both the training and test sets consistently — the model looks like it’s generalizing well when it’s actually exploiting a feature that encodes the answer. The leak only becomes visible once the model scores a brand-new city’s commuters at actual prediction time, when that future trip history genuinely doesn’t exist yet.

How to catch leakage before shipping

• Suspiciously high performance relative to the difficulty of the problem is the biggest red flag — if predicting a behavior change as noisy as mode-switching hits 98% AUC, be suspicious before celebrating.
• Check feature importance / SHAP values — if pct_trips_bike_90d dominates overwhelmingly, investigate exactly how it’s computed and whether it could only exist after the switch occurred.
• Time-travel the feature computation: explicitly ask, “would this exact value have been knowable at the moment I need to make this prediction, using only trips logged before that moment?”
• Point-in-time correct feature pipelines: build features off trips using an explicit “as-of” timestamp that matches when the prediction would actually be served, not the time the dataset happens to be pulled.

The general pattern to watch for

Leakage often hides in fields like primary_mode, recent trip counts, or anything derived from a process that only completes after the event you’re trying to predict — always trace a feature back to “when would this value have actually existed” before trusting it.

Why grid search breaks down

Grid search evaluates every combination on a fixed grid — with 6 hyperparameters and even just 5 values each, that’s 5^6 ≈ 15,625 combinations, or roughly 217 days at 20 minutes per run for GreenMile’s fraud-risk model. It also wastes effort: it spends just as much budget exploring clearly bad regions of the space as promising ones, because it doesn’t use information from prior runs.

Random search

Randomly sampling combinations from each hyperparameter’s distribution, for a fixed budget of trials, tends to outperform grid search per unit of compute — because a few hyperparameters usually matter far more than others, and random sampling explores the important ones more densely than a grid does (a grid “wastes” resolution on hyperparameters that barely matter).

Bayesian optimization

Builds a probabilistic model (often a Gaussian process) of how hyperparameters map to validation performance on held-out fraud claims, and uses it to pick the next combination to try — balancing exploring uncertain regions against exploiting regions known to work well. This typically finds strong configurations in far fewer trials than grid or random search, at the cost of some added complexity and sequential (harder to parallelize) evaluation.

grid search:      exhaustive, wasteful, guaranteed coverage
random search:    cheap, surprisingly effective, easy to parallelize
bayesian opt:     sample-efficient, sequential, more setup

The test-set trap

If you tune hyperparameters by repeatedly checking performance on the held-out test set of claims and keeping whatever change improves it, you’re indirectly fitting to the test set through your own decisions — the final reported fraud-detection performance will be optimistic and won’t hold up once the model reviews next month’s real reimbursement claims. The fix is structural: tune exclusively against a validation set or nested cross-validation, and evaluate on the test set exactly once, after all tuning decisions are locked in.

What each curve actually plots

ROC curve:  True Positive Rate (Recall)  vs.  False Positive Rate
PR curve:   Precision                    vs.  Recall

False Positive Rate = FP / (FP + TN) — with a huge number of true negatives (commuters correctly predicted not to switch) in the denominator, FPR stays tiny even when the absolute number of false positives is large enough to swamp a coordinator’s outreach list. ROC-AUC averages performance across all thresholds and is dominated by how well the model separates the (huge) non-switching majority from everything else, which makes it insensitive to how the model does specifically on the (tiny) group who actually switch.

Why precision-recall is more honest here

Precision explicitly uses false positives in its denominator relative to predicted positives, not relative to the whole (mostly non-switching) population — so it directly reflects “of the commuters we’re telling coordinators to prioritize, how many actually switch?” A PR curve that dips sharply as recall increases shows exactly the pattern the program team is describing: to catch more real switchers, precision collapses and the outreach list fills with commuters who were never going to switch.

A concrete illustration

Say there are 10,000 commuters, 30 of whom fully switch this month. A model with a 1% false positive rate produces about 100 false positives — more than 3x the number of actual switchers — even though a 1% FPR sounds tiny and barely moves ROC-AUC. Precision at that operating point could be as low as 30/(30+100) ≈ 23%, which is the number that actually matters to coordinators working through the outreach list.

What to do instead

• Report precision-recall AUC or examine the PR curve directly, especially under class imbalance like GreenMile’s switch-prediction problem.
• Pick an operating threshold based on a target precision (e.g., “keep at least 50% of flagged commuters actually likely to switch”) rather than a threshold chosen from the ROC curve.
• Consider precision@k (e.g., precision among the top 100 highest-likelihood commuters per week) if coordinators have fixed outreach capacity — it maps directly to what they can act on.

Why offline metrics alone aren’t enough

Offline evaluation (AUC, precision, RMSE on cost_usd or co2_kg predictions, etc.) is computed against historical trips and commuters records, which reflect how commuters responded to whoever the old model chose to target — it can’t fully capture how commuters behave when they’re the ones getting nudged based on the new model’s picks, since there’s no historical record of how a commuter who was never previously flagged would react to a bike-switch nudge. This is the counterfactual problem: you’re evaluating a new targeting policy using data generated by a different policy.

There’s also a gap between the offline metric and the actual goal — AUC measures how well the model ranks commuters by predicted switch likelihood, but GreenMile’s partner cities care about a downstream outcome: how many commuters actually reduce car trips and cut CO2, which a ranking metric only approximates.

What a live A/B test adds

• Real commuter behavior in response to who the new model actually targets with subsidy nudges and push notifications, not a backtest against old trip logs.
• Business-relevant metrics — the share of targeted commuters who log a bike trip in trips within the following 30 days, drop in average co2_kg per commuter, subsidy program cost per conversion — rather than a proxy ranking metric.
• Guardrail metrics to catch unintended harm (e.g., notification opt-out rate, complaints from commuters who feel over-messaged, subsidy budget burn rate) that offline AUC wouldn’t surface at all.

1. Offline validation: confirm new model beats old on historical proxy metrics
2. Shadow/canary: score live commuters with the new model without acting on it, compare picks to the old model
3. A/B test: route a subset of commuters/cities to new-model targeting, measure real switch rate + guardrails
4. Gradual rollout: expand to more cities as confidence grows
5. Full rollout + continued monitoring

What to tell the manager

Frame the offline win as a promising filter, not a launch decision: “the 6% AUC gain earns this model a spot in an A/B test — if it also moves actual car-to-bike switches and CO2 savings without spiking opt-outs or subsidy cost, we roll it out to every city; if it doesn’t, we’ve avoided messaging every commuter based on a proxy metric that didn’t hold up with real behavior.” This also protects against the case where the offline gain came from overfitting to quirks of the backtest period.