Statistics

What’s happening statistically

Any individual commuter’s monthly cost_usd total is a mix of their true underlying commuting pattern (distance, typical parking situation, usual number of trips) plus random noise (an unusual number of toll trips, a month with more downtown parking, a gas price spike). If you specifically select the commuters who had the highest cost last month, you’re disproportionately selecting commuters who got an unlucky cost month, not just commuters who are genuinely expensive drivers. Next month, that bad luck isn’t likely to repeat, so their cost naturally drifts back down toward their true average — this happens whether or not any outreach or subsidy nudge existed.

Why this trips people up

It’s tempting to interpret “we nudged them, then their cost dropped” as evidence the outreach worked, but this is exactly the pattern regression to the mean produces on its own, with zero causal contribution from the outreach. The bias is strongest whenever you select a group specifically because they were extreme (highest cost) in one period — that selection itself guarantees some reversion toward average next period.

How to actually test whether the outreach worked

• Use a control group. Take a comparable set of high-cost commuters last month who don’t get the outreach, and compare their month-over-month cost change to the outreach group’s. If the outreach group’s cost drops meaningfully more than the untargeted high-cost group’s, that’s better evidence the outreach itself is adding value beyond regression to the mean.
• Look at the full distribution, not just the selected extremes. If commuters across the whole cost spectrum (not just last month’s most expensive) also saw costs drop similarly — e.g., because gas prices fell city-wide — that’s a further hint something general changed, rather than the outreach specifically helping the targeted group.
• Be skeptical of any before/after comparison on a group chosen for being extreme — this is one of the most common false-positive traps in causal reasoning, and it applies just as much to “our highest-cost commuters improved” as it does to sports “slump” stories or clinical trial baselines.

The precise definition

A p-value is: the probability of observing data at least as extreme as what you observed, assuming the null hypothesis is true. It is a statement about the data given the null hypothesis, not a statement about the probability the null (or alternative) hypothesis is true given the data — that distinction is the single most common p-value misinterpretation, and it’s a real logical difference, not just pedantry.

Why “97% chance biking is cheaper” is wrong

The sponsor’s framing (1 minus the p-value equals the probability biking is really cheaper) reverses the conditional — it treats the p-value as if it directly answered “what’s the probability my hypothesis is true,” which frequentist p-values simply do not compute. To make a genuine probability statement about which mode is more likely to be cheaper, you’d need a Bayesian framework with a prior, which most standard experiment analysis tools don’t provide.

What a p-value also doesn’t tell you

• Effect size. A p-value of 0.001 with a $0.10 average cost difference and a p-value of 0.04 with a $40 average cost difference can both be “significant,” but only one is meaningful for the “biking saves you real money” pitch. Always look at the effect size (in dollars, per the vehicle_cost_reference benchmarks) and confidence interval alongside the p-value.
• Practical importance. With a large enough sample of trips, even a trivially small, business-irrelevant cost difference can produce a tiny p-value. Statistical significance is not the same as “this matters to a commuter deciding how to get to work.”
• Whether the comparison was well-designed. A p-value from an underpowered sample, a comparison with multiple untracked confounds (e.g., bike commuters living closer to work), or a metric that was peeked at repeatedly, is not trustworthy regardless of its numeric value.

The more accurate thing to tell the sponsor: “if car and bike commuting truly cost the same on average, we’d see a gap this large only 3% of the time by chance — that’s suggestive evidence, and combined with the actual dollar effect size and confidence interval, here’s what I’d recommend telling employees.”

The general definitions

In hypothesis testing, the null hypothesis here is “the e-bike subsidy has no effect on bike-mode share.”

• Type I error (false positive, α): rejecting the null when it’s actually true — concluding the subsidy increased bike-mode share when in reality the pilot’s result was just noise.
• Type II error (false negative, β): failing to reject the null when it’s actually false — concluding the subsidy had no effect when it truly would have increased biking if rolled out.

Why the significance threshold and sample size control this tradeoff

The pilot’s significance threshold (α) and sample size jointly determine how these two error rates trade off:

• A looser significance threshold (higher α) → more likely to declare the subsidy effective → catches more true effects (fewer Type II errors) but also greenlights more false wins (more Type I errors), risking a costly city-wide rollout of a program that doesn’t actually work.
• A stricter significance threshold (lower α), or an underpowered pilot with too few enrolled commuters → fewer false “it works” conclusions (fewer Type I errors) but a higher chance of missing a real, meaningful increase in bike-mode share (more Type II errors), risking shelving a genuinely good program.

Framing it for leadership

The business-relevant framing isn’t “which error is worse in the abstract,” it’s “what does each error cost GreenMile and the city here specifically”: a Type I error costs the money and operational effort of rolling out subsidies, e-bike fleets, and marketing city-wide for a program that doesn’t move the needle, while a Type II error costs the missed emissions reductions, health benefits, and program revenue GreenMile would have captured by rolling out something that actually works. The right threshold and pilot sample size aren’t a purely statistical question — they depend on the relative dollar and reputational cost of each error type, which is a business judgment call informed by, but not fully determined by, the statistics.

The confound hiding in plain sight

“Cities with more bike lanes” is not a random assignment of infrastructure — cities that build extensive bike networks tend to differ systematically from cities that don’t. They may have higher municipal budgets, denser and more walkable urban design overall, more progressive health and transit policy generally, or residents who self-select into those cities partly because they’re already more health- and environment-conscious. Any of those traits could independently drive lower BMI and health costs regardless of whether the bike lanes themselves add any value. This unmeasured shared cause is called a confounding variable.

Why this matters for the recommendation

If those city-level traits are the real driver, another city adding bike lanes without the rest of that surrounding context (walkability, culture, budget for health programs) may not replicate the observed 40% gap at all, because you’d be changing one input (bike infrastructure) while holding constant a very different city. The correlation was never evidence that bike lanes cause lower BMI and health costs; it’s equally consistent with “cities that already prioritize health and active living both build bike lanes and see better health outcomes, for reasons only partly related to the lanes.”

How to actually test causality

• Look for natural experiments. A city that adds a large network of bike lanes at a specific point in time, compared to similar cities that didn’t, lets you check whether BMI/health costs shifted after the change relative to the comparison cities — a difference-in-differences approach.
• If a true randomized rollout isn’t feasible (you can’t randomly assign bike lanes to city blocks), quasi-experimental methods like matching cities on observable traits (density, income, baseline health spending) or regression discontinuity around funding-eligibility cutoffs can partially control for confounding, though they’re weaker evidence than randomization.
• In the meantime, be explicit with the city-partnerships lead that the 40% gap is a correlation worth investigating further via GreenMile’s own commuter-level trips and co2_kg/cost data, not yet evidence for a “build more bike lanes and health costs will drop” pitch on its own.

The technically correct interpretation

In the frequentist framework, the true average co2_kg saved per commuter across the whole population of car-to-bike switchers is treated as a fixed, unknown number — not a random variable. So it doesn’t make sense to say “there’s a 95% chance the true value is in this range,” because the true value either is or isn’t in the range; there’s no probability left once the interval is computed from trips data. What “95% confidence” actually describes is the long-run behavior of the procedure: if GreenMile pulled a new sample of commuters and rebuilt this confidence interval over and over, about 95% of those intervals would contain the true CO2 savings.

Why this distinction still matters in practice

For decision-making purposes, most practitioners treat “95% CI [1.8 kg, 4.2 kg]” as reasonably close to “we’re fairly confident the true CO2 savings per switched commuter falls somewhere in this range,” and that informal shorthand is usually harmless for business communication with a city partner. But it’s worth knowing the precise definition for two reasons:

• If the partner asks a follow-up question that depends on the exact probabilistic meaning (e.g., “what’s the probability the true savings is above 4 kg?”), the frequentist CI genuinely cannot answer that — you’d need a Bayesian credible interval, which does make direct probability statements about the parameter, but requires specifying a prior.
• It guards against the common mistake of treating overlapping confidence intervals from two cities’ co2_kg estimates as definitive proof of “no difference between cities,” when the correct comparison is a direct test (or CI) on the difference between the cities, not eyeballing whether their individual intervals overlap.

Practical guidance

For most stakeholder conversations, it’s fine to say “we’re 95% confident the true CO2 savings per commuter is between 1.8 kg and 4.2 kg” as an approximation, while keeping the precise definition in your back pocket. The more useful conversation with the city partner is usually about what the interval implies for the decision: does it exclude zero (statistically distinguishable from no savings), and does the range include only savings sizes large enough to matter for the city’s emissions targets?

Why “run it for a month” is the wrong starting point

Deciding pilot duration first and hoping for significance backwards is how programs end up either wasting a month on a pilot that was never going to detect a realistic emissions difference (underpowered), or stopping early the moment a random fluctuation in co2_kg looks significant (inflating false positives). The right order is: decide what emissions difference matters, calculate the sample size needed to reliably detect it, then translate that into a duration based on expected commuter enrollment.

The four inputs to a power analysis

• Baseline variability — the standard deviation of per-trip or per-commuter co2_kg in the current data, since noisier emissions data requires a larger sample to detect the same difference.
• Minimum detectable effect (MDE) — the smallest emissions difference you’d actually want to act on (e.g., a 1.5 kg CO2 reduction per commute, or a 20% relative reduction). Smaller MDEs require dramatically larger samples to detect reliably.
• Significance level (α) — usually 0.05, the false-positive rate you’re willing to accept.
• Power (1 − β) — usually 0.80, the probability of detecting the emissions difference if it’s really there (i.e., avoiding a false negative).

Rough intuition, not exact formula

For a two-group comparison of a continuous metric like co2_kg, required sample size per group grows roughly with:

n ≈ 16 * σ² / (MDE)^2

(a commonly used rule-of-thumb approximation for α=0.05, power=0.80, where σ is the standard deviation of co2_kg). The key takeaway even without memorizing the formula: sample size requirements grow quadratically as the MDE shrinks — reliably detecting a 1 kg CO2 difference can require 4x the sample of detecting a 2 kg difference, not 2x.

Turning this into a conversation with the sponsor

Once you know the required number of enrolled commuters, divide by the expected weekly signup rate to get an actual required pilot duration — which might turn out to be longer than a month for a small employer site, or might reveal that a month is actually plenty for a large campus rollout. Either way, the number is now grounded in a real detectability requirement rather than an arbitrary calendar guess, and it also sets a clear “stopping rule” that helps resist the temptation to end the pilot early based on a promising early trend.

Bike-mode share: a proportion test

Bike-mode share is a binary outcome per trip (was mode = 'bike' or not) aggregated into a rate. Its variance is a known function of the rate itself — p(1-p)/n — which is exactly what a two-proportion z-test (or equivalently, a chi-square test of independence for a 2x2 table of subsidy-enrolled vs. control by bike-vs-other-mode) is built around:

z = (p1 - p2) / sqrt( p_pooled(1 - p_pooled) * (1/n1 + 1/n2) )

This works cleanly because the underlying distribution of a proportion is well understood and doesn’t require checking normality assumptions the way a t-test does.

Average trip cost: a continuous metric

Average cost_usd per trip is typically right-skewed (most commutes cost a routine amount, but a handful of trips get hit with tolls, surge fares, or downtown parking-garage fees). A standard two-sample t-test compares mean cost_usd between the subsidy-enrolled group and control, relying on the Central Limit Theorem to make the sampling distribution of the mean approximately normal even if individual trip costs aren’t — which usually holds fine with large enough sample sizes, but can break down if a few extreme parking/toll outliers dominate the mean.

Why you can’t just run one type of test for everything

Treating bike-mode share like a continuous metric (running a t-test on a column of 0s and 1s for “is this trip a bike trip”) technically often gives similar results to a proportion test at large sample sizes, but it’s not the standard or most defensible approach, and it obscures the fact that variance in a proportion is determined by the rate itself, not estimated independently. Conversely, treating average trip cost as a simple proportion doesn’t make sense at all — it’s not a binary outcome.

Handling the skew in trip cost

If outlier trips (tolls, parking garages) are severe enough to distort the mean comparison, consider:

• Log-transforming cost_usd before running the t-test (changes the interpretation to comparing geometric means).
• Using a non-parametric test like Mann-Whitney U, which compares distributions via ranks rather than raw values.
• Bootstrapping a confidence interval directly on the difference in mean cost_usd, which doesn’t require a normality assumption at all.

Running the right test for each metric type — rather than defaulting to one test for everything — is what makes the eventual read on the subsidy program defensible when it’s presented to the cities and employers footing the bill.

Matching the test to the data type

Metric	Data type	Test
Average trip cost (cost_usd)	Continuous, roughly symmetric	Two-sample t-test
Mode choice (mode: car/bike/bus/walk)	Categorical (multi-class)	Chi-square test of independence
Trip distance (distance_km)	Continuous, heavily skewed with outliers	Mann-Whitney U (or log-transformed t-test, or bootstrap)

Why the t-test needs (approximate) normality

A t-test compares means and relies on the sampling distribution of the mean being approximately normal, which the Central Limit Theorem guarantees as sample size grows — even for moderately skewed data. So average cost_usd per commute across a large, mostly-routine sample of trips is usually a reasonable fit for a t-test, as long as toll/parking outliers aren’t severe enough to dominate the mean.

Why chi-square fits categorical mode data

mode isn’t a rate or a continuous number — it’s a category with more than two possible values (car, bike, bus, walk). A chi-square test of independence checks whether the distribution of modes differs between two groups (e.g., subsidy-enrolled vs. control) by comparing observed counts in each mode-by-group cell against the counts you’d expect if mode and group assignment were independent. This is the standard choice whenever you’re comparing category proportions rather than a single binary rate or a continuous mean.

Why Mann-Whitney for skewed distance data

distance_km is often heavily right-skewed — most commutes cluster in a normal range, but a handful of commuters have unusually long trips (or occasional long detours) that inflate the mean and destabilize a t-test’s assumptions. Mann-Whitney U is a non-parametric test — it compares the ranks of observations between two groups rather than their raw values, which makes it robust to outliers and doesn’t require any assumption about the underlying distribution’s shape. The tradeoff is that it tests whether one group’s distances tend to be larger than the other’s (stochastic dominance), which is a slightly different question than “are the mean distances different” — worth being precise about when reporting results to a city partner who might assume any significance test is answering “are the averages different.”

The general principle

Choosing a statistical test should start from “what is the shape and type of this data” (continuous vs categorical, symmetric vs skewed), not from habit or whichever test is easiest to run in your tool of choice — using the wrong test can either miss real effects (reduced power) or produce misleading significance (violated assumptions).

Why the toll/parking outliers are hurting the test

A t-test’s power depends heavily on the variance of the metric — the noisier the data, the harder it is to distinguish a real effect from random fluctuation. A handful of extreme trips (a $40 toll bridge crossing, a $35 downtown parking-garage day) inflate the sample variance of cost_usd enormously, which widens the confidence interval around the mean and makes it much harder to reach statistical significance even when a real, meaningful cost difference exists in the bulk of everyday commute trips.

Remedies, and their tradeoffs

• Log-transform the metric (e.g., log(cost_usd + 1)) before running the t-test. This compresses the influence of extreme values and often makes the distribution closer to normal, but changes the hypothesis being tested — you’re now comparing geometric mean cost, not arithmetic mean cost, which is a subtly different (and sometimes harder to explain) business question.
• Winsorize or cap extreme values (e.g., cap cost_usd at the 99th percentile) before averaging. This reduces the influence of outlier toll/parking trips while keeping the metric in its original units, but it also means you’re intentionally under-crediting the true cost burden that occasional toll/parking days impose on car commuters — a real limitation if those spikes are exactly what you’re trying to show commuters they’d avoid by biking.
• Use a non-parametric test (Mann-Whitney U), which compares distributions via ranks and is robust to outliers, at the cost of testing a different question (stochastic dominance rather than difference in means).
• Bootstrap a confidence interval for the difference in means directly, resampling the raw trips data many times to build an empirical distribution of the mean cost difference — this doesn’t require a normality assumption and preserves the original dollar units and business meaning, though it’s more compute-intensive.

What to actually recommend

If the toll/parking-garage trips are a real and recurring part of the car-commuting experience (not data errors), capping or ignoring them can hide the very cost burden you want to highlight to employers — a bootstrap or a stratified analysis (analyzing toll/parking-heavy routes separately from typical local trips) is often more honest than a transformation that mutes their contribution. If the goal is a general “typical trip cost” comparison across all commuters, a log-transform or Mann-Whitney is usually a reasonable, defensible choice.

A concrete numeric example

Suppose:

	Lower-income bracket (subsidized, more bike trips)	Higher-income bracket (fewer bike trips)	Overall
Car	200 trips, $0.35/km	800 trips, $0.30/km	1,000 trips, $0.31/km
Bike	800 trips, $0.10/km	200 trips, $0.28/km	1,000 trips, $0.14/km

Bike beats car within the lower-income bracket ($0.10 vs. $0.35/km) and within the higher-income bracket ($0.28 vs. $0.30/km) — yet in this constructed setup, if the mix were skewed the other way (most bike trips coming from the cheaper-to-drive higher-income bracket, and most car trips concentrated in the more expensive-to-drive lower-income bracket, perhaps because that’s where parking is scarcest), the blended car average could come out lower than the blended bike average even though bike wins in every bracket. The reversal happens because the subsidy program enrolled a very different income mix of bike riders than car drivers, so the overall average isn’t comparing like with like.

Why this happens

The aggregate cost-per-km is a trip-weighted average of the bracket-level rates. Because the e-bike subsidy was rolled out unevenly across income brackets, the mix of trips feeding into “bike” and “car” differ by bracket in ways that don’t cancel out — whichever mode happens to have more of its trips concentrated in an inherently cheaper-per-km bracket can look better (or worse) overall even while losing (or winning) in every individual bracket. The mix, not the mode itself, is driving the aggregate difference.

What to check

• Subsidy allocation balance. Confirm the subsidy (and resulting bike trips) wasn’t concentrated in a way that skews the income-bracket mix differently for bike vs. car — an uneven rollout is often the root cause.
• Report by income bracket, not just in aggregate, whenever enrollment or trip mix could plausibly differ between modes across brackets.
• When brackets disagree with the aggregate, trust the bracket-level results for making the “is biking cheaper” claim to leadership, and treat the aggregate number as, at best, a mix-confounded summary rather than the “real” answer.

The lesson for the team

Never trust a single city-wide cost-per-km number without checking whether the underlying population (income bracket, distance, subsidy enrollment) was actually comparable between car and bike trips — this is one of the more common ways a mobility dashboard silently misleads a team into overstating (or understating) the case for biking.

The multiple comparisons problem

Each individual city-level test run at α = 0.05 has a 5% chance of a false positive if the subsidy truly has no effect there. But run 12 independent city-level tests, and the probability that at least one comes back “significant” by pure chance climbs to roughly 1 - (0.95)^12 ≈ 46% — meaning it’s more likely than not that at least one city shows a spurious “win” on bike-mode share even if the subsidy program isn’t actually doing anything anywhere. Layering on multiple metrics per city (mode share, cost, CO2) multiplies the problem further.

Fixes include:

• Bonferroni correction — divide α by the number of comparisons (0.05 / 12 ≈ 0.0042), a conservative but simple adjustment.
• False Discovery Rate control (e.g., Benjamini-Hochberg) — less conservative than Bonferroni, controls the expected proportion of false positives among significant results across cities.
• Best practice: designate one primary metric (e.g., overall bike-mode share pooled across all 12 cities) before the test starts, and treat individual city and secondary-metric breakdowns as exploratory, interpreted with appropriately more skepticism.

The peeking problem

Even looking at a single city’s bike-mode share, checking for significance every day during a fixed-duration test inflates the effective false-positive rate, because you’re implicitly running many sequential tests — one on each day you check — and stopping the moment any of them crosses the threshold. This is sometimes called “optional stopping,” and it can push the true false-positive rate well above the nominal 5%, even if you only look at one city and one metric.

Fixes include:

• Pre-commit to a fixed enrollment size or duration determined by a power analysis, and only make the rollout/no-rollout call at that predetermined point.
• Sequential testing methods (e.g., alpha-spending functions, always-valid p-values from sequential testing frameworks) that are specifically designed to allow continuous monitoring of the dashboard without inflating the false-positive rate — this is the statistically correct way to support “checking daily” if city partners genuinely need early visibility.

What to tell the team

Excitement over City #7 on day 6 should be treated as a hypothesis to watch, not a result to act on — testing 12 cities across multiple metrics and checking early makes a false positive here quite likely. The disciplined path is to keep the pilot running to its pre-determined endpoint and evaluate the pre-specified primary metric, pooled across all cities, first.

What’s happening

A novelty effect occurs when a change draws extra attention or curiosity purely because it’s unfamiliar, temporarily inflating a metric — here, bike-mode share among subsidy-enrolled commuters — regardless of whether the change represents a genuine, lasting shift in commuting behavior. As commuters acclimate to the e-bike subsidy, that curiosity-driven boost fades, and mode share settles toward its true steady-state effect, which might be smaller, nonexistent, or (less commonly discussed) even negative once the novelty wears off and the friction of biking (weather, distance, effort) reasserts itself.

The opposite pattern also exists: a change aversion effect, where a new program initially depresses adoption because commuters are unfamiliar with enrollment or skeptical of e-bikes, before recovering as they adapt. Both patterns share the same root cause — the change itself, not the change’s long-run merit, is driving the short-term signal.

Why reporting week-2 results would be a mistake

If the team reports the subsidy’s impact to city partners based on the week-2 spike alone, they’re making a claim based on a temporary artifact of novelty rather than the program’s true long-run impact on mode share — which, per the question, has mostly faded by week six. Recommending a city-wide rollout (or a bigger subsidy budget) on the basis of early results alone is one of the more common ways experimentation programs mislead themselves.

What to do differently

• Run the pilot long enough to reach a steady state, not just long enough to hit a sample-size threshold — for a subsidy plausibly subject to novelty, that often means several weeks beyond the point where bike-mode share first appears to move, not just once significance is technically reached.
• Segment new signups vs. existing commuters. Commuters who join the program well after launch have no “launch excitement” to be novelty-struck by, so their mode-share response is a cleaner read on the subsidy’s true effect; early adopters are the ones most susceptible to novelty dynamics.
• Plot bike-mode share over time within the pilot rather than only looking at a single cumulative number at the end — a declining trend line is itself diagnostic of a novelty effect and worth flagging to the employer sponsor even before the pilot formally ends.
• Be skeptical of unusually large early lifts, especially right after a highly visible launch event or free e-bike demo day, which are classic settings for novelty effects.

The core problem CUPED solves

Average trip cost per commuter varies enormously from person to person for reasons that have nothing to do with whether they were randomly assigned the e-bike subsidy — some commuters live far from work and drive long distances, some already bike, some pay steep downtown parking regardless of mode. That commuter-to-commuter variance shows up as noise in the experiment, widening confidence intervals and requiring larger samples (or longer run times) to detect a real subsidy effect against that noisy backdrop.

The intuition

If a commuter had high average cost_usd before the subsidy program even started, they’ll probably also tend to have relatively higher cost during the experiment — regardless of which group (subsidy or control) they got. CUPED uses this pre-period cost_usd as a covariate to statistically “subtract out” the portion of during-experiment variance that’s predictable from pre-experiment behavior, leaving a cleaner, lower-variance signal that isolates the actual subsidy effect more precisely.

The (simplified) formula

adjusted_cost_i = cost_usd_i - θ * (pre_period_cost_usd_i - mean(pre_period_cost_usd))

θ is typically chosen (via a simple regression) to minimize the variance of the adjusted metric — conceptually similar to a regression coefficient. Because θ * (pre_period_cost_usd_i - mean(pre_period_cost_usd)) has an expected value of zero across commuters (it’s a mean-centered covariate), subtracting it doesn’t change the expected value of the metric — meaning the estimate of the subsidy effect stays unbiased — it only reduces the variance around that estimate.

Why this matters practically

Lower variance directly translates to either:

• Detecting the same effect size with a smaller sample (shorter test duration for the same number of enrolled commuters), or
• Detecting a smaller effect size with the same sample that the un-adjusted analysis wouldn’t have had the power to find.

The catch

CUPED requires a pre-experiment period where you can measure each commuter’s cost_usd (or a correlated metric) before the subsidy test starts — it doesn’t help for brand-new commuters who signed up with no trips history, and the variance reduction is only as good as how correlated the pre-period cost is with the during-experiment cost. A commuter whose pre-period cost is only weakly correlated with their during-experiment cost (e.g., someone who just moved cities) won’t meaningfully shrink variance.

Absence of evidence is not evidence of absence

Failing to reach statistical significance only tells you that the observed difference in bike-mode share between the subsidy and control groups isn’t extreme enough to confidently rule out the null hypothesis at your chosen threshold — it does not tell you the null hypothesis is true. There are two very different scenarios that both produce “not significant”:

1. The subsidy genuinely doesn’t shift mode share (or shifts it too little to matter), and the data correctly reflects that.
2. The subsidy has a real effect but the test was underpowered to detect it — too few enrolled commuters, too much noise in individual mode choice, or too small an effect size relative to what the test could reliably distinguish from chance.

A single p-value of 0.34 can’t tell you which of these is true on its own.

What to actually check

• Look at the confidence interval, not just the p-value. A tight CI on the mode-share difference centered near zero (e.g., [-0.5pp, +0.8pp]) is much stronger evidence of a genuinely small/no effect than a wide CI (e.g., [-6pp, +9pp]) that simply couldn’t pin down the effect either way — the wide CI is consistent with a real, meaningful shift in bike-mode share that the test just didn’t have enough enrolled commuters to detect.
• Check whether the test was adequately powered for the minimum mode-share shift that would actually matter to the employer (e.g., a 3-percentage-point increase), ideally via a power analysis done before the test (a post-hoc power calculation using the observed effect size is generally discouraged, since it’s mathematically just a rescaling of the p-value and doesn’t add new information).
• Consider whether enrollment and test duration matched what was planned. If fewer commuters signed up for the subsidy than expected, or the test window was cut short, it may simply not have reached the power it needed.

How to frame this for the sponsor

“Not significant” should be reported as “we don’t have strong evidence the subsidy shifted mode share, and here’s how precise our estimate is” rather than “the subsidy does nothing.” If the confidence interval is wide, the honest recommendation is often to extend enrollment or run a longer pilot before concluding the program doesn’t work, rather than treating an inconclusive result as a definitive negative.

Why bike commuters aren’t a random sample

Choosing to bike to work is itself a behavior that correlates with the very things you’re trying to measure. A commuter who bikes is plausibly already younger, more physically fit, living in a denser neighborhood closer to work, or simply more inclined toward health- and environment-conscious choices in general — meaning the primary_mode = 'bike' commuters you observe systematically differ from car commuters on traits unrelated to the act of biking itself. This is a form of self-selection bias.

Why the conclusion doesn’t follow

“Biking causes better outcomes” treats the observed primary_mode groups as if they were randomly assigned, when in reality who chooses to bike is entangled with age, fitness, income, and commute distance. The true causal effect of biking (versus driving, holding everything else constant) is likely smaller than the raw gap between the two groups in the commuters and trips tables — the data is comparing “healthier, closer-to-work people who bike” against “the general car-commuting population,” not measuring the effect of biking in isolation.

Ways to check and mitigate this

• Compare observable traits between groups — if bike commuters skew younger, have shorter average distance_km, or are concentrated in denser home_city areas, that’s a signal these groups aren’t comparable to begin with.
• Look at employer_subsidy_enrolled and signup timing. Commuters who enrolled in the subsidy and switched from car to bike are a much better comparison group than commuters who were already biking, since a within-person before/after comparison controls for their own baseline traits.
• Use matching or regression adjustment — match car and bike commuters on distance, home city, and other observables, or include them as covariates in a regression, to isolate the mode effect from the confounding traits.
• Prefer the A/B test data over the observational primary_mode split whenever available — randomized subsidy enrollment breaks the self-selection link that plagues the raw observational comparison.

The general lesson: any time a “treatment” (like biking) is something people choose for themselves rather than something randomly assigned, always ask what kind of person tends to make that choice, and whether that’s correlated with the outcome you’re studying.