Data Scientist project
Analyze an A/B test and give a ship or no-ship recommendation
A real experiment-analysis writeup that checks test validity before trusting the p-value — the actual daily job of a product data scientist.
The brief
Using a public A/B test dataset, or a synthetic one you generate yourself with a documented, realistic data-generating process, run a proper experiment analysis: check for sample ratio mismatch between control and treatment, compute the statistically appropriate significance test for the metric type, report a confidence interval and effect size (not just a p-value), and give a clear ship/no-ship recommendation with stated caveats.
Suggested stack
What you hand in
- A public GitHub repository with the notebook and a README
- The README states the data source (or the documented generating process, if synthetic) and how to reproduce the analysis
- A short section stating the final recommendation and the caveats that would change it
Grading happens against the rubric below, so read it before you start — not after.
How this is graded
Published in advance and weighted out of 100. Nothing here is a surprise.
The test used matches the metric type (e.g. a proportions test for a conversion rate, not a t-test misapplied to a binary outcome).
Sample ratio mismatch between control and treatment is checked explicitly before trusting the result, and any other obvious validity threat (e.g. novelty effects, if relevant) is at least addressed.
The final recommendation states a clear ship/no-ship call along with the effect size and confidence interval, not just a p-value.
A reader can follow the analysis from raw data to recommendation, with markdown explaining each decision.
No console errors or crashes, no broken layout, no leftover placeholder text or commented-out code.
Why this project is worth your weekend
- Nearly every product data scientist's actual job is reading and acting on experiment results, not building models from scratch.
- Reporting a p-value without checking for sample ratio mismatch is a classic mistake that gets a genuinely broken experiment shipped.
- "Should we ship this" is the real question a stakeholder asks, and answering with only a statistic (no recommendation) is a common junior failure.
Where people lose points
- Reporting statistical significance without checking for sample ratio mismatch between control and treatment.
- Treating a p-value just under 0.05 as proof of an effect, instead of reporting the confidence interval and effect size.
- Giving a bare yes/no recommendation with no mention of practical significance or the cost of being wrong.
Other Data Scientist projects
Two or three of these turn an empty resume into a portfolio.
Built it? Get it scored against this rubric.
Submit your work and get a score on every criterion above, written feedback, and three resume bullets you can use straight away.