AI Engineer project
Build an eval harness to compare two prompt versions
A way to prove a prompt change actually improved things, instead of shipping on vibes — the actual daily question at any real AI product team.
The brief
Build a small evaluation harness for a chosen LLM task: a set of test cases (input plus expected behavior, not an exact string match, since the model's output is probabilistic), an automated scoring mechanism (rule-based checks and/or a second LLM-as-judge call), and a report comparing two versions of the prompt on the same test set.
Suggested stack
What you hand in
- A public GitHub repository with the harness, test cases, and a README
- A comparison report (in the repo) showing scores for both prompt versions on the same test set
- The README documents how the scoring mechanism works and its known limitations
Grading happens against the rubric below, so read it before you start — not after.
How this is graded
Published in advance and weighted out of 100. Nothing here is a surprise.
Test cases are chosen to probe real failure modes of the task (edge cases, ambiguous inputs, adversarial phrasing), not just easy happy-path examples.
Scoring reflects what actually matters for the task (correctness, format compliance, tone) rather than a brittle exact-match check on a non-deterministic output.
The report makes it clear which prompt version performed better, on which specific cases, and by how much.
Running the harness again on the same inputs produces stable, consistent scores, not wildly different results run to run.
No console errors or crashes, no broken layout, no leftover placeholder text or commented-out code.
Why this project is worth your weekend
- "How do you know your prompt change didn't make things worse" is the actual daily question at any real AI product team.
- Most junior AI engineers ship prompt changes with zero regression testing, so having a working harness is a genuine differentiator.
- This is the difference between vibes-based prompt iteration and an engineering process a team could actually rely on.
Where people lose points
- Test cases that only check for an exact string match against a probabilistic model's output, failing on trivially equivalent rewordings.
- No held-out cases — the prompt was tuned directly against the same cases it's now being scored on, inflating the result.
- Using an LLM-as-judge with no consistency check, so the "score" it produces is itself unreliable from run to run.
Other AI Engineer projects
Two or three of these turn an empty resume into a portfolio.
Built it? Get it scored against this rubric.
Submit your work and get a score on every criterion above, written feedback, and three resume bullets you can use straight away.