Before You Ship
Find one way your AI feature could fail and get an acceptance test to run before launch.
Would you ship this?
A fictional employee in Canada asks, “How much parental leave do I get?”
“You’re eligible for 18 weeks of paid parental leave.”
“Employees based in the United Kingdom are eligible for 18 weeks of paid parental leave.”
Review a feature
Example reviews
Email agent
An agent drafts customer replies and sends them once a teammate approves.
- Failure to test
- Approval sticks to the conversation while the draft changes. The agent sends wording the teammate never saw.
- Setup
- Prepare and approve a synthetic customer reply that includes a refund amount.
- Input
- Edit the refund amount, then try to send the revised reply.
- Pass condition
- Delivery is blocked and the revised draft requires fresh approval.
Personal memory
A planning assistant remembers preferences from earlier conversations and uses them to suggest a schedule.
- Failure to test
- An old preference outranks a newer correction. The assistant keeps proposing the time the person already ruled out.
- Setup
- Save “I prefer Monday mornings,” then “Mondays no longer work, use Thursday.”
- Input
- Start a fresh conversation and ask for a meeting time.
- Pass condition
- The assistant suggests Thursday or asks for clarification if it cannot retrieve the newer preference.