Quest 10 · Developers
Night at the Guild
Launch is where it really starts — evaluation · cost · operations
- Time
- 7 min
- Max XP
- 100
- Skill earned
- Operations · cost · evaluation
1 / 60 XP
A month later
Hanna
The library service from request 8 is used a lot. But we need a meeting tonight. There are two problems.
Hanna
One, the lantern workshop has released a new model. They say it is smarter. Two, this month's lantern bill is three times last month's.
You
I thought it was finished once it launched.
Ari
Actually, that is where it starts. A service that uses me keeps changing: its users, its documents, its model.
Summary
What this request teaches
Is an AI service done once it launches? Through two problems in the guild's library service after a month of running, learn how to evaluate before changing, how to find where cost leaks, and what to record while running.
Show the key points (best read after finishing the request)
- An AI service keeps changing after launch: its users, documents, and model. Operations are where it really starts.
- Build an evaluation set from real questions and expected answers, and compare before and after with the same set before changing a model or prompt.
- Open big changes to some users first and widen them if nothing goes wrong (a canary release).
- Record the tokens, response time, errors, and model used for each request. The causes of cost and problems are found in the records.
- Cost leaks easily through long conversation history, too many search pieces, and models bigger than needed. After cutting, check quality again.
Try it yourself
If you have a service or task that uses AI, make your first evaluation set with the request below. The more real user questions you mix in, the better.
My service is [an AI service that does what]. Make 20 questions to evaluate this service. - Split them into 10 common questions, 5 tricky ones, and 5 it must not answer. - For each question, write one line on what a good answer must have.