Development note 01
A real citation isn’t always enough.
In a set of synthetic test materials, the retry delay was 2 seconds in the test environment and 5 seconds in production. The model returned the right numbers, but its citations only described the environments. They did not include the numbers needed to verify the answer.
- Test environment
- 2 seconds
- Production environment
- 5 seconds
Environment descriptions.
No retry-delay numbers.
My original check could establish that a quotation appeared in the source. It could not establish that the quotation supported the claim being made.
I kept the verbatim quotation check and added citation-coverage checks for some time quantities and percentages, with one targeted correction attempt. I also tried a more complicated prompt structure. It made some examples worse, so I reverted it.
Development note 02
The source was gone. The answer was still on its way.
I reproduced a timing problem in a test: a user starts a question, then moves its source material to the recycle bin while the model is still generating an answer. By the time the answer returns, the material is no longer available for that question.
- 01Question starts
Record the material version.
- 02Source is recycled
The model is still generating.
- 03Check before saving
Recheck status, version and topic membership.
- 04Reject the late result
No late draft is saved.
I record the material’s version when the request begins. Before saving the result, I check its status, version and topic membership again. If any of those have changed, I reject the late result.
The model can take time to respond, so I avoid holding a database transaction open throughout inference.
A few other choices
What I’m keeping separate.
One source, several courses
Materials exist independently of courses. The week and purpose sit on the course–material link, so the same source can be reused in different contexts.
Keep sources and generated content separate
AI-generated content does not automatically become evidence for later questions. I want to avoid treating a previous answer as if it were an original source.
Some features can wait
I have deferred automatic question generation and marking to avoid reusing incorrect content. The immediate focus is the material and evidence behind an answer.