Measuring Answer Quality Without Labels
How do you tell whether a retrieval-based assistant is right, when nobody has labelled the correct answers?
Background
Every campus assistant project runs into the same wall: it demos well and nobody can say whether it is accurate.
Standard retrieval benchmarks do not transfer to a small, local, constantly changing corpus.
This question sits underneath several BuildHub projects, which makes it worth answering once, properly.
Literature
Reading themes rather than a citation list — the bibliography is maintained by the group, not published here.
Retrieval evaluation metrics
What precision and recall mean when relevance is subjective.
Model-graded evaluation
Using a model as a judge, and how badly that can fail.
Calibration and abstention
Knowing when a system should decline to answer.
Exploration
Build a small, hand-labelled question set as ground truth and treat it as the project’s real asset.
Compare human grading against model grading on the same answers and quantify the disagreement.
Test whether abstention can be tuned without collapsing usefulness.
Experiments
- Hand-labelled question setrunningSlow, unglamorous, and the foundation of everything else.
- Human vs. model grading agreementplannedMeasured on identical answers.
Analysis
There is nothing to analyse yet. Analysis appears here once there are experimental results worth arguing about — and not before.
Paper
No paper is written, submitted or under review. If this project produces a contribution worth defending, that is where it would appear.
Where it stands
IDEA
CompleteFramed from a problem three BuildHub projects share.
LITERATURE
In progressEvaluation methodology reading in progress.
EXPERIMENT
In progressQuestion set under construction.
ANALYSIS
Not startedPending enough labelled data to say anything.
PAPER
Not startedNot planned. A usable methodology would be enough.
Open to
Contributors who will write and label questions carefully
Anyone interested in evaluation methodology
Accuracy note · This is an ongoing student research initiative. Nothing on this page is a proven result, a peer-reviewed finding or an established contribution to the field, and none of it should be cited as one.