Most AI content projects fail when someone asks what "good" means and nobody has written it down. I have seen this happen on enough projects to know the fix. An LLM evaluation framework decomposes quality into measurable dimensions, agrees on a representative evaluation set, and commits to pass rates on that set before building. That is how we approach every LLM project at NineTwoThree.
In short
Two mistakes make quality feel impossible to define. First, treating "quality" as one thing instead of decomposing it. Second, implicitly promising that every output will be good. I have watched both happen on project after project.
The problem with the second mistake is mathematical. If your system takes human inputs, the input space is infinite. You cannot test every possible input, so you cannot promise every output will be good. What you can promise is that the system clears a bar on a representative sample. That bar is defined by the dimensions you decompose quality into, the measurement method for each, and the threshold you agree on.
This is the single most important move. It is how every serious LLM product is evaluated, and it is what makes the commitment both buildable and defensible.
The first mistake, treating quality as one thing, is what makes the second one feel inevitable. If "quality" is a single fuzzy concept, you cannot write a test for it. Once you decompose it into named dimensions, each one becomes testable. Factual accuracy is checkable. Requirement coverage is countable. Writing quality is gradable on a rubric. Each dimension gets its own definition, its own measurement method, and its own threshold.
We wrote about this in the context of our AI engineering SDLC, where evaluation suites are part of the build pipeline, not an afterthought.
The useful split is objective/checkable vs subjective/preference. They need fundamentally different measurement methods, and conflating them is where most teams get stuck.
Objective dimensions are binary or countable. Factual claims grounded in the source. No fabrication. All mandatory requirements addressed. Required sections present. Format constraints met. These can be checked programmatically or with a tight rubric. On an RFP automation project, a hallucinated capability statement is a compliance problem, not a style nit. That is the risk level.
Subjective dimensions are graded. Writing quality, tone, persuasiveness, depth. These need anchored rubrics with a written description per level, scored by humans or by an LLM judge calibrated against humans. The rubric is the shared definition of quality that you co-author with the team.
| Type | Examples | How to measure | Where the risk lives |
|---|---|---|---|
| Objective (binary or countable) | Factual claims grounded in source, no fabrication, mandatory requirements addressed, required sections present, format constraints met | Programmatic checks or tight rubric | A hallucinated capability in an RFP response is a compliance problem, not a style nit |
| Subjective (graded) | Writing quality, tone, persuasiveness, depth/sufficiency | Anchored rubrics (1 to 5 with written descriptions per level), scored by humans or LLM judge calibrated against humans | Lower stakes but harder to agree on |
Objective dimensions are where the real risk lives. We have seen AI failures stem from exactly this gap, where teams focused on style while factual accuracy went unchecked. We wrote about those patterns in our AI failures analysis.
Four methods, each matched to a type of dimension. The method determines what you can promise and how you defend it.
| Method | When to use | What it measures |
|---|---|---|
| Programmatic checks | Objective criteria | Requirement coverage, citation grounding, fabrication detection against the source |
| Anchored rubrics | Subjective criteria | Writing quality, tone, depth. The rubric is the shared definition of quality, co-authored with the team |
| LLM-as-judge (calibrated) | When you need scale | Have subject matter experts score a batch, tune the judge until it agrees at a reported rate, then evaluate at scale cheaply |
| Pairwise comparison | When absolute scoring is hard | "Beats the current human first draft X% of the time" is easier to agree on and measure than an absolute score |
Programmatic checks handle objective criteria. Requirement coverage, citation grounding, fabrication detection. These are binary or countable, and you can automate them.
Anchored rubrics handle subjective criteria. A rubric with a written description per level is the shared definition of quality. You co-author it with the team. Level 4 on writing means specific things, not a vibe.
LLM-as-judge lets you evaluate at scale. Have subject matter experts score a batch of outputs, then tune the judge until it agrees with them at a reported rate. That agreement rate is the evidence the judge is trustworthy. Below 0.6 Cohen's Kappa, the evaluation task itself is ambiguous. That insight comes from Galileo AI's research on calibrating LLM judges, and it matches what we have seen in practice.
Pairwise comparison works when absolute scoring is hard. "Beats the current human first draft X% of the time" is easier to agree on and measure than an absolute score. The baseline is usually the team's current human process.
Thresholds are pass rates on an agreed evaluation set, not promises about every future output. The shape matters.
On the agreed evaluation set: 90% or more of outputs address 100% of mandatory requirements. 80% or more score 4 out of 5 on depth and writing. 0 contain ungrounded factual claims.
Note the asymmetry. 100% on the safety-critical objective dimension. Distributional targets on the subjective ones. This is deliberate. A single hallucinated factual claim can cause real damage. A writing score that occasionally dips to 3 out of 5 is tolerable if the distribution holds.
Then tier inputs by quality and tie commitments to the tier. On a content creation project, we discovered that input quality varied wildly and you only find its problems over time. Garbage-in cannot be a quality miss you own. Define input quality tiers and state that targets hold for tier 1 to 2 inputs. Below that, the deliverable is flagging the input as insufficient.
This is the same principle we wrote about in our data quality analysis. The quality of what goes in determines what comes out. If you do not tier inputs, you are implicitly promising results regardless of what people feed the system.
Evaluating dozens of individual interactions is the trap. I fell into it early on and learned to avoid it. The unit of measurement becomes the checkpoint, not the keystroke.
Define 3 to 5 checkpoint outputs in the workflow: outline, first draft, post-edit draft, final. Put an acceptance gate at each. The gate is what the AI must deliver for the human to accept and proceed.
Then measure the AI's contribution at each gate:
The honest metric is responsiveness to direction. Given this input and these human instructions, did the AI produce a checkpoint output that met the bar? These metrics map directly to value. "The AI gets you to an acceptable draft in N fewer cycles" is the thing people are buying.
This cleanly separates our accountability (the AI hits the gate given reasonable direction) from theirs (the human's choices shape the final piece). We wrote about this separation in our human-in-the-loop analysis.
There is an apparent chicken-and-egg problem. You want to agree on "acceptable" before you have built a testable thing. It dissolves once you see what calibration rates.
Calibration doesn't measure our system. It measures whether the humans agree with each other on what "acceptable" means.
Calibration does not measure your system. It measures whether humans agree with each other on what "acceptable" means. You only need outputs to judge, and those do not have to come from your pipeline. You manufacture the sample.
Three sources for pre-build samples. First, existing work. The team already produces these artifacts today. Have their experts blind-rate a batch of their own historical output. This reveals the bar their real-world process clears. On a client project, we found their own "good" work scored a 3.5, not a 5. That reset the target honestly before we had promised anything.
Second, a hand-built spectrum. Deliberately assemble 15 to 20 samples spanning clearly great to clearly bad, including some near the boundary. The boundary cases are where the rubric gets sharpened.
Third, competitor or comparable outputs where available. These give you a range of quality that exists in the wild, which helps calibrate expectations.
Frame it as defining the answer key before the exam. A teacher writes the rubric and grades sample essays before students sit the test. The calibration samples exist to pin down the grading standard, and it is better that they are not yours, because nobody can accuse the standard of having been reverse-engineered to make your system pass.
The honest caveat: this first bar is a v1, set on proxy samples. You will re-confirm it against real system outputs at the first milestone. That is why the revision checkpoint matters.
The full sequence: calibrate on borrowed or built samples, lock the rubric and the scored reference examples, build against that bar, run system outputs through the agreed rubric, and re-confirm the bar against real outputs at the first milestone. The agreement on acceptable happens entirely in the human-judgment phase, before a single batch goes through the pipeline.
Write it down. The documentation should include:
The evaluation suite is a living artifact you refine together at checkpoints. That is more credible than claiming you can fully specify quality before you have seen the data. We have seen teams skip this step and pay for it later, when the lack of written agreement turns into a disagreement about what was promised. We wrote about the upstream version of this problem in our data preparation guide.
Two real projects show how the framework adapts to different content types and quality risks.
| Content creation project | RFP automation project | |
|---|---|---|
| Dimensions | Checkpoint gates: outline, first draft, post-edit, final | Coverage-across-topic, coverage-within-topic, accuracy, writing |
| Methods | Acceptance rate and edit distance per gate | Programmatic requirement check, depth rubric, grounding check (zero tolerance), pairwise vs past winners |
| Input tiers | Yes. Variable source content tiered | Yes. Past solicitations held out as evaluation set |
| Headline metric | Cycles and time saved at an acceptable quality bar | SME-acceptable-with-minimal-edits or win rate |
| What we promise | The system hits each gate given reasonable direction | The system passes the eval suite at threshold on the agreed set |
On the content creation project, we used checkpoints with acceptance gates. Per-gate metrics were acceptance rate and edit distance. Input tiers protected us on variable source content. The headline value was cycles and time saved at an acceptable quality bar.
On an RFP automation project for a healthcare company, we decomposed quality into named dimensions. Coverage-across-topic was a programmatic requirement-coverage check against the solicitation. Accuracy was a grounding and fabrication check against the knowledge base, with zero tolerance. We evaluated on a held-out set of past solicitations with known good answers. The ultimate proxy metric was SME-acceptable-with-minimal-edits or win rate.
Define quality relative to a baseline, not in absolutes. "As good as a competent junior's first draft, in a fraction of the time" is honest, achievable, and measurable. Absolute perfection is none of those things. Relative bars are easier to agree on, easier to measure against the baseline, and more defensible.
NineTwoThree by the numbers: 97% project success rate. 24 of 27 projects ROI-positive. 150+ AI projects shipped.
If you are evaluating an AI partner for a content or document automation project, ask how they define quality before they build. The answer should include named dimensions, measurement methods, and pass-rate thresholds. We bring that framework to every engagement, across 150+ projects with a 97% success rate. Talk to us about your project.
.jpg)