Evaluation

Five questions, five separate evaluations. Never merge them into one score. A profile can be valid while the prose is poor; prose can follow every rule and still not sound like the user; good prose that changes a fact is a failure.

Run an evaluation only when the user asks for one. Report each part as passed, failed or not tested, with its evidence and its limits.

1. Profile structure

Is the profile in the right shape?

Run python3 <skill>/scripts/vf.py validate. No errors means passed. Warnings (empty brief fields) are listed, not failed.
This says nothing about the quality of any text.

2. Rule compliance

Do texts follow the rules that apply, and does the reviewer judge that correctly?

Writers: draft or edit a few texts with the profile, then review each with an independent reviewer (review.md). Count VIOLATIONs by severity. Compare with the same task written without Voice Forge, reviewed the same way.
Reviewers: take a text, plant violations of specific rules, and add passages that look like violations but fall under an exception. Write the expected verdicts (- R05: VIOLATION, - R09: OK, - R12: ANY) and score the reviewer's report with vf.py review-score <expected> <report> --rules <rules.md>. It counts caught, missed, uncertain and false alarms.
The repo's tests/fixtures/review/ is a worked example with an invented profile.

3. Voice fidelity

Does the prose sound like the user? Only the user's own texts and the user's own judgment can answer this. Rule compliance and the absence of AI phrases are not evidence of fidelity.

Use the holdout texts: texts the user wrote that no round, sample or analysis has used.

Run vf.py holdout check. It confirms no holdout paragraph leaked into samples, rounds or analyses.
For one holdout text, ask the user for a short brief: the topic, the main points and the length. Do not open the holdout text. The brief comes from the user.
Write two or more versions from that brief, each in a fresh context (a subagent, or a new chat if your agent has none): one with Voice Forge and the profile, one baseline with the same model and no skill. Optionally a third with only the generic slop rules. Save each as <source>.md, always with the same names, such as voice-forge.md and baseline.md: the file name is how results are counted.
Optional numbers: python3 <skill>/scripts/measure.py compare <holdout text> <version files> compares sentence, paragraph and stance measures with the user's real text. A script reads the holdout text; you only report the table. The numbers describe surface habits; they do not prove fidelity.
Run vf.py blind make <name> <version files>. It shuffles the versions into evals/<date>-<name>/blind.md with letters. Never open key.json. The user reads blind.md, picks the version most like them, and may rate each from 1 to 5.
Run vf.py blind score <date>-<name> --pick <letter>. It reveals the source and logs the result. vf.py blind summary shows how often each source was picked.

Without a shell (a chat app): show the versions under letters in an order you choose, write the key down without showing it, and reveal it only after the user picks. Say that the shuffle was not done by a script. Log the result by hand in evals/fidelity-log.md.

Report the counts, not a percentage alone, and say how many comparisons there were. With fewer than about 10 comparisons, call the evidence thin. Never write "fidelity validated" from a few comparisons.

4. Writing quality

Is the text good for its genre and readers, whatever the voice?

Use a judge that does not see the profile: an independent subagent if possible. Give it the task, the readers and the text. It rates each from 1 to 5, with a short quote as evidence:

Clarity: each sentence is easy to follow on first reading.
Coherence: paragraphs follow from each other; the main point is clear.
Correctness: grammar, spelling, agreement, punctuation; accents and «» in Spanish.
Effectiveness: the argument or explanation does its job for these readers.
Fit: length, register and structure suit the genre and venue.

Report the five ratings separately. Do not substitute rule compliance for quality.

5. Meaning and facts

Did the text keep every fact, number, name, quote, attribution, qualification and logical link of its sources?

The reviewer's Meaning section checks this when it gets the sources. For drafts, the source is the facts the user gave.
Any unauthorized change in meaning is a critical failure for that text, whatever the other four results say.

Report

# Evaluation: <profile>, <date>

## 1. Profile structure: passed
- vf.py validate: 0 errors, 2 warnings (Scope, Genre empty)

## 2. Rule compliance: passed with limits
- 3 drafts, independent reviews: 0 high, 1 medium violation; baseline: 4 high, 2 medium
- reviewer test: 8 of 9 planted violations caught, 1 uncertain, 0 false alarms in 25
- limits: one model, invented test texts

## 3. Voice fidelity: not tested
- no holdout comparisons yet

## 4. Writing quality: passed
- clarity 4, coherence 4, correctness 5, effectiveness 4, fit 4 (one judge, one text)

## 5. Meaning and facts: passed
- 3 of 3 texts preserved meaning

Save it as evals/<YYYYMMDD>-<profile>.md in the voice home, then vf.py save.