Measure draft edit effort
Module · Stable
Grade a draft by how much a person typed to send it
Grades an AI-drafted reply against the reply a person actually sent, by the effort the edit cost them. Deterministic: no model, the same pair always gives the same numbers. Both texts are split into sentences (bullets and numbered steps count as sentences, markdown is flattened); greetings, sign-offs, signatures and courtesy lines (“Thanks for reaching out.”, “Let me know if you have any questions.”) in EN, CS, DE, ES, IT, FR and PL never count, and quoted mail under the sent reply is cut away. Each sent sentence is aligned to its closest draft sentence: kept costs nothing but a word fixed inside it, edited costs the words retyped inside it, new costs all its words; a draft sentence they dropped is counted as deleted but costs no typing. band reads the numbers: perfect (at most perfect_words typed, at most one sentence dropped), light (at most light_words typed and max_deleted_light dropped), partial (built on the draft), unusable (nothing of the draft survived, or same_answer is false), unknown (either text is empty). usable is perfect or light. The thresholds are Job config: calibrate them on the desk’s own labels, the defaults are a starting point. Example: a support Job grades each draft against the agent’s sent reply and charts band per week.
When to use
Section titled “When to use”Scoring a draft against the text a person really sent — a support copilot’s KPI, a weekly quality chart, a regression check after changing the drafting prompt. Pass same_answer from a separate check when the reply may give a different resolution than the draft.
When not to use
Section titled “When not to use”Not a judge of whether the answer is right or the same resolution — it measures typing, not meaning; ask a model for that and pass it as same_answer. Not for comparing two drafts with each other or for general text similarity.
Inputs
Section titled “Inputs”Configured per use: draft, sent, perfect_words, light_words, max_deleted_light, same_answer.
Outputs
Section titled “Outputs”Auto-generated from the skill registry (load_skills()). Do not edit by hand.