Benchmark

No benchmark has been published yet. This page shows only real measurements on a fixed test set, and it will say where another tool is ahead of us. Until then, nothing on this site claims to be the most accurate or the best.

What will be measured: AUROC, true-positive rate at 1% false positives, false positives on non-native English and academic writing, accuracy on humanized AI text and on short texts, and speed, for TrueScribe and the detectors whose providers allow benchmarking; for the humanizer, meaning kept, protected numbers and names, readability, detector scores, blind judge and human preference, and cost.

The test set will have at least

• 1,500 human texts (500 non-native English, 300 academic, 300 technical) and 1,500 AI texts
• 8 AI models, 8 genres, raw, lightly edited and humanized by 3 outside tools plus ours

Method

• Fixed test set. Versioned and checked against its SHA-256 before every run. A private hold-out is never used for training or tuning and never read by the benchmark code.
• Operating point. Tools score on different scales, so for each tool the threshold that gives at most 1% false positives is chosen on the human texts of a separate tuning split, then applied unchanged to the test split. The false-positive rate it really gave is shown next to the true-positive rate.
• Intervals. Figures in brackets are 95% intervals: bootstrap resampling for AUROC and rates that are averages, Wilson intervals for plain proportions. Differences inside the intervals are not differences.
• Outside tools. Run through their APIs only where the provider's terms allow it, with calls cached. Tools with only a web page were typed in by hand for a sample and are marked manual; they are not compared one-to-one with full-set numbers.
• Humanizer. Meaning similarity (embedding model when installed, otherwise word overlap; the method is stored in the results file), claim-by-claim comparison by a language-model judge, protected-item integrity, blind shuffled pairwise comparisons by at least two judge models from different families, and blind human votes.
• Limits. A detector score is an estimate. Results on this test set may not carry over to your texts, and tools change over time, so every table has a date.