I Spent Two Weekends Trying to Out-Engineer AI Detectors. Here's What I Learned About How They Work.
The ticket that started all this was titled "export flagged again." Our product has a feature that drafts client-ready summaries with an LLM. Users loved it right up until their own clients started running deliverables through AI detectors and bouncing them back. The text was accurate. The users had edited it. Didn’t matter: the gate said machine, and in that relationship the gate wins.
My first reaction was the standard engineer reaction: how hard can this be? Detectors are just classifiers. Classifiers have inputs. Inputs can be perturbed. I have a weekend.
Two weekends, actually. I lost both, and I am writing this so you can lose neither. Because the interesting part is not that my DIY humanizer failed. It is why it failed, which turns out to be a compact lesson in how these detectors actually work, and a case study in the oldest engineering question there is: when is a problem a feature of your system, and when is it someone else’s product?
Attempt one: the string-manipulation phase
Everyone’s first idea is the same idea. Detectors flag AI text; AI text has tells; find the tells, patch the tells.
So I built the naive pipeline. Synonym rotation with a thesaurus lookup so the model’s favorite words ("delve," "crucial," "comprehensive") got swapped out. Sentence splitting and merging on a random schedule to break up the uniform length. A contractions pass. A sprinkle of informality, the occasional sentence fragment. You have seen this code, or written it:
def humanize_naive(text):
text = swap_overused_words(text, THESAURUS)
text = vary_sentence_lengths(text, jitter=0.35)
text = apply_contractions(text)
return text
Results: on the detector I was testing against, my scores moved from "definitely AI" to "probably AI." Occasionally a paragraph squeaked into ambiguous territory. The prose, meanwhile, got noticeably worse, because a thesaurus does not know that "important" and "portentous" are not the same word. I had built a machine for making text read like a hostage negotiation, and it still got flagged.
The mistake, which took me embarrassingly long to see, is that I was editing the text when the detector was reading the distribution.
What detectors actually measure (the part I should have researched first)
Modern AI-text detectors mostly do not look for magic words. They score statistical properties of the whole passage. Two matter most.
Perplexity: how predictable each token is, given the preceding ones, under a reference language model. LLM output is generated by sampling from exactly that kind of probability distribution, so it is, by construction, high-probability text. Low perplexity is the native accent of a language model.
Burstiness: the variance in that predictability across the passage. Humans are erratic. We write three plain sentences and then one weird one, we start clauses and abandon them, we reach for an improbable word because it is funny. Models are smooth everywhere.
Here is why my weekend one failed, stated properly: swapping synonyms and jittering sentence lengths perturbs the surface, but the underlying probability structure, the smoothness, survives. It is the same reason you cannot anonymize a dataset by renaming columns. The signal detectors read lives at a deeper statistical layer than my string operations could reach. There is decent public research formalizing this: the PADBen benchmark work (arXiv 2511.00416) evaluates exactly this class of paraphrase-based evasion and shows how uneven naive transforms are against modern detectors, while the "Self-Disguise Attack" paper (arXiv 2508.15848) demonstrates the flip side, that even prompting a model to disguise its own output is a real but bounded technique. Reading those papers after my failed weekend was a familiar experience: oh. Everyone already knew this.
Attempt two: the model-in-the-loop phase
Weekend two, smarter plan. If the problem is distributional, use a model to fix the distribution. I set up a second LLM pass with an elaborate rewriting prompt: vary rhythm, inject asides, drop the register, allow imperfection. The prompt was three hundred words. I was proud of it.
This worked... better. Scores on my test detector dropped meaningfully. Then I ran the same outputs through two other detectors and got completely different verdicts, which was its own education: these tools disagree with each other constantly, because each one is a different classifier trained on different data with a different threshold. Optimizing against one is overfitting, the classic ML sin, just committed against someone else’s model.
Worse, my rewriting pass had a fidelity problem. About one paragraph in ten came back subtly wrong: a number rounded, a hedge deleted, a claim strengthened. In a feature that ships client deliverables, "subtly wrong one time in ten" is not a quality bar, it is an incident postmortem waiting for a date. So now I needed a meaning-diff check on top of the rewriter, and a multi-detector test harness on top of that, and regression runs every time any upstream model updated, because detector behavior shifts whenever the generators do.
Somewhere around 2 a.m. on the second Sunday I did the math on what I was actually building: a specialized rewriting model, an evaluation pipeline against a moving fleet of adversarial classifiers, and a fidelity-verification layer, all maintained forever, by me, adjacent to our actual product. I closed the laptop.
The build-vs-buy math nobody runs until 2 a.m.
Here is the framing I wish I had started with. "Make text read as human" is not a feature. It is an arms race, and arms races are terrible things to own as a side quest.
The detection side updates continuously. The generation side updates continuously. Any static transform you write decays as both sides move, silently, with no error thrown, until a user ticket tells you your humanizer stopped humanizing. Compare that to what it costs the vendors whose entire product is this one problem: they retrain against current detectors as a matter of survival, they benchmark daily, and their fidelity handling has survived contact with millions of documents instead of my forty test paragraphs.
This is the same judgment call as rolling your own auth, your own PDF renderer, your own timezone library. The difference between a competent engineer and an expensive one is knowing which rabbit holes are load-bearing for the business. Text-statistics warfare, for a team whose product is something else entirely, is not.
Run the actual numbers and it gets less romantic still. My two weekends were maybe twenty focused hours, call it a few thousand dollars of engineering time at any honest rate, and they produced a prototype that solved perhaps a third of the problem with an unbounded maintenance tail. The API alternative prices per word processed, at rates where our entire monthly volume costs less than one afternoon of my time. Even if a vendor bill grew tenfold, it would not approach the cost of me owning an adversarial ML problem part-time, badly, forever. The build-versus-buy spreadsheet was never close; I just refused to open it until 2 a.m. had softened me up.
What I ended up shipping
The production version is boring, which is how you know it is right.
Generation stays exactly as it was. After the draft, the pipeline makes one API call to a dedicated humanization service; we wired in a tool called UndetectedGPT after testing a few options against our own content types, and the integration was the least interesting part of the whole saga: POST the text, get the rewrite, done in an afternoon including retries and caching. Then, and this part we kept from my weekend-two wreckage, a cheap similarity check diffs the rewrite against the source, and anything that drifts past threshold falls back to the original with a flag for human review. Meaning stays ours; texture is the vendor’s problem.
If you are evaluating this space, two pieces of advice from the scar tissue. First, test candidates on your content, not the vendor demos; humanizers behave differently on technical summaries than on essays, and the deltas between providers are bigger than their landing pages suggest. There are decent engineering comparisons of humanizer APIs for content pipelines that are worth reading before you shortlist, mostly to calibrate on request shapes, latency, and where each provider sits on the fidelity-versus-transformation tradeoff. Second, keep the verification harness even after you buy. Trust, but diff.
And put the whole thing behind an interface. HumanizerClient is four methods in our codebase. If the vendor landscape shifts, we swap an adapter, not an architecture.
The harness I kept
One artifact from the failed weekends survived into production, and if you take a single practical thing from this post, take this: whatever you buy, keep your own evaluation harness.
Ours is embarrassingly small. A fixture set of about sixty documents that represent our real content types: short summaries, long reports, bulleted status updates, the weird half-structured exports users actually generate. A script that runs each through the humanization step and then through three public detectors, logging scores. A sentence-embedding similarity check between input and output, with a threshold we tuned by manually reviewing the worst fifty pairs (crude, but calibrated on our own tolerance for drift rather than a vendor’s). The whole thing runs nightly in CI and costs a few cents.
What it buys us is independence from marketing claims in every direction. When a detector updates and scores shift, we see it in the morning instead of in a support ticket. When our provider ships a model change, the fidelity distribution tells us whether the rewrites got more aggressive. When a new vendor pitches us, we run their trial key through the same fixtures and have a comparable answer in an hour, on our content, not their demo set.
The deeper lesson generalizes past this niche. Any time your product depends on someone else’s model fighting someone else’s other model, you are building on ground that moves. You cannot stop the movement. You can absolutely instrument it, and the difference between teams that instrument and teams that trust is the difference between reading about a regression in your dashboard versus reading about it in a churned customer’s exit survey.
The uncomfortable part, briefly
I want to be straight about the elephant: this technology sits close to an ethics line, and where it lands depends entirely on which side of the workflow you are standing on.
Our users are not laundering slop past anyone. They co-write with a model, they own the claims, and their clients' detectors were flagging texture, not dishonesty. People also get burned in the other direction; there is a growing pile of documented cases of fully human writing getting flagged, which is its own kind of failure with real consequences for the flagged person. The gate is real, it is often crude, and it is being enforced anyway. Building honestly around a crude gate, human substance, machine texture repair, verified fidelity, is a defensible position. Using the same tooling to pass off unreviewed machine output as considered human work is not, and no API contract will make it so. Know which product you are building.
What the weekends bought me
I did not ship a humanizer. I shipped an understanding, which is usually the consolation prize of failed builds, and occasionally the more valuable one.
Detectors read distributions, not words, so surface edits lose. Beating one detector is overfitting, because the fleet disagrees. Any rewriting pass strong enough to matter is strong enough to corrupt meaning, so fidelity checking is not optional. And a moving adversarial target is a product category, not a utility function, which means the right amount of it to build in-house is, for almost everyone, none.
Also: "export flagged again" is closed. The user whose client bounced three deliverables sent a one-line follow-up after the fix went live. "Whatever you did, it worked."
Two weekends. Worse ways to spend them.
Comments
Loading comments…