oct 28, 2026

I tricked Anthropic’s new legal tool and it failed 4 of 6 traps.

Anthropic launched a tool for contract lawyers today. You give it your playbook (i.e. rules and guidance) and a contract, and it marks each clause:

• Standard (fine)
• Fallback (acceptable)
• Deviation (breaks a rule)

I tested it on a classic negotiation trick: the other side gives you what you asked for BUT takes away a protection in the same edit. I took 5 real contracts (public SEC filings), wrote a playbook for my client in each, and made a revised draft with one clause changed this way (6 changes in all). Since each change goes against my playbook, the right answer should have been “Deviation” all 6 times. However, it said “Standard” for 4 out of 6 (of the other two, it caught one and flagged the other for an unrelated reason - so it missed the actual problem in 5 of 6!).

1. The deposit

My client keeps a deposit with the platform (Party A), which can take fines out of it. The tool noticed the clause doesn’t say “reasonable”, said so, and passed it anyway. My client can still dispute a deduction, but the clause no longer requires deductions to be reasonable.

Key: red is deleted, green is added, yellow is unchanged (same for all pictures). The tool only saw the final text.

2. The adviser

My client is an adviser who can be fired for repeatedly refusing work the Board asks for. Without “reasonably”, that now includes unreasonable requests.

3. The bad batch

My client (Intersect) buys an ingredient from Hovione and can reject a batch that’s off spec, with a lab settling any dispute. The picture shows two sentences:

• The first decides who wins the dispute. The edit deleted “materially” so the batch now has to meet spec exactly. Good for my client.
• The second, starting at “Hovione shall pay all the fees”, decides who pays for the lab. The edit left “materially” in. Bad for my client.

Say the spec is 99.5% pure and the batch is 99.4%: my client wins the dispute but still pays for the lab and the replacement.

4. The licence

My client (NLS) gets a cut of Eurofarma’s sales, so it wants a say in its sales plan. My playbook says Eurofarma can’t change its plan without my client’s approval. After the edit, Eurofarma only needs that OK to: sell fewer units, launch later or spend less. Under this clause, any other change is its call alone. The tool marked it Standard. (It flagged a different clause nearby, just not this one.)

What its notes show

• It grades the clause as a whole. Once the clause contains what you asked for, it sees that it is a match and the missing word becomes a side note: “matching the standard … although”.
• It shortens your rule. When it restates my playbook, the small words drop out: “only Board-requested services” instead of “services the Board reasonably requests”. That's a problem because it checks the clause against its own shorter version.
• Its note and its verdict disagree. It wrote down the missing word and still chose Standard.

What fixes it

Plain Opus with a simple prompt caught all 6, so the model can do this.

So I rebuilt the tool’s exact prompt (it calls a clause standard when it “meets our standard position in substance”) and ran it on Opus. At low effort it made most of the same misses, almost word for word.

Two things fixed it: raising the effort or changing the prompt to “standard only when the clause meets every requirement”. Either way it caught at least 5 of 6. I can’t see which model or settings the tool runs on, so this is tested on Opus.

Limitations

Full disclosure, I’ve been researching exactly this for a bit: AI contract edits that look like a win but actually give something away. The 6 changes came from that research. They’re real AI edits I had already flagged as bad, and here they play the other side’s revised draft. The tool only saw the final text either way. I picked them on purpose, so don’t read 4/6 as an error rate !! (or 5/6, depending on whether you count the one it flagged for the wrong reason)

When this tool launched, it was the perfect test: would a tool built to check contracts catch them?

I also wrote the playbooks so each rule says exactly what my client needs, including the exact words that later went missing. So the tool had everything it needed, and in 3 of 4 it marked Standard, its own notes show it saw the gap. The one it caught was where my playbook also said, in a separate sentence, what wasn’t acceptable.

I kept the explanation to each example to a few lines so this stays short but there’s plenty more reasoning behind every call. If you’d read one differently, my DMs are open!

Everything is public so you can open the actual tool with my playbooks, contracts and every verdict here: https://claude.ai/artifact/Sba8sk5iudvnBDPRHKrvCE

first published on x