The Play — Issue #26: The Redlines Lawyers Turn Down

Microsoft put a set of legal tools inside Word last week. If your firm pays for Microsoft 365 Copilot, they're probably already switched on, and the next time someone opens a contract and asks Copilot a question, Copilot will use them. Microsoft's headline number is that lawyers testing it accepted about seven out of every ten changes it suggested. I'm more interested in the other three.

Here's what shipped. On October 15 Microsoft released Copilot for Legal, a bundle of more than twenty legal skills that live inside Word and inside Cowork, Microsoft's workspace for longer, multi-step jobs. It can review a contract against your playbook, draft from a precedent, pull out deadlines and obligations, and build a timeline for a case. Plugins are rolling out for tools you may already use, including iManage, NetDocuments, Harvey, Legora, Bloomberg Law, Everlaw and Relativity. Westlaw and Lexis aren't on the list. Microsoft is the fourth big AI company to launch a legal product since May, after Anthropic, Google and OpenAI.

The part your lawyers will actually touch is contract review. You give Copilot your playbook as a Word document, and it reads the contract against it and suggests changes, one issue at a time. For each suggestion the lawyer has two choices. Accept it, and it goes into the document as an ordinary tracked change. Or turn it down (Microsoft calls this dismissing), and it never touches the document. There is also an Accept All button, which does what you'd expect.

Microsoft says that since April, more than 1,800 early users have produced nearly 100,000 of these suggested edits and kept "roughly seven in ten suggestions." It also says its own testing scored the edits above 90 percent for fluency, 95 for conciseness and 85 for relevance, without saying what any of those mean. Notice what isn't on that list: whether the edits were right. Microsoft's product documentation is more candid than its launch post: "Apparent confidence or fluency in output is not an indicator of legal accuracy."

Seven in ten tells you how often a lawyer agreed with the machine. It doesn't tell you how often the machine was right. A bad suggestion that someone accepted on a busy afternoon counts as a success in that number.

The three in ten that got turned down are more interesting. Each one is a small moment where a lawyer read a suggested change, thought "no, that's wrong," and knew why. The reasons come in a short list:

That last one is the thing a legal ops director most wants to know and almost never finds out.

Right now all of that disappears. A rejected suggestion leaves no tracked change, no comment and no note. The lawyer clicks, moves to the next issue, and the reason goes with them. By Microsoft's own ratio, a contract that draws twenty suggestions produces about six of those decisions. Over a month of NDAs, that's a lot of judgment nobody writes down.

Back in Issue #24 we suggested writing a one-page playbook for the contract you see most. The rejected suggestions are how that page gets better, using feedback your lawyers are already giving, just not to anyone.

If your expert uses ChatGPT while preparing a report, how much of that history does the other side get? Two federal courts have now given different answers.

In Villanueva v. Las Vegas Metropolitan Police Department, a civil rights case in Nevada, a magistrate judge ordered the plaintiff's expert to hand over his entire ChatGPT history, all 40 pages of it. On September 29, District Judge Anne Traum pulled that back and sorted the history into piles:

And using ChatGPT didn't waive anything, because, as she put it, "ChatGPT is not a person, but was an application."

Back in May, a magistrate judge in Connecticut went the other way on the questions. In Conservation Law Foundation v. Shell Oil, Judge Thomas Farrish treated the expert's prompts as part of her methodology and ordered them produced. CLF argued that an existing agreement between the parties covered this material, but the judge said an agreement limiting discovery has to be "quite clear," and theirs wasn't. CLF's objection gets argued November 3.

The read: Both courts agree that whatever the AI handed your expert, and your expert used, is fair game. They disagree about the questions. The CLF lesson is the cheap one: the parties had an agreement, and it wasn't clear enough to reach AI prompts. If you're negotiating an expert stipulation now, one sentence saying whether prompts, AI outputs and AI-assisted drafts are in or out could save you a motion later.

Starting October 30, if your firm uses Meta for business, whether that's a Facebook page, Instagram or ads, you're agreeing to new terms. They let Meta send automated crawlers through the websites connected to that account and use what they find to train its products, AI included. The terms also let Meta create test accounts to get in. You agree just by continuing to post or run ads after the 30th. According to Nieman Lab, which broke the story October 15, there's no opt-out short of stopping business use of Meta.

Cornell law professor James Grimmelmann told Nieman Lab those test accounts could let Meta sign up like a regular user and get past a registration page. Meta says the update pulls together language it already had in other documents. It also says website owners can still keep crawlers out with robots.txt, the small file that tells bots where they're allowed to go. Meta's own crawler documentation, though, says a couple of its crawlers can ignore that file in some situations.

The read: Most firms have a Facebook page, and plenty buy ads there. Before the 30th, have someone pull up every website and subdomain tied to the firm's Meta account and look at what's actually on them: practice pages, ad landing pages, intake forms, anything behind a login. Then check that your robots.txt file mentions Meta's crawlers. It's an open question whether that file beats terms you've already accepted, so if you have an intake portal where anyone can create an account, start there.

Microsoft's instructions cover how to accept a suggestion and how to reject one. They don't cover writing down why.

See you in the next one.