As developers continue to expand the capabilities of generative AI, lawyers have begun to question how GenAI could impact litigation in the future. To address these questions, Vanderbilt Law School’s Branstetter Litigation and Dispute Resolution Program and the Vanderbilt AI Law Lab hosted a discussion on GenAI and its implications on the legal field. The presentation was moderated by Professor Brian Fitzpatrick and featured Robert Keeling, the head of eDiscovery and Data Analytics at Sidley Austin.
Discovery and document review
In his role as head of eDiscovery at Sidley, Keeling is responsible for running large document reviews, privilege reviews, and producing documents. Projects can involve document populations of tens of millions of documents. Such large document populations require enormous amounts of work and funding to review, so Keeling’s group has been an early adopter of technology to help alleviate some of those burdens.
For the past year and a half, Keeling has been experimenting with GenAI as either a complement or a substitute to traditional AI prevalent in the legal field. To do this, Keeling works with colleagues who have strong backgrounds in technology.
“I’m very blessed to have former computer scientists [and] council on our team who have advanced degrees in machine learning,” Keeling said. “We’ve been working by ourselves or in partnership with other vendors to either create or test different applications for GenAI for different legal use cases.”
Keeling believes that large language models can be used most effectively in two areas: discovery and certain legal tasks associated more with practice.
Implementing GenAI
Generative AI derives its name from its ability to create new content based on large language models. At first, Keeling wasn’t sure whether GenAI could help to identify if a document was relevant to a case like traditional AI.
“I was very skeptical when we first started down this path that Generative AI could do this [identify relevancy],” Keeling said. “Particularly whether it could do it better than our current suite of technology.”
At the time, Technology Assisted Review (TAR) was the norm. Otherwise known as predictive coding, TAR can score millions of documents on relevancy based on other sample documents that it is given.
Keeling eventually found that GenAI could not only replace, but actually outperform TAR.
“What we’re seeing is, with sufficient iteration, it does better than traditional AI in the review of documents,” Keeling said. “In other words, when we run traditional machine learning on a data set, then run generative AI on the same data set, we are finding Generative AI is doing better.”
Keeling conceded that GenAI was not yet an adequate full substitute for human review but could significantly reduce the labor involved in document review.
“Generative AI can review, right now, about two documents per second, compared to the traditional human review of about 40 documents per hour,” he said.
The GenAI also goes a step further than TAR and gives a one-sentence summary, relevant snippets as well as possible flaws in methodology. The methodological flaws are particularly helpful for Keeling and his team.
“When we’re seeing things that it’s getting wrong, it’s telling us what it’s thinking,” Keeling said. “We can use that to revise the prompt.”
Keeling then explained that GenAI has a successful document identification rate of 86% and an accuracy rate of 77%. “That’s better than traditional machine learning,” he said. “It also compares pretty favorably to human review.”
GenAI can also mitigate costs associated with document review. According to Keeling, the cost for traditional AI was 70 to 80 cents per document a year ago, while the rate for humans stands at just over a dollar per document. However, with GenAI, those costs dropped to 15 cents per document.
Other uses for GenAI
Keeling’s job also involves HSR second requests—broad document requests by the government in the midst of a merger. 15 years ago, Keeling would have needed a team of 300 to 500 attorneys for these requests. With traditional AI, that number fell to 80 to 120 attorneys—with GenAI, firms only need 30 to 50 attorneys.
“We still need contract attorneys,” Keeling said. “But I would estimate in the future we will use about 90% less compared to no machine learning or AI help whatsoever.”
He also discussed the potential for GenAI as a chatbot for cases. In this application, the AI would be given a repository of all the documents for the case and be able to answer questions about it.
“As it stands right now, we don’t have GenAI that can do that,” Keeling said. “It’s too expensive still to have GenAI look at a large repository.”
Keeling does believe a GenAI with this capability will be available soon.
Implications of GenAI on the legal field
Keeling admits that lawyers are very slow to adopt technology, so he doesn’t believe there will be widespread adoptions of GenAI across big law firms in the near future. Firms like Sidley are also reluctant to allow firmwide implementation, because they do not want accidental misuse to result in sanctions. Still, Keeling believes that traditional AI will be adopted by more firms soon as people grow more familiar with the idea of AI in everyday life.
