According to a new study released Wednesday, artificial intelligence can now outperform most law school graduates on the bar exam, the grueling two-day test prospective attorneys must pass to practice law in the United States.
GPT-4, the updated AI model released this week by Microsoft-backed OpenAI, scored 297 points on the bar exam in an experiment conducted by two law professors and two employees at legal technology company Casetext.
That puts GPT-4 in the 90th percentile of actual test takers and enough to get a license to practice law in most states, the researchers found.
The bar exam assesses knowledge and reasoning, and includes essays and proficiency tests designed to simulate legal work, as well as multiple-choice questions.
“Large language models can meet the standard set for human attorneys in almost every jurisdiction in the United States by tackling complex tasks that require sound legal knowledge, literacy, and literacy,” the authors write.
Less than four months ago, two of the same researchers concluded that OpenAI’s previous major language model, ChatGPT, failed to achieve the required score on the bar exam, underscoring how rapidly the technology is improving.
The newer GPT-4 correctly answered nearly 76 percent of the multiple-choice questions on the bar exam, versus about 50 percent for ChatGPT, outperforming the average human test-taker by more than 7 percent.
The National Conference of Bar Examiners, which designs the multiple-choice section, said in a statement Wednesday that attorneys have unique skills acquired through education and experience that “AI currently cannot match.”
Study co-author Daniel Martin Katz, a professor at Chicago-Kent College of Law, said in an interview that he was most surprised by GPT-4’s ability to produce broadly relevant and coherent responses to essays and proficiency tests.
“I’ve heard so many people say, ‘Well, it might get the multiple choice, but never the essays,'” Katz said.
The AI has also performed well on other standardized tests, including the SAT and the GRE, but the bar exam has attracted more attention. OpenAI touted its passing score when it announced the latest model on Tuesday.
Bar exam instructor Sean Silverman attributed the focus on the bar exam to its widely recognized difficulty. This year’s first-time pass rate on the bar exam was 78% among test-takers who completed three years of law school.
Silverman said people might be less impressed when they learn that AI can pass a test designed for high schoolers like the SAT, “rather than the test to become a lawyer.”
© Thomson Reuters 2023
