A new study released on Wednesday reveals that artificial intelligence (AI) can outperform most law school graduates on the bar exam, a two-day test that aspiring attorneys must pass to practice law in the United States. The study was conducted by two law professors and two employees of legal technology company Casetext, who tested GPT-4, the upgraded AI model released by Microsoft-backed OpenAI this week. GPT-4 scored 297 on the bar exam, placing it in the 90th percentile of actual test takers and enough to be admitted to practice law in most states.
The bar exam assesses knowledge and reasoning, including essays, performance tests to simulate legal work, and multiple-choice questions. The study’s authors wrote, “Large language models can meet the standard applied to human lawyers in nearly all jurisdictions in the United States by tackling complex tasks requiring deep legal knowledge, reading comprehension, and writing ability.”
Less than four months ago, two of the same researchers concluded that OpenAI’s earlier large language model, ChatGPT, fell short of a passing score on the bar exam, highlighting how rapidly the technology is improving. The newer GPT-4 got nearly 76% of the bar exam’s multiple-choice questions right, up from about 50% for ChatGPT, outperforming the average human test-taker by more than 7%.
On Wednesday, the National Conference of Bar Examiners, which designs the multiple choice section, said that attorneys have unique skills gained through education and experience that “AI cannot currently match.” Study co-author Daniel Martin Katz, a Chicago-Kent College of Law professor, said in an interview that he was most surprised by GPT-4’s ability to produce broadly relevant and coherent essay and performance test answers.
“I heard so many people say, ‘Well, it might get the multiple choice, but it will never get the essays,'” Katz said. AI has also performed well on standardized tests, including the SAT and the GRE, but the bar exam has garnered more attention. OpenAI touted its passing score when it announced the latest model on Tuesday.
Bar exam tutor Sean Silverman attributed the focus on the bar exam to its widely recognized difficulty. This year’s first-time pass rate on the attorney licensing exam was 78% among test takers who spent three years in law school. Silverman said people may be less impressed to learn that AI can pass a test designed for high schoolers, like the SAT, “rather than the test to become a lawyer.”
Despite AI’s impressive performance on the bar exam, the study also acknowledges that the legal profession requires unique skills gained through education and experience that AI cannot currently match. The researchers noted that AI could assist human lawyers in many tasks and be particularly useful in legal research, but ultimately, the decision-making power remains with human lawyers.
In conclusion, the new study conducted by two law professors and two employees of Casetext shows that artificial intelligence has significantly progressed in outperforming most law school graduates on the bar exam. However, while AI can assist human lawyers in many tasks, the legal profession still requires unique skills gained through education and experience that AI cannot currently match. This study highlights the rapid pace of technological advancement and the potential of AI in the legal field. Ultimately, human lawyers remain the final decision-makers, and AI is expected to enhance legal services rather than replace them.