In recent years, artificial intelligence (AI) has made substantial strides, especially in the realm of legal services. The advancements have led major firms to explore the potential of AI tools in their operations. A notable investigation was conducted by the prominent British law firm Linklaters, which aimed to assess the effectiveness of AI chatbots in responding to legal queries. The study involved presenting these chatbots with a series of 50 challenging questions centered on English law, thereby placing them under scrutiny to evaluate their performance relative to human legal experts.
Linklaters’ findings were illuminating. The firm observed that OpenAI’s earlier model, GPT-2, which dates back to 2019, performed poorly, being described as “hopeless”. In contrast, its newer OpenAI o1 model, introduced in December 2024, exhibited marked improvements. This suggests that while AI tools are becoming progressively capable, they are not yet able to rival even a junior lawyer’s proficiency without substantial human oversight. According to Linklaters, these tools are on the verge of being valuable in real-world legal applications, though they need guidance from experienced legal professionals.
The legal sector is currently grappling with the implications of AI advancements. The rapid evolution of AI technologies potentially presents both threats and opportunities to traditional law practices. Hill Dickinson, an international law firm, exemplified this quandary by restricting access to several AI tools after observing a surge in employee usage. This highlights the growing concern regarding the unregulated incorporation of AI within legal frameworks.
Amidst this backdrop, there is an ongoing global discourse surrounding the risks associated with AI and the need for stringent regulations. The recent failure of the UK and the US to endorse an international agreement on AI further underscores these tensions. US Vice President JD Vance openly criticized European nations for allegedly prioritizing safety over innovation, revealing a divide in perspectives on how best to incorporate AI into various sectors, including law.
Linklaters’ second round of AI assessments, dubbed LinksAI benchmark tests, follows an original test conducted in October 2023. During the first round, different models, including OpenAI’s GPT-3 and GPT-4 and Google’s Bard, were evaluated. The latest assessment has been expanded to encompass the performance of Google’s newly launched Gemini 2.0, alongside OpenAI’s o1. Notably absent from this round was DeepSeek’s R1, a low-cost AI model that garnered attention recently for its impressive capabilities.
The test was designed to mimic the type of inquiries that a competent mid-level lawyer with two years of experience would typically navigate. The results indicated that while newer AI models demonstrated a substantial enhancement over earlier iterations, they still fell short of the standards expected of qualified lawyers. Incidents of factual inaccuracies, omission of critical information, and erroneous citations, although less frequent than with previous models, persisted.
Despite these imperfections, Linklaters recognized the potential utility of AI tools in legal research scenarios. The firm posited that these advancements could offer initial drafts for legal documents or assist in verifying answers. However, they cautioned that relying solely on AI without a foundational understanding of the answers could lead to significant pitfalls. There are ongoing inquiries regarding whether the current pace of AI competence will continue or if there are inherent limitations that might hinder further advancements.
Ultimately, Linklaters emphasized that the essence of legal practice—maintaining client relationships—remains a cornerstone of the profession. Even as AI technologies evolve, the “fleshy bits” of legal service delivery, which hinge on interpersonal skills and human judgment, are unlikely to be fully supplanted. This perspective highlights the importance of integrating AI into legal frameworks with caution, ensuring that human expertise continues to play a pivotal role in the delivery of legal services.
