Assessing the Legal Acumen of GPT Exploring its Capabilities and Limitations in the Field of Law

The GPT-4 model, presented by OpenAI, achieved a high score, placing in the top 12% overall, in the Law School Admission Test (LSAT), a prominent entrance examination for law schools in the United States. However, OpenAI did not disclose the specific problem-solving process behind this achievement. In this study, we utilize both GPT-3.5 and GPT-4 models to solve "The Official PrepTest June Sample" from the LSAT exam administered in 2007. Through this, we aim to evaluate the analytical reasoning and reading comprehension abilities of GPT models in the context of LSAT exams, by having human evaluators assess their suitability in the legal domain. This study compares the performance differences of GPT models based on the same questions and prompts, with the objective of exploring potential applications of GPT in the field of law.