The most prestigious law school admissions discussion board in the world.
|
 |
 
|
Crazy that the AI Legal Agent benchmarks are amongst the lowest
Poast new message in this thread
Date: July 28th, 2026 1:59 PM Author: jet-lagged pearly native STEMmos have always looked down on us, but it turns out automating that stuff is way easier than putting together A DEAL! (http://www.autoadmit.com/thread.php?thread_id=5887079&forum_id=2#50029944) |
Date: July 28th, 2026 3:28 PM Author: jet-lagged pearly native (http://www.autoadmit.com/thread.php?thread_id=5887079&forum_id=2#50030208) |
|
Date: July 28th, 2026 3:32 PM Author: Onyx roast beef My irl lawyer friends don't believe me when I tell them that it's only a matter of time before enough money is thrown at RL environments to make LLMs almost as good at law as they are at programming
Right now they are already better than most attorneys (most attorneys suck) |
 |
Date: July 28th, 2026 3:41 PM Author: jet-lagged pearly native I agree it's better than many attorneys already and that someday it may get to that point you're saying. I'm just saying it's a bit suprising how much of a gap there is between legal performance benchmarks and other things.
https://x.com/i/status/2064394151441863006 |
|
Date: July 28th, 2026 3:48 PM Author: Onyx roast beef It's because they haven't thrown resources at RL for law specifically. The models are extremely good at programming because there has been an enormous amount of resources sunk into training them for that specifically. It's true that programming is easier (less resource intensive) than law to RL train for, but it still required a lot of effort to get models this good at programming
Eventually the same thing will happen for law. The incentives are there because lawyers are outrageously overpaid for the actual level of expertise and training that they have. They're able to command the excessive compensation they get only because they're a monopoly/cartel
AI displacing lawyers is going to be one of the most positive outcomes from AI because lawyers have such disproportionate power and influence in our society and their interests conflict with those of most people. They are an extremely anti-social force in modern society and responsible for a lot of our society's ills |
|
Date: July 28th, 2026 4:00 PM Author: jet-lagged pearly native But that monopoly exists because you cannot provide legal services without a license. The power and influence you're talking about won't change just because the work gets easier. According to you the work is already over overpaid relative to the effort required. (http://www.autoadmit.com/thread.php?thread_id=5887079&forum_id=2#50030280) |
|
Date: July 28th, 2026 4:20 PM Author: Onyx roast beef Law's power in the US comes from it being an unassailable religion first, and a cartel second
AI will do to law in the public consciousness what "covid" and AI have done to medicine in the public consciousness. Exposed industry professionals as largely fraudulent rent-seekers rather than the Experts and Priests they supposedly are
It will happen a lot faster than people think. It's a question of technical capability. Once LLMs can straight up beat almost all human lawyers, the flood of pro se pressure on the legal system will cause huge societal changes. People will not respect The Law anymore. It will become just another scam industry in the public mind |
|
Date: July 28th, 2026 4:35 PM Author: jet-lagged pearly native Everyone already knows lawyering is a bullshit quasi-religion. If AI starts crafting deal docs better than Wachtell that's not going to wake anyone up to that.
There's really no way to "beat" lawyers work by some objective metric other than if it has actual errors. What's "better" and who "wins" is decided by people. It's going to be a long ass time before the legal system even permits AI.
The people that decide whether the legal cartel exist are the legal cartel. Judges (all cartel) and lawmakers (mostly cartel) decide, and they're going to be *most* interested it kn preserving it. And I don't see some popular backlash coming on such a niche issue. People will be more concerned about their jobs being eliminated than dragging lawyers into the same bucket. Fuck, PACER and other court records are still often paywalled when that should have been free 20 years ago. |
|
Date: July 28th, 2026 5:14 PM Author: Onyx roast beef Unfortunately, the general public absolutely does not think that law is bullshit. It's the only "institution" that is still widely respected across demographics. This will change
The legal system already permits AI. It's being used right now across the country. Everyone is using it
It's only a question of capability, which is only a question of time and resources invested. It's going to happen. The incentives are too powerful
|
|
Date: July 28th, 2026 5:22 PM Author: jet-lagged pearly native I don't see that being a high priority or more apparent than now when AI gets to the point you're envisioning. It will create so many other issues that toppling the legal priesthood will be a lower priority than it already is.
I agree with you about it being a rent-seeking priesthood and that it would be good for society if that was dimished. I just don't see it happening before 1000x other things that will be more pressing. It's just too niche for people to even care about, let alone take an interest it correcting. And I don't think the market will take care of it, biglaw doesn't command their fees because of higher competence, plenty of firms could do the work the most elite firms do just as well.
Anyway, we'll see. Hope you're right and I'm wrong. |
|
Date: July 28th, 2026 6:57 PM Author: bat shit crazy boyish den ape interesting anti-lawyer screed. i think it's an interesting question how much salaries would go down if the cartel were eliminated. as is most students who want to go to law school can get loans for the whole thing. but yeah it costs 3 years of training and delaying gratification (for those of you who don't enjoy scholarship anyway).
i think getting that last mile on AI lawyers will be tough because how is it going to tell you the right supreme court answer on a politically loaded topic? we all know scalia saw the result first and then figured out how to get there afterwards. if AI says law is "solved" it will need to flatten some of these nebulous/grey areas - which is weird to think about. |
|
Date: July 28th, 2026 4:02 PM Author: Burgundy useless idiot You seem retarded. What makes you think the scores are normalized axroas fields idiot? (http://www.autoadmit.com/thread.php?thread_id=5887079&forum_id=2#50030285) |
|
Date: July 28th, 2026 4:11 PM Author: jet-lagged pearly native Well it can't be normalized because we're talking about different tasks. But it's hard for me to think of a legal task that's so much "harder" than those on Humanity's Last Exam that the difference in correct answers is attributable to something like the legal questions being "harder."
I think the correct answer is that there's much less data to train AIs on for legal work because the nature of the work means much of it is confidential. |
|
Date: July 28th, 2026 4:14 PM Author: Burgundy useless idiot Or it could be that harder than average problems are a bigger part of the legal benchmark's question mix than of the other benchmark. You have no idea. (http://www.autoadmit.com/thread.php?thread_id=5887079&forum_id=2#50030308) |
|
Date: July 28th, 2026 4:17 PM Author: jet-lagged pearly native Humanity's Last Exam is designed to extremely difficult. It's hard for me to even conceive of how a legal task could be harder than questions on that. (http://www.autoadmit.com/thread.php?thread_id=5887079&forum_id=2#50030313) |
|
Date: July 28th, 2026 4:21 PM Author: Onyx roast beef They are trained to beat humanity's last exam. They aren't trained to be lawyers
It really is that simple. The "benchmarks" are heavily gamed because there's enormous incentive to do so |
|
Date: July 28th, 2026 4:25 PM Author: jet-lagged pearly native Ok. But you're just explaining why the models perform worse or legal tasks than similar ones. But they still do perform worse currently, which was my point. (http://www.autoadmit.com/thread.php?thread_id=5887079&forum_id=2#50030329) |
|
Date: July 28th, 2026 4:16 PM Author: cracking flesh police squad tanning salon I, for one, am optimistic for the future of this majestic profession. (http://www.autoadmit.com/thread.php?thread_id=5887079&forum_id=2#50030311) |
|
Date: July 28th, 2026 4:23 PM Author: Angry National Security Agency Therapy I Philosophize, therefore I Poast (http://www.autoadmit.com/thread.php?thread_id=5887079&forum_id=2#50030324) |