The California bar exam is a multi-year, multi-agency saga, and stories regularly appear in the local news. My standard caveat with anything about the law is that I'm not an attorney. Over the time period in question, the %pass rates declined from 62% to 44%. I have no doubt (OK, maybe a tiny doubt) that one day, there will be a Broadway musical or a Netflix made-for-TV movie about the exam, exam takers, the California Supreme Court, and numerous other exam stakeholders and "Influencers." Maybe Brad Pitt will star as the statistician/psychometrician at the heart of the saga. The story goes back to at least 2016, when the California bar and the California Supreme Court were responsible for changing the exam cut score (among their many responsibilities). For the scorekeepers, the California bar decided that bar exam takers' scores had declined. They recommended that the California Supreme Court change the "cut score". The Supreme Court has that authority but refused the change.
In 2016, due to declining pass rates, the state bar investigated revising the cut score.
https://www.calbar.ca.gov/Admissions/Examinations/California-Bar-Examination/California-Bar-Examination-Studies-Fact-Sheet2
excerpting "This trend raised the question of whether the current pass line, or cut score, of 1440 is appropriate". IN 2017, the California Supreme Court got involved because they are the highest authority about the bar exam, and it ordered at least three studies. Those studies identified changes in demographics as one of the contributing factors to the decline
Excerpting
...A historical analysis of California Bar Exam pass rate trends from 2008, 2012, and 2016, which suggested that downward shifts in law school applicant performance are a contributing factor in declining bar exam pass rates….
Somewhere there is an explanation why the years 2009, -10, -11, and -13 through 15 were not considered.
In 2018, the state bar appears to have used regression to evaluate whether changing demographics played a role in the declining scores
https://www.calbar.ca.gov/Portals/0/documents/admissions/Examinations/Bar-Exam-Report-Final.pdf
The California Supreme Court ordered that psychometricians and other experts get involved. Two independent Psychometricians, Bolus and Buckendahl, got involved as early as 2017. Buckendahl, in particular, evaluated the "content validity." What's content validity? The FDA requires pharmaceutical sponsors to evaluate content validity when developing PROs (Patient Reported Outcomes) for use as a primary endpoint in pharmaceutical clinical trials. Some digging and the Educational Testing Service evaluates the content validity of the SAT's, etc. Caveat emptor, with oversimplification, "content validity" can be thought of as "does this test measure what we think it measures-and how can we be sure of that?"
https://www.calbar.ca.gov/Portals/0/documents/reports/2017-Final-Bar-Exam-Report.pdf
Buckendahl is a psychometrician on the payroll of the California supreme court.
The findings of that report
https://www.calbar.ca.gov/Admissions/Examinations/California-Bar-Examination/California-Bar-Examination-Studies
Then, in 2025, there was the now nearly legendary California "meltdown," aka "the disaster" of the Bar examination. The students had been well trained and immediately filed class action suits
https://iclg.com/news/22339-california-bar-exam-disaster-sparks-class-action-lawsuit
The psychometrician appears to be the one who used the AI
https://dailyjournal.com/articles/385120-california-bar-exam-plunges-to-new-low-amid-scandal
------------This recent news may only be the 'eye of the California Bar Exam hurricane." ----------
------------------------------
Chris Barker, Ph.D.
Adjunct Associate Professor of Biostatistics
www.barkerstats.com---
"In composition you have all the time you want to decide what to say in 15 seconds, in improvisation you have 15 seconds."
-Steve Lacy
------------------------------
Original Message:
Sent: 05-02-2025 08:52
From: Ralph O'Brien
Subject: ... not even Super Statistician can fix it.
There is a story in the 4/30/25 NY Times by Orlando Mayorquín about how California's much-revised bar exam given last February was compromised by both the poor quality and bizarre nature of numerous questions (some written by AI) and by the computing technologies developed to administer and score the exam itself.
I found this paragraph amusing (emphasis added):
The state bar added that it would evaluate whether Meazure Learning, the vendor that provided the technology and proctoring services to administer the exam, had failed to meet its contractual obligations. It also said it would enlist a psychometrician - a specialist who focuses on measuring intangible qualities such as knowledge or intelligence - to come up with score adjustments for test-takers who had experienced difficulties.
Of course, statisticians of all stripes know that measurement is a cornerstone of our work, and if that becomes chaotic, we have no wizardry to truly fix it.
------------------------------
Ralph O'Brien
Professor of Biostatistics (officially retired; still keenly active)
Case Western Reserve University
http://rfuncs.weebly.com/about-ralph-obrien.html
------------------------------