Judge scolds state over AI errors in ‘deepfakes’ lawsuit

In a case concerning a challenge to a Minnesota law regulating election-related “deepfake” audios and videos, a U.S. District Court judge has excluded an expert legal submission for the state that cited AI-generated references that did not exist.

“The Court thus adds its voice to a growing chorus of courts around the country declaring the same message: Verify AI-generated content in legal submissions!” implored U.S. District Judge Laura Provinzino, who has chambers in St. Paul.

The Hamilton Lincoln Law Institute and Upper Midwest Law Center are representing plaintiffs Christoper Kohls and Mary Franson in a lawsuit challenging a state law prohibiting the dissemination of “deepfakes” when there is intent to injure a political candidate or influence the result of an election, alleging that the ban violates the First Amendment and is unconstitutionally vague.

The plaintiffs moved for a preliminary injunction. The state responded by submitting two expert declarations, one of which was authored by Jeff Hancock, a professor of communication at Stanford University and director of the Stanford Social Media Lab. The plaintiffs moved to exclude the declaration, claiming that it included fabricated material.

Specifically, a 2023 study titled “The Influence of Deepfake Videos on Political Attitudes and Behavior” featured in the Journal of Information Technology & Politics was cited. However, the study did not exist.

UCLA law professor Eugene Volokh also determined that another citation, “Deepfakes and the Illusion of Authenticity: Cognitive Processes Behind Misinformation Acceptance. Computers in Human Behavior,” also did not exist.

In November 2024, the plaintiffs suggested in their memorandum to exclude Hancock’s declarations that, “Likely, the study was a ‘hallucination’ generated by an AI large language model like ChatGPT.”

Attorney General Keith Ellison subsequently contacted Hancock, who admitted that the declaration inadvertently cited authors of a third article and included citations to two nonexistent academic articles. The errors arose from Hancock’s use of GPT-4o, a generative AI tool, when he drafted his declaration. Hancock failed to verify the fake citations before including those.

Ellison acknowledged to the court that the citations were fake upon becoming aware of it. Hancock maintained that, despite the fake citations, he stood by the substantive propositions in his declaration. Before the court was the question of whether to consider the declaration when deciding the motion for a preliminary injunction.

“The Court takes Attorney General Ellison at his word and appreciates his candor in rectifying the issue,” Provinzino wrote. But this was not enough to get Provinzino to consider Hancock’s declaration when deciding the preliminary injunction.

“The irony,” Provinzino wrote. “Professor Hancock, a credentialed expert on the dangers of AI and misinformation, has fallen victim to the siren call of relying too heavily on AI — in a case that revolves around the dangers of AI, no less.”

“It is particularly troubling to the Court that Professor Hancock typically validates citations with a reference software when he writes academic articles but did not do so when submitting the Hancock Declaration as part of Minnesota’s legal filing,” wrote Provinzino. “One would expect that greater attention would be paid to a document submitted under penalty of perjury than academic articles. Indeed, the Court would expect greater diligence from attorneys, let alone an expert in AI misinformation at one of the country’s most renowned academic institutions.”

Provinzino concluded that Hancock’s citation to AI-generated, nonexistent sources “shatters his credibility.”

“At a minimum, expert testimony is supposed to be reliable,” Provinzino wrote. “The Court should be able to trust the ‘indicia of truthfulness’ that declarations made under penalty of perjury carry, but that trust was broken here.”

“[A]t the end of the day, even if the errors were an innocent mistake, and even if the propositions are substantively accurate, the fact remains that Professor Hancock submitted a declaration made under penalty of perjury with fake citation,” Provinzino emphasized.

“Attorney General Ellison’s attorneys are reminded that Federal Rule of Civil Procedure 11 imposes a ‘personal, nondelegable responsibility’ to ‘validate the truth and legal reasonableness of the papers filed’ in an action,” Provinzino added.

While Provinzino did not consider Hancock’s declaration in deciding the motion for a preliminary injunction against the deepfakes law, she did ultimately deny the motion, finding that Kohls “lacks standing to bring this pre-enforcement challenge” and that Franson “fails to demonstrate that she will be irreparably harmed absent injunctive relief.”

Provinzino did not completely reject the use of AI in the legal profession.

“To be clear, the Court does not fault Professor Hancock for using AI for research purposes. AI, in many ways, has the potential to revolutionize legal practice for the better,” Provinzino wrote. “But when attorneys and experts abdicate their independent judgment and critical thinking skills in favor of ready-made, AI-generated answers, the quality of our legal profession and the Court’s decisional process suffer.”