Anabelle Colaco
03 Nov 2025, 12:34 GMT+10
NEW YORK CITY, New York: Judges around the world are grappling with an unexpected new problem: legal filings drafted with the help of artificial intelligence that cite cases that don't exist.
The rise of these flawed documents, riddled with "hallucinations", fabricated facts, or case law produced by AI tools, is becoming an urgent concern for courts, lawyers, and professionals across industries experimenting with the technology at work.
Damien Charlotin, a French data scientist and lawyer who studies the intersection of technology and law, has cataloged nearly 500 such filings globally over the past six months. Many of them were submitted in U.S. courts by self-represented litigants, but some came from practicing attorneys and even major companies. "Even the more sophisticated player can have an issue with this," said Charlotin, a senior research fellow at HEC Paris. "AI can be a boon. It's wonderful, but also there are these pitfalls."
Charlotin's database tracks cases where judges determined that lawyers or litigants had relied on generative AI systems that produced non-existent citations, inaccurate summaries, or misleading arguments. In most instances, judges issued warnings, but in some cases, they imposed fines or sanctions. In one notable example, a federal judge in Colorado ruled that a lawyer for MyPillow Inc. had submitted a brief containing nearly 30 defective citations in a defamation case involving the company's founder, Michael Lindell.
The problem reflects a larger issue: the growing use of AI tools at work, often without clear guidelines or sufficient oversight. Across professions, people are turning to chatbots and digital assistants to speed up research, draft documents, or brainstorm ideas. But as the legal world is learning, AI's confident tone can mask serious errors — and in high-stakes situations, those mistakes can have real consequences.
Maria Flynn, CEO of the nonprofit Jobs for the Future, said AI should be treated like a junior assistant, not a replacement for human judgment. "Think about AI as augmenting your workflow," she said. Flynn often uses her organization's in-house AI system to help prepare for meetings or develop discussion questions from articles she shares with her team. "Some of the questions it proposed weren't the right context for our organization," she said. "I was able to give it feedback, and it came back with better, more thoughtful questions."
Still, Flynn cautions that AI's output can be unreliable. When she asked the system to compile a list of her organization's past projects, it mixed up completed work with pending proposals. "In that case, our AI tool wasn't able to identify the difference between something that had been proposed and something that had been completed," she said. She caught the error only because she knew the background. "If you're new in an organization, ask coworkers if the results look accurate," she suggested.
Legal experts say the most significant risk is complacency. "People assume because it sounds so plausible that it's right," said Justin Daniels, an Atlanta-based attorney at Baker Donelson. "Having to go back and check all the cites, or reread the contract an AI summarized, that's inconvenient and time-consuming. But as much as you think the AI can substitute for that, it can't."
Another emerging concern is privacy. Many AI systems collect or store user inputs to improve future responses, meaning sensitive data can inadvertently be exposed. Flynn warned against uploading confidential material such as client information, proprietary data, or unreleased financials into public AI platforms. "It doesn't discern whether something is public or private," she said. "Once you've entered that information, it could resurface in someone else's results."
Even simple workplace applications of AI can carry legal risks. Many employees use AI-powered note-taking tools that record meetings or generate summaries, but in some jurisdictions, recording without consent is illegal. "Before using AI notetakers, pause and consider whether the conversation should remain privileged and confidential," said Danielle Kays, a Chicago-based partner at Fisher Phillips. She recommends checking with legal or human resources departments before deploying such tools in sensitive discussions, such as performance reviews or legal investigations.
Despite these risks, experts emphasize that avoiding AI entirely is not the answer. As more organizations integrate the technology into daily operations, understanding its strengths and weaknesses is becoming a critical skill. "The largest potential pitfall in learning to use AI is not learning to use it at all," Flynn said. "We're all going to need to become fluent in AI, and taking the early steps of building your familiarity, your literacy, your comfort with the tool is going to be critically important."
Get a daily dose of Milwaukee Sun news through our daily email, its complimentary and keeps you fully up to date with world and business news as well.
Publish news of your business, community or sports group, personnel appointments, major event and more by submitting a news release to Milwaukee Sun.
More InformationKYIV, Ukraine: A wave of Russian missiles and drones hit Kyiv and around a dozen Ukrainian regions on the night of October 6, killing...
JERUSALEM, Israel: Three years after Hamas attacked Israel on October 7, 2023, Israelis marked the anniversary with vigils and memorial...
PARIS, France: French high school students blocked schools again on October 6 as a nationwide protest movement demanding more resources...
TEL AVIV, Israel: Three years after Hamas' October 7 attack, at least nine Israelis from families directly affected by the assault...
SYDNEY, Australia: Australia's conservative opposition coalition has proposed major immigration cuts, targeting temporary and humanitarian...
RIYADH, Saudi Arabia: Saudi-backed Yemeni government forces have taken effective control of the Bab el-Mandeb Strait, a key route for...
WASHINGTON, D.C.: A sharp slowdown in U.S. hiring in September and hefty downward revisions to previous months pushed the prospect...
WASHINGTON, D.C.: U.S. consumer spending surged in August while inflation rose less than economists expected, a combination that could...
(Photo credit: David Frerker) After an impressive postseason, San Diego Padres right-hander Michael King faces a decision that could...
(Photo credit: Matt Marton) Chicago Bears running back D'Andre Swift was a limited participant in practice Thursday, one day after...
(Photo credit: Stephen Lew) Three weeks ago, fans and media members alike scoffed at the fact that the national audience would be...
(Photo credit: Wendell Cruz) Rookie Victor Mesa Jr. belted a tiebreaking two-run homer in the sixth inning and the Tampa Bay Rays...
