Your AI Bias Audit Might Be Worthless — Regulators Just Proved It

Your AI hiring tool passed its bias audit. Congratulations — but here's the uncomfortable truth: that audit may not mean what you think it means. New York State auditors just reviewed how NYC enforces Local Law 144 — the landmark law requiring bias audits for AI hiring tools — and the findings are alarming. Three out of four complaints filed by New Yorkers never even reached the right agency. They got logged in the 311 system and disappeared. And when state auditors reviewed the same employers the city had cleared? They found 17 potential violations the city's own investigators had missed. But it gets worse. A major study from ACM FAccT analyzed 116 published LL 144 bias audits and found widespread methodological problems — incomplete data, opaque methods, even synthetic proxy data standing in for real applicants. Some tools that received clean audit reports were likely violating the four-fifths adverse impact rule all along. Meanwhile, the regulatory landscape is expanding fast. Illinois, Colorado, and the EU AI Act are all adding new obligations — with key deadlines hitting in 2026. For HR leaders, the audit-and-forget era is officially over.