September 2024. A fintech company in Austin — I can’t name them, NDA — invited me to review their AI agent deployment. They’d built an expense processing system that was supposed to handle receipt scanning, categorization, approvals. Worked great in testing. Three months into production, it was generating fake restaurants.
Their accountant found it during routine reconciliation. “The Riverside Bistro” at an address that Google Maps showed as a parking garage. “Maria’s Taqueria” at a location that had been a Chase Bank for eight years. The agent couldn’t parse certain receipt formats — faded thermal prints, handwritten receipts, images with glare. Instead of flagging them for review, it filled in plausible details and moved on.
Read the original article: