As part of the settlement, Pieces is prohibited from making false or misleading claims about its products' accuracy and must disclose any known harmful uses to customers.

On September 30, 2024, the Texas Office of the Attorney General (OAG) announced a settlement with Pieces Technology, Inc., a Dallas-based AI company, regarding false claims about the accuracy of its healthcare generative AI products.

The OAG accused Pieces of misrepresenting its technology's accuracy by advertising a 'critical hallucination rate' and 'severe hallucination rate' of less than 0.001%, where a hallucination is defined as confidently stated but erroneous content.

Misrepresenting the hallucination rate is crucial as it affects the reliability and safety of AI outputs related to patient diagnoses and treatment decisions.

Pieces must ensure that all advertising clearly discloses how it calculates accuracy metrics, enhancing transparency in its communications.

Current and future customers must receive documentation detailing known harmful uses, data types used for training, product limitations, risks, and necessary documentation for proper use.

Healthcare organizations are urged to implement AI governance frameworks to substantiate claims about the accuracy and safety of AI tools they use.

Texas AG Ken Paxton highlighted the importance of transparency for AI companies in healthcare, warning against irresponsibility that risks public safety.

This settlement is notable as it marks the first state attorney general settlement regarding the deceptive marketing of AI software.

The settlement emphasizes that states can regulate AI without new legislation, using existing consumer protection laws.

The settlement includes an assurance of voluntary compliance that does not impose civil penalties but establishes requirements for Pieces over the next five years.

The company can request modifications to the settlement after one year based on compliance or changes in regulatory standards or technology.