Why ChatGPT Should Never Decide a Child's Future

This article warns against using AI systems like ChatGPT as decision-makers in child welfare agencies. It outlines the structural risks of AI hallucinating details, encoding biases, and lacking accountability when making life-altering decisions for children.

Details

The article explains that large language models (LLMs) like ChatGPT generate the most likely next word, not verified facts. They can produce different, sometimes fabricated responses to the same prompt, hallucinate details, omit context, and contradict themselves - even with safeguards in place. In the child welfare system, an AI that invents a risk indicator or misstates legal thresholds can have serious consequences. Real-world examples show how 'non-malicious' AI errors have contributed to wrongful arrests and given incorrect medical/financial guidance. When these issues arise in child protection, the liability and accountability become diffuse across the AI development and deployment chain. Privacy risks are also a concern, as pasting sensitive family histories into public chatbots can expose highly confidential information. Bias is intrinsic, as LLMs inherit biases from their training data and can amplify discrimination against over-surveilled communities. Overall, the article argues that a system prone to hallucination, drift, and bias is structurally incompatible with being treated as a decision-maker in child welfare.