The Algorithm May Not See You

For millions of Americans struggling to afford health care, a new source of medical advice is only a few keystrokes away.

As use of AI tools like ChatGPT, Claude, Google Gemini and others become more widespread, national surveys suggest that uninsured people — particularly Black Americans and young adults — are among the groups most likely to use them for physical health information and mental health guidance. 

Some users appear to be turning to AI because they can’t afford a doctor’s visit, can’t schedule an appointment, or lack a regular health care provider. While instant, free advice is hard to resist, researchers and patient advocates warn that AI can produce inaccurate, misleading, or potentially dangerous recommendations. 

The stakes are even higher for Black patients: researchers have consistently found that chatbots often echo racial biases, generalizations and stereotypes. Using one for medical advice, experts say, could reproduce the same problems that plague Black Americans in healthcare spaces. 

Higher Use Among the Young and the Uninsured

A new study, published in JAMA Pediatrics earlier this month, illustrates the trend. It found that Black youth are around five times more likely than white youth to seek mental health advice from a chatbot at least once a month. This seems to be a snapshot of a larger trend: Black and Hispanic adults are also more likely to use AI for mental health information compared to white adults and people who have health insurance, according to KFF Health poll results published in March.

The findings come amid what is widely seen as a mental health crisis among Black people, including suicide rates that have risen sharply over the last two decades.

Kamal Grewal, a health technology entrepreneur, also worries that an AI therapist can give a young person in crisis the illusion of support, even though it potentially does more harm than good.  Several major AI companies, including OpenAI and Google, are fighting lawsuits alleging their chatbots contributed to or encouraged user suicides. 

“Consumer AI tools are optimized to keep users engaged: they validate, they agree, they make you feel heard,” says Grewal, creator of Therapy Companion, an AI tool designed to help therapists with non-patient services. “That’s great for retention metrics, but dangerous for someone in a mental health crisis, where what you need isn’t validation but a trained clinician who can challenge your thinking and hold you accountable,” 

It’s no surprise that AI use is higher among younger people and is gaining traction among adults. 

In fact, roughly 1 in 3 adults responding to a nationwide poll said they used AI chatbots last year to get health information. Of those who said they used AI for health information, one-third were looking for advice about physical health matters and just over 15% searched for mental health advice. 

Most respondents to the poll said they were looking for immediate advice. But there are indications that problems affording or finding health care also played a role for a slice of the respondents, especially for younger adults.

The RAND study published in JAMA estimates that roughly 8.2 million young people nationwide reported using artificial intelligence for mental health advice this year, an increase of 40% compared to last year. That’s a major jump from the findings of a similar RAND survey conducted a year earlier.

The study sampled 1009 youth, of whom 95 were Black. But the study’s authors noted the sample size for this analysis was small, and the issue needs more investigation. Nevertheless, when it came to using AI chatbots for mental health advice, Black youth were over five times more likely than white youth to seek advice at least once a month.

The RAND study finds that among youth who asked AI chatbots for mental health advice, 42.8% did so at least once a month and a whopping 91.7% said the advice ranged from “somewhat” to “very” helpful. Equally noteworthy: more than 2 of 3 adolescents said they hadn’t told anyone they used AI chatbots for mental health support. 

Chatbots used include ChatGPT, Gemini, Character.AI, and Meta AI. Young people tended to consult them when they felt angry, nervous, sad, or stressed.

Bias Hidden in the Code

A landmark 2024 study published in Nature magazine found that when large language models were given prompts written in African American English, they produced stereotypes as negative as — or worse than — those recorded from humans during the Jim Crow era, without the users’ race even being mentioned. 

The AI assigned the AAE speakers’ information for lower-prestige jobs, convicted them more often in hypothetical criminal cases, and more frequently recommended the death penalty when that punishment was an option.

Sharese King, assistant professor of Linguistics at the University of Chicago, said the data did not make her optimistic. 

“If we continue to ignore the field of AI as a space where racism can emerge, then we’ll continue to perpetuate the stereotypes and the harms against African Americans, ” she said.

The implications for teen mental health are serious since a large number of Black youth communicate in some form of AAE. If the chatbot they’re trusting to advise them about their depression, anxiety, or a crisis is unknowingly penalizing them because of how they speak, the “help” being offered could be biased against them.

Among Black youth, the use of AI to support mental health breaks along gender lines. 

Black teenage girls are more likely to use a chatbot to deal with anxiety, depression, and relationship stress. By comparison, Black teenage boys face stronger cultural pressure against vulnerability and are less likely to disclose if they use AI. But researchers say when they do turn to a chatbot, they are often already in acute distress.

Grewal of Therapy Companion says the overreliance on AI for mental health is a worrisome sign among Black people in general. 

“Black adults are already turning to these tools at nearly double the rate of white adults, which means the people most underserved by the mental health system are also the most exposed to AI that isn’t built to help them,” he says.

The post The Algorithm May Not See You appeared first on Word In Black.