A new health-focused version of a popular chatbot is now available to all users over 18, including those on free plans. This development raises concerns due to a significant downgrade in privacy protections compared to the original health-focused setup. The chatbot allows users to connect their medical records or health data, and while it is not intended to replace a doctor, it can provide advice or help interpret medical data.
Despite claims that the chatbot will ask for permission to access health data and that conversations will be kept separate, the terms have shifted. Connected health information will not be used to train AI, but only conversations that use this information will be considered protected health conversations. This means that users must share their health data to receive privacy protections, as conversations without connected data will not be protected.
The company behind the chatbot claims it outperforms doctors in giving medical advice, citing its HealthBench evaluations. However, these evaluations have limitations, including the use of carefully constructed questions and the omission of real-world conversation complexities. The tests are also limited to short conversations, which may not reflect real-life interactions. As a result, the chatbot's responses may not be reliable or trustworthy.
Previous studies have shown that language models like the one used in the chatbot do not provide reliable medical advice. The HealthBench framework has been criticized for its limitations, including the lack of rare diseases in the testing material and sloppy rubrics. The company's own example of a rubric appears to award double points for similar responses, which raises questions about the validity of the results.
Using chatbots for medical advice can have real-world consequences, including harm or even death. There have been lawsuits against the company alleging that its products contributed to medical problems and suicide. A medical safety nonprofit has named the misuse of AI chatbots as the number one health technology hazard of 2026. As a result, users should exercise caution when using the chatbot for health advice.
If users still want to use the chatbot, there are some steps they can take to protect their privacy. These include using temporary chats, reviewing and editing imported health data, and disconnecting health information at any time. Users can also set the chatbot to ask for permission to access health records every time, rather than granting blanket permission. However, users should be aware that the company's safeguards are still being tested and may not catch everything.
