AI is becoming an emotional confidant for millions of people. But have we considered what happens to the intimate information we’re sharing?
People are telling AI their deepest fears.
Their insecurities. Their relationship problems. The things they’re ashamed of. Thoughts they’ve never shared with another human being.
The entity listening is an artificial intelligence developed and operated by a technology company.
And depending on the service and your privacy settings, those conversations may be stored, processed, or used to improve its systems.
Why Are So Many People Turning to AI for Emotional Support?
In 2025, therapy and companionship ranked first in Marc Zao-Sanders’ analysis of generative AI use cases, published in Harvard Business Review.
This was an analysis of emerging use cases, but the finding is still revealing and it isn’t difficult to understand.
AI is available at 3 a.m. It doesn’t interrupt. It doesn’t appear impatient. It doesn’t tell you that you’re overthinking things.
And, most importantly, it doesn’t judge you.
For someone struggling with anxiety, loneliness, or difficult emotions, it can be very appealing.
A study published in the Journal of Public Health in January 2026 provides additional insight.
Researchers surveyed 500 Catalan-speaking participants aged 16 to 90. Of those surveyed, 17.2% reported having used ChatGPT for mental health support.
People aren’t just asking AI to write emails, summarize documents, or plan vacations. They’re asking it to help them understand themselves.
What Happens to the Things We Tell AI?
When we share something personal with a friend, we generally understand the relationship.
We know who we’re talking to. We have some understanding of how that person might use the information. We can decide whether to trust them.
With AI, the situation is considerably more complicated.
The friendly conversational interface can make us forget that we’re interacting with a commercial technology system.
Depending on the provider, product, and privacy settings, conversations may be:
Stored – conversations can remain on company servers, sometimes beyond the point at which users delete them from their visible history, subject to retention policies and legal requirements.
Used to improve AI models – Some consumer AI services use eligible conversations for model training or improvement, although users may have options to opt out.
Accessible to authorized personnel – Selected conversations may be reviewed for purposes such as safety investigations, abuse prevention, or quality assurance.
Processed to generate additional information – AI systems can potentially identify patterns and make inferences from what users disclose.
Subject to legal disclosure – Companies may be required to provide information in response to valid legal processes.
These practices vary significantly between providers.
Some providers also implement substantial privacy safeguards.
For example, OpenAI states that ChatGPT users can disable the use of new conversations for model training and use Temporary Chat for conversations they don’t want saved in their regular history.
But that doesn’t eliminate the broader question.
Do we truly understand what we’re agreeing to when we share our most intimate thoughts with an AI system?
And more importantly, should understanding complex data-processing policies be a prerequisite for receiving emotional support?
The Bigger Risk Isn’t What You Tell AI. It’s What AI Could Infer About You.
Imagine a woman who regularly uses an AI chatbot to discuss difficulties at work.
She’s successful, experienced, and generally confident in her abilities.
But she sometimes struggles with self-doubt.
She talks about feeling overlooked for promotions, worrying about disappointing her manager, and finding it difficult to say no to additional responsibilities.
She also discusses her relationships, her childhood experiences, and her fear of letting people down.
Individually, these conversations seem harmless.
But collectively, they reveal patterns she may not have even consciously recognized.
Perhaps she seeks reassurance before making important decisions.
Perhaps she’s particularly sensitive to criticism from authority figures.
A sufficiently capable system analyzing those conversations might infer aspects of her emotional vulnerabilities and decision-making patterns.
Now imagine that information being used to personalize commercial messages.
An advertisement for an expensive confidence-building program appears at exactly the moment she’s questioning her abilities.
A product is presented as the solution to an insecurity she never publicly acknowledged.
To be clear, this is a hypothetical scenario illustrating a potential risk. It is not evidence that AI chatbot providers are currently selling users’ psychological profiles to advertisers.
But the possibility deserves consideration.
AI can be a useful tool for reflection, but it shouldn’t be our only source of emotional support. There are also [free and accessible mental health resources] worth exploring, including options that offer human support.
When Consent Isn’t Really Informed
We’re all familiar with the ritual.
A new application asks us to accept its terms and conditions.
We click Agree.
Most of us haven’t even read the document.
And I say this as someone who has worked in technology for years.
Privacy policies can be lengthy, technical, and difficult to interpret, even for people familiar with data protection principles.
Now consider someone experiencing emotional distress.
Perhaps they’re going through a divorce, grieving a loss, or struggling with anxiety.
They open an AI chatbot because they need someone to talk to.
Is it reasonable to expect that person to carefully evaluate the platform’s data-retention policies, model-training practices, and potential long-term privacy implications before describing what they’re going through?
I don’t think so.
And while privacy laws don’t always rely on consent as the legal basis for processing personal information, the broader issue remains.
When people are emotionally vulnerable, meaningful privacy protection shouldn’t depend entirely on their ability to navigate complicated settings and legal documents.
The responsibility should also rest with the organizations collecting and processing the information.
Especially when that information concerns someone’s psychological wellbeing.
Are Existing Privacy Laws Enough?
In Europe, the General Data Protection Regulation (GDPR) provides important protections for personal information, including special categories of data such as information concerning health.
It establishes obligations around lawful processing, transparency, data minimization, security, and individuals’ rights.
The EU AI Act also introduces restrictions and obligations intended to address certain risks associated with artificial intelligence, including prohibited manipulative practices.
These are meaningful safeguards.
But having legal rights doesn’t automatically mean having practical control over how information about us is processed.
When someone shares sensitive information with a licensed psychotherapist, that relationship is generally subject to professional confidentiality obligations and relevant healthcare regulations.
It is not the same when someone shares information with a general-purpose AI chatbot.
Of course, the exact safeguards depend on the service, jurisdiction, and nature of the information being processed.
But this raises an important question:
If someone uses AI to discuss trauma, depression, relationship abuse, or other deeply personal experiences, should that information receive protections comparable to those governing clinical mental health records?
What Happens When Business Models Change?
There’s another aspect of this discussion that deserves more attention.
Trust is built in the present.
Data can remain valuable for years.
Someone may be sharing personal information with a technology company today because they trust the current privacy practices.
What happens in five or ten years?
The business model might evolve.
It could be acquired by another organization.
Regulations might change.
Of course, existing privacy laws and contractual commitments limit what companies can legitimately do with previously collected information.
A change in ownership doesn’t automatically give a company unrestricted rights to use personal data however it chooses.
But legal protections cannot eliminate every possibility of misuse, unauthorized access, or security breaches.
And the consequences of exposure aren’t equal for every type of information.
Having your shopping preferences exposed is unpleasant.
Having years of intimate conversations about your fears, relationships, mental health, and personal struggles exposed could be devastating.
The Human Connection Question
There’s something else about this trend that I find both fascinating and unsettling.
Why are so many people comfortable discussing their deepest fears with artificial intelligence?
Convenience and affordability are obvious explanations.
But perhaps AI offers something more:
The feeling of being heard.
Without interruption.
Without judgment.
Without worrying that they’re becoming a burden.
We shouldn’t dismiss the potential value of that experience.
AI can help people organize their thoughts, consider different perspectives, and reflect on difficult emotions.
For some, it may even provide the encouragement needed to seek professional support.
But I also wonder what it says about our society that an artificial system can feel like a safer place for emotional disclosure than another human being.
Are people choosing AI because they prefer it?
Or because meaningful human connection has become increasingly difficult to find?
And if AI becomes a primary source of emotional support, what happens to our willingness or ability to develop those connections in the real world?
We don’t yet have definitive answers.
But I believe these questions deserve as much attention as the technological capabilities themselves.
How Can You Protect Your Privacy When Using AI for Emotional Support?
People won’t stop using AI for personal reflection, and I don’t think they should.
But we need to approach these conversations with greater awareness.
Here are a few practical precautions worth considering.
1. Review your data controls.
Check whether your AI provider uses conversations for model training and whether you can disable that option.
2. Understand the difference between temporary and regular conversations.
Some services offer temporary conversations that aren’t retained in your regular history or used for model training. However, temporary doesn’t necessarily mean that no information is retained or processed.
3. Avoid unnecessary identifying details.
You can explore difficult situations without always providing full names, addresses, workplaces, or other information that identifies you or someone else.
4. Be mindful of information about other people.
When discussing relationships or workplace conflicts, remember that you’re potentially sharing sensitive information about individuals who haven’t agreed to participate in the conversation.
5. Understand what deleting a conversation actually means.
Review your provider’s retention and deletion policies. Removing something from your chat history doesn’t necessarily mean every copy disappears immediately.
6. Don’t confuse emotional support with professional confidentiality.
A general-purpose AI chatbot may be useful for reflection, but it shouldn’t be treated as equivalent to a licensed mental health professional operating under professional confidentiality obligations.
None of these precautions guarantees complete privacy.
But they help reduce unnecessary exposure.
Sources and Further Reading
Research
- Marc Zao-Sanders (2025). How People Are Really Using Gen AI in 2025. Harvard Business Review.
Read the HBR analysis - Díaz, L., & Huguet, A. (2026). Spontaneous use of ChatGPT for mental health support: an exploratory study. Journal of Public Health.
Read the research study
Privacy and Regulation
- European Commission. Data Protection in the European Union.
European Commission: Data Protection - European Commission. AI Act: Regulatory Framework for Artificial Intelligence.
EU AI Act information - OpenAI. Data Controls in ChatGPT.
ChatGPT privacy and data controls
Leave a Reply