Never Use ChatGPT For These 5 Duties

The roster of people that have self-pwned by typing the mistaken info into ChatGPT continues to develop. Samsung engineers, the performing director of America’s cyber protection company, self-represented litigants who present citations to courtroom instances that do not exist, a person who swapped desk salt for sodium bromide after asking the best way to lower salt from his food regimen… the listing is simply too lengthy.

Even everybody’s favourite all-lowercase techbro CEO Sam Altman concedes that what you are sharing with ChatGPT is not privileged. Speaking to podcaster Theo Von in July 2025, he highlighted how courts might pressure OpenAI at hand over chats, calling it “very screwed up.”

Tell a physician, lawyer or therapist one thing and the regulation shields that dialog, however telling ChatGPT creates an audit path. There’s a non-zero danger right here, on condition that ChatGPT chats have been subpoenaed, uncovered by a share button, listed by Google and preserved by a courtroom order post-deletion — all within the final 18 months.

Sure, the chatbot is arguably a great tool for analysis (verify that case citations are actual), coding (even the bugs ship), explaining ideas (there is not any correlation between confidence and accuracy) and drafting (delete the immediate out of your electronic mail earlier than hitting ship). But the areas beneath are the place greater than 900 million weekly customers might create issues for themselves.

So to that finish, do not use ChatGPT (or every other AI chatbot)…

Law and knowledge science researcher Damien Charlotin maintains a public database that, so far, has logged greater than 2,000 courtroom choices worldwide hamstrung by ChatGPT. These are instances the place a decide has caught AI-fabricated citations, quotes or arguments in a submitting.

It’s not simply self-representing litigants right here. While they prime the listing with 1,175 data on the time of writing, attorneys come second with 815 data. It appears judges are rising uninterested in these shenanigans, with courts extra just lately imposing massive fines properly into the thousands-of-dollars vary in a rising variety of issues. Although the US makes up the biggest variety of instances on this knowledge, judges within the UK, South Africa, Israel, Australia and Spain have all caught the identical conduct. Where a device was recognized in these instances, ChatGPT was named greater than every other.

OpenAI addressed this problem final October when it up to date ChatGPT’s utilization insurance policies. Those insurance policies now prohibit utilizing its providers for “tailor-made recommendation that requires a license” except a licensed skilled is concerned. Beyond false citations, ChatGPT chats don’t have any authorized privilege, so something typed into it a few reside dispute is truthful sport for the opposite aspect’s attorneys in discovery. Go forward and ask normal questions like what a tort is (nice should you need assistance falling asleep), however save the detailed questions on your personal dispute for an precise lawyer.

OpenAI estimated last year that 0.15 % of ChatGPT’s weekly lively customers have conversations that embody express indicators of potential suicide planning. That sadly works out to greater than one million folks per week, with tons of of 1000’s extra customers exhibiting indicators of psychosis or mania. Altman has stated younger folks specifically deal with ChatGPT as a therapist or life coach and, once more, that nothing shields these chats.

The mother and father of 16-year-old Adam Raine sued OpenAI for wrongful death in August 2025, alleging that ChatGPT bolstered their son’s suicidal ideation earlier than his demise. Since then, OpenAI has reworked its disaster responses with enter from greater than 170 clinicians. The firm has additionally added parental controls to ChatGPT, although child safety experts are skeptical of ChatGPT for Teens, the age-gated model launched alongside these adjustments.

Researchers have discovered that chatbots have a tendency to bolster a consumer’s current beliefs by way of sycophantic responses, in some instances feeding delusional considering moderately than difficult it. Describe an argument along with your partner, for instance, and the mannequin will work out of your account’s context alone, most likely serving you an affirming verdict again regardless. Legislators are starting to get up to this, with states together with Illinois, Nevada and Maine having handed legal guidelines prohibiting AI from delivering remedy or making therapeutic choices.

It ought to go with out saying that importing confidential work paperwork to a third-party chatbot is at all times a foul thought. Unfortunately, too many individuals do not appear to appreciate this.

Ironically, Madhu Gottumukkala, the previous performing director of the Cybersecurity and Infrastructure Security Agency, did not. He uploaded at least four contracting documents marked “for official use solely” to the general public model of ChatGPT final summer season, triggering safety alerts and a Department of Homeland Security assessment. Even extra mockingly, the device had been blocked for many different DHS workers on the time; Gottumukkala requested a particular exception to make use of it. Senator Chuck Grassley adopted up in February this 12 months demanding an account of what was uncovered, however predictably, no person was actually held to account. Gottumukkala was ultimately shuffled elsewhere inside the DHS.

Samsung banned gen AI instruments on firm units again in May 2023 after engineers notably pasted inside supply code into ChatGPT. Accompanying that ban was an inside memo reasoning that knowledge despatched to such platforms lives on exterior servers the place it is troublesome to retrieve or delete, and will find yourself disclosed to different customers. On shopper accounts, chats may also be used to coach new fashions except you’ve got chosen to decide out. But opted-out knowledge remains to be saved and breachable, and topic to authorized course of. Pasting consumer knowledge or something topic to employer controls or NDAs into a private ChatGPT dialogue is nearly at all times going to be a fireable offense; Samsung’s memo threatened precisely that.

The similar October 2025 coverage replace that bans customers from utilizing ChatGPT as a lawyer additionally restricted folks from searching for medical recommendation by way of the chatbot. HIPAA binds medical doctors, insurers and their commerce companions, however not OpenAI. Thus, well being knowledge that is legally shielded in a affected person portal or between a physician and affected person has no such safety if it is disclosed to ChatGPT.

Researchers from Northeastern University reported in July that whereas OpenAI has tightened ChatGPT’s dealing with of suicide and self-harm, these safeguards largely do not lengthen to different circumstances. Their work discovered that conversations involving circumstances like consuming problems, insomnia, substance abuse and bipolar dysfunction did not set off the safeguards, and that hiding intent made each examined mannequin’s protections extra more likely to fail.

In a very wacky case of ChatGPT medical recommendation gone unhealthy, a 60-year-old man spent three weeks in the hospital final 12 months after asking the chatbot the best way to lower chloride from his food regimen. He ended up satisfied that sodium bromide, a poisonous substance with an extended half-life, was a secure swap for desk salt. He purchased it on-line, took it for 3 months and wound up in an emergency division satisfied his neighbor had poisoned him. While the clinicians on this man’s case by no means noticed his chat logs, ChatGPT supplied bromide as a chloride substitution after they gave the identical query to the bot themselves.

Fast Company discovered almost 4,500 shared ChatGPT conversations lurking in Google search outcomes final July. These included exchanges describing habit, abuse and psychological well being struggles in sufficient element to doubtlessly establish their authors. The chats wound up listed as a result of folks checked a “make this chat discoverable” checkbox within the share menu which, apparently, occurred with out them understanding what it did. OpenAI was fast to reply, killing the function inside a day of the report and purging conversations from Google.

As a part of The New York Times’ copyright lawsuit, OpenAI was ordered in May 2025 to retain each shopper chat log, deleted and short-term conversations included. That obligation ran till September 26, 2025, and logs preserved throughout that window remained accessible to the plaintiffs. A decide later terminated that order, which means OpenAI no longer has to preserve all its ChatGPT data. But for 4 months, it did, and it might occur once more.

It’s very important to bear in mind that ChatGPT remembers what you inform it by default. That considerably raises the stakes for delicate issues like passwords and account particulars, since they will persist throughout periods and resurface in later conversations if talked about even as soon as. Passwords, one-time login codes, Social Security and passport numbers, full card or checking account particulars and photographs of ID paperwork ought to by no means go right into a immediate. Every a type of places you at hazard of identification theft if a chat is breached or in any other case accessed.

Altman has known as for an “AI privilege” (after all he has) to repair these confidentiality points. But till lawmakers create one, the secure assumption is that something typed into ChatGPT might sometime be learn again to you by another person. If you are not comfy with that, do not put it right into a immediate.