Prompting for Legal and Medical Topics: The Hard Limits

TL;DR: If you searched AI legal medical prompts, you're asking a chatbot to do something its maker restricts. OpenAI, Anthropic, and Google all treat unsupervised legal and medical advice as a policy violation. HIPAA and UK GDPR separately limit what data you should type in. Route advice through a license-holder, and use AI only for the parts that don't require one.

Two kinds of AI usage get discussed on the same page, and they aren't the same problem. One is "will the chatbot refuse my prompt." The other is "even if it answers, should I rely on that for a legal filing or a medical decision." This post is about the second, and it has no clever-prompt workaround, only a real answer: what the providers' policies say, what regulators require, and what a professional's workflow does instead of asking a chatbot to stand in for a license.

What Do OpenAI, Anthropic, and Google Actually Say About Legal and Medical Prompts?

Not "we recommend caution." Each provider behind the models people actually use here (ChatGPT, Claude, Gemini) writes legal and medical use into its usage policy as a restricted or high-risk category, not a footnote.

OpenAI's Usage Policies (last updated 2025-10-29, read 3 Sep 2026) list, among prohibited uses, the "provision of tailored advice that requires a license, such as legal or medical advice, without appropriate involvement by a licensed professional". The same policy separately restricts automated decisions "in sensitive areas without human review", naming "legal" and "medical" explicitly on that list.

Anthropic's Usage Policy (effective September 15, 2025, read 3 Sep 2026) is more structured about it. It defines a category called "High-Risk Use Cases", and both Legal and Healthcare are named entries: "Legal: Use cases related to legal interpretation, legal guidance, or decisions with legal implications" and "Healthcare: Use cases related to healthcare decisions, medical diagnosis, patient care, therapy, mental health, or other medical guidance." (Anthropic carves out an exception: "Wellness advice (e.g., advice on sleep, stress, nutrition, exercise, etc.) does not fall under this category"). For every High-Risk Use Case, the policy requires that "a qualified professional in that field must review the content or decision prior to dissemination or finalization", plus disclosure to the end user "at a minimum at the beginning of each session."

Google's Generative AI Prohibited Use Policy (last modified December 17, 2024, read 3 Sep 2026, applies across Google's generative AI products) takes a slightly different angle: it doesn't ban legal or medical topics outright, but its restricted-content list includes anything that "Makes automated decisions that have a material detrimental impact on individual rights without human supervision in high-risk domains" (its own examples for that clause: employment, healthcare, finance, legal, housing, insurance, or social welfare), and, separately, "Facilitating misleading claims of expertise or capability in sensitive areas" (its own examples: health, finance, government services, or the law).

Three companies, three documents, one landing spot: legal and medical advice is the one category none of them will let a chatbot handle unsupervised. That's the actual hard limit; everything past this point is about why, and what a workflow that respects it looks like.

Can ChatGPT Give Legal Advice? What the Bar Actually Says

The American Bar Association answered this directly. ABA Formal Opinion 512 (July 29, 2024, Standing Committee on Ethics and Professional Responsibility) opens by stating that lawyers using generative AI tools have to weigh a full set of ethical duties at once: competent representation, protecting client information, communicating with clients, supervising staff who use these tools, advancing only meritorious arguments, staying candid with the court, and charging reasonable fees. This is US guidance for US-licensed lawyers, and it doesn't bind a self-represented person asking ChatGPT a question, but it tells you exactly what a professional must check before relying on the same output you'd get.

Two obligations matter most here, professional or not:

• Competence. The opinion is blunt about why: "Some GAI tools are also prone to “hallucinations,” providing ostensibly plausible responses that have no basis in fact or reality." That's not a hedge; it's the working definition the ABA uses for why hallucination is a professional-conduct problem and not just a UX quirk.
• Confidentiality. "A lawyer using GAI must be cognizant of the duty under Model Rule 1.6 to keep confidential all information relating to the representation of a client, regardless of its source, unless the client gives informed consent". If a licensed attorney can't paste a client's facts into a general-purpose chatbot without clearing that hurdle first, the same caution applies with more force to a non-lawyer typing in the details of their own dispute, custody case, or contract.

The case that made this concrete is Mata v. Avianca (S.D.N.Y., docket 1:22-cv-01461, Opinion and Order on Sanctions, June 22, 2023). An attorney used ChatGPT for legal research and it fabricated several cases complete with fake citations. What makes the case worth reading in full rather than trusting a summary of it: the attorney actually tried to verify the work, by asking ChatGPT itself. His own affidavit, quoted in the court's opinion, shows him asking "Is Varghese a real case" and "Are the other cases you provided fake". ChatGPT responded that it had supplied "real" authorities that could be found through Westlaw, LexisNexis, and the Federal Reporter, and when he pushed further, it told him Varghese "does indeed exist". It didn't. The court's own framing is worth quoting exactly, because it's more measured than most retellings: "there is nothing inherently improper about using a reliable artificial intelligence tool for assistance." The sanction wasn't for using AI. It was for never checking the output against an actual case reporter, and for standing behind it after being told it was fake.

The pattern to take from the legal side: the tool didn't just fail once, it failed to catch its own failure when asked directly. Verification has to happen against a real source, not the model that produced the claim.

Can AI Diagnose You? The Medical Hard Limits

Start with what actually reviews AI for medical use, because it explains why a chatbot answer and a cleared device aren't the same category of thing. The FDA maintains an AI-Enabled Medical Device List (page content current as of 06/16/2026, read 3 Sep 2026), described as identifying devices "authorized for marketing in the United States." Getting onto that list requires that a device has "met the FDA’s applicable premarket requirements, including a focused review of the device’s overall safety and effectiveness". General-purpose chatbots are not marketed as medical devices and are not on that list. That's not an oversight the maker forgot to fix; it's a different regulatory lane entirely, and it's the concrete reason "the AI sounded confident" isn't equivalent to "the AI was cleared for this."

This lines up with Anthropic's own policy, which puts "healthcare decisions, medical diagnosis, patient care, therapy, mental health, or other medical guidance" in the same High-Risk bucket as legal advice, requiring a licensed professional's review before any output reaches a patient. Note the jurisdiction split: Anthropic's policy is a private company's global usage terms, applying wherever its products are used; the FDA's device-clearance regime is specifically a US authority over marketed devices, not a claim about chatbots generally.

The honest medical hard limit is narrower than "don't ask AI about health": it's fine to ask a chatbot to explain a diagnosis code, summarize a condition in plain language, or help write down a symptom timeline before an appointment. It becomes unsafe the moment the output is used in place of an exam, a test result, or a clinician's judgment, which is the line both the FDA's device framework and Anthropic's policy draw from different directions.

Does HIPAA (or GDPR) Protect What You Type Into a Chatbot?

This is the part people get backwards most often, and the two jurisdictions aren't interchangeable.

In the United States, HIPAA's privacy rule has a narrower scope than most people assume. The regulation itself, at 45 CFR § 160.103, defines who it actually reaches: a "covered entity" is a health plan, a health care clearinghouse, or a health care provider who transmits health information electronically in connection with a covered transaction, and the rule extends to their "business associates" (contractors who handle that data on a covered entity's behalf). A consumer typing symptoms into a general-purpose AI chatbot on a personal account isn't, by itself, a HIPAA-covered exchange, because neither the person nor (typically) the chatbot provider is acting as a covered entity there. That doesn't mean the data is safe; it means the specific federal law people invoke here usually isn't doing any protecting. Where HIPAA does bite: a clinic or hospital piping patient records through an AI tool as part of treatment or billing, a business-associate relationship needing a signed agreement and the same safeguards as any other vendor touching protected health information.

In the United Kingdom, the calculus runs through a different instrument. Health information is a "special category" of personal data under the UK GDPR. The Information Commissioner's Office (ICO), the UK's data protection regulator, defines it directly: "‘data concerning health’ means personal data related to the physical or mental health of a natural person, including the provision of health care services, which reveal information about his or her health status" (UK GDPR Article 4(15), as quoted in ICO guidance, read 3 Sep 2026). Processing that category of data requires identifying a specific legal condition under Article 9 before you do it at all, a materially stricter starting point than most US context, and it applies to organizations, not to an individual asking about their own health. The EU's own GDPR runs on an equivalent Article 9 structure, but its primary text on eur-lex.europa.eu could not be fetched directly for this piece (it returns an unresolved script challenge); the ICO's UK GDPR guidance is cited here as the closest available regulator's-own-page equivalent, and the two frameworks diverge in enforcement detail.

The practical rule that survives both jurisdictions: treat anything you type into a general-purpose AI tool as if someone other than your doctor or lawyer could read it, because in the plain reading of both frameworks it usually isn't protected by the health-privacy law you assume covers it.

What Should You Actually Prompt AI to Do Instead?

Once the "advice" and "diagnosis" boxes are off the table, there's a genuinely useful, lower-risk band of work left, most of what people actually needed in the first place. A workflow that keeps the license-holder in charge looks like this:

1. Write down your own facts first, without the AI. A symptom timeline, or a plain sequence of what happened in a dispute. This part must stay accurate and shouldn't be generated for you.
2. Ask AI to turn that into organized questions, not conclusions. "Turn this timeline into a list of questions for my appointment" is a fundamentally different prompt than "what's wrong with me."
3. Ask it to explain terms, not interpret them for your situation. "Explain what this diagnosis code means in plain language" is safe; "does this diagnosis code apply to me" isn't.
4. Send the output to the professional before you act on anything. Not after. The ABA's competence and confidentiality duties, and Anthropic's human-in-the-loop rule, both assume review happens before dissemination, not as a retroactive check.
5. Never paste identifying details you wouldn't want on a third-party server. Names, case numbers, dates of birth, and full addresses aren't necessary for steps 2 or 3.

Here's a template that follows that shape, written to produce organizing output only, with no invented facts and no diagnosis or legal conclusion baked into the request:

Context: I'm preparing for a meeting with my [attorney / doctor] about [general topic, no identifying details].
I am NOT asking you to diagnose, advise on, or interpret my specific situation.

Please help me:
1. List the general questions someone in my position would want to bring to that meeting.
2. Explain these terms in plain language, without applying them to my case: [list terms].
3. Summarize the typical procedural steps or timeline for this kind of situation, in general terms.

Do not:
- Guess at a diagnosis, a legal conclusion, or an outcome.
- Cite specific case law, statutes, or medical studies unless I've pasted them in myself.
- Tell me what decision to make.

This is also where a well-organized reusable context (see how to give ChatGPT context without repeating yourself) earns its keep: a saved, non-sensitive template for "turn my notes into questions for my [professional]" means you're not rebuilding the safe version of the prompt from scratch, and you're not tempted to paste in more detail than the task needs. If you're comfortable with more advanced prompting, advanced ChatGPT prompting for people past the basics covers structuring a multi-step request like the template above without it drifting into a conclusion.

Where a Prompt Manager Fits, and Where It Doesn't

To be direct about the product angle, because a "hard limits" post should say plainly where a tool does and doesn't belong: Prompt Architects is a prompt manager and enhancer. It organizes and improves the wording of a prompt; it does not, and should not be asked to, generate legal or medical advice, and using it doesn't change any policy or regulatory limit described above. Where it's genuinely useful here is narrow: saving the "turn my notes into questions, don't diagnose or advise" template as a reusable prompt with variables for the topic and the terms list, so the safe version of the request is the default one you reach for, not something rebuilt under pressure at 11pm before an appointment. That's a workflow improvement, not a workaround, and the professional review is still the part doing the actual work.