AI Data Leakage: How It Happens and How to Stop It

AI data leakage is sensitive, regulated or proprietary information leaving your control through an AI tool. Most of the time nobody means harm. A staff member pastes a client email into a chatbot to tidy the wording, an AI feature switches on inside software you already pay for, or an assistant surfaces a document to someone who should never have seen it. You can’t stop what you can’t see, so the first job is visibility. After that come clear rules, sensible access controls and training that people actually follow.

This guide is for Australian business owners, managers and IT leads. It covers how leakage happens, what the regulators say, and what to do in the next 90 days.

CalibreforumSee Where AI Use Needs AttentionCalibre Forum’s AI readiness assessment gathers employee insights and produces a scored report with a 90-day action plan.Explore the AI assessment

TL;DR

AI leakage is mostly accidental and everyday: pasted prompts, embedded AI features and over-broad access, not Hollywood-style hacking.
The Verizon 2026 DBIR, as summarized by Forcepoint, reports 45 percent of employees were regular AI users on corporate devices by mid-2026, up from 15 percent a year earlier.
Australian regulators advise against entering personal or sensitive information into public generative AI tools, and they expect clear internal policies and vendor due diligence.
Blanket bans rarely work because AI is built into everyday software. Know where your sensitive data lives and who can reach it.
Start by finding out what your people actually use, then write rules around reality.

What AI data leakage actually looks like
Why it’s now a mainstream business risk
What Australian rules say
Why banning AI doesn’t work
A practical 90-day approach
Perspective: visibility comes before policy
Getting a clear picture with Calibre Forum
FAQ
Sources

What AI data leakage actually looks like

The term gets used for two different things, and search results mix them up. In machine learning, “leakage” means training information contaminating a model’s test results. That is a modeling error, not a security problem. This article is about the business meaning: your information escaping through AI.

The common routes are less dramatic than most people expect.

Pasting into public tools

This is the obvious one. Someone drops a contract, a customer list, a patient note or source code into a public chatbot to summarize or fix it. Once the information is in a tool you don’t control, you may not be able to get it back. The Office of the Australian Information Commissioner (OAIC) says that once personal information is entered into a public tool, control over it can be lost, and recommends that organizations avoid entering it.

Features switched on inside tools you already use

AI now sits inside email, document suites, meeting software and operating systems. A staff member may never open a chatbot and still send data through an AI feature. This is why bans on named tools have limited effect. The policy has to cover the data, not just the brand names.

Memory, retrieval and context

Internal assistants create their own exposure. Microsoft’s security documentation notes that real-world exposure increasingly comes from memory, context and retrieval artifacts: working summaries, retrieved document snippets and cached data can reach authorized users who lack clearance for that particular information. Often the person who triggers it simply asks the assistant a question.

The practical lesson: if an assistant is connected to your file shares, it inherits every permission mistake you’ve ever made. A folder that was “technically open” for years but never browsed can suddenly be one question away.

AI retrieval exposes a restricted file snippet

Memorization in models

OAIC guidance notes a further risk with models trained on large datasets: fragments of a document, such as a phone number or a piece of code, can sometimes be pulled out later. Deleting a file afterward doesn’t guarantee its content is gone from a model that already learned from it. That’s one reason to check whether a vendor trains on your inputs before you adopt its tool.

Operational data

If you run industrial or operational systems, even routine readings like temperatures or flow rates can matter. The Australian Cyber Security Centre explains that when this data is used to train or tune a model, it can become statistically stored inside it, turning transient noise into a lasting intellectual property risk. Engineering firms and similar businesses should weigh that before sending project data to an AI vendor.

Why it’s now a mainstream business risk

The speed of adoption is the problem. According to the Verizon 2026 Data Breach Investigations Report as summarized by Forcepoint, regular AI use on corporate devices tripled in a year. Policies written in 2024 often describe a workplace that no longer exists.

The gap that hurts most is between what management thinks is happening and what staff do. People use AI because it saves them time, and many don’t know what counts as sensitive. If the only guidance is a vague line in a handbook, they’ll fill the silence with their own judgment.

The ACSC’s guidance for small business lists data leaks and privacy breaches among the key cyber security risks to understand before adopting AI tools. Small and medium businesses are explicitly in scope.

What Australian rules say

This section covers Australia only. New Zealand has its own privacy law, so if you operate there, check the Office of the Privacy Commissioner’s guidance. This isn’t legal advice: if you suspect a breach, or aren’t sure whether the Privacy Act applies to you, speak to a privacy lawyer.

Who the Privacy Act covers

The Privacy Act 1988 covers organizations with annual turnover above $3 million and Australian Government agencies. Turnover alone doesn’t settle every case, so confirm your status with the OAIC or your adviser rather than assuming you’re exempt.

What the OAIC expects when staff use AI

The OAIC’s guidance on commercially available AI products makes several points that matter for leakage:

Privacy obligations apply to personal information put into an AI system and to AI output that contains personal information.
Information that is inferred or generated, including hallucinations, counts as personal information if it relates to an identifiable person.
As best practice, the OAIC recommends not entering personal information, particularly sensitive information, into publicly available generative AI tools.
Organizations should do due diligence on a product before adopting it and make sure its privacy terms fit their obligations.

What the ACSC recommends

The ACSC’s AI data security guidance points businesses to review internal data governance, set clear do-not-upload rules, train staff on AI risks and anonymize data before processing. For anyone integrating AI, it says the traditional controls still matter, including role-based access and encryption.

Data sovereignty

Where an AI service is hosted matters. The ACSC warns that services accessed from foreign jurisdictions may be subject to local laws or directions that conflict with your own obligations, so you need to know where data is stored and who can reach it. Ask vendors directly.

Why banning AI doesn’t work

Many businesses start with a ban. It feels decisive and it’s cheap. The trouble is that AI is now built into software people have to use. And a ban pushes use onto personal phones and home accounts, where you have even less visibility.

A more durable approach is data-centric. Work out what sensitive data you hold, who should be able to reach it, and enforce those boundaries whichever tool is involved. NIST’s generative AI profile frames the issue in similar terms, covering data retention, containment and leakage as lifecycle concerns. Note that the version we reference is a withdrawn draft, so treat it as background rather than current standard.

A practical 90-day approach

This is a sequence we’d suggest for a business of 10 to 200 people. Adjust the pace to your size.

Days 1 to 30: find out what’s really happening

Ask every staff member, not just the IT team, which AI tools they use, for what tasks and with what kinds of information. Make it anonymous. People admit to shortcuts when they won’t be named for them.
List the AI features already inside your approved software.
Map your sensitive data: client records, health or financial information, employee files, source code, pricing, contracts.
Check each vendor’s terms: does it train on your inputs, where is data stored, how long is it retained?

Days 31 to 60: write rules people can follow

Publish a short do-not-upload list in plain language, with examples from your own work. “No client names or case details in public tools” is clearer than “no confidential data”.
Name the approved tools for each common task, so staff have a safe route.
Tidy permissions before connecting any assistant to shared drives. Apply least privilege and remove stale access.
Decide who staff ask when they’re unsure, and make that person easy to reach.

Days 61 to 90: train, test, repeat

Run short training around real scenarios from your first-month findings.
Set an incident process: who is told, how fast, and who decides whether a notification is needed. Take legal advice on notification duties before an incident, not during one.
Re-check usage. Compare against the first month to see whether habits changed.

Trust matters on the customer side too. If your teams use AI for marketing content, a 2026 analysis of how audiences react to AI-generated ads is worth a read alongside your data rules.

Perspective: visibility comes before policy

We see the same pattern again and again. Leadership writes a policy first, then discovers it describes tools nobody uses and misses the ones everybody does. We think that order is backwards.

Our view is that the most useful thing a business can have is an honest, organization-wide picture of how AI is really used day to day. That picture only comes from asking everyone, and people only answer honestly when they trust the answers can’t be traced back to them. So we built our method around anonymous surveys of every staff member, with no software to install and no individual data collected. We also don’t make compliance or legal claims, because those are decisions for your advisers.

When management sees the map, the conversation shifts. It stops being “who broke the rules?” and becomes “which workflows need a safe, approved path?” That is a conversation staff are happy to join.

— Anthony

Getting a clear picture with Calibre Forum

If you want the visibility step done properly, our AI readiness assessment is built for it. It’s an anonymous survey of your staff on current AI tool use, workflows and technology integration. The result is one scored report that maps your capabilities, flags risks and sets out a tailored 90-day action plan covering operational and security improvements.

The Report is A$1,990 as a one-off. If you want to track change quarter by quarter, the Membership is A$2,388 per year and covers recurring assessments and reports. You can see what’s inside the report on our what you get page, and how we protect staff answers on our security and privacy page.

Industry matters because the data differs. Our industry pages cover areas such as medical and allied health clinics, law firms and recruitment agencies.

Getting a clear picture with Calibre Forum — overview diagram

FAQ

What causes AI data leakage?

Mostly ordinary behavior: staff pasting sensitive text into public tools, AI features embedded in everyday software, and assistants connected to data with loose permissions. Vendor practices, such as training on customer inputs or retaining data, add to the risk. Deliberate attacks exist but are not the main source for most small and medium businesses.

Is an AI data leak the same as a data breach?

Not always. A leak is information escaping your control. Whether it’s a notifiable breach under the Privacy Act depends on the facts, including whether personal information is involved and the likelihood of serious harm. Talk to a privacy lawyer or the OAIC before deciding.

Can we just block ChatGPT and similar tools?

Blocking helps a little, but it doesn’t cover AI features built into email, documents and operating systems. It can also drive use onto personal devices. A better plan combines clear do-not-upload rules, approved tools, tight permissions and training.

Do data leakage prevention tools solve this?

Traditional controls such as access management and encryption remain important, and the ACSC still recommends them. But software alone can’t tell you which tools staff use or why. Pair any technical control with a clear view of real usage and plain-language rules.

How does AI affect data privacy for Australian businesses?

Privacy obligations apply to personal information you enter into an AI system and to output that contains personal information, including inferences. The OAIC recommends not entering personal or sensitive information into public generative AI tools. Check vendor terms and hosting location too.