Putting AI into a workflow that touches personal information

We built an AI-assisted NDIS documentation platform and ran it in production: a support worker describes a shift in plain English, and the platform generates a structured draft case note, incident report or progress note and flags content for human review. That build is where everything below comes from.

What we did not expect is how little of the work was about the model. The Claude API call is a small piece of code. What took the time — and what actually decided what happened to personal information — was authentication, roles, database permissions, and deciding what a person had to approve before a document counted as real.

The AI is already in your workflow

The decision in front of most Australian businesses is not whether to introduce AI to a process. It is whether to acknowledge the version already running.

Deloitte Access Economics surveyed more than 1,000 Australian small and medium businesses for a report released in November 2025 and found two-thirds already use AI in some form, while only 5% of those using it were what the report called “fully enabled” to realise the benefits. Governance and standards gaps were named as one of five barriers.

Meanwhile the exposure is measurably growing. The OAIC reported in July 2026 that it received 1,205 data breach notifications in calendar 2025 — the highest since the scheme began in 2018, and an 8% increase on the year before. Health service providers were the top sector, at 225 notifications. IBM’s 2026 Cost of a Data Breach study, covering 602 organisations, put the global average breach cost at USD $4.99 million and found more than 20% of organisations reported a breach targeting AI models or applications.

The four questions that decide the risk

Every AI step that touches personal information comes down to four questions:

1. What leaves your system?
2. Where does it go, and under what terms?
3. Who inside your own system can already see it?
4. Who checks the output before anyone acts on it?

Apply them to the two situations most businesses are choosing between.

The read-out rule is blunt: if you cannot answer all four in one sentence each, the step is not ready to turn on.

Notice that three of the four have nothing to do with AI. That is why most AI vendor checklists miss them — and why a business with a system that already knows who is allowed to see what is in a far better position than one that does not, regardless of which model it picks.

1. What leaves your system

The safest record is the one the AI step never receives.

There is a hierarchy here, and most implementations start at the bottom of it. Sending the whole row is worst. Redacting identifiers before the request leaves is better. Best is scoping at the field level from the start — the step only ever receives the fields the prompt actually needs, because that is the only thing the code passes it.

Practical patterns that follow from this:

• Pass identifiers, not names, wherever the model does not need the name to do the job.
• Pass the fields the prompt needs, not the record they live on.
• Use structured input and structured output, in both directions. Free text in and free text out makes it impossible to say later what was sent.
• Decide retention for prompts and responses at design time. This is the one I got wrong: it is much harder to answer “how long do we keep the prompts?” after you have been keeping them for a year.

Over-collection is not only a question of principle. It sets the blast radius of every later mistake. A step that only ever saw three fields cannot leak a record.

De-identification, if you can manage it, removes the problem

In August 2025 the OAIC closed preliminary inquiries into I-MED Radiology Network, Harrison.ai and Annalise.ai without a breach finding, because the patient data that had been shared was de-identified sufficiently that it was no longer personal information under the Privacy Act.

Two things to take from that, and the second matters as much as the first. Information that is genuinely no longer about a reasonably identifiable person is outside the Act, so the cheapest privacy design is often to do the work without the identifiers. But the OAIC expressly said the closure “should not be taken as an endorsement” of the organisation’s broader practices. De-identification is a documented technique assessed in context and in totality, not a synonym for deleting the name column.

2. Where it goes, and on what terms

The tier your staff are signed in to changes the data-handling terms more than any control you can bolt on afterwards.

The general shape across major vendors is that consumer plans lean toward the vendor and business or developer tiers invert that. Anthropic’s position is the one I can state precisely, because it is the vendor I build on and it is documented. Its consumer terms update of 28 August 2025 applies to Free, Pro and Max plans: users who allow their data to be used for model training have chats retained for five years, and users who decline keep a 30-day retention period. That update expressly does not apply to services under the commercial terms — the API, Claude for Work, Bedrock, Vertex. On the API, Anthropic’s documentation says conversation content is not retained by default, retained data is never used for training without express permission, and zero-retention arrangements are available — with the honest caveat that even under zero retention, inputs and outputs can be held for up to two years where content is flagged by trust and safety systems or where the law requires it.

Other vendors publish their own defaults and change them. Read the current terms for the product your staff actually use, on the plan they are actually on, rather than trusting a summary — including this one.

Where the request runs, and why APP 8 is not about servers

Australian data residency for frontier models is narrower than the marketing around it suggests, and the clearest example is my own preferred vendor. As at the date of writing, Anthropic’s documentation says the inference_geo parameter supports two values — global and us — and that workspace geo, which controls where data is stored at rest, is available only as us. There is no Australian option on the first-party Claude API. Australian-region inference for Claude is available through Amazon Bedrock, which announced Australia cross-region inference in October 2025, routing between Sydney and Melbourne across the whole request lifecycle.

More importantly, it is easy to get the underlying rule backwards. Under APP 8, the question that triggers the cross-border disclosure obligations is who receives the information — whether an overseas recipient can access it — not which data centre it sits in. Choosing an Australian region is a good thing to do. It is not, on its own, an answer to APP 8.

3. Who inside your own system can already see it

This is the question nobody asks, and it is usually where the real exposure lives.

APP 11.3, inserted by the Privacy and Other Legislation Amendment Act 2024 and in force since 11 December 2024, provides that the reasonable steps required by APP 11 “include technical and organisational measures”. Policy alone was always a weak answer; it is a harder argument now. The OAIC’s APP 11 guidelines, updated 3 October 2025, set out what shapes “reasonable”: the size and resources of the entity, the amount and sensitivity of the information, the possible adverse consequences, the practicability of the measure, and whether the measure is itself privacy-invasive. Inconvenience alone does not excuse leaving it undone.

The technical measure that does the most work is the least glamorous one: the database should refuse the query, rather than relying on the application remembering to filter. Row-level security means a support worker sees only their own records, an admin only their own organisation, and a participant only what has been explicitly shared with them — enforced at the data layer, where forgetting is not possible.

An AI step tends to widen internal read access, quietly. A summariser typically needs to see more rows than any individual user is allowed to see. That is worth stating as a rule:

Least privilege applies to the AI step specifically. A feature that can read one approved table has a much smaller blast radius than an agent wired into email, drive, CRM and production database. And prompt injection remains the top-ranked risk, LLM01, in the OWASP Top 10 for LLM Applications — held there in the 2026 edition published in August 2026 — on the basis that models process instructions and data through the same channel with no clear separation. The narrower the step’s reach, the less a successful injection can do.

4. Who checks the output before anyone acts on it

The OAIC’s 2026 Australian Community Attitudes to Privacy Survey asked 1,511 Australians what they consider essential before an organisation uses AI to make a decision affecting them. The answers read like a design brief:

For context on the environment you are launching into: the same survey found only 4% of Australians believe AI companies are worthy of trust, and 93% consider it not fair and reasonable for organisations to use personal information to train AI models.

The OAIC’s guidance on commercially available AI products puts the same point as an expectation: a human user should be responsible for verifying the accuracy of personal information obtained through AI and be able to overturn decisions, and records should clearly show where information is the product of an AI output rather than established fact.

Designing a review point people actually use

A review step that gets rubber-stamped is worse than no AI at all, because it launders an unchecked output as a checked one. Four things make the difference between a real gate and a checkbox:

• Frame drafts as drafts in the product, not just in the terms of service. The word people see on screen does more than the clause they never read.
• Show what the model added — a diff or a highlight — so review is a specific act rather than re-reading a wall of text.
• Require an explicit approve action. Not a timeout, not a default-accept.
• Keep a record of what was generated, what was flagged, and who approved it.

One pattern worth naming, because it is the one I found most useful: a second AI pass whose only job is to flag content for human attention. It does not replace the reviewer. It directs them — which is what makes review survivable when there are forty documents rather than four.

What I built, and what I would do differently

The platform is a Next.js and Supabase application on Vercel, with the Claude API, Stripe for subscriptions and Resend for transactional email. It has multi-organisation accounts with role-based access, subscription billing kept in sync by webhooks, file storage and document history. It generates a structured draft in under three minutes and flags content for human review. It is not a prototype with a model bolted on.

Five decisions did most of the privacy work, and only one of them is about AI.

• Two front-ends on separate subdomains — a provider dashboard for support workers and admins, and a participant portal for the people receiving support — rather than one login with hidden permissions. A boundary you can point at is a boundary you can test.
• Row-level security on every table. The worker sees their records, the admin sees their organisation, the participant sees what was shared. Enforced by the database, not by the application remembering.
• Sharing is explicit and opt-in, one document at a time. Not a folder, not a feed, not a standing permission. The narrowest sharing primitive that still solved the problem.
• A second AI pass that only flags for review, feeding an admin review queue. The model checks the model; the human stays the gate.
• Full history of what was generated, what was flagged and who approved it, exportable. That record is what makes an AI step reviewable after the fact, and it is the part most AI features never build.

The decision that changed the data-handling picture most, though, was not an engineering decision at all. It was using an API under commercial terms instead of a consumer chatbot. That cost nothing in engineering effort and did more than any code I wrote.

What I would do differently

• Decide retention for prompts and model responses at design time, rather than discovering later that nobody had.
• Scope the model’s read access field by field from the first commit, rather than passing a whole record and narrowing later.
• Write the collection notice alongside the feature, not at launch.
• Run the platform’s own security advisor as part of the build and treat its output as work, not as a post-launch list to get to eventually.

Where Australian law actually sits

This section is dated deliberately, because parts of it will move. It is general information, not legal advice.

In force now

• The 13 Australian Privacy Principles, for entities the Privacy Act covers.
• APP 11.3 — reasonable steps include technical and organisational measures. In force 11 December 2024.
• The statutory tort for serious invasions of privacy, commenced 10 June 2025. It requires the invasion to have been intentional or reckless, so an accidental or negligent data breach is not caught by it. It is not limited to entities covered by the APPs.

Coming

From 10 December 2026, new obligations at APP 1.7 to 1.9 require a covered entity’s privacy policy to disclose where a computer program makes — or does something substantially and directly related to making — a decision that could reasonably be expected to significantly affect an individual’s rights or interests using their personal information. If you are building something this year that will make or materially shape decisions about people, that date is worth putting in the plan.

The small business exemption, honestly

Many Australian businesses reading this are outside the Privacy Act entirely. A business with annual turnover of $3 million or less in the previous financial year is generally a small business for the purposes of the Act.

Three things people get wrong about it.

• It is a rolling annual test, not a one-way door. Section 6D(1) asks whether annual turnover for the previous financial year was $3 million or less, and it is re-applied each year. A business that goes over and later comes back under can be a small business again. Plenty of pages say crossing the threshold removes the exemption permanently; no provision in the Act says that.
• The carve-outs are real and easy to trip. Providing a health service and holding any health information ends the exemption, as does disclosing personal information for a benefit, providing a benefit to collect it, trading in personal information, or being a Commonwealth contracted service provider.
• Removal of the exemption is a policy proposal. No Bill has been enacted and no commencement date exists. You will find Australian pages confidently naming a date. Treat them with suspicion, and check the Act.

There is no Australian AI Act

AI use in Australia is governed by existing law — privacy, consumer, anti-discrimination, work health and safety — rather than by a dedicated statute. Anyone selling you certainty about how that law applies to a novel AI system is overselling. A useful reminder of that: in February 2026 the Administrative Review Tribunal partly overturned the Privacy Commissioner’s 2024 Bunnings determination, finding the collection of biometric information was authorised, while affirming the findings that Bunnings had breached APP 1 and APP 5. Regulators can be wrong, and positions move.

When the honest answer is not to build anything

I make a living building custom software, so read this section with that in mind. It is still true.

Most shadow AI problems are a procurement problem, not a software problem. Buy the business tier of the tool your staff already use, turn on single sign-on, write a one-page note about what may and may not be pasted into it. That is days of somebody’s attention rather than months, and it resolves the majority of the exposure. No build competes with that on cost or speed.

If a vendor already does your workflow properly, buy it. A custom build has to beat an existing product on more than novelty — and if that product has a real data processing agreement and an Australian data story, it is starting ahead.

If nobody will actually read the drafts, do not ship the feature. An unreviewed AI output in a workflow that touches personal information is worse than having no AI in it, because it carries the authority of having been checked.

If you can do the work without the identifiers, do that instead. The I-MED outcome is the clearest available demonstration that de-identified data can take the whole question outside the Act.

Building earns its place in a narrow set of cases: the AI step needs to read your own records under controlled scope, produce something a person then approves, and leave a record of who approved it. That is a system requirement, not an AI requirement — which is rather the point.

Before you turn it on

Written so each item is something you can verify, rather than something you can intend.

What leaves

• The exact fields the step reads are written down, and match what the code passes.
• Identifiers are removed or never sent where the task does not need them.
• Retention for prompts and responses is decided, and implemented.

Where it goes

• You know which account and tier the request goes through, for every path in the product.
• You have read the vendor’s current terms on training and retention for that tier — not a summary of them.
• You know where inference runs, and you have not confused that with the APP 8 question of who receives the information.

Internal access

• Access is enforced at the database, not only in the application.
• The AI step’s read scope is no broader than the user it acts for.
• The step cannot reach systems it does not need.

Review

• A named person approves output before anyone acts on it, with an explicit action.
• Output is visibly marked as generated, in the product.
• There is a record of what was generated, what was flagged and who approved it.

Common questions