Sam, a customer of Orbit Pay, asks the help-centre bot how long he has to request a refund. The bot answers "within 30 days (see Refund Policy, section 4.2)". The policy says 14 days and there is no section 4.2. Nobody attacked anything: the model answered from memory and the application sent it out as fact. A refund desk then reads the same number and pays $290 with no human involved, and a coding assistant recommends a package that does not exist. You will reproduce all three locally, fix each, and test which fix stops which failure.

For authorized testing and learning only. The lab listens on 127.0.0.1, uses a made-up company and a fake package registry, runs a built-in simulated model (no API key, offline), and never runs a real package.

What is misinformation in an LLM app?

LLM09: Misinformation is the ninth entry in the OWASP Top 10 for LLM Applications (2025). A model produces fluent, confident output that is false, and a person or system acts on it. OWASP lists factual errors, unsupported claims, misrepresented expertise and unsafe generated code as consequences.

The root bug is not the model being wrong. It is the application treating model output as verified fact. A fluent sentence carries no evidence: the application must supply it or refuse.

Why it happens

The lab covers factual errors, unsupported claims and unsafe generated code; misrepresented expertise (for example medical or legal advice) is not modelled. OWASP also advises labelling AI output, stating its limits in the UI and training users not to over-trust it; the lab does not build those process controls.

Real-world grounding

In Moffatt v. Air Canada (2024 BCCRT 149), a chatbot gave wrong bereavement-fare rules and the tribunal held the airline responsible. Spracklen et al. tested 16 code models on 576,000 samples and found 19.7% of recommended packages did not exist: the basis of "slopsquatting".

Impact

What goes wrong Consequence
Confident false answer from the help bot A customer acts on it; the company can be held liable (Moffatt v. Air Canada)
Invented package name A developer or CI job installs attacker code (supply chain)
AI decision executed without review Money leaves out of policy, with no approver in the ledger

Attack flow

One model, three places where its output is trusted, with F1-F4 and D1-D4

Lab setup: the Orbit Pay AI features

Hands-on lab · LLM09
Orbit Pay AI features
A self-contained practice lab that runs only on your own machine. Attack it, fix it, and prove each fix works.
Download the lab (zip) Sign in to downloadllm09-misinformation-lab.zip · 52 KB
Requirements
Docker with Compose v2, or Python 3 (standard library only; tested on 3.13 locally and 3.12 in Docker), a browser, free port 8309.
SHA-256
efebc3d7e43a03a74e263d4f3690912a48b28793b85d53389798be596236d70e
How to check the file is genuine
Run the command for your system in the folder you downloaded to. The output must match the SHA-256 above exactly; if it does not, delete the file and download it again.
Linuxsha256sum llm09-misinformation-lab.zip
macOSshasum -a 256 llm09-misinformation-lab.zip
WindowsGet-FileHash llm09-misinformation-lab.zip
Files, tests, Windows notes and clean-up are in the lab reference at the end.

Orbit Pay has three AI features: the help-centre bot (OrbitBot), an AI refund desk that asks the bot and pays refunds, and a coding assistant. You are Sam Carter (customer); Dev Kim writes code; Mallory Quinn is the attacker you play in Mission 2. Start the lab with the run commands in the lab reference and open http://127.0.0.1:8309. The badge reads VULNERABLE: no fixes. Press Reset a password and Run: a correct answer, but no page is cited. The Knowledge base tab holds the ground truth: six pages, none about a certification.

Vulnerable baseline: correct answer, no source cited

Simulation limits. SimLLM is a deterministic table of "memories" with one stale habit: it writes 30 days even when shown a page that says 14. A real model is probabilistic; the fixes are plain code and hold with any model. A prompt saying "answer only from the context" is not a control.

Exploiting it

Each mission has three hints; the server issues a flag only when it sees the outcome.

Mission 1: make the help centre state something false

Goal: as Sam, get two confident answers the documentation does not back.

  1. As Sam Carter, press M1a – refund window and Run. The reply says 30 days and cites section 4.2. Model & checks shows the model got only the question and the claim check marks both as NO.
  2. Press M1b – PCI certification and Run. The documentation says nothing; the bot answers "Yes, Orbit Pay is fully PCI DSS certified".

OrbitBot says 30 days and cites a non-existent section 4.2

OrbitBot answers yes to PCI although no page mentions it

Option 2: from the terminal

curl -s -X POST 127.0.0.1:8309/api/chat -H 'X-Session-User: 2001' \
  -d '{"question":"How long do I have to request a refund on an annual plan?"}' \
  | python3 -c 'import json,sys; r=json.load(sys.stdin); print(r["final"], r["false_claims"])'
# ...within 30 days of purchase (see Refund Policy, section 4.2). ['refund window stated as 30 days (the policy says 14)']
Show solution

Solution: Mission 1

Nothing connects the answer to the documentation. Fix: ground and check it (F1); abstain when there is no answer (F2).

Mission 2: squat the package name the assistant made up

Goal: Dev Kim asks the coding assistant for JWT help and is told to install a package that does not exist. Register the name and let his install print your hook.

  1. As Mallory Quinn, open Dev assistant and press Ask: validate a JWT in Flask. flask-jwt-validator-pro shows as not found on the registry.
  2. Press Publish this name (the "install hook" is a string the lab only prints).
  3. As Dev Kim, ask again and press Install the packages in this answer. Your hook text appears in the sandbox terminal.

Mallory publishes the made-up name, 0 days old, with a hook

Dev Kim's install prints the attacker's hook text

Option 2:

curl -s -X POST 127.0.0.1:8309/api/registry/publish -H 'X-Session-User: 2004' \
  -d '{"name":"flask-jwt-validator-pro","version":"1.0.0","hook":"SIMULATED PAYLOAD"}'
curl -s -X POST 127.0.0.1:8309/api/dev/install -H 'X-Session-User: 2003' \
  -d '{"names":["flask","pyjwt","flask-jwt-validator-pro"]}'
# the last result carries "hook": "[install hook output] SIMULATED PAYLOAD"  (printed, never run)
Show solution

Solution: Mission 2

"The package exists" is not proof of safety: the attacker made it exist. Fix: check age, downloads and an approved list (F3).

Mission 3: let the AI pay a refund the policy forbids

Goal: the refund desk asks the bot for the window and executes its decision. Ticket T-2041: $290, bought 20 days ago, policy window 14.

Open Refund desk and press Let the AI triage T-2041. The desk reads 30 days, decides approve and pays with no human approval.

Refund desk pays T-2041 against the 14-day policy, no approver

Option 2:

curl -s -X POST 127.0.0.1:8309/api/tickets/triage -H 'X-Session-User: 2001' -d '{"ticket":"T-2041"}' \
  | python3 -c 'import json,sys; r=json.load(sys.stdin); print(r["decision"]["action"], r["release"], r["payment"])'
# approve pay {'id': 'P-0xx', 'ticket': 'T-2041', 'amount': 290, ... 'approved_by': None, 'out_of_policy': True}
Show solution

Solution: Mission 3

The desk trusts a model decision as if it were a rule: overreliance. Fix: a person approves any AI decision that moves money (F4); the model reads the verified answer (F1).

Fixing it

The Defences tab has one switch per fix and a What changed? drawer showing the real source.

F1: ground the answer, then verify its claims

This is retrieval-augmented generation (RAG) plus a check. The model gets the retrieved pages; every number, amount and duration in its answer is checked against the cited page, and an unbacked claim is replaced by what the page says.

def context_fixed(hits):
    return [h for h in hits if h["score"] >= USE_SCORE][:TOP_K]

def verify_fixed(text, chunks):
    claims = claim_check(text, chunks)
    bad = {c["sentence"] for c in claims if not c["supported"]}
    if not bad:
        return {"text": text, "corrected": False, "dropped": 0}
    # ... replace each bad sentence by best_sentence(s, src["text"]) + "[" + src["id"] + "]"

The check is narrow: "we are certified" has nothing to compare, so it passes.

F2: abstain when the documentation has no answer

def gate_fixed(support):
    if support < MIN_SUPPORT:
        return {"ok": False, "why": "the documentation covers this question with support %.2f, below the %.2f threshold" % (support, MIN_SUPPORT)}
    return {"ok": True}

F3: verify every package

# toy values: calibrate for a real deployment
# REGISTRY_MIN_AGE_DAYS = 90, REGISTRY_MIN_DOWNLOADS = 10000, MIN_SUPPORT = 0.34
def package_verdict_fixed(name, registry, allow):  # abridged
    meta, reasons = registry.get(name), []
    if meta is None:
        reasons.append("not found on the registry")
    elif meta["age_days"] < REGISTRY_MIN_AGE_DAYS or meta["downloads"] < REGISTRY_MIN_DOWNLOADS:
        reasons.append("too new or too few downloads")
    if name not in allow:
        reasons.append("not on the approved dependency list")
    return {"ok": not reasons, "reasons": reasons}

F4: human review before money moves

def release_fixed(decision):
    return {"deny": "deny"}.get(decision["action"], "hold")   # approve or escalate: a support lead decides

Re-testing the fixes

Mission 4: prove the fixes hold

Goal: every attack fails with all fixes on and normal work still runs.

  1. In Refund desk, press Clear tickets, log and impact, then Secure (badge: SECURE: 4 of 4 fixes).
  2. As Sam run M1a (corrected to 14 days, cited [kb01]) and M1b (I can't find this in our documentation, so I won't guess). As Dev Kim ask the JWT question: the package is dropped and refused. Triage T-2041: the desk reads 14, declines, and the ledger stays empty.
  3. Run Reset a password: it still works, and the M4 flag appears.

Secure mode: the unverified package is dropped and refused

Secure mode: the desk reads 14 days, declines T-2041, ledger empty

Option 2: tools/attack.sh vulnerable, then tools/attack.sh secure: it shows flask-jwt-validator-pro refused, AI decision: deny (window used: 14 days) and the M4 flag.

Each fix alone

The Matrix tab runs four attacks and five normal tasks against eight settings; normal tasks pass 5/5 in all. LEAK means the attack succeeded.

Setting S1 wrong window S2 invented cert S3 squatted package S4 AI pays out of policy
No fixes LEAK LEAK LEAK LEAK
F1 only blocked LEAK LEAK blocked
F2 only LEAK blocked LEAK LEAK
F3 only LEAK LEAK blocked LEAK
F4 only, careful reviewer LEAK LEAK LEAK blocked
F4 only, rubber-stamp reviewer LEAK LEAK LEAK LEAK
F1 + F2 blocked blocked LEAK blocked
All four blocked blocked blocked blocked

Matrix tab: every attack against every fix setting

Four lessons:

F4 alone: a careful reviewer rejects T-2041, nothing is paid

F4 alone: a rubber-stamp approval pays the refund anyway

Vulnerable vs. fixed

Vulnerable Fixed
Model context The question only Retrieved pages, claims checked (F1)
Unknown questions A fluent guess "I can’t find this" (F2)
Dependencies Whatever the model names Approved list, age, downloads (F3)
AI payouts Executed at once Held for a person (F4)

Detecting this class of bug

Log per reply the support score, cited pages and claim-check result; log every recommended or installed package and every payout with its approver. Never trust the model to report its own doubt.

Rule Fires when False positive
D1 A reply is delivered as fact with low support Small talk
D2 A number or duration in the output is not in the cited page A calculation
D3 An unapproved, missing or brand-new package is recommended or installed A new internal package
D4 Money moved with no approver, or a flagged mismatch approved An auto-approve limit

D2 and D3 still fire in Secure mode: the model drifted or someone tried. In production, diff installs against your lockfile and mirror (D3) and alert on payouts with an empty approver (D4).

Audit log: rules D1 to D4 firing on the four attacks

Offline, python3 tools/detect.py applies the same rules to the audit file (8 of 12 events flagged after the vulnerable run).

Self-check

  1. Why does F1 alone not stop the invented PCI claim? (a) the page is missing; (b) the claim check has no number to compare; (c) F1 runs first.
  2. Which fix stops the slopsquatted install? (a) F2; (b) F3; (c) F4.
  3. True or false: telling the model "answer only from the context" is a control. Which OWASP cause is Mission 3?
  4. A bot cites a non-existent [kb07]. Name the bug, two controls, one log signal.
Show answers

Answers

  1. (b). F2 handles it.
  2. (b).
  3. False. It is a request, not enforcement. Mission 3 is overreliance.
  4. Citations are never checked. Controls: resolve each citation to a retrieved page (F1); abstain on low support (F2). Signal: D1 or D2.

Key takeaways

Lab reference: files, tests and clean-up

llm09-misinformation/
  README.md  run.sh  run.ps1  Dockerfile  docker-compose.yml
  app/        server.py agent.py defenses.py simllm.py missions.py scenarios.py
              detection.py matrix.py policy.py data.py  ui/index.html
  tools/      attack.sh  detect.py
  tests/      test_lab.py
unzip llm09-misinformation-lab.zip
cd llm09-misinformation
./run.sh              # Docker ... or:
./run.sh local        # plain python3, no Docker

Windows: .\run.ps1 is untested; otherwise run python server.py in app.

Expect Ready: open http://127.0.0.1:8309 (Docker) or LLM09 lab http://127.0.0.1:8309 (model: SimLLM, fixes on: 0 of 4) (local), then the badge VULNERABLE: no fixes.

Tests: python3 -m unittest discover -s tests -v (70 tests, no server needed).

Final checklist complete, four flags collected

Stop and clean up:

docker compose down --rmi all     # Docker; plain Python: Ctrl+C

References