Sam, a customer of Orbit Pay, asks the help-centre bot how long he has to request a refund. The bot answers "within 30 days (see Refund Policy, section 4.2)". The policy says 14 days and there is no section 4.2. Nobody attacked anything: the model answered from memory and the application sent it out as fact. A refund desk then reads the same number and pays $290 with no human involved, and a coding assistant recommends a package that does not exist. You will reproduce all three locally, fix each, and test which fix stops which failure.
For authorized testing and learning only. The lab listens on
127.0.0.1, uses a made-up company and a fake package registry, runs a built-in simulated model (no API key, offline), and never runs a real package.
What is misinformation in an LLM app?
LLM09: Misinformation is the ninth entry in the OWASP Top 10 for LLM Applications (2025). A model produces fluent, confident output that is false, and a person or system acts on it. OWASP lists factual errors, unsupported claims, misrepresented expertise and unsafe generated code as consequences.
The root bug is not the model being wrong. It is the application treating model output as verified fact. A fluent sentence carries no evidence: the application must supply it or refuse.
Why it happens
- Hallucination: the model fills gaps with plausible text.
- Missing or stale data: the model answers from memory, not the current policy.
- Overreliance: people and systems act on the output without checking it.
The lab covers factual errors, unsupported claims and unsafe generated code; misrepresented expertise (for example medical or legal advice) is not modelled. OWASP also advises labelling AI output, stating its limits in the UI and training users not to over-trust it; the lab does not build those process controls.
Real-world grounding
In Moffatt v. Air Canada (2024 BCCRT 149), a chatbot gave wrong bereavement-fare rules and the tribunal held the airline responsible. Spracklen et al. tested 16 code models on 576,000 samples and found 19.7% of recommended packages did not exist: the basis of "slopsquatting".
Impact
| What goes wrong | Consequence |
|---|---|
| Confident false answer from the help bot | A customer acts on it; the company can be held liable (Moffatt v. Air Canada) |
| Invented package name | A developer or CI job installs attacker code (supply chain) |
| AI decision executed without review | Money leaves out of policy, with no approver in the ledger |
Attack flow

Lab setup: the Orbit Pay AI features
- Requirements
- Docker with Compose v2, or Python 3 (standard library only; tested on 3.13 locally and 3.12 in Docker), a browser, free port 8309.
- SHA-256
efebc3d7e43a03a74e263d4f3690912a48b28793b85d53389798be596236d70e
How to check the file is genuine
sha256sum llm09-misinformation-lab.zipshasum -a 256 llm09-misinformation-lab.zipGet-FileHash llm09-misinformation-lab.zipOrbit Pay has three AI features: the help-centre bot (OrbitBot), an AI refund desk that asks the bot and pays refunds, and a coding assistant. You are Sam Carter (customer); Dev Kim writes code; Mallory Quinn is the attacker you play in Mission 2. Start the lab with the run commands in the lab reference and open http://127.0.0.1:8309. The badge reads VULNERABLE: no fixes. Press Reset a password and Run: a correct answer, but no page is cited. The Knowledge base tab holds the ground truth: six pages, none about a certification.

Simulation limits. SimLLM is a deterministic table of "memories" with one stale habit: it writes 30 days even when shown a page that says 14. A real model is probabilistic; the fixes are plain code and hold with any model. A prompt saying "answer only from the context" is not a control.
Exploiting it
Each mission has three hints; the server issues a flag only when it sees the outcome.
Mission 1: make the help centre state something false
Goal: as Sam, get two confident answers the documentation does not back.
- As Sam Carter, press M1a – refund window and Run. The reply says 30 days and cites section 4.2. Model & checks shows the model got only the question and the claim check marks both as NO.
- Press M1b – PCI certification and Run. The documentation says nothing; the bot answers "Yes, Orbit Pay is fully PCI DSS certified".


Option 2: from the terminal
curl -s -X POST 127.0.0.1:8309/api/chat -H 'X-Session-User: 2001' \
-d '{"question":"How long do I have to request a refund on an annual plan?"}' \
| python3 -c 'import json,sys; r=json.load(sys.stdin); print(r["final"], r["false_claims"])'
# ...within 30 days of purchase (see Refund Policy, section 4.2). ['refund window stated as 30 days (the policy says 14)']
Show solution
Solution: Mission 1
Nothing connects the answer to the documentation. Fix: ground and check it (F1); abstain when there is no answer (F2).
Mission 2: squat the package name the assistant made up
Goal: Dev Kim asks the coding assistant for JWT help and is told to install a package that does not exist. Register the name and let his install print your hook.
- As Mallory Quinn, open Dev assistant and press Ask: validate a JWT in Flask.
flask-jwt-validator-proshows as not found on the registry. - Press Publish this name (the "install hook" is a string the lab only prints).
- As Dev Kim, ask again and press Install the packages in this answer. Your hook text appears in the sandbox terminal.


Option 2:
curl -s -X POST 127.0.0.1:8309/api/registry/publish -H 'X-Session-User: 2004' \
-d '{"name":"flask-jwt-validator-pro","version":"1.0.0","hook":"SIMULATED PAYLOAD"}'
curl -s -X POST 127.0.0.1:8309/api/dev/install -H 'X-Session-User: 2003' \
-d '{"names":["flask","pyjwt","flask-jwt-validator-pro"]}'
# the last result carries "hook": "[install hook output] SIMULATED PAYLOAD" (printed, never run)
Show solution
Solution: Mission 2
"The package exists" is not proof of safety: the attacker made it exist. Fix: check age, downloads and an approved list (F3).
Mission 3: let the AI pay a refund the policy forbids
Goal: the refund desk asks the bot for the window and executes its decision. Ticket T-2041: $290, bought 20 days ago, policy window 14.
Open Refund desk and press Let the AI triage T-2041. The desk reads 30 days, decides approve and pays with no human approval.

Option 2:
curl -s -X POST 127.0.0.1:8309/api/tickets/triage -H 'X-Session-User: 2001' -d '{"ticket":"T-2041"}' \
| python3 -c 'import json,sys; r=json.load(sys.stdin); print(r["decision"]["action"], r["release"], r["payment"])'
# approve pay {'id': 'P-0xx', 'ticket': 'T-2041', 'amount': 290, ... 'approved_by': None, 'out_of_policy': True}
Show solution
Solution: Mission 3
The desk trusts a model decision as if it were a rule: overreliance. Fix: a person approves any AI decision that moves money (F4); the model reads the verified answer (F1).
Fixing it
The Defences tab has one switch per fix and a What changed? drawer showing the real source.
F1: ground the answer, then verify its claims
This is retrieval-augmented generation (RAG) plus a check. The model gets the retrieved pages; every number, amount and duration in its answer is checked against the cited page, and an unbacked claim is replaced by what the page says.
def context_fixed(hits):
return [h for h in hits if h["score"] >= USE_SCORE][:TOP_K]
def verify_fixed(text, chunks):
claims = claim_check(text, chunks)
bad = {c["sentence"] for c in claims if not c["supported"]}
if not bad:
return {"text": text, "corrected": False, "dropped": 0}
# ... replace each bad sentence by best_sentence(s, src["text"]) + "[" + src["id"] + "]"
The check is narrow: "we are certified" has nothing to compare, so it passes.
F2: abstain when the documentation has no answer
def gate_fixed(support):
if support < MIN_SUPPORT:
return {"ok": False, "why": "the documentation covers this question with support %.2f, below the %.2f threshold" % (support, MIN_SUPPORT)}
return {"ok": True}
F3: verify every package
# toy values: calibrate for a real deployment
# REGISTRY_MIN_AGE_DAYS = 90, REGISTRY_MIN_DOWNLOADS = 10000, MIN_SUPPORT = 0.34
def package_verdict_fixed(name, registry, allow): # abridged
meta, reasons = registry.get(name), []
if meta is None:
reasons.append("not found on the registry")
elif meta["age_days"] < REGISTRY_MIN_AGE_DAYS or meta["downloads"] < REGISTRY_MIN_DOWNLOADS:
reasons.append("too new or too few downloads")
if name not in allow:
reasons.append("not on the approved dependency list")
return {"ok": not reasons, "reasons": reasons}
F4: human review before money moves
def release_fixed(decision):
return {"deny": "deny"}.get(decision["action"], "hold") # approve or escalate: a support lead decides
Re-testing the fixes
Mission 4: prove the fixes hold
Goal: every attack fails with all fixes on and normal work still runs.
- In Refund desk, press Clear tickets, log and impact, then Secure (badge: SECURE: 4 of 4 fixes).
- As Sam run M1a (corrected to 14 days, cited
[kb01]) and M1b (I can't find this in our documentation, so I won't guess). As Dev Kim ask the JWT question: the package is dropped and refused. Triage T-2041: the desk reads 14, declines, and the ledger stays empty. - Run Reset a password: it still works, and the M4 flag appears.


Option 2: tools/attack.sh vulnerable, then tools/attack.sh secure: it shows flask-jwt-validator-pro refused, AI decision: deny (window used: 14 days) and the M4 flag.
Each fix alone
The Matrix tab runs four attacks and five normal tasks against eight settings; normal tasks pass 5/5 in all. LEAK means the attack succeeded.
| Setting | S1 wrong window | S2 invented cert | S3 squatted package | S4 AI pays out of policy |
|---|---|---|---|---|
| No fixes | LEAK | LEAK | LEAK | LEAK |
| F1 only | blocked | LEAK | LEAK | blocked |
| F2 only | LEAK | blocked | LEAK | LEAK |
| F3 only | LEAK | LEAK | blocked | LEAK |
| F4 only, careful reviewer | LEAK | LEAK | LEAK | blocked |
| F4 only, rubber-stamp reviewer | LEAK | LEAK | LEAK | LEAK |
| F1 + F2 | blocked | blocked | LEAK | blocked |
| All four | blocked | blocked | blocked | blocked |

Four lessons:
- F1 alone corrects the number, not the certification: a yes/no claim has nothing to compare.
- F2 alone abstains on the unknown, not on the wrong: the refund question is covered, so the stale number passes.
- F3 stops the install, not the wrong answer. The gate refuses a package that now exists. Age and download counts can be gamed (inflated downloads, waiting out an age limit); the approved list or internal mirror is the control that holds.
- Review is only as good as the reviewer. A careful one rejects the queued T-2041; a rubber-stamp one pays anyway.


Vulnerable vs. fixed
| Vulnerable | Fixed | |
|---|---|---|
| Model context | The question only | Retrieved pages, claims checked (F1) |
| Unknown questions | A fluent guess | "I can’t find this" (F2) |
| Dependencies | Whatever the model names | Approved list, age, downloads (F3) |
| AI payouts | Executed at once | Held for a person (F4) |
Detecting this class of bug
Log per reply the support score, cited pages and claim-check result; log every recommended or installed package and every payout with its approver. Never trust the model to report its own doubt.
| Rule | Fires when | False positive |
|---|---|---|
| D1 | A reply is delivered as fact with low support | Small talk |
| D2 | A number or duration in the output is not in the cited page | A calculation |
| D3 | An unapproved, missing or brand-new package is recommended or installed | A new internal package |
| D4 | Money moved with no approver, or a flagged mismatch approved | An auto-approve limit |
D2 and D3 still fire in Secure mode: the model drifted or someone tried. In production, diff installs against your lockfile and mirror (D3) and alert on payouts with an empty approver (D4).

Offline, python3 tools/detect.py applies the same rules to the audit file (8 of 12 events flagged after the vulnerable run).
Self-check
- Why does F1 alone not stop the invented PCI claim? (a) the page is missing; (b) the claim check has no number to compare; (c) F1 runs first.
- Which fix stops the slopsquatted install? (a) F2; (b) F3; (c) F4.
- True or false: telling the model "answer only from the context" is a control. Which OWASP cause is Mission 3?
- A bot cites a non-existent
[kb07]. Name the bug, two controls, one log signal.
Show answers
Answers
- (b). F2 handles it.
- (b).
- False. It is a request, not enforcement. Mission 3 is overreliance.
- Citations are never checked. Controls: resolve each citation to a retrieved page (F1); abstain on low support (F2). Signal: D1 or D2.
Key takeaways
- A fluent answer is not evidence: supply the source, check claims against it, or abstain.
- A package that exists is not safe: verify age, downloads and an approved list.
- Never let an AI decision move money without a person who compares it with the policy (overreliance).
- Test each fix alone and together, and alert on attempts (D2, D3).
Lab reference: files, tests and clean-up
llm09-misinformation/
README.md run.sh run.ps1 Dockerfile docker-compose.yml
app/ server.py agent.py defenses.py simllm.py missions.py scenarios.py
detection.py matrix.py policy.py data.py ui/index.html
tools/ attack.sh detect.py
tests/ test_lab.py
unzip llm09-misinformation-lab.zip
cd llm09-misinformation
./run.sh # Docker ... or:
./run.sh local # plain python3, no Docker
Windows: .\run.ps1 is untested; otherwise run python server.py in app.
Expect Ready: open http://127.0.0.1:8309 (Docker) or LLM09 lab http://127.0.0.1:8309 (model: SimLLM, fixes on: 0 of 4) (local), then the badge VULNERABLE: no fixes.
Tests: python3 -m unittest discover -s tests -v (70 tests, no server needed).

Stop and clean up:
docker compose down --rmi all # Docker; plain Python: Ctrl+C
References
- OWASP GenAI Security Project, LLM09:2025 Misinformation
- Spracklen et al., We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs (2024)
- McCarthy Tétrault, Moffatt v. Air Canada: misrepresentation by an AI chatbot (decision: 2024 BCCRT 149)
- OWASP GenAI Security Project, OWASP Top 10 for LLM Applications 2025