Incidents, consequences and accountability
Treat every occurrence as an incident, keep records in an AI registrar, and know the five usual causes - brittleness, lack of robustness, lack of quality data, insufficient testing, and model or data drift. Forecast downstream consequences (resentment, false safety, a roadmap for malicious actors). Tools like AI Verify (11 ethics principles) automate accountability.
Treat every occurrence as an incident, learn why it happened, and keep the receipts in one place.
- Incident discipline → identify the issue and who it must be reported to, inside and outside the organisation → keep incident and issue information in an AI registrar → document the mitigation and communication.
- Why incidents happen → the five usual suspects → brittleness, lack of robustness, lack of quality data, insufficient testing, model or data drift. An incident's impact rarely stops at one tool, so notify partner and internal groups.
- Resentment → poorly implemented interventions and opacity read as unfairness, arbitrariness or ideological influence; superficial policies meet the letter but not the spirit of the law.
- False sense of safety → assuming all risks were addressed, likelier when there are incentives to obscure or reframe risks; one-time evaluation vs continuous monitoring is the failure pattern.
- Unintended consequences → asking developers to reflect on misuse can itself create a "roadmap" for malicious actors
Timing → review downstream consequences early in the research and development pipeline, Risk levels → categorise research and consequences by risk, Normalise discussions of downstream consequences, negative and positive, be fully transparent and proactive in identifying negatives, develop common protocols for responsible development, deployment and continuous improvement. Updates should be freely available in plain language, with use guidance and mitigation strategies.
- Accountability mechanisms → governments increasingly require audits and assessments by use case and risk level.
- Human review → respect automated decision-making rules for personal data; reviewers must be trained and empowered to override automated decisions (beware automation bias).
- Automation tools → manual validation is slow, costly and error-prone, so automation institutionalises processes and continuously collects evidence → AI Verify (Singapore) validates systems against 11 ethics principles; the Model Card Regulatory Check app automates compliance checks from model cards.
1) Review policies (privacy, security, IP, model ops, open source), 2) Evaluate options → cloud scales, on-prem controls, edge localises, 3) Manage third-party risk → two contexts, five riding risks, one screening strategy, 4) Prioritise ethics → document competing-value calls, keep humans in the loop, 5) Assess readiness → Well-Tested Code Deserves Model-cards, 6) Monitor continuously → baselines, drift detection, snapshots, red teaming, challenger models, 7) Document incidents → AI registrar, the five causes, human shutdown, 8) Ensure transparency → disclosure that AI is in place is the global threshold. Mnemonic bank → Cloud scales, On-prem controls, Edge localises, Foundation → Risk → Society, Well-Tested Code Deserves Model-cards.
Key terms - quick answers
What is “AI registrar”?
What is “Five incident causes”?
What is “AI Verify”?
What is “Model Card Regulatory Check app”?
Sources and study method
This independent lesson uses active recall, spaced retrieval and scenario practice. Read the full study method.