Ethics, Errors and Explainability
15 min read
AI systems make mistakes. They reflect biases from their training data. Their decision-making processes are often opaque. Understanding these limitations helps you use AI responsibly and recognise when human judgment is essential.
What are hallucinations?
In the context of LLMs, "hallucinations" refer to responses that sound plausible but are incorrect or entirely fabricated. This happens because LLMs generate text based on what's statistically likely, not what's factually accurate.
Unlike a search engine, LLMs don't check facts. They predict the most likely next words based on their training data. There's no verification step between generation and output. The system produces text that fits the patterns it learned, regardless of whether that text is true.
LLMs generate answers even when they lack sufficient information. They don't say "I don't know." They don't indicate uncertainty. They produce plausible-sounding responses that may be completely wrong. This confident tone makes hallucinations particularly dangerous. The response sounds authoritative even when it's fabricated.
When LLMs lack context or knowledge, they fill gaps with text that fits the pattern. If you ask about a government report from last year, the AI might confidently cite sections, page numbers, and quotes. All convincingly detailed. None of it real. The system has learned patterns about how reports are structured and referenced. It applies these patterns even when it has no knowledge of the specific document.
This is why you need to verify factual claims. Statistics, dates, legal requirements, and recent events all require checking against authoritative sources. The more plausible the response sounds, the more carefully you should verify it.
Why are hallucinations particularly dangerous in the public sector?
Government work depends on accuracy. Policy decisions affect people's lives. Public communications must be trustworthy. Legal interpretations carry consequences. When AI hallucinates in these contexts, the risks are significant.
An AI might cite non-existent research to support a policy position. This undermines evidence-based decision-making. Decisions get made based on fabricated evidence that sounds credible but doesn't exist. The policy rests on a false foundation.
Fabricated case citations or misrepresented legislation can lead to incorrect legal advice or flawed regulatory interpretation. Legal work requires precision. A hallucinated case reference might seem perfectly plausible until someone tries to verify it. By then, flawed advice may have already shaped decisions.
False information in public-facing materials damages trust and can mislead citizens about their rights or obligations. When government publishes information, people rely on it. They make decisions based on what they believe to be authoritative guidance. Discovering it was AI-generated fiction erodes trust in public institutions.
Hallucinated summaries of stakeholder input can misrepresent public opinion or miss critical concerns. If you're using AI to summarise consultation responses and it fabricates themes or concerns that weren't raised, you're basing policy on fiction. If it omits legitimate concerns because they don't fit familiar patterns, important voices get lost.
How can you identify potential hallucinations?
When AI provides detailed citations, statistics, dates, or quotes, treat them with skepticism. The more specific and authoritative the claim sounds, the more carefully you should verify it. Specificity creates an illusion of reliability. A fabricated citation with a page number and publication date sounds more credible than a vague reference. But the detail doesn't make it real.
If different parts of a response contradict each other, the AI is likely combining conflicting patterns from its training data. This internal inconsistency reveals the limitation of pattern matching. The AI has pulled from incompatible sources without any ability to recognise they conflict. When you spot contradictions, treat the entire response with caution.
Any factual claim that will influence policy, operations, or public communications requires independent verification. Don't rely on AI alone for information that carries consequences. Cross-reference against authoritative sources. Check official publications, verified databases, and established reference materials.
The rule is straightforward. If the information matters, verify it. If you cannot verify it, don't use it. AI outputs are starting points for your work, not endpoints you can publish without review.
What is bias in AI systems?
AI systems learn from data created by humans. That data reflects human biases, prejudices, and inequalities. The AI absorbs and reproduces these patterns.
Bias in AI isn't intentional. It's structural. The system learns what it's shown. If training data includes more examples of men in leadership roles, the AI learns this as a pattern. If training data underrepresents certain communities, the AI's responses reflect this gap.
How does bias manifest in AI outputs?
AI might use gendered language inappropriately or make assumptions about roles, capabilities, or circumstances based on demographic characteristics. When discussing examples or generating scenarios, AI might consistently default to certain demographics while underrepresenting others.
AI trained predominantly on Western content may not understand or appropriately represent other cultural contexts. The nuances of non-Western communication styles, social structures, or cultural references may be missed or misrepresented.
AI trained on historical texts absorbs outdated attitudes, terminology, or assumptions that no longer reflect current understanding or values. Language that was once standard may now be offensive. Assumptions that were widely held may now be recognised as discriminatory.
Why does bias matter in government use?
Government services must be equitable. Policy development should consider diverse perspectives. Public communications need to be inclusive. When AI systems carry biases, the consequences extend beyond poor outputs.
Biased AI outputs can perpetuate existing disparities rather than address them. If AI-generated policy analysis consistently overlooks impacts on certain communities, those communities remain underserved. If AI-assisted service design reflects assumptions about "typical" users that exclude significant populations, access becomes inequitable.
If AI-generated consultation summaries or policy briefs don't fairly represent minority views, important perspectives get lost. Decisions get made without full consideration of affected communities. This undermines the purpose of consultation and risks policies that serve some citizens better than others.
When citizens see bias in AI-generated government communications or services, it damages trust in public institutions. Trust is harder to rebuild than to maintain. A single instance of obvious bias can undermine years of work to build inclusive services.
Bias in AI-assisted decisions about service provision, benefit eligibility, or regulatory enforcement can create discrimination claims. Even unintentional bias creates legal liability when it results in differential treatment based on protected characteristics.
How can you address bias?
Review AI-generated content critically. Check whether it makes assumptions about people based on demographics. Examine whether it uses inclusive language. Consider whether it represents diverse perspectives or defaults to narrow examples.
If asking for examples or scenarios, explicitly request diverse representation in your prompts. AI won't automatically provide this. You need to specify it. Even then, review outputs to ensure diversity is meaningful rather than tokenistic.
Have people with different backgrounds and perspectives review AI-generated content before it's finalised or published. Your own blind spots may prevent you from spotting bias. Diverse reviewers bring different perspectives and notice different issues.
Recognise that AI reflects patterns in its training data. It won't automatically correct for historical biases or structural inequalities. If anything, it may amplify them by treating them as patterns to reproduce. Human oversight remains essential.
What is the black box problem?
When AI generates a response, you see the output. You don't see how it arrived at that output.
The system processes millions of mathematical calculations. It weighs countless patterns. The final response emerges from this complex process. But the steps aren't transparent. This creates a fundamental accountability problem.
Why does explainability matter?
When you can't see the reasoning, you can't check whether it's sound. Verification becomes difficult. You can test whether the output seems correct, but you cannot assess whether the process that produced it was appropriate.
When AI contributes to a decision, explaining that decision to stakeholders becomes complicated. "The AI recommended this" doesn't explain why the recommendation makes sense. Stakeholders need to understand the reasoning, not just the result.
People need to understand how decisions are made, especially in government contexts where transparency is essential. Black box AI undermines transparency. Citizens have a right to understand decisions that affect them. Unexplainable AI processes make this impossible.
How can you work with this limitation?
When AI contributes to your work, document what you asked, what it provided, and how you reviewed and modified the output. This creates an audit trail. It shows that you applied judgment rather than accepting AI outputs uncritically.
Use AI outputs as inputs to your decision-making, not as the decision itself. The AI generates options, analysis, or drafts. You make the actual decision based on your expertise, judgment, and understanding of context.
When communicating decisions, focus on your reasoning and judgment, informed by various sources including AI tools. Don't try to explain how the AI reached its conclusion. Explain how you reached yours, acknowledging that AI-generated analysis informed your thinking.
Ensure multiple people review AI-assisted work, particularly when it has significant implications. Different reviewers catch different issues. Multiple perspectives strengthen quality control and reduce the risk of uncaught errors.
What does this mean for you?
These constraints don't make AI unusable.
AI can help draft, summarise, and explore ideas. But it cannot replace human judgment, verify facts, eliminate bias, or explain its reasoning. You provide the judgment, accountability, and ethical oversight that AI cannot.
When you understand what AI can't do, you make better decisions about what it should do.