When AI Harms the Vulnerable: Lessons from Refugee, Justice, and Humanitarian Contexts¶
Introduction¶
Artificial intelligence systems are rarely tested most rigorously in comfortable conditions. They are tested at borders in the middle of the night, in bail hearings where a wrong prediction can mean months in pre-trial detention, and in disaster zones where connectivity is intermittent and data is incomplete. It is in these environments - high-stakes, resource-constrained, and populated by people who have the least power to push back - that the weaknesses of AI systems are exposed first and felt most acutely.
At TechEthics, our mission is to develop technology that strengthens democratic resilience and social cohesion - which means engaging seriously with the cases where technology has done the opposite. This article examines documented and representative failures of AI deployment across three domains: refugee and asylum processing, criminal justice, and humanitarian response. In each domain, we trace the specific mechanisms through which harm occurred, identify the design and governance failures that enabled it, and set out the practical safeguards that responsible practitioners should build in before deployment - not after.
The goal is not to condemn AI as unsuitable for these contexts. On the contrary, thoughtfully designed systems can dramatically improve efficiency, consistency, and outcomes. But the path to those benefits runs directly through an honest reckoning with what has already gone wrong.
Part One: Refugee and Asylum Processing¶
The Promise and the Problem¶
Asylum systems are chronically under-resourced. Caseloads in many jurisdictions have grown faster than adjudicator capacity for years, creating backlogs that stretch to years and impose profound uncertainty on people fleeing persecution. AI tools that can triage cases, flag inconsistencies, or prioritise urgent claims hold genuine appeal.
The problem is that asylum adjudication is among the most context-sensitive, linguistically complex, and existentially consequential decisions a government makes. A missed nuance in a survivor’s testimony - a culturally specific expression of distress, an indirect reference to sexual violence common in certain communities, a tribal term with no clean translation - can convert a legitimate fear of persecution into a score that suggests fabrication. The model does not know what it does not know, and neither does the caseworker who trusts it.
Case: Risk Scoring Without Transparency or Appeal¶
Several national immigration authorities have piloted automated risk scoring tools to assist in processing asylum claims. These systems typically ingest application data, cross-reference it against watchlists and inconsistency flags, and produce a score or recommendation that influences how quickly a case is heard and whether the claimant is detained pending decision.
In practice, these scores have been used to de-prioritise or deny services on the basis of correlations that are opaque to applicants, their legal representatives, and, in many cases, the caseworkers themselves. When a claimant’s risk score is elevated because their route of travel matched a pattern associated with smuggling networks - without any mechanism to account for the fact that asylum seekers frequently have no legal means of travel - there is no channel through which that error can surface and be corrected. The applicant is simply in a slower lane, or in detention, without knowing why.
This opacity is not merely an ethical problem - it is increasingly a legal one. As our guide to UK and EU AI regulation sets out, Article 22 of the GDPR establishes a general prohibition on decisions “based solely on automated processing” that produce legal or similarly significant effects on individuals, with mandatory requirements for human intervention, transparency, and the right to contest. Asylum and immigration decisions squarely meet this threshold. Authorities deploying scoring tools without genuine contestation mechanisms are not merely falling short of best practice - they are operating in likely violation of data protection law on both sides of the Channel.
The harm is also structural. When AI systems create outcomes without legible reasons, identifying systematic bias - say, that claims from a particular country of origin are scored less favourably - requires external audit rather than routine process. Errors compound silently.
Case: Language Model Misclassification of Asylum Narratives¶
A separate class of failures involves natural language processing tools applied to the written or transcribed accounts that form the core of many asylum claims. These tools have been deployed to assess credibility, detect inconsistency across multiple interviews, and in some pilots, produce summaries for adjudicators. The training data for such models is almost always dominated by formal, literate, Western-European-language text.
Asylum seekers frequently communicate in ways that create systematic disadvantages in these systems. Oral narrative traditions from parts of sub-Saharan Africa, the Middle East, and Southeast Asia do not organise chronology the way that Western legal testimony expects. Trauma affects memory and coherence in ways that are well-documented in clinical and legal scholarship but poorly represented in model training data. Dialects and regional idioms - particularly those of minority communities who may face persecution precisely because of their linguistic identity - are underrepresented or absent. A model trained on standard Modern Standard Arabic will not handle Levantine dialect in the same way; a model trained on Northern Somali variants may mishandle Southern ones.
When these systems flag narrative features as indicators of unreliability, they do so based on patterns that reflect the structure of the training corpus more than the truth-value of the claimant’s account. The result is that individuals most at risk - members of minority communities, people with significant trauma histories, those without formal education - receive systematically worse outputs than those whose communication style happens to match the training distribution. This is precisely the kind of proxy discrimination that the EU AI Act’s risk classification targets as a high-risk application requiring conformity assessment before deployment.
Safeguards: What Responsible Deployment Looks Like¶
The failures above share a common structure: consequential decisions are made or heavily influenced by AI systems that lack transparency, mechanisms for challenge, and were never evaluated against the populations they would actually serve. The fixes are demanding, but they are not mysterious.
Human-in-the-loop adjudication must be a genuine requirement, not a formality. This means AI outputs are advisory inputs to a human decision-maker who is trained to understand the tool’s limitations and is not incentivised simply to accept machine recommendations. Targets that reward rapid case resolution create structural pressure to defer to automation; governance structures must counteract this. Our AI Ethics & Governance Reviews are specifically designed to help organisations assess whether their human oversight mechanisms are genuine or nominal - a distinction that matters both legally and ethically.
Culturally aware evaluation datasets should be built in partnership with linguistic communities and civil society organisations that work directly with asylum seekers. Evaluation of NLP tools must include performance disaggregated by language, dialect, nationality, and trauma history before deployment, and must be repeated at intervals after.
Mandatory explanation and contestation channels mean that any AI-influenced outcome must be accompanied by a legible account of the factors that contributed to it, expressed in terms that the applicant and their representative can engage with. There must be a formal process for contesting that account, accessible without requiring legal expertise or resources the applicant does not have.
Part Two: Criminal Justice¶
The Legitimacy Problem¶
Criminal justice is the domain where AI failures carry the clearest coercive power. A miscalibrated recidivism score does not merely inconvenience someone - it can keep them in prison or impose conditions that destabilise their life and employment. A predictive policing algorithm that over-predicts crime in a neighbourhood does not just misallocate patrol resources - it generates the arrests that then train future models to over-predict again.
The use of algorithmic tools across the justice system has expanded significantly over the past two decades, driven by a combination of genuine efficiency pressures, a cultural faith in quantification as objectivity, and commercial interests from vendors who are rarely accountable for the downstream effects of their products. The academic and journalistic record of failures is now substantial.
Case: Predictive Policing and the Feedback Loop¶
Predictive policing systems use historical crime data - typically arrest records and incident reports - to generate risk scores for geographic areas or, in some implementations, individuals. The output is used to direct patrol resources: more officers to high-score areas, potentially more stops and searches, and more resulting arrests.
The structural problem is that arrest data does not measure crime - it measures policing. Communities that have historically been over-policed generate more arrest records, which feed into models that predict more risk, which justify more policing. The system is not learning about crime; it is learning about itself. This feedback loop has been documented in deployments in Chicago, Los Angeles, New Orleans, and a number of European cities, and it produces outcomes that are both empirically invalid and deeply discriminatory.
The legal landscape has responded. The EU AI Act now explicitly prohibits AI-based individual predictive policing based solely on profiling - one of the banned practices enforceable since February 2025. In the UK, the landmark R (Bridges) v Chief Constable of South Wales Police [2020] EWCA Civ 1058 established that public authorities must proactively investigate potential algorithmic bias before deployment, not retrospectively - a precedent with implications well beyond facial recognition. Despite these developments, existing systems in many jurisdictions continue operating without the retrospective review these rulings demand.
Beyond the statistical problem is the question of legitimacy. When a resident of a high-score neighbourhood is stopped or searched, the basis for that interaction is, in part, an algorithm that learned to flag their neighbourhood from data that reflected prior discriminatory policing. The accountability chain that should connect police action to articulable suspicion has been replaced by an opaque output from a commercial system, the details of which are often protected as proprietary information.
Case: Recidivism Scoring in Bail and Sentencing¶
Tools like COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) and its equivalents have been used across the United States and in several other jurisdictions to inform decisions about pre-trial detention, bail conditions, and in some cases sentencing. The ProPublica investigation of COMPAS in 2016, examining records from Broward County, Florida, found that the system’s false positive rate for Black defendants was nearly double that for white defendants: Black defendants who did not reoffend were predicted to be higher risk at significantly greater rates than white defendants in the same situation.
The features these models use are rarely disclosed in detail, but research has shown that proxies for race - neighbourhood, education, family history of incarceration - are frequently included. Since these proxies are correlated with race through decades of discriminatory policy in housing, education, and law enforcement, the model encodes historical structural disadvantage and applies it to individuals as if it were a predictive signal about their future behaviour.
The judicial oversight problem compounds this. Judges using these scores often receive a number or category - “medium risk” - without the feature contributions that drove it, without the error rates for the demographic group the defendant belongs to, and without training adequate to critically assess what they are being given. Studies have found that judges who defer to algorithmic recommendations rather than exercising independent judgment produce worse outcomes and perpetuate bias more reliably than those who treat scores as one input among many. This is the “rubber-stamping” problem that GDPR Article 22 guidance specifically identifies as failing to constitute meaningful human oversight - an issue discussed in detail in our AI regulation overview.
Safeguards: What Responsible Deployment Looks Like¶
Exclude protected proxies and their correlates. Any feature substantially correlated with a protected characteristic requires explicit justification and should be subject to adversarial review. The argument that a proxy is “predictive” does not justify its use if it is predictive partly because it encodes structural disadvantage. Our algorithmic bias audits provide exactly this kind of adversarial scrutiny, working through feature sets to identify indirect discrimination pathways before systems go live.
Conduct mandatory disparate impact audits before and after deployment. Performance metrics - accuracy, false positive rates, false negative rates - must be disaggregated by demographic group, and a decision to deploy must require that disparities fall within pre-specified tolerances. The UK’s Algorithmic Transparency Recording Standard, now mandatory for central government and recommended for all public bodies, provides a useful baseline framework for documentation.
Publish model cards and evaluation documentation. Vendors who sell AI tools to justice system actors should be required to publish detailed model cards specifying training data, feature sets, known limitations, evaluation methodology, and demographic performance breakdowns. Proprietary protection of these details is legally incompatible with their use in consequential decisions. Defendants must have the right to access and challenge the algorithmic basis of decisions made about them.
Require genuine judicial oversight. AI tools should have no automatic authority in the justice system. Scores should be advisory, accompanied by confidence intervals and error rates for the relevant demographic group, and the decision-maker who relies on them should be able to articulate why the score was or was not weighted in their reasoning.
Part Three: Humanitarian Response¶
Data Poverty and the Invisible¶
Humanitarian response - the logistical challenge of delivering aid, shelter, medical care, and protection to people in the acute phase of disaster or conflict - has long attracted interest from the data science community. The problems are real and significant: resources are scarce, needs are vast and geographically dispersed, and the window for effective intervention is often narrow. AI tools for needs assessment, supply chain optimisation, and beneficiary registration hold genuine promise.
The critical challenge is that the populations most in need of humanitarian assistance are often the least represented in the data that AI systems train and operate on. Communities that are geographically remote, linguistically marginalised, or deliberately displaced are not generating digital footprints at the rate of connected urban populations. When resource allocation algorithms are trained on connectivity patterns, mobile data, or administrative records, they systematically underestimate need in precisely the places where need is greatest.
This problem sits at the heart of why TechEthics’ Atlas conflict mapping platform is designed around geospatial and community-level data sources rather than digital footprint proxies - recognising that the communities who need early warning most urgently are those whose signals are hardest to detect through conventional data pipelines.
Case: Resource Allocation and the Unconnected¶
During several large-scale displacement crises, AI-assisted needs assessment tools were piloted by humanitarian organisations. These tools aggregated mobile phone data, satellite imagery, and social media signals to map population movement and estimate resource requirements. The outputs informed decisions about where to position food distribution points, medical teams, and shelter materials.
Post-distribution monitoring consistently showed that the populations furthest from distribution points - those who walked multiple days to access aid, those in areas with no mobile coverage, those from communities that did not use formal mobile networks - had been systematically under-counted by the algorithmic assessment. Resources were directed toward populations the model could see, not necessarily toward those with the greatest need. The model had operationalised connectivity as a proxy for presence and visibility as a proxy for need.
This is not a technical edge case - it is a predictable consequence of training data that reflects the digital divide. Addressing it requires conflict-sensitive technology design that begins with explicit interrogation of whose data is absent from the training set, and what that absence means for the outputs. It also requires institutional willingness to prioritise the unconnected, which is not always present when speed and scale are the dominant operational pressures.
Case: Facial Recognition at Checkpoints and the Cost of False Positives¶
Facial recognition has been piloted in humanitarian settings for beneficiary verification - preventing duplicate registration and ensuring aid reaches intended recipients. The efficiency rationale is genuine: registration fraud is a real problem that diverts resources from intended beneficiaries.
The asymmetry of error costs has not been adequately reckoned with. In commercial facial recognition deployments, a false positive - incorrectly matching an individual to a record - is typically a minor inconvenience. In a humanitarian checkpoint context, a false positive match to a watchlist or a deduplication error can mean denial of food or medical care, detention for questioning, or, in conflict contexts, exposure to armed actors. The tolerance for false positives appropriate in a smartphone unlock system is several orders of magnitude higher than what is appropriate at a food distribution point in an active conflict zone.
Systems trained predominantly on faces from North American and European datasets perform measurably worse on darker-skinned faces, as documented extensively in research from the MIT Media Lab and NIST. The populations served by humanitarian operations - predominantly from Africa, the Middle East, South Asia, and Southeast Asia - are exactly the populations for whom commercial facial recognition systems have the highest error rates. The EU AI Act’s prohibition on real-time biometric identification in public spaces except under strictly limited conditions, now enforceable, is relevant even in humanitarian contexts: the humanitarian label does not exempt a system from the obligation to perform reliably on the people it is processing.
Safeguards: What Responsible Deployment Looks Like¶
Data minimisation should be the default stance in humanitarian AI. The question is not “what data could we collect and use?” but “what is the minimum data required to achieve the specific operational purpose?” This matters both for privacy - humanitarian data in conflict zones can endanger lives if accessed by the wrong parties - and for model validity, since minimal and targeted data collection is more likely to be representative than comprehensive collection that systematically excludes the most vulnerable.
Offline-first design ensures that systems degrade gracefully when connectivity is unavailable, rather than excluding entire populations when the system cannot reach them. Critical functions - registration, verification, resource allocation - must operate without continuous connectivity, with synchronisation as a supplement rather than a requirement. TechEthics’ bespoke development approach explicitly prioritises solutions designed for complex, real-world environments where infrastructure cannot be assumed.
Fallback manual processes must be designed, resourced, and tested before deployment, not improvised when the system fails. Humanitarian operations in crisis conditions will always encounter system failures; the contingency plan must be as robust as the primary system.
Independent humanitarian ethics review, distinct from standard organisational ethics review, should assess AI deployments against specialised frameworks - including the humanitarian principles of humanity, neutrality, impartiality, and operational independence - before deployment and at defined intervals thereafter. Our Conflict & PeaceTech Advisory service is designed precisely for this kind of contextual review, drawing on decades of experience in post-conflict and fragile-state settings.
Part Four: Cross-Cutting Protections¶
The domain-specific failures above share deeper structural causes. Addressing them requires governance measures that cut across all AI deployments in high-stakes contexts involving vulnerable populations.
Harm Thresholds and Kill Switches¶
Every consequential AI deployment should be accompanied by pre-specified harm thresholds: defined conditions under which the system is automatically suspended or rolled back pending investigation. These thresholds should be set during the design phase, not retrospectively. They should include demographic performance disparities beyond defined tolerances, error rates above specified limits, and any pattern of outcomes that diverges significantly from baseline expectations established during evaluation. The EU AI Act’s risk management requirements under Article 9 - which mandate continuous monitoring and iterative risk assessment throughout a system’s lifecycle - formalise this principle as a legal obligation for high-risk systems. Our governance framework development service helps organisations design these mechanisms before deployment, rather than reaching for them in crisis.
Participatory Design with Affected Communities¶
The most consistently valuable insight about how AI systems will fail in a given context comes from the people who live in that context. Participatory design - engaging affected communities as genuine co-designers, not as focus group subjects or checkbox consultees - surfaces failure modes that technical teams will not identify in the lab. For asylum systems, this means working with refugee community organisations and legal aid providers from the earliest stages of specification. For justice tools, it means involving public defenders, impacted community groups, and formerly incarcerated people. For humanitarian tech, it means systematic partnership with local NGOs and community leaders.
This is not simply an ethical commitment. It is an epistemic one. Affected communities are the most reliable source of information about how a system’s assumptions will fail in the specific context where it is deployed. Excluding that knowledge produces worse systems. TechEthics’ co-design approach is built on this principle - every platform we develop is designed with end users rather than for them, with accessibility and data protection built in from the ground up rather than retrofitted. Our DialogAI platform extends this further, providing structured digital consultation infrastructure for facilitated dialogue and real-time consensus detection in exactly the kinds of divided or fragile settings where participatory design is most important and most challenging.
Monitoring, Logging, and Redress¶
Consequential decisions made with AI assistance must be logged in sufficient detail to allow retrospective audit. This means preserving the model version, input data, the output, and the human decision that followed, associated with each case. Without this, identifying systematic errors is impossible, accountability is illusory, and learning from failures cannot happen. The UK’s Algorithmic Transparency Recording Standard and the EU AI Act’s technical documentation requirements under Article 11 both operationalise this principle - though compliance remains uneven in practice.
Alongside logging, affected individuals and communities must have meaningful access to redress: the ability to appeal AI-influenced decisions, to access the reasoning behind them, and to receive compensation where harm is documented. The cost of building redress mechanisms is far lower than the cost of repairing institutional trust after a pattern of unexplained harm becomes public.
The Role of Vendors and Procurement¶
A significant share of AI tools used in these contexts are procured from commercial vendors rather than built in-house. Procurement processes have rarely been adequate to the risks. Vendor contracts should specify evaluation requirements, performance standards disaggregated by demographic group, disclosure obligations, and liability for documented harm. Proprietary protection should not extend to the information required for meaningful accountability. As our AI regulation guide notes, the enforcement trend is clear: regulators across jurisdictions are actively penalising AI systems that violate fundamental rights, and the accumulated penalties from cases like Clearview AI make robust procurement governance a business necessity, not just an ethical aspiration.
Organisations that lack in-house capacity to conduct this level of vendor due diligence can commission ethical impact assessments from independent specialists. This is consistently more effective than relying on vendor-provided documentation alone.
Conclusion¶
The argument for AI in refugee processing, criminal justice, and humanitarian response is not wrong - it is incomplete. These systems face genuine capacity and consistency problems, and thoughtfully designed AI can help address them. But the same power asymmetries that make these populations vulnerable to other forms of institutional harm make them vulnerable to algorithmic harm as well. They have less capacity to identify when a system has erred, less leverage to demand correction, and less ability to absorb the consequences of mistakes.
The safeguards described here - human oversight with genuine authority, culturally aware evaluation, transparency and contestation, participatory design, harm thresholds, and meaningful redress - are practical requirements that distinguish responsible from irresponsible deployment. They cost more at the design phase. They cost less than repairing harm after the fact, and incomparably less than the damage done to individuals and to the institutional trust that humanitarian and justice systems depend on.
AI deployed in contact with vulnerable populations must earn the right to scale by demonstrating that it can be governed. That demonstration begins not with a successful pilot, but with a design process that treats the people who will be most affected not as data points, but as the primary stakeholders whose interests the system must serve. At TechEthics, that principle sits at the centre of everything we build - from Veritas and Metis to the advisory work we do with governments, NGOs, and civil society organisations navigating these challenges.
If you are deploying AI in high-stakes contexts and want to ensure your governance frameworks are fit for purpose, get in touch with our team. We offer AI ethics reviews, algorithmic bias audits, conflict-sensitive design consultancy, and bespoke development - all grounded in the same commitment to accountability and dignity that this article argues for.
Related reading: UK and EU AI Regulation: What Organisations Need to Know in 2025 · Misinformation and ‘Fake News’: A Guide to Critical Information Literacy · The Hidden Architects of Division: How Social Media’s Recommendation Engines Shape Our Reality
