Who is responsible when AI goes wrong?

In 2020, iTutorGroup's hiring software quietly rejected job applicants for being too old, a case later resolved under US age discrimination law, not the EU AI Act. A year earlier, a widely used healthcare algorithm developed by Optum and used by hospitals and insurers was found to identify fewer Black patients for additional care than similarly sick white patients. In both cases, the hard question wasn't whether something had gone wrong. It was who had to answer for it.

The EU AI Act tries to make that less a matter of luck. It places obligations on the organisations that develop, supply, deploy, import, distribute or modify an AI system. The role an organisation plays helps determine which obligations apply to it.

A provider develops an AI system, or has it developed, and puts it on the market under its own name. A deployer is the organisation that uses the AI system: an employer, a hospital, a bank. Importers and distributors can carry obligations too, and a single incident can involve more than one of these roles.

The iTutorGroup case illustrates this distinction, but it was not decided under the EU AI Act. In 2020, the online tutoring company iTutorGroup used hiring software that automatically rejected female applicants aged 55 or older and male applicants aged 60 or older, screening out more than 200 people before anyone noticed. One rejected applicant only found the pattern by resubmitting an identical application with a younger birth date, and got an interview. The US Equal Employment Opportunity Commission sued, and in 2023 iTutorGroup agreed to pay $365,000 and accept at least five years of EEOC monitoring, without admitting wrongdoing. The case ran under US age discrimination law, not the AI Act, but it illustrates why obligations can arise for both the organisation providing a system and the organisation deploying it.

Recruitment, education, essential services and credit assessment are among the uses the Act treats as high-risk, meant to carry its strongest safeguards: risk management, data governance, documentation and human oversight for providers, plus supervision duties for deployers.

Under a 2026 amendment to the Act, the Digital Omnibus on AI, the relevant high-risk requirements for systems covered by these rules now apply from 2 December 2027, while high-risk AI embedded in regulated products follows on 2 August 2028. Other parts of the Act, including its bans on certain practices and many of its transparency obligations, are already applicable, although some have transitional deadlines.

A widely used healthcare algorithm developed by Optum and used by hospitals and insurers shows why this matters. In 2019, researchers found that the system underestimated the healthcare needs of Black patients because it used past healthcare spending as a proxy for illness. Optum worked with the researchers to modify the algorithm, and the revised approach reduced the disparity identified in the study by 84 per cent. It is a reminder that hospitals and insurers using such systems cannot treat the software as a substitute for their own responsibilities. That is the deployer side of the same coin.

Responsibility can also shift along the supply chain. Under Article 25, a distributor, importer, deployer or other third party can be treated as a provider of a high-risk system in certain circumstances. An organisation that substantially modifies a high-risk system, or changes the intended purpose of an AI system that was not classified as high-risk so that it becomes high-risk, can become the provider for the purposes of the Act and be subject to the provider obligations under Article 16.

For example, imagine a distributor takes an AI system originally designed for a general purpose and adapts its intended purpose to assess people's creditworthiness. If that change makes the system high-risk under Article 6, the distributor can become the provider under Article 25 and be subject to the provider obligations under Article 16. The point is simple: changing what an AI system is intended to do can also change who is legally responsible for it.

Regulators have real enforcement power. For prohibited practices, fines can reach €35 million or 7 per cent of worldwide turnover, whichever is higher. Other breaches, including high-risk violations once that chapter applies, can bring fines of up to €15 million or 3 per cent of turnover, under Article 99. People can complain to their national authority if they believe the Regulation has been breached, and in some cases can ask what role a high-risk system played in a decision made about them.

That is a separate, usually older, question. The AI Act regulates how AI systems are developed and used; other laws determine whether a person is entitled to compensation.

Whether someone can be compensated, and by whom, is usually a separate question for product liability, contract, employment, or anti-discrimination law. The first of these is being updated under the EU's own Product Liability Directive to cover software and AI, though member states have until December 2026 to write it into national law. The iTutorGroup case above, however, was resolved under US employment discrimination law.

iTutorGroup paid $365,000 and accepted at least five years of EEOC monitoring for a rule its own software applied without a human ever reviewing it.

• EU AI Act, Regulation (EU) 2024/1689
• Digital Omnibus on AI, Regulation (EU) 2026/1744
• Product Liability Directive, Directive (EU) 2024/2853
• EEOC v. iTutorGroup, Inc., No. 1:22-cv-02565 (E.D.N.Y. 2023)
• Obermeyer et al., 'Dissecting Racial Bias in an Algorithm Used to Manage the Health of Populations,' Science 366, no. 6464 (2019)