The High-Level Expert Group on Artificial Intelligence (AI HLEG), organized under the aegis of the European Commission, has asked the members of the European AI Alliance for contribution on the topics to be discussed at the first workshop of the AI HLEG, one of them concerning Trusted AI.
Here are a few of my thoughts on how trust can be gained.
Note : For the purpose of the present intervention, AI is considered primarily an automated decision system (ADS). The operator is the party deploying the AI/ADS.
Trust
It is generally acknowledged that the future of Artificial Intelligence depends on trust. If we want AI to really benefit people, we need to find a way to get people to trust it. To the divisions running deep in our societies we should not add that between those who trust (and benefit from) AI and those who don't.Trust is a complex and multifaceted phenomenon, context- and domain-specific. I have attempted an operationalization of the concept, accompanied by some specific measures for building trust in AI/ADS, by no means exhaustive.
Value-compliance
We must be mindful about the values reflected in the design and operation of AI/ADS, about what we choose to optimize and automate. We should also acknowledge that apparently neutral data reflect historical or systemic inequality and discrimination. Another topic for consideration and decision concerns the algorithms' role in correcting discrimination and injustice.
The set of principles underlying the functioning of any socially impactful AI/ADS should incorporate and embody fairness and justice. This could mean that the system has some necessary bias built in - e.g., the relaxation of economic or logistical criteria for optimization when they conflict with protection of vulnerable persons and of the environment.
The set of principles underlying the functioning of any socially impactful AI/ADS should incorporate and embody fairness and justice. This could mean that the system has some necessary bias built in - e.g., the relaxation of economic or logistical criteria for optimization when they conflict with protection of vulnerable persons and of the environment.
Values can enter an AI/ADS as rules (e.g. in the justice system the preference is to minimize either false positives - no one wrongfully convicted - or false negatives - no one guilty acquitted; in the design and delivery of public services the preference is either to minimize public expenditures or to maximize coverage) and/or constraints (optimize transportation network, but minimize environmental impact; optimize human resources, but make sure single mothers are not disproportionately fired).
Value-compliance could be enacted as a checklist of predefined, mandatory societal values, to be filled in by the developer with the approach implemented to ensure compliance, and by the operator with the respective verifications. Preferably, the values included in the checklist are supplemented by their operationalization (see examples above), or should at least be operationalizable.
Opt-out
The possibility to be excluded from any data operations and automated decision-making systems and processes should be explicitly stated and implemented as a straightforward, easy, trouble-free, and inexpensive procedure.Challenge and reparations
The operator should provide a clear, straightforward, and affordable avenue for challenging automated decisions. Additionally, the operator has the obligation to respond to the concerns of people who feel they have been incorrectly or unfairly assessed, and to rectify any harmful outcomes.
* * *
There are also two more complex components of trust, namely transparency and accountability; together, they form the foundation for a meaningful implementation of the right to know and challenge automated decisions. Their respective operationalization and implementation is presented further.
Transparency
Disclosure to the public and regulators
The operator should provide a definition of AI/ADS understandable by the general public. It should also explain the purpose of deploying AI/ADS, and (preferably) make explicit the expected performance of the system and the underlying assumptions about its operation.
The operator should disclose their current use of AI/ADS, accompanied by any related self-assessments and outside review processes and their respective results; for proposed use, the disclosure should ideally occur before a system has been acquired. The information should be detailed by types and classes of tools and processes, as well as areas of application (e.g., administrative; medical; with large-scale and long-lasting public impact - urban planning; impacting rights and entitlements - criminal risk assessment, employment, compensation) in order to facilitate review.
Use of an AI/ADS should be clearly indicated on any document (including web pages) that communicates the decision (or any output of the decision-making process) to the affected person / institution / group. It could (should?) include elements to allow the unambiguous identification of the specific algorithm used and possibly the training data set.
AI/ADS description should allow the assessment of:
- Data (sources, quality, bias, prohibited info - race, ethnicity, sexual preferences, etc., or proxies);
- Correctness / appropriateness of mathematical / statistical approach;
- Correct understanding of the subject matter;
- Proper usage, i.e., in the contexts and for the purposes the system has been designed for.
Understandable design
The AI/ADS should be specifically designed to be reviewed. Key among the disclosed information should be the properties that matter to automated decision making (see below, Explainability). Technical standards should be developed to this effect.Explainability
This is a key trust-inducing feature - witness the emerging field of explainable AI (XAI), which aims to create new AI methods that are accountable to human reasoning.
The AI/ADS should be accompanied by a natural language explanation of the decision input, process, and output. Although output is the most relevant, the system can generate the right output for the wrong reasons, which means that continued delivery of error-free decisions is not ensured.
As mentioned in a paper on legal accountability of AI, "explanations are usually required to answer questions like these: What were the main factors in a decision? Would changing a certain factor have changed the decision? Why did two similar-looking cases lead to different decisions?"
Accountability
Accountability is the antithesis of the assumption that AI-based systems and decisions are correct and don’t need to be verified or audited, and the twin concept of "moral outsourcing - the removal of human agency and responsibility from the outcomes of AI/ADS".
I suggest that the responsibility for the consequences of using AI/ADS be assigned to the operator. One key obligation of the operator should be to rectify any harmful outcomes.
Human-in-the-loop
The decision to deploy an AI/ADS, as well as any automated decision of such a system, should link back to a specific human decision-maker.Monitoring and Auditing
The AI/ADS should be regularly audited; the operator should develop meaningful review processes, subject to independent control, that ensure continuous validation and testing, in order to discover, measure, and track impacts over time. This could include the compilation of a record of decisions and the respective context, as well as error rates and magnitudes, available (anonymized) to outside examination. The review process should allow the assessment of procedural regularity, i.e., that the same process was applied in all cases (see above, Explainability).
The review and testing processes should allow external experts to apply the AI/ADS on their own data sets and then collect and analyze the output. The systems should be tested under a variety of conditions (sample size, data quality and completeness/ gaps, different operationalizations of the same concepts), to make sure that it behaves as expected, and to identify the circumstances under which it doesn't perform satisfactorily. Non-restricted access of external experts is necessary in order to compensate for the fact that the potential bugs in the system cannot be reasonably expected to be identified in the development phase. In time, after significant experience is acquired, regulators could demand a set of mandatory tests before certification.