Measuring and monitoring the societal impact of AI



Nations that rise ahead in the AI revolution will be those that not only harness the advantages of AI, but also have an effective plan for managing the societal disruption that it will bring.
(Strategic Competition in an Era of Artificial Intelligence)
I would like to start with some general considerations.
First, the societal impact of AI is not directly measurable, except maybe for a few relatively uninteresting aspects (investments, number of people working in AI-related fields, number of deployed AI, energy consumption of AI systems, etc); the most politically relevant forms and facets of this impact are to be inferred analytically. Some of the impact might very well be qualitative, subtle, like a change in the mechanism of influence of one societal dimension upon another. Last but not least, when assessing the impact we should step beyond the aggregate effects, otherwise we risk disregarding the distributional aspects of rewards and loses, the fact that AI produces both winners and victims.
Second, in order to measure impact, we have to decide what is the input. Thus, we have to agree on a definition / extension of AI, because this is definitely not an objective, homogeneous, undifferentiated class of existence. For example, do we include chatbots, smart contracts, routing software (Waze), and Google Translate? We should also keep in mind that the generally accepted view of what constitutes AI might change over time : once we reach the stage of semantic AI (capable of discerning the meaning of words, objects, and actions), will we still consider the old non-semantic face recognition systems, chatbots, or autonomous cars as intelligent?

Third, there are several sides / components of the input, which generate different forms and mechanisms of impact on society :
(1) Objective side (tools, explicit instructions, process knowledge)
(2) Semantic / ideological load, the meaning associated, the narratives that establish the background, role, and character of technology, its relationships with other societal facets and phenomena, notably politics. There is meaning given (bestowed by public figures and the intelligentsia), and meaning embedded (the values and ideals, dreams and fears, of the industries / organized groups that develop and implement AI, in which the technology is rooted). This sets the arena in which technology can manifest itself, and shapes the influence mechanisms.
(3) Interaction of "cultural areas", with different visions / perceptions / ideologies / approach regarding AI. The same culture will not be dominant and undisputed worldwide. It appears that the cultural camps are already gaining shape : simplifying a bit, and echoing the official European Commission vision on AI, Europe would position itself as the promoter of "AI for society" (human-centered, ethical, secure, and true to EU core values), a third cultural pole beside and apart from "AI for profit" (US) and "AI for control" (China, where the development and deployment of AI is subordinated to the national rejuvenation project, aimed at "enhancing national power" and authoritarian control of society).

Let's look in turn at first two sides of the input. Both should be considered, in order to allow the direct examination of the interplay between objective and inter-subjective factors; the latter type of factors cannot be dismissed since public perceptions and preferences shape or restrict policy - like the anti-GMO movement, or Germany's decision to abandon nuclear power.
(1) In order to determine the characteristics and magnitude of the objective side of AI, we could (upon agreeing on a definition / extension) conduct a continuous, administrative census. This could be achieved by implementing legislation requiring the disclosure, and possibly registration, of every AI used.  The data is collected by the public administration, in the same way it collects data on SME or vehicles. At a minimum, use of an AI or Automated Decision System should be clearly indicated on any document (including web pages) that communicates the decision (or any output of the decision-making process) to the affected person / institution / group. Such universal disclosure requirements should contribute to the building of public trust. Among the difficulties of this approach is the phenomenon of fauxtomation / "Potemkin AI", meaning that algorithms are given undue credit, and processes are misrepresented as AI to devalue the human labor that is actually doing the work.
(2) Let's start with a brief example of semantic load and its potential impact. If we perceive AI as inherently benign and trustworthy, we are more likely to embrace it (higher speed and scope of adoption), to be oblivious or at least permissive regarding inbuilt biases (allowing for the reproduction of inequality and discrimination structures), and to assume that those left behind will also get well in the end (resulting in less support for social security and reskilling policies and programs). If AI is seen as neutral, objectively inevitable, we might relinquish our societal right to shape and direct it (thus perceiving as illegitimate or at least useless the efforts to establish an AI ethics), disregard the stewardship and even entrepreneurial role of the state, and leave the initiative to Silicon Valley and venture capitalists.
The object of analysis here is the public discourse on AI (including, but not limited to, legal texts, articles, speeches, handbooks and other educational material, internal company documents, memoirs and interviews of key figures of technology, advertisements), collected e.g. by web scraping and document collection, and de/re-coded through content analysis and hermeneutics. This is arguably the most work-intensive part, but I would say it is crucial because it would illuminate the mechanisms of influence, and elucidate differences between cases that are similar on the objective component.
A special category are AI-related laws and regulations : they have a quasi-objective, reified nature (so they can be seen as a complement of the objective side of technology), but they are also the embodiment and balance of value systems and interests.


 A few words now on the impact.
Technology is reflected at personal level in experiences and practices, as well as in attitudes, opinions, value systems, and worldviews. At the level of society, technology can impact the size of the workforce and its skill and demographic composition, labor and demographic mobility, forms of sociability and patterns of human settlement, consumption and crime. It can also force us to redefine fundamental concepts as work, family, life, and death.
At personal level, the experience and impact of new technologies is mediated and moderated by personal resources and constraints, such as skills and material resources (safety net). Subjective factors that also shape the individual or group response, like the general outlook (active or passive, optimistic or pessimistic, assertive or submissive, future- or past-oriented) can be inferred from observable consequences / circumstances, like experience of a technology-rich environment (see below), meaning the existence of a long-term pattern of the individual encountering and dominating technology.
More directly, the impact of new technologies, and AI in particular, can be seen in forms like :
  • Job polarization (decrease in number and incomes of middle-skill, middle-wage, routine occupations jobs, and widening inequalities between those who have access to good-quality and skills-intensive work and those who end up being low-paid employees in inferior jobs)
  • Technological unemployment (societal inability to invent new jobs faster than technology is destroying the old ones)
  • Task automation and human replacement / Change in labor status due to automation
  • Job quality (working time and work-life balance, job security, flexibility, autonomy and control over schedule, task variety and complexity).
Experience of a technology-rich environment, as mediating and moderating factor, can be measured by specific experiences, of by possession / use of specific items or services
  • In daily life : self-driving car, smart & connected appliances, household digital hub, robotic child / elder care
  • At work : regular human-machine interaction : with smart robots (design/development, supervision, training, service, collaborative robots) or AI (in medicine, services, administration), new occupations (robot coach, image labeller, algorithmization and automation of non-routine cognitive jobs)
All these can be measured by business and citizen surveys; Big Data and mobile apps could be used to investigate e.g. mobility and sociability patterns, or time use.

Of course, this is just a general sketch; concept refinement and operationalization is necessary before actual work can begin.

To sum up, measuring and monitoring the societal impact of AI is not going to be easy, fast, or cheap. We should aim, as well acknowledged by the HLEG request, to produce a multidimensional picture, where numbers are complemented by narratives illuminating the mechanisms of influence - which means, among other things, that a synthetic indicator or even a scoreboard, although attractive for policymakers, would be rather myopic and simplistic.
The key data collection methods, as already mentioned by other AI Alliance members, would be : an administrative AI census, business and citizen surveys, Big Data, web scraping and document collection. To all these I would suggest adding a funding line for small-scale, specialized research : ethnographic and organizational studies, psychological and behavioral economics experiments, medical and neurological investigations, econometric modeling.

I hope the information coming out of this project will acquire the same importance as macroeconomic data or the statistics underlying the Social Pillar. The information delivered should inform concerted policy and political response at EU level, therefore the raw data should be of verifiable quality and sustainable (continuously available to institutions). As a consequence, we cannot rely on data generated spontaneously by businesses and general public, but the AI monitoring project should be included in the EU statistical programme, supported by adequate legislation and funding.