Processes of algorithmic decision-making (ADM) now evaluate people in many areas of life. To take advantage of the opportunities ADM offers in the area of participation, one overall goal must be set when ADM processes are planned, designed and implemented: ensuring that participation actually increases. If this is not the case, the use of these tools could in fact lead to greater social inequality.

In sum, the opportunities and risks in the examples presented in our working paper “When Machines Judge People – International Case Studies of Algorithmic Decision-making” point to a number of general factors related to ADM processes that can critically affect participation. These factors involve different aspects of the overall socio-informatic process and can be found on different levels. Here are three examples:

  • Shaping ADM processes on the micro and macro level: Choosing data and setting criteria at the startof a development process can themselves reflect normative principles which sometimes touch on fundamental social issues.
  • Structure of suppliers and operators on the macro level: Having a range of ADM processes andoperators can increase participation (e.g. through credit assessments of people who have not been partof the system in the past), can make it easier to avoid the ADM process and can expand possibilities forfalsification. Conversely, monopolistic structures increase the risk that individuals will “fall out of the system” and get left behind.
  • Use of ADM forecasts on the micro, meso and macro level: The interplay of technology, society andindividuals has a major impact on how and when algorithms are used and the influence they thus have.Key questions that must therefore be asked are: How do people (ADM developers and users, and thegeneral public) deal with automated predictions? Do the processes include the possibility of challengingADM results?

Criteria are needed for determining the benefits of ADM processes on all levels and in all steps – from the definition of the goals and the efforts to measure the issues at hand, to data collection, the selecting of algorithms and the embedding of processes in the relevant social context.

The nine corrective responses discussed below can provide initial impetus for addressing these issues:

1. Ensure falsifiability

ADM processes can learn asymmetrically from mistakes. “Asymmetric” means that the system, by virtue of the design of the overall process, can only recognize in retrospect certain types of its own predictions which proved incorrect. When algorithms learn asymmetrically, the danger always ex-ists that self-reinforcing feedback loops will occur.

Example: Recidivism predictions used in the legal system

2. Ensure proper use

Institutional logic can lead to ADM processes being used for completely different purposes than originally envisioned by their developers. Such inappropriate uses must be avoided.

Example: Predicting individual criminal behavior

3. Identify appropriate logic model for social impact

Algorithm-driven efficiency gains in individual process steps can obscure the question of whether the means used to solve a social problem are generally appropriate.

Example: Predicting lead poisoning

4. Make concepts properly measurable

Social phenomena or issues such as poverty and social ine-quality are often hard to operationalize. Robust benchmarks developed through public discussion are therefore helpful.

Example: Predicting patterns of poverty

5. Ensure comprehensive evaluation

The normative power of what is technically feasible all too easily eclipses the discussion of what makes sense from a social point of view. For example, the scalability of machine-based decisions can quickly lead to situations in which the appropriateness and consequences for society of using ADM processes have neither been debated nor verified.

Example: Automatic face-recognition systems

6. Ensure diversity of ADM processes

Once developed, the decision-making logic behind an ADM process can be applied in a great number of instances with-out any substantial increase in cost. One result is that a limited number of ADM processes can predominate in certain areas of application. The more extensive the reach, the more difficult it is for individuals to escape the process or its conse-quences.

Example: Preselection of candidates using online personality tests

7. Facilitate verifiability

Frequently, no effort is made to determine if an ADM process is sufficiently fair. Doing so is even impossible if the logic and nature of an algorithm is kept secret. Without verification by independent third parties, no informed debate on the opportu-nities and risks of a specific ADM process can take place.

Example: University admissions in France

8. Consider social interdependencies

Even when use is very limited, the interdependences between ADM processes and their environment are highly complex. Only an analysis of the entire socio-informatic process can reveal the relationship between opportunities and risks.

Example: Location-specific pre-dictions of criminal behavior

9. Prevent misuse

Easily accessible predictions such as scoring results can be used for inappropriate purposes. Such misuse must be pre-vented at all costs.

Example: Credit scoring in the US

This is an excerpt from the working paper “When Machines Judge People – International Case Studies of Algorithmic Decision-making”, written by Konrad Lischka and Anita Klingel, published by the Bertelsmann Stiftung under CC BY-SA 3.0 DE.

This publication documents the preliminary results of our investigation of the topic. We are publishing it as a working paper to contribute to this rapidly developing field in a way that others can build on.