Empowerment Week 1: Automation Bias¶

How to use this notebook

Simply read the text and follow the instructions.
This notebook contains code cells, which can be modified and must be executed to see the result of their content.
To execute a cell, select it and click on the play button (▶) in the tool bar, or type Shift + Enter or Ctr + Enter.

As the variables contained in a cell are stored in memory, the order of execution of the cells is important!

Notebook by Mattéo Berthet, Cécile Hardebolle and the Responsible Software TAs team (2025).
Except where otherwise noted, the content of this notebook is licensed under a Creative Commons Attribution International License (CC BY 4.0 International).
Creative Commons License

Introduction¶

Welcome to this week's exercise session of CS-290 Responsible Software!

We start this Empowerment 1 module with an exploration of the issue of automation bias, that is to say the tendency of humans to over-trust and over-rely on automated systems.
For this, we are going to reproduce a simplified version of the study published in 2020 (which you don't need to read): Suresh, H., Lao, N., & Liccardi, I. (2020). Misplaced Trust: Measuring the Interference of Machine Learning in Human Decision-Making. 12th ACM Conference on Web Science, 315–324. https://doi.org/10.1145/3394231.3397922

Let's get started!

Learning Goals

What will be covered:

  • Part 1: The concept of automation bias on human decision-making for a specific task.
  • Part 2: Genreralization of this concept to other tasks.
  • Part 3: The impact of automation bias on medical diagnosis.

By the end of the session you will be able to:

  • ✅ Explain how recommendations from software can influence human decisions (in particular in terms of performance)
  • ✅ Describe the phenomenon of automation bias: how it manifests itself, when and for whom
  • ✅ Identify risks related to automation bias in real-life applications

Part 1: Crowd analysis¶

In this first part, we are analyzing how individuals respond to recommendations from a Machine Learning model in a specific tasks: identifying the number of people in a crowd.

Crowd Analysis Task:
For this study, individual participants had to respond to a series of questions, where each question displayed a pair of crowd images and participants were asked to identify which image had the most people.
The image selected by the participant could be either correct i.e., had the most people (scored 1) or incorrect (scored 0).

ML recommendations:
For some of the questions (not all), participants received recommendations from a ML model.
These recommendations could be either correct (the ML model provided the right answer) or incorrect (the ML model provided the wrong answer).
However, participants were not informed of the correctness of the ML recommendations nor how accurate the ML was.

The interface in the experiment looked like the image below (for this example the recommendation from the ML model is shown):

No description has been provided for this image

Results:
There was a total of 20 questions and the results of the experiment are presented for 4 groups of questions:

  • 5 questions with a recommendation from the ML model that is correct: these questions are denoted Cor Rec
  • 5 questions with a recommendation from the ML model that is incorrect: these questions are denoted Incor Rec
  • 5 questions where the recommendation of the ML model was correct but not shown: these questions serve as control, they are denoted Cor NoRec
  • 5 questions where the recommendation of the ML model was incorrect but not shown: these questions serve as control, they are denoted Incor NoRec.

The score of the participants is presented by question group, and can take values between 0 (no question answered correctly) to a maximum of 5 (all questions from the question group answered correctly).

Expertise levels:
One of the goals of the study by Suresh et al. was to determine if there is a difference in the responses between experts and beginners and to evaluate whether the recommendations from the ML model influence their answers in the same way, especially when the recommendations are incorrect. Therefore participants were categorized into two groups based on their expertise:

  • Experts: Individuals with extensive knowledge of ML and strong mathematical skills.
  • Beginners: Individuals with limited knowledge of ML and less advanced mathematical skills.

1.1 - Dataset exploration¶

Instructions

Execute the cell below to load the libraries we need in the notebook.

In [1]:
%load_ext autoreload
%autoreload 2

import otter
test = otter.Notebook()

import pandas as pd
import numpy as np
import math
import os
import warnings
from copy import deepcopy
import pandas.testing as pdt

from res.utils import *

np.random.seed(42)
warnings.filterwarnings("ignore")

Instructions

Execute the cell below to load the dataset.

In [2]:
# Load the data
path_notebook = os.getcwd()
df = pd.read_csv(os.path.join(path_notebook, 'dataset', 'automation_bias.csv'))

crowds_df = df.copy()
crowds_df = crowds_df.filter(regex='Crowds|Expertise')
crowds_df.head()
Out[2]:
Crowds Cor Rec Crowds Incor Rec Crowds Cor NoRec Crowds Incor NoRec Expertise
0 4 1 2 4 Beginner
1 4 1 4 4 Beginner
2 5 2 3 3 Expert
3 5 2 4 3 Beginner
4 5 2 3 2 Beginner

Reflection time

  • What does each row represent?
  • What do the numbers in the columns Crowds Cor Rec, Crowds Incor Rec, Crowds Cor NoRec, Crowds Incor NoRec represent?

Feedback - Click on the "..." below only once you have really tried to answer the question!

Each row represents a participant, with her score for each question group and her level of expertise.

The numbers represent the score from each participant i.e. the number of questions they answered correctly, between 0 and 5, for each group of 5 questions:

  • Cor Rec: 5 questions with a recommendation from the ML model that is correct
  • Incor Rec: 5 questions with a recommendation from the ML model that is incorrect
  • Cor NoRec: 5 questions where the recommendation of the ML model was correct but not shown (serve as control)
  • Incor NoRec: 5 questions where the recommendation of the ML model was incorrect but not shown (serve as control)

1.2 - Performance with and without ML recommendations¶

You can refer to crowds_df as the dataset containing the results:

In [3]:
crowds_df.head()
Out[3]:
Crowds Cor Rec Crowds Incor Rec Crowds Cor NoRec Crowds Incor NoRec Expertise
0 4 1 2 4 Beginner
1 4 1 4 4 Beginner
2 5 2 3 3 Expert
3 5 2 4 3 Beginner
4 5 2 3 2 Beginner

We would like to understand how the ML recommendations influence the participants performance i.e. how often the participants answered correctly, depending on whether they see the recommendation for the ML model or not and whether the recommendation from the ML model is correct or not.

Instructions

For each category of question (Cor Rec, Incor Rec, Cor NoRec, Incor NoRec), compute the performance rate of participants i.e. the average percentage of questions answered correctly.

The goal is to obtain the dataframe performance_group_category_df with the following structure:

NoRec Rec
Crowds Cor ... ...
Crowds Incor ... ...

Complete the cell below.

In [4]:
# Compute the values
performance_category_cor_rec = df["Crowds Cor Rec"].mean() * 100 / 5 # SOLUTION
performance_category_incor_rec = df["Crowds Incor Rec"].mean() * 100 / 5 # SOLUTION
performance_category_cor_no_rec = df["Crowds Cor NoRec"].mean() * 100 / 5 # SOLUTION
performance_category_incor_no_rec = df["Crowds Incor NoRec"].mean() * 100 / 5 # SOLUTION

# Bundle the values into a dataframe
performance_crowds_category_df = pd.DataFrame({
    "NoRec": [
        performance_category_cor_no_rec,
        performance_category_incor_no_rec,
    ],
    "Rec": [
        performance_category_cor_rec,
        performance_category_incor_rec,
    ]
}, index=["Correct", "Incorrect"])

Check your code with the following tests.

In [6]:
test.check("performance_category")
Out[6]:

performance_category
passed! 🚀

Instructions

Execute the cell below to plot the results.

In [7]:
plot_performance_group(performance_crowds_category_df, data_type="crowds")

performance_crowds_category_df
Out[7]:
NoRec Rec
Correct 68.533333 90.533333
Incorrect 69.466667 38.933333
No description has been provided for this image

Reflection time

  1. What influence do correct and incorrect ML recommendations have on the participants' performance?
  2. Which seems to have the greater imapact?

Feedback - Click on the "..." below only once you have really tried to answer the question!

Participants perform better when provided with a correct recommendation but worse when provided with an incorrect recommendation.

For the questions where the ML model had a correct recommendation, seeing the recommendation improved the performance rate by 22%.

For the questions where the ML model had an incorrect recommendation, seeing the recommendation reduced the performance rate by 30%.

Incorrect recommendations have a greater impact on the performance rate.

We would like to understand how could the incorrect ML recommendations influence very negatively the participants' performance. We could ask ourserveles if it comes from the expertise of the participants? Maybe the beginners rely too much on the ML recommendations?

1.2 - The influence of expertise¶

We now want to explore the influence of the level of expertise of the participants.

Instructions

Execute the cell below to get the average score of participants by question category and by level of expertise.

In [8]:
crowds_expertise_df = crowds_df.groupby("Expertise").mean()
crowds_expertise_df
Out[8]:
Crowds Cor Rec Crowds Incor Rec Crowds Cor NoRec Crowds Incor NoRec
Expertise
Beginner 4.520000 1.493333 3.533333 3.640000
Expert 4.533333 2.400000 3.320000 3.306667

Instructions

For this analysis, we will focus on incorrect recommendations only. From the crowds_expertise_df dataframe, extract the average score for incorrect recommendations (Incor Rec and Incor NoRec) by expertise level and compute the corresponding performance rate i.e. the percentage of questions answered correctly.

The goal is to obtain the dataframe performance_crowds_incorrect_df with the following structure:

NoRec Rec
Beginner ... ...
Expert ... ...

Complete the cell below.

In [9]:
# Compute the values
performance_incorrect_no_rec_begginer = (crowds_expertise_df["Crowds Incor NoRec"] / 5)[0] * 100 # SOLUTION
performance_incorrect_no_rec_expert = (crowds_expertise_df["Crowds Incor NoRec"] / 5)[1] * 100 # SOLUTION
performance_incorrect_rec_begginer = (crowds_expertise_df["Crowds Incor Rec"] / 5)[0] * 100 # SOLUTION
performance_incorrect_rec_expert = (crowds_expertise_df["Crowds Incor Rec"] / 5)[1] * 100 # SOLUTION

# Bundle the results into a dataframe
performance_crowds_incorrect_df = pd.DataFrame({
    "NoRec": [performance_incorrect_no_rec_begginer, performance_incorrect_no_rec_expert],
    "Rec": [performance_incorrect_rec_begginer, performance_incorrect_rec_expert]
}, index=["Beginner", "Expert"])

Check your code with the following tests.

In [11]:
test.check("performance_incorrect")
Out[11]:

performance_incorrect
passed! 🍀

Instructions

Execute the cell below to plot the results.

In [12]:
plot_performance_incorrect(performance_crowds_incorrect_df)

performance_crowds_incorrect_df
Out[12]:
NoRec Rec
Beginner 72.800000 29.866667
Expert 66.133333 48.000000
No description has been provided for this image

Reflection time

  1. Does the poor performance of the participants with incorrect recommendations only come from beginners?
  2. Are participants able to trust themselves or do they rely on the recommendations from the ML model?

Feedback - Click on the "..." below only once you have really tried to answer the question!

For both groups, performance drasticly decreases when they are provided with an incorrect recommendation. For the beginner group, the performance rate goes from 73% to 30% when provided with a recommendation. For the expert group, it goes from 66% to 48%. Even if the expert group is less impacted by the incorrect recommendation, they still are.

Participants rely too much on the recommendations of the ML model. They should be more critical about the system output.

1.3 - Agreement rates¶

We can guess that participants are greatly infuenced by the ML recommendations. We would like to evaluate to which extent participants tend to agree ML with the recommendations depending on their expertise.

Agreement is defined as follows:

  • When the recommendation from the ML model is correct (shown or not), agreement means that the participants had the correct answer and so agreement = score.
  • When the recommendation is incorrect (shown or not), agreement means that the participants had the incorrect answer, and so in this case agreement = 5 - score (since the score reflects the number of correct answers).

From there, the agreement rate is the percentage of questions where the participant agreed with the recommendation of the ML model.

Agreement rate of one participant¶

Let's take as an example the first participant of the dataframe crowds_df, stored in the first_participant variable below (don't forget to execute the cell to have the variable defined!):

In [13]:
first_participant = crowds_df.head(1).copy().iloc[0]
crowds_df.head(1)
Out[13]:
Crowds Cor Rec Crowds Incor Rec Crowds Cor NoRec Crowds Incor NoRec Expertise
0 4 1 2 4 Beginner

Instructions

Compute the agreement rate of this participant with the ML model:

  • for NoRec (recommendation not shown) questions,
  • for Rec (recommendation shown) questions.

Remember to compute the agreement differently depending on whether the participant had the correct answer (in which case agreement = score) or incorrect answer (in which case agreement = 5 - score).

Express the agreement rate as a percentage of the number of questions.

In [14]:
first_particpant_agreement_no_rec = (first_participant["Crowds Cor NoRec"] + 5 - first_participant["Crowds Incor NoRec"])/10*100 # SOLUTION
first_particpant_agreement_rec = (first_participant["Crowds Cor Rec"] + 5 - first_participant["Crowds Incor Rec"])/10*100 # SOLUTION

print("The agreement rate for the first participant without ML recommandation is ", first_particpant_agreement_no_rec, "%", "and with ML recommandation is ", first_particpant_agreement_rec, "%") 

The agreement rate for the first participant without ML recommandation is  30.0 % and with ML recommandation is  80.0 %

Check your code with the following tests.

In [16]:
test.check("agreement_first_participant")
Out[16]:

agreement_first_participant
passed! 🚀

Agreement rate by level of expertise¶

Now that you successfully computed the agreement rate for the first participant, the goal is to compute the agreement rate depending on the expertise level of the participants.

For this analysis we are going to start from the crowds_expertise_df dataframe, which you can see again below:

In [17]:
crowds_expertise_df
Out[17]:
Crowds Cor Rec Crowds Incor Rec Crowds Cor NoRec Crowds Incor NoRec
Expertise
Beginner 4.520000 1.493333 3.533333 3.640000
Expert 4.533333 2.400000 3.320000 3.306667

Instructions

From the crowds_expertise_df dataframe, extract the average score for the 4 categories of questions by expertise level and compute the corresponding agreement rate i.e. the percentage of questions where participants agreed with the model.

Remember to compute the agreement differently depending on whether the participants had the correct answer (in which case agreement = score) or incorrect answer (in which case agreement = 5 - score).

The goal is to obtain the agreement_crowds_expertise_df dataframe with the following structure:

NoRec Rec
Beginner ... ...
Expert ... ...

Complete the cell below.

In [18]:
# Compute the agreement for the NoRec and Rec categories of questions
agreement_expertise_no_rec =  (crowds_expertise_df["Crowds Cor NoRec"] + 5 - crowds_expertise_df["Crowds Incor NoRec"]) / 10 * 100 # SOLUTION
display(agreement_expertise_no_rec)
agreement_expertise_rec =  (crowds_expertise_df["Crowds Cor Rec"] + 5 - crowds_expertise_df["Crowds Incor Rec"]) / 10 * 100 # SOLUTION
display(agreement_expertise_rec)

# Extract the agreement rates by expertise
agreement_begginer_no_rec = agreement_expertise_no_rec[0] # SOLUTION
agreement_expert_no_rec = agreement_expertise_no_rec[1] # SOLUTION
agreement_begginer_rec = agreement_expertise_rec[0] # SOLUTION
agreement_expert_rec = agreement_expertise_rec[1] # SOLUTION

# Bundle the results into a dataframe
agreement_crowds_expertise_df = pd.DataFrame({
    "NoRec": [agreement_begginer_no_rec, agreement_expert_no_rec],
    "Rec": [agreement_begginer_rec, agreement_expert_rec]
}, index=["Beginner", "Expert"])

Expertise
Beginner    48.933333
Expert      50.133333
dtype: float64
Expertise
Beginner    80.266667
Expert      71.333333
dtype: float64

Check your code with the following tests.

In [20]:
test.check("agreement_expertise")
Out[20]:

agreement_expertise
passed! 💯

Instructions

Execute the cell below to plot the results.

In [21]:
plot_agreement_rate_expertise(agreement_crowds_expertise_df)

agreement_crowds_expertise_df
Out[21]:
NoRec Rec
Beginner 48.933333 80.266667
Expert 50.133333 71.333333
No description has been provided for this image

Reflection time

Compare the agreement rate for experts and beginners when provided with a recommandation and whitout one. Which group relies less on the recommandations from the ML model?

Feedback - Click on the "..." below only once you have really tried to answer the question!

When provided with a recommendation, the agreement rate is higher for the beginner group (from 50% to 80%) than for the expert group (from 50% to 71%). The expert group relies less on the ML recommendations.

1.4 - Conclusion¶

Instructions

Execute the cell below to see the graphs of all the results we have obtained so far.

In [22]:
plot_all_subplots(agreement_crowds_expertise_df, performance_crowds_category_df, performance_crowds_incorrect_df, data_type="crowds")
No description has been provided for this image

Reflection time

Based on your analysis of the results, explain the main characteriscits of automation bias (e.g. in which situation does it arise and for whom).

Feedback - Click on the "..." below only once you have really tried to answer the question!

From the results above we can conclude that the main characteristics of automation bias are:

  • Users tend to have more confidence in automated systems than in themselves.
  • Users tend to follow recommendations even when:
    • The output is incorrect
    • They would have made the correct decision on their own

Part 2: Generalization of the results¶

These conclusions we formulated might be only valid for the crowd analysis task. We would like to see if the same conclusions can be drawn for another task or if it is specific to the crowd analysis task.

We now consider a different task: Animal Analysis Questions. Participants were shown images of animals and asked to identify which animal was present in the image out of two close species.

Here is one example image from this task where the two possible choices are Beaver or Marmot:

No description has been provided for this image

We created new columns on the datraframe to include the Animals results.

In [23]:
df.head()
Out[23]:
Crowds Cor Rec Crowds Incor Rec Crowds Cor NoRec Crowds Incor NoRec Animals Cor Rec Animals Incor Rec Animals Cor NoRec Animals Incor NoRec Expertise
0 4 1 2 4 5 1 3 4 Beginner
1 4 1 4 4 4 1 5 3 Beginner
2 5 2 3 3 4 1 4 3 Expert
3 5 2 4 3 5 1 3 1 Beginner
4 5 2 3 2 5 0 3 3 Beginner

Here are the scores by expertise level for the different types of questions:

In [24]:
expertise_df = df.groupby("Expertise").mean()
expertise_df
Out[24]:
Crowds Cor Rec Crowds Incor Rec Crowds Cor NoRec Crowds Incor NoRec Animals Cor Rec Animals Incor Rec Animals Cor NoRec Animals Incor NoRec
Expertise
Beginner 4.520000 1.493333 3.533333 3.640000 4.520000 1.066667 3.040000 3.013333
Expert 4.533333 2.400000 3.320000 3.306667 4.466667 1.213333 2.946667 3.000000

2.1 - Result from the animal analysis task¶

Here are the same plots as in the crowd task but for the animal analysis task:

Instructions

Execute the cell below to plot the results.

In [25]:
plot_animal_all_subplots(df, expertise_df)
No description has been provided for this image

Reflection time

Do you see any differences in the results for this task? Do your conclusions from PART 1 apply also to the animal analysis task?

Feedback - Click on the "..." below only once you have really tried to answer the question!

The automation bias seems to have even more impact on the participants' performance for the animal analysis task. The performance rate decreases by 40% when the participants are provided with an incorrect recommendation (from 60% to 20%) compared to the 30% decrease for the crowd analysis task.

Overall the graphs are very similar and we can conclude that the observations are the same hence our conclusions also apply to this other task.

2.2 - Comparison between the tasks¶

Lets remember that our model gives as many correct as incorrect recommendations in the setup used for this study: on a total of 10 questions with the ML recommendations (Rec) there are 5 questions with correct recommendations and 5 with incorrect recommendations.
This means that, as seen from the partcipants, it has an accuracy of 50%.

Let's plot the performance again with animal and crowd analysis tasks. We want to better understand the impact of the ML recommendations on the participants' performance.

In [26]:
plot_whole_performance_group(df)
Out[26]:
NoRec Rec
Crowds Cor 68.533333 90.533333
Crowds Incor 69.466667 38.933333
Animals Cor 59.866667 89.866667
Animals Incor 60.133333 22.800000
No description has been provided for this image

Reflection time

  1. How accurate do you think the participants think the model is: more accurate or less accurate than themselves?
  2. What could be the effect(s) of using a more accurate model?
  3. How could we reduce the effect of automation bias?

Feedback - Click on the "..." below only once you have really tried to answer the question!

  1. While particpants know how to answer the questions with a relatively good accuracy by themselves (60% for animals and 70% for crowds), they seem to think the ML model is better than them, which it is not, and hence discard their own judgment.

  2. With a more accurate model, the overall performance would probably improve. However, they would still be less performant when provided with an incorrect recommendation. Even worse, over time they could tend to rely even more on the model and would be even more impacted by the incorrect recommendations on the long term (we will see in the videos that this effect exist and is called "learned carelessness").

  3. One possibility to mitigate automation bias would be to provide participants with more information about the model to make them aware of the type of model and its errors. Another option would be to provide information with the output of the model, like confidence intervals for instance, so that participants can evaluate how much they can trust the recommendation. We could also see if the way we present the recommendation of the model has an effect on the reliance of participants. Another option would be to provide them with training on automation bias.
    NB: all these options have been and are explored by researchers, some of them have an effect (modifying the display, training), others not so much or even have an opposite effect (e.g. confidence information).

Part 3: Implications of the study¶

The two tasks we studied highlighted the presence of automation bias. However these tasks do not really have consequences on the real world.
Let's now focus on a task where ML models are increasingly used and that has real-world consequences: medical diagnosis.

We take the example of mammography, a medical imaging technique used to detect breast cancer.
A ML model could be used to help radiologists in their diagnosis.

We want to understand the impact automation bias could have in this context.

3.1 - Reflection time

  1. When the ML model is correct, what would be the impact on the radiologists' performance?
  2. When the ML model is incorrect, what would be the impact on the radiologists' performance?

Feedback - Click on the "..." below only once you have really tried to answer the question!

  1. When the mammography ML model is correct, the radiologists' performance would improve. They would be able to detect more cases of breast cancer and would be more confident in their diagnosis.
  2. When the mammography ML model is incorrect, the radiologists' performance would decrease. In this case we need to distinguish between two categories of incorrect classifications:
    • False positives where a cancer is diagnosed where there is none,
    • False negatives where a cancer is not detected where it should.
    • While both errors have negative impacts (as we have seen in other notebooks), the false negatives combined with automation bias would mean that radiologists would be at higher risk of missing cases of breast cancer.

3.2 - Reflection time

What is the dilemma of using such a ML model to assist with medical diagnosis?

Feedback - Click on the "..." below only once you have really tried to answer the question!

The dilema is that the radiologist would be overall more performant with the help of the ML model. However, when provided with an incorrect recommendation, while the radiologist would probably be able to detect the error, they would be less confident in their diagnosis and would be more likely to miss cases of breast cancer.

For this part of the notebook, we have derived our analysis from this study:

Dratsch, Chen et. al. (2023). Automation Bias in Mammography: The Impact of Artificial Intelligence BI-RADS Suggestions on Reader Performance. Radiology, 307(4), e222176. https://doi.org/10.1148/radiol.222176

Synthesis¶

It is now time to step back and reflect on what you have discovered in this notebook!

Final reflection time!

Summarize what you have learned from this notebook:

  • Describe the phenomenon of automation bias:
    • What is it?
    • How does it manifests itself?
    • When does it arise and for whom?
  • Explain how automation bias can lead to ethical risks in real-life applications such as medical diagnosis

Feedback - Click on the "..." below only once you have really tried to answer the question!

Automation bias is the tendency of humans to over-rely on / over-trust recommendations from software (more generally automated systems) in the context of decision-making.

Its manifestation is that human decisions are influenced by recommendations in the following ways:

  • The correctness of human decisions varies depending on whether recommendations are shown or not, correctness increases when the recommendations are correct but decreases when the recommendations are incorrect.
  • Humans tend to agree with the recommendations provided by software when these are shown, whether correct or incorrect.

Automation bias arises both when the recommendations provided are correct and when they are incorrect (in the tasks we have analyzed the incorrect recommendations seemed to have more effect), and it affects everyone including experts (but novices seemed to more affected in the tasks we have analyzed).

In real-life applications, automation bias can lead to ethical risks because humans tend to follow recommendations from software even when they are incorrect. To evaluate the potential negative impacts we need to identify the possible consequences of two types of errors, false negatives and false positives, in the context of the application.

Congratulations! You have finished this notebook!