Sustainability Week 1: the Carbon Footprint of Algorithms¶
How to use this notebook
Simply read the text and follow the instructions.
This notebook contains code cells, which can be modified and must be executed to see the result of their content.
To execute a cell, select it and click on the play button (▶) in the tool bar, or type Shift + Enter or Ctr + Enter.
As the variables contained in a cell are stored in memory, the order of execution of the cells is important!
Introduction¶
Welcome to this week's exercise session of Responsible Software!
As software engineers and data scientists, it's essential that we consider the impact of the software we create on the environment and society. Beyond merely evaluating how well our software operates, we must also scrutinize its resource consumption and pollution emissions, ensuring that our digital solutions minimize their environmental footprint as much as possible.
In this notebook, you will explore the environmental impacts of classic algorithms and Machine Learning (ML) models by measuring their carbon footprints.
Scenario¶
Imagine that you work as an engineer for Poodle Inc., a software company that develops an extremely successful learning platform used by schools and universities worldwide to manage their courses, assignments, and grades.
Poodle Inc. has committed to the ambitious goal of reaching net zero emissions, i.e. remove from the environment as much carbon as it emits, by 2030. You have been tasked with measuring the carbon footprint of the company's products and making recommendations for reducing their environmental impact.
Let's get started!
Learning Goals
What will be covered:
- Part 1: Measuring and evaluating the carbon footprint of classic algorithms
- Part 2: Measuring and evaluating the carbon footprint of machine learning models
By the end of the session you will be able to:
- ✅ Measure the carbon footprint of a piece of software
- ✅ Identify the parameters that impact the carbon footprint of an algorithm
- ✅ Identify the parameters that impact the carbon footprint of Machine Learning models
- ✅ Find solutions to make algorithms less energy intensive
Part 1: Measuring the carbon footprint of classic algorithms¶
1.1 - Toolbox¶
You are going to use a tool called code carbon for tracking CO2 emissions of code. It works by tracking the power usage of the CPU, RAM and GPU during the execution of the code and then translating that into CO2 emissions, based on the energy mix (fossil fuels, renewables) of the region where the code is running (or the cloud provider, if the code is running in the cloud).
Let's first see the basic usage of codecarbon (you can check the documentation for more details).
Instructions
Run the cell below the libraries we need for this notebook, including the codecarbon library.
import pandas as pd
import numpy as np
import math
import pickle
import matplotlib.pyplot as plt
from res.utils import *
from codecarbon import OfflineEmissionsTracker
To use codecarbon, you need to instantiate a tracker object, call the function start() before the code you want to track and stop() afterwards.
Instructions
Execute the cell below to measure the carbon footprint of computing 10 000 000 times the sum 1 + 1 (NB: the name _ for a variable is a convention to indicate that we don't care about the result of the sum).
# Instantiate the tracker, setting the country to Switzerland and the log level to "error" (to avoid a verbose output).
tracker = OfflineEmissionsTracker(country_iso_code="CHE", log_level="error")
tracker.start()
try:
# Compute intensive code goes here
for i in range(10000000):
_ = 1 + 1
finally:
emissions = tracker.stop()
print("Total emissions: ", emissions, "kg of CO₂-equivalents [CO₂eq]")
[codecarbon INFO @ 14:33:40] Codecarbon is taking the configuration from the local file /home/git_Noto/responsible-software-notebooks/Notebook-5-Sustainability-1/.codecarbon.config [codecarbon INFO @ 14:33:40] offline tracker init [codecarbon INFO @ 14:33:40] Codecarbon is taking the configuration from the local file /home/git_Noto/responsible-software-notebooks/Notebook-5-Sustainability-1/.codecarbon.config
Total emissions: 1.2022217297298163e-07 kg of CO₂-equivalents [CO₂eq]
Codecarbon returns the carbon footprint of our code in **kilograms of CO₂-equivalents, noted *CO₂eq***. CO₂-equivalents provide a measure of the carbon footprint of a process by expressing the total impact of all greenhouse gases produced by that process in terms of the amount of CO₂ that would cause the same level of global warming. This is to take into account the fact that electricity production emits other greenhouse gases besides CO₂ (methane and nitrous oxide for example), each of them with different global warming potential. You will learn more about this in the videos.
If you want to know more about how codecarbon computes the amount of CO₂-equivalents, see their methodology.
1.2 - Comparing the carbon emissions of two algorithms¶
As a first step in your task to reduce the environmental impact of Poodle Inc., you decide to audit the company's codebase, measuring the carbon footprint of some pieces of code that are frequently used and see if they can be optimized.
You run across two functions that seem to have very similar purposes, but you are not sure which one is more efficient in terms of energy consumption, so you decide to measure the carbon footprint of these two functions to compare them.
Instructions
Execute the two cells below so that the two functions are defined.
def get_top_ten_popular_courses(course_data):
"""Finds and returns the ten courses of the input DataFrame with the largest number of enrolled students, sorted in descending order. If two or more courses have the same number of students, they are also sorted in alphabetical order.
Args:
course_data (Pandas DataFrame): A DataFrame with two columns: `course_id` and `number of students`
Returns:
Pandas Dataframe: A slice of course_data, containing the ten courses with the largest number of students.
"""
sorted_course = course_data.sort_values(
kind="quicksort",
by=["Number of students", "Course ID"],
ascending=[False, True],
)
return sorted_course.head(10)
def get_most_popular_course(course_data):
"""Finds and returns the course of the input DataFrame with the largest number of enrolled students. If two or more courses have the same number of students, the first one in alphabetical order is returned.
Args:
course_data (Pandas DataFrame): A DataFrame with two columns: `course_id` and `number of students`
Returns:
list: A list containing the id of the course with the most students and the number of students enrolled in it.
"""
# create a list of all courses included in the DataFrame
course_list = (
course_data.values.tolist()
) # 2-dimensional list of courses of the form [[course_id, number_of_students], ...]
course_list_size = len(course_list)
# iterate over all courses, comparing each one with those that come after it in the list
for i in range(course_list_size):
max_i = i # index of the course with the maximum number of students
for j in range(i + 1, course_list_size):
# select the index of the course with the maximum number of students between the two that are compared in every iteration
if course_list[j][1] > course_list[max_i][1]:
max_i = j # update the index of the course with the maximum number of students
# swap the courses to sort the list
(course_list[i][1], course_list[max_i][1]) = (
course_list[max_i][1],
course_list[i][1],
)
# if two or more courses have the same number of students, find the first one alphabetically
max_n_students = course_list[0][1]
first_alphabetically = course_list[0][0]
most_popular_course = course_list[0]
# iterate over all courses to find the first one alphabetically among those with the maximum number of students
for i in range(course_list_size):
# if the number of students of the course is equal to the maximum number of students, compare the course with the first one alphabetically and update the most popular course
if (
course_list[i][1] == max_n_students
and course_list[i][0] < first_alphabetically
):
first_alphabetically = course_list[i][0]
most_popular_course = course_list[i]
return most_popular_course
Both of those functions take as an argument a Pandas DataFrame with the following columns:
course_id: the ID of the coursenumber_of_students: the number of students enrolled in the course
You can find the database of all courses on the platform in the file res/course_data.csv.
Instructions
Load the data from the file res/course_data.csv and display the first few rows.
course_data = pd.read_csv('res/course_data.csv')
print(course_data.shape)
course_data.head()
(50000, 2)
| Course ID | Number of students | |
|---|---|---|
| 0 | 5kwhgd3xru | 1191 |
| 1 | 60q7kkkcri | 1797 |
| 2 | i5gji8bivg | 1033 |
| 3 | bthotbrt2q | 363 |
| 4 | fn46jly2oq | 1480 |
Footprint for one sample¶
Instructions
Complete the two code cells below to measure the carbon footprint of the two functions.
Give as an argument the first 1000 rows of the course_data DataFrame (using the full dataset would make the code run too slow).
The company's servers are located in Ireland, so we set the emission tracker to this location.
First measure the footprint of get_top_ten_popular_courses:
# BEGIN SOLUTION NO PROMPT
# get a sample of 1000 courses from the data
sample_course_data = course_data.head(1000)
# instantiate tracker, setting location to Ireland
tracker = OfflineEmissionsTracker(log_level="error", country_iso_code="IRL")
# start tracking emissions
tracker.start()
try:
# call the function get_top_ten_popular_courses on the sample courses
top_ten_popular_courses = get_top_ten_popular_courses(sample_course_data)
finally:
# stop the tracker and get the emissions
emissions = tracker.stop()
# END SOLUTION
""" # BEGIN PROMPT
# get a sample of 1000 courses from the data
sample_course_data = course_data.head(...)
# instantiate tracker, setting location to Ireland
tracker = OfflineEmissionsTracker(log_level="error", country_iso_code="IRL")
# start tracking emissions
... # look at the cell in 1.1 for the missing code
try:
# call the function get_top_ten_popular_courses on the sample courses
top_ten_popular_courses = ...
finally:
# stop the tracker and get the emissions
emissions = ...
"""; # END PROMPT
print("Total emissions of the `get_top_ten_popular_courses` function: ", emissions, "kg of CO₂-equivalents [CO₂eq]")
Total emissions of the `get_top_ten_popular_courses` function: 4.071163981597638e-07 kg of CO₂-equivalents [CO₂eq]
Do the same for get_most_popular_course:
# BEGIN SOLUTION NO PROMPT
# instantiate tracker
tracker = OfflineEmissionsTracker(log_level="error", country_iso_code="IRL")
# start tracking emissions
tracker.start()
try:
# call the function get_most_popular_course on the sample courses
most_popular_course = get_most_popular_course(sample_course_data)
finally:
# stop the tracker and get the emissions
emissions = tracker.stop()
# END SOLUTION
""" # BEGIN PROMPT
# instantiate tracker
tracker = OfflineEmissionsTracker(log_level="error", country_iso_code="IRL")
# start tracking emissions
... # look at the cell in 1.1 for the missing code
try:
# call the function get_most_popular_course on the sample courses
most_popular_course = ...
finally:
# stop the tracker and get the emissions
emissions = ...
"""; # END PROMPT
print("Total emissions of the `get_most_popular_course` function: ", emissions, "kg of CO₂-equivalents [CO₂eq]")
Total emissions of the `get_most_popular_course` function: 4.3031517711786244e-07 kg of CO₂-equivalents [CO₂eq]
Reflection time
Compare the carbon footprint of the two functions: what do you observe?
Feedback - Click on the "..." below only once you have really tried to answer the question!
The two functions seem to have very similar carbon footprints, around $4\times10^{-7}$ kg of CO₂eq (the footprint of the function get_most_popular_course being very slightly higher than that of the get_top_ten_popular_courses function).
Let's now examine what happens when we use different input sizes for the two functions and see how the carbon footprint changes.
Varying the size of the samples¶
Important: To prevent useless consumption of energy, we have executed the following code and saved the data in the file res/code_emissions.csv.
If you want to run the code yourself, you can change the type of the cell below from Raw to Code (select the cell and use the dropdown menu at the top of the notebook to change its type). Watch out, execution will take some time. Some small differences in the results are expected due to the difference in hardware, but the overall trends should be similar.
# sample sizes: from 1000 to the full dataset (50 000)
input_sizes = [1000, 10000, 50000]
emissions_most_popular = []
emissions_top_ten = []
# footprint of get_most_popular_course on the different sample sizes
for input_size in input_sizes:
tracker = OfflineEmissionsTracker(log_level="error", country_iso_code="IRL")
tracker.start()
try:
get_most_popular_course(course_data.head(input_size))
finally:
emissions_most_popular.append(tracker.stop())
# footprint of get_top_ten_popular_courses on the different sample sizes
for input_size in input_sizes:
tracker = OfflineEmissionsTracker(log_level="error", country_iso_code="IRL")
tracker.start()
try:
get_top_ten_popular_courses(course_data.head(input_size))
finally:
emissions_top_ten.append(tracker.stop())
# bundling the results into a dataframe
emissions = pd.DataFrame({
"Input size": input_sizes,
"get_most_popular_course()": emissions_most_popular,
"get_top_ten_popular_courses()": emissions_top_ten,
}).set_index("Input size")
# Export to CSV
# emissions.to_csv("res/code_emissions.csv")
# and display the resulting dataframe
emissions
Instructions
Load and display res/code_emissions.csv.
emissions = pd.read_csv('res/code_emissions.csv')
emissions = emissions.set_index('Input size')
emissions
| get_most_popular_course() | get_top_ten_popular_courses() | |
|---|---|---|
| Input size | ||
| 1000 | 4.199884e-07 | 4.060972e-07 |
| 10000 | 2.623492e-05 | 4.111655e-07 |
| 50000 | 1.014281e-03 | 4.409159e-07 |
Now, let's plot the carbon footprint of the two functions as a function of the input size.
Instructions
Complete the parameters of the plot below to display the carbon footprint of get_most_popular_course and get_top_ten_popular_courses as a function of the input size.
Hint: use a log scale for the y-axis; it makes it easier to compare values that differ by orders of magnitude.
emissions.plot(
kind="bar", # SOLUTION
logy=True, # SOLUTION
ylabel="Emissions (kg of CO₂-equivalents [CO₂eq])",
xlabel="Input Size",
title="CO2 Emissions vs Input Size",
);
Reflection time
What do you observe in the plot? How does the carbon footprint of the two functions change with the input size?
Which function is more efficient in terms of energy consumption and why?
Hint: consider computational complexity.
Feedback - Click on the "..." below only once you have really tried to answer the question!
From the plot above, we can see that for relatively small inputs, the carbon footprint of the two functions is similar (in the same order of magnitude). However, as the input size increases, the carbon footprint of get_most_popular_course() grows exponentially!
This is due to the due to the difference in computational complexity of the two algorithms:
- The function
get_most_popular_courseuses an inefficient sorting algorithm, a version of selection sort, which has a computational complexity of $O(n^2)$. - The function
get_top_ten_popular_coursesuses quicksort, which has a $O(n.log(n))$ complexity.
As the processor has to make more operations to sort the list in get_most_popular_course, it consumes more energy and as a result has a higher carbon footprint! You will further explore in the video the impact of the energy consumed by the CPU in the carbon footprint of software.
So, we have an indication that computational complexity is important to consider when trying to reduce the carbon footprint of a piece of code. The question is how much of an impact a specific function may have. For example, a highly inefficient algorithm that is only ran once in a while might not have a big impact on the overall carbon footprint, however one that is used frequently, perhaps by millions of users, might have a significant impact.
1.3 - Annual carbon footprint of a function¶
Since get_most_popular_course is clearly the most inefficient function of the two, refactoring it would be beneficial in terms of energy consumption. Since refactoring can be costly, the company would like to know if it is worth it and asks you to calculate its annual carbon footprint. (Refactoring this particular function is in fact not that complicated, but in a real-world scenario it could involve rewriting many parts of the codebase and potentially introducing new bugs, thus creating a lot of work for the developers and quality insurance team).
You decide to check the codebase and figure out how often this function is called and with what input sizes. You find out that when a user registers for the first time, the function is called to display the most popular courses on the platform, with the whole dataset of courses (included in course_data.csv) as input.
To calculate the carbon footprint of this function for a year, you will use the following formula:
$$ \text{annual emissions} = \text{estimated number of calls per year} \times \text{emissions per call} $$
Let's make this computation step by step.
Estimated number of calls per year¶
To calculate the number of calls per year, we need to know how many new users register annually.
The number of new users who register annually is available in the file res/registrations.csv. You will need to load the dataframe and use it to calculate the average new users per year.
Instructions
Load the file res/registrations.csv and display it.
registrations = pd.read_csv('res/registrations.csv')
registrations
| Year | Registrations | |
|---|---|---|
| 0 | 2010 | 483411 |
| 1 | 2011 | 1020981 |
| 2 | 2012 | 2004793 |
| 3 | 2013 | 2107358 |
| 4 | 2014 | 2384946 |
| 5 | 2015 | 2898856 |
| 6 | 2016 | 2973002 |
| 7 | 2017 | 3203307 |
| 8 | 2018 | 4347951 |
| 9 | 2019 | 4924770 |
| 10 | 2020 | 5049288 |
| 11 | 2021 | 5513708 |
mean_registrations_per_year = registrations['Registrations'].mean() # SOLUTION
mean_registrations_per_year_rounded = mean_registrations_per_year.round() # SOLUTION
print("Mean number of new users per year, i.e. estimated number of calls of `get_most_popular_course()` per year: ", mean_registrations_per_year_rounded)
Mean number of new users per year, i.e. estimated number of calls of `get_most_popular_course()` per year: 3076031.0
Emissions per call¶
We have previously calculated the emissions per call for input size 50 000 (the size of the dataset) and saved it in res/code_emissions.csv that has been loaded into the dataframe emissions.
Instructions
Find the emissions per call for the function get_most_popular_course for input size 50 000 in the dataframe emissions.
Hint: Use the dataframe.at[index, column name] function to get the value at a specific row and column.
emissions_per_call = emissions.at[50000, 'get_most_popular_course()'] # SOLUTION
print("Emissions per call of the `get_most_popular_course()` function: ", emissions_per_call, "kg of CO₂-equivalents [CO₂eq]")
Emissions per call of the `get_most_popular_course()` function: 0.001014280523401 kg of CO₂-equivalents [CO₂eq]
Annual emissions¶
Instructions
Finally, calculate the annual emissions of the function get_most_popular_course using the formula seen at the beginning of the section.
annual_emissions = mean_registrations_per_year_rounded * emissions_per_call # SOLUTION
print("Annual carbon footprint of the `get_most_popular_course()` function: ", annual_emissions, "kg of CO₂-equivalents [CO₂eq]")
Annual carbon footprint of the `get_most_popular_course()` function: 3119.9583326777015 kg of CO₂-equivalents [CO₂eq]
Instructions
Let's now create a function to be able to reuse this code in the future. Use the code you wrote above to create a function calculate_annual_emissions that takes as inputs the registrations dataframe and the emissions per call and returns the annual carbon emissions.
# BEGIN SOLUTION NO PROMPT
def calculate_annual_emissions(registrations, emissions_per_call):
mean_registrations_per_year_rounded = registrations["Registrations"].mean().round()
annual_emissions = mean_registrations_per_year_rounded * emissions_per_call
return annual_emissions
# END SOLUTION
""" # BEGIN PROMPT
def calculate_annual_emissions(registrations, emissions_per_call):
mean_registrations_per_year_rounded = ...
annual_emissions = ...
return annual_emissions
"""; # END PROMPT
Instructions
Call the function below and make sure the result is the same to the one you calculated above (the annual_emissions variable).
Note: We are using math.isclose to compare the two values, as we are dealing with floating point numbers, which often cannot be represented exactly in binary.
annual_emissions_withfunction = calculate_annual_emissions(registrations, emissions_per_call) # SOLUTION
print("Annual carbon footprint of the `get_most_popular_course()` function: ", annual_emissions_withfunction, "kg of CO₂-equivalents [CO₂eq]")
# test that the values are similar (math.isclose will return a boolean)
math.isclose(annual_emissions_withfunction, annual_emissions)
Annual carbon footprint of the `get_most_popular_course()` function: 3119.9583326777015 kg of CO₂-equivalents [CO₂eq]
True
Run the cell below to check your work.
import otter
test = otter.Notebook()
test.check("annual_emissions")
annual_emissions
passed! 🚀
Reflection time
What do you think of this annual footprint: is it a lot or not?
Feedback - Click on the "..." below only once you have really tried to answer the question!
It is often quite difficult to grasp whether carbon footprints expressed in CO2-eq are "a lot" or not. Here are some criteria to use:
- Comparing to overall climate targets can help: for instance, a well known target is the 2 tons of CO2-eq per person per year which got popularized after the Paris Agreement in 2016.
- Converting to equivalent quantities from other products or services can help by providing comparisons that are more concrete.
- Finally, to be able to judge, we also need to understand what we get for that footprint: is the footprint reasonable compared to the service or product obtained?
Let's start with point 1: a footprint of 3120 kg i.e. of 3.1 tons of CO2-eq is more than the 2 tons target. This means that this represents more than the annual footprint of one person in one year. Based on this criteria, this footprint is A LOT!
For point 2: we will develop ways to convert a footprint in the next exercise!
And finally for point 3: we are getting a list of the most popular courses on the platform from Poodle Inc. While this may be a useful feature, the value added by the feature is clearly not reasonable for the service obtained.
1.4 - Carbon emissions conversion¶
In the previous section, we calculated the annual emissions of the function get_most_popular_course in kg CO₂ equivalents. But what does this number mean? How can we make it more understandable to the company's stakeholders?
It is often useful for communication purposes to translate carbon emissions into more relatable units. For example, we could express the emissions in terms of the number of kilometers driven by a car, the number of trees needed to offset the emissions, or the number of hours a light bulb could be powered by the same amount of energy.
For your report, you decide to not only present the raw numbers but also translate them to kilometers traveled using different modes of transportation.
Instructions
Complete the function below to convert the emissions to kilometers travelled by car, plane or rail. Round the results to the nearest integer.
See the table below for the conversion factors:
| Mode of transport | Emissions per kilometer per passenger (kg CO₂eq) |
|---|---|
| Plane (long haul international flight) | 0.1758 |
| Car (average petrol car) | 0.1639 |
| Rail (national UK rail) | 0.03546 |
Source: "Conversion factors" by Carbon Trust, 2023 based on data published by the Department for Energy Security and Net Zero (DESNZ) of the UK.
def translate_emissions(CO2_kg):
# compute converted values
co2_kg_to_km_plane = round(CO2_kg / 0.1758) # SOLUTION
co2_kg_to_km_car = round(CO2_kg / 0.1639) # SOLUTION
co2_kg_to_km_rail = round(CO2_kg / 0.03546) # SOLUTION
# bundle values into a dataframe and return it
return pd.DataFrame({
"CO2_kg": [round(CO2_kg)],
"car_km": [co2_kg_to_km_car],
"plane_km": [co2_kg_to_km_plane],
"rail_km": [co2_kg_to_km_rail],
})
# test the function by converting 1 kg of CO2-eq
translate_emissions(1)
| CO2_kg | car_km | plane_km | rail_km | |
|---|---|---|---|---|
| 0 | 1 | 6 | 6 | 28 |
Run the cell below to check your work.
test.check("emissions_translation")
emissions_translation
passed! 🌟
We provide a function called print_emissions_translation that generates a pretty display of the result of your conversion function.
Instructions
Execute the cell below to see the results of the conversion on the annual footprint of the get_most_popular_course function.
print("The function `get_most_popular_course` emits", round(annual_emissions), "kg of CO₂eq per year.")
print_emissions_translation(translate_emissions(annual_emissions))
The function `get_most_popular_course` emits 3120 kg of CO₂eq per year. 3 120 kg of CO₂eq are equivalent to: 🛫 taking an international flight for 17 747 km 🚙 driving an average passenger car for 19 036 km or 🚂 travelling by train for 87 985 km.
This seems high enough to warrant refactoring. Moreover, during your audit, you discover other functions with similarly high carbon footprints that could be optimized. When you present your findings, however, the company decides that the refactoring cost is too high and ask you to propose an alternative solution.
Reflection time
What could be some alternative ways to reduce the carbon footprint of the codebase without refactoring?
Feedback - Click on the "..." below only once you have really tried to answer the question!
Caching: if a function is called multiple times, we could cache the result, reducing the number of calls to the function and therefore the carbon footprint.
Re-evaluating the need for the function: if the function is not serving an important business purpose, it could be removed altogether.
We will see in the next section that location is another factor in the footprint: the carbon footprint of code depends on the energy mix of the region where the servers are located. One possible solution could be to transfer the company's servers to a region with a low carbon intensity, i.e. where the electricity mix has a lower proportion of fossil fuels.
1.5 - Regional differences in carbon emissions¶
The company is interested in the potential energy savings if the servers were transferred to a region with a cleaner energy mix and task you with investigating which location would be the best choice.
Comparing locations¶
To do this, you decide to calculate the carbon footprint of the function get_most_popular_course for different regions.
Important: As before, we have already calculated the emissions by country to avoid you having to run an energy-intensive calculation. However, if you would like to execute the code yourself, you can do so by changing the type of the cell below from Raw to Code and running it. Watch out as this may take quite some time to execute. You may observe small variations between the result you calculate and the one we provide, due to differences in hardware, but the overall trends should be similar.
# loading the list of countries
import json
with open('res/global_energy_mix.json') as f:
data = json.load(f)
countries = list(data.keys())
# empty dict for storing results
emissions_by_country = {}
# looping over countries to execute 'get_most_popular_course' on 100 samples and get the corresponding emissions
for country_iso_code in countries:
tracker = OfflineEmissionsTracker(log_level="error", country_iso_code=country_iso_code)
tracker.start()
try:
get_most_popular_course(course_data.head(100))
finally:
emissions_by_country[country_iso_code] = tracker.stop()
# bundling the results into a dataframe
emissions_by_country = pd.DataFrame(emissions_by_country.items(), columns=['Country', 'Emissions'])
# replacing the country codes by their names
country_iso_codes_to_names = pd.read_csv('res/countries.csv')
emissions_by_country['Country'] = country_iso_codes_to_names['Country name']
emissions_by_country.set_index('Country', inplace=True)
# Export to CSV
# emissions_by_country.to_csv("res/emissions_by_country.csv")
# Display
emissions_by_country
Instructions
Load the file res/emissions_by_country.csv and display the first rows.
emissions_by_country = pd.read_csv('res/emissions_by_country.csv')
emissions_by_country.set_index('Country', inplace=True)
emissions_by_country.head()
| Emissions | |
|---|---|
| Country | |
| United Arab Emirates | 7.820542e-07 |
| Argentina | 4.933163e-07 |
| Australia | 7.649192e-07 |
| Austria | 1.543713e-07 |
| Burundi | 3.483467e-07 |
We would like to find the ten countries with the lowest emissions and compare them with the emissions in Ireland, where the servers are currently located.
Instructions
First, filter the dataset to include only the ten countries with the lowest emissions.
- Create a new dataframe with the values of the column "Emissions" sorted by increasing emissions.
Hint: thesort_valuesfunction can be useful here. - Keep only the first 10 rows.
- Add a row with the emissions in Ireland for comparison
# sort the dataframe by increasing emissions
emissions_by_country_sorted = emissions_by_country.sort_values('Emissions') # SOLUTION
# get the first 10 rows
top_ten_countries_with_lowest_emissions = emissions_by_country_sorted.head(10) # SOLUTION
# add the emissions from Ireland to the result
countries_with_lowest_emissions_and_IRL = pd.concat([top_ten_countries_with_lowest_emissions, emissions_by_country[emissions_by_country.index == 'Ireland']])
# display
countries_with_lowest_emissions_and_IRL
| Emissions | |
|---|---|
| Country | |
| Norway | 4.192164e-08 |
| Switzerland | 4.856022e-08 |
| Sweden | 5.670422e-08 |
| Costa Rica | 7.439660e-08 |
| France | 7.807667e-08 |
| Kenya | 9.820134e-08 |
| Finland | 1.102740e-07 |
| Brazil | 1.371396e-07 |
| Luxembourg | 1.466698e-07 |
| Austria | 1.543713e-07 |
| Ireland | 4.050890e-07 |
Instructions
Now, plot the emissions of the ten countries with the lowest emissions and Ireland by completing the parameters of the plot below.
countries_with_lowest_emissions_and_IRL.plot(
kind="bar", # SOLUTION
ylabel="Emissions (kg of CO₂-equivalents [CO₂eq])",
xlabel="Country",
title="Emissions by Country",
);
Reflection time
In which country would the function get_most_popular_course have the lowest carbon footprint?
What could explain the differences you see among countries?
Feedback - Click on the "..." below only once you have really tried to answer the question!
According to the data, the function get_most_popular_course would have its lowest carbon footprint in Norway.
This is because Norway's electricity production is primarily based on renewable sources, mainly hydropower followed by wind power. This means that electricity consumption in Norway will generate less carbon emissions than in other countries.
You present your findings to the company and they decide to investigate the possibility of moving their infrastructure to Kenya, which is heavily investing in sustainable energy sources.
Relocating to Kenya¶
So Poodle Inc. asks you to investigate the annual carbon footprint of get_most_popular_course() if the servers were located in Kenya.
We have already calculated the emissions per call of get_most_popular_course() for Kenya to avoid unecessary emissions. You can run the code yourself if you want by transforming the Raw cell into a Code cell.
As before, there may be small variations between the provided result and the one you calculate but the trends should be similar.
# Measuring the emissions of 'get_most_popular_course' on the whole dataset in Kenya
tracker = OfflineEmissionsTracker(log_level="error", country_iso_code="KEN")
tracker.start()
try:
get_most_popular_course(course_data)
finally:
emissions_per_call_KEN = tracker.stop()
print("Total emissions: ", emissions_per_call_KEN, "kg of CO₂-equivalents [CO₂eq]")
Instructions
Execute the cell below to get the emissions per call of get_most_popular_course() for Kenya.
emissions_per_call_KEN = 0.00025184247935055605
Instructions
Calculate the yearly emissions of the function get_most_popular_course() if the Poodle Inc. servers were located in Kenya.
Use the calculate_annual_emissions() function you created before.
annual_emissions_KEN = calculate_annual_emissions(registrations, emissions_per_call_KEN) # SOLUTION
print("Annual emissions in Kenya: ", round(annual_emissions_KEN), "kg of CO₂-equivalents [CO₂eq]")
Annual emissions in Kenya: 775 kg of CO₂-equivalents [CO₂eq]
Instructions
Convert the annual emissions in Ireland and Kenya to kilometers traveled by car, plane or rail using the translate_emissions() function you created before, then print the results.
# BEGIN SOLUTION NO PROMPT
print("Annual emissions in Kenya: ", round(annual_emissions_KEN), "kg of CO₂-equivalents [CO₂eq]")
print_emissions_translation(translate_emissions(annual_emissions_KEN))
annual_emissions_IRL = annual_emissions
print("\nAnnual emissions in Ireland: ", round(annual_emissions_IRL), "kg of CO₂-equivalents [CO₂eq]")
print_emissions_translation(translate_emissions(annual_emissions_IRL))
# END SOLUTION
""" # BEGIN PROMPT
print("Annual emissions in Kenya: ", round(annual_emissions_KEN), "kg of CO₂-equivalents [CO₂eq]")
print_emissions_translation(...)
annual_emissions_IRL = annual_emissions
print("\nAnnual emissions in Ireland: ", round(annual_emissions_IRL), "kg of CO₂-equivalents [CO₂eq]")
print_emissions_translation(...)
"""; # END PROMPT
Annual emissions in Kenya: 775 kg of CO₂-equivalents [CO₂eq] 775 kg of CO₂eq are equivalent to: 🛫 taking an international flight for 4 407 km 🚙 driving an average passenger car for 4 727 km or 🚂 travelling by train for 21 846 km. Annual emissions in Ireland: 3120 kg of CO₂-equivalents [CO₂eq] 3 120 kg of CO₂eq are equivalent to: 🛫 taking an international flight for 17 747 km 🚙 driving an average passenger car for 19 036 km or 🚂 travelling by train for 87 985 km.
Run the cell below to check your work.
test.check("country_emissions")
country_emissions
passed! 🎉
Following the presentation of your results, the company decides to move the servers to Kenya. They also commit to gradually refactoring the codebase to reduce the carbon footprint of the most energy-consuming functions.
Part 2: Measuring the carbon footprint of Machine Learning models¶
Poodle Inc. wants to develop a new product: a predictive analytics tool that helps law schools identify students who might need extra support to pass the bar exam. The tool will use a machine learning model that, based on data from past students, will predict which candidates are most likely to succeed and which ones are at risk of failing. It then provides this information to the school administration, who can use it to make decisions about resource allocation.
The product is in the early stages of development and the company is interested in anticipating its potential environmental impacts. They ask you to examine different types of machine learning models and make recommendations on how to reduce the carbon footprint of the product without compromising on performance.
In this section, we will explore the different factors in the carbon footprint of machine learning models.
For this you will follow a simplified machine learning life cycle with the following phases:
- Data collection: Poodle Inc. provides you with a dataset to test different machine learning models
- Data preparation: you will perform simplified data preprocessing steps
- Model development: you will do one round of model training
- Model evaluation: you will evaluate the accuracy of the obtained models
2.1 - The dataset¶
To train the predictive analytics tool, Poodle Inc. provides you with a dataset containing information about students who have taken the bar exam in the past.
It includes the following columns:
| Column name | Values | Description |
|---|---|---|
| Decile Year 1 | [1, 10] | Decile ranking of the student in the first year of law school. Higher values indicate better performance. For example, a student in the 10th decile is in the top 10% of their class. |
| Decile Year 3 | [1, 10] | Decile ranking of the student in the third year of law school |
| LSAT score | [11, 48] | Law School Admission Test results of the student. |
| Undergraduate GPA | [1.5, 4.0] | Grade point average of the student during their undergraduate studies. |
| Tier | [1, 6] | Tier of the law school attended by the student. Higher values indicate higher-ranked schools. |
| Fulltime | {True, False} | Whether the student was enrolled full-time or part-time. |
| Family income bracket | [1, 5] | Family income by quintile. Higher values indicate higher income. |
| Gender | {"female", "male"} | Student gender |
| Passed bar exam | {True, False} | Whether the student passed the bar exam. |
(Note: in reality this dataset has been adapted from the Law School Admission Council (LSAC) survey conducted across 163 law schools in the United States in 1991).
Instructions
Execute the cell to load the file res/law_school_data.csv and display the first few rows.
# Load the dataset
law_school_data = pd.read_csv("res/law_school_data.csv")
# Display the first rows
law_school_data.head() # SOLUTION
| Decile Year 1 | Decile Year 3 | LSAT score | Undergraduate GPA | Tier | Fulltime | Family income bracket | Gender | Passed bar exam | |
|---|---|---|---|---|---|---|---|---|---|
| 0 | 10.0 | 10.0 | 44.0 | 3.5 | 4.0 | True | 5.0 | female | True |
| 1 | 5.0 | 4.0 | 29.0 | 3.5 | 2.0 | True | 4.0 | female | True |
| 2 | 8.0 | 7.0 | 37.0 | 3.4 | 4.0 | True | 3.0 | male | True |
| 3 | 8.0 | 7.0 | 43.0 | 3.3 | 4.0 | True | 4.0 | female | True |
| 4 | 3.0 | 2.0 | 41.0 | 3.3 | 5.0 | True | 4.0 | female | True |
Instructions
Display some basic information about the dataset, such as the number of rows and columns, the data types of each column, and the number of non-null values.
Hint: you don't need to compute it yourself, a Pandas method can do it for you: check this function!
law_school_data.info() # SOLUTION
<class 'pandas.core.frame.DataFrame'> RangeIndex: 20798 entries, 0 to 20797 Data columns (total 9 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 Decile Year 1 20798 non-null float64 1 Decile Year 3 20798 non-null float64 2 LSAT score 20798 non-null float64 3 Undergraduate GPA 20798 non-null float64 4 Tier 20798 non-null float64 5 Fulltime 20798 non-null bool 6 Family income bracket 20798 non-null float64 7 Gender 20798 non-null object 8 Passed bar exam 20798 non-null bool dtypes: bool(2), float64(6), object(1) memory usage: 1.2+ MB
2.2 - Data preparation¶
As you have learned in the Fairness 2 notebook, we need to prepare the data before being able to fitting a model on it.
Splitting into label and features¶
Here are the design choices that we make:
- We will try to predict whether the student will pass the bar exam, therefore the label that we will use is
Passed bar exam - We will train the model on the student's other information, therefore the features are:
Decile Year 1,Decile Year 3,LSAT score,Undergraduate GPA,Tier,Fulltime,Family income bracketandGender
Let's create the features (model input) and label (model expected output) dataframes.
Instructions
Create two DataFrames by selecting the right columns from law_school_data:
featuresshould contain the input columnslabelshould contain the expected output column
# BEGIN SOLUTION NO PROMPT
features = law_school_data[
[
"Decile Year 1",
"Decile Year 3",
"LSAT score",
"Undergraduate GPA",
"Tier",
"Fulltime",
"Family income bracket",
"Gender",
]
]
label = law_school_data[["Passed bar exam"]]
# END SOLUTION
""" # BEGIN PROMPT
features = law_school_data[['Decile Year 1', ..., ..., ..., ..., ..., ..., ...]]
label = law_school_data[...]
"""; # END PROMPT
Reflection time
Should we include all columns of the original dataset in the features? Why? What could we do about the issue?
Feedback - Click on the "..." below only once you have really tried to answer the question!
In the original dataset, there are attributes that are sensitive, such as Gender, or that can be proxies for sensitive attributes, such as Family income bracket and Tier.
- Keeping this information in the training data could lead to biased predictions, as the model might learn to associate certain characteristics with student success or failure, resulting in indirect discrimination.
- On the other hand, not including those columns in the input values could reduce the accuracy, and we have also seen that not including them will not necessarily mean that our model is fairer because of hidden correlations.
To address this issue we should first analyse the data and check whether there are biases in terms of representativeness of the different groups and how the outcome (i.e. success at the bar exam) relates to the sensitive attributes. Based on this information, we could decide on a strategy to either act on the data using pre-processing techniques (e.g. resampling, reweighting, transformation) or act on the model using in-processing techniques (train our model with a fairness objective) - see the Fairness 2 notebook for more details.
In this notebook, for reasons of simplicity and brevity, we will preserve all columns in the features and not explore the fairness questions further.
Challenge yourself: analyze the biases in the dataset and try to come up with solutions to obtain a fair model!
Normalization and indicator variables¶
Our label is already an indicator variable (i.e. it takes binary values True/False) so there is no need to further process it.
However we need to further process our features: we need to normalize the numerical values so they are all on the same scale and to convert the categorical columns (e.g. Gender) into indicator variables. We have seen in the Fairness 2 notebook how to do this using a StandardScaler and dummy variables.
Instructions
Execute the cell below to convert the column Gender into a dummy variable and to normalize all the numerical values.
from sklearn.preprocessing import StandardScaler
# replacing categorical variables by dummy variables
features["Female"] = pd.get_dummies(features["Gender"])["female"]
features.drop(columns=['Gender'], inplace=True)
# normalizing numerical values
scaler = StandardScaler()
features[["Decile Year 1", "Decile Year 3", "LSAT score", "Undergraduate GPA", "Tier", "Family income bracket"]] = scaler.fit_transform(
features[["Decile Year 1", "Decile Year 3", "LSAT score", "Undergraduate GPA", "Tier", "Family income bracket"]])
# display result
features.head()
| Decile Year 1 | Decile Year 3 | LSAT score | Undergraduate GPA | Tier | Fulltime | Family income bracket | Female | |
|---|---|---|---|---|---|---|---|---|
| 0 | 1.552592 | 1.551948 | 1.343487 | 0.664031 | 0.242777 | True | 1.797074 | True |
| 1 | -0.196999 | -0.547669 | -1.441033 | 0.664031 | -1.528276 | True | 0.625831 | True |
| 2 | 0.852755 | 0.502139 | 0.044044 | 0.420965 | 0.242777 | True | -0.545412 | False |
| 3 | 0.852755 | 0.502139 | 1.157852 | 0.177899 | 0.242777 | True | 0.625831 | True |
| 4 | -0.896836 | -1.247542 | 0.786583 | 0.177899 | 1.128304 | True | 0.625831 | True |
Splitting into training and evaluation sets¶
Let's now split the dataset into:
- a training dataset, which will be used to train the model
- a test dataset, which we will use to evaluate the performance of the model on data it has not seen yet.
This practice, which is very important in machine learning, helps assess how well the model generalizes while preventing overfitting to the training data.
Instructions
Use the test_train_split function to split the dataset into two sets of rows:
- a training set with 70% of the rows
- a test set with 30% of the rows.
Hint: train_size and test_size represent proportions of the dataset and should be between 0 and 1.
Note: the random_state parameter allows setting a seed value for the random number generator used in the splitting process. This ensures that the same split is obtained every time the code is executed with the same seed value.
from sklearn.model_selection import train_test_split
# BEGIN SOLUTION NO PROMPT
train_features, test_features, train_label, test_label = train_test_split(features, label, train_size=0.7, test_size=0.3, random_state=42)
# END SOLUTION
""" # BEGIN PROMPT
train_features, test_features, train_label, test_label = train_test_split(features, label, train_size=..., test_size=..., random_state=42)
"""; # END PROMPT
print("The training dataset contains", len(train_features), "rows, while the evaluation dataset contains", len(test_features), "rows.")
The training dataset contains 14558 rows, while the evaluation dataset contains 6240 rows.
Run the cell below to check your work.
test.check("data_preparation")
data_preparation
passed! 🚀
2.3 - Model training¶
We will now measure and compare the carbon footprint of training different types machine learning models:
- random forest: builds many separate decision trees (an ensemble of models) and combines their predictions.
- k-nearest neighbors: predicts a data point’s label based on the most common labels among its closest neighbors in the training data.
- logistic regression: predicts the probability of a binary outcome using a weighted combination of input features.
- multi-layer perceptron: type of neural network made up of layers of interconnected nodes that can adapt to non-linear patterns in data.
Important: To avoid generating carbon emissions, we have done the training ourselves and stored the results in a CSV file.
The cell below contains the code for the training. If you want to execute it, you should transform it from a Raw cell into a Code cell.
Note: we have defined a helper function called train_model in the res.utils module that trains the models using the selected algorithm.
This is the function signature and description, feel free to go and have a look at it in the res/utils.py file:
def train_model(algorithm, train_features, train_label)
"""
Trains a machine learning model using the specified algorithm.
Parameters:
algorithm (str): The algorithm to use for training the model.
train_features (array-like): The training data.
train_label (array-like): The target variable for training.
Returns:
trained_model: The trained machine learning model.
"""
# List of the model types we want to compare
algorithm_list = ["Random forest", "K-nearest neighbors", "Logistic regression", "Multi-layer perceptron"]
# Create a table to store the results
train_algorithm_comparison = pd.DataFrame({'Training Emissions [kg of CO2 eq.]': pd.Series(dtype='float')},
index = algorithm_list)
# Iterate over our model types
for algorithm in algorithm_list:
# Initialize the tracker
tracker = OfflineEmissionsTracker(log_level="error", country_iso_code="IRL")
# Start the tracker
tracker.start()
try:
# Train the ML model
model = train_model(algorithm, train_features, train_label)
finally:
# Stop the tracker and get the emissions
emissions = tracker.stop()
# Save results
train_algorithm_comparison.at[algorithm, 'Training Emissions [kg of CO2 eq.]'] = emissions
# Save the trained model in a file
with open('models/'+algorithm+'.pkl','wb') as f:
pickle.dump(model, f, protocol=pickle.HIGHEST_PROTOCOL)
# Export emissions to CSV
# train_algorithm_comparison.to_csv("res/train_algorithm_comparison.csv")
# Display
display(train_algorithm_comparison)
Instructions
Load the results in terms of carbon footprint for training the different machine learning models.
# Load file
train_algorithm_comparison = pd.read_csv("res/train_algorithm_comparison.csv", index_col=0)
# Display the result
train_algorithm_comparison
| Training Emissions [kg of CO2 eq.] | |
|---|---|
| Random forest | 8.090307e-07 |
| K-nearest neighbors | 4.165912e-07 |
| Logistic regression | 4.163862e-07 |
| Multi-layer perceptron | 2.762248e-05 |
Now, let's visualize the results by plotting the carbon footprint of training each model.
Instructions
Complete the parameters of the plot below.
See the Pandas documentation for more information on how to customize the plot.
train_algorithm_comparison.plot(
kind="bar", # SOLUTION
logy=True
);
Reflection time
- How do the carbon footprints of the different models at training time compare?
- What could be some reasons for these differences?
- Which model would you recommend Poodle Inc. to use if they want to minimize their environmental impact?
Feedback - Click on the "..." below only once you have really tried to answer the question!
From the plots above, we can see that the carbon footprint of the Multi-Layer Perceptron is way higher than that of the other models.
The Multi-Layer Perceptron uses a technology that is fundamentally different from the other three models since it relies on neural networks, which is the foundational architecture of deep learning (which is built on the idea of stacking multiple layers of neurons in a neural network to automatically learn increasingly complex features from data). The carbon footprint of training a Multi-Layer Perceptron is much higher than that of the three other models because it involves iteratively updating the weights on the connections between all the nodes through many passes over the data (epochs) using computationally intensive matrix operations.
If we compare only the three other models, we notice that the Random Forest model has the highest training footprint, while the K-Nearest Neighbors and Logistic Regression models have a similar training footprint. This is because the Random Forest model builds and optimizes many decision trees, each requiring multiple passes through the data and intensive computations.
Overall, this highlights that different types of models will have a different carbon footprint at training time because of their architecture.
To minimize the carbon footprint of training their model, Poodle Inc. should use either the K-Nearest Neighbors or Logistic Regression models. However we should check other decision factors such as the accuracy and carbon footprint of predictions before committing to a design decision!
2.4 - Model evaluation¶
Having trained our models, we can now evaluate their performance on the test dataset we have created earlier.
Poodle Inc. would like to determine which model makes the most accurate predictions, as well as which one has the lowest carbon footprint when making predictions.
Important: Again, to avoid generating carbon emissions, we have done the evaluation ourselves and stored the results in a CSV file.
The cell below contains the code for the evaluation. If you want to execute it, you should transform it from a Raw cell into a Code cell.
# List of the model types we want to compare
algorithm_list = ["Random forest", "K-nearest neighbors", "Logistic regression", "Multi-layer perceptron"]
# Create a table to store the results
eval_algorithm_comparison = pd.DataFrame({'Evaluation Emissions [kg of CO2 eq.]': pd.Series(dtype='float'),
'Accuracy': pd.Series(dtype='float')},
index = algorithm_list)
# Iterate over our model types
for algorithm in algorithm_list:
# Load the trained model from file (saved at the end of training)
with open('models/'+algorithm+'.pkl','rb') as f:
trained_model = pickle.load(f)
# Initialize the tracker
tracker = OfflineEmissionsTracker(log_level="error", country_iso_code="IRL")
# Start the tracker
tracker.start()
try:
# Run the model on the test dataset
predicted_output = trained_model.predict(test_features)
# Compute the accuracy
accuracy = accuracy_score(test_label, predicted_output)
finally:
# Stop the tracker and get the emissions
emissions = tracker.stop()
# Save results
eval_algorithm_comparison.at[algorithm, 'Evaluation Emissions [kg of CO2 eq.]'] = emissions
eval_algorithm_comparison.at[algorithm, 'Accuracy'] = accuracy
# Export to CSV
# eval_algorithm_comparison.to_csv("res/eval_algorithm_comparison.csv")
# Display
display(eval_algorithm_comparison)
Instructions
Load the results in terms of carbon footprint and accuracy of evaluating the different machine learning models.
# Load file
eval_algorithm_comparison = pd.read_csv("res/eval_algorithm_comparison.csv", index_col=0)
# Display the result
eval_algorithm_comparison
| Evaluation Emissions [kg of CO2 eq.] | Accuracy | |
|---|---|---|
| Random forest | 4.243377e-07 | 0.904006 |
| K-nearest neighbors | 4.724056e-07 | 0.880609 |
| Logistic regression | 4.074052e-07 | 0.902083 |
| Multi-layer perceptron | 1.210589e-05 | 0.901442 |
Carbon footprint¶
Instructions
Complete the parameters of the plot below to display the carbon footprint the different models at evaluation time.
# BEGIN SOLUTION NO PROMPT
eval_algorithm_comparison['Evaluation Emissions [kg of CO2 eq.]'].plot(
kind="bar",
logy=True,
ylabel="kg of CO2 eq.",
title="Carbon Footprint by Algorithm",
);
# END SOLUTION
""" # BEGIN PROMPT
eval_algorithm_comparison[...].plot(
kind="bar",
logy=True,
ylabel="kg of CO2 eq.",
title="Carbon Footprint by Algorithm",
);
"""; # END PROMPT
Reflection time
- How do the carbon footprints of the different models compare at evaluation time?
- What could be some reasons for these differences?
Feedback - Click on the "..." below only once you have really tried to answer the question!
Again, we can see that the carbon footprint of the Multi-Layer Perceptron is way higher than that of the other models.
The carbon footprint of evaluating a Multi-Layer Perceptron is much higher because making a prediction requires passing the input through all its layers and performing many weighted calculations, which is computationally intensive.
If we compare only the three other models, K-Nearest Neighbors has the highest carbon footprint, while the footprints of the Random Forest and the Logistic Regression models are comparable. This is because the K-Nearest Neighbors model must compare each new data point to all training samples to make a prediction. In contrast, Random Forest and Logistic Regression use pre-computed model parameters, making their predictions much faster and less energy-intensive.
Overall, this highlights that different types of models will have a different carbon footprint at evaluation time because of their architecture.
The results we obtain here highlight that models built on artificial neural networks often have higher carbon footprints than other types of models, especially large or deep networks, because of the many computations involved. However, there are other important factors that play a role, such as the size of the dataset, the size of the model (or other parameters such as depth) or the type of hardware used for execution (more on that in the videos!). For instance, small or shallow neural networks can sometimes consume less energy than complex models like large ensembles (e.g., Random Forests with hundreds of trees) or extremely large datasets used in K-Nearest Neighbors models.
Measuring the footprint while developing the model is essential to be able to make informed choices.
Accuracy¶
Instructions
Complete the parameters of the plot below to compare the accuracy of the different models.
# BEGIN SOLUTION NO PROMPT
eval_algorithm_comparison['Accuracy'].plot(
kind="bar",
ylabel="Accuracy",
ylim=(0.5, 1),
title="Model Accuracy by Algorithm",
);
# END SOLUTION
""" # BEGIN PROMPT
....plot(
kind="bar", # SOLUTION
ylabel="Accuracy",
ylim=(0.5, 1),
title="Model Accuracy by Algorithm",
);
"""; # END PROMPT
Reflection time
- Compare the accuracy of the different models, what do you observe?
- Which model would you recommend Poodle Inc. to use if they want to minimize their environmental impact without compromising on performance?
Feedback - Click on the "..." below only once you have really tried to answer the question!
- We observe that the accuracy of Random Forest, Logistic Regression and Multi-Layer Perceptron are very similar, around 90%, while the accuracy of K-Nearest Neighbors is slightly lower but in the same order of magnitude. All in all, the performance of these different models at predicting students who will pass the bar exam is relatively similar.
This result illustrates that models relying on artificial neural networks do not necessarily outperform simpler models. The type of model that will be the best choice highly depends on the data available for the task at hand (quality, type of patterns, quantity). - Based on the analysis of the accuracy above and of the carbon footprint earlier, the Logistic Regression model seems to be scoring best on our two criteria of accuracy and carbon footprint compared with the other types of models.
The accuracy-footprint trade-off
It is important to know that there is often a trade-off between the accuracy of a model and its carbon footprint. In general, more complex or accurate models (like deep neural networks or large ensembles) require more computation and energy during both training and inference, increasing their carbon footprint. However, model efficiency, data quality, and algorithm design can also improve accuracy without significantly raising emissions.
If you want to learn more about this issue:
Brownlee, A., Adair, J., Haraldsson, S., & Jabbo, J. (2021). Exploring the Accuracy—Energy Trade-off in Machine Learning. Genetic Improvement Workshop at 43rd International Conference on Software Engineering. https://doi.org/10.1109/GI52543.2021.00011
2.5 - Overall ML life-cycle¶
Focusing on its ambitious goal to reach net zero emissions, Poodle Inc. wants to anticipate the environmental impact of the whole machine learning life-cycle of the predictive tool under development.
They asks you the following two questions:
- What will be the overall carbon footprint of developing a model from scratch, for the most and the least carbon efficient types of models?
- Once deployed, which type of model will have the lowest carbon footprint on the long term?
You use the experiments you have made so far to answer these two questions.
The footprint of model development¶
For the first question, you identify that the carbon footprint of the overall development of a model includes:
- the footprint of training the model
- and the footprint of evaluating it.
You already have this data for the four types of models, you just need to combine it.
Instructions
Complete the cell below to concatenate the results from the training and from the evaluation of the four models into a single dataframe.
The result should look like this:
| Training Emissions [kg of CO2 eq.] | Evaluation Emissions [kg of CO2 eq.] | Accuracy | |
|---|---|---|---|
| Random forest | |||
| K-nearest neighbors | |||
| Logistic regression | |||
| Multi-layer perceptron |
# BEGIN SOLUTION NO PROMPT
algorithm_comparison = pd.concat([train_algorithm_comparison, eval_algorithm_comparison], axis="columns")
algorithm_comparison
# END SOLUTION
""" # BEGIN PROMPT
algorithm_comparison = pd.concat([...], axis="columns")
algorithm_comparison
"""; # END PROMPT
Instructions
Create a new column 'Total Dev Emissions [kg of CO2 eq.]' which will contain the total carbon footprint of the development (training + evaluation) for each model.
# Sum the training and evaluation footprints
algorithm_comparison['Total Dev Emissions [kg of CO2 eq.]'] = algorithm_comparison[['Training Emissions [kg of CO2 eq.]','Evaluation Emissions [kg of CO2 eq.]']].sum(axis=1) # SOLUTION
# Display the result
display(algorithm_comparison)
# Plot the result
algorithm_comparison['Total Dev Emissions [kg of CO2 eq.]'].plot(kind="bar", logy=True);
| Training Emissions [kg of CO2 eq.] | Evaluation Emissions [kg of CO2 eq.] | Accuracy | Total Dev Emissions [kg of CO2 eq.] | |
|---|---|---|---|---|
| Random forest | 8.090307e-07 | 4.243377e-07 | 0.904006 | 1.233368e-06 |
| K-nearest neighbors | 4.165912e-07 | 4.724056e-07 | 0.880609 | 8.889968e-07 |
| Logistic regression | 4.163862e-07 | 4.074052e-07 | 0.902083 | 8.237914e-07 |
| Multi-layer perceptron | 2.762248e-05 | 1.210589e-05 | 0.901442 | 3.972837e-05 |
Using this data, you identify that the development footprint would range from $8.2 \times 10^{-7}$ to $4 \times 10^{-5}$ kg of CO2 eq.
Reflection time
What are the limitations of these results? Do they represent the footprint range to expect for the development of real ML models?
Feedback - Click on the "..." below only once you have really tried to answer the question!
There are two limitations that impact the realism of these results:
- Your experiment included only one training iteration for each model type, which is quite unrealistic as developing a model will generally involve a search of the best parameters for each of the models;
- You have chosen small and computationally simple models for this experiment (3 simple statistical models, 1 very small neural network) so that they can be trained on CPUs, but training larger models would necessarily require more data, more compute and the use of GPUs, which would consume way more energy.
The footprint of inference¶
To answer the second question "Once deployed, which type of model will have the lowest carbon footprint on the long term?", you think about using the same approach as in Section 1.3 with the get_most_popular_course function:
- Measure the carbon footprint of a single prediction from each model;
- Multiply this footprint by the expected average number of predictions per year, and then multiply by the expected lifetime of the model (number of years).
However, you don't have the carbon footprint of a single prediction...
But you have the carbon footprint of evaluating the models, where each model has been used to make predictions on the test dataset!
Reflection time
Based on the footprint obtained at evaluation time, which type of model would you say will have the lowest carbon footprint on the long term?
Feedback - Click on the "..." below only once you have really tried to answer the question!
Based on the footprints at evaluation time, we can identify that the Logistic Regression model will have the lowest carbon footprint on the long term, while having a good accuracy!
The footprint of inference
While the carbon footprint of developing a machine learning model seems way higher than that of inference at first sight, the carbon footprint of inference has actually a much longer term impact and ultimately accounts for the largest part of the overall footprint. According to Patterson (2021), "Most companies spend more energy on serving a DNN model (performing inference) than on training it. For example, NVIDIA estimated that 80–90% of the ML workload is inference processing". Therefore it should never be underestimated, especially for (very) large models.
This is why it is extremely important to select models that will have the lowest possible footprint at inference time.
If you want to learn more about this issue:
Patterson, D., Gonzalez, J., Le, Q., Liang, C., Munguia, L.-M., Rothchild, D., So, D., Texier, M., & Dean, J. (2021). Carbon Emissions and Large Neural Network Training (No. arXiv:2104.10350). arXiv. https://doi.org/10.48550/arXiv.2104.10350
[Optional] 2.6 - Impact of dataset size on training emissions¶
After you present the above analysis to the company, they ask you to investigate the possibility to reduce the carbon footprint of the development of the product. To do that, you decide to explore whether reducing the size of the dataset will have an impact on the training emissions and the accuracy of the models.
Important: Again, to avoid having to do an energy-intensive calculation, we have run the training and evaluation of the models with different fractions of the initial training dataset (60%, 70%, 80% and 90%) and saved the results in the file res/training_results.csv. The corresponding code is contained in the Raw cell below, that you need to transform into a Code cell to execute.
# List of the model types we want to compare
algorithm_list = ["Random forest", "K-nearest neighbors", "Logistic regression", "Multi-layer perceptron"]
# Percentages of training data to use
prop_data = [60, 70, 80, 90]
# Create a list to store the results
rows_list = []
# Iterate over data proportions
for prop in prop_data:
# Split dataset
train_features, test_features, train_label, test_label = train_test_split(features, label, train_size=prop/100, test_size=0.1, random_state=30)
# Iterate over our model types
for algorithm in algorithm_list:
# Initialize the tracker
tracker = OfflineEmissionsTracker(log_level="error", country_iso_code="IRL")
# Start the tracker
tracker.start()
try:
# Train the ML model
trained_model = train_model(algorithm, train_features, train_label)
finally:
# Stop the tracker and get the emissions
emissions = tracker.stop()
# Run the model on the test dataset
predicted_output = trained_model.predict(test_features)
# Compute the accuracy
accuracy = accuracy_score(test_label, predicted_output)
# Save results
dict_data = {
'Algorithm': algorithm,
'% of original dataset': prop,
'Training Emissions [kg of CO2 eq.]': emissions,
'Accuracy': accuracy
}
print(dict_data)
rows_list.append(dict_data)
# Resulting dataframe
training_results = pd.DataFrame(rows_list)
# Export to CSV
# training_results.to_csv("res/training_results.csv")
# Display
training_results
Instructions
Load the file res/training_results.csv and display it.
# Load file
training_results = pd.read_csv("res/training_results.csv")
# Reshape the data for better visualization
training_results = training_results.pivot(index='% of original dataset', columns='Algorithm', values=['Accuracy', 'Training Emissions [kg of CO2 eq.]'])
# Display the results table
training_results
| Accuracy | Training Emissions [kg of CO2 eq.] | |||||||
|---|---|---|---|---|---|---|---|---|
| Algorithm | K-nearest neighbors | Logistic regression | Multi-layer perceptron | Random forest | K-nearest neighbors | Logistic regression | Multi-layer perceptron | Random forest |
| % of original dataset | ||||||||
| 60 | 0.879808 | 0.898077 | 0.899519 | 0.899519 | 4.167818e-07 | 4.312415e-07 | 0.000076 | 8.986941e-07 |
| 70 | 0.883173 | 0.898558 | 0.897596 | 0.897596 | 4.216075e-07 | 4.289986e-07 | 0.000057 | 9.405118e-07 |
| 80 | 0.880769 | 0.898558 | 0.898558 | 0.897115 | 4.214659e-07 | 4.267468e-07 | 0.000081 | 1.036902e-06 |
| 90 | 0.878846 | 0.898558 | 0.897115 | 0.896154 | 4.238362e-07 | 4.273712e-07 | 0.000117 | 1.085629e-06 |
Instructions
In the two cells below, plot the carbon footprint and accuracy of training the different models as a function of the dataset size.
# BEGIN SOLUTION NO PROMPT
training_results['Training Emissions [kg of CO2 eq.]'].plot(
kind="bar",
subplots=True,
legend=False,
ylabel="kg of CO2 eq.",
title="Carbon Footprint of Training by Dataset Size",
figsize=(5,8)
)
plt.tight_layout();
# END SOLUTION
""" # BEGIN PROMPT
....plot(
kind="bar",
subplots=True,
legend=False,
ylabel="kg of CO2 eq.",
title="Carbon Footprint of Training by Dataset Size",
figsize=(5,8)
)
plt.tight_layout();
"""; # END PROMPT
# BEGIN SOLUTION NO PROMPT
training_results['Accuracy'].plot(
kind="bar",
ylim=(0.5, 1),
ylabel="Accuracy",
title="Model Accuracy by Training Dataset Size",
).legend(bbox_to_anchor=(1, 1));
# END SOLUTION
""" # BEGIN PROMPT
....plot(
kind="bar",
ylim=(0.5, 1),
ylabel="Accuracy",
title="Model Accuracy by Training Dataset Size",
).legend(bbox_to_anchor=(1, 1))
"""; # END PROMPT
Reflection time
- How does the carbon footprint of the models change as the dataset size increases?
- What is the relationship between the dataset size used for training and the accuracy?
- What recommendations would you make to the company based on these results?
Feedback - Click on the "..." below only once you have really tried to answer the question!
We can see that the carbon footprint of training increases as the dataset size increases for the Multi-Layer Perceptron and the Random Forest models. This is because larger datasets require more computational resources to train these two models, while the Logistic Regression and the K-Nearest Neighbours are not affected, due to the respective architectures of the models (see sections above).
Note: The Multi-Layer Perceptron requires overall a larger amount of data for training compared to the 3 statistical models. For instance it would be difficult to train it will less than 60% of our dataset. In addition, the training process for the Multi-Layer Perceptron can be sensitive to the way the data is split (especially with smaller datasets), which means that the training algorithm will need more or less training samples to obtain a converging solution depending on the samples present in the training dataset.
Regarding accuracy, it seems that training with larger datasets does not improve it significantly, independently from the type of model.
We have to keep in mind though that we have limited ourselves to taking a percentage of the approx. 21 000 samples from the original dataset, which means the variation in size we have explored is limited!This suggests that using a smaller dataset to train our model could be a viable strategy to reduce the carbon footprint of the developing the product without compromising on performance.
However, in our example we have reduced the dataset by picking a random subset of it, which is a risky strategy as it may impact the patterns learned by the model. Other techniques exist to reduce the size of a dataset while preserving the patterns and quality of the data to achieve a good model performance.
If you want to learn more about this:
- Brownlee, A., Adair, J., Haraldsson, S., & Jabbo, J. (2021). Exploring the Accuracy—Energy Trade-off in Machine Learning. Genetic Improvement Workshop at 43rd International Conference on Software Engineering. https://doi.org/10.1109/GI52543.2021.00011
- Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020). Scaling Laws for Neural Language Models (No. arXiv:2001.08361). arXiv. https://doi.org/10.48550/arXiv.2001.08361
[Optional] Part3: Code Optimization for Sustainability¶
3.1 - Introduction¶
Remember the first function of the notebook in 1.2 get_most_popular_course() to calculate the most popular course? The complexity of the function was $O(n^2)$ and the execution time was quite long. Calling this function multiple times could have a significant impact on the overall carbon footprint especially if it is used at large scale.
We propose this optional exercise to optimize and hence reducing the total emissions of this function. If you don't have time to do it it's fine!
Instructions
Load the dataset and the function definition below.
import os
from res.utils import AlgorythmOptimizationTest
from time import perf_counter
df_course = pd.read_csv(os.path.join('res', 'course_data.csv'))
df_course = df_course.head(7000)
df_course.head()
| Course ID | Number of students | |
|---|---|---|
| 0 | 5kwhgd3xru | 1191 |
| 1 | 60q7kkkcri | 1797 |
| 2 | i5gji8bivg | 1033 |
| 3 | bthotbrt2q | 363 |
| 4 | fn46jly2oq | 1480 |
We recall that the function get_most_popular_course() retrieves the most popular course (maximum number of students enrolled) from the dataset. If there are multiple courses with the same maximum number of students, the function returns the first course ID in the alphabetical order.
We just copy pasted the function definition below for you to have it handy:
def slow_function(course_data):
"""Finds and returns the course of the input DataFrame with the largest number of enrolled students. If two or more courses have the same number of students, the first one in alphabetical order is returned.
Args:
course_data (Pandas DataFrame): A DataFrame with two columns: `course_id` and `number of students`
Returns:
list: A list containing the id of the course with the most students and the number of students enrolled in it.
"""
# create a list of all courses included in the DataFrame
course_list = (
course_data.values.tolist()
) # 2-dimensional list of courses of the form [[course_id, number_of_students], ...]
course_list_size = len(course_list)
# iterate over all courses, comparing each one with those that come after it in the list
for i in range(course_list_size):
max_i = i # index of the course with the maximum number of students
for j in range(i + 1, course_list_size):
# select the index of the course with the maximum number of students between the two that are compared in every iteration
if course_list[j][1] > course_list[max_i][1]:
max_i = j # update the index of the course with the maximum number of students
# swap the courses to sort the list
(course_list[i], course_list[max_i]) = (
course_list[max_i],
course_list[i],
)
# if two or more courses have the same number of students, find the first one alphabetically
max_n_students = course_list[0][1]
first_alphabetically = course_list[0][0]
most_popular_course = course_list[0]
# iterate over all courses to find the first one alphabetically among those with the maximum number of students
for i in range(course_list_size):
# if the number of students of the course is equal to the maximum number of students, compare the course with the first one alphabetically and update the most popular course
if (
course_list[i][1] == max_n_students
and course_list[i][0] < first_alphabetically
):
first_alphabetically = course_list[i][0]
most_popular_course = course_list[i]
return most_popular_course
3.2 - Optimiztion¶
Instructions
Complete the function optimized_function(course_data) such that it returns the same result as the function slow_function(course_data) but faster. You can achieve results as fast as 1000-2000 times faster than the original function.
Performance hints:
- Medium : use only one for loop
- Fast : No for loops and use sorting algorithms
- Very fast : Use only Pandas functions
# BEGIN SOLUTION NO PROMPT
def optimized_function(course_data):
"""Finds and returns the course of the input DataFrame with the largest number of enrolled students. If two or more courses have the same number of students, the first one in alphabetical order is returned.
Args:
course_data (Pandas DataFrame): A DataFrame with two columns: `course_id` and `number of students`
Returns:
list: A list containing the id of the course with the most students and the number of students enrolled in it.
"""
### FIRST SOLUTION (x5-8)
# max_course = -1
# id_course = None
# for index, row in df_course.iterrows():
# if row['Number of students'] > max_course:
# max_course = row['Number of students']
# id_course = row['Course ID']
# elif row['Number of students'] == max_course:
# if id_course is None or row['Course ID'] < id_course:
# id_course = row['Course ID']
# return [id_course, max_course]
### SECOND SOLUTION (x150-200)
# course_data.sort_values(
# kind="quicksort",
# by=["Number of students", "Course ID"],
# ascending=[False, True],
# inplace=True,
# )
# result = course_data.head(1)
# return [result["Course ID"].values[0], result["Number of students"].values[0]]
### THIRD SOLUTION (x1500-2000)
#Find the maximum number of students
max_students = df_course['Number of students'].max()
# Filter the DataFrame to get the courses with the maximum number of students
max_courses = df_course[df_course['Number of students'] == max_students]
# Find the course ID that comes first alphabetically among the max courses
chosen_course_id = max_courses['Course ID'].min()
# Get the number of students for the chosen course
chosen_course_students = max_students
# Return both the course ID and the number of students
result = [chosen_course_id, chosen_course_students]
return result
# END SOLUTION
""" # BEGIN PROMPT
def optimized_function(course_data):
...
"""; # END PROMPT
Instructions
You can test your function by running the cell below. To test for the speed you first need to be sure that the function is correct.
tester = AlgorythmOptimizationTest(perf_counter, optimized_function, df_course)
tester.test(n=3)
Your function is correct. Optimized function is 1731.00 times faster than the slow function.
Synthesis¶
It is now time to step back and reflect on what you have discovered in this notebook!
Final reflection time!
Summarize what you have learned from this notebook:
- Explain how you can measure the carbon footprint of a piece of software
- List 3 different factors that affect the carbon footprint of classic algorithms
- Identify 2 important phases in the machine learning life cycle in terms of CO² emissions
- List one important factor that affects the carbon footprint of a machine learning model
Feedback - Click on the "..." below only once you have really tried to answer the question!
Using a codecarbon tracker, you can measure the energy consumned by the hardware executing the code and codecarbon converts this energy into a carbon footprint using information about the geographical location of the machines.
We have seen in this notebook 3 different factors which affect the carbon footprint of classic algorithms:
- the computational complexity of the algorithm and the quantity of data,
- the frequency and duration of use,
- and the geographical location.
In the machine learning life cycle we can distinguish two different phases in terms of CO² emissions:
- the development phase, which encompasses the training of the model (which can be done multiple times to optimize the performances) and its evaluation
- the inference phase, when the model is used to make predictions by users (i.e. after deployment)
We have seen in this notebook that the type of model (i.e. its architecture) will have an important impact on the overall carbon footprint of the model. The model type will affect the training and inference footprints differently: some types of models will have a higher footprint at training and lower at inference, and vice versa.
It is worth noting that the size and parameters of the models as well as the characteristics of the data also play an important role in their overall carbon footprint, but we have not really explored this question in this notebook.
Congratulations! You have finished this notebook!
