How to Build an AI-powered Toaster

2022-09-19 | By ShawnHymel

License: Attribution Arduino

We can treat the toasting process like a predictive maintenance problem: how do we stop the toasting before the bread in question becomes irrevocably damaged (i.e. burnt)? We’ll use a variety of gas sensors and machine learning to accomplish this task.

See this video demonstration of the Perfect Toast Machine in operation:

Required Hardware

You will need the following components:

Wio Terminal
Grove Multichannel Gas Sensor v2
Grove SPG VOC and eCO2 gas sensor
Grove BME680 temperature, pressure, and humidity sensor
Grove I2C Hub (6 port)
Grove cable (100 cm)
Ammonia gas sensor
Pololu Carrier for MQ Gas Sensors
Fan (40 mm, 5V)
2x 10kΩ resistors
N-channel MOSFET
Mounting plate (e.g. a small piece of aluminum)
Various wire, screws, nuts, standoffs

Hardware Connections

First, we need to hack the toaster. Open the toaster and find the circuit board that controls the toasting process. Use a multimeter to identify the following 3 nodes:

Ground (GND)
Node that becomes 3.3 or 5 V during the toasting process (for example, an LED that turns on when you press the lever down)
Node that connects to GND when the “cancel” button is pressed

Tack-solder 3 wires to each of these nodes. In the photo below, the black wire goes to ground, the green wire goes to the high side of the limiting resistor for the “toasting” LED (i.e. 5V during “toasting” and 0V otherwise), and the yellow wire goes to the “cancel” button node opposite GND.

Hacking a toaster

Attach all of the sensors and fan to the mounting plate. You’ll want to position the fan to steadily blow air over the sensors. Use the I2C hub to connect all of the sensors together, and use the long Grove cable to connect the hub to the Wio Terminal. You’ll also want long wires to run from the Wio Terminal to the ammonia sensor (as it is an analog sensor, not I2C).

You need the MOSFET in open-drain configuration to successfully control the “cancel” button. The Wio Terminal might support open-drain GPIO, but I was too lazy to dig through the SAMD51 datasheet to figure out how to do this in code. The voltage divider is needed to convert the 5V “toasting” node to 2.5 V (still considered logic HIGH for 3.3V pins).

IMPORTANT: the SAMD51 GPIO pins are NOT 5V tolerant! Make sure you use a divider, diode, etc. to drop the voltage if you’re trying to sense something from 5V logic.

Perfect Toast Machine circuit diagram and schematic

Screw/bolt everything to the aluminum plate (or some other mounting device).

Sensor array for the perfect toast machine

Mechanical Build

Construct a cage or arm that suspends the collection of sensors above the toaster. The microcontroller (Wio Terminal) should not be placed with the sensors to avoid letting it get too hot.

Toaster with cage and sensors

Data Collection

The model I created worked in my environment. It may or may not work for you, which means you’ll likely need to collect data in your environment. Head to https://github.com/ShawnHymel/perfect-toast-machine to view all of the code for this project. Upload toast-odor-data-collection to the Wio Terminal. Read the comments in the code to determine which libraries you need to install prior to running the code.

Make sure the Wio Terminal is plugged into a computer for the data collection process. I recommend waiting 15-30 minutes to let the gas sensors warm up. Use Python (v3+) to run serial-data-collect-csv.py to have it listen for serial data from the Wio Terminal. This will log each toasting instance to a CSV file on your computer. See this readme to learn how to use serial-data-collect-csv.py.

Start the toasting process with a piece of bread. Press button C (on the top of the Wio Terminal) to tag the data in one of 3 states: background (not toasting), toasting, or burnt. Use your senses (sight, smell) to determine when you think the toast is “burnt.”

Collecting burnt toast data

See the CSV files in the datasets/ directory to see how I labeled the raw data. You’re welcome to use that data as a starting point (I just can’t promise that it will give you a model that works in your environment). Each folder is the brand and a description of the bread used. The prefix of each CSV file is where I pulled the bread from (e.g. room temperature, refrigerator, freezer).

Data Curation

Run this notebook in Google Colab, following all of the directions: https://github.com/ShawnHymel/perfect-toast-machine/blob/main/ptm_dataset_curation.ipynb

Note that the script will automatically download the dataset from the GitHub directory. You can skip those cells and manually upload your data to the dataset/ folder in Colab if you wish to use your own data.

At the end of Step 2, you can view plots of your captured raw gas samples (one at a time). Blue is “background,” green is “toasting,” and red is “burnt.”

Raw gas data over time

At the end of Step 3, you should see means, standard deviations, minimums, and ranges printed out. Copy those down–you’ll need the means and standard deviations for your inference code.

Step 4 will produce an out.zip file containing your curated dataset. Download this file and unzip it.

Machine Learning Model Training

Clone this Edge Impulse project as a starting point: https://studio.edgeimpulse.com/public/129477/latest

If you wish to use your own data, delete all of the data in the project and upload the data from the unzipped out.zip file. Note that you should upload files in the training/ directory to training in Edge Impulse and upload files in testing/ to testing.

Go to Raw data. Click Save parameters and then Generate features.

When that’s done, go to Regression. Feel free to modify the model, if you’d like. Click Start training and wait for training to finish.

Go to Model testing and click Classify all. When that’s done, you should see some estimates. Expected outcome is the ground-truth label that expresses the number of seconds until the toast is burned, which is calculated based on when the state label transitioned from “toasting” to “burnt” (0 being the moment we believe the toast went from "toast" to "burnt toast"). The result is the output of our model, given the test input data. For the most part, the model is capable of predicting “time till burnt” within a few seconds. Interestingly, the model seems to be more accurate the closer to 0.

Model testing in Edge Impulse

Deployment

Go to the Deployment page, select Arduino library, and click Build. When the library has been downloaded (do not unzip it!) open the Arduino IDE, click Sketch > Include Library > Add .ZIP Library… Select the Arduino library that you just downloaded from Edge Impulse. Note the name of the library! If it is different than ei-perfect-toast-machine-arduino-x.x.x.zip, you will need to change the .h include file name in your inference code.

Copy the perfect-toast-machine Arduino code found here to a new Arduino project. Read the comments at the beginning to see which libraries you need to install. Rename perfect-toast-machine_inferencing.h if you used a different project name in Edge Impulse. Upload the code to your Wio Terminal.

Copy the means and standard deviations from the Colab script to the means and std_devs arrays, respectively.

The first time you try the project, pay attention to the number displayed on the Wio Terminal. This shows the number of predicted seconds until the toast is burned. If your toast comes out to light, lower the CANCEL_THRESHOLD in the code (i.e. wait until it is closer to being burned before popping the toast up). If the toast is too dark, increase the CANCEL_THRESHOLD (i.e. stop the toasting process sooner).

With a little tweaking, you should be able to make perfect toast! Try different types of bread, different thicknesses, different starting temperatures, etc. It should also work for 2 slices of bread and even bagels!

Perfect toast from the perfect toast machine

Recommended Reading

All code in this project can be found in this GitHub repository. The Edge Impulse project used for training the machine learning model is found here.

Interestingly enough, creating the perfect toast is an exercise in predictive maintenance. Instead of “toast,” imagine we’re talking about an expensive piece of machinery. Is it possible to create a system that predicts when the machinery will fail (analogous to the toast burning) and notify us before it does fail (e.g. stopping the toasting process just before burning)?

Performing routine maintenance might help limit or prevent machine failure, but it is often done more than necessary, which increases equipment downtime. Predictive maintenance systems help limit this downtime by notifying us before a machine breaks so we can repair it at the appropriate times. You can read more about predictive maintenance in this blog post.