How to Create a Custom Skill for Mycroft AI on Raspberry Pi

2022-03-21 | By ShawnHymel

License: Attribution Raspberry Pi

Smart speakers are incredibly popular home devices that people can use to get local weather updates, order items, search for information, and play games by just using their voice. While the popular smart speakers, like the Amazon Echo and Google Home, are robust and can hold impressive (albeit short) conversations, they are built on proprietary systems.

In this tutorial, I will show you how to create a custom skill in Python for the open source smart speaker system known as Mycroft AI. We will use the Raspberry Pi version of Mycroft known as Picroft. Additionally, we will connect an LED and a servo motor to the Raspberry Pi and control such hardware with our voice via the skill.

See here if you would like to watch this tutorial in video form:

Requried Hardware

For this project, you will need the following hardware:

Raspberry Pi 3B+ or 4B
Pololu Micro Maestro servo controller
Servo motor
Microphone
LED
220 Ohm resistor
Breadboard
F/F and F/M jumper wires
Headphones or powered speaker

Maestro Configuration

You will need to configure the Maestro board using a PC, Mac, or Linux program first before being able to control it using a USB cable. This is because, by default, the Maestro listens for commands on its UART port.

Download the Maestro Control Center and install it on your computer. Run the program and connect your Maestro board to your computer using a USB cable. You should see a serial number appear in the Connected to drop-down menu.

Go to the Serial Settings tab. Under Serial mode, select USB Dual Port. This will allow your Maestro board to communicate with this software application as well as receive serial commands to control motors.

Configure Maestro servo controller

Wiring

Connect the components as follows:

Wiring for Picroft Raspberry Pi

Note that we are using 5V from the Raspberry Pi to power the servos connected to the Maestro board. This works for a single, small servo motor. However, if you wish to power more than one servo or larger servos, you will need to use a different power supply for the servos, as the Raspberry Pi can only output a few hundred milliamps.

Use a USB mini cable to connect the Maestro board to the Rasbperry Pi. We will send serial commands over this USB cable to control the servos.

Plug your headphones or powered speaker into the audio jack on the Raspberry Pi. Finally, connect a microphone to the Pi. The Adafruit USB microphone worked well for this demo. You may have luck with other mics, but I make no promises (it can be a struggle to get microphones to work in Linux).

Picroft Setup

Download the Picroft image (a Raspberry Pi-ready version of Mycroft AI) from this link in the README of this repository and use it to flash a microSD card. Note that I used the 2021-06-04 version of Picroft for this demo.

You will need to connect a keyboard, mouse, and monitor the first time.

Alternatively, you can connect a USB-to-UART serial converter (e.g. a 3.3V FTDI board) to the Raspberry Pi to get a serial terminal in order to configure the Pi. Note that in order to enable the UART port for the serial terminal, plug the SD card into your computer and open the boot partition. Add the following line to boot/config.txt

Copy Code

# Enable UART console
enable_uart=1

Plug the SD card into yoru Pi and apply power. Whether you are using a keyboard, mouse, monitor or a serial terminal setup, you should be presented with a login prompt. Use the following default credentials:

Username: pi
Password: mycroft

Run the following command:

Copy Code

sudo raspi-config

You should see the Raspberry Pi configuration screen.

Raspi-config menu

Go into System Options > Wireless LAN and follow the prompts to connect the Pi to your local WiFi (you could use Ethernet instead if you wish). You also want to go into Interface Options > SSH and enable SSH.

When your Pi reboots, log in again and enter the following:

Copy Code

ifconfig

Copy the IP address of your Raspberry Pi.

You should be able to reboot and log in via SSH now. Use whatever SSH program you like (I use PuTTY on Windows)

Configure Mycroft

When you first log in via SSH, you should be presented with a guided setup of Mycroft (enter ‘Y’ if asked to go through the guided setup). Follow the prompts to set up the Mycroft service on the Pi. Note that several of the steps require installing additional packages, which could take some time.

Mycroft setup screen

The setup process includes selecting microphone and speaker input/outputs. You will be given the option to test their functionality.

When the setup process is done, you should be presented with the Mycroft CLI environment, which allows you to view debugging information, see what Mycroft is doing, and send commands to the Mycroft interpreter.

Assuming you have an active Internet connect, you should see Mycroft install additional packages and skills. Be patient while those install. When that process is complete, you should see (and hear) Mycroft output a registration code. You’ll want to write it down.

For example, in the screen below, the registration code is “U9ANFV.”

Mycroft device registration code

Open a browser on your computer and head to home.mycroft.ai. Create an account (if you have not done so already). Log into your account, click on your user icon, and select Devices. Click Add Device. Enter your registration code and fill out the rest of the information about your Picroft device.

With any luck, your Mycroft device (Picroft, in this case) should connect to the Mycroft backend services and come to life. Feel free to ask it some questions, like “Hey Mycroft, what’s the weather like?”

I find that Mycroft (as it is an open source project) can be a little slow to respond sometimes. You’ll need to speak slowly and clearly.

Create a Mycroft Skill

Press ctrl+c to exit the Mycroft CLI interface. Note that the Mycroft service will still run in the background. To start, we can use some Mycroft tools to help us create a skill template.

Copy Code

mycroft-msk create

This tool will walk us through the process of creating a skill.

Note: you may get an error that says something like “error importing ‘get_config_path’ from ‘git.config’. If so, you will need to reinstall gitpython:

Copy Code

mycroft-pip uninstall gitpython
mycroft-pip install gitpython

mycroft-pip is a wrapper for pip, which helps us install Python packages in the Mycroft virtual environment (so they won’t affect the rest of the Linux or Python system).

Once gitpython is done installing, try running `mycroft-msk create` again. Follow the prompts to give your skill a name (I’ll use the name robo-interact for this demo), short description, long description, author, icon, and so on. If asked about providing GitHub details, you can skip that, if you wish (or fill them out, if you would like to check your skill into GitHub for version control).

You’ll be asked to define some phrases to trigger your skill. These are stored in an intent file. For example, you might enter:

Copy Code

can you {action}
give me a {action}

Where {action} is a variable that can change, depending on what the user says. We’ll be able to get that variable in string form in our Python code.

Next, we’ll be asked to define what Mycroft says in response to your skill, known as the dialog. For example:

Copy Code

of course I can {action}
how’s this for a {action}

Once again, we can pass arguments to this spoken dialog by using curly braces {}. Mycroft will speak the word given in that argument, which we can define in our skill.

When asked if the skill requires other packages or skills, enter ‘Y’ for yes. You do not need to create a GitHub repository around your skill, unless you plan to check it into GitHub.

Let’s create a new dialog file that details what Mycroft will say when it cannot perform one of the requested actions.

Copy Code

cd /opt/mycroft/skills/robo-interact/
nano locale/en-us/negative.interact.robo.dialog

Note that your locale directory may change (e.g. en-us for me, as I set up Mycroft for US English). In that file, enter some phrases that Mycroft can choose (randomly):

Copy Code

I don’t think so
I’m afraid I can’t do that

Save and exit.

Add Dependencies to Your Skill

Edit manifest.yml in the main skill directory. This is how we tell Mycroft to install dependencies and libraries when it installs our skill.

Copy Code

nano manifest.yml

Add the following lines:

Copy Code

dependencies:
  python:
    - RPi.GPIO
    - pyserial

Save and exit. Now, when someone installs this skill for the first time, the RPi.GPIO and pyserial Python packages will be installed. However, note that because we are developing the skill, Mycroft will not automatically install these dependencies for us. To do that in our development environment, we have to install them manually:

Copy Code

mycroft-pip install RPi.GPIO pyserial

We’ll use FRC4564’s maestro.py module to control the Maestro board over the USB connection. However, Mycroft struggles to load local Python modules. To get it to work, we’ll need to wrap this module in a package.

Copy Code

mkdir maestro
cd maestro
nano __init__.py

In that file, enter:

Copy Code

from .maestro import *

Save and exit. The `.` in front of maestro tells Python to look for a local module, which we’ll download using the following:

Copy Code

wget https://raw.githubusercontent.com/FRC4564/maestro/master/maestro.py
cd ..

Define Custom Skill Actions

Open the main skill file:

Copy Code

nano __init__.py

Copy in the following code:

Copy Code

import time
import RPi.GPIO as GPIO
from .maestro import *
from importlib import reload

from mycroft import MycroftSkill, intent_file_handler

# Settings
led_pin = 18            # Physical pin 12
servo_ch = 0            # Servo channel on Maestro board
servo_accel = 50        # Max acceleration (0..255)
servo_speed = 100       # Max speed
servo_pos_begin = 3000  # 0 deg
servo_pos_end = 9000    # 180 deg 

# Skill class that inherits from MycroftSkill
class RoboInteract(MycroftSkill):

    # Constructor
    def __init__(self):
        MycroftSkill.__init__(self)
        
        # Set up LED
        GPIO.setwarnings(False)
        GPIO.setmode(GPIO.BCM)
        GPIO.setup(led_pin, GPIO.OUT)
        
        # Set up servo controller
        reload(maestro)
        self.servo = maestro.Controller()
        self.servo.setAccel(servo_ch, servo_accel)
        self.servo.setSpeed(servo_ch, servo_speed)
        self.servo.setTarget(servo_ch, servo_pos_begin)
        
    # Called after skill loads
    def initialize(self):
        self.log.info("Robo interact skill loaded")

    # Called whenever skill is activated
    @intent_file_handler('interact.robo.intent')
    def handle_interact_robo(self, message):
    
        # Get string value from 'action' variable
        action = message.data.get('action')
        
        # Do a blink
        if action.casefold() == "blink":
            self.log.info("Blinking!")
            self.speak_dialog('interact.robo', data={
                'action': action
            })
            for i in range(2):
                GPIO.output(led_pin, GPIO.HIGH)
                time.sleep(0.5)
                GPIO.output(led_pin, GPIO.LOW)
                time.sleep(0.5)
        
        # Do a wave
        elif action.casefold() == "wave":
            self.log.info("Waving!")
            self.speak_dialog('interact.robo', data={
                'action': action
            })
            self.servo.setTarget(servo_ch, servo_pos_end)
            time.sleep(1.0)
            self.servo.setTarget(servo_ch, servo_pos_begin)
        
        # Unsupported action
        else:
            self.log.info("No can do")
            self.speak_dialog('negative.interact.robo')

# Called by Mycroft to create skill object
def create_skill():
    return RoboInteract()

Our skill is implemented as a RoboInteract class that inherits from the generic MycroftSkill parent class.

The __init__() function is our constructor, which is called as soon as the object is instantiated. Once the skill is finished loading, the initialize() function will be called. We can log messages to the console and Mycroft log file with the self.log.x() function, where x is debug, info, warning, error, or exception. You can read more about Mycroft logging here.

Skill interactions are implemented as callback functions, which are called whenever the intent of the speaker is matched to one of the lines we put in the .intent file. To define these callback functions, we must wrap them in the decorator @intent_file_handler(file), where file is the name of our intent file (that we created in the setup process).

Inside this callback function, you can get argument values from the {} parameters by calling message.data.get(‘variable-name’). In our example, we get a string from the argument defining the {action} variable. We can then parse this string to determine how Mycroft should react: blinking an LED, moving a servo arm, or claiming that the action cannot be done.

At the bottom of the code, you’ll see a create_skill() function that simply instantiates an object from our skill class. This must be present, as the Mycroft service looks for and calls this function during the setup and skill registration process.

Run It

Save your work and open the Mycroft CLI environment.

Copy Code

mycroft-cli-client

Mycroft should automatically find and register your skill. Pay attention to the log to ensure there are no errors.

Try saying “Hey Mycroft, can you blink?” If everything works, you should see the LED flash twice while the voice assistant gives you an audio affirmative.

Mycroft AI blinking an LED