Debugging interfaces

Chromebook devices are automated in the lab using Servo boards, which gives access to low-level hardware controls. It connects via USB on the host and either a dedicated debug power on the Chromebook or a USB Type-C for more recent models.

Alternatively, SuzyQ USB-C cables can be used for the same purposes. The documentation below is common for both interfaces.

Dependencies

Using Servo boards requires some software packages to be installed on the host. These are available in the Collabora Sysadmin Debian repository. On LAVA dispatchers controlled by Chef, they are automatically installed by the controller.rb recipe.

The steps below are useful when debugging Chromebooks on an arbitrary computer, to install these packages and dependencies manually after installing Collabora’s apt key:

$ wget https://repositories.collabora.co.uk/sysadmin/debian/pool/tools/c/collabora-archive-keyring/collabora-archive-keyring_0.3debian10.1_all.deb
$ sudo dpkg -i collabora-archive-keyring_0.3debian10.1_all.deb
$ sudo sh -c 'echo "deb https://repositories.collabora.co.uk/sysadmin/debian bullseye tools" >> /etc/apt/sources.list'
$ sudo apt update
$ sudo apt install docker-servod-tools

Developer mode

In order to use Chromebooks with Servo boards, it is necessary to enable developer mode. Follow the steps below to do this on any Chrome OS device:

Reboot the Chromebook and put it in recovery mode with Esc + Refresh + Power. Note that the power button could be on the side of the case.

Note: to enter Recovery mode on Chromeboxes (e.g. puff), press the recovery button (located right above the K-Slot) with a paper clip and then press the power button. Release the recovery button after a second.
Enter developer mode with Ctrl-D, then Enter to turn OS verification off
Wait for the Chromebook to reboot and set up developer mode
Press Ctrl-D to let it boot Chrome OS
Open a terminal on the device with Ctrl + Alt + → and log in as root (no password required).

Note: With some Dell Chromebooks and Chromeboxes, open the terminal with Ctrl + Alt + Fn + F2.

Note: If root doesn’t work, log in with chronos and then run sudo su.

Note: With some other Chromebooks, such as trogdor and volteer, open the terminal with Ctrl + Alt + t. Then open the shell with shell followed by sudo su.

See the Chromium OS documentation for more details about the Firmware Keyboard Interface and Opening a VT-2 terminal.

Recovery

In case something went wrong, for example if the device fails to boot again or if an unknown root password has been set up, the factory OS image can be reinstalled on the Chromebook. Follow the Device OS recovery instructions provided by Google.

In a nushell, on Linux:

create a recovery media (USB stick or SD card) using this script:

$ wget https://dl.google.com/dl/edgedl/chromeos/recovery/linux_recovery.sh
$ chmod 0755 linux_recovery.sh
$ sudo bash linux_recovery.sh

follow the instructions to create the recovery media
insert it in the Chromebook and restart it in recovery mode

It should automatically perform a full OS recovery.

Servo board serial number and running servod

Predefined udev rules will match the VID/PID of known Servo boards and CCD capable devices upon connection with the host and google-docker-servo services will start accordingly.

The rules can be found in /etc/udev/rules.d/99-docker-servo.rules:

# google-docker-servo* matches on our usbmisc child device, then we specify
# SUBSYSTEMS/DRIVERS to walk up from the specific interface, to the overall
# device, which will actually have a serial

# Servo v2
ACTION=="add", SUBSYSTEMS=="usb", ATTRS{idVendor}=="18d1", ATTRS{idProduct}=="5002", ENV{MAJOR}=="189", TAG+="systemd", ENV{SYSTEMD_WANTS}+="google-docker-servo@$name.service", ENV{ID_MODEL}="Google Servo control board %n"

# Servo Micro
ACTION=="add", SUBSYSTEMS=="usb", ATTRS{idVendor}=="18d1", ATTRS{idProduct}=="501a", ENV{MAJOR}=="189", TAG+="systemd", ENV{SYSTEMD_WANTS}+="google-docker-servo@$name.service", ENV{ID_MODEL}="Google Servo Micro control board %n"

# Cr50 (Case Close Debugging)
ACTION=="add", SUBSYSTEMS=="usb", ATTRS{idVendor}=="18d1", ATTRS{idProduct}=="5014", ENV{MAJOR}=="189", TAG+="systemd", ENV{SYSTEMD_WANTS}+="google-docker-servo@$name.service", ENV{ID_MODEL}="Google Cr50 %n"

The idVendor should always be 18d1 for Google products. The idProduct tells which kind of Servo board has been detected.

To find the VID/PID of the connected Servo/CCD capable device, look for this kind of kernel messages while plugging the USB cable of the Servo board or the SuzyQable:

$ sudo journalctl -kf

Oct 26 18:20:51 debian kernel: usb 5-2.3.3.1.3.2: new full-speed USB device number 12 using xhci_hcd
Oct 26 18:20:51 debian kernel: usb 5-2.3.3.1.3.2: New USB device found, idVendor=18d1, idProduct=5014, bcdDevice= 1.00
Oct 26 18:20:51 debian kernel: usb 5-2.3.3.1.3.2: New USB device strings: Mfr=1, Product=2, SerialNumber=11
Oct 26 18:20:51 debian kernel: usb 5-2.3.3.1.3.2: Product: Cr50
Oct 26 18:20:51 debian kernel: usb 5-2.3.3.1.3.2: Manufacturer: Google Inc.
Oct 26 18:20:51 debian kernel: usb 5-2.3.3.1.3.2: SerialNumber: 20006046-8D23B055

All the known types should be covered by the predefined udev rules; if this is a new type of Servo board, add a new rule.

The logs also report the serial number for the connected device. The SerialNumber value is used to create a unique google-docker-servo service and it needs to be added to the/etc/google-docker-servo.conf file on the host.

Each line in the file should have:

A device name which will be used to create symlinks to device nodes and to name the servod docker container
The device serial number
A unique network port
The device board name
The device model (only needed for certain Chromebooks)

For example, using the kernel log above to configure a servod service on port 9902 to control the jelboz Chromebook:

jelboz-local, 20006046-8D23B055, 9902, zork, dalboz

After the file has been edited, replug the Servo board and check that a google-docker-servo service is starting as expected:

$ sudo journalctl -f
Oct 26 18:22:59 debian systemd[1]: Starting "Google Servo control board manager"...
Oct 26 18:23:00 debian run-servod[100102]: INFO:root:Device path: /dev/bus/usb/005/013
Oct 26 18:23:00 debian run-servod[100102]: INFO:root:Board serial: b'20006046-8D23B055'
Oct 26 18:23:00 debian run-servod[100102]: INFO:root:Board configuration: {'servo-name': 'jelboz-local', 'serial-number': '20006046-8D23B055', 'port-number': '9902', 'board-name': 'zork', 'board-model': 'dalboz'}

[...]

Oct 26 18:23:07 debian systemd[1]: Started "Google Servo control board manager".

The google-docker-servo service will start a docker container that runs the servod server to control the connected device. The server’s port is mapped to the network port on the host specified in the /etc/google-docker-servo.conf file:

$ docker ps
CONTAINER ID   IMAGE                                                                 COMMAND                  CREATED              STATUS              PORTS                    NAMES
4763462e7cf8   registry.gitlab.collabora.com/lava/hdctools-docker:release-1022.1-1   "bash /start_servod.…"   About a minute ago   Up About a minute   0.0.0.0:9902->9999/tcp   jelboz-local-docker_servod

The docker container is named after the device’s local alias defined in the /etc/google-docker-servo.conf file.

Some device nodes will be automatically created with symlinks for the cr50 and EC debug consoles:

$ ls -l /dev/google-servo/jelboz-local/
total 0
lrwxrwxrwx 1 root root 10 Oct 26 18:23 cr50-uart -> /dev/pts/6
lrwxrwxrwx 1 root root 10 Oct 26 18:23 ec-uart -> /dev/pts/8

A /dev/ttyUSB* device will be created for the AP debug console:

$ ls -la /dev/ttyUSB*
crw-rw---- 1 root dialout 188, 10 Oct 26 18:23 /dev/ttyUSB1

Low-level Chromebook controls can be accessed through the dut-control tool available in the docker container, or through the XML-RPC API.

A quick manual check to confirm that the Servo interface is working as expected can be done with this command:

$ docker exec jelboz-local-docker_servod dut-control -i
*************
* CONTROL
*************
accum_clear_ctrls               DOC: List of available controls to clear underlying accumulators. Use this in conjunction with 'avg_power_rails' to clear after reading, to ensure a fresh sample on the next timestamp.
------------------------------- GET: {'drv': 'servo_metadata', 'subtype': 'tagged_controls', 'interface': 'servo', 'tag': 'accum_clear_ctrls', 'cmd': 'get', 'interface_prefix': '', 'control_name': 'accum_clear_ctrls'}
------------------------------- SET: {'drv': 'servo_metadata', 'subtype': 'tagged_controls', 'interface': 'servo', 'tag': 'accum_clear_ctrls', 'cmd': 'set', 'interface_prefix': '', 'control_name': 'accum_clear_ctrls'}
adc_ez_config_ctrls             DOC: List of available controls to run easy configuration on all known ADC controls.
[...]

Controls can also be get/set using the XML-RPC API, by connecting to the servod server running in the docker container.

For example, to get the cold_reset state using the API in a Python script:

#!/usr/bin/env python3

from xmlrpc import client

remote_uri = 'http://0.0.0.0:9902'
client = client.ServerProxy(remote_uri, verbose=False)

print(client.get('cold_reset'))

Serial console and power control

The main control to be used for automation is the cold_reset one, which essentially turns the device off when enabled.

To get the current status:

$ docker exec jelboz-local-docker_servod dut-control cold_reset
cold_reset:on

In a separate terminal, monitor the embedded controller console output to check when the firmware starts booting:

$ sudo cat /dev/google-servo/jelboz-local/ec-uart

Then turn the power on and watch the firmware boot, and turn it off again:

$ docker exec jelboz-local-docker_servod dut-control cold_reset:off

# check the ec-uart console output for some activity

$ docker exec jelboz-local-docker_servod dut-control cold_reset:on

If this worked as expected, congratulations! Your Chromebook is now ready to be automated in the LAVA test lab.

Upgrading the Servo v4 firmware

There are many versions of the Servo v4 firmware. It’s recommended to upgrade it to a recent version whenever possible, as older versions have shown some issues with certain boards and Depthcharge versions.

You can check the current Servo v4 firmware version using the CLI that the Servo runs in the first serial terminal it enumerates (/dev/ttyUSB0 in the example below, it could be different in your setup):

$ miniterm /dev/ttyUSB0

> version
Board:   3
RO:      servo_v4_v2.3.22-ecb74cc56
RW:      servo_v4_v2.3.22-ecb74cc56
Build:   servo_v4_v2.3.22-ecb74cc56
         2019-07-23 18:07:15 @chromeos-legacy-release-us-central2-c-x32-40-2efr

The servo_updater tool (installed from the dependencies above) can be used to flash the Servo v4 firmware. To use it, first clone the firmware-tools repo. Then make sure the Servo v4 is connected to your host and run:

$ cd firmware-tools
$ sudo PATH=$PWD/bin:$PATH LD_LIBRARY_PATH=$PWD/lib servo_updater -f <firmware_file>

The servo_v4_fw directory in firmware-tools includes some tested Servo v4 firmware binaries. You can usually find these binaries in the ChromeOS SDK (usr/share/servo_updater/firmware/ inside the cros-build chroot).

If everything goes well you should see an output like this:

Current servo_v4 version is   servo_v4_v2.3.22-ecb74cc56
Available servo_v4 version is servo_v4_v2.4.35-f1113c92b
Updating to recommended version.
usb_updater2 -d 18d1:501b -n servo_v4_fw/servo_v4_v2.4.35-f1113c92b.bin
read 131072(0x20000) bytes from servo_v4_fw/servo_v4_v2.4.35-f1113c92b.bin
Found device.
found interface 4 endpoint 6, chunk_len 64
READY
-------
start
target running protocol version 6 (type 1)
maximum PDU size: 1024
Flash protection status: 0008
version:       servo_v4_v2.3.22-ecb74cc56
key_version: 0
min_rollback: -1
offset: writable at 0x10000
sending 0xe760 bytes to 0x10000
-------
update complete
image updated
usb_updater2 -d 18d1:501b -n servo_v4_fw/servo_v4_v2.4.35-f1113c92b.bin
read 131072(0x20000) bytes from servo_v4_fw/servo_v4_v2.4.35-f1113c92b.bin
Found device.
found interface 4 endpoint 6, chunk_len 64
READY
-------
start
target running protocol version 6 (type 1)
maximum PDU size: 1024
Flash protection status: 0008
version:       servo_v4_v2.3.22-ecb74cc56
key_version: 0
min_rollback: -1
offset: writable at 0
sending 0xe8d4 bytes to 0
-------
update complete
image updated

If you have more than one Servo v4 interface connected to the host, you can specify which one you want to flash using the -s <serial_number> option of servo_updater. The serial number of an interface can also be retrieved from its CLI using the serialno command:

> serialno
Serial number: C1706310161