Measuring Rust Test Coverage using mozilla/grcov

How do you usually measure test coverage in Rust?

I think many people use kennytm/cargo-kcov or xd009642/tarpaulin, but there is also an official(?) tool called mozilla/grcov. This project was started for Firefox's coverage measurement, and it is expected to continue to be developed stably in the future. In fact, frequent commits can be seen.

https://github.com/mozilla/grcov

In this article, I will create a shell script that easily measures test coverage using mozilla/grcov. Since it is not yet mature, the usage may change in the future, so please confirm it yourself when using it.

Preparation

Prepare the necessary tools for execution.

• grcov
  • This is the main tool. Easy to install with cargo install grcov
• Nightly Rust
  • As usual, the Nightly version is required for test coverage measurement. Easy to install with rustup install nightly
• lcov
  • Used to convert coverage reports to HTML (genhtml). On Mac, you can install it with brew install lcov etc.

Execution

TL;DR

#!/usr/bin/env bash

set -eux

PROJ_NAME=$(cat Cargo.toml | grep -E "^name" | sed -E 's/name[[:space:]]=[[:space:]]"(.*)"/\1/g' | sed -E 's/-/_/g')
rm -rf target/debug/deps/${PROJ_NAME}-*

export CARGO_INCREMENTAL=0
export RUSTFLAGS="-Zprofile -Ccodegen-units=1 -Cinline-threshold=0 -Clink-dead-code -Coverflow-checks=off -Zno-landing-pads"

cargo +nightly build --verbose
cargo +nightly test --verbose

grcov ./target/debug/deps -s . -t lcov --llvm --branch --ignore-not-existing --ignore-dir "/*" -o lcov.info
genhtml -o report/ --show-details --highlight --ignore-errors source --legend lcov.info

If you run this at the project root, the report will appear in report/index.html.

Details

I will create a shell script based on the travis version script in grcov's README. The goal is to generate a measurement report just by running it.

First, let's check the flags. Set CARGO_INCREMENTAL and RUSTFLAGS to configure appropriate settings for measurement. Please change these according to your use case.

export CARGO_INCREMENTAL=0
export RUSTFLAGS="-Zprofile -Ccodegen-units=1 -Cinline-threshold=0 -Clink-dead-code -Coverflow-checks=off -Zno-landing-pads"

Since we're using nightly features in the flags, specify the channel to use with +nightly.

cargo +nightly build --verbose
cargo +nightly test --verbose

After the test is completed, it's finally time for grcov. It's a bit surprising that ./target/debug/deps is specified, but GCNA/GCNO files are generated here. The overview is explained in cov, so please read it if you're interested.

grcov ./target/debug/deps -s . -t lcov --llvm --branch --ignore-not-existing --ignore-dir "/*" -o lcov.info

A temporary file is generated at the location specified by -o. This file is converted to HTML to make it readable for ordinary humans. The result is output to the directory specified by -o.

genhtml -o report/ --show-details --highlight --ignore-errors source --legend lcov.info

Now, while these steps should work well, when actually operating, you'll notice that errors occur when the code changes.

+ grcov ./target/debug/deps -s . -t lcov --llvm --branch --ignore-not-existing --ignore-dir '/*' -o lcov.info
Error in computing counters:
Unexpected number of edges (in xxx) in xxx

Is it because it's incremental? Running cargo clean every time works well, but it takes a very long time to rebuild dependencies, which is not practical. Since I couldn't pinpoint the cause (if you know, please comment), I dealt with it by deleting the relevant GCNA/GCNO files as shown below for now. To increase versatility, PROJ_NAME is not hardcoded but taken from Cargo.toml.

PROJ_NAME=$(cat Cargo.toml | grep -E "^name" | sed -E 's/name[[:space:]]=[[:space:]]"(.*)"/\1/g' | sed -E 's/-/_/g')
rm -rf target/debug/deps/${PROJ_NAME}-*

This might not work well in some cases, so if build speed is not important, it's better to run cargo clean beforehand every time.

This completes it. Putting all of this together results in the shell script shown in the TL;DR.

Conclusion

This article created a script to measure test coverage using mozilla/grcov. Even when changes are made, dependencies are not rebuilt, so it's thought that measurement can be done at a practical speed. As grcov is being actively developed, I want to keep following it in the future.