Measuring Rust Test Coverage using mozilla/grcov

How do you usually measure test coverage in Rust?

I think many people use kennytm/cargo-kcov or xd009642/tarpaulin, but there is also an official tool called mozilla/grcov. This project was started for Firefox's coverage measurement, and it is expected to continue to be developed stably in the future. In fact, frequent commits can be seen.

https://github.com/mozilla/grcov

In this article, we will create a shell script that easily measures test coverage using mozilla/grcov. Since it is not yet mature, the usage may change in the future, so please confirm it yourself when using it.

Preparation

We will prepare the necessary tools for execution.

• grcov
  • This is the tool itself. It can be easily installed with cargo install grcov.
• Nightly Rust
  • The Nightly version is necessary for test coverage measurement. It can be easily installed with rustup install nightly.
• lcov
  • This is used to generate an HTML coverage report (genhtml). On Mac, it can be installed with brew install lcov.

Execution

TL;DR

#!/usr/bin/env bash

set -eux

PROJ_NAME=$(cat Cargo.toml | grep -E "^name" | sed -E 's/name[[:space:]]=[[:space:]]"(.*)"/\1/g' | sed -E 's/-/_/g')
rm -rf target/debug/deps/${PROJ_NAME}-*

export CARGO_INCREMENTAL=0
export RUSTFLAGS="-Zprofile -Ccodegen-units=1 -Cinline-threshold=0 -Clink-dead-code -Coverflow-checks=off -Zno-landing-pads"

cargo +nightly build --verbose
cargo +nightly test --verbose

grcov ./target/debug/deps -s . -t lcov --llvm --branch --ignore-not-existing --ignore-dir "/*" -o lcov.info
genhtml -o report/ --show-details --highlight --ignore-errors source --legend lcov.info

Running this in the project root will generate a report in report/index.html.

Details

We will create a shell script by referring to the grcov README's Travis version script. The goal is to " generate a measurement report with a single command ".

First, we check the flags. We set CARGO_INCREMENTAL and RUSTFLAGS to make the settings suitable for measurement. You may need to change them depending on your use case.

export CARGO_INCREMENTAL=0
export RUSTFLAGS="-Zprofile -Ccodegen-units=1 -Cinline-threshold=0 -Clink-dead-code -Coverflow-checks=off -Zno-landing-pads"

Since we are using Nightly's functionality with flags, we specify the channel with +nightly.

cargo +nightly build --verbose
cargo +nightly test --verbose

After the test is complete, it's finally time for grcov. We specify ./target/debug/deps, which may seem unexpected, but the GCNA/GCNO files are generated here. For details, see the cov documentation.

grcov ./target/debug/deps -s . -t lcov --llvm --branch --ignore-not-existing --ignore-dir "/*" -o lcov.info

The temporary file specified with -o is generated. We convert this file to HTML to make it readable by humans. The result is output to the directory specified with -o.

genhtml -o report/ --show-details --highlight --ignore-errors source --legend lcov.info

However, when we actually run it, we notice that an error occurs when the code changes.

+ grcov ./target/debug/deps -s . -t lcov --llvm --branch --ignore-not-existing --ignore-dir '/*' -o lcov.info
Error in computing counters:
Unexpected number of edges (in xxx) in xxx

It seems that incremental compilation is causing the issue. Running cargo clean every time solves the problem, but it takes a very long time due to the rebuild of dependencies. Since we couldn't identify the cause (please comment if you know), we temporarily solved it by deleting the corresponding GCNA/GCNO files. We also take PROJ_NAME from Cargo.toml instead of hardcoding it to increase versatility.

PROJ_NAME=$(cat Cargo.toml | grep -E "^name" | sed -E 's/name[[:space:]]=[[:space:]]"(.*)"/\1/g' | sed -E 's/-/_/g')
rm -rf target/debug/deps/${PROJ_NAME}-*

There may be cases where this doesn't work, so it's a good idea to run cargo clean beforehand if build speed is not important.

That's it. The above process is summarized in the TL;DR shell script.

Summary

We created a script that measures test coverage using mozilla/grcov. Since it doesn't rebuild dependencies when the code changes, it can measure coverage at a practical speed. mozilla/grcov is actively being developed, so we'd like to keep an eye on it in the future.