Rust code coverage with Github workflows

2020.01.20 Rust CI Coverage

contents

• Coveralls.io using Cargo Tarpaulin
• Coveralls.io using official Github action
• Codecov.io
• Bonus: adding a badge for the workflow

Github workflows being quite new, there seems to be no straight forward instructions on how to set up coverage uploads to Coveralls from Github workflows and Tarpaulin.

Uploads to Codecov is also covered here for similar use cases.

Coveralls.io using Cargo Tarpaulin¶

EDIT: this seems to have some serious issues, it may be worth using the Coveralls Github action, as described in the next section.

First, add your Coveralls repo token to the repository's settings (doc).

Repository settings -> Secrets -> Add a new secret

Tarpaulin supports uploads to Coveralls natively, so in theory all that has to be done is to add something like the following to your .github/workflows/build.yml:

cargo install cargo-tarpaulin
cargo tarpaulin --ciserver github-ci --coveralls $COVERALLS_REPO_TOKEN

The repo token is added to the environment for this step, as follows:

    - name: Push to coveralls.io
      env:
        COVERALLS_REPO_TOKEN: ${{ secrets.COVERALLS_REPO_TOKEN }}
      run: |
        cargo install cargo-tarpaulin
        cargo tarpaulin --ciserver github-ci --coveralls $COVERALLS_REPO_TOKEN

There is one caveat here, that by default Github workflows checks out one commit only, and no refs, presumably for speed, so Tarpaulin is unable to find the Git information it needs so that it can push the coverage data identifying what branch it pertains to. As such, by default you end up with Coveralls having coverage data for an unknown branch.

This can be fixed by using the Checkout v2 action provided by github, rather than v1, and providing it with the parameter fetch-depth: 0, to checkout full git history.

With this change, we end up with a complete yaml like:

name: build

on: [push, pull_request]

jobs:
  build:

    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v2
      with:
        fetch-depth: 0
    - name: Push to coveralls.io
      env:
        COVERALLS_REPO_TOKEN: ${{ secrets.COVERALLS_REPO_TOKEN }}
      run: |
        cargo install cargo-tarpaulin
        cargo tarpaulin --ciserver github-ci --coveralls $COVERALLS_REPO_TOKEN

And this seems to work for now, uploading coverage data, with a known branch.

Coveralls helpfully shows you how to embed a badge for the repo on its main page for the repository, e.g. in markdown:

[![Coverage Status](https://coveralls.io/repos/github/<user>/<repo>/badge.svg?branch=master)](https://coveralls.io/github/<user>/<repo>?branch=master)

EDIT: This doesn't seem to work properly, leaking the repo token (as build ID) on the Coveralls web UI. Some investigation needed here.

EDIT: In the meantime, the below method works for coverage analysis and upload using grcov and Coveralls' Github action.

Coveralls.io using official Github action¶

Coveralls provides an action for uploading LCOV coverage data to coveralls.io. This is in fact the only method mentioned on their CI support documentation for Github workflows.

This says to add the following to your wokflow yaml:

- name: Coveralls GitHub Action
  uses: coverallsapp/github-action@v1.0.1

Getting LCOV data for your Rust build¶

First, for this to work, we need some LCOV data. LCOV, it appears, is a front-end for the GNU goverage tool, gcov. Thankfully there is a just project called grcov, a Mozilla project which appears to be currently maintained, for profiling during Rust builds to obtain LCOV data.

Use the nightly compiler. The following uses the actions-rs/toochain action to install the nightly toolchain:

    - uses: actions-rs/toolchain@v1
      with:
        profile: minimal
        toolchain: nightly
        override: true

This seems to be required due to:

 error: the option `Z` is only accepted on the nightly compiler

Adding this to the workflow as follows, mostly modified according to the grcov readme's travis.yml example to a Github workflow format:

    - name: Run grcov
       env:
         PROJECT_NAME: "<your_project_name>"
         RUSTFLAGS: "-Zprofile -Ccodegen-units=1 -Cinline-threshold=0 -Clink-dead-code -Coverflow-chec  ks=off -Zno-landing-pads"
         CARGO_INCREMENTAL: 0
       run: |
         cargo build --verbose
         cargo test --verbose
         zip -0 cov.zip $(find . -name "$PROJECT_NAME*.gc*" -print)
         grcov cov.zip -s . -t lcov --llvm --ignore-not-existing --ignore "/*" -o lcov.info

Note: remember to substitute hyphens in your project name with underscores.

Adding CARGO_INCREMENTAL=0, to disable incremental builds of the crate itself, appears to be required to allow the profiling build to complete without the error:

 error: can't instrument with gcov profiling when compiling incrementally

And then another step to upload using the Github action mentioned above:

    - name: Push grcov results to Coveralls via GitHub Action
      uses: coverallsapp/github-action@v1.0.1
      with:
        github-token: ${{ secrets.GITHUB_TOKEN }}
        path-to-lcov: "lcov.info"

Works, quite nicely.

According to the documentation, Codecov uses the GITHUB_TOKEN provided (and handled) by the action to authenticate the upload, so no messing with a repo token as above, which seemed problematic when interacting with Coveralls from a Github action. This also allows it to comment on PRs.

Codecov.io¶

Add a CODECOV_TOKEN secret (as above).

To get coverage data and upoad it to Codecov we need to run the following:

cargo install cargo-tarpaulin
cargo tarpaulin --out Xml 
bash <(curl -s https://codecov.io/bash) -X gcov -t $CODECOV_TOKEN

And add the secret as an env var.

With this we get:

name: build

on: [push, pull_request]

jobs:
  build:

    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v2
      with:
        fetch-depth: 0
    - name: Push to codecov.io
      env:
        CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
      run: |
        cargo install cargo-tarpaulin
        cargo tarpaulin --out Xml
        bash <(curl -s https://codecov.io/bash) -X gcov -t $CODECOV_TOKEN

It will probably help to have the history available in the checkout as described above for Coveralls, but this wasn't clear from any errors reported by the Codecov upload script.

Bonus: adding a badge for the workflow¶

See the Github doc.

For example:

![build](https://github.com/<user>/<repo>/workflows/build/badge.svg)

The name of the badge, and the name displayed is taken from the name field in the workflow yaml.