Page MenuHomePhabricator

Support Rust repositories in CI
Closed, ResolvedPublic

Description

There is a slowly growing group of Wikimedia developers migrating to using Rust for its various performance and memory safety features (c.f. T194953, Tools using Rust). Personally I'm starting to switch some of my Toolforge tools (hosted in Gerrit) to use it.

I imagine we'd have the following jobs as a starting point:

  • rust-test, runs the following:
    • cargo fmt -- --check # code formatting check
    • cargo clippy --all-features -- -D warnings # linter/static analysis
    • cargo test --all-features --verbose # build and run tests
  • rust-coverage, (coverage and postmerge)
    • cargo install cargo-tarpaulin
    • cargo tarpaulin --all-features --out=Xml --out=Html --output-dir=coverage
      • Note that this is in cobertura format, we need to convert it to clover using the script we already have
  • rust-rustdoc (publish/postmerge)
    • cargo rustdoc (using nightly) # publish local docs similar to https://docs.rs

We might also have a rust-binary postmerge job that builds the binary in release mode and provides the binary as a jenkins artifact. For each job we also need a variant for the nightly version of rust, which I'll explain in a bit.

The main non-standard thing we have to do is how we install rust/cargo itself. The version packaged in Debian Buster is already dated and doesn't support our need to support nightly (see upstream's explanation at https://doc.rust-lang.org/stable/book/appendix-07-nightly-rust.html). Upstream's solution for distributing the language is to use rustup, a Rust program that downloads and manages pre-built versions or rustc, cargo, etc..

I think we should take the same approach that rust's own docker images use: https://github.com/rust-lang/docker-rust/blob/master/Dockerfile-debian.template. We download rustup-init, use it to bootstrap rustup, which is then in turn used to install rustc/cargo/clippy/rustdoc/etc. We would want to update the image every 6 weeks or so based on the release schedule.

For caching, we need to set the $CARGO_HOME env variable to /cache. We also want to cache ~/src/target to avoid rebuilding everything from scratch, so we might want some cp/mv helpers to put that directory into /cache before and after jobs.

Event Timeline

Change 620284 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[integration/config@master] docker: Add Rust image

https://gerrit.wikimedia.org/r/620284

Change 620284 merged by jenkins-bot:
[integration/config@master] docker: Add Rust image

https://gerrit.wikimedia.org/r/620284

Mentioned in SAL (#wikimedia-releng) [2020-08-14T13:06:03Z] <hashar> Successfully tagged docker-registry.discovery.wmnet/releng/rust:1.45.2-1 # T256827

Change 620395 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[integration/config@master] Add rust-test job

https://gerrit.wikimedia.org/r/620395

Change 620395 merged by jenkins-bot:
[integration/config@master] Add rust-test job

https://gerrit.wikimedia.org/r/620395

The rust-test job seems to be working properly now: https://gerrit.wikimedia.org/r/620148

The main thing it's lacking is caching (tbh I'm surprised how good the performance is without any caching). I think we should store the crates.io cache in castor and use sccache for build caching. I've requested T260414: Request creation of sccache VPS project for experimenting with that.

Change 620804 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[integration/config@master] Add rust-coverage docker image

https://gerrit.wikimedia.org/r/620804

Change 620804 merged by jenkins-bot:
[integration/config@master] Add rust-coverage docker image

https://gerrit.wikimedia.org/r/620804

Change 620817 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[integration/config@master] jjb: Add rust-coverage-publish job

https://gerrit.wikimedia.org/r/620817

Change 620817 merged by jenkins-bot:
[integration/config@master] jjb: Add rust-coverage-publish job

https://gerrit.wikimedia.org/r/620817

coverage report is now generated: https://doc.wikimedia.org/cover/labs-tools-shorturls/ ! except...the HTML view loads external JS from unpkg.com - I've filed https://github.com/xd009642/tarpaulin/issues/534 asking if it can be embedded instead.

At least we're generating clover.xml which allows for historical tracking (it's correct that the repo is at 0%).

Change 621115 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[integration/config@master] rust: Add script to generate documentation

https://gerrit.wikimedia.org/r/621115

Change 621116 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[integration/config@master] Add rust-doc-publish job

https://gerrit.wikimedia.org/r/621116

Change 621115 merged by jenkins-bot:
[integration/config@master] rust: Add script to generate documentation

https://gerrit.wikimedia.org/r/621115

Change 621116 merged by jenkins-bot:
[integration/config@master] Add rust-doc-publish job

https://gerrit.wikimedia.org/r/621116

Legoktm claimed this task.

There are now 3 jobs:

  • rust-test (test/gate-and-submit): checks rustfmt, clippy, and runs any tests
  • rust-coverage-publish (postmerge): runs coverage and publishes XML/HTML reports
  • rust-doc-publish (postmerge): generates HTML documentation and publishes it

Unlike what I originally proposed, all of these run using the latest stable Rust version. If someone has a use case for nightly or using an older version (to match distro packaged), then we should be able to fullfil those needs when its needed, but for now this setup is good enough for the current usecase of Toolforge tools.

Two remaining issues have been split to their own tasks: