Page MenuHomePhabricator

Setup session storage service testing/continuous integration
Closed, ResolvedPublic

Description

The session storage service (aka kask) has unit, functional, and integration tests, and will eventually include a number of other checks (formatting, lint, etc). For the time being, we are proposing to execute integration tests manually against the staging environment prior to release/deployment, (not automatically by CI). The remainder of these tests/checks though need to be invoked automatically on patch submission, in the customary fashion.

Some considerations:

Compilation for deployment

Kask is written in Go, and so its release artifact is a statically compiled binary executable. Given the security-sensitive nature of this service, we've opted to use build dependencies sourced entirely from what is packaged in Debian Stretch (to piggy-back on the excellent lifecycle/security management provided there). This will make the compilation of release artifacts too error prone to be done on developer workstations; An improperly applied environment variable could result in compilation against the wrong dependencies.

The CI environment should be able to compile Kask in a pristine Debian Stretch environment, (with packaged build dependencies applied), and then make the resulting binary available for deployments.

Functional tests

Kask's functional tests require connectivity to a Cassandra database. Additionally, while care has been taken to ensure that tests won't be influenced by previous runs, ideally the database would be spun up fresh on each new run, and tore down after.

Event Timeline

Eevans triaged this task as Medium priority.Nov 8 2018, 8:22 PM
Eevans created this task.
Eevans renamed this task from Testing and continuous integration of the session storage service to Setup session storage service testing/continuous integration.Jan 15 2019, 8:54 PM

Change 490683 had a related patch set uploaded (by Clarakosi; owner: Clarakosi):
[mediawiki/services/kask@master] Add .pipeline/blubber.yaml

https://gerrit.wikimedia.org/r/490683

Change 490678 had a related patch set uploaded (by Clarakosi; owner: Clarakosi):
[integration/config@master] Setup Basic CI for mediawiki/services/kask

https://gerrit.wikimedia.org/r/490678

Just wanted to offer some feedback based on the patches you already have and some of the things @Eevans, @thcipriani and I talked about during All Hands.

What the pipeline will do given your current patches

The patches you've submitted look great, and I've left some more detailed comments there. What you have so far will get you the following for each patchset pushed to gerrit:

  1. A "test" variant image built with the packages you've specified installed and the project source copied in to the lives.in directory.
  2. That image's variants.test.entrypoint will be executed just as you've given it. It's up to you to implement that test process in whatever way makes sense for the project, which you've done. (As of now, we only support a single "test" variant/entrypoint, but we're building out something more flexible/advanced. See comments in T210267 if you're interested in participating in the changes.)

What it won't do given your patches

Since you've configured your project to use the service-pipeline-test, it will stop there.

  • No "production" image will be built
  • Nothing will be published to the docker registry
  • No test deployment w/ end-to-end tests will happen either

Based on what we talked about at All Hands and what you've described in this task, that all seems okay except for the last point.

It seems like you do want a level of end-to-end functional testing performed. We can do that, theoretically, and I only include the qualification because we haven't yet deployed an application with a backend dependency like cassandra into the "ci" k8s namespace. If you're willing to be our guinea pigs, let's try it out!

Getting a test deployment going requires implementation of a Helm chart. We're still fleshing out documentation around this, but if you want to brave this part yourself there is a minimal README in the repo. We can also help over in #wikimedia-pipeline (<- RelEng, Services, and SRE folks involved in the project) with any questions.

Getting a binary artifact out of the built image

Again, this isn't really available yet, but based on our conversations Release-Engineering-Team is working on adding support for extraction of arbitrary files from the images build in pipeline stages. The work is being tracked in T216272: The pipeline should provide a way to save artifacts from a stage so please comment!

Why not go "full pipeline"?

Talking with @thcipriani today about your project and how we can better support it, we were left wondering if we might be able to get this project fully deployed via the pipeline. IIRC @Eevans, you mentioned some reservations about that, but I thought it might be worthwhile checking in again.

Of course @akosiaris and @Joe would probably want to chime in about the production side of things, but from our perspective this project seems like a good candidate to be run on k8s, having the same access to Cassandra as it would being deployed to the current service backends, but with the added benefit of the full service pipeline workflow.

In any case, I hope these comments are helpful. We're excited to roll out more services on the Release Pipeline and thus very eager to help.

Thanks for this thorough feedback @dduvall! We really appreciate it.

We are still developing various parts of Kask so, for now, we are okay with using the service-pipeline-test since it'll test our linters and unit tests on every change.

It seems like you do want a level of end-to-end functional testing performed. We can do that, theoretically, and I only include the qualification because we haven't yet deployed an application with a backend dependency like cassandra into the "ci" k8s namespace. If you're willing to be our guinea pigs, let's try it out!

Eventually, we do want to be able to run end-to-end functional tests on each change and on every merge create a Kask base image that can be used by others for testing their Kask configurations. I believe the latter part is currently doable with service-pipeline-test-and-publish. And for the first half, we are excited to be your guinea pigs! :)

Why not go "full pipeline"?

We have three machines in each database that we are planning on using for session storage hence the push for a standalone service. We could probably run docker images on them but I'm not sure if it's worth setting up Kubernetes.

Change 490683 merged by Eevans:
[mediawiki/services/kask@master] Add .pipeline/blubber.yaml

https://gerrit.wikimedia.org/r/490683

Change 493069 had a related patch set uploaded (by Eevans; owner: Eevans):
[mediawiki/services/kask@master] .pipeline/blubber.yaml: Use internal (wmf) golang image

https://gerrit.wikimedia.org/r/493069

Change 493069 abandoned by Eevans:
.pipeline/blubber.yaml: Use internal (wmf) golang image

Reason:
See https://gerrit.wikimedia.org/r/#/c/mediawiki/services/kask/ /493004/ instead

https://gerrit.wikimedia.org/r/493069

Change 490678 merged by jenkins-bot:
[integration/config@master] Setup Basic CI for mediawiki/services/kask

https://gerrit.wikimedia.org/r/490678

Change 493333 had a related patch set uploaded (by Eevans; owner: Eevans):
[mediawiki/services/kask@master] Override GOPATH for Debian packaged build dependencies

https://gerrit.wikimedia.org/r/493333

Change 493333 merged by Eevans:
[mediawiki/services/kask@master] Override GOPATH for Debian packaged build dependencies

https://gerrit.wikimedia.org/r/493333

Change 493338 had a related patch set uploaded (by Eevans; owner: Eevans):
[mediawiki/services/kask@master] Install prometheus client library from backports

https://gerrit.wikimedia.org/r/493338

Change 493338 abandoned by Eevans:
Install prometheus client library from backports

Reason:
This is a mess; Pulling in the backported golang-github-prometheus-client-golang-dev package means (manually) pulling in a number of transitive dependencies, including a newer version of golang-gopkg-yaml.v2-dev.

We should first make every effort to see if golang-github-prometheus-client-golang-dev=0.8.0 (Stretch version) will satisfy our needs before heading down this road.

https://gerrit.wikimedia.org/r/493338

Change 493717 had a related patch set uploaded (by Eevans; owner: Eevans):
[mediawiki/services/kask@master] Use Debian provided code checkers

https://gerrit.wikimedia.org/r/493717

Change 493717 merged by Clarakosi:
[mediawiki/services/kask@master] Use Debian provided code checkers

https://gerrit.wikimedia.org/r/493717

Anything left to do here?

I think we can close it.

Anything left to do here?

I think we can close it.

/me does