Page MenuHomePhabricator

Implement CI rules for new kartotherian repo
Closed, ResolvedPublic

Description

Background information

Kartotherian is now monorepo and is hosted on Gerrit. It now should have a CI config to run tests.

Requirements

Kartotherian is running with node 10 and needs mapnik installed for proper testing, the list of dependencies are listed as follows:

"dependencies": {
  "_all": [
    "fonts-dejavu",
    "libboost-filesystem-dev",
    "libboost-regex-dev",
    "libboost-system-dev",
    "libcairo2-dev",
    "libfreetype6-dev",
    "libgdal-dev",
    "libharfbuzz-dev",
    "libjpeg-dev",
    "libpng-dev",
    "libpq-dev",
    "libproj-dev",
    "libtiff-dev",
    "libwebp-dev",
    "libxml2-dev",
    "libmapbox-variant-dev",
    "libboost-program-options-dev",
    "libboost-thread-dev",
    {
      "repo_url": "https://apt.wikimedia.org/wikimedia",
      "release": "stretch-wikimedia",
      "pool": "main",
      "packages": [
        "libmapnik-dev",
        "mapnik-utils",
        "mapnik-doc",
        "libmapnik3.0"
      ]
    }
  ]
}

Acceptance criteria

  • Tests run and do not fail when pushing a new change to Gerrit

Event Timeline

MSantos created this task.Jul 16 2019, 2:40 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 16 2019, 2:40 PM

I'll add node10, and also the pipeline as an experimental job.

Change 523769 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[integration/config@master] layout: [mediawiki/services/kartotherian] Initial task, node10 plus experimental pipeline

https://gerrit.wikimedia.org/r/523769

Change 523769 merged by jenkins-bot:
[integration/config@master] layout: [mediawiki/services/kartotherian] Initial task, node10 plus experimental pipeline

https://gerrit.wikimedia.org/r/523769

Mentioned in SAL (#wikimedia-releng) [2019-07-16T16:48:44Z] <James_F> Zuul: Adding first tasks for mediawiki/services/kartotherian T228170

OK, our bit here is done.

Can the ticket be closed, then?

LGoto added a subscriber: LGoto.Jul 17 2019, 3:36 PM

@MSantos is this done? Feel free to close if so, or move on the board as needed. Thanks!

OK, our bit here is done.

Can the ticket be closed, then?

It's not passing CI yet, so the acceptance criteria aren't meant. I meant "our" == RelEng not "our" == Product Infrastructure, sorry. :-)

OK, our bit here is done.

Can the ticket be closed, then?

It's not passing CI yet, so the acceptance criteria aren't meant. I meant "our" == RelEng not "our" == Product Infrastructure, sorry. :-)

This still needs some tweaking. The CI is failing when building node-mapnik because the Docker image doesn't have mapnik and other dependencies installed:

12:28:41 /bin/sh: 1: mapnik-config: not found
12:28:41 gyp: Call to 'mapnik-config --cflags' returned exit status 127 while in binding.gyp. while trying to load binding.gyp

The requirements session in the description lists all linux packages that should be present when building this service.

Oh, meh. Custom images suck.

Oh, meh. Custom images suck.

It's never easy with maps 😑

Gods, this repo desperately needs to be in the pipeline. This CI test is taking forever.

Change 524271 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[integration/config@master] dockerfiles: Provide node10-kartotherian until it moves to the services pipeline

https://gerrit.wikimedia.org/r/524271

Change 524272 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[integration/config@master] jjb: Provide kartotherian-node10-browser-docker based on custom image

https://gerrit.wikimedia.org/r/524272

Change 524273 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[integration/config@master] layout: [wikimedia/services/kartotherian] Switch to custom job

https://gerrit.wikimedia.org/r/524273

Gods, this repo desperately needs to be in the pipeline. This CI test is taking forever.

Building node-mapnik is the bottleneck, not sure how to make it better though (hosting the built binary somewhere?)

Pushing it as a pre-built binary into a .deb on our infrastructure would be a lot of effort and special-casing, yeah. :-(

Change 524271 merged by jenkins-bot:
[integration/config@master] dockerfiles: Provide node10-kartotherian until it moves to the services pipeline

https://gerrit.wikimedia.org/r/524271

Change 524272 merged by jenkins-bot:
[integration/config@master] jjb: Provide kartotherian-node10-browser-docker based on custom image

https://gerrit.wikimedia.org/r/524272

Change 524273 merged by jenkins-bot:
[integration/config@master] layout: [wikimedia/services/kartotherian] Switch to custom job

https://gerrit.wikimedia.org/r/524273

Mentioned in SAL (#wikimedia-releng) [2019-07-18T18:10:51Z] <James_F> Zuul: [wikimedia/services/kartotherian] Switch to custom job T228170

MSantos claimed this task.Jul 18 2019, 6:44 PM

Change 524304 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[integration/config@master] dockerfiles: [node10-kartotherian] Use SRE-backported custom components where needed

https://gerrit.wikimedia.org/r/524304

Change 524308 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[integration/config@master] jjb: [kartotherian-node10-browser-docker] Bump to latest custom image

https://gerrit.wikimedia.org/r/524308

Change 524304 merged by jenkins-bot:
[integration/config@master] dockerfiles: [node10-kartotherian] Use SRE-backported custom components where needed

https://gerrit.wikimedia.org/r/524304

LGoto removed a subscriber: LGoto.Jul 18 2019, 7:18 PM

Change 524308 merged by jenkins-bot:
[integration/config@master] jjb: [kartotherian-node10-browser-docker] Bump to latest custom image

https://gerrit.wikimedia.org/r/524308