Page MenuHomePhabricator
Create Task
Maniphest T228170

Implement CI rules for new kartotherian repo
Closed, ResolvedPublic
Actions

Description

Background information

Kartotherian is now monorepo and is hosted on Gerrit. It now should have a CI config to run tests.

Requirements

Kartotherian is running with node 10 and needs mapnik installed for proper testing, the list of dependencies are listed as follows:

"dependencies": {
  "_all": [
    "fonts-dejavu",
    "libboost-filesystem-dev",
    "libboost-regex-dev",
    "libboost-system-dev",
    "libcairo2-dev",
    "libfreetype6-dev",
    "libgdal-dev",
    "libharfbuzz-dev",
    "libjpeg-dev",
    "libpng-dev",
    "libpq-dev",
    "libproj-dev",
    "libtiff-dev",
    "libwebp-dev",
    "libxml2-dev",
    "libmapbox-variant-dev",
    "libboost-program-options-dev",
    "libboost-thread-dev",
    {
      "repo_url": "https://apt.wikimedia.org/wikimedia",
      "release": "stretch-wikimedia",
      "pool": "main",
      "packages": [
        "libmapnik-dev",
        "mapnik-utils",
        "mapnik-doc",
        "libmapnik3.0"
      ]
    }
  ]
}

Acceptance criteria

  • Tests run and do not fail when pushing a new change to Gerrit

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
jjb: [kartotherian-node10-browser-docker] Bump to latest custom imageintegration/configmaster+1 -1
dockerfiles: [node10-kartotherian] Use SRE-backported custom components where neededintegration/configmaster+18 -1
dockerfiles: Provide node10-kartotherian until it moves to the services pipelineintegration/configmaster+73 -0
layout: [wikimedia/services/kartotherian] Switch to custom jobintegration/configmaster+2 -2
jjb: Provide kartotherian-node10-browser-docker based on custom imageintegration/configmaster+19 -0
layout: [mediawiki/services/kartotherian] Initial task, node10 plus experimental pipelineintegration/configmaster+12 -0
Customize query in gerrit

Event Timeline

MSantos created this task.Jul 16 2019, 2:40 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 16 2019, 2:40 PM
Jdforrester-WMF claimed this task.Jul 16 2019, 4:35 PM
Jdforrester-WMF added a comment.Jul 16 2019, 4:38 PM

I'll add node10, and also the pipeline as an experimental job.

gerritbot added a comment.Jul 16 2019, 4:39 PM

Change 523769 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[integration/config@master] layout: [mediawiki/services/kartotherian] Initial task, node10 plus experimental pipeline

https://gerrit.wikimedia.org/r/523769

gerritbot added a project: Patch-For-Review.Jul 16 2019, 4:39 PM
gerritbot added a comment.Jul 16 2019, 4:48 PM

Change 523769 merged by jenkins-bot:
[integration/config@master] layout: [mediawiki/services/kartotherian] Initial task, node10 plus experimental pipeline

https://gerrit.wikimedia.org/r/523769

Stashbot added a comment.Jul 16 2019, 4:48 PM

Mentioned in SAL (#wikimedia-releng) [2019-07-16T16:48:44Z] <James_F> Zuul: Adding first tasks for mediawiki/services/kartotherian T228170

Jdforrester-WMF added a project: Release-Engineering-Team-TODO (201907).Jul 16 2019, 5:02 PM
Maintenance_bot removed a project: Patch-For-Review.Jul 16 2019, 5:10 PM
Jdforrester-WMF moved this task from INBOX to Done (within RelEng) on the Release-Engineering-Team-TODO (201907) board.Jul 16 2019, 5:10 PM

OK, our bit here is done.

Jdforrester-WMF reassigned this task from Jdforrester-WMF to MSantos.Jul 16 2019, 5:10 PM
• Mholloway added a comment.Jul 17 2019, 3:36 PM
In T228170#5338071, @Jdforrester-WMF wrote:

OK, our bit here is done.

Can the ticket be closed, then?

LGoto subscribed.Jul 17 2019, 3:36 PM

@MSantos is this done? Feel free to close if so, or move on the board as needed. Thanks!

Jdforrester-WMF added a comment.Jul 17 2019, 5:18 PM
In T228170#5341437, @Mholloway wrote:
In T228170#5338071, @Jdforrester-WMF wrote:

OK, our bit here is done.

Can the ticket be closed, then?

It's not passing CI yet, so the acceptance criteria aren't meant. I meant "our" == RelEng not "our" == Product Infrastructure, sorry. :-)

MSantos added a comment.Jul 18 2019, 3:48 PM
In T228170#5342317, @Jdforrester-WMF wrote:
In T228170#5341437, @Mholloway wrote:
In T228170#5338071, @Jdforrester-WMF wrote:

OK, our bit here is done.

Can the ticket be closed, then?

It's not passing CI yet, so the acceptance criteria aren't meant. I meant "our" == RelEng not "our" == Product Infrastructure, sorry. :-)

This still needs some tweaking. The CI is failing when building node-mapnik because the Docker image doesn't have mapnik and other dependencies installed:

12:28:41 /bin/sh: 1: mapnik-config: not found
12:28:41 gyp: Call to 'mapnik-config --cflags' returned exit status 127 while in binding.gyp. while trying to load binding.gyp

The requirements session in the description lists all linux packages that should be present when building this service.

MSantos edited projects, added Product-Infrastructure-Team-Backlog-Deprecated (Kanban); removed Product-Infrastructure-Team-Backlog-Deprecated.Jul 18 2019, 3:49 PM
Jdforrester-WMF claimed this task.Jul 18 2019, 3:51 PM

Oh, meh. Custom images suck.

Jdforrester-WMF moved this task from Done (within RelEng) to Doing on the Release-Engineering-Team-TODO (201907) board.Jul 18 2019, 3:51 PM
MSantos added a comment.Jul 18 2019, 3:56 PM
In T228170#5346418, @Jdforrester-WMF wrote:

Oh, meh. Custom images suck.

It's never easy with maps 😑

Jdforrester-WMF added a comment.Jul 18 2019, 4:38 PM

Gods, this repo desperately needs to be in the pipeline. This CI test is taking forever.

gerritbot added a comment.Jul 18 2019, 4:47 PM

Change 524271 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[integration/config@master] dockerfiles: Provide node10-kartotherian until it moves to the services pipeline

https://gerrit.wikimedia.org/r/524271

gerritbot added a project: Patch-For-Review.Jul 18 2019, 4:47 PM

Change 524272 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[integration/config@master] jjb: Provide kartotherian-node10-browser-docker based on custom image

https://gerrit.wikimedia.org/r/524272

gerritbot added a comment.Jul 18 2019, 4:47 PM

Change 524273 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[integration/config@master] layout: [wikimedia/services/kartotherian] Switch to custom job

https://gerrit.wikimedia.org/r/524273

MSantos added a comment.Jul 18 2019, 5:43 PM
In T228170#5346681, @Jdforrester-WMF wrote:

Gods, this repo desperately needs to be in the pipeline. This CI test is taking forever.

Building node-mapnik is the bottleneck, not sure how to make it better though (hosting the built binary somewhere?)

Jdforrester-WMF added a comment.Jul 18 2019, 5:47 PM

Pushing it as a pre-built binary into a .deb on our infrastructure would be a lot of effort and special-casing, yeah. :-(

gerritbot added a comment.Jul 18 2019, 5:49 PM

Change 524271 merged by jenkins-bot:
[integration/config@master] dockerfiles: Provide node10-kartotherian until it moves to the services pipeline

https://gerrit.wikimedia.org/r/524271

gerritbot added a comment.Jul 18 2019, 5:51 PM

Change 524272 merged by jenkins-bot:
[integration/config@master] jjb: Provide kartotherian-node10-browser-docker based on custom image

https://gerrit.wikimedia.org/r/524272

gerritbot added a comment.Jul 18 2019, 6:09 PM

Change 524273 merged by jenkins-bot:
[integration/config@master] layout: [wikimedia/services/kartotherian] Switch to custom job

https://gerrit.wikimedia.org/r/524273

Maintenance_bot removed a project: Patch-For-Review.Jul 18 2019, 6:10 PM
Stashbot added a comment.Jul 18 2019, 6:10 PM

Mentioned in SAL (#wikimedia-releng) [2019-07-18T18:10:51Z] <James_F> Zuul: [wikimedia/services/kartotherian] Switch to custom job T228170

MSantos claimed this task.Jul 18 2019, 6:44 PM
gerritbot added a comment.Jul 18 2019, 6:51 PM

Change 524304 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[integration/config@master] dockerfiles: [node10-kartotherian] Use SRE-backported custom components where needed

https://gerrit.wikimedia.org/r/524304

gerritbot added a project: Patch-For-Review.Jul 18 2019, 6:51 PM
gerritbot added a comment.Jul 18 2019, 7:17 PM

Change 524308 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[integration/config@master] jjb: [kartotherian-node10-browser-docker] Bump to latest custom image

https://gerrit.wikimedia.org/r/524308

gerritbot added a comment.Jul 18 2019, 7:17 PM

Change 524304 merged by jenkins-bot:
[integration/config@master] dockerfiles: [node10-kartotherian] Use SRE-backported custom components where needed

https://gerrit.wikimedia.org/r/524304

• Mholloway unsubscribed.Jul 18 2019, 7:18 PM
LGoto unsubscribed.Jul 18 2019, 7:18 PM
gerritbot added a comment.Jul 18 2019, 7:26 PM

Change 524308 merged by jenkins-bot:
[integration/config@master] jjb: [kartotherian-node10-browser-docker] Bump to latest custom image

https://gerrit.wikimedia.org/r/524308

Maintenance_bot removed a project: Patch-For-Review.Jul 18 2019, 8:10 PM
Jdforrester-WMF closed this task as Resolved.Jul 18 2019, 8:31 PM

https://gerrit.wikimedia.org/r/c/mediawiki/services/kartotherian/+/523743 landed.

Jdforrester-WMF updated the task description. (Show Details)Jul 18 2019, 8:31 PM
Jdforrester-WMF moved this task from Doing to Completed on the Release-Engineering-Team-TODO (201907) board.
MSantos added a comment.Jul 18 2019, 8:54 PM

@Jdforrester-WMF thank you so much!

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits