Change 510456 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[maps/kartotherian/package@master] add .pipeline/blubber.yaml

https://gerrit.wikimedia.org/r/510456

To run the pipeline, we need a CI configuration change in integration/config.git which would look like:

diff --git a/zuul/layout.yaml b/zuul/layout.yaml
index 44c8e843..a14f9368 100644
--- a/zuul/layout.yaml
+++ b/zuul/layout.yaml
@@ -1522,6 +1522,11 @@ projects:
     experimental:
       - kartotherian-deploy-npm-node-6-docker
 
+  - name: maps/kartotherian/package
+    template:
+      - name: service-pipeline-test
+      - name: service-pipeline-test-and-publish
+
   - name: maps/tilerator
     test:
       - noop

It seems developers are willing to use maps/kartotherian/package.git it is unclear to me what that repository is about. It seems to fetch the code from npmjs.org??? It seems to me the pipeline file should instead by in the source repo maps/kartotherian, and that would take care of running tests and ultimately publish the Docker image.

Anyway, I lack most of the context anyway, but @dduvall and @thcipriani would be able to provide guidance

@hashar and @Mathew.onipe, kartotherian is considered a third-party and is supposed to have its own community. The packaging adds it and possible specific code to attend wmf needs

If we need the CI to read the source repo, we might need to have a specific WMF fork of kartotherian/tilerator under WMF, we do have a task for that (T177008) conflicted with the will of kartotherian being a separate project with its own community.

Change 510635 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[integration/config@master] enable maps/kartotherian/package in CI

https://gerrit.wikimedia.org/r/510635

Change 510635 merged by jenkins-bot:
[integration/config@master] enable maps/kartotherian/package in CI

https://gerrit.wikimedia.org/r/510635

I have deployed the change so that the blubber/pipeline magic is now enabled on the repository maps/kartotherian/package.

I can't tell though whether it is suitable for production deployment later on. For most services we just use the actual source repository (eg mediawiki/services/mathoid) and I would rather have Kartotherian to follow the same model for sake of simplicity/coherency between the various MediaWiki services we have. But maybe there are some unique use cases that would prevent it.

Anyway, you should at least be able to trigger the blubber/pipeline and progress from there. If the workflow is not suiting production needs, we can just adjust it later on :]

Sorry to come to this late, but I don't think it makes sense to Kubernetize the package repo(s). Their only purpose is to provide a layer of indirection for including Wikimedia-specific dependencies when building deploy repo commits; see T162241 (and particularly T162241#3711873) for background. (In particular, they were introduced to handle dependencies for the Wikimedia map style update work that was ultimately postponed indefinitely.) It certainly wouldn't make sense to deploy a package repo to production.

IMO this is a good opportunity to retire the rather cumbersome package repo setup and streamline maps services deployment. Is there anything we're doing now that couldn't be done in a blubberfile in the main repo?

Here's what I think the potentially tricky/unusual bits are:

• The main Kartotherian and Tilerator repos were recently merged along with various Kartotherian dependency modules into a monorepo managed with Lerna. They live in the packages/ directory of the overall Kartotherian monorepo. Is there a way to specify (1) an alternate Blubberfile location and (2) include multiple Blubberfiles for multiple services living in the same code repo?

• We'll need to ensure that mapnik is built from source (i.e., that --build-from-source=@kartotherian/mapnik is included in the Blubberfile's npm install step).

In T223275#5187300, @Mholloway wrote:

• We'll need to ensure that mapnik is built from source (i.e., that --build-from-source=@kartotherian/mapnik is included in the Blubberfile's npm install step).

Actually, looks like the npm build step definition just points at package.json, so this shouldn't be difficult either.

We can use the main repo instead of using the package repo. It will align better with other docker related project at WMF.

I don't think mono repo works well with our deployment pipeline. Have more than one .pipeline/blubber.yaml will most likely bring more troubles and that's if its possible

I believe there's a way we can make the monorepo work with a single pipeline. Since both tilerator and kartotherian share a lot modules together, we can include a flag in helm chart to run or not run tilerator. This is ugly, I know. They should run in different containers.

To memorialize our convo in standup:

• We're best off handling these services from Gerrit-hosted, per-service repos (i.e., one each for Kartotherian and Tilerator) in order to work with service pipeline convention rather than against it
• What those should look like is TBD:
  • separate forks of the monorepo, with Tilerator stuff removed for the Kartotherian repo (and vice versa)?
  • identical forks of the monorepo, but with documentation for each identifying which service it corresponds to, and configuration in Docker and/or Helm to ignore the other service?
  • copies of the kartotherian and tilerator subfolders of the monorepo's packages/ subdirectory, manually synced as needed?
  • something else?
• Note that maps/kartotherian and maps/tilerator repos currently exist in Gerrit, but are badly out of date, having not been updated since the project was moved to GitHub 2+ years ago

Looking forward to discussing further with @MSantos and @Gehel next week. @Mathew.onipe Feel free to correct or expand on my notes above. Thanks!

Change 523743 had a related patch set uploaded (by MSantos; owner: MSantos):
[mediawiki/services/kartotherian@master] Prepare packages for CI

https://gerrit.wikimedia.org/r/523743

In T223275#5187350, @Mathew.onipe wrote:

We can use the main repo instead of using the package repo. It will align better with other docker related project at WMF.

I don't think mono repo works well with our deployment pipeline. Have more than one .pipeline/blubber.yaml will most likely bring more troubles and that's if its possible

It seems to be fixed now, see T210267#4874631.

The current strategy is to have one single repo and configuring the pipeline for kartotherian and tilerator for it.

The main reason is to keep the development environment easier to use and create consistency across all the packages we need to maintain.

Change 523743 merged by jenkins-bot:
[mediawiki/services/kartotherian@master] Prepare packages for CI

https://gerrit.wikimedia.org/r/523743

@MSantos Thank you!

Change 530330 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[integration/config@master] maps: enable pipeline build for kartotherian.

https://gerrit.wikimedia.org/r/530330

Change 530330 merged by jenkins-bot:
[integration/config@master] maps: enable pipeline build for kartotherian.

https://gerrit.wikimedia.org/r/530330

Mentioned in SAL (#wikimedia-releng) [2019-08-15T09:05:46Z] <James_F> Zuul: Kartotherian pipeline enablement T223275

Is https://gerrit.wikimedia.org/r/c/maps/kartotherian/package/+/510456 meant to be in mediawiki/services/kartotherian instead?

In T223275#5415510, @Jdforrester-WMF wrote:

Is https://gerrit.wikimedia.org/r/c/maps/kartotherian/package/+/510456 meant to be in mediawiki/services/kartotherian instead?

Yes, cc/ @Mathew.onipe

Change 530389 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[mediawiki/services/kartotherian@master] add .pipeline/blubber.yaml

https://gerrit.wikimedia.org/r/530389

Change 530390 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[mediawiki/services/kartotherian@master] add .pipeline/blubber.yaml

https://gerrit.wikimedia.org/r/530390

Change 530389 abandoned by Jforrester:
add .pipeline/blubber.yaml

Reason:
Too confusing.

https://gerrit.wikimedia.org/r/530389

Change 530836 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[mediawiki/services/kartotherian@master] (WIP) prepare pipeline for deployment

https://gerrit.wikimedia.org/r/530836

Change 510456 abandoned by Jforrester:
add .pipeline/blubber.yaml

Reason:
We're moving away from this repo.

https://gerrit.wikimedia.org/r/510456

Change 531134 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[integration/config@master] layout: [kartotherian] Disable pipeline on gate as it doesn't pass

https://gerrit.wikimedia.org/r/531134

Change 531134 merged by jenkins-bot:
[integration/config@master] layout: [kartotherian] Disable pipeline on gate as it doesn't pass

https://gerrit.wikimedia.org/r/531134

Mentioned in SAL (#wikimedia-releng) [2019-08-20T09:39:08Z] <James_F> Zuul: [kartotherian] Disable pipeline on gate as it doesn't pass T223275

Change 534033 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[integration/config@master] layout: [kartotherian] enable pipeline on gate

https://gerrit.wikimedia.org/r/534033

Change 530836 merged by jenkins-bot:
[mediawiki/services/kartotherian@master] Prepare pipeline for deployment

https://gerrit.wikimedia.org/r/530836

Change 534178 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[integration/config@master] layout: [mediawiki/services/kartotherian] Enable the publish pipeline now this works

https://gerrit.wikimedia.org/r/534178

Change 534178 merged by jenkins-bot:
[integration/config@master] layout: [mediawiki/services/kartotherian] Enable the publish pipeline now this works

https://gerrit.wikimedia.org/r/534178

Mentioned in SAL (#wikimedia-releng) [2019-09-03T15:57:25Z] <James_F> Zuul: Enable the publish pipeline for mediawiki/services/kartotherian T223275

I believe that this is now Done?

Change 534033 abandoned by Jforrester:
layout: [kartotherian] enable pipeline on gate

Reason:
Oh, oops, just done in I216ba3a00.

https://gerrit.wikimedia.org/r/534033

Change 530390 abandoned by Mathew.onipe:
add .pipeline/blubber.yaml

https://gerrit.wikimedia.org/r/530390