Page MenuHomePhabricator

Migrate changeprop to kubernetes
Closed, ResolvedPublic

Description

Subtask of the TEC3:O3:O3.1:Q3 Goal to migrate changeprop to use the deployment pipeline

This is done after all the below items have been checked off:

Details

ProjectBranchLines +/-Subject
operations/deployment-chartsmaster+219 -198
operations/deployment-chartsmaster+228 -197
operations/deployment-chartsmaster+1 -1
operations/deployment-chartsmaster+213 -194
operations/deployment-chartsmaster+8 -2
labs/privatemaster+13 -0
operations/deployment-chartsmaster+209 -190
operations/deployment-chartsmaster+3 -4
operations/deployment-chartsmaster+209 -187
operations/deployment-chartsmaster+205 -190
operations/deployment-chartsmaster+204 -185
operations/deployment-chartsmaster+4 -1
operations/deployment-chartsmaster+203 -184
operations/deployment-chartsmaster+202 -183
operations/deployment-chartsmaster+198 -179
operations/deployment-chartsmaster+11 -6
operations/deployment-chartsmaster+1 -0
operations/deployment-chartsmaster+5 -0
operations/deployment-chartsmaster+75 -0
operations/deployment-chartsmaster+184 -165
operations/deployment-chartsmaster+77 -28
operations/docker-images/production-imagesmaster+10 -0
operations/deployment-chartsmaster+182 -163
operations/deployment-chartsmaster+2 -2
operations/deployment-chartsmaster+182 -163
operations/deployment-chartsmaster+189 -161
operations/puppetproduction+14 -0
operations/deployment-chartsmaster+354 -0
labs/privatemaster+8 -0
operations/deployment-chartsmaster+2 -2
mediawiki/services/change-propagationmaster+25 -310
mediawiki/services/change-propagationmaster+25 -310
mediawiki/services/change-propagationmaster+25 -310
mediawiki/services/change-propagationmaster+25 -310
integration/configmaster+3 -3
Show related patches Customize query in gerrit

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 565137 abandoned by Ppchelko:
Changes for Changeprop and cpjobqueue to K8s migration

Reason:
Transferring commits from GH doesn't seem to work

https://gerrit.wikimedia.org/r/565137

Change 565130 merged by Ppchelko:
[mediawiki/services/change-propagation@master] Changes for Changeprop and cpjobqueue to K8s migration

https://gerrit.wikimedia.org/r/565130

Change 565144 had a related patch set uploaded (by Ppchelko; owner: Holger Knust):
[mediawiki/services/change-propagation@master] Changes for Changeprop and cpjobqueue to K8s migration

https://gerrit.wikimedia.org/r/565144

Change 565144 merged by jenkins-bot:
[mediawiki/services/change-propagation@master] Changes for Changeprop and cpjobqueue to K8s migration

https://gerrit.wikimedia.org/r/565144

After a bunch of fighting with corrupted change-prop repo on gerrit and with help from RelEng, we now have a working image built via the pipeline.

See https://gerrit.wikimedia.org/r/c/mediawiki/services/change-propagation/+/565150

Adding @hnowlan so that he is aware and perhaps maybe help move this along.

akosiaris updated the task description. (Show Details)Feb 11 2020, 2:48 PM

Updated the list of actions that have to be taken at the task description. Number #1 is done, we are doing now the helm chart. As soon as the chart is reviewed and is merged, the other 2 items are SRE deploys (and should be done fairly quickly). The define what a "safe" deploy is for changeprop is a pretty good question. I guess @Pchelolo might be able to help with that, there's already some discussion in T244387 about it. Last step is pretty obvious I think :-)

akosiaris updated the task description. (Show Details)Feb 20 2020, 9:21 AM
akosiaris updated the task description. (Show Details)Feb 20 2020, 4:16 PM

Change 574094 had a related patch set uploaded (by Holger Knust; owner: Holger Knust):
[operations/deployment-charts@master] WIP changeprop: New helmfiles for deployment

https://gerrit.wikimedia.org/r/574094

Change 574713 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[labs/private@master] Showcase redis pass population for changeprop

https://gerrit.wikimedia.org/r/574713

Change 574719 had a related patch set uploaded (by Hnowlan; owner: Hnowlan):
[operations/deployment-charts@master] Admin: Add changeprop namespace

https://gerrit.wikimedia.org/r/574719

Change 574806 had a related patch set uploaded (by Hnowlan; owner: Hnowlan):
[labs/private@master] hierdata: Add stub values for changeprop

https://gerrit.wikimedia.org/r/574806

Change 574811 had a related patch set uploaded (by Hnowlan; owner: Hnowlan):
[operations/puppet@production] changeprop: add hierdata k8s entries and LVS entry

https://gerrit.wikimedia.org/r/574811

Change 574806 merged by Alexandros Kosiaris:
[labs/private@master] hierdata: Add stub values for changeprop

https://gerrit.wikimedia.org/r/574806

I've committed tokens on the private repo. Stubs have been merged to labs-private.

Change 574719 merged by Alexandros Kosiaris:
[operations/deployment-charts@master] Admin: Add changeprop namespace

https://gerrit.wikimedia.org/r/574719

Change 574811 merged by Hnowlan:
[operations/puppet@production] changeprop: add hierdata k8s entries

https://gerrit.wikimedia.org/r/574811

hnowlan updated the task description. (Show Details)Feb 28 2020, 12:24 PM

Change 576335 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/deployment-charts@master] changeprop: Package 0.9.5

https://gerrit.wikimedia.org/r/576335

Change 576335 merged by jenkins-bot:
[operations/deployment-charts@master] changeprop: Package 0.9.5

https://gerrit.wikimedia.org/r/576335

Change 576344 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/deployment-charts@master] changeprop: Correctly align the prometheus-statsd.conf call

https://gerrit.wikimedia.org/r/576344

Change 576344 merged by jenkins-bot:
[operations/deployment-charts@master] changeprop: Correctly align the prometheus-statsd.conf call

https://gerrit.wikimedia.org/r/576344

Change 576377 had a related patch set uploaded (by Hnowlan; owner: Hnowlan):
[operations/deployment-charts@master] changeprop: Bump CPU usage

https://gerrit.wikimedia.org/r/576377

Change 576377 merged by jenkins-bot:
[operations/deployment-charts@master] changeprop: Bump CPU usage

https://gerrit.wikimedia.org/r/576377

Change 576400 had a related patch set uploaded (by Hnowlan; owner: Hnowlan):
[operations/deployment-charts@master] changeprop: Create new changeprop release

https://gerrit.wikimedia.org/r/576400

Change 576400 merged by jenkins-bot:
[operations/deployment-charts@master] changeprop: Create new changeprop release

https://gerrit.wikimedia.org/r/576400

hnowlan updated the task description. (Show Details)Mar 3 2020, 5:53 PM

changeprop now deploys to staging and creates pods successfully. Errors out with this, as expected. Nutcracker sidecar will remedy this.

{"name":"change-propagation","hostname":"changeprop-staging-7bfbb7798c-2xx5j","pid":17,"level":50,"err":{"message":"","name":"changeprop-staging","stack":"Error: Redis connection to localhost:6379 failed - connect ENOENT localhost:6379\n    at PipeConnectWrap.afterConnect [as oncomplete] (net.js:1104:14)","code":"ENOENT","errno":"ENOENT","syscall":"connect","address":"localhost:6379","levelPath":"error/redis"},"msg":"Redis connection to localhost:6379 failed - connect ENOENT localhost:6379","time":"2020-03-03T17:53:31.387Z","v":0}

Change 576827 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/deployment-charts@master] changeprop: Add nutcracker sidecar

https://gerrit.wikimedia.org/r/576827

Change 576829 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/docker-images/production-images@master] nutcracker: Add entrypoint and user directives

https://gerrit.wikimedia.org/r/576829

Change 576829 merged by Alexandros Kosiaris:
[operations/docker-images/production-images@master] nutcracker: Add entrypoint and user directives

https://gerrit.wikimedia.org/r/576829

Change 577239 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/deployment-charts@master] admin: Add redis databases for changeprop

https://gerrit.wikimedia.org/r/577239

Change 577307 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/deployment-charts@master] changeprop: Release 0.9.8

https://gerrit.wikimedia.org/r/577307

Change 576827 merged by jenkins-bot:
[operations/deployment-charts@master] changeprop: Add nutcracker sidecar

https://gerrit.wikimedia.org/r/576827

Change 577307 merged by jenkins-bot:
[operations/deployment-charts@master] changeprop: Release 0.9.8

https://gerrit.wikimedia.org/r/577307

Change 577239 merged by jenkins-bot:
[operations/deployment-charts@master] admin: Add redis databases for changeprop

https://gerrit.wikimedia.org/r/577239

Change 578526 had a related patch set uploaded (by Hnowlan; owner: Hnowlan):
[operations/deployment-charts@master] changeprop: configure redis servers for staging.

https://gerrit.wikimedia.org/r/578526

Change 578526 merged by jenkins-bot:
[operations/deployment-charts@master] changeprop: configure redis servers for staging.

https://gerrit.wikimedia.org/r/578526

Change 578567 had a related patch set uploaded (by Hnowlan; owner: Hnowlan):
[operations/deployment-charts@master] changeprop: enable nutcracker

https://gerrit.wikimedia.org/r/578567

Change 578567 merged by jenkins-bot:
[operations/deployment-charts@master] changeprop: enable nutcracker

https://gerrit.wikimedia.org/r/578567

Change 578946 had a related patch set uploaded (by Hnowlan; owner: Hnowlan):
[operations/deployment-charts@master] changeprop: Change redis connect string to use host/port rather than file path

https://gerrit.wikimedia.org/r/578946

Change 578946 merged by jenkins-bot:
[operations/deployment-charts@master] changeprop: Change redis connect string to use host/port rather than file path

https://gerrit.wikimedia.org/r/578946

Change 578964 had a related patch set uploaded (by Hnowlan; owner: Hnowlan):
[operations/deployment-charts@master] changeprop: New release

https://gerrit.wikimedia.org/r/578964

Change 578964 merged by jenkins-bot:
[operations/deployment-charts@master] changeprop: New release

https://gerrit.wikimedia.org/r/578964

Change 579016 had a related patch set uploaded (by Hnowlan; owner: Hnowlan):
[operations/deployment-charts@master] changeprop: Nutcracker sidecar fixes

https://gerrit.wikimedia.org/r/579016

Change 579240 had a related patch set uploaded (by Hnowlan; owner: Hnowlan):
[operations/deployment-charts@master] changeprop: Release new version of changeprop

https://gerrit.wikimedia.org/r/579240

Change 579257 had a related patch set uploaded (by Hnowlan; owner: Hnowlan):
[operations/deployment-charts@master] changeprop: Release new version of changeprop

https://gerrit.wikimedia.org/r/579257

Change 579240 abandoned by Hnowlan:
changeprop: Release new version of changeprop

https://gerrit.wikimedia.org/r/579240

Change 579257 merged by jenkins-bot:
[operations/deployment-charts@master] changeprop: Release new version of changeprop

https://gerrit.wikimedia.org/r/579257

Change 579016 merged by jenkins-bot:
[operations/deployment-charts@master] changeprop: Nutcracker sidecar fixes

https://gerrit.wikimedia.org/r/579016

Change 579270 had a related patch set uploaded (by Hnowlan; owner: Hnowlan):
[operations/deployment-charts@master] changeprop: Release version with fixes for nutcracker behaviour.

https://gerrit.wikimedia.org/r/579270

Change 579270 merged by jenkins-bot:
[operations/deployment-charts@master] changeprop: Release version with fixes for nutcracker behaviour.

https://gerrit.wikimedia.org/r/579270

Change 579339 had a related patch set uploaded (by Hnowlan; owner: Hnowlan):
[operations/deployment-charts@master] changeprop: Remove HTTP liveness probe

https://gerrit.wikimedia.org/r/579339

Change 579339 merged by jenkins-bot:
[operations/deployment-charts@master] changeprop: Remove HTTP liveness probe

https://gerrit.wikimedia.org/r/579339

Change 579352 had a related patch set uploaded (by Hnowlan; owner: Hnowlan):
[operations/deployment-charts@master] changeprop: use tcpsocket for readiness and liveness

https://gerrit.wikimedia.org/r/579352

Change 579352 merged by jenkins-bot:
[operations/deployment-charts@master] changeprop: use tcpsocket for readiness and liveness

https://gerrit.wikimedia.org/r/579352

Change 579458 had a related patch set uploaded (by Holger Knust; owner: Holger Knust):
[operations/deployment-charts@master] changeprop: Add readiness and liveness check delay

https://gerrit.wikimedia.org/r/579458

Change 579458 merged by jenkins-bot:
[operations/deployment-charts@master] changeprop: Add readiness and liveness check delay

https://gerrit.wikimedia.org/r/579458

Change 579604 had a related patch set uploaded (by Hnowlan; owner: Hnowlan):
[operations/deployment-charts@master] changeprop: new release

https://gerrit.wikimedia.org/r/579604

Change 579604 merged by jenkins-bot:
[operations/deployment-charts@master] changeprop: new release

https://gerrit.wikimedia.org/r/579604

Change 574713 merged by Alexandros Kosiaris:
[labs/private@master] Showcase redis pass population for changeprop

https://gerrit.wikimedia.org/r/574713

Change 581667 had a related patch set uploaded (by Hnowlan; owner: Hnowlan):
[operations/deployment-charts@master] calico: add changeprop access to varnish multicast address

https://gerrit.wikimedia.org/r/581667

Change 581667 merged by jenkins-bot:
[operations/deployment-charts@master] calico: add changeprop access to varnish multicast address

https://gerrit.wikimedia.org/r/581667

Change 581997 had a related patch set uploaded (by Hnowlan; owner: Hnowlan):
[operations/deployment-charts@master] changeprop: Release new version

https://gerrit.wikimedia.org/r/581997

Change 581997 merged by jenkins-bot:
[operations/deployment-charts@master] changeprop: Release new version

https://gerrit.wikimedia.org/r/581997

Change 582018 had a related patch set uploaded (by Hnowlan; owner: Hnowlan):
[operations/deployment-charts@master] calico: Fix rdb1009 IP address

https://gerrit.wikimedia.org/r/582018

Change 582018 merged by jenkins-bot:
[operations/deployment-charts@master] calico: Fix rdb1009 IP address

https://gerrit.wikimedia.org/r/582018

Change 583056 had a related patch set uploaded (by Hnowlan; owner: Hnowlan):
[operations/deployment-charts@master] changeprop: enable nutcracker in production, reenable rate limiting

https://gerrit.wikimedia.org/r/583056

Change 583056 merged by jenkins-bot:
[operations/deployment-charts@master] changeprop: enable nutcracker in production, reenable rate limiting

https://gerrit.wikimedia.org/r/583056

Change 583111 had a related patch set uploaded (by Hnowlan; owner: Hnowlan):
[operations/deployment-charts@master] changeprop: Move prometheus query to use regex

https://gerrit.wikimedia.org/r/583111

Change 583111 merged by jenkins-bot:
[operations/deployment-charts@master] changeprop: Move prometheus query to use regex

https://gerrit.wikimedia.org/r/583111

holger.knust closed this task as Resolved.Mar 26 2020, 6:45 PM
holger.knust updated the task description. (Show Details)

Are you planning to file a new ticket for migrating all the rules to k8s change-prop? The work here is not done yet.

Yes, Hugh Nolan will be creating another ticket.