Page MenuHomePhabricator

Changeprop config management in beta cluster
Closed, ResolvedPublic

Description

After we migrate production change-prop to k8s, we would be left with a working deployment in beta cluster. For the rest of the services, it seems like beta deploys were replaced with a simple docker container runner. This would be enough for change-prop, but the problem would be that change-prop config is extremely complicated, so duplicating the config for beta cluster would not be a satisfying experience.

Instead, we should experiment with reusing the k8s config in beta. Perhaps, it could involve some manual steps, but the important part is to have the source of truth for the config shared. How would it work:

  1. We would have special 'beta' values.yaml
  2. We would do helm template to render the config. It will render much more then just the config itself for us, be it would be easy enough to trim the output to get the config.
  3. We would deploy the rendered config on beta note automatically.

This could be done via some updater script calling to k8s api from the beta node.

Event Timeline

Change 593559 had a related patch set uploaded (by Hnowlan; owner: Hnowlan):
[operations/deployment-charts@master] changeprop: allow additional configuration of metrics

https://gerrit.wikimedia.org/r/593559

Change 593559 merged by jenkins-bot:
[operations/deployment-charts@master] changeprop: allow additional configuration of metrics

https://gerrit.wikimedia.org/r/593559

Change 593561 had a related patch set uploaded (by Hnowlan; owner: Hnowlan):
[mediawiki/services/change-propagation/deploy@master] Add tool and configuration for generating beta configuration from kubernetes

https://gerrit.wikimedia.org/r/593561

Change 593561 abandoned by Hnowlan:
Add tool and configuration for generating beta configuration from kubernetes

Reason:
Moving to deployment-charts

https://gerrit.wikimedia.org/r/593561

Change 596209 had a related patch set uploaded (by Hnowlan; owner: Hnowlan):
[operations/deployment-charts@master] Add tool and configuration for generating beta configuration from kubernetes

https://gerrit.wikimedia.org/r/596209

Change 596686 had a related patch set uploaded (by Hnowlan; owner: Hnowlan):
[mediawiki/services/change-propagation/deploy@master] changeprop: only process purges outside of Docker in beta

https://gerrit.wikimedia.org/r/596686

Change 596686 merged by Hnowlan:
[mediawiki/services/change-propagation/deploy@master] changeprop: only process purges outside of Docker in beta

https://gerrit.wikimedia.org/r/596686

Change 596209 merged by jenkins-bot:
[operations/deployment-charts@master] Add tool and configuration for generating beta configuration from kubernetes

https://gerrit.wikimedia.org/r/596209

Change 598026 had a related patch set uploaded (by Hnowlan; owner: Hnowlan):
[operations/deployment-charts@master] changeprop: simplify config writing. make beta config write puppet-friendly YAML.

https://gerrit.wikimedia.org/r/598026

This work is currently blocked on us getting our configuration into beta somehow. As it stands we can't post the full configuration to Horizon for a few reasons:

  • The default config with anchors cannot be safely loaded by puppet. Trying to write the configuration out will cause puppet to barf because it's not a safe loading of YAML
  • I've made some changes in https://gerrit.wikimedia.org/r/596209 to write "unrolled" YAML - with the whole blocks inserted wherever they would just be referenced as an anchor. However, this significantly increases the size of the generated YAML, unsurprisingly (100KB!)
  • Horizon rejects data submissions larger than 64KB. You can see our configuration being truncated at 64KB exactly here However, our configuration is also submitted to git successfully, but unfortunately this is *not* the source of truth used by hiera.

Not sure how to proceed on this one - we could put the configuration into puppet somehow (deployment-prep will allow you to include cherry picked reviews via this tag) but I would sincerely like to avoid putting _anything_ changeprop-specific into puppet for docker so that we can purge any and all changeprop configuration from puppet.

Change 598753 had a related patch set uploaded (by Hnowlan; owner: Hnowlan):
[operations/puppet@production] deployment-docker-changeprop01: override docker configuration

https://gerrit.wikimedia.org/r/598753

Change 601717 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] beta: Allow using docker volumes

https://gerrit.wikimedia.org/r/601717

Change 598026 merged by jenkins-bot:
[operations/deployment-charts@master] changeprop: simplify config writing. make beta config write puppet-friendly YAML.

https://gerrit.wikimedia.org/r/598026

Change 601717 merged by Hnowlan:
[operations/puppet@production] beta: Allow using docker volumes

https://gerrit.wikimedia.org/r/601717

Change 598753 abandoned by Hnowlan:
deployment-docker-changeprop01: override docker configuration

Reason:
Fixed by using a local Docker volume via puppet

https://gerrit.wikimedia.org/r/c/operations/puppet/ /598753