Page MenuHomePhabricator

Create a mwdebug deployment for mediawiki on kubernetes
Closed, ResolvedPublic

Description

We need to create a deployment on k8s corresponding to our current mwdebug cluster, for functional testing of our mediawiki on kubernetes installation first, and to use as mwdebug service in the future.

  • Create namespace, user on all kubernetes service clusters
  • Create a mediawiki "mwdebug" deployment
  • Functional testing of the new mwdebug cluster on k8s
  • Add a mechanism to the traffic layer to point to this installation
  • Add a kubernetes option to the mwdebug browser extension
  • Add a new build variant to mediawiki-multiversion that installs php-tideways-xhprof
  • Filter MediaWiki logs from mediawiki on kubernetes to a separate dashboard.
  • Implement a simple mechanism for deployers to deploy to mediawiki on kubernetes when they deploy with scap

Related Objects

StatusSubtypeAssignedTask
ResolvedJoe
ResolvedJoe
ResolvedNone
Resolvedjijiki
Resolvedjijiki
Resolveddancy
Resolveddancy
ResolvedJoe
ResolvedJoe
Resolvedjeena
ResolvedJoe
ResolvedJoe
Resolveddancy
ResolvedJoe
Resolveddpifke
Resolveddancy
OpenJoe
Resolvedjijiki
Resolveddpifke
ResolvedLegoktm

Event Timeline

Joe triaged this task as High priority.May 18 2021, 7:46 AM
Joe created this task.

Change 692667 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] Add tokens and users for mwdebug service

https://gerrit.wikimedia.org/r/692667

Change 692672 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[labs/private@master] Add tokens and users for mwdebug

https://gerrit.wikimedia.org/r/692672

Change 692672 merged by Effie Mouzeli:

[labs/private@master] Add tokens for mwdebug service

https://gerrit.wikimedia.org/r/692672

Change 692667 merged by Effie Mouzeli:

[operations/puppet@production] Add tokens and users for mwdebug service

https://gerrit.wikimedia.org/r/692667

Change 693124 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] Add a namespace for mwdebug service

https://gerrit.wikimedia.org/r/693124

Change 693124 merged by jenkins-bot:

[operations/deployment-charts@master] Add a namespace for mwdebug service

https://gerrit.wikimedia.org/r/693124

Change 693169 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[labs/private@master] Add kubernetes mwdebug user

https://gerrit.wikimedia.org/r/693169

Change 693169 merged by Effie Mouzeli:

[labs/private@master] Add kubernetes mwdebug user

https://gerrit.wikimedia.org/r/693169

Change 693875 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] mwdebug: add helmfile configuration

https://gerrit.wikimedia.org/r/693875

Change 693875 merged by Effie Mouzeli:

[operations/deployment-charts@master] mwdebug: add helmfile configuration

https://gerrit.wikimedia.org/r/693875

After solving various problems with the deployment, the situation now is:

curl -H 'Host: en.wikipedia.org' http://10.64.75.196:8080/wiki/Main_Page
<br />
<b>Fatal error</b>:  Uncaught ConfigException: Failed to load configuration from etcd: (curl error: 60) SSL peer certificate or SSH remote key was not OK in /srv/mediawiki/php-1.37.0-wmf.4/includes/config/EtcdConfig.php:205

We definitely need to add the puppet CA at the very least to the php container. We also have the alternative of adding new discovery endpoints from envoy, but I think it's worth actually adding the certificate to the image inside /etc/ssl/certs/ca-certificates.crt which we can easily add via a configmap.

I've deployed the mwdebug deployment to all clusters; for now it's running pinned to kubernetes1017/kubernetes2017 via a nodeSelector constraint, so that new copies of the mediawiki image can be deployed by pre-pulling them with

$ sudo cumin 'kubernetes*017*' 'docker --config /var/lib/kubelet pull docker-registry.discovery.wmnet/restricted/mediawiki-multiversion:2021-07-01-232810-publish'

before running helmfile.

Change 704799 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] conftool-data: add mwdebug discovery

https://gerrit.wikimedia.org/r/704799

Change 704948 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/dns@master] Add entries for mwdebug service

https://gerrit.wikimedia.org/r/704948

Change 704964 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] service::catalog: add mwdebug service 2

https://gerrit.wikimedia.org/r/704964

Change 704948 merged by Effie Mouzeli:

[operations/dns@master] Add entries for mwdebug service

https://gerrit.wikimedia.org/r/704948

Change 705860 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/dns@master] Add discovery for mwdebug

https://gerrit.wikimedia.org/r/705860

Change 704799 merged by Effie Mouzeli:

[operations/puppet@production] conftool-data: add mwdebug discovery 1

https://gerrit.wikimedia.org/r/704799

Change 705860 abandoned by Effie Mouzeli:

[operations/dns@master] Add discovery for mwdebug

Reason:

nah

https://gerrit.wikimedia.org/r/705860

Change 705862 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/dns@master] Add discovery for mwdebug

https://gerrit.wikimedia.org/r/705862

Change 705862 merged by Effie Mouzeli:

[operations/dns@master] Add discovery for mwdebug

https://gerrit.wikimedia.org/r/705862

Change 704964 merged by Effie Mouzeli:

[operations/puppet@production] add mwdebug service to LVS 2

https://gerrit.wikimedia.org/r/704964

Change 706340 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] add mwdebug service to LVS 3

https://gerrit.wikimedia.org/r/706340

Change 706355 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] add mwdebug service to LVS 4

https://gerrit.wikimedia.org/r/706355

Change 706356 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] add mwdebug service to LVS 5

https://gerrit.wikimedia.org/r/706356

Change 706340 merged by Effie Mouzeli:

[operations/puppet@production] add mwdebug service to LVS 3

https://gerrit.wikimedia.org/r/706340

Change 706355 merged by Effie Mouzeli:

[operations/puppet@production] add mwdebug service to LVS 4

https://gerrit.wikimedia.org/r/706355

Change 706356 abandoned by Effie Mouzeli:

[operations/puppet@production] add mwdebug service to LVS 5

Reason:

rebase hell

https://gerrit.wikimedia.org/r/706356

Change 706502 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] add mwdebug service to LVS 5

https://gerrit.wikimedia.org/r/706502

Change 706502 merged by Effie Mouzeli:

[operations/puppet@production] add mwdebug service to LVS 5

https://gerrit.wikimedia.org/r/706502

Change 706506 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/dns@master] Add mwdebug discovery record

https://gerrit.wikimedia.org/r/706506

Change 706506 merged by Effie Mouzeli:

[operations/dns@master] Add mwdebug discovery record

https://gerrit.wikimedia.org/r/706506

Change 708248 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] profile::trafficserver: fix mwdebug on k8s support

https://gerrit.wikimedia.org/r/708248

Change 708248 merged by Giuseppe Lavagetto:

[operations/puppet@production] profile::trafficserver: fix mwdebug on k8s support

https://gerrit.wikimedia.org/r/708248

Change 708255 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/mediawiki-config@master] Add the experimental kubernetes backend to mwdebug

https://gerrit.wikimedia.org/r/708255

Change 708255 merged by jenkins-bot:

[operations/mediawiki-config@master] Add the experimental kubernetes backend to mwdebug

https://gerrit.wikimedia.org/r/708255

Mentioned in SAL (#wikimedia-operations) [2021-07-27T11:20:32Z] <oblivian@deploy1002> Synchronized debug.json: Config: [[gerrit:708255|Add the experimental kubernetes backend to mwdebug (T283056)]] (duration: 00m 56s)

Status update:

the mwdebug installation is now reachable from external users via the Wikimedia Debug browser extension. As of today, the following things don't work:

  • The release currently working on the mwdebug server is not the one that is currently deployed everywhere else. We need to make it simple for deployers to also deploy the code to mwdebug on k8s.
  • Profiling doesn't work because we need to create a new variant of the mediawiki image which includes tideways-xhprof
Joe updated the task description. (Show Details)

Change 709392 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] trafficserver::text: open mwdebug on k8s again

https://gerrit.wikimedia.org/r/709392

Change 709392 merged by Giuseppe Lavagetto:

[operations/puppet@production] trafficserver::text: open mwdebug on k8s again

https://gerrit.wikimedia.org/r/709392