Page MenuHomePhabricator
Create Task
Maniphest T283056

Create a mwdebug deployment for mediawiki on kubernetes
Closed, ResolvedPublic
Actions

Description

We need to create a deployment on k8s corresponding to our current mwdebug cluster, for functional testing of our mediawiki on kubernetes installation first, and to use as mwdebug service in the future.

  • Create namespace, user on all kubernetes service clusters
  • Create a mediawiki "mwdebug" deployment
  • Functional testing of the new mwdebug cluster on k8s
  • Add a mechanism to the traffic layer to point to this installation
  • Add a kubernetes option to the mwdebug browser extension
  • Add a new build variant to mediawiki-multiversion that installs php-tideways-xhprof
  • Filter MediaWiki logs from mediawiki on kubernetes to a separate dashboard.
  • Implement a simple mechanism for deployers to deploy to mediawiki on kubernetes when they deploy with scap

Details

Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 StalledNoneT255792 Quibble runs core:unit tests twice!
 OpenNoneT328919 Upgrade to PHPUnit 10
 OpenNoneT338103 Micro-optimize ApiResult::isMetadataKey with str_starts_with once we support PHP8+
 OpenNoneT328921 Drop PHP 7.4 support from MediaWiki
 StalledNoneT334726 Use return type `never` in Wikibase
 OpenNoneT328922 Drop PHP 8.0 support from MediaWiki
 StalledNoneT319055 Upgrade to psr/container 2.x
 StalledKrinkleT319432 Migrate WMF production from PHP 7.4 to PHP 8.1
 OpenNoneT291916 Tracking task for Bullseye migrations in production
 StalledNoneT356293 Migrate MW appservers' base images to bullseye
 OpenNoneT290536 Serve production traffic via Kubernetes
 ResolvedJoeT283056 Create a mwdebug deployment for mediawiki on kubernetes
 ResolvedJoeT284417 Add the puppet CA to the MediaWiki deployment
 ResolvedNoneT284418 Add conditional to mediawiki-config for stuff running on kubernetes
 ResolvedjijikiT284420 Add all redis and memcached backends to mw on k8s automatically
 ResolvedjijikiT284421 Enable TLS termination on the mwdebug deployment. fix the service definition in the chart
 ResolveddancyT284581 Ownership of the /tmp/mw-cache directories should be www-data in the mediawiki-multiversion image
 ResolveddancyT284582 The restricted/mediawiki-multiversion image should include the production version of private/PrivateSettings.php
 ResolvedJoeT285298 Make all httpbb tests pass on the mwdebug deployment.
 ResolvedJoeT285309 Install wiki-specific php extensions in the mediawiki production image
 ResolvedjeenaT285325 Check out www-portals repo in the mediawiki-webserver and in the mediawiki-multiversion images
 ResolvedJoeT285232 The restricted/mediawiki-webserver image should include skins and resources
 ResolvedJoeT286491 Access mwdebug kubernetes deployment via the 'X-Wikimedia-Debug' header
 ResolveddancyT287512 GitInfo is missing from mwdebug-kubernetes deployment
 ResolvedJoeT287570 Ensure the code is deployed to mediawiki on k8s when it is deployed to production
 Resolved• dpifkeT288164 Ensure WikimediaDebug "log" and "profile" features work with k8s-mwdebug
 ResolveddancyT287495 Create a variant of mediawiki-multiversion which installs php-tideways-xhprof
 ResolvedJoeT288851 Make logging work for mediawiki in k8s
 ResolvedClement_GoubertT326794 Ingest php-slowlog in logstash
 ResolvedcolewhiteT328318 Index orchestrator object fields from ECS 1.11.0 in OpenSearch
 ResolvedjijikiT290485 mediawiki-debug image does not produce profiling info
Resolved• dpifkeT288165 Create separate ArcLamp pipeline for k8s-mwdebug
 ResolvedLegoktmT288848 Make HTTP calls work within mediawiki on kubernetes

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Joe triaged this task as High priority.May 18 2021, 7:46 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 18 2021, 7:46 AM
jijiki updated the task description. (Show Details)May 18 2021, 5:21 PM
gerritbot added a comment.May 18 2021, 5:30 PM

Change 692667 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] Add tokens and users for mwdebug service

https://gerrit.wikimedia.org/r/692667

gerritbot added a project: Patch-For-Review.May 18 2021, 5:30 PM
gerritbot added a comment.May 18 2021, 5:48 PM

Change 692672 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[labs/private@master] Add tokens and users for mwdebug

https://gerrit.wikimedia.org/r/692672

gerritbot added a comment.May 20 2021, 6:56 AM

Change 692672 merged by Effie Mouzeli:

[labs/private@master] Add tokens for mwdebug service

https://gerrit.wikimedia.org/r/692672

gerritbot added a comment.May 20 2021, 6:57 AM

Change 692667 merged by Effie Mouzeli:

[operations/puppet@production] Add tokens and users for mwdebug service

https://gerrit.wikimedia.org/r/692667

jijiki mentioned this in rLPRI4c4b84f0929d: Add tokens for mwdebug service.May 20 2021, 6:57 AM
Maintenance_bot removed a project: Patch-For-Review.May 20 2021, 7:10 AM
gerritbot added a comment.May 20 2021, 7:33 AM

Change 693124 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] Add a namespace for mwdebug service

https://gerrit.wikimedia.org/r/693124

gerritbot added a project: Patch-For-Review.May 20 2021, 7:33 AM
gerritbot added a comment.May 20 2021, 7:38 AM

Change 693124 merged by jenkins-bot:

[operations/deployment-charts@master] Add a namespace for mwdebug service

https://gerrit.wikimedia.org/r/693124

Maintenance_bot removed a project: Patch-For-Review.May 20 2021, 8:10 AM
Joe updated the task description. (Show Details)May 20 2021, 12:41 PM
gerritbot added a comment.May 20 2021, 2:39 PM

Change 693169 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[labs/private@master] Add kubernetes mwdebug user

https://gerrit.wikimedia.org/r/693169

gerritbot added a project: Patch-For-Review.May 20 2021, 2:39 PM
gerritbot added a comment.May 20 2021, 4:11 PM

Change 693169 merged by Effie Mouzeli:

[labs/private@master] Add kubernetes mwdebug user

https://gerrit.wikimedia.org/r/693169

jijiki updated the task description. (Show Details)May 20 2021, 4:12 PM
jijiki mentioned this in rLPRI9a0c30a3f2e6: Add kubernetes mwdebug user.May 20 2021, 4:13 PM
Maintenance_bot removed a project: Patch-For-Review.May 20 2021, 5:10 PM
gerritbot added a comment.Jun 1 2021, 10:10 AM

Change 693875 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] mwdebug: add helmfile configuration

https://gerrit.wikimedia.org/r/693875

gerritbot added a project: Patch-For-Review.Jun 1 2021, 10:10 AM
gerritbot added a comment.Jun 1 2021, 10:24 AM

Change 693875 merged by Effie Mouzeli:

[operations/deployment-charts@master] mwdebug: add helmfile configuration

https://gerrit.wikimedia.org/r/693875

Maintenance_bot removed a project: Patch-For-Review.Jun 1 2021, 11:10 AM
Joe moved this task from Backlog to In Progress on the MW-on-K8s board.Jun 4 2021, 8:43 AM
Joe updated the task description. (Show Details)Jun 5 2021, 2:48 PM
Joe added a comment.Jun 5 2021, 2:51 PM

After solving various problems with the deployment, the situation now is:

curl -H 'Host: en.wikipedia.org' http://10.64.75.196:8080/wiki/Main_Page
<br />
<b>Fatal error</b>:  Uncaught ConfigException: Failed to load configuration from etcd: (curl error: 60) SSL peer certificate or SSH remote key was not OK in /srv/mediawiki/php-1.37.0-wmf.4/includes/config/EtcdConfig.php:205

We definitely need to add the puppet CA at the very least to the php container. We also have the alternative of adding new discovery endpoints from envoy, but I think it's worth actually adding the certificate to the image inside /etc/ssl/certs/ca-certificates.crt which we can easily add via a configmap.

dancy closed subtask T284581: Ownership of the /tmp/mw-cache directories should be www-data in the mediawiki-multiversion image as Resolved.Jun 8 2021, 6:24 PM
dancy closed subtask T284582: The restricted/mediawiki-multiversion image should include the production version of private/PrivateSettings.php as Resolved.Jun 8 2021, 7:35 PM
jijiki added a project: User-jijiki.Jun 24 2021, 3:49 PM
Joe closed subtask T284417: Add the puppet CA to the MediaWiki deployment as Resolved.Jun 25 2021, 3:07 PM
Joe closed subtask T284421: Enable TLS termination on the mwdebug deployment. fix the service definition in the chart as Resolved.Jun 28 2021, 9:46 AM
jijiki closed subtask T284420: Add all redis and memcached backends to mw on k8s automatically as Resolved.Jun 28 2021, 9:50 AM
Joe added a comment.Jul 8 2021, 2:25 PM

I've deployed the mwdebug deployment to all clusters; for now it's running pinned to kubernetes1017/kubernetes2017 via a nodeSelector constraint, so that new copies of the mediawiki image can be deployed by pre-pulling them with

$ sudo cumin 'kubernetes*017*' 'docker --config /var/lib/kubelet pull docker-registry.discovery.wmnet/restricted/mediawiki-multiversion:2021-07-01-232810-publish'

before running helmfile.

jijiki added a subtask: T286491: Access mwdebug kubernetes deployment via the 'X-Wikimedia-Debug' header.Jul 12 2021, 4:48 PM
jijiki moved this task from Inbox 🐅 to In Progress 🏋️‍♀️ on the User-jijiki board.Jul 12 2021, 4:52 PM
gerritbot added a comment.Jul 15 2021, 2:44 PM

Change 704799 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] conftool-data: add mwdebug discovery

https://gerrit.wikimedia.org/r/704799

gerritbot added a project: Patch-For-Review.Jul 15 2021, 2:44 PM
gerritbot added a comment.Jul 16 2021, 11:59 AM

Change 704948 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/dns@master] Add entries for mwdebug service

https://gerrit.wikimedia.org/r/704948

gerritbot added a comment.Jul 16 2021, 1:11 PM

Change 704964 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] service::catalog: add mwdebug service 2

https://gerrit.wikimedia.org/r/704964

gerritbot added a comment.Jul 21 2021, 11:04 AM

Change 704948 merged by Effie Mouzeli:

[operations/dns@master] Add entries for mwdebug service

https://gerrit.wikimedia.org/r/704948

gerritbot added a comment.Jul 21 2021, 11:16 AM

Change 705860 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/dns@master] Add discovery for mwdebug

https://gerrit.wikimedia.org/r/705860

gerritbot added a comment.Jul 21 2021, 11:21 AM

Change 704799 merged by Effie Mouzeli:

[operations/puppet@production] conftool-data: add mwdebug discovery 1

https://gerrit.wikimedia.org/r/704799

gerritbot added a comment.Jul 21 2021, 11:30 AM

Change 705860 abandoned by Effie Mouzeli:

[operations/dns@master] Add discovery for mwdebug

Reason:

nah

https://gerrit.wikimedia.org/r/705860

gerritbot added a comment.Jul 21 2021, 11:35 AM

Change 705862 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/dns@master] Add discovery for mwdebug

https://gerrit.wikimedia.org/r/705862

gerritbot added a comment.Jul 21 2021, 11:41 AM

Change 705862 merged by Effie Mouzeli:

[operations/dns@master] Add discovery for mwdebug

https://gerrit.wikimedia.org/r/705862

gerritbot added a comment.Jul 21 2021, 1:00 PM

Change 704964 merged by Effie Mouzeli:

[operations/puppet@production] add mwdebug service to LVS 2

https://gerrit.wikimedia.org/r/704964

gerritbot added a comment.Jul 22 2021, 9:58 AM

Change 706340 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] add mwdebug service to LVS 3

https://gerrit.wikimedia.org/r/706340

gerritbot added a comment.Jul 22 2021, 10:38 AM

Change 706355 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] add mwdebug service to LVS 4

https://gerrit.wikimedia.org/r/706355

gerritbot added a comment.Jul 22 2021, 10:38 AM

Change 706356 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] add mwdebug service to LVS 5

https://gerrit.wikimedia.org/r/706356

gerritbot added a comment.Jul 22 2021, 10:39 AM

Change 706340 merged by Effie Mouzeli:

[operations/puppet@production] add mwdebug service to LVS 3

https://gerrit.wikimedia.org/r/706340

gerritbot added a comment.Jul 22 2021, 2:21 PM

Change 706355 merged by Effie Mouzeli:

[operations/puppet@production] add mwdebug service to LVS 4

https://gerrit.wikimedia.org/r/706355

gerritbot added a comment.Jul 22 2021, 2:35 PM

Change 706356 abandoned by Effie Mouzeli:

[operations/puppet@production] add mwdebug service to LVS 5

Reason:

rebase hell

https://gerrit.wikimedia.org/r/706356

gerritbot added a comment.Jul 22 2021, 2:39 PM

Change 706502 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] add mwdebug service to LVS 5

https://gerrit.wikimedia.org/r/706502

gerritbot added a comment.Jul 22 2021, 2:45 PM

Change 706502 merged by Effie Mouzeli:

[operations/puppet@production] add mwdebug service to LVS 5

https://gerrit.wikimedia.org/r/706502

gerritbot added a comment.Jul 22 2021, 3:00 PM

Change 706506 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/dns@master] Add mwdebug discovery record

https://gerrit.wikimedia.org/r/706506

gerritbot added a comment.Jul 22 2021, 3:25 PM

Change 706506 merged by Effie Mouzeli:

[operations/dns@master] Add mwdebug discovery record

https://gerrit.wikimedia.org/r/706506

gerritbot added a comment.Jul 27 2021, 8:35 AM

Change 708248 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] profile::trafficserver: fix mwdebug on k8s support

https://gerrit.wikimedia.org/r/708248

gerritbot added a comment.Jul 27 2021, 9:09 AM

Change 708248 merged by Giuseppe Lavagetto:

[operations/puppet@production] profile::trafficserver: fix mwdebug on k8s support

https://gerrit.wikimedia.org/r/708248

Joe updated the task description. (Show Details)Jul 27 2021, 9:23 AM
gerritbot added a comment.Jul 27 2021, 9:24 AM

Change 708255 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/mediawiki-config@master] Add the experimental kubernetes backend to mwdebug

https://gerrit.wikimedia.org/r/708255

gerritbot added a comment.Jul 27 2021, 11:13 AM

Change 708255 merged by jenkins-bot:

[operations/mediawiki-config@master] Add the experimental kubernetes backend to mwdebug

https://gerrit.wikimedia.org/r/708255

Stashbot added a comment.Jul 27 2021, 11:20 AM

Mentioned in SAL (#wikimedia-operations) [2021-07-27T11:20:32Z] <oblivian@deploy1002> Synchronized debug.json: Config: [[gerrit:708255|Add the experimental kubernetes backend to mwdebug (T283056)]] (duration: 00m 56s)

Joe claimed this task.Jul 27 2021, 3:54 PM

Status update:

the mwdebug installation is now reachable from external users via the Wikimedia Debug browser extension. As of today, the following things don't work:

  • The release currently working on the mwdebug server is not the one that is currently deployed everywhere else. We need to make it simple for deployers to also deploy the code to mwdebug on k8s.
  • Profiling doesn't work because we need to create a new variant of the mediawiki image which includes tideways-xhprof
Joe updated the task description. (Show Details)Jul 27 2021, 4:10 PM
Joe updated the task description. (Show Details)
dancy subscribed.Jul 27 2021, 4:14 PM
Joe closed subtask T286491: Access mwdebug kubernetes deployment via the 'X-Wikimedia-Debug' header as Resolved.Jul 27 2021, 4:14 PM
• dpifke subscribed.Jul 27 2021, 4:51 PM
ArielGlenn subscribed.Jul 28 2021, 9:52 AM
Tarrow subscribed.Jul 28 2021, 9:58 AM
• ema subscribed.Aug 2 2021, 8:55 AM
gerritbot added a comment.Aug 2 2021, 10:11 AM

Change 709392 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] trafficserver::text: open mwdebug on k8s again

https://gerrit.wikimedia.org/r/709392

gerritbot added a comment.Aug 3 2021, 12:45 PM

Change 709392 merged by Giuseppe Lavagetto:

[operations/puppet@production] trafficserver::text: open mwdebug on k8s again

https://gerrit.wikimedia.org/r/709392

dancy closed subtask T287512: GitInfo is missing from mwdebug-kubernetes deployment as Resolved.Aug 4 2021, 5:53 PM
dancy closed subtask T287495: Create a variant of mediawiki-multiversion which installs php-tideways-xhprof as Resolved.Aug 4 2021, 6:47 PM
Krinkle added a subtask: T285309: Install wiki-specific php extensions in the mediawiki production image.Aug 10 2021, 8:19 PM
dancy closed subtask T284418: Add conditional to mediawiki-config for stuff running on kubernetes as Resolved.Aug 11 2021, 3:40 PM
thcipriani closed subtask T287570: Ensure the code is deployed to mediawiki on k8s when it is deployed to production as Resolved.Aug 23 2021, 4:28 PM
• dpifke closed subtask T288165: Create separate ArcLamp pipeline for k8s-mwdebug as Resolved.Sep 9 2021, 10:24 PM
jijiki updated the task description. (Show Details)Sep 20 2021, 5:25 AM
Joe closed this task as Resolved.Nov 16 2021, 2:59 PM
Legoktm closed subtask T288848: Make HTTP calls work within mediawiki on kubernetes as Resolved.Dec 23 2021, 6:56 PM
Joe closed subtask T285298: Make all httpbb tests pass on the mwdebug deployment. as Resolved.Jan 19 2022, 5:10 PM
• dpifke closed subtask T288164: Ensure WikimediaDebug "log" and "profile" features work with k8s-mwdebug as Resolved.Feb 22 2022, 9:41 PM
Krinkle merged a task: T308763: Performance review of MediaWiki on Kubernetes for Test2Wiki.May 24 2022, 11:11 PM
Krinkle added subscribers: thcipriani, Krinkle, Peter and 2 others.
Krinkle mentioned this in T308763: Performance review of MediaWiki on Kubernetes for Test2Wiki.May 25 2022, 12:25 AM
Clement_Goubert closed subtask T288851: Make logging work for mediawiki in k8s as Resolved.Apr 3 2023, 10:20 AM
Krinkle added a parent task: T290536: Serve production traffic via Kubernetes.Aug 2 2023, 10:19 PM
Maintenance_bot removed a project: Patch-For-Review.Aug 2 2023, 10:30 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL