Page MenuHomePhabricator

Split search.wikimedia.org out of ops/mediawiki-config into separate service
Open, Needs TriagePublic

Description

search.wikimedia.org is the gateway for Apple's dictionary search. The entire thing is ~100 lines of PHP: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/mediawiki-config/+/refs/heads/master/docroot/search.wikimedia.org/index.php and purely just a wrapper around a public API.

It has no tests, logging, stats, etc. AFAICT. There is no real reason to keep deploying it with MediaWiki like this.

I propose that search.wikimedia.org is turned into a PHP microservice deployed via k8s and separated from MediaWiki. Putting it in a separate repository would it make it easy to add PHPUnit tests and deployment pipeline. Deploying via k8s will give us logging and basic monitoring/stats via envoy.

I came across this via T288848: Make HTTP calls work within mediawiki on kubernetes , where the proposed solution would not work for search.wikimedia.org, creating some extra work for MW-on-K8s.


  • Review charts
  • apple-search namespaces in k8s
  • apple-search accounts in k8s.
  • apple-search puppet private tokens.
  • Generate TLS certificates
  • Review helmfile.d files:
  • LVS setup
  • DNS for LVS records
  • Discovery DNS
  • Monitoring dashboard
  • Integration and Acceptance tests

Event Timeline

+1 to moving this out of ops/mw-config. I'm happy to try to get this moving if the stakeholders (Search team I assume) are fine with it.

Gehel added a subscriber: Gehel.

The Search Platform team isn't responsible for this service, so from my point of view, feel free to do whatever needs doing!

The Search Platform team isn't responsible for this service, so from my point of view, feel free to do whatever needs doing!

This is not what I was expecting, but thanks for the comment. I think we'll go ahead with the split and migration to k8s and I'll file a separate Code-Stewardship-Reviews for it.

+1 to moving this out of ops/mw-config. I'm happy to try to get this moving if the stakeholders (Search team I assume) are fine with it.

Please go ahead :) Originally on IRC I had suggested a repository name of search/apple-gateway but that was assuming Search was responsible for it. I think we can use something like: mediawiki/services/apple-search (bonus points if you can use git filter-branch etc. to keep the VCS history!).

I think you/we should be able to copy most of the setup from Shellbox, so the architecture is envoy (TLS termination) -> httpd (apache) ->php-fpm -> 100 line PHP script.

In the 1/128 sampled web request logs, this service has 1.7k entries over 30 days, so theoretically ~217,600 reqs/month, or ~5reqs/min. For reference in 2013, we were getting 11 reqs/second (per T81982). It really should be fine with 2 replicas and low memory/CPU limits.

Legoktm renamed this task from Consider splitting search.wikimedia.org out of ops/mediawiki-config into separate service to Split search.wikimedia.org out of ops/mediawiki-config into separate service.Aug 23 2021, 6:23 PM

Change 715317 had a related patch set uploaded (by Majavah; author: Majavah):

[integration/config@master] Setup CI for mediawiki/services/apple-search

https://gerrit.wikimedia.org/r/715317

Change 715317 merged by jenkins-bot:

[integration/config@master] jjb, Zuul: [mediawiki/services/apple-search] Add pipeline CI

https://gerrit.wikimedia.org/r/715317

Mentioned in SAL (#wikimedia-releng) [2021-08-30T15:16:45Z] <James_F> Zuul: [mediawiki/services/apple-search] Add pipeline CI for T289224

https://integration.wikimedia.org/ci/job/apple-search-pipeline-publish/1/console:

'publishedImage':'docker-registry.wikimedia.org/wikimedia/mediawiki-services-apple-search:2021-08-30-191349-production'

Change 716049 had a related patch set uploaded (by Legoktm; author: Legoktm):

[labs/private@master] Add k8s users/tokens for apple-search

https://gerrit.wikimedia.org/r/716049

Change 716052 had a related patch set uploaded (by Legoktm; author: Legoktm):

[operations/puppet@production] Add k8s users/tokens for apple-search

https://gerrit.wikimedia.org/r/716052

Change 716049 merged by Legoktm:

[labs/private@master] Add k8s users/tokens for apple-search

https://gerrit.wikimedia.org/r/716049

Change 716052 merged by Legoktm:

[operations/puppet@production] Add k8s users/tokens for apple-search

https://gerrit.wikimedia.org/r/716052

Change 716056 had a related patch set uploaded (by Legoktm; author: Legoktm):

[operations/deployment-charts@master] admin: Add namespace for apple-search

https://gerrit.wikimedia.org/r/716056

Change 716056 merged by jenkins-bot:

[operations/deployment-charts@master] admin: Add namespace for apple-search

https://gerrit.wikimedia.org/r/716056