Patches:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Andrew-WMDE | T330998 🟪 Update ElasticSearch to 7.10.2 | |||
| Resolved | Tarrow | T335854 Configure MediaWiki to support two Elasticsearch clusters | |||
| Resolved | Evelien_WMDE | T335891 🟣 Wire up and initialize Elasticsearch 7.10.2 cluster | |||
| Resolved | Evelien_WMDE | T335893 🟣 Switch reads over to Elasticsearch 7.10.2 cluster | |||
| Resolved | Evelien_WMDE | T335894 🟣 Remove Elasticsearch 6.8.22 cluster |
Guide on how to smoke test these changes:
Checkout the following:
make minikube-delete make minikube-start make init-local make apply-local make minikube-tunnel
skaffold run -m mediawiki-139 skaffold run -m api
kubectl port-forward elasticsearch-master-0 9200:9200 kubectl port-forward elasticsearch-honey-master-0 9201:9200
kubectl --context minikube-wbaas exec -it sql-mariadb-primary-0 -- /bin/bash -c 'mysql -u root -p${MARIADB_ROOT_PASSWORD} -e "SELECT wikis.id, sitename, prefix, name FROM apidb.wikis LEFT JOIN apidb.wiki_dbs ON wikis.id = wiki_dbs.wiki_id;"'curl http://localhost:9200/_cat/indices?v
curl http://localhost:9200/_cat/indices?v
kubectl exec -it deployments/api-scheduler -- bash -c "php artisan job:dispatchNow CirrusSearch\\\\ElasticSearchIndexInit 1 default" kubectl exec -it deployments/api-scheduler -- bash -c "php artisan job:dispatchNow CirrusSearch\\\\QueueSearchIndexBatches 1 default"
kubectl exec -it deployments/api-scheduler -- bash -c "php artisan job:dispatchNow CirrusSearch\\\\ForceSearchIndex id 1 0 1000 default"
kubectl --context minikube-wbaas exec -it sql-mariadb-primary-0 -- /bin/bash -c 'mysql -u root -p${MARIADB_ROOT_PASSWORD} -e "SELECT wikis.id, sitename, prefix, name FROM apidb.wikis LEFT JOIN apidb.wiki_dbs ON wikis.id = wiki_dbs.wiki_id;"'curl http://localhost:9200/_cat/indices?v
kubectl exec -it deployments/api-scheduler -- bash -c "php artisan job:dispatchNow DeleteWikiDispatcherJob"
kubectl --context minikube-wbaas exec -it sql-mariadb-primary-0 -- /bin/bash -c 'mysql -u root -p${MARIADB_ROOT_PASSWORD} -e "SELECT wikis.id, sitename, prefix, name FROM apidb.wikis LEFT JOIN apidb.wiki_dbs ON wikis.id = wiki_dbs.wiki_id;"'curl http://localhost:9200/_cat/indices?v
Checkout the following:
Skaffold the following services again:
skaffold run -m mediawiki-139 skaffold run -m api
curl http://localhost:9201/_cat/indices?v
kubectl exec -it deployments/api-scheduler -- bash -c "php artisan job:dispatchNow CirrusSearch\\\\ElasticSearchIndexInit 1 secondary" kubectl exec -it deployments/api-scheduler -- bash -c "php artisan job:dispatchNow CirrusSearch\\\\QueueSearchIndexBatches 1 secondary"
kubectl exec -it deployments/api-scheduler -- bash -c "php artisan job:dispatchNow CirrusSearch\\\\ForceSearchIndex id 1 0 1000 secondary"
curl http://localhost:9200/_cat/indices?v curl http://localhost:9201/_cat/indices?v
Note: Try querying the index directly if the queries above aren't updating with:
curl http://localhost:9201/NAME_content_first/_search
kubectl --context minikube-wbaas exec -it sql-mariadb-primary-0 -- /bin/bash -c 'mysql -u root -p${MARIADB_ROOT_PASSWORD} -e "SELECT wikis.id, sitename, prefix, name FROM apidb.wikis LEFT JOIN apidb.wiki_dbs ON wikis.id = wiki_dbs.wiki_id;"'curl http://localhost:9200/_cat/indices?v curl http://localhost:9201/_cat/indices?v
curl http://localhost:9200/_cat/indices?v curl http://localhost:9201/_cat/indices?v
kubectl exec -it deployments/api-scheduler -- bash -c "php artisan job:dispatchNow DeleteWikiDispatcherJob"
kubectl --context minikube-wbaas exec -it sql-mariadb-primary-0 -- /bin/bash -c 'mysql -u root -p${MARIADB_ROOT_PASSWORD} -e "SELECT wikis.id, sitename, prefix, name FROM apidb.wikis LEFT JOIN apidb.wiki_dbs ON wikis.id = wiki_dbs.wiki_id;"'curl http://localhost:9200/_cat/indices?v curl http://localhost:9201/_cat/indices?v
There seems to still be a bug in C.2.
This step doesn't always exit cleanly on the first run but it does still create the indices:
kubectl exec -it deployments/api-scheduler -- bash -c "php artisan job:dispatchNow CirrusSearch\\\\ElasticSearchIndexInit 1 secondary"
Task was marked stalled both to represent reality: it had been hanging around since Mid may with no movement but also to focus on other work that was inflight at the same time (I think specifically T330389).
Marked it as open again
Right, unfortunately I'm now off for a few days and therefore am about to un-assign myself from reviewing this. Sorry that it doesn't feel like I've managed to make some great progress on it but there are certainly a couple of thoughts.
There are a couple of comments but I would also add some general ones here. While it does definitely make it harder to write and prepare the patches it would probably be easier to review if these were a little more broken up. Particularly if they were broken up in an order where it felt like merging some of the work was very low risk.
For example I wasn't sure if I could merge the api or MW changes without that then breaking ES functionality. While there would be no change in production to things if there was no corresponding chart or repo change merging some breaking change here would result in us then being unable to push and deploy new api patches.
The other big picture thought I had here was that having the platform api talk directly to ES is an inconvenience that we don't even benefit from that this point since we don't use the delete functionality. Without this it would be possible to encapsulate all the ES config to be set in MW and we wouldn't then need to worry about passing things like cluster names over the wire.
We had a look at the api and mediawiki patches and decided that the main thing that made the couple was changing the parameters of the api calls that triggers the elasticsearch jobs to run on mediawiki.
We wondered if instead it would be possible to remove these and have MediaWiki always write to all configured clusters when these apis are called. This would mean that all of the changes that are being made on both sides (api and mediawiki) would not need to happen at the same time (because they probably don't need to happen at all).
When, mid migration, we need to call specific clusters one at a time we could do this in another way (for example, following the "use a kubernetes job" pattern that we are generally aiming towards).
We'll look at adding on (or adjusting?) the mediawiki patch to see what this would look like.
Documenting some failed updates form dailies over the last few dasy: Marked stalled since we at least want a second cluster setup that is working for all devs before people move further on this. See: T335853
Marked open again now that there is a second cluster both on the local environment and in staging