For now we will be using persistent configuration in the cluster settings.
I created a small python script on mwmaint1002:~/home/dcausse/es_multi_instance/push_cross_cluster_conf.py and can be used like this:
python push_cross_cluster_conf.py https://search.svc.codfw.wmnet:9443/_cluster/settings --ccc psi=psi_codfw_masters.lst chi=chi_codfw_masters.lst
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| Add scripts/push_cross_cluster_conf.py | mediawiki/extensions/CirrusSearch | master | +59 -0 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | EBernhardson | T183281 [epic] ELK upgrade to 6.x (elasticsearch, kibana, logstash) | |||
| Resolved | None | T183282 [epic] Search cluster upgrade to 6.x | |||
| Resolved | debt | T193654 [epic] Run multiple elasticsearch clusters on same hardware | |||
| Resolved | dcausse | T213150 Configure elasticsearch crosscluster on production search servers |
Change 482770 had a related patch set uploaded (by DCausse; owner: DCausse):
[mediawiki/extensions/CirrusSearch@master] Add scripts/push_cross_cluster_conf.py
Mentioned in SAL (#wikimedia-operations) [2019-01-08T10:34:16Z] <dcausse> elastic@eqiad setting crosscluster conf on production search cluster (T213150)
curl -s https://search.svc.codfw.wmnet:9443/_cluster/settings | jq .persistent.search.remote
{ "chi": { "seeds": [ "elastic2042.codfw.wmnet:9300", "elastic2025.codfw.wmnet:9300", "elastic2031.codfw.wmnet:9300" ] }, "psi": { "seeds": [ "elastic2027.codfw.wmnet:9700", "elastic2029.codfw.wmnet:9700", "elastic2048.codfw.wmnet:9700" ] } }
curl -s https://search.svc.codfw.wmnet:9643/_cluster/settings | jq .persistent.search.remote
{ "chi": { "seeds": [ "elastic2042.codfw.wmnet:9300", "elastic2025.codfw.wmnet:9300", "elastic2031.codfw.wmnet:9300" ] }, "omega": { "seeds": [ "elastic2042.codfw.wmnet:9500", "elastic2047.codfw.wmnet:9500", "elastic2038.codfw.wmnet:9500" ] } }
curl -s https://search.svc.codfw.wmnet:9243/_cluster/settings | jq .persistent.search.remote
{ "omega": { "seeds": [ "elastic2042.codfw.wmnet:9500", "elastic2047.codfw.wmnet:9500", "elastic2038.codfw.wmnet:9500" ] }, "psi": { "seeds": [ "elastic2027.codfw.wmnet:9700", "elastic2029.codfw.wmnet:9700", "elastic2048.codfw.wmnet:9700" ] } }
curl -s https://search.svc.eqiad.wmnet:9443/_cluster/settings | jq .persistent.search.remote
{ "chi": { "seeds": [ "elastic1036.eqiad.wmnet:9300", "elastic1030.eqiad.wmnet:9300", "elastic1040.eqiad.wmnet:9300" ] }, "psi": { "seeds": [ "elastic1052.eqiad.wmnet:9700", "elastic1048.eqiad.wmnet:9700", "elastic1050.eqiad.wmnet:9700" ] } }
curl -s https://search.svc.eqiad.wmnet:9643/_cluster/settings | jq .persistent.search.remote
{ "chi": { "seeds": [ "elastic1036.eqiad.wmnet:9300", "elastic1030.eqiad.wmnet:9300", "elastic1040.eqiad.wmnet:9300" ] }, "omega": { "seeds": [ "elastic1034.eqiad.wmnet:9500", "elastic1040.eqiad.wmnet:9500", "elastic1038.eqiad.wmnet:9500" ] } }
curl -s https://search.svc.eqiad.wmnet:9243/_cluster/settings | jq .persistent.search.remote
{ "omega": { "seeds": [ "elastic1034.eqiad.wmnet:9500", "elastic1040.eqiad.wmnet:9500", "elastic1038.eqiad.wmnet:9500" ] }, "psi": { "seeds": [ "elastic1052.eqiad.wmnet:9700", "elastic1048.eqiad.wmnet:9700", "elastic1050.eqiad.wmnet:9700" ] } }
Change 482770 merged by jenkins-bot:
[mediawiki/extensions/CirrusSearch@master] Add scripts/push_cross_cluster_conf.py
Note we ran this same procedure in the future (well, future relative to this ticket) to set the cross cluster replication properly after changing the masters. x-linking for posterity: https://phabricator.wikimedia.org/T294805#7701855