Page MenuHomePhabricator
Create Task
Maniphest T183281

[epic] ELK upgrade to 6.x (elasticsearch, kibana, logstash)
Closed, ResolvedPublic
Actions

Description

ElasticSearch 6.0 was released in November 2017 and that means we'll need to plan out the upgrade — for the search cluster and the logging clusters.

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedEBernhardsonT183281 [epic] ELK upgrade to 6.x (elasticsearch, kibana, logstash)
 ResolvedNoneT183282 [epic] Search cluster upgrade to 6.x
 ResolvedEBernhardsonT192615 Add 'type' field to store same information as was in es5 types.
 DuplicateNoneT192616 Update saneitizer to reindex documents that havn't been indexed in N days
 ResolvedEBernhardsonT192680 Update ttmserver for elasticsearch 6
 DeclinedNoneT192681 Identify and update all query building to filter on the new 'type' field
 DeclinedNoneT192693 Resolve document id clashes with unified type
 ResolvedEBernhardsonT192699 Build a unified namespace index
 ResolvedEBernhardsonT192972 Evaluate impact of adding ~2700 new shards to production cluster
 ResolvedEBernhardsonT215969 Measure mutation latency across the newly split elasticsearch clusters
 Resolved• debtT193654 [epic] Run multiple elasticsearch clusters on same hardware
 ResolvedEBernhardsonT194678 Update OtherIndex to operate on a cluster other than the one holding the wiki
 ResolvedGehelT198351 Refactor puppet to support multiple elasticsearch instances on same node
 ResolvedEBernhardsonT198490 Use kafka for communication from analytics cluster to elasticsearch
 ResolvedEBernhardsonT200215 Create kafka topic for mjolinr bulk daemon and decide on cluster
 ResolvedEBernhardsonT200740 Deploy mjolnir msearch daemon to the elasticsearch clusters
 ResolvedEBernhardsonT201948 Add stats collection for observability of mjolnir daemons
 ResolvedGehelT198352 Setup two elasticsearch clusters on relforge to test multi-instance
 ResolvedGehelT207195 Configure LVS endpoints for new elasticsearch clusters
 Resolved• dcausseT210381 Update mw-config to use the psi&omega elastic clusters
 Resolved• dcausseT211752 Adapt or configure mjolnir so that it knows all search clusters within a DC
 Resolved• Mathew.onipeT212434 Allow elasticsearch machines to communicate with each others on port 9500 and 9700
 Resolved• dcausseT213150 Configure elasticsearch crosscluster on production search servers
 ResolvedEBernhardsonT213959 Decide order of operations for elastic 6 upgrade
 Resolved• dcausseT214052 Delete indices moved from chi to psi/omega
 ResolvedLegoktmT190024 Elastica 5.3 uses the each keyword which is deprecated in PHP 7.2
ResolvedNoneT194199 [Epic] Prepare for Elasticsearch 6 upgrade
 ResolvedEBernhardsonT192614 Resolve current deprecation warnings in elasticsearch 5
 ResolvedTJonesT194849 Investigate language analyzers in ElasticSearch 6
 ResolvedTJonesT214439 Review Manually re-built Hebmorph plugin
 Resolved• dcausseT197960 Upgrade plugins to elastic 6.3
 Resolved• dcausseT198066 Upgrade the LTR plugin to elastic 6.3
 Resolved• dcausseT198067 Upgrade the extra plugin to elastic 6.3
 Resolved• dcausseT198068 Upgrade the extra-analysis plugin to elastic 6.3
 Resolved• dcausseT198069 Upgrade the experimental highlighter to elastic 6.3
 Resolved• dcausseT199791 Build wmf-elasticsearch-search-plugins.deb that supports elastic 6.5.4
 Resolved• dcausseT206468 Upgrade the elasticsearch plugin ltr to elasticsearch 6.4.2
 Resolved• dcausseT206469 Upgrade the elasticsearch plugin extra to elasticsearch 6.4.2
 Resolved• dcausseT206470 Upgrade the elasticsearch plugin extra-analysis to elasticsearch 6.4.2
 Resolved• dcausseT206472 Upgrade the elasticsearch plugin experimental highlighter to elasticsearch 6.4.2
 Resolved• dcausseT214312 Upgrade the elasticsearch plugins extra, extra-analysis and highlighter to elasticsearch 6.5.4
Resolved• dcausseT200198 Drop the "archive" type in the general index
 ResolvedEBernhardsonT213851 Drop "archive" from generic index on testwiki
 Resolved• dcausseT209348 Port the elasticsearch plugin extra-analysis-surrogates to 6.4.2 as a noop plugin
 DuplicateEBernhardsonT215345 Verify ttmserver continues to operate in elastic 6
 ResolvedEBernhardsonT215475 Ensure mjolnir daemons work seamlessly with elasticsearch 5 or 6
 Resolved• dcausseT215621 Make sure that ApiFeatureUsage still works with elasticsearch 6.5.4
 Declined• mmodellT181393 Make sure elasticsearch 6 is supported in phabricator
 Resolved• mmodellT230787 Make search context highlights work with the ferret search engine
 ResolvedGehelT218113 Upload elastic 6.5.4 packages to reprepro
 Resolved• dcausseT198734 Rework how source_regex timeout is done in Cirrus
 Resolved• dcausseT199256 Setup a cirrussearch branch for es 6.x and make it work
 ResolvedhasharT200669 Help needed to setup the elastic6 dev branch on CirrusSearch, Elastica and mediawiki-vendor
 ResolvedGehelT215916 ElasticSearch 6 migration plan checklist (search cluster)
 Resolved• debtT215931 Upgrade elasticsearch to 5.6.14
 Resolved• dcausseT215932 Prepare a debian package with search plugins compatible with elastic 5.6.14
 ResolvedGehelT216047 Create new elastic56 component in reprepro and upload elasticsearch and plugins
 ResolvedGehelT216052 upgrade logstash and the logstash elasticsearch cluster to 5.6.14
 Resolved• Mathew.onipeT216993 Upgrade logstash plugins to 5.6.14
 Resolved• debtT217196 Adapt elasticsearch puppet module for elasticsearch6
 Resolved• debtT218116 Upgrade relforge to elasticsearch 6.5.4
 Resolved• debtT218247 Upgrade deployment-elastic* to 6.5.4
 ResolvedGehelT218878 Upgrade to elasticsearch 6.5.4 for cirrus / codfw
 ResolvedGehelT218879 Upgrade to elasticsearch 6.5.4 for cirrus / eqiad
 DeclinedNoneT183283 [epic] Logging cluster upgrade to 6.x
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits