Page MenuHomePhabricator

๐ŸŸฃ Wire up and initialize Elasticsearch 7.10.2 cluster
Closed, ResolvedPublic

Event Timeline

Restricted Application added a subscriber: Aklapper. ยท View Herald TranscriptMay 3 2023, 5:14 PM
Tarrow updated the task description. (Show Details)

It appears that the image we custom build from an older bitnami starting image doesn't seem to be compatible with the latest charts. This means that the node roles aren't correctly assigned. We talked about this a little and drew up a similar options list to the one drawn up before:

Option 1 we stick with 6 now we can't update mediawiki / we're now running an unsupported version -F

Option 2 we keep using the latest bitnami chart and build an older image that is compatible with it (and licesned correctly)

Option 3 we go back to using a historic elasticco built chart, fork it for k8s comptibility and also continue building an older image that is compatible and open licensed - F

Option 4 we use the new bitnami chart and build our custom image off a new image and add the plugins to this (we enter some license minefield)

Option 5 OpenSearch???

Option 6 Revisit product requirements - try to find alternative that isn't wikibase cirrussearch

Deniz_WMDE renamed this task from Wire up and initialize Elasticsearch 7.10.2 cluster to ๐ŸŸ  Wire up and initialize Elasticsearch 7.10.2 cluster.Aug 21 2023, 9:10 AM
Deniz_WMDE renamed this task from ๐ŸŸ  Wire up and initialize Elasticsearch 7.10.2 cluster to ๐ŸŸฃ Wire up and initialize Elasticsearch 7.10.2 cluster.Aug 21 2023, 9:19 AM

Aaand... now that I'm off for a week something really needs to be added to this ticket so that there is a chance other people can make some headway in may absence given that there was no opportunity for some in person handover.

After looking at this I invested (probably too much) time in trying to build something analogous to option 2.

My strategy was to lightly tweak the Dockerfile used by the latest bitnami image to load an older (FOSS) version of the code. From this image I then derived an image in the same way we currently do with the repo at https://github.com/wbstack/elasticsearch.

I then attempted somewhat repeatedly to use this with the modern chart and hoped to end with a working cluster.

The lite tweak to the bitnami containers repo was as at: https://github.com/tarrow/containers (basically using the FOSS tarball extracted with the elasticsearch.yml removed) as the source but leaving all the bitnami special sauce present. This file must not be present or the special bitnami scripts do not generate the config but instead assume the baked in one is what you want.

I then spent much time debugging the libelasticsearch.sh script to try to determine if the reasons my cluster was falling over was bad config, transient, or other things.

n.b. removing the liveness probes is very useful to prevent pods getting killed before their time (or before you can inspect them). Adding something like:

coordinating:
  livenessProbe:
    enabled: false

The current hurdle I was stuck on (although probably not far from success) was preventing the generated config from including references to xpack this being a non-FOSS plugin that we don't provide or load (but causes it to bail early) even though all the settings there were to ensure these features (security and ml) were disabled.

Currently untested before I really needed to leave was the following addition to the values file:

extraConfig:
  # T309378
  action.auto_create_index: false
  xpack:

in the hope that leaving this xpack key empty would allow the nodes to boot. Maybe this is a place to start.

I was repeatedly considering and making half-hearted attempts at Option 3 / fork it in the simplest way and write my own custom minimalist chart that would be easy for us to maintain.

Courage mon brave

Tarrow removed Tarrow as the assignee of this task.Sep 8 2023, 8:48 PM
Tarrow added a subscriber: Tarrow.

https://github.com/wbstack/elasticsearch/pull/1 is an attempt to continue with option 2. Still needs a little work on the CI build process

Tarrow removed Tarrow as the assignee of this task.Oct 4 2023, 3:39 PM
Tarrow moved this task from Doing to In Review on the Wikibase Cloud (Kanban board Q4 2023) board.
Andrew-WMDE changed the task status from Open to Stalled.Oct 26 2023, 8:57 AM

Waiting for indexing jobs to finish

Evelien_WMDE claimed this task.