These queries look to be running as expected now.

To get an idea of what we need to optimize i ran an experiment. This experiment stands up a fresh elasticsearch instance, creates 100 indexes with the same settings, and restarts the instance every 10 indexes. I measure how long the instance takes to come up and how long indices take to create. Ran this experiment with 4 different index configurations:

reindex complete, looks to have resolved the issue as expected.

Search backend error during entity_full_text search for 'test' after 35: Parse error on Cannot search on field [labels.en] since it is not indexed.

has been deployed for a month without issues, lets hope this is resolved.

The most notable difference in metrics I see is in the disk utilization per host of the cluster overview dashboard. During the backfilling period all the other codfw hosts are reporting a max per-disk value of around 10%. For 2002 half the disks were at 25% and the other half at 45-50%.

In T336076#8842456, @colewhite wrote:

Flink logs are ECS-formatted and can be found under the ecs-* index pattern. Query: orchestrator.namespace:"rdf-streaming-updater" On April 18th, we allowed ECS-formatted logs < 1.7.0 to enter later ECS versioned indexes: T292585.

In T334194#8833365, @Gehel wrote:

If this is about Wikibase.cloud, how much should we de-duplicate vs reduce the numbers of supported languages? My intuition is that on Wikibase.cloud, the number of languages per instance is unlikely to be as high as on Wikibase. Would that also be an option? Would that help?

In T334194#8827123, @Lydia_Pintscher wrote:

Is there anywhere where I can briefly read up on what analyzers, token filters and char filters are in this context? Then I can probably help.

As part of migrating our integration testing out of vagrant I put together an environment based off of mwcli / mwdd that runs enough of CirrusSearch to pass the integration tests: https://gitlab.wikimedia.org/repos/search-platform/cirrus-integration-test-runner/

Released a new version of the plugin to maven central, 7.10.2-wmf8. Once the debian packaging is done and available we can start updating the places that use it.

We can use an elasticsearch query to find the oldest dated completion indices. This query will give us the 5 titlesuggest indices with the oldest batch_id (~= indexing timestamp) when issued against the :9243 cluster:

Poking at the logs for the script that builds the daily autocomplete indices, we may be missing errors that happen there. The logs show that the enwiki completion index failed its daily build from dec 9 2022 through jan 20 2023. This was not identified by any of our monitoring, we should correct that so these errors bubble up sooner and get fixed immediately.

To check that titles are making it into the primary search index i ran a quick python script (P47281) and ran it for the last 7 days worth of new pages according to the recent changes api. It found 12572 pages that were created and should exist in the enwiki search index. Of these 12 were not found in the search index. A manual check shows them all to be redirects to redirects which we don't index. This looks to be generally working, although there could certainly be edge cases that are not handled.

I tried to clarify the section of docs about updates to include the distinction between full-text and title completion search indexes: https://www.mediawiki.org/w/index.php?title=Help:CirrusSearch&diff=prev&oldid=5897819

Took a quick pass at the docs on wikitech as well: https://wikitech.wikimedia.org/wiki/Cindy_The_Browser_Test_Bot

Updated the dependencies on the cirrus side from node 10 / wdio v5 to node 14 / wdio v7. This brings us a few years into the future. Didn't try the node 16/ wdio v8. One issue we will soon run into is that this test suite was written with synchronous wdio, but thats been discontinued starting in nodejs v16.

We should measure the gain. We already have a component that can deduplicate (AnalysisFilter) but we should test if this has any useful effect. Based on the change in index creation time we can decided if it should move forward.

Progress so far:

re-index is currently running, mappings look as expected for indices that have completed reindexing already. It's up to r, will probably take a few more days and we can ship the keyword in next weeks train with any luck.

@TJones can you take a look agian now? I see a number of results now for a korean search for 가마우지 but I'm not familiar with what it was returning before and can't be certain this is correct.

We talked a bit about this, the plan right now is to prevent the library we use from disabling the last available connection. That should allow the retries to work as we expect regardless of the error type. It seems this connection disabling is more suited for cases that have many instances in their pool, rather than a single DNS backed by LVS.

needs https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/304 to properly pass templated values to the submitted skein spec and let this run

@bking has delt with these issues before, might have ideas

In T331580#8683625, @nfraison wrote:

Command to change right is runnning

To prevent that we can remove write for the analytics-search-users group:

From:
drwxrwxr-x   - analytics-search analytics-search-users               0 2023-03-10 01:06 hdfs://analytics-hadoop/wmf/data/discovery

To:
drwxr-xr-x   - analytics-search analytics-search-users               0 2023-03-10 01:06 hdfs://analytics-hadoop/wmf/data/discovery

I would just need some confirmation from someone having more history knowledge on if it is ot not expected to have users belonging to analytics-search-users group writing on that folder

confirmed the instance seems to be working, remaining updates are to be made in the data-engineering airflow-dags repo

In T327970#8671765, @Ottomata wrote:

Re https://gerrit.wikimedia.org/r/894740, we should ask @mforns @Milimetric @JAllemandou about this. I think there might be a better way? Maybe the logic to get MW db hostnames and ports should be moved out of refinery python? Or, wmfdata-python uses the refinery bin/analytics-mysql CLI. Perhaps the whole thing should move out of refinery so it is installable without deploying refinery?

Took a closer look for the wikidata failure, but i've turned up nothing. With the output ending without any failure messages it suggests to me that the process died, perhaps a force kill or a segfault. I couldn't find anything in the system logs that correlate. Sadly syslog doesn't go back that far (oldest syslog entry is feb 27, this died on feb 22) but theres no certainty it would have had useful information.