Page MenuHomePhabricator

EBernhardson (EBernhardson)
User

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Friday

  • Clear sailing ahead.

User Details

User Since
Oct 7 2014, 4:49 PM (426 w, 1 d)
Availability
Available
LDAP User
EBernhardson
MediaWiki User
EBernhardson (WMF) [ Global Accounts ]

Recent Activity

Today

EBernhardson moved T324247: Incorrect stats returning from 7.10.2 ltr plugin for non-matching terms from Ready for Dev -- SWE to Needs review on the Discovery-Search (Current work) board.
Wed, Dec 7, 6:24 PM · Patch-For-Review, Discovery-Search (Current work), CirrusSearch
EBernhardson moved T323628: Optimize the WikibaseCirrusSearch elasticsearch mapping and filter query for non-english users from In Progress to Needs review on the Discovery-Search (Current work) board.
Wed, Dec 7, 6:24 PM · Patch-For-Review, Discovery-Search (Current work), CirrusSearch

Yesterday

EBernhardson added a comment to T324247: Incorrect stats returning from 7.10.2 ltr plugin for non-matching terms.

Plugin patch: https://github.com/ebernhardson/elasticsearch-learning-to-rank/commit/c9a59cb840f872d29263a02275cedae16ba43aa4
Based on upstream change (which mixed solutions to several related problems into one patch): https://github.com/o19s/elasticsearch-learning-to-rank/pull/380

Tue, Dec 6, 10:22 PM · Patch-For-Review, Discovery-Search (Current work), CirrusSearch
EBernhardson created T324525: The module 'wikibase.lexeme.lexemeview' must not have target 'mobile' because its dependency 'jquery.ui.languagesuggester' does not have it.
Tue, Dec 6, 12:05 AM · Wikidata-Campsite, MediaWiki-ResourceLoader, Performance-Team, ci-test-error (WMF-deployed Build Failure), Wikidata, Wikidata Lexicographical data
EBernhardson committed rWDAN1d3ba411524a: import_cirrus: Update doc cleaning to match cirrus updates (authored by EBernhardson).
import_cirrus: Update doc cleaning to match cirrus updates
Tue, Dec 6, 12:04 AM
EBernhardson committed rWDAN57d62fee158d: Add bin/ directory to flake8 (authored by EBernhardson).
Add bin/ directory to flake8
Tue, Dec 6, 12:04 AM

Mon, Dec 5

EBernhardson added a comment to T323628: Optimize the WikibaseCirrusSearch elasticsearch mapping and filter query for non-english users.

Thanks for checking! I'm tempted to decline, the purpose of ticket was to fix a missed opportunity and I think that adding yet another 500+ indexed fields is not something we could do without evaluating the impact and there might be other things (less costly) we could do to improve recall in languages other than English (e.g. use the icu tokenizer for the labels_all.plain field?).

Pinging @TJones for advises on this matter.

Mon, Dec 5, 6:12 PM · Patch-For-Review, Discovery-Search (Current work), CirrusSearch
EBernhardson moved T317023: Investigate moving incoming_links computation to a batch job from To Be Deployed to Needs Reporting on the Discovery-Search (Current work) board.
Mon, Dec 5, 4:07 PM · MW-1.40-notes (1.40.0-wmf.12; 2022-11-28), Patch-For-Review, Discovery-Search (Current work), CirrusSearch

Fri, Dec 2

EBernhardson added a comment to T323628: Optimize the WikibaseCirrusSearch elasticsearch mapping and filter query for non-english users.

As of today the mapping for the labels.ko field is:
...
This does index a field name labels.ko using the elasticsearch default text analyzer.

Fri, Dec 2, 8:49 PM · Patch-For-Review, Discovery-Search (Current work), CirrusSearch
EBernhardson claimed T323628: Optimize the WikibaseCirrusSearch elasticsearch mapping and filter query for non-english users.
Fri, Dec 2, 6:39 PM · Patch-For-Review, Discovery-Search (Current work), CirrusSearch

Thu, Dec 1

EBernhardson moved T322327: CirrusSearch should generate a document consistent to a given schema from In Progress to Needs review on the Discovery-Search (Current work) board.
Thu, Dec 1, 9:51 PM · Patch-For-Review, MW-1.40-notes (1.40.0-wmf.12; 2022-11-28), Discovery-Search (Current work), CirrusSearch
EBernhardson added a comment to T322327: CirrusSearch should generate a document consistent to a given schema.

After reviewing the results of previous work I'm seeing that file_text => false is still possible. This occurs on files where the media handler does not support extracting text from the file format. This is a significant number of files, most of them really. While other bits only changed the values being returned, here we need to set the default value on all pages to null to ensure it gets cleared out.

Thu, Dec 1, 9:45 PM · Patch-For-Review, MW-1.40-notes (1.40.0-wmf.12; 2022-11-28), Discovery-Search (Current work), CirrusSearch
EBernhardson moved T318575: Restore mjolnir live-indices querying from To Be Deployed to Needs Reporting on the Discovery-Search (Current work) board.

The live-indices querying seems to be working fine now. Unfortunately there is a regression in the LTR plugin (T324247) that will need to be addressed before the mjolnir dag can be re-enabled. For now I've disabled the DAG.

Thu, Dec 1, 8:28 PM · Discovery-Search (Current work), CirrusSearch
EBernhardson created T324247: Incorrect stats returning from 7.10.2 ltr plugin for non-matching terms.
Thu, Dec 1, 8:25 PM · Patch-For-Review, Discovery-Search (Current work), CirrusSearch

Wed, Nov 30

EBernhardson committed rWDAN32871248980c: airflow: Set mjolnir max_active_runs to 1 (authored by EBernhardson).
airflow: Set mjolnir max_active_runs to 1
Wed, Nov 30, 10:18 PM
EBernhardson moved T323687: Enable the wmf_capped doc size limiter in the mediawiki-config for CirrusSearch from To Be Deployed to Needs Reporting on the Discovery-Search (Current work) board.
Wed, Nov 30, 10:08 PM · Discovery-Search (Current work), CirrusSearch
EBernhardson added a comment to T318575: Restore mjolnir live-indices querying.

This is deployed and currently working away at the new reduced rates. It looks like we are now around 30-ish hours to run all the queries necessary for model building.

Wed, Nov 30, 7:06 PM · Discovery-Search (Current work), CirrusSearch
EBernhardson added a comment to T265056: Make Cirrus Search dump script more resilient to failures (elasticsearch restarts).

The error this time around is:

Elastica\Exception\ClientException from line 26 of /srv/mediawiki/php-1.40.0-wmf.10/vendor/ruflin/elastica/src/Connection/Strategy/Simple.php: No enabled connection
#0 /srv/mediawiki/php-1.40.0-wmf.10/vendor/ruflin/elastica/src/Connection/ConnectionPool.php(86): Elastica\Connection\Strategy\Simple->getConnection(Array)
#1 /srv/mediawiki/php-1.40.0-wmf.10/vendor/ruflin/elastica/src/Client.php(394): Elastica\Connection\ConnectionPool->getConnection()
#2 /srv/mediawiki/php-1.40.0-wmf.10/vendor/ruflin/elastica/src/Client.php(508): Elastica\Client->getConnection()
#3 /srv/mediawiki/php-1.40.0-wmf.10/vendor/ruflin/elastica/src/Search.php(278): Elastica\Client->request('cowiki_content/...', 'POST', Array, Array)
#4 /srv/mediawiki/php-1.40.0-wmf.10/extensions/CirrusSearch/includes/Elastica/SearchAfter.php(90): Elastica\Search->search()
#5 /srv/mediawiki/php-1.40.0-wmf.10/extensions/CirrusSearch/includes/Elastica/SearchAfter.php(70): CirrusSearch\Elastica\SearchAfter->runSearch()
#6 /srv/mediawiki/php-1.40.0-wmf.10/extensions/CirrusSearch/maintenance/DumpIndex.php(163): CirrusSearch\Elastica\SearchAfter->next()
#7 /srv/mediawiki/php-1.40.0-wmf.10/maintenance/includes/MaintenanceRunner.php(309): CirrusSearch\Maintenance\DumpIndex->execute()
#8 /srv/mediawiki/php-1.40.0-wmf.10/maintenance/doMaintenance.php(85): MediaWiki\Maintenance\MaintenanceRunner->run()
#9 /srv/mediawiki/php-1.40.0-wmf.10/extensions/CirrusSearch/maintenance/DumpIndex.php(288): require_once('/srv/mediawiki/...')
#10 /srv/mediawiki/multiversion/MWScript.php(120): require_once('/srv/mediawiki/...')
#11 {main}
Wed, Nov 30, 5:13 PM · Patch-For-Review, MW-1.40-notes (1.40.0-wmf.14; 2022-12-12), Discovery-Search (Current work), CirrusSearch, Dumps-Generation

Tue, Nov 29

EBernhardson moved T318575: Restore mjolnir live-indices querying from In Progress to To Be Deployed on the Discovery-Search (Current work) board.
Tue, Nov 29, 7:14 PM · Discovery-Search (Current work), CirrusSearch
EBernhardson claimed T323687: Enable the wmf_capped doc size limiter in the mediawiki-config for CirrusSearch.
Tue, Nov 29, 4:32 PM · Discovery-Search (Current work), CirrusSearch
EBernhardson added a comment to T323687: Enable the wmf_capped doc size limiter in the mediawiki-config for CirrusSearch.

It does not seem that we need a per-wiki approach to this so adding this config option in CirrusSearch-common.php seems appropriate.

Tue, Nov 29, 4:32 PM · Discovery-Search (Current work), CirrusSearch

Mon, Nov 28

EBernhardson claimed T318575: Restore mjolnir live-indices querying.
Mon, Nov 28, 4:57 PM · Discovery-Search (Current work), CirrusSearch
EBernhardson moved T318575: Restore mjolnir live-indices querying from Ready for Dev -- SWE to In Progress on the Discovery-Search (Current work) board.
Mon, Nov 28, 4:57 PM · Discovery-Search (Current work), CirrusSearch

Wed, Nov 23

EBernhardson added a comment to T318575: Restore mjolnir live-indices querying.

After thinking about this for awhile, the method with 429's seem overly complicated and hard to implement. While we can reject queries based on system load in the degraded query router, that would only reject individual shard queries. The other shards would still run, but the end result would be unusable. Additionally removing the production side daemon would reduce the metrics we have available, we don't currently have a good story around sending runtime metrics from spark applications to prometheus. Removing the metrics would ake the system harder to reason about.

Wed, Nov 23, 10:53 PM · Discovery-Search (Current work), CirrusSearch

Tue, Nov 22

EBernhardson moved T318270: Avoid overloading individual Elastic nodes with popular shards from In Progress to Needs Reporting on the Discovery-Search (Current work) board.
Tue, Nov 22, 7:46 PM · Discovery-Search (Current work), Patch-For-Review
EBernhardson added a comment to T318270: Avoid overloading individual Elastic nodes with popular shards.

Checked the stats, commonswiki_file is pretty reasonably distributed across the cluster:

:) (ebernhardson@stat1006)-~$ curl https://search.svc.eqiad.wmnet:9243/_cat/shards/commonswiki_file | awk '{print $8}' | sort | uniq -c | sort                                                                   
     1 elastic1071-production-search-eqiad                                                                                                                                                                       
     1 elastic1076-production-search-eqiad                                                                                                                                                                       
     1 elastic1085-production-search-eqiad                                                                                                                                                                       
     1 elastic1090-production-search-eqiad                                                                                                                                                                       
     2 elastic1053-production-search-eqiad
     2 elastic1054-production-search-eqiad
     2 elastic1055-production-search-eqiad
     2 elastic1056-production-search-eqiad
     2 elastic1057-production-search-eqiad
     2 elastic1058-production-search-eqiad
     2 elastic1059-production-search-eqiad
     2 elastic1060-production-search-eqiad
     2 elastic1061-production-search-eqiad
     2 elastic1062-production-search-eqiad
     2 elastic1063-production-search-eqiad
     2 elastic1064-production-search-eqiad
     2 elastic1065-production-search-eqiad
     2 elastic1066-production-search-eqiad
     2 elastic1067-production-search-eqiad
     2 elastic1068-production-search-eqiad
     2 elastic1069-production-search-eqiad
     2 elastic1070-production-search-eqiad
     2 elastic1072-production-search-eqiad
     2 elastic1073-production-search-eqiad
     2 elastic1074-production-search-eqiad
     2 elastic1075-production-search-eqiad
     2 elastic1077-production-search-eqiad
     2 elastic1078-production-search-eqiad
     2 elastic1079-production-search-eqiad
     2 elastic1080-production-search-eqiad
     2 elastic1081-production-search-eqiad
     2 elastic1082-production-search-eqiad
     2 elastic1083-production-search-eqiad
     2 elastic1084-production-search-eqiad
     2 elastic1086-production-search-eqiad
     2 elastic1087-production-search-eqiad
     2 elastic1088-production-search-eqiad
     2 elastic1089-production-search-eqiad
     2 elastic1091-production-search-eqiad
     2 elastic1092-production-search-eqiad
     2 elastic1093-production-search-eqiad
     2 elastic1094-production-search-eqiad
     2 elastic1095-production-search-eqiad
     2 elastic1096-production-search-eqiad
     2 elastic1097-production-search-eqiad
     2 elastic1098-production-search-eqiad
     2 elastic1099-production-search-eqiad
     2 elastic1100-production-search-eqiad
     2 elastic1101-production-search-eqiad
     2 elastic1102-production-search-eqiad
Tue, Nov 22, 7:44 PM · Discovery-Search (Current work), Patch-For-Review
EBernhardson added a comment to T322327: CirrusSearch should generate a document consistent to a given schema.

labels: should be null or an empty map instead of an empty array
descriptions should be null or an empty map instead of an empty array

Making this an empty map is a bit awkward, we would have to pass around \stdClass in places and document that it should be expected. Instead converted the sources to return either a populated array or null.

\Wikibase\MediaInfo\Content\MediaInfoHandler::getContentDataForSearchIndex might need a small tweak too I think, it enforces the description field to $fieldsData[DescriptionsField::NAME] = [] when no labels are found.

Tue, Nov 22, 4:45 PM · Patch-For-Review, MW-1.40-notes (1.40.0-wmf.12; 2022-11-28), Discovery-Search (Current work), CirrusSearch

Mon, Nov 21

EBernhardson added a comment to T322327: CirrusSearch should generate a document consistent to a given schema.

defaultsort should be null when not set instead of being false

Mon, Nov 21, 11:42 PM · Patch-For-Review, MW-1.40-notes (1.40.0-wmf.12; 2022-11-28), Discovery-Search (Current work), CirrusSearch
EBernhardson added a reverting change for rWDAN48c230a16853: transfer_to_es: Allow first run of wait_for_incoming_links: rWDAN62b613a752fa: Revert "transfer_to_es: Allow first run of wait_for_incoming_links".
Mon, Nov 21, 9:14 PM
EBernhardson committed rWDAN62b613a752fa: Revert "transfer_to_es: Allow first run of wait_for_incoming_links" (authored by EBernhardson).
Revert "transfer_to_es: Allow first run of wait_for_incoming_links"
Mon, Nov 21, 9:14 PM
EBernhardson committed rWDAN48c230a16853: transfer_to_es: Allow first run of wait_for_incoming_links (authored by EBernhardson).
transfer_to_es: Allow first run of wait_for_incoming_links
Mon, Nov 21, 8:16 PM
EBernhardson moved T317023: Investigate moving incoming_links computation to a batch job from Needs Reporting to To Be Deployed on the Discovery-Search (Current work) board.
Mon, Nov 21, 4:10 PM · MW-1.40-notes (1.40.0-wmf.12; 2022-11-28), Patch-For-Review, Discovery-Search (Current work), CirrusSearch

Thu, Nov 17

EBernhardson moved T321170: Wikidata query service does not allow mwapi queries to incubator.wikimedia.org from Needs review to Needs Reporting on the Discovery-Search (Current work) board.

Example query seems to work:

Thu, Nov 17, 6:53 PM · Discovery-Search (Current work), Wikidata, Wikidata-Query-Service
EBernhardson moved T317023: Investigate moving incoming_links computation to a batch job from Needs review to Needs Reporting on the Discovery-Search (Current work) board.
Thu, Nov 17, 6:53 PM · MW-1.40-notes (1.40.0-wmf.12; 2022-11-28), Patch-For-Review, Discovery-Search (Current work), CirrusSearch

Tue, Nov 15

EBernhardson committed rEWLC7a2be692ce3c: Remove now unused getStoredFields method (authored by EBernhardson).
Remove now unused getStoredFields method
Tue, Nov 15, 12:07 AM

Mon, Nov 14

EBernhardson claimed T322327: CirrusSearch should generate a document consistent to a given schema.
Mon, Nov 14, 9:54 PM · Patch-For-Review, MW-1.40-notes (1.40.0-wmf.12; 2022-11-28), Discovery-Search (Current work), CirrusSearch
EBernhardson claimed T321170: Wikidata query service does not allow mwapi queries to incubator.wikimedia.org.
Mon, Nov 14, 9:50 PM · Discovery-Search (Current work), Wikidata, Wikidata-Query-Service
EBernhardson moved T265056: Make Cirrus Search dump script more resilient to failures (elasticsearch restarts) from In Progress to Needs review on the Discovery-Search (Current work) board.
Mon, Nov 14, 8:09 PM · Patch-For-Review, MW-1.40-notes (1.40.0-wmf.14; 2022-12-12), Discovery-Search (Current work), CirrusSearch, Dumps-Generation
EBernhardson committed rEWLC3b69a5774717: Update fixtures for stored_fields -> fields conversion (authored by EBernhardson).
Update fixtures for stored_fields -> fields conversion
Mon, Nov 14, 6:49 PM
EBernhardson moved T317023: Investigate moving incoming_links computation to a batch job from In Progress to Needs review on the Discovery-Search (Current work) board.
Mon, Nov 14, 4:11 PM · MW-1.40-notes (1.40.0-wmf.12; 2022-11-28), Patch-For-Review, Discovery-Search (Current work), CirrusSearch

Thu, Nov 10

EBernhardson added a comment to T317023: Investigate moving incoming_links computation to a batch job.

After some analysis I've found we don't even need to filter pages with low numbers of incoming links directly. The current state of the index is somewhat far out from the exact values, expected due to our usage of super_detect_noop with a 20% threshold. The first run of this will send ~115M pages to be updated. Analysis of two database dumps more than a week apart suggests that most pages have relatively constant incoming_links counts across weeks. Of 185M pages in the dumps with incoming_links, only 6M of them changed between 10-30 and 11-11.

Thu, Nov 10, 5:15 PM · MW-1.40-notes (1.40.0-wmf.12; 2022-11-28), Patch-For-Review, Discovery-Search (Current work), CirrusSearch

Nov 7 2022

EBernhardson added a comment to T265056: Make Cirrus Search dump script more resilient to failures (elasticsearch restarts).

splitting this up to run a few separate groups of wikis seems much more doable than trying to parallelize the bash function, will look into how that can be done. Perhaps we can run one for each db group since thats a reasonably convenient split of wikis by size that already exists.

Nov 7 2022, 5:24 PM · Patch-For-Review, MW-1.40-notes (1.40.0-wmf.14; 2022-12-12), Discovery-Search (Current work), CirrusSearch, Dumps-Generation
EBernhardson moved T312175: Create alerts when cirrus jobs backlog from Needs review to Needs Reporting on the Discovery-Search (Current work) board.

this will be auto-deployed within 30 minutes of the merge and should be live.

Nov 7 2022, 5:15 PM · Discovery-Search (Current work)
EBernhardson added a comment to T322408: CirrusSearch support for PHP 8.1.

This is used for experimenting with different variations of search, it wont be found in any automation. It's still used but not all that often. There is a patch, gerrit:812401 that was previously failing due to an unrelated test failure in Echo that I didn't understand. Checking back into this patch today it seems to be working, will need to get this reviewed now.

Nov 7 2022, 4:58 PM · MW-1.40-notes (1.40.0-wmf.10; 2022-11-14), Patch-For-Review, Discovery-Search (Current work), CirrusSearch, PHP 8.1 support
EBernhardson moved T265056: Make Cirrus Search dump script more resilient to failures (elasticsearch restarts) from To Be Deployed to In Progress on the Discovery-Search (Current work) board.
Nov 7 2022, 4:09 PM · Patch-For-Review, MW-1.40-notes (1.40.0-wmf.14; 2022-12-12), Discovery-Search (Current work), CirrusSearch, Dumps-Generation

Oct 31 2022

EBernhardson moved T262630: ProductionServices.php has cloudelastic-{psi,omega}-eqiad ports mixed up from To Be Deployed to Needs Reporting on the Discovery-Search (Current work) board.
Oct 31 2022, 8:36 PM · Discovery-Search (Current work)
EBernhardson updated the task description for T147505: [tracking] CirrusSearch: what is updated during re-indexing.
Oct 31 2022, 7:46 PM · Tracking-Neverending, Epic, Discovery-Search (Current work), Discovery-ARCHIVED
EBernhardson updated the task description for T147505: [tracking] CirrusSearch: what is updated during re-indexing.
Oct 31 2022, 7:46 PM · Tracking-Neverending, Epic, Discovery-Search (Current work), Discovery-ARCHIVED
EBernhardson added a comment to T265056: Make Cirrus Search dump script more resilient to failures (elasticsearch restarts).

I mistakenly didn't attach to this ticket, but another patch has been merged to use backoff with retries that will hopefully manage to get past the previous zhwiki failure and stabilize the dumps.

Oct 31 2022, 7:36 PM · Patch-For-Review, MW-1.40-notes (1.40.0-wmf.14; 2022-12-12), Discovery-Search (Current work), CirrusSearch, Dumps-Generation
EBernhardson moved T265056: Make Cirrus Search dump script more resilient to failures (elasticsearch restarts) from In Progress to To Be Deployed on the Discovery-Search (Current work) board.
Oct 31 2022, 4:20 PM · Patch-For-Review, MW-1.40-notes (1.40.0-wmf.14; 2022-12-12), Discovery-Search (Current work), CirrusSearch, Dumps-Generation

Oct 28 2022

EBernhardson committed rWDAN2326f9c67c5f: Import cirrus indexes to hdfs (authored by EBernhardson).
Import cirrus indexes to hdfs
Oct 28 2022, 9:41 PM

Oct 26 2022

EBernhardson added a comment to T321677: Use user_random sort in LocalSearchTaskSuggester.

Here's a use case from the SD team:

We need to get ALL pages-with-suggestions for a particular wiki, in order to send notifications to experienced users who have those pages on their watchlists. ATM we query elasticsearch directly to get these because it's tricky to get more than 10k results from elastic, so we have to use elastic's scroll API

ATM we do not exclude results with wiki-specific infbox templates. The simplest way to do this would be to just allow Growth's code to do it, but atm they can only return a limited number of results, and we can't page through them

Would this change make it possible to page through everything? Or will we still hit the 10k results cap? @dcausse @EBernhardson

Oct 26 2022, 7:37 PM · GrowthExperiments, Growth-Team
EBernhardson merged T321680: Problems with autocompletion in Search bar into T317381: Reduction in helpfulness and quantity of autocomplete search results.
Oct 26 2022, 5:57 PM · Wikimedia-Incident, Discovery-Search (Current work), Elasticsearch, CirrusSearch
EBernhardson merged task T321680: Problems with autocompletion in Search bar into T317381: Reduction in helpfulness and quantity of autocomplete search results.
Oct 26 2022, 5:57 PM · CirrusSearch, Discovery-Search

Oct 25 2022

EBernhardson created P36346 (An Untitled Masterwork).
Oct 25 2022, 7:28 PM

Oct 24 2022

EBernhardson added a comment to T321006: Special:Search: "429 Too Many Requests" when number of returned search results set to more than 50 .

Digging a bit more into this...

Oct 24 2022, 11:01 PM · MW-1.40-notes (1.40.0-wmf.8; 2022-10-31), Structured-Data-Backlog (Current Work), Discovery-Search, SDAW-Search-Improvements (Milestone 1: CSS changes to all wikis)
EBernhardson updated the language for P36132 Test thumbnail timing from autodetect to php.
Oct 24 2022, 10:39 PM
EBernhardson created P36132 Test thumbnail timing.
Oct 24 2022, 10:29 PM
EBernhardson added a comment to T321006: Special:Search: "429 Too Many Requests" when number of returned search results set to more than 50 .

For the search side of things we can attach &cirrusDumpResult to any Special:Search request, this will short-circuit the request once elasticsearch responds and tell us how long the search engine took. In my quick testing:

Oct 24 2022, 10:09 PM · MW-1.40-notes (1.40.0-wmf.8; 2022-10-31), Structured-Data-Backlog (Current Work), Discovery-Search, SDAW-Search-Improvements (Milestone 1: CSS changes to all wikis)
EBernhardson set the point value for T318270: Avoid overloading individual Elastic nodes with popular shards to 3.
Oct 24 2022, 3:45 PM · Discovery-Search (Current work), Patch-For-Review
EBernhardson moved T320408: Monitor CirrusSearch update lag from Ready for Dev -- SWE to Ready for Dev -- SRE/Ops on the Discovery-Search (Current work) board.
Oct 24 2022, 3:34 PM · Discovery-Search, CirrusSearch
EBernhardson moved T143553: Switching search traffic between datacenters should be faster from Ready for Dev -- SWE to Blocked/Waiting on the Discovery-Search (Current work) board.
Oct 24 2022, 3:33 PM · Discovery-Search, Sustainability (Incident Followup), Discovery-ARCHIVED, Elasticsearch
EBernhardson moved T303134: Should wdqs LVS checks page from Ready for Dev -- SWE to Ready for Dev -- SRE/Ops on the Discovery-Search (Current work) board.
Oct 24 2022, 3:30 PM · Discovery-Search, wdwb-tech, Wikidata, Sustainability (Incident Followup), Wikidata-Query-Service
EBernhardson moved T319020: Reset to upstream java GC options and remove redundant JVM options from Ready for Dev -- SWE to Ready for Dev -- SRE/Ops on the Discovery-Search (Current work) board.
Oct 24 2022, 3:30 PM · Patch-For-Review, Discovery-Search (Current work)
EBernhardson moved T317816: Enable 10G networking in cirrus elastic clusters from Ready for Dev -- SWE to Ready for Dev -- SRE/Ops on the Discovery-Search (Current work) board.
Oct 24 2022, 3:29 PM · Discovery-Search (Current work)
EBernhardson moved T313095: Improve Search team alerting for missing masters from Ready for Dev -- SWE to Ready for Dev -- SRE/Ops on the Discovery-Search (Current work) board.
Oct 24 2022, 3:29 PM · Discovery-Search, Sustainability (Incident Followup), SRE-OnFire, Observability-Alerting
EBernhardson moved T305818: Perform a data transfer to wdqs2004 & wdqs1004 to reclaim burnt allocators from Ready for Dev -- SWE to Ready for Dev -- SRE/Ops on the Discovery-Search (Current work) board.
Oct 24 2022, 3:29 PM · Discovery-Search (Current work), Wikidata, Wikidata-Query-Service
EBernhardson moved T318270: Avoid overloading individual Elastic nodes with popular shards from Waiting to In Progress on the Discovery-Search (Current work) board.
Oct 24 2022, 3:23 PM · Discovery-Search (Current work), Patch-For-Review
EBernhardson added a comment to T318270: Avoid overloading individual Elastic nodes with popular shards.

We should look into if commonswiki is having similar issues, when commonswiki gets heavily loaded we often have a few instances with significantly more load than the others.

Oct 24 2022, 3:21 PM · Discovery-Search (Current work), Patch-For-Review
EBernhardson moved T265056: Make Cirrus Search dump script more resilient to failures (elasticsearch restarts) from Needs review to In Progress on the Discovery-Search (Current work) board.
Oct 24 2022, 3:20 PM · Patch-For-Review, MW-1.40-notes (1.40.0-wmf.14; 2022-12-12), Discovery-Search (Current work), CirrusSearch, Dumps-Generation
EBernhardson moved T178328: Create maintenance script to clean up stale indexes from To Be Deployed to Needs Reporting on the Discovery-Search (Current work) board.
Oct 24 2022, 3:20 PM · Discovery-Search (Current work), CirrusSearch

Oct 20 2022

EBernhardson added a comment to T317023: Investigate moving incoming_links computation to a batch job.

With the data now available in hdfs I've run some stats over the data available:

Oct 20 2022, 11:14 PM · MW-1.40-notes (1.40.0-wmf.12; 2022-11-28), Patch-For-Review, Discovery-Search (Current work), CirrusSearch
EBernhardson added a comment to T317023: Investigate moving incoming_links computation to a batch job.

The first step, to load the cirrus indices into the hadoop cluster so we can further process them, is mostly ready. Already have data in yarn to work with and the automated job should be deployeed next week. Some notes:

Oct 20 2022, 10:54 PM · MW-1.40-notes (1.40.0-wmf.12; 2022-11-28), Patch-For-Review, Discovery-Search (Current work), CirrusSearch
EBernhardson added a comment to T317682: Make new Vector search navigate to search result URL when selecting search result using keyboard.

Poked over the history and the related tests. There are tests in tests/browser/SearchSatisfactionTests.php that expect to log a -1 as the position when the user submits their own query and not something provided by the autocomplete. This seems to have been provided as data.index to the autocomplete track function.

Oct 20 2022, 6:31 PM · MW-1.40-notes (1.40.0-wmf.12; 2022-11-28), Wikidata Dev Team (Sprint-∞), Patch-For-Review, Codex, Design-Systems-Team, Discovery-Search, Desktop Improvements (Vector 2022), MediaWiki-Interface (autocomplete search), wmde-wikidata-tech, Wikidata

Oct 18 2022

EBernhardson claimed T317023: Investigate moving incoming_links computation to a batch job.
Oct 18 2022, 10:42 PM · MW-1.40-notes (1.40.0-wmf.12; 2022-11-28), Patch-For-Review, Discovery-Search (Current work), CirrusSearch
EBernhardson moved T317023: Investigate moving incoming_links computation to a batch job from Ready for Dev -- SWE to In Progress on the Discovery-Search (Current work) board.
Oct 18 2022, 10:42 PM · MW-1.40-notes (1.40.0-wmf.12; 2022-11-28), Patch-For-Review, Discovery-Search (Current work), CirrusSearch
EBernhardson updated the task description for T316236: Reload WCQS from dumps.
Oct 18 2022, 6:24 PM · Patch-For-Review, Discovery-Search (Current work), Wikidata, Wikidata-Query-Service

Oct 17 2022

EBernhardson added a comment to T178328: Create maintenance script to clean up stale indexes.

Related patch takes a slightly different approach than suggested in the ticket. The ticket was to clean up these stale indices at some point after they fail, but the solution we went with was delete the bad indices when they fail instead of leaving the broken indices on the cluster.

Oct 17 2022, 6:16 PM · Discovery-Search (Current work), CirrusSearch
EBernhardson moved T316711: Reduce shard count on all wikis in beta cluster to 1 from To Be Deployed to Needs Reporting on the Discovery-Search (Current work) board.

commonswiki_file is now complete as well. Test reports no indices with multiple shards:

curl -s https://deployment-elastic09.deployment-prep.eqiad1.wikimedia.cloud:9243/_cat/indices | awk '$5 > 1 { print $0 }'
Oct 17 2022, 5:58 PM · Discovery-Search (Current work)
EBernhardson claimed T262630: ProductionServices.php has cloudelastic-{psi,omega}-eqiad ports mixed up.
Oct 17 2022, 5:51 PM · Discovery-Search (Current work)
EBernhardson added a comment to T262630: ProductionServices.php has cloudelastic-{psi,omega}-eqiad ports mixed up.

Trying to track down what happened here. Was it ever mixed up? Did we fix it at some point in the past?

Oct 17 2022, 5:46 PM · Discovery-Search (Current work)
EBernhardson added a comment to T262630: ProductionServices.php has cloudelastic-{psi,omega}-eqiad ports mixed up.

As a secondary check, compared the first 5 sorted content/general indices by cluster (ignoring _titlesuggest and _archive since those don't exist on cloudelastic):

for port in 9{2,4,6}43; do
    echo '*****'
    echo
    for cluster in search.svc.{eqiad,codfw}.wmnet cloudelastic.wikimedia.org; do
        echo https://$cluster:$port/
        curl -s https://$cluster:$port/_cat/indices | awk '{print $3}' | grep '_\(content\|general\)' | sort | head -n 5 | sed 's/^/    /'
    done
done
Oct 17 2022, 4:51 PM · Discovery-Search (Current work)
EBernhardson added a comment to T262630: ProductionServices.php has cloudelastic-{psi,omega}-eqiad ports mixed up.

Not sure this is still the case,

Oct 17 2022, 4:38 PM · Discovery-Search (Current work)

Oct 13 2022

EBernhardson moved T319136: Allow federated queries with the Eu Knowledge Graph from To Be Deployed to Needs Reporting on the Discovery-Search (Current work) board.

This has been deployed. If anything isn't working right please ping us here.

Oct 13 2022, 8:51 PM · Discovery-Search (Current work), Wikidata, Wikidata-Query-Service
EBernhardson added a comment to T143553: Switching search traffic between datacenters should be faster.

The above patches would mostly take care of cirrussearch, but apifeatureusage and translate still need to be handled. Apifeatureusage should be easy, it's read-only from the wiki side. translate will need some consideration, it needs different logic for reads and writes.

Oct 13 2022, 7:26 PM · Discovery-Search, Sustainability (Incident Followup), Discovery-ARCHIVED, Elasticsearch
EBernhardson added a comment to T265056: Make Cirrus Search dump script more resilient to failures (elasticsearch restarts).

With these errors dated oct 4-6 it's not clear that they would have been running the new search_after code. Hopefully this fails less on this weeks run. I've checked the current log outputs and all seems happy (except bclwikiquote which was a brand new wiki without an index yet), but it's only up to commonswiki_file. Unfortunately this weeks dumps with the new code seem to be running signficantly slower, might have to investigate that.

Oct 13 2022, 7:18 PM · Patch-For-Review, MW-1.40-notes (1.40.0-wmf.14; 2022-12-12), Discovery-Search (Current work), CirrusSearch, Dumps-Generation
EBernhardson added a comment to T317546: Add new elasticsearch field to index the number of outgoing links.

@Tgr reindexing is completed, this field is now available on all indices in all clusters (prod and beta)

Oct 13 2022, 6:42 PM · MW-1.40-notes (1.40.0-wmf.1; 2022-09-12), Discovery-Search (Current work), Growth-Team (Current Sprint), Add-Link
EBernhardson updated the task description for T147505: [tracking] CirrusSearch: what is updated during re-indexing.
Oct 13 2022, 6:32 PM · Tracking-Neverending, Epic, Discovery-Search (Current work), Discovery-ARCHIVED
EBernhardson moved T317200: Reindex all wikis to fix nnbsp regression from In Progress to Needs Reporting on the Discovery-Search (Current work) board.

All indices have been reindexed, additionally ran the check_indices.py script and cleaned up extra indexes that were left behind

Oct 13 2022, 6:31 PM · MW-1.40-notes (1.40.0-wmf.6; 2022-10-17), Discovery-Search (Current work)

Oct 5 2022

EBernhardson added a comment to T316016: Limit the size of the documents indexed by CirrusSearch.

Couldn't help poking this a bit. I adjusted the previous query to get percentiles to instead perform an aggregation and report the top 4 (arbitrary) pages by size from each index (P35367), then import it into pandas (P35368) and get the following list (which excludes private wikis) for the top 100 pages by size from those results: P35369

Oct 5 2022, 7:10 PM · MW-1.40-notes (1.40.0-wmf.8; 2022-10-31), Patch-For-Review, Discovery-Search (Current work), CirrusSearch
EBernhardson created P35369 some of the top pages by search index source size.
Oct 5 2022, 7:06 PM
EBernhardson edited P35368 code to extract and source top source size from results of P35367.
Oct 5 2022, 7:01 PM
EBernhardson created P35368 code to extract and source top source size from results of P35367.
Oct 5 2022, 6:59 PM
EBernhardson created P35367 top pages by source size from all elasticsearch indices query.
Oct 5 2022, 6:57 PM

Oct 4 2022

EBernhardson added a comment to T316711: Reduce shard count on all wikis in beta cluster to 1.

This is 99% of the way there, only remaining index is commonswiki_file. When applying the new shard settings I failed to set an explicit value for the file index suffix, which only exists on commonswiki, which results in cirrus refusing to create a new index there. Once the next patch is shipped we should be able to close this out.

Oct 4 2022, 9:29 PM · Discovery-Search (Current work)

Oct 3 2022

EBernhardson added a comment to T143553: Switching search traffic between datacenters should be faster.

other loose ends, we report the default search cluster via APIQuerySetInfoGeneralInfo which is read by SRE tools to ensure we don't forget that traffic is directed at non-local clusters. We probably need to either kill those or replace the checks with something that looks at the etcd based data.

Oct 3 2022, 8:20 PM · Discovery-Search, Sustainability (Incident Followup), Discovery-ARCHIVED, Elasticsearch
EBernhardson added a comment to T143553: Switching search traffic between datacenters should be faster.

Realized while looking at this in mediawiki-config that we also need to deploy an envoy proxy that handles these connections.

Oct 3 2022, 5:44 PM · Discovery-Search, Sustainability (Incident Followup), Discovery-ARCHIVED, Elasticsearch
EBernhardson added a comment to T316711: Reduce shard count on all wikis in beta cluster to 1.

started up reindexing on deployment-prep again, after verifying that $wgCirrusSearchShardCount is appropriately set via shell.php

Oct 3 2022, 5:22 PM · Discovery-Search (Current work)
EBernhardson moved T319136: Allow federated queries with the Eu Knowledge Graph from Incoming to Needs review on the Discovery-Search (Current work) board.
Oct 3 2022, 5:19 PM · Discovery-Search (Current work), Wikidata, Wikidata-Query-Service
EBernhardson added a project to T319136: Allow federated queries with the Eu Knowledge Graph: Discovery-Search (Current work).
Oct 3 2022, 5:18 PM · Discovery-Search (Current work), Wikidata, Wikidata-Query-Service