Page MenuHomePhabricator

EBernhardson (EBernhardson)
User

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Thursday

  • Clear sailing ahead.

User Details

User Since
Oct 7 2014, 4:49 PM (237 w, 1 h)
Availability
Available
LDAP User
EBernhardson
MediaWiki User
EBernhardson (WMF) [ Global Accounts ]

Recent Activity

Today

EBernhardson added a comment to T221547: Elastica\Exception\Connection\HttpException: Malformed URL.

Try defining the configuration as follows. Likely we need to improve the documentation in this area.

Tue, Apr 23, 4:51 PM · Discovery-Search, CirrusSearch

Fri, Apr 19

EBernhardson added a comment to T219234: Job runner request timeouts in Elastica\Task.

Only seeing timeouts against php-1.33.0-wmf.25, nothing against 1.34-wmf.1 yet. Should let this run for a week before declaring victory on the timeouts though.

Fri, Apr 19, 9:18 PM · Discovery-Search (Current work), CirrusSearch, MW-1.34-notes (1.34.0-wmf.3; 2019-04-30), MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Wikimedia-production-error
EBernhardson edited projects for T219234: Job runner request timeouts in Elastica\Task, added: Discovery-Search (Current work); removed Discovery-Search.
Fri, Apr 19, 7:22 PM · Discovery-Search (Current work), CirrusSearch, MW-1.34-notes (1.34.0-wmf.3; 2019-04-30), MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Wikimedia-production-error

Thu, Apr 18

EBernhardson added a comment to T221121: Capacity planning for elastic search .

A few variations that might be useful to test (using gor middleware to modify the queries). These would mostly inform our options for reducing server load if necessary for incident response:

  • Reduce LTR rescore window
  • Removing the LTR rescore
  • Reduce popularity rescore window
Thu, Apr 18, 6:09 PM · Discovery-Search (Current work)
EBernhardson moved T221135: Include WikibaseCirrusSearch into CirrusSearch tests from needs triage to elastic / cirrus on the Discovery-Search board.
Thu, Apr 18, 6:05 PM · Discovery-Search, Discovery
EBernhardson moved T221315: Determine future of bare-metal hosting for services like WDQS, ElasticSearch, RESTBase Cassandra, etc. from needs triage to watching / waiting on the Discovery-Search board.
Thu, Apr 18, 6:05 PM · Core Platform Team Backlog (Designing), Services (designing), RESTBase, Discovery-Search, Wikidata, serviceops, Release Pipeline, Elasticsearch, Wikidata-Query-Service

Wed, Apr 17

EBernhardson moved T18237: Sort results by date from Waiting/Blocked to Done on the Discovery-Search (Current work) board.
Wed, Apr 17, 12:01 AM · Discovery-Search (Current work), Discovery, MediaWiki-Search
EBernhardson added a comment to T18237: Sort results by date.

Sorting functionality deployed and appears to be returning correct results now. Some followup will be needed for UI elements, probably T197525

Wed, Apr 17, 12:01 AM · Discovery-Search (Current work), Discovery, MediaWiki-Search
EBernhardson moved T215487: search sorted by creation date missing some items from Waiting/Blocked to Done on the Discovery-Search (Current work) board.
Wed, Apr 17, 12:00 AM · Discovery-Search (Current work), MW-1.33-notes (1.33.0-wmf.17; 2019-02-12), Patch-For-Review, CirrusSearch, Discovery
EBernhardson added a comment to T215487: search sorted by creation date missing some items.

Example query now returns appropriate results. It seems the processes involved here are all working as intended, calling this complete.

Wed, Apr 17, 12:00 AM · Discovery-Search (Current work), MW-1.33-notes (1.33.0-wmf.17; 2019-02-12), Patch-For-Review, CirrusSearch, Discovery

Tue, Apr 16

EBernhardson moved T221058: Fatal error when submitting "insource" query on mediawiki.org from in progress to Done on the Discovery-Search (Current work) board.
Tue, Apr 16, 11:58 PM · Discovery-Search (Current work), CirrusSearch, Wikimedia-production-error
EBernhardson moved T221019: Metrics reported for wrong cluster when $wgCirrusSearchClusterOverrides is used from in progress to Done on the Discovery-Search (Current work) board.
Tue, Apr 16, 11:58 PM · MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Discovery-Search (Current work), MW-1.34-notes (1.34.0-wmf.1; 2019-04-16), Patch-For-Review
EBernhardson moved T221058: Fatal error when submitting "insource" query on mediawiki.org from needs triage to Current work on the Discovery-Search board.
Tue, Apr 16, 11:58 PM · Discovery-Search (Current work), CirrusSearch, Wikimedia-production-error
EBernhardson moved T221019: Metrics reported for wrong cluster when $wgCirrusSearchClusterOverrides is used from needs triage to Current work on the Discovery-Search board.
Tue, Apr 16, 11:58 PM · MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Discovery-Search (Current work), MW-1.34-notes (1.34.0-wmf.1; 2019-04-16), Patch-For-Review
EBernhardson claimed T221019: Metrics reported for wrong cluster when $wgCirrusSearchClusterOverrides is used.
Tue, Apr 16, 11:58 PM · MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Discovery-Search (Current work), MW-1.34-notes (1.34.0-wmf.1; 2019-04-16), Patch-For-Review
EBernhardson updated the task description for T220901: Elasticsearch nodes overloading in eqiad.
Tue, Apr 16, 11:39 PM · Patch-For-Review, Operations, Discovery-Search (Current work)
EBernhardson claimed T221058: Fatal error when submitting "insource" query on mediawiki.org.
Tue, Apr 16, 11:35 PM · Discovery-Search (Current work), CirrusSearch, Wikimedia-production-error
EBernhardson merged T221027: Terminating slash in insource regexp breaks PHP into T221058: Fatal error when submitting "insource" query on mediawiki.org.
Tue, Apr 16, 11:22 PM · Discovery-Search (Current work), CirrusSearch, Wikimedia-production-error
EBernhardson merged task T221027: Terminating slash in insource regexp breaks PHP into T221058: Fatal error when submitting "insource" query on mediawiki.org.
Tue, Apr 16, 11:22 PM · Elasticsearch, Discovery-Search
EBernhardson updated the name of F28682359: www.wikipedia.org linux, Firefox 66.0.2 (64-bit) from "www.wi" to "www.wikipedia.org linux, Firefox 66.0.2 (64-bit)".
Tue, Apr 16, 11:12 PM
EBernhardson added a comment to T221121: Capacity planning for elastic search .

While not fully documented, our the results of previous load testing rounds and the methodology used are described here:

Tue, Apr 16, 10:20 PM · Discovery-Search (Current work)
EBernhardson merged T221042: Got "TypeError" when searching with prefix: in the Spanish Wikipedia into T221058: Fatal error when submitting "insource" query on mediawiki.org.
Tue, Apr 16, 2:42 AM · Discovery-Search (Current work), CirrusSearch, Wikimedia-production-error
EBernhardson merged task T221042: Got "TypeError" when searching with prefix: in the Spanish Wikipedia into T221058: Fatal error when submitting "insource" query on mediawiki.org.
Tue, Apr 16, 2:42 AM · Wikimedia-production-error, Discovery-Search, CirrusSearch
EBernhardson merged task T220927: PHP fatal error in Elastica extension into T221058: Fatal error when submitting "insource" query on mediawiki.org.
Tue, Apr 16, 2:41 AM · Wikimedia-production-error, Discovery-Search, Elasticsearch
EBernhardson merged T220927: PHP fatal error in Elastica extension into T221058: Fatal error when submitting "insource" query on mediawiki.org.
Tue, Apr 16, 2:41 AM · Discovery-Search (Current work), CirrusSearch, Wikimedia-production-error
EBernhardson added a comment to T221058: Fatal error when submitting "insource" query on mediawiki.org.

Looks like because insource uses a different connection than the standard connection (mitigation of cluster overloads over the weekend) the attempt to source the last sent request from the connection fails. Will need to get the right connection object into ElasticsearchIntermediary::multiFailure()

Tue, Apr 16, 2:14 AM · Discovery-Search (Current work), CirrusSearch, Wikimedia-production-error

Mon, Apr 15

EBernhardson added a comment to T220901: Elasticsearch nodes overloading in eqiad.

Looked into a few angles but nothing conclusive:

Mon, Apr 15, 10:20 PM · Patch-For-Review, Operations, Discovery-Search (Current work)
EBernhardson added a project to T221019: Metrics reported for wrong cluster when $wgCirrusSearchClusterOverrides is used: Discovery-Search.
Mon, Apr 15, 4:58 PM · MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Discovery-Search (Current work), MW-1.34-notes (1.34.0-wmf.1; 2019-04-16), Patch-For-Review
EBernhardson created T221019: Metrics reported for wrong cluster when $wgCirrusSearchClusterOverrides is used.
Mon, Apr 15, 4:58 PM · MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Discovery-Search (Current work), MW-1.34-notes (1.34.0-wmf.1; 2019-04-16), Patch-For-Review

Sun, Apr 14

EBernhardson added a comment to T220901: Elasticsearch nodes overloading in eqiad.

I don't know it's necessarily related, but i noticed that full text qps is up in the last month. Over the last year we've been pretty consistent between 400-500qps, but since late march we've been at 550-650 or so.

Sun, Apr 14, 6:52 AM · Patch-For-Review, Operations, Discovery-Search (Current work)
EBernhardson added a comment to T220901: Elasticsearch nodes overloading in eqiad.

Patch does not fix overall problem, it fixes the per-node percentiles data collection which usually helps tracking down these kinds of problems.

Sun, Apr 14, 6:30 AM · Patch-For-Review, Operations, Discovery-Search (Current work)
EBernhardson updated subscribers of T220901: Elasticsearch nodes overloading in eqiad.
Sun, Apr 14, 6:29 AM · Patch-For-Review, Operations, Discovery-Search (Current work)
EBernhardson added a comment to T220901: Elasticsearch nodes overloading in eqiad.

A previous time this happened we added some new metrics endpoints inside elasticsearch and started logging them to prometheus to collect per-node latency metrics based on stats buckets we provide at query time. Unfortunately the prometheus graphs seem empty. Should also see how to get these back, they would potentially help.

Sun, Apr 14, 6:16 AM · Patch-For-Review, Operations, Discovery-Search (Current work)
EBernhardson created T220901: Elasticsearch nodes overloading in eqiad.
Sun, Apr 14, 6:15 AM · Patch-For-Review, Operations, Discovery-Search (Current work)

Fri, Apr 12

EBernhardson added a comment to T220823: Use ElasticSearch for bulk Wikidata entity term lookup.

As a very rough comparison, i pulled sum(irate(elasticsearch_indices_search_query_total[5m])) from prometheus, which gives 5 min averages for total shard queries executed per second across the cluster as 5 minute averages. We vary between about 12k and 21k shard queries per second, or about 840k to 1.25M per minute. This at least puts the volume of requests discussed here in the plausible range.

Fri, Apr 12, 4:15 PM · Discovery-Search, User-Addshore, wikidata-tech-focus, Wikidata

Thu, Apr 11

EBernhardson added a comment to T220758: Flaky wdio test "Special:Version has WikibaseMediaInfo listed".

Should we rollback the addition to shared build until this can be resolved?

Thu, Apr 11, 8:18 PM · MW-1.34-notes (1.34.0-wmf.1; 2019-04-16), Multimedia, SDC Engineering, WikibaseMediaInfo, Wikimedia-production-error (Shared Build Failure)
EBernhardson added a comment to T220723: Install extension require-dev dependencies in wmf-quibble-vendor-mysql-hhvm-docker.

Related workaround: https://gerrit.wikimedia.org/r/#/c/mediawiki/vendor/+/503068/

Thu, Apr 11, 6:11 PM · TechCom, Librarization, Quibble
EBernhardson created T220723: Install extension require-dev dependencies in wmf-quibble-vendor-mysql-hhvm-docker.
Thu, Apr 11, 5:48 PM · TechCom, Librarization, Quibble
EBernhardson triaged T220637: [Bug] Unexpected result set returned by Elasticsearch as Normal priority.
Thu, Apr 11, 5:06 PM · Readers-Web-Backlog (Tracking), MW-1.34-notes (1.34.0-wmf.1; 2019-04-16), Patch-For-Review, Discovery-Search, Elasticsearch, GeoData
EBernhardson moved T220637: [Bug] Unexpected result set returned by Elasticsearch from needs triage to Geodata on the Discovery-Search board.
Thu, Apr 11, 5:06 PM · Readers-Web-Backlog (Tracking), MW-1.34-notes (1.34.0-wmf.1; 2019-04-16), Patch-For-Review, Discovery-Search, Elasticsearch, GeoData
EBernhardson added a comment to T220637: [Bug] Unexpected result set returned by Elasticsearch.

Not expected, although hard to say what the error is. GeoData error handling needs to be updated log whatever the response it didn't like was

Thu, Apr 11, 5:06 PM · Readers-Web-Backlog (Tracking), MW-1.34-notes (1.34.0-wmf.1; 2019-04-16), Patch-For-Review, Discovery-Search, Elasticsearch, GeoData
EBernhardson moved T55652: Special:Search doesn't use labels and descriptions for suggestions but just the item ID from needs triage to Wikidata Search on the Discovery-Search board.
Thu, Apr 11, 4:05 PM · Discovery-Search, Patch-For-Review, User-Smalyshev, Wikidata, MediaWiki-extensions-WikibaseRepository
EBernhardson moved T220557: Prepare hackathon presentation about how to use cloudelastic from needs triage to elastic / cirrus on the Discovery-Search board.
Thu, Apr 11, 4:04 PM · Discovery-Search, Cloud-Services, Elasticsearch, Discovery
EBernhardson moved T220554: Open cloudelastic to wmf cloud hosts from needs triage to Ops / SRE on the Discovery-Search board.
Thu, Apr 11, 4:04 PM · Discovery-Search, Cloud-Services, Elasticsearch, Discovery
EBernhardson moved T220625: Initialize CirrusSearch on cloudelastic from needs triage to Current work on the Discovery-Search board.
Thu, Apr 11, 4:04 PM · Discovery-Search (Current work), Patch-For-Review, Cloud-Services, Elasticsearch, Discovery
EBernhardson claimed T220547: Document CirrusSearch schema.
Thu, Apr 11, 3:55 PM · Discovery-Search (Current work), Documentation, Cloud-Services, Elasticsearch, Discovery
EBernhardson moved T220547: Document CirrusSearch schema from needs triage to Current work on the Discovery-Search board.
Thu, Apr 11, 3:54 PM · Discovery-Search (Current work), Documentation, Cloud-Services, Elasticsearch, Discovery

Wed, Apr 10

EBernhardson updated the task description for T220625: Initialize CirrusSearch on cloudelastic.
Wed, Apr 10, 4:26 PM · Discovery-Search (Current work), Patch-For-Review, Cloud-Services, Elasticsearch, Discovery
EBernhardson renamed T220625: Initialize CirrusSearch on cloudelastic from Initialize CirrusSeearch on cloudelastic to Initialize CirrusSearch on cloudelastic.
Wed, Apr 10, 4:24 PM · Discovery-Search (Current work), Patch-For-Review, Cloud-Services, Elasticsearch, Discovery
EBernhardson updated the task description for T220625: Initialize CirrusSearch on cloudelastic.
Wed, Apr 10, 4:24 PM · Discovery-Search (Current work), Patch-For-Review, Cloud-Services, Elasticsearch, Discovery
EBernhardson updated the task description for T220625: Initialize CirrusSearch on cloudelastic.
Wed, Apr 10, 4:23 PM · Discovery-Search (Current work), Patch-For-Review, Cloud-Services, Elasticsearch, Discovery
EBernhardson updated the task description for T220625: Initialize CirrusSearch on cloudelastic.
Wed, Apr 10, 4:15 PM · Discovery-Search (Current work), Patch-For-Review, Cloud-Services, Elasticsearch, Discovery
EBernhardson updated the task description for T220625: Initialize CirrusSearch on cloudelastic.
Wed, Apr 10, 4:14 PM · Discovery-Search (Current work), Patch-For-Review, Cloud-Services, Elasticsearch, Discovery
EBernhardson created T220625: Initialize CirrusSearch on cloudelastic.
Wed, Apr 10, 4:14 PM · Discovery-Search (Current work), Patch-For-Review, Cloud-Services, Elasticsearch, Discovery

Tue, Apr 9

EBernhardson added a comment to T220547: Document CirrusSearch schema.

Created a horrible first draft that lists most of the properties and provides a short description for the ones used across most wikis. We should figure out how we want to format this before going much further:

Tue, Apr 9, 11:47 PM · Discovery-Search (Current work), Documentation, Cloud-Services, Elasticsearch, Discovery
EBernhardson created T220557: Prepare hackathon presentation about how to use cloudelastic.
Tue, Apr 9, 9:31 PM · Discovery-Search, Cloud-Services, Elasticsearch, Discovery
EBernhardson created T220554: Open cloudelastic to wmf cloud hosts.
Tue, Apr 9, 9:19 PM · Discovery-Search, Cloud-Services, Elasticsearch, Discovery
EBernhardson updated subscribers of T220205: Define constraints for cloudelastic use cases.
Tue, Apr 9, 9:11 PM · Discovery-Search (Current work)
EBernhardson closed T190010: JavaScript redirect shows irrelevant internal information in search page as Resolved.

This is an intentional feature added by the people behind AdvancedSearch. The high level goal there is for the URL to represent what is being searched. In particular if a user has a set of namespaces saved as their default search namespaces their search URL's will not be shareable. The exact implementation details are debatable, but the overall goal is reasonable. See T217445 for more details, discussion of the feature should likely also happen there.

Tue, Apr 9, 8:58 PM · Performance-Team (Radar), MW-1.32-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)), Patch-For-Review, Discovery-Search (Current work), MediaWiki-Search, Discovery
EBernhardson added a subtask for T109715: Replicate production elasticsearch indices to labs: T220205: Define constraints for cloudelastic use cases.
Tue, Apr 9, 8:14 PM · Discovery-Search, Cloud-Services, Elasticsearch, Discovery
EBernhardson added a parent task for T220205: Define constraints for cloudelastic use cases: T109715: Replicate production elasticsearch indices to labs.
Tue, Apr 9, 8:14 PM · Discovery-Search (Current work)
EBernhardson created T220547: Document CirrusSearch schema.
Tue, Apr 9, 8:13 PM · Discovery-Search (Current work), Documentation, Cloud-Services, Elasticsearch, Discovery
EBernhardson merged T220545: [epic] Deploy replica of CirrusSearch search indices to cloud into T109715: Replicate production elasticsearch indices to labs.
Tue, Apr 9, 8:07 PM · Discovery-Search, Cloud-Services, Elasticsearch, Discovery
EBernhardson merged task T220545: [epic] Deploy replica of CirrusSearch search indices to cloud into T109715: Replicate production elasticsearch indices to labs.
Tue, Apr 9, 8:07 PM · Discovery-Search, Epic
EBernhardson created T220545: [epic] Deploy replica of CirrusSearch search indices to cloud.
Tue, Apr 9, 8:06 PM · Discovery-Search, Epic
EBernhardson edited projects for T55652: Special:Search doesn't use labels and descriptions for suggestions but just the item ID, added: Discovery-Search; removed Discovery-Search (Current work).
Tue, Apr 9, 5:22 PM · Discovery-Search, Patch-For-Review, User-Smalyshev, Wikidata, MediaWiki-extensions-WikibaseRepository
EBernhardson moved T218954: Default to article search on commons + wikibase (aka SDC) from Needs review to Done on the Discovery-Search (Current work) board.
Tue, Apr 9, 5:18 PM · MW-1.33-notes (1.33.0-wmf.24; 2019-04-02), Patch-For-Review, Discovery-Search (Current work), Wikidata
EBernhardson moved T219234: Job runner request timeouts in Elastica\Task from Needs review to Waiting/Blocked on the Discovery-Search (Current work) board.
Tue, Apr 9, 5:17 PM · Discovery-Search (Current work), CirrusSearch, MW-1.34-notes (1.34.0-wmf.3; 2019-04-30), MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Wikimedia-production-error
EBernhardson created T220519: Build presentation about using interleaved methods to evaluate search.
Tue, Apr 9, 3:17 PM · Discovery-Search (Current work)
EBernhardson moved T218945: Exception from CirrusSearch/Sanity/Checker: Cannot fetch ids from index from in progress to Done on the Discovery-Search (Current work) board.
Tue, Apr 9, 3:15 PM · MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Discovery-Search (Current work), Patch-For-Review, CirrusSearch, Wikimedia-production-error

Mon, Apr 8

EBernhardson moved T219234: Job runner request timeouts in Elastica\Task from Waiting/Blocked to Needs review on the Discovery-Search (Current work) board.
Mon, Apr 8, 6:56 PM · Discovery-Search (Current work), CirrusSearch, MW-1.34-notes (1.34.0-wmf.3; 2019-04-30), MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Wikimedia-production-error
EBernhardson added a comment to T220282: Investigate CirrusSearch as a Suggested Edits suggestions backend.
  1. Can WikibaseCirrusSearch easily be updated to support the above queries?
Mon, Apr 8, 4:50 PM · Discovery-Search, WikimediaEditorTasks, Reading-Infrastructure-Team-Backlog
EBernhardson added a comment to T219234: Job runner request timeouts in Elastica\Task.

Actually there might be a minimum delay hardcoded into the deleteByQuery code, will write up something to decay the delay from perhaps 100ms up to 5s over time

Mon, Apr 8, 4:11 PM · Discovery-Search (Current work), CirrusSearch, MW-1.34-notes (1.34.0-wmf.3; 2019-04-30), MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Wikimedia-production-error
EBernhardson added a comment to T219234: Job runner request timeouts in Elastica\Task.

Some new 1200s timeouts from job's in Task.php came up today: dhttps://logstash.wikimedia.org/goto/314bfac86ad15374a5cd8223f8867cbd

Mon, Apr 8, 4:06 PM · Discovery-Search (Current work), CirrusSearch, MW-1.34-notes (1.34.0-wmf.3; 2019-04-30), MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Wikimedia-production-error
EBernhardson added a comment to T219234: Job runner request timeouts in Elastica\Task.

If out of sync isn't a big deal, it seems the most direct and simplest way to resolve is to set conflicts=proceed and let them continue deleting instead of failing the delete-by-query.

Mon, Apr 8, 3:31 PM · Discovery-Search (Current work), CirrusSearch, MW-1.34-notes (1.34.0-wmf.3; 2019-04-30), MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Wikimedia-production-error
EBernhardson moved T219932: Convert mjolnir from KafkaRDD to direct kafka-python usage from Needs review to Done on the Discovery-Search (Current work) board.
Mon, Apr 8, 3:30 PM · Patch-For-Review, Discovery-Search (Current work)
EBernhardson moved T219268: Elasticsearch 6: the classic similarity is deprecated from Needs review to Done on the Discovery-Search (Current work) board.
Mon, Apr 8, 3:29 PM · MW-1.33-notes (1.33.0-wmf.24; 2019-04-02), Patch-For-Review, Discovery-Search (Current work), CirrusSearch
EBernhardson moved T174116: Another look at multi-hyphen tokens on enwiki and zhwiki from Needs review to Done on the Discovery-Search (Current work) board.
Mon, Apr 8, 3:28 PM · Discovery-Search (Current work), Chinese-Sites, Discovery

Fri, Apr 5

EBernhardson closed T219824: Mjolnir model builds generating models with `inf` loss as Invalid.

After further review the inf loss wasn't actually a problem, that was just hyperopt reporting before any training runs had completed.

Fri, Apr 5, 4:13 PM · Discovery-Search (Current work)
EBernhardson added a comment to T148843: Remove computational bottlenecks in stats machine via adding a GPU that can be used to train ML models.

Doesn't seem to be needed anymore, feel free to start moving this to a more production configuration.

Fri, Apr 5, 1:55 PM · Patch-For-Review, User-Elukey, Operations, Analytics, Research-management

Thu, Apr 4

EBernhardson added a comment to T148843: Remove computational bottlenecks in stats machine via adding a GPU that can be used to train ML models.

Synthetic benchmarks of runtime performance of CNN training in images/sec between CPU and WX9100. This oessentially confirms what we already know, that even a GPU that is not top of the line is an order of magnitude faster than training on cpu. Distributed training isn't a linear speedup, so it would likely take a significant portion of the hadoop cluster to achieve the same runtime performance as a single GPU. It's good to get a verification that the GPU is mostly working in this configuration. Note also that the current case can only fit a single gpu, but ideally future hardware would be purchased with the ability to fit at least 2 cards, or possibly 4 cards, in a single server.

Thu, Apr 4, 11:05 PM · Patch-For-Review, User-Elukey, Operations, Analytics, Research-management
EBernhardson added projects to T219590: Autocomplete in search box not working on mobile: MobileFrontend, MinervaNeue.
Thu, Apr 4, 7:50 PM · MinervaNeue, Mobile, Discovery-Search, CirrusSearch
EBernhardson added a comment to T148843: Remove computational bottlenecks in stats machine via adding a GPU that can be used to train ML models.

per https://github.com/RadeonOpenCompute/ROCm/issues/703#issuecomment-462598966

that means no, miopen-opencl functionality is not supported within TF.
Thu, Apr 4, 5:47 PM · Patch-For-Review, User-Elukey, Operations, Analytics, Research-management
EBernhardson added a comment to T219234: Job runner request timeouts in Elastica\Task.

Something that would also need to be investigated, queries only return documents that have been refreshed (on 5s intervals). I suspect that documents that have been written to elasticsearch but not yet refreshed would not be deleted by a delete-by-query in that timespan. At a high level reading your writes is not guaranteed, or even expected, in elasticsearch as it is eventually consistent.

Thu, Apr 4, 5:33 PM · Discovery-Search (Current work), CirrusSearch, MW-1.34-notes (1.34.0-wmf.3; 2019-04-30), MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Wikimedia-production-error
EBernhardson added a comment to T219234: Job runner request timeouts in Elastica\Task.

Well it must be this query: https://gerrit.wikimedia.org/g/mediawiki/extensions/Translate/+/51089073db26ffd69096c9d7f805c37703e55b92/ttmserver/ElasticSearchTTMServer.php#266

Editing the same translation multiple times in a succession could result this, as it essentially does delete+create. Assuming last one would win, it's safe to ignore failures.

Thu, Apr 4, 5:26 PM · Discovery-Search (Current work), CirrusSearch, MW-1.34-notes (1.34.0-wmf.3; 2019-04-30), MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Wikimedia-production-error
EBernhardson moved T220069: Build authenticating reverse proxy for Cloud CirrusSearch replicas from needs triage to making others happy on the Discovery-Search board.
Thu, Apr 4, 5:22 PM · Data-Services, Discovery-Search, Elasticsearch, Discovery
EBernhardson added a comment to T219590: Autocomplete in search box not working on mobile.

While related to search, someone from mobile frontend will likely need to take a look at this. Will leave it to them to triage the priority.

Thu, Apr 4, 5:21 PM · MinervaNeue, Mobile, Discovery-Search, CirrusSearch
EBernhardson moved T219712: When search contains "insource:", add "edit source" link to each result from needs triage to UI tickets on the Discovery-Search board.
Thu, Apr 4, 5:20 PM · Discovery-Search, MediaWiki-Special-pages, CirrusSearch
EBernhardson triaged T219712: When search contains "insource:", add "edit source" link to each result as Normal priority.

As insource in a special functionality of CirrusSearch, and the search UI is all in core, this is unfortunately not a 5-line patch. Not sure the best route to an implementation. There is also a design question of clutter in the UI that I'm not an expert on.

Thu, Apr 4, 5:19 PM · Discovery-Search, MediaWiki-Special-pages, CirrusSearch
EBernhardson triaged T219601: Create checks that alerts on cirrussearch update lags as Normal priority.
Thu, Apr 4, 5:17 PM · Discovery-Search (Current work), Patch-For-Review, Wikimedia-Incident, Operations, CirrusSearch, Elasticsearch
EBernhardson moved T219601: Create checks that alerts on cirrussearch update lags from needs triage to Current work on the Discovery-Search board.
Thu, Apr 4, 5:17 PM · Discovery-Search (Current work), Patch-For-Review, Wikimedia-Incident, Operations, CirrusSearch, Elasticsearch
EBernhardson moved T219590: Autocomplete in search box not working on mobile from needs triage to UI tickets on the Discovery-Search board.
Thu, Apr 4, 5:14 PM · MinervaNeue, Mobile, Discovery-Search, CirrusSearch
EBernhardson removed projects from T219839: Bump mediawiki core Mustache version from 1.0.0 to 3.0.1: Discovery-Search, CirrusSearch.
Thu, Apr 4, 5:09 PM · MW-1.34-notes (1.34.0-wmf.1; 2019-04-16), Patch-For-Review, Performance-Team, Readers-Web-Backlog (Tracking), MinervaNeue, RelatedArticles, MediaWiki-extensions-CentralNotice, Front-end-Standards-Group, MobileFrontend (MobileFrontend and MinervaNeue architecture)
EBernhardson added a comment to T148843: Remove computational bottlenecks in stats machine via adding a GPU that can be used to train ML models.

With the changes in packages now trying to run any model returns:

Thu, Apr 4, 4:54 PM · Patch-For-Review, User-Elukey, Operations, Analytics, Research-management
EBernhardson added a comment to T219234: Job runner request timeouts in Elastica\Task.

Still not seeing any job runner timeouts that are obviously related to this since the last once at 2019-03-25T22:52:37, should still probably leave this task around for a little while to check into this a few more times.

Thu, Apr 4, 2:57 PM · Discovery-Search (Current work), CirrusSearch, MW-1.34-notes (1.34.0-wmf.3; 2019-04-30), MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Wikimedia-production-error
EBernhardson updated subscribers of T220069: Build authenticating reverse proxy for Cloud CirrusSearch replicas.
Thu, Apr 4, 6:31 AM · Data-Services, Discovery-Search, Elasticsearch, Discovery

Wed, Apr 3

EBernhardson added a comment to T219234: Job runner request timeouts in Elastica\Task.

Not timeouts, but seeing a few delete-by-query failures in the logs now. These appear to be due to version conflicts, we can set conflicts=proceed to at least let the delete by query complete rather than abort mid-delete. This might require some input from @Nikerabbit and @abi_ regarding what is appropriate here. Basically what is happening is that during the delete-by-query operation some document that was supposed to be deleted was updated. What should happen in that case?

Wed, Apr 3, 9:37 PM · Discovery-Search (Current work), CirrusSearch, MW-1.34-notes (1.34.0-wmf.3; 2019-04-30), MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Wikimedia-production-error
EBernhardson moved T217809: inlabel / incaption keywords should highlight the label/caption from Needs review to Done on the Discovery-Search (Current work) board.
Wed, Apr 3, 9:21 PM · MW-1.34-notes (1.34.0-wmf.3; 2019-04-30), MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Patch-For-Review, Discovery-Search (Current work)
EBernhardson added a comment to T217809: inlabel / incaption keywords should highlight the label/caption.

All results to demo query now show one highlighted thing, so that is progress. Some items still don't show a snippet, but clicking through to the result page I'm not sure what could have been displayed in the snippet anyways. The initial goal seems to be complete.

Wed, Apr 3, 9:21 PM · MW-1.34-notes (1.34.0-wmf.3; 2019-04-30), MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Patch-For-Review, Discovery-Search (Current work)
EBernhardson moved T219266: Elasticsearch 6 nested_path and nested_filter are deprecated from Needs review to Done on the Discovery-Search (Current work) board.
Wed, Apr 3, 5:49 PM · MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), Patch-For-Review, Discovery-Search (Current work), GeoData