Page MenuHomePhabricator

EBernhardson (EBernhardson)
User

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Monday

  • Clear sailing ahead.

User Details

User Since
Oct 7 2014, 4:49 PM (269 w, 4 d)
Availability
Available
LDAP User
EBernhardson
MediaWiki User
EBernhardson (WMF) [ Global Accounts ]

Recent Activity

Yesterday

EBernhardson added a comment to T73405: Medium-sized image dump.

What about instead of dumps, which have the obvious difficulty of size and scope, could we dump lists of public urls? The use case here is allowing researchers to collect a set of all image thumbnails at approximately a 300 pixel width. Currently they need to go through the mediawiki api's and request thumbnails to be generated, not knowing if close enough thumbnails already exist or not.

Fri, Dec 6, 8:48 PM · Internet-Archive, Dumps-Generation, Datasets-Archiving
EBernhardson committed rLPRI47885e25b43b: secret: dummy credentials for airflow (authored by EBernhardson).
secret: dummy credentials for airflow
Fri, Dec 6, 7:12 AM
EBernhardson added a comment to T235263: Make it possible to bypass automatic redirection to exact matches in commons.

Doesn't seem to have worked? http://commons.wikimedia.org/wiki/?search=cat is redirecting to https://commons.wikimedia.org/wiki/Felis_silvestris_catus

Fri, Dec 6, 12:18 AM · MW-1.35-notes (1.35.0-wmf.8; 2019-11-26), Structured Data Engineering, Structured-Data-Backlog (Current Work), Discovery-Search (Current work), MediaWiki-Search, Discovery, Commons

Thu, Dec 5

EBernhardson committed rWDANc29a75856ff7: Deploy via scap to search-airflow dsh group (authored by EBernhardson).
Deploy via scap to search-airflow dsh group
Thu, Dec 5, 5:37 PM
EBernhardson committed rWDANb1cc28b0cdb0: Airflow for mjolnir (authored by EBernhardson).
Airflow for mjolnir
Thu, Dec 5, 5:37 PM
EBernhardson updated the task description for T239879: CirrusSearch "no such index" error from sister search.
Thu, Dec 5, 12:17 AM · Discovery-Search
EBernhardson created T239879: CirrusSearch "no such index" error from sister search.
Thu, Dec 5, 12:14 AM · Discovery-Search

Wed, Dec 4

EBernhardson claimed T238246: Add "source" to A/B test schema for DYM suggestions.
Wed, Dec 4, 7:59 PM · Discovery-Search (Current work), Patch-For-Review

Tue, Dec 3

EBernhardson added a comment to T236180: Deploy search platform airflow service.

fwiw, somebody on mysql support channel told me about "explicit_defaults_for_timestamp" that "scope is system and session.You can set it per session".

Tue, Dec 3, 7:25 PM · Patch-For-Review, Discovery-Search
EBernhardson added a comment to T222321: Make /entity/ alias work for Commons.

In summary, it seems we need to merge the patch[1] for the /entity/ endpoint, and this should be resolved?

Tue, Dec 3, 7:13 PM · Discovery-Search (Current work), Patch-For-Review, Wikimedia-Apache-configuration, WikibaseMediaInfo, Wikidata-Query-Service, SDC General, Commons, Wikidata

Mon, Dec 2

EBernhardson added a comment to T230495: Partition CirrusSearch mediawiki jobs by cluster.

Thanks! I'll keep an eye on things and see how this goes as the train rolls forward this week and we shift all the updates into these partitioned jobs.

Mon, Dec 2, 5:38 PM · MW-1.35-notes (1.35.0-wmf.8; 2019-11-26), Discovery-Search (Current work), Core Platform Team Workboards (Clinic Duty Team), Cloud-Services, Elasticsearch, Discovery
EBernhardson moved T230495: Partition CirrusSearch mediawiki jobs by cluster from Needs review to To Be Deployed on the Discovery-Search (Current work) board.
Mon, Dec 2, 5:36 PM · MW-1.35-notes (1.35.0-wmf.8; 2019-11-26), Discovery-Search (Current work), Core Platform Team Workboards (Clinic Duty Team), Cloud-Services, Elasticsearch, Discovery
EBernhardson added a comment to T236180: Deploy search platform airflow service.

Apologies for the delay, I didn't see the ping :(
I added the grants for the IPv6 IP, was only for IPv4 before. Can you retry and see if it works?

Mon, Dec 2, 4:47 PM · Patch-For-Review, Discovery-Search

Wed, Nov 27

EBernhardson added a comment to T236180: Deploy search platform airflow service.

After the above deployments things are looking mostly in order, but stuck on a mariadb access error:

Wed, Nov 27, 7:50 PM · Patch-For-Review, Discovery-Search

Tue, Nov 26

jcrespo awarded T237559: wfEscapeWikiText() emits error "PHP Notice: Array to string conversion" on Special:Search a Like token.
Tue, Nov 26, 5:55 PM · MW-1.35-notes (1.35.0-wmf.10; 2019-12-10), Discovery-Search (Current work), Wikimedia-production-error, affects-translatewiki.net, MediaWiki-Search
EBernhardson added a comment to T237559: wfEscapeWikiText() emits error "PHP Notice: Array to string conversion" on Special:Search.

I am seeing ATM errors:

/wiki/Special:Search?search=<search string>&ns0=1   ErrorException from line 1591 of /srv/mediawiki/php-1.35.0-wmf.5/includes/GlobalFunctions.php: PHP Notice: Array to string conversion

Such as: https://logstash.wikimedia.org/goto/ba65dd4317ecfef44eac4372c9c13a62
Could you confirm they are the same origin, and will be fixed after wmf8 gets deployed, and not different kind of errors with the same error message?

Tue, Nov 26, 5:42 PM · MW-1.35-notes (1.35.0-wmf.10; 2019-12-10), Discovery-Search (Current work), Wikimedia-production-error, affects-translatewiki.net, MediaWiki-Search
EBernhardson added a comment to T237605: Create kerberos principals for users.

discovery-analytics: We run analytics jobs (submit to oozie, etc) from this user. The upcoming airflow installation will also submit our jobs as this user. I suppose send this to my email as well? Otherwise maybe discovery-private@lists.wikimedia.org would work, but that is far from private and subscribed by many across the org.

I am aware of the analytics-search system user, and for that I have already created a keytab (basically a file with user+password that only the owner can read) on stat1007 (and all systemd timers will use it as soon as we enable kerberos, no action needed from your side). Is it the user that you mentioned? If so, I think that we'd just need to create a keytab for an-airflow1001 right? (the keytabs are host-specific).

Tue, Nov 26, 4:07 PM · Analytics-Kanban, Analytics

Mon, Nov 25

EBernhardson added a comment to T237605: Create kerberos principals for users.

I'd like to request two credentials for hadoop access:

Mon, Nov 25, 8:08 PM · Analytics-Kanban, Analytics
EBernhardson added a comment to T237363: Undeploy Glent M0 A/B test.

This is still waiting for a train deploy before rolling the remaining config patches out.

Mon, Nov 25, 6:03 PM · MW-1.35-notes (1.35.0-wmf.8; 2019-11-26), Patch-For-Review, Discovery-Search (Current work)
EBernhardson moved T197129: Increase sampling rates for search metrics on smaller language wikis from Needs review to Done on the Discovery-Search (Current work) board.
Mon, Nov 25, 6:03 PM · MW-1.35-notes (1.35.0-wmf.8; 2019-11-26), Product-Analytics, MW-1.34-notes (1.34.0-wmf.15; 2019-07-23), Patch-For-Review, Discovery-Search (Current work)
EBernhardson added a comment to T239002: Regex search in labels, descriptions and statements.

labels and descriptions are covered with the existing insource:// functionality, statements might be considered but wikidata already has indices that are almost too big to manage, and use more fields than elasticsearch supports. It's not likely we will be adding many more indexing methods to wikidata unless we end up buying a wikidata search cluster.

Mon, Nov 25, 5:25 PM · Wikidata, Discovery-Search
EBernhardson moved T239000: Deploying GeoData extension in the Russian Wikinews from In Progress to Waiting on the Discovery-Search (Current work) board.
Mon, Nov 25, 5:19 PM · Discovery-Search (Current work), User-Zoranzoki21, GeoData, Wikimedia-Site-requests
EBernhardson moved T239000: Deploying GeoData extension in the Russian Wikinews from needs triage to Current work on the Discovery-Search board.
Mon, Nov 25, 5:19 PM · Discovery-Search (Current work), User-Zoranzoki21, GeoData, Wikimedia-Site-requests
EBernhardson triaged T239003: New keyword for exact match of label/description as Medium priority.
Mon, Nov 25, 5:16 PM · Wikidata, Discovery-Search
EBernhardson moved T239003: New keyword for exact match of label/description from needs triage to Wikidata Search on the Discovery-Search board.
Mon, Nov 25, 5:16 PM · Wikidata, Discovery-Search
EBernhardson triaged T239002: Regex search in labels, descriptions and statements as Low priority.
Mon, Nov 25, 5:16 PM · Wikidata, Discovery-Search
EBernhardson moved T239002: Regex search in labels, descriptions and statements from needs triage to Wikidata Search on the Discovery-Search board.
Mon, Nov 25, 5:16 PM · Wikidata, Discovery-Search
EBernhardson added a comment to T239004: Search return incorrect number of results.

Looks like it's hitting this: https://gerrit.wikimedia.org/r/c/search/highlighter/+/435282/5/experimental-highlighter-lucene/src/main/java/org/wikimedia/highlighter/experimental/lucene/hit/AutomatonHitEnum.java#189

Mon, Nov 25, 5:11 PM · Discovery-Search, Wikidata
EBernhardson moved T237559: wfEscapeWikiText() emits error "PHP Notice: Array to string conversion" on Special:Search from Needs review to Done on the Discovery-Search (Current work) board.
Mon, Nov 25, 5:11 PM · MW-1.35-notes (1.35.0-wmf.10; 2019-12-10), Discovery-Search (Current work), Wikimedia-production-error, affects-translatewiki.net, MediaWiki-Search
EBernhardson moved T237606: Regex searches no longer display a timeout message when showing partial results from Needs review to Done on the Discovery-Search (Current work) board.
Mon, Nov 25, 5:11 PM · MW-1.35-notes (1.35.0-wmf.8; 2019-11-26), Discovery-Search (Current work), CirrusSearch
EBernhardson added a comment to T239004: Search return incorrect number of results.

Hmm, the request with only 222 items is interesting, will look at that

Mon, Nov 25, 4:48 PM · Discovery-Search, Wikidata
EBernhardson added a comment to T239004: Search return incorrect number of results.

While it doesn't say timeout, the request spins for some time and eventually the backend gives up and reports a failure. As i said the error messages shown to users could be improved, but the request would still fail.

Mon, Nov 25, 4:47 PM · Discovery-Search, Wikidata
EBernhardson added a comment to T239004: Search return incorrect number of results.

The request basically asks to run a regex against 76M titles and fails with timeouts. While the error messages could be improved, this is such a niche thing that I don't think it's particularly important. This happens to fail the timeout in a different way than other things, but the only improvement i would probably be able to offer is to replace the whole search results pages with a message that effectively says "dont do that"

Mon, Nov 25, 4:39 PM · Discovery-Search, Wikidata

Fri, Nov 22

EBernhardson committed rECIR462c1ed50552: Don't pass arrays into Status i18n messages (authored by EBernhardson).
Don't pass arrays into Status i18n messages
Fri, Nov 22, 9:00 AM
EBernhardson committed rECIRf37102f1ffae: Attach query timeout messages to search context (authored by EBernhardson).
Attach query timeout messages to search context
Fri, Nov 22, 9:00 AM
EBernhardson triaged T238883: wgCirrusSearchClusters should not merge arrays as Medium priority.
Fri, Nov 22, 12:19 AM · Discovery-Search, CirrusSearch
EBernhardson moved T238883: wgCirrusSearchClusters should not merge arrays from needs triage to elastic / cirrus on the Discovery-Search board.
Fri, Nov 22, 12:18 AM · Discovery-Search, CirrusSearch
EBernhardson moved T237606: Regex searches no longer display a timeout message when showing partial results from In Progress to Needs review on the Discovery-Search (Current work) board.
Fri, Nov 22, 12:15 AM · MW-1.35-notes (1.35.0-wmf.8; 2019-11-26), Discovery-Search (Current work), CirrusSearch
EBernhardson claimed T237606: Regex searches no longer display a timeout message when showing partial results.
Fri, Nov 22, 12:15 AM · MW-1.35-notes (1.35.0-wmf.8; 2019-11-26), Discovery-Search (Current work), CirrusSearch
EBernhardson moved T237559: wfEscapeWikiText() emits error "PHP Notice: Array to string conversion" on Special:Search from In Progress to Needs review on the Discovery-Search (Current work) board.
Fri, Nov 22, 12:03 AM · MW-1.35-notes (1.35.0-wmf.10; 2019-12-10), Discovery-Search (Current work), Wikimedia-production-error, affects-translatewiki.net, MediaWiki-Search
EBernhardson moved T237606: Regex searches no longer display a timeout message when showing partial results from Needs review to In Progress on the Discovery-Search (Current work) board.
Fri, Nov 22, 12:03 AM · MW-1.35-notes (1.35.0-wmf.8; 2019-11-26), Discovery-Search (Current work), CirrusSearch
EBernhardson moved T237606: Regex searches no longer display a timeout message when showing partial results from In Progress to Needs review on the Discovery-Search (Current work) board.
Fri, Nov 22, 12:03 AM · MW-1.35-notes (1.35.0-wmf.8; 2019-11-26), Discovery-Search (Current work), CirrusSearch
EBernhardson moved T237560: Failed executing job: cirrusSearchElasticaWrite [...] from In Progress to Done on the Discovery-Search (Current work) board.
Fri, Nov 22, 12:03 AM · Discovery-Search (Current work), CirrusSearch

Thu, Nov 21

EBernhardson created T238883: wgCirrusSearchClusters should not merge arrays.
Thu, Nov 21, 11:40 PM · Discovery-Search, CirrusSearch
EBernhardson moved T238802: Remove $wgCirrusSearchEnableSearchLogging from In Progress to Done on the Discovery-Search (Current work) board.
Thu, Nov 21, 11:15 PM · Discovery-Search (Current work)
EBernhardson claimed T238802: Remove $wgCirrusSearchEnableSearchLogging.
Thu, Nov 21, 11:14 PM · Discovery-Search (Current work)
EBernhardson moved T237332: Regex in CirrusSearch can't find Anatolian Hieroglyphs from needs triage to elastic / cirrus on the Discovery-Search board.
Thu, Nov 21, 10:04 PM · Discovery-Search, CirrusSearch
EBernhardson triaged T237332: Regex in CirrusSearch can't find Anatolian Hieroglyphs as Medium priority.
Thu, Nov 21, 10:04 PM · Discovery-Search, CirrusSearch
EBernhardson moved T236588: Include subpages in "mwgrep" from needs triage to making others happy on the Discovery-Search board.
Thu, Nov 21, 10:03 PM · Discovery-Search, Performance-Team (Radar)
EBernhardson triaged T237519: Search starting in small-caps in Wiktionary as Medium priority.
Thu, Nov 21, 10:03 PM · MediaWiki-Search, MediaWiki-Interface, Discovery-Search, Wiktionary
EBernhardson moved T237519: Search starting in small-caps in Wiktionary from needs triage to UI tickets on the Discovery-Search board.
Thu, Nov 21, 10:03 PM · MediaWiki-Search, MediaWiki-Interface, Discovery-Search, Wiktionary
EBernhardson added a comment to T237560: Failed executing job: cirrusSearchElasticaWrite [...].

The error message is: Couldn't connect to host, Elasticsearch down?

Thu, Nov 21, 10:01 PM · Discovery-Search (Current work), CirrusSearch
EBernhardson moved T237268: Add the ability to search in contributions of a specific user from needs triage to later on... on the Discovery-Search board.
Thu, Nov 21, 9:26 PM · MediaWiki-Special-pages, Discovery-Search, Discovery
EBernhardson removed a project from T237268: Add the ability to search in contributions of a specific user: CirrusSearch.

While perhaps interesting, I can't see this fitting into CirrusSearch. Dealing with the entire history of revisions is outside it's scope.

Thu, Nov 21, 9:25 PM · MediaWiki-Special-pages, Discovery-Search, Discovery
EBernhardson moved T113840: Offer a titles-only search result from needs triage to UI tickets on the Discovery-Search board.
Thu, Nov 21, 9:20 PM · Discovery-Search, CirrusSearch, Discovery
EBernhardson claimed T237559: wfEscapeWikiText() emits error "PHP Notice: Array to string conversion" on Special:Search.
Thu, Nov 21, 9:13 PM · MW-1.35-notes (1.35.0-wmf.10; 2019-12-10), Discovery-Search (Current work), Wikimedia-production-error, affects-translatewiki.net, MediaWiki-Search
EBernhardson moved T237559: wfEscapeWikiText() emits error "PHP Notice: Array to string conversion" on Special:Search from needs triage to Current work on the Discovery-Search board.
Thu, Nov 21, 9:13 PM · MW-1.35-notes (1.35.0-wmf.10; 2019-12-10), Discovery-Search (Current work), Wikimedia-production-error, affects-translatewiki.net, MediaWiki-Search
EBernhardson triaged T237559: wfEscapeWikiText() emits error "PHP Notice: Array to string conversion" on Special:Search as Medium priority.
Thu, Nov 21, 9:13 PM · MW-1.35-notes (1.35.0-wmf.10; 2019-12-10), Discovery-Search (Current work), Wikimedia-production-error, affects-translatewiki.net, MediaWiki-Search
EBernhardson moved T237560: Failed executing job: cirrusSearchElasticaWrite [...] from needs triage to Current work on the Discovery-Search board.
Thu, Nov 21, 8:21 PM · Discovery-Search (Current work), CirrusSearch
EBernhardson claimed T237560: Failed executing job: cirrusSearchElasticaWrite [...].
Thu, Nov 21, 8:21 PM · Discovery-Search (Current work), CirrusSearch
EBernhardson added a comment to T237560: Failed executing job: cirrusSearchElasticaWrite [...].

For whatever reason beta cluster isn't logging any of the CirrusSearch channels that production does, so there is no information about what actually went wrong. Will need to fix beta cluster logging first.

Thu, Nov 21, 8:21 PM · Discovery-Search (Current work), CirrusSearch
EBernhardson moved T238130: Create a ES wildcard/prefix/fuzzy query that supports normalization and max_determinized_states (extra plugin) from needs triage to elastic / cirrus on the Discovery-Search board.
Thu, Nov 21, 8:09 PM · Discovery-Search, CirrusSearch
EBernhardson triaged T238130: Create a ES wildcard/prefix/fuzzy query that supports normalization and max_determinized_states (extra plugin) as Medium priority.
Thu, Nov 21, 8:09 PM · Discovery-Search, CirrusSearch
EBernhardson moved T237606: Regex searches no longer display a timeout message when showing partial results from needs triage to Current work on the Discovery-Search board.
Thu, Nov 21, 8:08 PM · MW-1.35-notes (1.35.0-wmf.8; 2019-11-26), Discovery-Search (Current work), CirrusSearch
EBernhardson triaged T237606: Regex searches no longer display a timeout message when showing partial results as High priority.
Thu, Nov 21, 8:08 PM · MW-1.35-notes (1.35.0-wmf.8; 2019-11-26), Discovery-Search (Current work), CirrusSearch
EBernhardson moved T238004: Implement sonarcloud integration for Java projects in the same way as PHP projects from needs triage to Ops / SRE on the Discovery-Search board.
Thu, Nov 21, 8:07 PM · Code-Health, Discovery-Search
EBernhardson triaged T238004: Implement sonarcloud integration for Java projects in the same way as PHP projects as Medium priority.
Thu, Nov 21, 8:07 PM · Code-Health, Discovery-Search
EBernhardson moved T238131: Cleanup: assume that the all field is always indexed from needs triage to elastic / cirrus on the Discovery-Search board.
Thu, Nov 21, 8:07 PM · CirrusSearch, Discovery-Search
EBernhardson triaged T238131: Cleanup: assume that the all field is always indexed as Medium priority.
Thu, Nov 21, 8:07 PM · CirrusSearch, Discovery-Search
EBernhardson moved T238166: Have Maryum learn a little bit about language analysis with Trey from needs triage to Current work on the Discovery-Search board.
Thu, Nov 21, 8:06 PM · Discovery-Search (Current work)
EBernhardson triaged T238166: Have Maryum learn a little bit about language analysis with Trey as Medium priority.
Thu, Nov 21, 8:06 PM · Discovery-Search (Current work)
EBernhardson triaged T238362: Blazegraph write performance tuning as Medium priority.
Thu, Nov 21, 8:06 PM · Wikidata-Query-Service, Wikidata
EBernhardson moved T238362: Blazegraph write performance tuning from needs triage to Current work on the Discovery-Search board.
Thu, Nov 21, 8:05 PM · Wikidata-Query-Service, Wikidata
EBernhardson moved T238498: index date statements from needs triage to Wikidata Search on the Discovery-Search board.
Thu, Nov 21, 8:05 PM · Discovery-Search, Wikidata
EBernhardson triaged T238498: index date statements as Medium priority.
Thu, Nov 21, 8:05 PM · Discovery-Search, Wikidata
EBernhardson moved T238802: Remove $wgCirrusSearchEnableSearchLogging from needs triage to Current work on the Discovery-Search board.
Thu, Nov 21, 8:03 PM · Discovery-Search (Current work)
EBernhardson triaged T238802: Remove $wgCirrusSearchEnableSearchLogging as Medium priority.
Thu, Nov 21, 8:03 PM · Discovery-Search (Current work)

Wed, Nov 20

EBernhardson committed rWDAN78f76e9b0dba: Drop earlist legal ts from glent m0prep (authored by EBernhardson).
Drop earlist legal ts from glent m0prep
Wed, Nov 20, 8:09 PM
EBernhardson committed rECIR807114b1b069: [integ] Wait for document to have expected incoming link count before searching (authored by EBernhardson).
[integ] Wait for document to have expected incoming link count before searching
Wed, Nov 20, 9:30 AM

Tue, Nov 19

EBernhardson moved T238703: Investigate applying PDGD to our datasets from In Progress to Done on the Discovery-Search (Current work) board.
Tue, Nov 19, 11:33 PM · Discovery-Search (Current work)
EBernhardson claimed T238703: Investigate applying PDGD to our datasets.
Tue, Nov 19, 11:33 PM · Discovery-Search (Current work)
EBernhardson added a comment to T238703: Investigate applying PDGD to our datasets.

Should have created this task before, did most of the investigation last week. I applied this technique both with a linear model and a neural model to one of our frwiki folds. This contains ~140k queries to train against and another 40k for evaluation. Our standard xgboost models trained over this dataset achieved ndcg@10 of 0.888. For reference the labels used here were estimated with a DBN click model and required seeing the same query at least 5 times within the prior 12 weeks.

Tue, Nov 19, 11:33 PM · Discovery-Search (Current work)
EBernhardson created T238703: Investigate applying PDGD to our datasets.
Tue, Nov 19, 11:07 PM · Discovery-Search (Current work)
EBernhardson added a comment to T223046: Lack of case sensitivity with hastemplate:.

Old servers have been replaced with new servers now, should be able to unblock this.

Tue, Nov 19, 11:01 PM · MediaWiki-Search, Discovery-Search (Current work)
EBernhardson added a comment to T233403: Unassigned shards in eqiad.

shard allocation looks pretty happy since we pooled the new servers and depooled the old ones. I'd be willing to call this complete for now, re-open if it's an issue later.

Tue, Nov 19, 10:50 PM · Discovery-Search (Current work), Operations, Elasticsearch
EBernhardson moved T236186: Move bulk content out of the ElasticaWrite job from Needs review to Done on the Discovery-Search (Current work) board.
Tue, Nov 19, 10:49 PM · MW-1.35-notes (1.35.0-wmf.8; 2019-11-26), Discovery-Search (Current work), Core Platform Team Workboards (Clinic Duty Team), Cloud-Services, Elasticsearch, Discovery
EBernhardson added a comment to T237550: Increase Glent M0 retention.

I've deployed the new glent jar along with an updated oozie workflow.xml, going forward m0prep should retain search session based suggestions indefinitely.

Tue, Nov 19, 10:46 PM · Discovery-Search (Current work)
EBernhardson moved T237550: Increase Glent M0 retention from Needs review to Done on the Discovery-Search (Current work) board.
Tue, Nov 19, 10:45 PM · Discovery-Search (Current work)
EBernhardson added a comment to T238686: Deepcat search returns incomplete results.

The SPARQL query endpoint that provides the categories to search against doesn't appear to be returning all expected sub-categories.:

ebernhardson@mwmaint1002:~$ curl -s -XPOST http://wdqs-internal.discovery.wmnet/bigdata/namespace/categories/sparql?format=json -d 'query=SELECT ?out WHERE {
      SERVICE mediawiki:categoryTree {
          bd:serviceParam mediawiki:start <https://en.wikipedia.org/wiki/Category:Musicals_by_topic> .
          bd:serviceParam mediawiki:direction "Reverse" .
          bd:serviceParam mediawiki:depth 5 .
      }
} ORDER BY ASC(?depth)
LIMIT 50' | jq '.results.bindings | map(.out.value)'
[
  "https://en.wikipedia.org/wiki/Category:Musicals_by_topic",
  "https://en.wikipedia.org/wiki/Category:Musicals_about_writers",
  "https://en.wikipedia.org/wiki/Category:Musicals_about_World_War_II",
  "https://en.wikipedia.org/wiki/Category:Musicals_set_in_the_Roaring_Twenties",
  "https://en.wikipedia.org/wiki/Category:Plays_and_musicals_about_disability",
  "https://en.wikipedia.org/wiki/Category:Musicals_about_World_War_I",
  "https://en.wikipedia.org/wiki/Category:Musicals_about_the_Great_Depression"
]
Tue, Nov 19, 9:47 PM · Wikidata, Wikidata-Query-Service, Discovery-Search
EBernhardson added a comment to T230495: Partition CirrusSearch mediawiki jobs by cluster.

@Pchelolo This patch will rollout with the next train (early dec?). What will happen at that point is:

Tue, Nov 19, 6:36 PM · MW-1.35-notes (1.35.0-wmf.8; 2019-11-26), Discovery-Search (Current work), Core Platform Team Workboards (Clinic Duty Team), Cloud-Services, Elasticsearch, Discovery

Thu, Nov 14

EBernhardson committed rECIR618a53c8c51a: Enqueue a job per cluster to write to (authored by EBernhardson).
Enqueue a job per cluster to write to
Thu, Nov 14, 5:53 PM
EBernhardson moved T237849: Commons search seems to have stopped indexing statements since 30 October 2019 from Waiting to Done on the Discovery-Search (Current work) board.
Thu, Nov 14, 4:45 PM · MW-1.35-notes (1.35.0-wmf.5; 2019-11-05), Discovery-Search (Current work), Commons, CirrusSearch, SDC-Statements, Structured-Data-Backlog
EBernhardson added a comment to T237849: Commons search seems to have stopped indexing statements since 30 October 2019.

Backfill has completed, this should be resolved.

Thu, Nov 14, 4:45 PM · MW-1.35-notes (1.35.0-wmf.5; 2019-11-05), Discovery-Search (Current work), Commons, CirrusSearch, SDC-Statements, Structured-Data-Backlog
EBernhardson committed rWDAN4cf0720dcbc2: Update pyspark to 2.4.4 (authored by EBernhardson).
Update pyspark to 2.4.4
Thu, Nov 14, 1:36 PM
EBernhardson committed rECIR21af6448ec59: Restore CirrusSearchBuildDocumentParse hook (authored by EBernhardson).
Restore CirrusSearchBuildDocumentParse hook
Thu, Nov 14, 12:34 AM

Wed, Nov 13

EBernhardson moved T237849: Commons search seems to have stopped indexing statements since 30 October 2019 from Needs review to Waiting on the Discovery-Search (Current work) board.
Wed, Nov 13, 11:19 PM · MW-1.35-notes (1.35.0-wmf.5; 2019-11-05), Discovery-Search (Current work), Commons, CirrusSearch, SDC-Statements, Structured-Data-Backlog
EBernhardson closed T130329: Icinga should alert on free disk space < 15% (now < 12%) on Elasticsearch hosts as Resolved.

These servers (elastic1017-31) no longer have any data on them and are being decomissioned. The elasticsearch servers are now all under 50% disk used

Wed, Nov 13, 8:26 PM · Discovery-Search (Current work), Patch-For-Review, Operations, Discovery, Elasticsearch
EBernhardson committed rECIRc3178010382e: Restore CirrusSearchBuildDocumentParse hook (authored by EBernhardson).
Restore CirrusSearchBuildDocumentParse hook
Wed, Nov 13, 4:49 PM
EBernhardson moved T237849: Commons search seems to have stopped indexing statements since 30 October 2019 from In Progress to Needs review on the Discovery-Search (Current work) board.
Wed, Nov 13, 4:30 PM · MW-1.35-notes (1.35.0-wmf.5; 2019-11-05), Discovery-Search (Current work), Commons, CirrusSearch, SDC-Statements, Structured-Data-Backlog
EBernhardson created P9622 pdgd frwiki 20191029 v5.
Wed, Nov 13, 4:16 PM