Page MenuHomePhabricator

dcausse (David Causse)
User

Projects

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Friday

  • Clear sailing ahead.

User Details

User Since
Jun 9 2015, 9:03 AM (362 w, 1 d)
Availability
Available
IRC Nick
dcausse
LDAP User
DCausse
MediaWiki User
DCausse (WMF) [ Global Accounts ]

Recent Activity

Today

dcausse changed the status of T271777: Bump rufin/elastica (and related libraries) to versions that support PHP 8.0 from Stalled to Open.

Marking as Open as this should be resolved "soon" as we plan to ship ruflin/Elastica 7.1.5 & elasticsearch/elasticsearch 7.11.0 as part of https://gerrit.wikimedia.org/r/c/mediawiki/vendor/+/791634 in the coming weeks.

Wed, May 18, 6:42 PM · MW-1.38-notes (1.38.0-wmf.16; 2022-01-03), CirrusSearch, Upstream, Discovery-Search, PHP 8.0 support
dcausse changed the status of T271777: Bump rufin/elastica (and related libraries) to versions that support PHP 8.0, a subtask of T268861: CirrusSearch uses Elastica's Match class, from Stalled to Open.
Wed, May 18, 6:41 PM · MW-1.37-notes (1.37.0-wmf.20; 2021-08-23), Patch-For-Review, Discovery-Search, Upstream, PHP 8.0 support, CirrusSearch
dcausse changed the status of T271777: Bump rufin/elastica (and related libraries) to versions that support PHP 8.0, a subtask of T268863: Translate uses Elastica's Match class, from Stalled to Open.
Wed, May 18, 6:41 PM · MW-1.37-notes (1.37.0-wmf.20; 2021-08-23), PHP 8.0 support, MediaWiki-extensions-Translate
dcausse changed the status of T271777: Bump rufin/elastica (and related libraries) to versions that support PHP 8.0, a subtask of T268864: WikibaseCirrusSearch uses Elastica's Match class, from Stalled to Open.
Wed, May 18, 6:41 PM · wdwb-tech, PHP 8.0 support, CirrusSearch, Wikidata, Discovery-Search
dcausse changed the status of T271777: Bump rufin/elastica (and related libraries) to versions that support PHP 8.0, a subtask of T268865: WikibaseLexemeCirrusSearch uses Elastica's Match class, from Stalled to Open.
Wed, May 18, 6:41 PM · MW-1.37-notes (1.37.0-wmf.20; 2021-08-23), wdwb-tech, Discovery-Search, Wikidata, PHP 8.0 support, CirrusSearch, Wikidata Lexicographical data
dcausse changed the status of T271777: Bump rufin/elastica (and related libraries) to versions that support PHP 8.0, a subtask of T268866: WikibaseMediaInfo uses Elastica's Match class, from Stalled to Open.
Wed, May 18, 6:41 PM · Structured-Data-Backlog, PHP 8.0 support, CirrusSearch, Discovery-Search, WikibaseMediaInfo
dcausse changed the status of T271777: Bump rufin/elastica (and related libraries) to versions that support PHP 8.0, a subtask of T283275: Make MW master tests pass on PHP 8.0, from Stalled to Open.
Wed, May 18, 6:41 PM · MW-1.37-notes, MW-1.35-notes, MW-1.36-notes, MW-1.38-notes (1.38.0-wmf.19; 2022-01-24), PHP 8.0 support, MediaWiki-General
dcausse updated the task description for T308676: Elasticsearch 7.10.2 rollout plan.
Wed, May 18, 2:27 PM · Discovery-Search, CirrusSearch
dcausse moved T301959: Upgrade Search elasticsearch cluster / eqiad to elasticsearch 6.8.23 from Ready for Development to Needs Reporting on the Discovery-Search (Current work) board.
Wed, May 18, 2:26 PM · Discovery-Search (Current work)
dcausse added a comment to T301959: Upgrade Search elasticsearch cluster / eqiad to elasticsearch 6.8.23.

@Jdforrester-WMF indeed! I wish I had noticed your comment before! Reedy's patch on vendor was merged so I think we're good now :)

Wed, May 18, 2:26 PM · Discovery-Search (Current work)
dcausse closed T218995: re-enable deprecation warning logger on elasticsearch once issues are solved as Resolved.
Wed, May 18, 2:23 PM · CirrusSearch, Discovery-Search, SRE
dcausse closed T218995: re-enable deprecation warning logger on elasticsearch once issues are solved, a subtask of T218994: Deprecation warning on elasticsearch 6 , as Resolved.
Wed, May 18, 2:23 PM · Patch-For-Review, Discovery-Search (Current work), CirrusSearch
dcausse added a subtask for T263142: [EPIC] Upgrade Elasticsearch to version 7.10: T308676: Elasticsearch 7.10.2 rollout plan.
Wed, May 18, 2:18 PM · MW-1.39-notes (1.39.0-wmf.8; 2022-04-18), Discovery-Search (Current work), Epic
dcausse added a parent task for T308676: Elasticsearch 7.10.2 rollout plan: T263142: [EPIC] Upgrade Elasticsearch to version 7.10.
Wed, May 18, 2:18 PM · Discovery-Search, CirrusSearch
dcausse created T308676: Elasticsearch 7.10.2 rollout plan.
Wed, May 18, 2:17 PM · Discovery-Search, CirrusSearch
dcausse renamed T308647: elastic2054 is having H/W issues from elasticsearch2054 is having H/W issues to elastic2054 is having H/W issues.
Wed, May 18, 9:36 AM · SRE, ops-codfw, CirrusSearch, DC-Ops, Discovery-Search
dcausse added a project to T308647: elastic2054 is having H/W issues: ops-codfw.
Wed, May 18, 9:36 AM · SRE, ops-codfw, CirrusSearch, DC-Ops, Discovery-Search
dcausse created T308647: elastic2054 is having H/W issues.
Wed, May 18, 9:30 AM · SRE, ops-codfw, CirrusSearch, DC-Ops, Discovery-Search
dcausse claimed T308645: CirrusSearch should include GeoData in its phan analysis.
Wed, May 18, 9:23 AM · Discovery-Search (Current work), CirrusSearch
dcausse edited projects for T308645: CirrusSearch should include GeoData in its phan analysis, added: Discovery-Search (Current work); removed Discovery-Search.
Wed, May 18, 9:22 AM · Discovery-Search (Current work), CirrusSearch
dcausse created T308645: CirrusSearch should include GeoData in its phan analysis.
Wed, May 18, 9:17 AM · Discovery-Search (Current work), CirrusSearch
dcausse added a comment to T308640: Error: Call to undefined method CirrusSearch\Connection::getPageType().

Cause is https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CirrusSearch/+/790734 removing this function but this patch should have had https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GeoData/+/790732 tagged as Depends-On.
We might have avoided this mistake by including GeoData in CirrusSearch phan analysis. I'll file a task to change this.

Wed, May 18, 9:07 AM · Discovery-Search, GeoData, Wikimedia-production-error

Mon, May 16

dcausse renamed T308044: Remove reference to Elastica\Type from CirrusSearch and related extensions and upgrade to Elastica 7.1.5 from Remove reference to Elastica\Type from CirrusSearch and related extensions to Remove reference to Elastica\Type from CirrusSearch and related extensions and upgrade to Elastica 7.1.5.
Mon, May 16, 8:38 AM · MW-1.39-notes (1.39.0-wmf.12; 2022-05-16), Discovery-Search (Current work), Patch-For-Review, CirrusSearch

Thu, May 12

dcausse added a comment to T307862: Search on betacommons is not indexing anything.

https://commons.wikimedia.beta.wmflabs.org/wiki/File:Jason_Shaw_-_Big_Car_Theft.ogg?action=cirrusDump shows it being indexed and seems to appear in search results now, might just that beta is bit slow to index pages.

Thu, May 12, 12:19 PM · Discovery-Search (Current work), Beta-Cluster-reproducible, Beta-Cluster-Infrastructure, MediaWiki-Core-JobQueue, CirrusSearch

Tue, May 10

dcausse created T308044: Remove reference to Elastica\Type from CirrusSearch and related extensions and upgrade to Elastica 7.1.5.
Tue, May 10, 5:24 PM · MW-1.39-notes (1.39.0-wmf.12; 2022-05-16), Discovery-Search (Current work), Patch-For-Review, CirrusSearch
dcausse added a comment to T307959: [Shared Event Platform] Design and Implement POC Flink Service to Combine Existing Streams, Enrich and Output to New Topic.

any others? Undelete?

Probably. Let's ask @dcausse, @RBrounley_WMF and @Protsack.stephan

Likely:

  • mediawiki.revision-visibility-change
  • mediawiki.page-undelete
  • mediawiki.page-suppress
  • mediawiki.page-move
Tue, May 10, 1:16 PM · Epic, Generated Data Platform

Mon, May 9

dcausse moved T306168: Better classification of CirrusSearch errors from Ready for Development to In Progress on the Discovery-Search (Current work) board.
Mon, May 9, 3:19 PM · MW-1.39-notes (1.39.0-wmf.12; 2022-05-16), Discovery-Search (Current work), CirrusSearch
dcausse moved T288764: Set include_type_name in all get requests from To Be Deployed to Needs Reporting on the Discovery-Search (Current work) board.
Mon, May 9, 3:13 PM · Patch-For-Review, Discovery-Search (Current work), CirrusSearch
dcausse moved T288765: Always provide minimum_should_match in bool queries from To Be Deployed to Needs Reporting on the Discovery-Search (Current work) board.
Mon, May 9, 3:13 PM · MW-1.39-notes (1.39.0-wmf.12; 2022-05-16), Patch-For-Review, Discovery-Search (Current work), CirrusSearch
dcausse moved T209859: Wikidata autocomplete (wbsearchentities) results with score <= 0 from To Be Deployed to Needs Reporting on the Discovery-Search (Current work) board.
Mon, May 9, 3:13 PM · MW-1.39-notes (1.39.0-wmf.9; 2022-04-25), Discovery-Search (Current work), Wikidata
dcausse added a comment to T307635: Query service results are missing some variables on some servers.

This is extremely weird and I suspect a serious blazegraph bug that causes this. I could not reproduce the problem at the moment running the python script provided but it might certainly happen again in the future.
I'm not sure how to proceed here but perhaps capturing the full blazegraph response when it occurs might help?

Mon, May 9, 8:25 AM · Discovery-Search (Current work), Wikidata, Wikidata-Query-Service

Tue, May 3

dcausse added a comment to T306798: Investigate using Flink as an Event Service Platform.

@dcausse are the 'schema changes' you are talking about just for Flink managed state?

Tue, May 3, 4:27 PM · Spike, Generated Data Platform
dcausse added a comment to T306798: Investigate using Flink as an Event Service Platform.

I've experimented with jsonschema2pojo to auto generate Java POJOs from our JsonSchemas, but the way we do $refs and materialized schemas makes it a little strange (every schema that uses the meta fragment makes a new Meta class. Maybe there's a way around this, but it would almost certainly require code changes in jsonschema2pojo.

My question then is...does Flink handle case class serialization well now? From the docs it seems to be yes? I recall in the past that case classes weren't actually well supported. @dcausse do you know?

Tue, May 3, 10:42 AM · Spike, Generated Data Platform

Mon, May 2

dcausse moved T305689: Elasticsearch chi@eqiad cluster contains invalid cross cluster settings from In Progress to Needs Reporting on the Discovery-Search (Current work) board.

I could not reproduce such duplicated settings even when doing the following updates: 5.5 -> 6.3 -> 6.3.1 -> 6.4.2 -> 6.5.4.
Testing using the node state taken from the master node (e.g. elastic1054.eqiad.wmnet:/srv/elasticsearch/production-search-eqiad/nodes/0/_state/) I was able to boot elasticsearch v7.10.2 locally without errors, the duplicated settings disappeared and I could update them properly.
I'm going to assume that this won't cause any problem for us and will resolve itself by upgrading to 7.10.

Mon, May 2, 2:12 PM · Discovery-Search (Current work), CirrusSearch

Fri, Apr 29

dcausse committed rEWLC01e71ae523f6: Do not rely on existing translations during tests (authored by dcausse).
Do not rely on existing translations during tests
Fri, Apr 29, 7:43 PM
dcausse committed rEWLC9d3c9277fea0: Re-enable and fix tests and drop disable_coord (authored by dcausse).
Re-enable and fix tests and drop disable_coord
Fri, Apr 29, 7:37 PM
dcausse added a comment to T306797: [Shared Event Platform] Investigate Event Service Platforms.

Here are some thoughts we compiled while figuring out the possible deployment options on the wikikube k8s cluster: https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Flink_On_Kubernetes .

Fri, Apr 29, 3:25 PM · Epic, Generated Data Platform

Thu, Apr 28

dcausse created T307074: Requesting access to wmf ldap group for ejoseph .
Thu, Apr 28, 8:50 AM · SRE, SRE-Access-Requests

Mon, Apr 25

dcausse moved T306681: Download results of a CONSTRUCT query in the UI in different RDF serialisations from Incoming to GUI on the Wikidata-Query-Service board.
Mon, Apr 25, 3:38 PM · Wikidata, Wikidata-Query-Service
dcausse added a project to T306054: Upgrade deployment-wdqs01 host to Buster: Discovery-Search (Current work).
Mon, Apr 25, 3:30 PM · Discovery-Search (Current work), wdwb-tech, Wikidata, Wikidata-Query-Service, Beta-Cluster-Infrastructure
dcausse triaged T305983: query.wikidata.org/bigdata/ldf - Language string should include language tag as Medium priority.
Mon, Apr 25, 3:27 PM · Wikidata, Wikidata-Query-Service
dcausse moved T305960: wdqs-tutorial.toolforge.org loads external resources from Incoming to GUI on the Wikidata-Query-Service board.
Mon, Apr 25, 3:25 PM · Wikidata, Wikidata-Query-Service, Privacy Engineering, Privacy
dcausse moved T305961: Make https://wdqs-tutorial.toolforge.org/ work on mobile from Incoming to GUI on the Wikidata-Query-Service board.
Mon, Apr 25, 3:25 PM · Wikidata, Mobile, Wikidata-Query-Service
dcausse added a comment to T305215: 1.39.0-wmf.9 deployment blockers.
Mon, Apr 25, 7:45 AM · User-brennen, Patch-For-Review, Release-Engineering-Team (🌱 Spring Cleaning — April 2022), Release, Train Deployments

Thu, Apr 21

dcausse added a comment to T306272: [M] Notify a list of users on demand about image suggestions in a specific category.

@matthiasmullie I don't think this would be problematic but it will be limited to the ability of blazegraph to return the full category hierarchy and by the way deepcat is currently constructing its elasticsearch query.

Thu, Apr 21, 4:13 PM · Structured-Data-Backlog (Current Work), Image-Suggestions

Wed, Apr 20

dcausse claimed T305689: Elasticsearch chi@eqiad cluster contains invalid cross cluster settings.
Wed, Apr 20, 2:19 PM · Discovery-Search (Current work), CirrusSearch

Tue, Apr 19

dcausse moved T305428: Upgrade the Translate TTM Elasticsearch implementation to elasticsearch 6.8 and onwards from In Progress to Waiting on the Discovery-Search (Current work) board.
Tue, Apr 19, 4:11 PM · MW-1.39-notes (1.39.0-wmf.8; 2022-04-18), Patch-For-Review, Discovery-Search (Current work), MediaWiki-extensions-Translate, CirrusSearch
dcausse added a comment to T306422: Upgrade translatewiki elasticsearch version to 6.8.23.

@Nikerabbit ideally we would like to have migrated all MW extensions depending on elasticsearch by may 13th so addressing this task in the first half of May would be ideal for us.

Tue, Apr 19, 2:33 PM · Discovery-Search (Current work), Unplanned-Sprint-Work, Language-Team (Language-2022-April-June), translatewiki.net, CirrusSearch
dcausse moved T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead from Needs review to Needs Reporting on the Discovery-Search (Current work) board.
Tue, Apr 19, 12:29 PM · Patch-For-Review, Discovery-Search (Current work), Wikidata, Wikidata-Query-Service
dcausse created T306422: Upgrade translatewiki elasticsearch version to 6.8.23.
Tue, Apr 19, 8:52 AM · Discovery-Search (Current work), Unplanned-Sprint-Work, Language-Team (Language-2022-April-June), translatewiki.net, CirrusSearch
dcausse created T306420: Reindex the ttm index with elasticsearch 6.8.
Tue, Apr 19, 8:37 AM · Discovery-Search (Current work), CirrusSearch

Apr 14 2022

dcausse created T306168: Better classification of CirrusSearch errors.
Apr 14 2022, 8:59 AM · MW-1.39-notes (1.39.0-wmf.12; 2022-05-16), Discovery-Search (Current work), CirrusSearch
dcausse added a comment to T306054: Upgrade deployment-wdqs01 host to Buster.

I can confirm, this host is not used.

Apr 14 2022, 8:43 AM · Discovery-Search (Current work), wdwb-tech, Wikidata, Wikidata-Query-Service, Beta-Cluster-Infrastructure

Apr 12 2022

dcausse moved T209859: Wikidata autocomplete (wbsearchentities) results with score <= 0 from Ready for Development to In Progress on the Discovery-Search (Current work) board.
Apr 12 2022, 9:30 AM · MW-1.39-notes (1.39.0-wmf.9; 2022-04-25), Discovery-Search (Current work), Wikidata
dcausse moved T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead from Waiting to Needs review on the Discovery-Search (Current work) board.
Apr 12 2022, 7:52 AM · Patch-For-Review, Discovery-Search (Current work), Wikidata, Wikidata-Query-Service

Apr 11 2022

dcausse created T305818: Perform a data transfer to wdqs2004 & wdqs1004 to reclaim burnt allocators.
Apr 11 2022, 8:37 AM · Discovery-Search (Current work), Wikidata, Wikidata-Query-Service

Apr 8 2022

dcausse updated the task description for T305689: Elasticsearch chi@eqiad cluster contains invalid cross cluster settings.
Apr 8 2022, 2:29 PM · Discovery-Search (Current work), CirrusSearch
dcausse updated the task description for T305689: Elasticsearch chi@eqiad cluster contains invalid cross cluster settings.
Apr 8 2022, 2:19 PM · Discovery-Search (Current work), CirrusSearch
dcausse created T305689: Elasticsearch chi@eqiad cluster contains invalid cross cluster settings.
Apr 8 2022, 12:58 PM · Discovery-Search (Current work), CirrusSearch

Apr 5 2022

dcausse added a comment to T302189: Regularly purge orphaned sitelink, value and reference nodes.

Reason is that this data may be referenced by other items and thus cannot be deleted blindly without asking blazegraph: "is this data used by another item?" which would be too costly to ask for every edit.
Another approach is to reload blazegraph from the dumps at regular intervals (TBD: once, twice or four times a year).

Apr 5 2022, 8:21 PM · Wikidata, Wikidata-Query-Service
dcausse moved T305169: TypeError: Argument 2 passed to WikitextContentHandler::getDataForSearchIndex() must be an instance of ParserOutput, null given, called in /srv/mediawiki/php-1.39.0-wmf.5/extensions/CirrusSearch/includes/BuildDocument/ParserOutputPageProperties.php on line 88 from In Progress to Needs review on the Discovery-Search (Current work) board.
Apr 5 2022, 5:34 PM · MW-1.39-notes (1.39.0-wmf.7; 2022-04-11), Discovery-Search (Current work), CirrusSearch, Discovery, Wikimedia-production-error
dcausse claimed T305169: TypeError: Argument 2 passed to WikitextContentHandler::getDataForSearchIndex() must be an instance of ParserOutput, null given, called in /srv/mediawiki/php-1.39.0-wmf.5/extensions/CirrusSearch/includes/BuildDocument/ParserOutputPageProperties.php on line 88.
Apr 5 2022, 7:26 AM · MW-1.39-notes (1.39.0-wmf.7; 2022-04-11), Discovery-Search (Current work), CirrusSearch, Discovery, Wikimedia-production-error
dcausse moved T305428: Upgrade the Translate TTM Elasticsearch implementation to elasticsearch 6.8 and onwards from Incoming to In Progress on the Discovery-Search (Current work) board.
Apr 5 2022, 7:23 AM · MW-1.39-notes (1.39.0-wmf.8; 2022-04-18), Patch-For-Review, Discovery-Search (Current work), MediaWiki-extensions-Translate, CirrusSearch
dcausse triaged T305428: Upgrade the Translate TTM Elasticsearch implementation to elasticsearch 6.8 and onwards as Medium priority.
Apr 5 2022, 7:22 AM · MW-1.39-notes (1.39.0-wmf.8; 2022-04-18), Patch-For-Review, Discovery-Search (Current work), MediaWiki-extensions-Translate, CirrusSearch
dcausse added a subtask for T263142: [EPIC] Upgrade Elasticsearch to version 7.10: T305428: Upgrade the Translate TTM Elasticsearch implementation to elasticsearch 6.8 and onwards .
Apr 5 2022, 7:21 AM · MW-1.39-notes (1.39.0-wmf.8; 2022-04-18), Discovery-Search (Current work), Epic
dcausse added a parent task for T305428: Upgrade the Translate TTM Elasticsearch implementation to elasticsearch 6.8 and onwards : T263142: [EPIC] Upgrade Elasticsearch to version 7.10.
Apr 5 2022, 7:21 AM · MW-1.39-notes (1.39.0-wmf.8; 2022-04-18), Patch-For-Review, Discovery-Search (Current work), MediaWiki-extensions-Translate, CirrusSearch
dcausse created T305428: Upgrade the Translate TTM Elasticsearch implementation to elasticsearch 6.8 and onwards .
Apr 5 2022, 7:21 AM · MW-1.39-notes (1.39.0-wmf.8; 2022-04-18), Patch-For-Review, Discovery-Search (Current work), MediaWiki-extensions-Translate, CirrusSearch

Apr 4 2022

dcausse moved T288765: Always provide minimum_should_match in bool queries from In Progress to To Be Deployed on the Discovery-Search (Current work) board.
Apr 4 2022, 3:24 PM · MW-1.39-notes (1.39.0-wmf.12; 2022-05-16), Patch-For-Review, Discovery-Search (Current work), CirrusSearch
dcausse moved T301147: The WDQS streaming updater went unstable for several hours (2022-02-06T23:00:00 - 2022-02-07T06:20:00) from Needs review to Needs Reporting on the Discovery-Search (Current work) board.
Apr 4 2022, 3:22 PM · Discovery-Search (Current work), Wikidata, Wikidata-Query-Service
dcausse moved T305068: Alert when flink does not have the number of expected task managers from Needs review to Needs Reporting on the Discovery-Search (Current work) board.
Apr 4 2022, 3:22 PM · Discovery-Search (Current work), Wikidata, Wikidata-Query-Service

Apr 1 2022

dcausse added a comment to T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead.

Actually wdqs2007, wdqs2004 and wdqs2003 also triggered jvmquake, GC activity increased and wdqs2007 & wdqs2003 were unresponsive for a couple minutes. For wdqs2004 there are no visible blips in the various graph. I guess we should relax the settings a bit more.

Apr 1 2022, 1:05 PM · Patch-For-Review, Discovery-Search (Current work), Wikidata, Wikidata-Query-Service
dcausse added a comment to T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead.

With the settings we properly detected wdqs1006 going down for 30minutes at 2022-04-01T12:30:00 (this 2minutes after the first blip in the graph).
Unfortunately there was a false positive wdqs1012 at 2022-04-01T10:00:00 as this machine was unavailable from 2 minutes only.
Unsure if it's still too sensitive or if we can accept having a couple false positives.

Apr 1 2022, 12:50 PM · Patch-For-Review, Discovery-Search (Current work), Wikidata, Wikidata-Query-Service
dcausse moved T304365: Add property predicates to WCQS from Incoming to Scaling on the Wikidata-Query-Service board.

I agree that federation is adding a lot of boiler plate and inspecting the shape of the IRIs is very fragile. But merging multiple graphs into the same store for ease of use is going against the recent discussions we had around the future of the WDQS architecture, it is also a bit more complex than it seems within the current data flows. Nevertheless I think the concern you raise is very valid and should be taken into account while we figure out if splitting the graph and building on top SPARQL federation is something we have to pursue and/or if some sub-graph are very central that they'd better be replicated to all sub-graphs.

Apr 1 2022, 7:52 AM · Wikidata, Wikidata-Query-Service, SDC General
dcausse moved T305068: Alert when flink does not have the number of expected task managers from Incoming to Current work on the Wikidata-Query-Service board.
Apr 1 2022, 7:39 AM · Discovery-Search (Current work), Wikidata, Wikidata-Query-Service

Mar 31 2022

dcausse updated the task description for T301147: The WDQS streaming updater went unstable for several hours (2022-02-06T23:00:00 - 2022-02-07T06:20:00).
Mar 31 2022, 2:50 PM · Discovery-Search (Current work), Wikidata, Wikidata-Query-Service
dcausse added a comment to T301147: The WDQS streaming updater went unstable for several hours (2022-02-06T23:00:00 - 2022-02-07T06:20:00).

Thanks for the quick answer! (response inline)

Mar 31 2022, 2:50 PM · Discovery-Search (Current work), Wikidata, Wikidata-Query-Service
dcausse moved T301147: The WDQS streaming updater went unstable for several hours (2022-02-06T23:00:00 - 2022-02-07T06:20:00) from Ready for Development to Needs review on the Discovery-Search (Current work) board.

Tentatively moving this ticket to needs review as I'm not sure sure we can do much more from the search team perspective.
I think the last point to discuss was to investigate the reasons why a single k8s node that misbehaves could make a deployment unstable.
@JMeybohm do you see any additional action items that would improve the resilience of k8s in such scenario?

Mar 31 2022, 12:52 PM · Discovery-Search (Current work), Wikidata, Wikidata-Query-Service
dcausse claimed T305068: Alert when flink does not have the number of expected task managers.
Mar 31 2022, 12:45 PM · Discovery-Search (Current work), Wikidata, Wikidata-Query-Service

Mar 30 2022

dcausse placed T209859: Wikidata autocomplete (wbsearchentities) results with score <= 0 up for grabs.
Mar 30 2022, 4:09 PM · MW-1.39-notes (1.39.0-wmf.9; 2022-04-25), Discovery-Search (Current work), Wikidata
dcausse assigned T209859: Wikidata autocomplete (wbsearchentities) results with score <= 0 to EJoseph.
Mar 30 2022, 4:09 PM · MW-1.39-notes (1.39.0-wmf.9; 2022-04-25), Discovery-Search (Current work), Wikidata
dcausse added a project to T238751: Only generate maxlag from pooled query service servers.: Discovery-Search.
Mar 30 2022, 3:06 PM · Discovery-Search (Current work), User-ItamarWMDE, SRE-OnFire, wdwb-tech, Sustainability (Incident Followup), Patch-For-Review, User-Addshore, Wikidata
dcausse updated the task description for T301147: The WDQS streaming updater went unstable for several hours (2022-02-06T23:00:00 - 2022-02-07T06:20:00).
Mar 30 2022, 2:22 PM · Discovery-Search (Current work), Wikidata, Wikidata-Query-Service
dcausse updated the task description for T301147: The WDQS streaming updater went unstable for several hours (2022-02-06T23:00:00 - 2022-02-07T06:20:00).
Mar 30 2022, 2:10 PM · Discovery-Search (Current work), Wikidata, Wikidata-Query-Service
dcausse created T305068: Alert when flink does not have the number of expected task managers.
Mar 30 2022, 2:09 PM · Discovery-Search (Current work), Wikidata, Wikidata-Query-Service
dcausse assigned T304796: "Search completion" setting is not changed. to Func.
Mar 30 2022, 1:03 PM · Discovery-Search (Current work), Regression, CirrusSearch
dcausse moved T174745: Enable debug API's, like cirrusDumpQuery, cirrusDumpResult and cirrusExplain for prefix search and completion suggester from elastic / cirrus to needs triage on the Discovery-Search board.
Mar 30 2022, 10:34 AM · Discovery-Search (Current work), Discovery, CirrusSearch
dcausse moved T161863: Support searching for external links in CirrusSearch from Feature Requests to needs triage on the Discovery-Search board.
Mar 30 2022, 10:26 AM · Discovery-Search, Discovery, CirrusSearch
dcausse moved T284499: Make CirrusSearch word count available via API from Feature Requests to needs triage on the Discovery-Search board.
Mar 30 2022, 10:12 AM · Discovery-Search (Current work), CirrusSearch

Mar 29 2022

dcausse moved T294076: Blazegraph and MariaDB contain different sitelinks at Wikidata from Waiting to Needs Reporting on the Discovery-Search (Current work) board.

The reconciliation process is running and should auto-correct missed updates couple hours after they're performed.
I also fixed the inconsistencies listed here and other related tickets. Please let me know if you still find errors.

Mar 29 2022, 5:09 PM · Discovery-Search (Current work), Wikidata-Query-Service, Wikidata
dcausse added a project to T304954: Import data from hdfs to commonswiki_file: Discovery-Search.
Mar 29 2022, 2:22 PM · Discovery-Search (Current work), Image-Suggestions
dcausse updated the task description for T302494: The WDQS Streaming Updater should use S3 to access thanos-swift instead of the native swift protocol.
Mar 29 2022, 8:18 AM · Discovery-Search (Current work), Patch-For-Review, Wikidata, Wikidata-Query-Service
dcausse moved T302494: The WDQS Streaming Updater should use S3 to access thanos-swift instead of the native swift protocol from In Progress to Needs Reporting on the Discovery-Search (Current work) board.

Moved remaining work in T304914.

Mar 29 2022, 8:17 AM · Discovery-Search (Current work), Patch-For-Review, Wikidata, Wikidata-Query-Service
dcausse updated the task description for T302494: The WDQS Streaming Updater should use S3 to access thanos-swift instead of the native swift protocol.
Mar 29 2022, 8:16 AM · Discovery-Search (Current work), Patch-For-Review, Wikidata, Wikidata-Query-Service
dcausse created T304914: Remove the presto client for swift from the flink image.
Mar 29 2022, 8:15 AM · Wikidata, Wikidata-Query-Service

Mar 28 2022

dcausse committed rWDANb5b63c3fe32a: wdqs reconciliation: fix entity to namespace map (authored by dcausse).
wdqs reconciliation: fix entity to namespace map
Mar 28 2022, 4:09 PM
dcausse updated the task description for T242453: Detect and alert and/or remediate Blazegraph deadlocks.
Mar 28 2022, 3:59 PM · Discovery-Search (Current work), Wikidata-Query-Service, Wikidata
dcausse moved T304290: Create docker elasticsearch images with wmf search plugins from Needs review to Blocked (from outside the team) on the Discovery-Search (Current work) board.
Mar 28 2022, 3:08 PM · dev-images, Discovery-Search (Current work), CirrusSearch
dcausse reopened T242453: Detect and alert and/or remediate Blazegraph deadlocks as "Open".

re-opening, seems to happen more frequently

Mar 28 2022, 2:23 PM · Discovery-Search (Current work), Wikidata-Query-Service, Wikidata

Mar 25 2022

dcausse added projects to T304290: Create docker elasticsearch images with wmf search plugins: Release-Engineering-Team, dev-images.

Pinging releng for help on how to proceed with the gitlab MR and the deployment of the images to the docker repo.

Mar 25 2022, 3:24 PM · dev-images, Discovery-Search (Current work), CirrusSearch

Mar 24 2022

dcausse added a comment to T304224: Archiva's disk partiton space is getting filled up.

I notice that you didn't include https://archiva.wikimedia.org/#artifact~releases/org.wikidata.query.rdf/blazegraph-service in that list of releases to prune @dcausse
Should I drop anything below 0.3.80 in there was well?

Mar 24 2022, 11:44 AM · Data-Engineering-Kanban, Data-Engineering