Page MenuHomePhabricator

dcausse (David Causse)
User

Projects

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Thursday

  • Clear sailing ahead.

User Details

User Since
Jun 9 2015, 9:03 AM (407 w, 12 h)
Availability
Available
IRC Nick
dcausse
LDAP User
DCausse
MediaWiki User
DCausse (WMF) [ Global Accounts ]

Recent Activity

Today

dcausse updated the task description for T333373: The WDQS streaming updater should support connecting to kafka with SSL.
Tue, Mar 28, 3:37 PM · Wikidata, Wikidata-Query-Service
dcausse created T333373: The WDQS streaming updater should support connecting to kafka with SSL.
Tue, Mar 28, 3:34 PM · Wikidata, Wikidata-Query-Service
dcausse added a comment to T330507: New Service Request mediawiki-page-content-change-enrichment.

Oh, I misunderstood, I thought that WDQS updater was using the Wikidata content?

Tue, Mar 28, 12:37 PM · Event-Platform Value Stream (Sprint 10), Patch-For-Review, Data-Engineering-Planning, serviceops, Service-deployment-requests
Restricted Application added a project to T331405: Query service maxlag calculation should exclude datacenters that don't receive traffic and where the updater is turned off: User-ItamarWMDE.

@ItamarWMDE once https://gerrit.wikimedia.org/r/900729 is deployed we should be able to create a grafana query like the one suggested by Joe and adjust the threshold to double check that the query does what we need, I'll ping you once ready :)

Tue, Mar 28, 10:23 AM · User-ItamarWMDE, serviceops, wdwb-tech, Wikidata.org, Wikidata-Query-Service, Wikidata
dcausse added a comment to T330507: New Service Request mediawiki-page-content-change-enrichment.

Sorry, I'm getting confused; to my understanding, WDQS/search will use mediawiki.page_change which AIUI are generated from mediawiki, not mediawiki.page_content_change.

Tue, Mar 28, 9:44 AM · Event-Platform Value Stream (Sprint 10), Patch-For-Review, Data-Engineering-Planning, serviceops, Service-deployment-requests

Wed, Mar 22

dcausse edited projects for T325565: Add support for page re-renders, added: Discovery-Search (Current work); removed Discovery-Search.
Wed, Mar 22, 9:39 AM · Discovery-Search (Current work), CirrusSearch
dcausse edited projects for T325672: Re-order and optimize change events, added: Discovery-Search (Current work); removed Discovery-Search.
Wed, Mar 22, 9:38 AM · Discovery-Search (Current work), CirrusSearch
dcausse renamed T332763: The search update pipeline should support events compatible with the /mediawiki/page/change/1.0.0 schema from Add support for /mediawiki/page/change/1.0.0 to The search update pipeline should support events compatible with the /mediawiki/page/change/1.0.0 schema.
Wed, Mar 22, 9:34 AM · Discovery-Search (Current work), CirrusSearch
dcausse edited projects for T332763: The search update pipeline should support events compatible with the /mediawiki/page/change/1.0.0 schema, added: CirrusSearch, Discovery-Search (Current work); removed Discovery-Search.
Wed, Mar 22, 9:32 AM · Discovery-Search (Current work), CirrusSearch
dcausse edited projects for T332763: The search update pipeline should support events compatible with the /mediawiki/page/change/1.0.0 schema, added: Discovery-Search; removed Discovery-Search (Current work).
Wed, Mar 22, 9:32 AM · Discovery-Search (Current work), CirrusSearch
dcausse created T332763: The search update pipeline should support events compatible with the /mediawiki/page/change/1.0.0 schema.
Wed, Mar 22, 9:31 AM · Discovery-Search (Current work), CirrusSearch
dcausse moved T325315: Add support for redirects from Ready for Dev -- SWE to Incoming on the Discovery-Search (Current work) board.
Wed, Mar 22, 9:24 AM · Discovery-Search (Current work)
dcausse edited projects for T325315: Add support for redirects, added: Discovery-Search (Current work); removed Discovery-Search.
Wed, Mar 22, 9:24 AM · Discovery-Search (Current work)

Tue, Mar 21

dcausse claimed T328330: Create SLI / SLO on Search update lag and error rate.
Tue, Mar 21, 4:11 PM · Patch-For-Review, Discovery-Search (Current work), Epic

Mon, Mar 20

dcausse moved T329879: Migrate rdf_streaming_updater_reconcile.py from airflow 1 to airflow 2 from To Be Deployed to Needs Reporting on the Discovery-Search (Current work) board.
Mon, Mar 20, 4:10 PM · Discovery-Search (Current work), Data Pipelines, Data-Engineering-Planning
dcausse moved T328625: Federated queries with AGROVOC are not working from To Be Deployed to Needs Reporting on the Discovery-Search (Current work) board.
Mon, Mar 20, 4:09 PM · Discovery-Search (Current work), Wikidata, Wikidata-Query-Service
dcausse moved T331271: Add https://data.europa.eu/sparql to WDQS federated services allow list from To Be Deployed to Needs Reporting on the Discovery-Search (Current work) board.
Mon, Mar 20, 4:09 PM · Discovery-Search (Current work), Wikidata, Wikidata-Query-Service

Thu, Mar 16

dcausse closed T297870: WDQS Streaming Updater fails with Timeout expired after 60000milliseconds while awaiting InitProducerId as Declined.

We might have to switch to KafkaSource and this might change the behavior of flink during this kind of scenario.

Thu, Mar 16, 2:53 PM · Wikidata, Wikidata-Query-Service
dcausse added a comment to T330693: Storage request: swift s3 bucket for mediawiki-page-content-change-enrichment checkpointing.

Hi everyone and sorry to jump into this conversion but just wanted to add a quick note on the usefulness of cross-DC replication as it's something that was not obvious to us when we started to use swift containers in thanos for our search jobs.
If you consider all the dependent services&platforms:

  • wikikube
  • kafka-main
  • MW mw-async* (hopefully a new mw-async-ro should be available at some point)
  • thanos*

(* for services that I think have an automatic failover via dns discovery in place)
Thanos being replicated this brings down to only 2 the services/platforms that would cause a downtime to our job if they're down.

Thu, Mar 16, 8:49 AM · Event-Platform Value Stream (Sprint 10), Data-Engineering-Planning, SRE-swift-storage

Mon, Mar 13

dcausse added a comment to T323628: Optimize the WikibaseCirrusSearch elasticsearch mapping and filter query for non-english users.

@TJones indeed I think that the query should explicitly add labels.ko to to the filter, it does seem to only add a scoring clause. Pushed a small patch to change how the filter is constructed.

Mon, Mar 13, 7:17 PM · MW-1.41-notes (1.41.0-wmf.1; 2023-03-20), MW-1.40-notes (1.40.0-wmf.27; 2023-03-13), Discovery-Search (Current work), CirrusSearch
dcausse moved T329879: Migrate rdf_streaming_updater_reconcile.py from airflow 1 to airflow 2 from Needs review to To Be Deployed on the Discovery-Search (Current work) board.
Mon, Mar 13, 4:06 PM · Discovery-Search (Current work), Data Pipelines, Data-Engineering-Planning
dcausse claimed T331271: Add https://data.europa.eu/sparql to WDQS federated services allow list.
Mon, Mar 13, 2:02 PM · Discovery-Search (Current work), Wikidata, Wikidata-Query-Service
dcausse added a comment to T331719: When searching by keyword, results sorted by relevance should prioritize family names in the title: please improve search results for articles with DEFAULTSORT.

Thanks for the detailed report!
Adding DEFAULTSORT to autocomplete searches is a feature that we can enable on a per-wiki basis, due to the way this tag is used it can't be enabled on every wiki without prior evaluation (see T145427#3515817). This feature was first enabled on mongolian wikipedia a couple weeks ago (see T327878). If this is something the frwiki community would like to experiment with we could enable it.

Mon, Mar 13, 1:53 PM · Discovery-Search, CirrusSearch, Community Feedback (Web)

Thu, Mar 9

dcausse added a comment to T331405: Query service maxlag calculation should exclude datacenters that don't receive traffic and where the updater is turned off.

WDQS lag issues should be rare now, a node not serving traffic should (as of today) be able to ingest ~8x the throughput that we usually see on wikidata so we should not worry about them.
Using blazegraph_queries_done_total to identify nodes that should be part of the maxlag propagation makes a ton of sense, there might be some adjustments to make in how this metric is produced. It currently also include update queries which might make the threshold a bit hard to adapt esp. on codfw that receives a lot less trafic than eqiad.
If there are no strong reasons against this idea I'm happy to make the necessary adjustments to blazegraph_queries_done_total (or possibly introduce a new metric suited for this purpose).

Thu, Mar 9, 1:22 PM · User-ItamarWMDE, serviceops, wdwb-tech, Wikidata.org, Wikidata-Query-Service, Wikidata

Tue, Mar 7

dcausse closed T316882: RdfStreamingUpdaterHighConsumerUpdateLag alert is not fired as Resolved.

can confirm it's fixed by the above patch, saw multiple alerts of this kind being fired during the k8s upgrade. Thanks!

Tue, Mar 7, 3:47 PM · Wikidata, Wikidata-Query-Service
dcausse added a comment to T331127: phantom redirects lingering in incategory searches after page moves.

I'll note that there is one new page so far that ended up in the incategory search today for the same reasons, but I'm not immediately doing anything about it because of what you said about how the process fix may take time to work through the database -- and also because I've nominated the mainspace move target for AFD as improperly sourced possible self-promotion anyway -- but I also wanted to ask: despite there only being one page currently in the search, the number of pages is being reported as three by the "results" counter in the top right corner. Would this simply be an artifact of the same problem, which will clean itself up as the fix that was already applied here propagates, or would this be a different problem that has to be looked at separately?

Tue, Mar 7, 2:06 PM · MW-1.40-notes (1.40.0-wmf.25; 2023-02-27), Discovery-Search (Current work), CirrusSearch
dcausse added a comment to T331405: Query service maxlag calculation should exclude datacenters that don't receive traffic and where the updater is turned off.

Updates shouldn't depend on where the discovery dns record points to, but rather go to the local datacenter directly.

I think the bug here is with wdqs-updater rather than with the script.

Tue, Mar 7, 1:15 PM · User-ItamarWMDE, serviceops, wdwb-tech, Wikidata.org, Wikidata-Query-Service, Wikidata
dcausse moved T331127: phantom redirects lingering in incategory searches after page moves from To Be Deployed to Needs Reporting on the Discovery-Search (Current work) board.

The problem should be resolved, new page moves across namespaces should properly delete the page in the old namespace. Some phantom redirects created because of this bug will stay until the process that cleanups the index fixes these pages (it can take up to 8weeks), if you identify annoying ones please do let us know in this ticket so that we can clean them up manually quicker. Sorry for the inconvenience this has caused.

Tue, Mar 7, 8:52 AM · MW-1.40-notes (1.40.0-wmf.25; 2023-02-27), Discovery-Search (Current work), CirrusSearch

Mon, Mar 6

dcausse moved T331127: phantom redirects lingering in incategory searches after page moves from Needs review to To Be Deployed on the Discovery-Search (Current work) board.
Mon, Mar 6, 8:51 PM · MW-1.40-notes (1.40.0-wmf.25; 2023-02-27), Discovery-Search (Current work), CirrusSearch
dcausse raised the priority of T331127: phantom redirects lingering in incategory searches after page moves from Medium to High.
Mon, Mar 6, 6:12 PM · MW-1.40-notes (1.40.0-wmf.25; 2023-02-27), Discovery-Search (Current work), CirrusSearch
dcausse moved T331127: phantom redirects lingering in incategory searches after page moves from In Progress to Needs review on the Discovery-Search (Current work) board.
Mon, Mar 6, 6:04 PM · MW-1.40-notes (1.40.0-wmf.25; 2023-02-27), Discovery-Search (Current work), CirrusSearch
dcausse claimed T331127: phantom redirects lingering in incategory searches after page moves.
Mon, Mar 6, 5:56 PM · MW-1.40-notes (1.40.0-wmf.25; 2023-02-27), Discovery-Search (Current work), CirrusSearch
dcausse edited projects for T331127: phantom redirects lingering in incategory searches after page moves, added: Discovery-Search (Current work); removed Discovery-Search.
Mon, Mar 6, 5:54 PM · MW-1.40-notes (1.40.0-wmf.25; 2023-02-27), Discovery-Search (Current work), CirrusSearch
dcausse moved T322869: Fewer results from wdqs nodes running in codfw than eqiad from Needs review to Needs Reporting on the Discovery-Search (Current work) board.
Mon, Mar 6, 4:21 PM · Discovery-Search (Current work), Wikidata
dcausse moved T328332: Add a new keyword to filter pages based on their "length" from Needs review to Blocked/Waiting on the Discovery-Search (Current work) board.
Mon, Mar 6, 3:55 PM · MW-1.40-notes (1.40.0-wmf.24; 2023-02-20), Patch-For-Review, Discovery-Search (Current work), CirrusSearch, GrowthExperiments-Homepage, Growth-Team
dcausse moved T317609: Create a schema for fetch failures from Needs review to Blocked/Waiting on the Discovery-Search (Current work) board.
Mon, Mar 6, 3:55 PM · Patch-For-Review, Discovery-Search (Current work)
dcausse moved T323628: Optimize the WikibaseCirrusSearch elasticsearch mapping and filter query for non-english users from Needs review to To Be Deployed on the Discovery-Search (Current work) board.
Mon, Mar 6, 3:49 PM · MW-1.41-notes (1.41.0-wmf.1; 2023-03-20), MW-1.40-notes (1.40.0-wmf.27; 2023-03-13), Discovery-Search (Current work), CirrusSearch
dcausse created T331271: Add https://data.europa.eu/sparql to WDQS federated services allow list.
Mon, Mar 6, 9:50 AM · Discovery-Search (Current work), Wikidata, Wikidata-Query-Service

Feb 24 2023

dcausse added a parent task for T294133: Expose rdf-streaming-updater.mutation content through EventStreams: T330521: Make WDQS update stream public.
Feb 24 2023, 7:34 PM · Data-Engineering, Event-Platform Value Stream, Analytics, Wikidata, EventStreams, Wikidata-Query-Service
dcausse added a subtask for T330521: Make WDQS update stream public: T294133: Expose rdf-streaming-updater.mutation content through EventStreams.
Feb 24 2023, 7:34 PM · Wikidata, Wikidata Analytics, Wikidata-Query-Service

Feb 23 2023

dcausse updated the task description for T241128: EPIC: Reduce the time needed to do the initial WDQS import.
Feb 23 2023, 5:26 PM · Epic, Wikidata-Query-Service, Wikidata
dcausse added a comment to T330236: Event partitions missing since 2023-02-21T10:00 for stream without events (canary events not produced?).

Thanks for fixing the issue and dealing with failed sensors! :)
Regarding what we could do to mitigate this in the future I don't think there exists great solution out of the box...
One could set soft_fail=True on their sensors but that means you have to set a meaningful timeout and not sure the benefit would be worth the possible harm in case the data finally appears a couple hours later...
Perhaps the solution to creating these "empty" partitions could be done without relying on canary events?

Feb 23 2023, 8:45 AM · Patch-For-Review, Data-Engineering-Planning, Event-Platform Value Stream

Feb 22 2023

dcausse added a comment to T330236: Event partitions missing since 2023-02-21T10:00 for stream without events (canary events not produced?).

Are there ways to unblock it? It's causing plenty of hourly jobs to fail on our side.

Feb 22 2023, 5:15 PM · Patch-For-Review, Data-Engineering-Planning, Event-Platform Value Stream
dcausse updated subscribers of T330236: Event partitions missing since 2023-02-21T10:00 for stream without events (canary events not produced?).
Feb 22 2023, 4:04 PM · Patch-For-Review, Data-Engineering-Planning, Event-Platform Value Stream
dcausse updated subscribers of T330236: Event partitions missing since 2023-02-21T10:00 for stream without events (canary events not produced?).

I think @Ottomata used to take care of this and was not sure what tags to add so please feel free to remove it from here :)

Feb 22 2023, 11:23 AM · Patch-For-Review, Data-Engineering-Planning, Event-Platform Value Stream
dcausse added a comment to T330236: Event partitions missing since 2023-02-21T10:00 for stream without events (canary events not produced?).

org.wikimedia.analytics.refinery.job.ProduceCanaryEvents seems stuck since yesterday maint operation on the k8s cluster, one thread seems blocked on:

"main" #1 prio=5 os_prio=0 tid=0x00007f301000b000 nid=0x138e runnable [0x00007f3015216000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:464)
	at sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:68)
	at sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1346)
	at sun.security.ssl.SSLSocketImpl.access$300(SSLSocketImpl.java:73)
	at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:962)
	at org.apache.http.impl.conn.LoggingInputStream.read(LoggingInputStream.java:84)
	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
	at org.wikimedia.eventutilities.core.http.BasicHttpClient.post(BasicHttpClient.java:107)
	at org.wikimedia.eventutilities.monitoring.CanaryEventProducer.postEvents(CanaryEventProducer.java:286)
	at org.wikimedia.eventutilities.monitoring.CanaryEventProducer.lambda$postEventsToUris$2(CanaryEventProducer.java:253)
	at org.wikimedia.eventutilities.monitoring.CanaryEventProducer$$Lambda$52/478182173.apply(Unknown Source)
	at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1321)
	at java.util.stream.Collectors$$Lambda$37/1365790282.accept(Unknown Source)
	at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
	at java.util.HashMap$EntrySpliterator.forEachRemaining(HashMap.java:1723)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
	at org.wikimedia.eventutilities.monitoring.CanaryEventProducer.postEventsToUris(CanaryEventProducer.java:251)
	at org.wikimedia.analytics.refinery.job.ProduceCanaryEvents$.produceCanaryEvents(ProduceCanaryEvents.scala:307)
	at org.wikimedia.analytics.refinery.job.ProduceCanaryEvents$$anonfun$4$$anonfun$apply$1.apply$mcZ$sp(ProduceCanaryEvents.scala:212)
	at org.wikimedia.analytics.refinery.job.ProduceCanaryEvents$$anonfun$4$$anonfun$apply$1.apply(ProduceCanaryEvents.scala:212)
	at org.wikimedia.analytics.refinery.job.ProduceCanaryEvents$$anonfun$4$$anonfun$apply$1.apply(ProduceCanaryEvents.scala:212)
	at scala.util.Try$.apply(Try.scala:192)
	at org.wikimedia.analytics.refinery.job.ProduceCanaryEvents$$anonfun$4.apply(ProduceCanaryEvents.scala:211)
	at org.wikimedia.analytics.refinery.job.ProduceCanaryEvents$$anonfun$4.apply(ProduceCanaryEvents.scala:210)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
	at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
	at scala.collection.AbstractTraversable.map(Traversable.scala:104)
	at org.wikimedia.analytics.refinery.job.ProduceCanaryEvents$.apply(ProduceCanaryEvents.scala:210)
	at org.wikimedia.analytics.refinery.job.ProduceCanaryEvents$.main(ProduceCanaryEvents.scala:147)
	at org.wikimedia.analytics.refinery.job.ProduceCanaryEvents.main(ProduceCanaryEvents.scala)
Feb 22 2023, 10:08 AM · Patch-For-Review, Data-Engineering-Planning, Event-Platform Value Stream

Feb 21 2023

dcausse created T330236: Event partitions missing since 2023-02-21T10:00 for stream without events (canary events not produced?).
Feb 21 2023, 9:24 PM · Patch-For-Review, Data-Engineering-Planning, Event-Platform Value Stream

Feb 20 2023

dcausse moved T327381: Migrate RDF Tooling to Spark 3 from To Be Deployed to Needs Reporting on the Discovery-Search (Current work) board.
Feb 20 2023, 4:08 PM · Discovery-Search (Current work)

Feb 17 2023

dcausse renamed T329842: Some jobs in refreshLinksPrioritized seems to repeat themselves for ever from Some jobs (refreshLinksPrioritized?) seems to repeat themselves for ever (rootJobTimestamp not propagated?) to Some jobs in refreshLinksPrioritized seems to repeat themselves for ever.
Feb 17 2023, 5:12 PM · WMF-JobQueue
dcausse added a comment to T328276: Add outlink topic model predictions to CirrusSearch indices.

@dcausse Hi! I am working in T328576 to split mediawiki.revision-score into multiple substreams, and I am wondering which ones are really needed. IIUC from reading this task your team would still need drafttopic right? If so we could start from it so that you'll be able to migrate away from ORES' revision-score. Does it make sense?

Moreover, do you folks use or plan to use other ORES revision score streams in the future? I recall that we discussed briefly something about it, but I don't remember the exact list of things needed :)

Feb 17 2023, 9:21 AM · Machine-Learning-Team, Discovery-Search, CirrusSearch

Feb 16 2023

dcausse updated subscribers of T329842: Some jobs in refreshLinksPrioritized seems to repeat themselves for ever.

pinging @daniel, in case you have ideas on who could help us to understand and possibly fix this behavior, thanks!

Feb 16 2023, 5:05 PM · WMF-JobQueue
dcausse updated subscribers of T329842: Some jobs in refreshLinksPrioritized seems to repeat themselves for ever.
Feb 16 2023, 4:02 PM · WMF-JobQueue
dcausse created T329842: Some jobs in refreshLinksPrioritized seems to repeat themselves for ever.
Feb 16 2023, 2:47 PM · WMF-JobQueue
Sj awarded T198318: Change how SpecialSearch/SearchEngine handles the 'prefix' URI param set by the InputBox extension a Burninate token.
Feb 16 2023, 12:35 AM · MW-1.32-notes (WMF-deploy-2018-07-24 (1.32.0-wmf.14)), Discovery-Search (Current work), Technical-Debt, Advanced-Search, Discovery-ARCHIVED, TCB-Team (now WMDE-TechWish), CirrusSearch

Feb 14 2023

dcausse added a comment to T328276: Add outlink topic model predictions to CirrusSearch indices.

Thanks for all the input! I've updated the task description accordingly.

Feb 14 2023, 2:58 PM · Machine-Learning-Team, Discovery-Search, CirrusSearch
dcausse updated the task description for T328276: Add outlink topic model predictions to CirrusSearch indices.
Feb 14 2023, 2:57 PM · Machine-Learning-Team, Discovery-Search, CirrusSearch
dcausse claimed T320408: Monitor CirrusSearch update lag.
Feb 14 2023, 1:15 PM · MW-1.41-notes (1.41.0-wmf.1; 2023-03-20), Discovery-Search (Current work), CirrusSearch
dcausse renamed T328332: Add a new keyword to filter pages based on their "length" from implement some kind of charlengthlessthan:1500 search keyword to Add a new keyword to filter pages based on their "length".
Feb 14 2023, 10:50 AM · MW-1.40-notes (1.40.0-wmf.24; 2023-02-20), Patch-For-Review, Discovery-Search (Current work), CirrusSearch, GrowthExperiments-Homepage, Growth-Team
dcausse moved T328332: Add a new keyword to filter pages based on their "length" from In Progress to Needs review on the Discovery-Search (Current work) board.

Sadly I had to change the elasticsearch mapping to allow this and thus this will have to wait for a full re-index after the first patch is merged before enabling the new textbytes keyword.

Feb 14 2023, 10:24 AM · MW-1.40-notes (1.40.0-wmf.24; 2023-02-20), Patch-For-Review, Discovery-Search (Current work), CirrusSearch, GrowthExperiments-Homepage, Growth-Team
dcausse moved T328675: Create a dse-k8s service demonstrating how to run the rdf-streaming-updater using the flink-app chart from Ready for Dev -- SRE/Ops to In Progress on the Discovery-Search (Current work) board.
Feb 14 2023, 9:07 AM · Patch-For-Review, Discovery-Search (Current work), Wikidata, Wikidata-Query-Service
dcausse moved T328675: Create a dse-k8s service demonstrating how to run the rdf-streaming-updater using the flink-app chart from in progress to incoming on the Wikidata board.
Feb 14 2023, 9:06 AM · Patch-For-Review, Discovery-Search (Current work), Wikidata, Wikidata-Query-Service
dcausse claimed T328675: Create a dse-k8s service demonstrating how to run the rdf-streaming-updater using the flink-app chart.
Feb 14 2023, 9:04 AM · Patch-For-Review, Discovery-Search (Current work), Wikidata, Wikidata-Query-Service
dcausse placed T326328: Create helm charts for the cirrus-streaming-updater flink jobs up for grabs.
Feb 14 2023, 9:03 AM · Discovery-Search (Current work), CirrusSearch
dcausse claimed T326328: Create helm charts for the cirrus-streaming-updater flink jobs.
Feb 14 2023, 9:02 AM · Discovery-Search (Current work), CirrusSearch
dcausse claimed T328332: Add a new keyword to filter pages based on their "length".
Feb 14 2023, 8:57 AM · MW-1.40-notes (1.40.0-wmf.24; 2023-02-20), Patch-For-Review, Discovery-Search (Current work), CirrusSearch, GrowthExperiments-Homepage, Growth-Team

Feb 13 2023

dcausse claimed T328625: Federated queries with AGROVOC are not working.
Feb 13 2023, 7:32 PM · Discovery-Search (Current work), Wikidata, Wikidata-Query-Service
dcausse updated the task description for T328675: Create a dse-k8s service demonstrating how to run the rdf-streaming-updater using the flink-app chart.
Feb 13 2023, 4:48 PM · Patch-For-Review, Discovery-Search (Current work), Wikidata, Wikidata-Query-Service
dcausse moved T327878: Tweak Autocomplete search results on the Mongolian Wikipedia from Needs review to Needs Reporting on the Discovery-Search (Current work) board.

@chinneeb the change is applied to https://mn.wikipedia.org, typing Мөрдорж does seem to suggest Лувсанжамбын Мөрдорж first now. Please let us know if you see anything unusual.

Feb 13 2023, 3:27 PM · Discovery-Search (Current work), CirrusSearch

Feb 10 2023

dcausse claimed T327878: Tweak Autocomplete search results on the Mongolian Wikipedia.
Feb 10 2023, 9:52 AM · Discovery-Search (Current work), CirrusSearch

Feb 9 2023

dcausse committed rWDANcaf48084da32: scap: keep only 3 revisions (authored by dcausse).
scap: keep only 3 revisions
Feb 9 2023, 4:35 PM
dcausse committed rWDANdc3cd56b553a: rdf-spark-tools: bump to version 0.3.120 (authored by dcausse).
rdf-spark-tools: bump to version 0.3.120
Feb 9 2023, 3:47 PM
dcausse moved T329089: The rdf-streaming-updater does not reconcile missed page-undelete events from Needs review to Needs Reporting on the Discovery-Search (Current work) board.

Q115608572 was restored after applying the fix, items that were not edited after being undeleted will sadly remain absent from WDQS until an edit is made or after a full reload.

Feb 9 2023, 2:17 PM · Discovery-Search (Current work), Wikidata, Wikidata-Query-Service

Feb 8 2023

dcausse awarded T329070: Automated event stream throughput alerting for important state change streams a Like token.
Feb 8 2023, 2:07 PM · Data-Engineering-Planning, Event-Platform Value Stream

Feb 7 2023

dcausse claimed T329089: The rdf-streaming-updater does not reconcile missed page-undelete events.
Feb 7 2023, 6:18 PM · Discovery-Search (Current work), Wikidata, Wikidata-Query-Service
dcausse moved T329089: The rdf-streaming-updater does not reconcile missed page-undelete events from Incoming to Needs review on the Discovery-Search (Current work) board.
Feb 7 2023, 6:18 PM · Discovery-Search (Current work), Wikidata, Wikidata-Query-Service
dcausse added a project to T329089: The rdf-streaming-updater does not reconcile missed page-undelete events: Discovery-Search (Current work).
Feb 7 2023, 6:17 PM · Discovery-Search (Current work), Wikidata, Wikidata-Query-Service
dcausse added a comment to T329089: The rdf-streaming-updater does not reconcile missed page-undelete events.

The reconcile batch job seems the one to blame, it reports:
23/01/12 03:29:25 INFO ReconcileCollector: Collected 0 inconsistencies from event.rdf_streaming_updater_state_inconsistency/datacenter=eqiad/year=2023/month=1/day=12/hour=0

Feb 7 2023, 5:32 PM · Discovery-Search (Current work), Wikidata, Wikidata-Query-Service
dcausse added a comment to T329089: The rdf-streaming-updater does not reconcile missed page-undelete events.

The inconsistencies were properly detected by the updater:

select * from rdf_streaming_updater_state_inconsistency where year=2023 AND month=01 AND day=12 AND meta.domain="www.wikidata.org" AND item="Q115608572" AND datacenter="codfw";
_schema	meta	item	dt	original_ingestion_dt	revision_id	original_event_info	action_type	parent_revision_id	inconsistency	state_revision_id	state_status	is_wmf_domain	normalized_host	datacenter	year	month	day	hour
NULL	{"uri":null,"request_id":null,"id":"5eae0041-d477-4231-a31d-fe60d09b1992","dt":"2023-01-12T00:21:02.243793Z","domain":"www.wikidata.org","stream":"rdf-streaming-updater.state-inconsistency"}	Q115608572	2023-01-12T00:21:02.243557Z	2023-01-12T00:18:54.174Z	1785108184	{"_schema":null,"meta":{"uri":null,"request_id":"ab00575f-7a12-43cb-9f73-a50bf7e98540","id":"748301dd-194a-40c5-8db1-8bf79f67c797","dt":"2023-01-12T00:18:53Z","domain":"www.wikidata.org","stream":"mediawiki.revision-create"},"dt":"2023-01-12T00:18:53Z"}	revision-create	1785104250	newer_revision_seen	1785111610	DELETED	true	{"project_class":"wikidata","project":"-","qualifiers":[],"tld":"org","project_family":"wikidata"}	codfw	2023	1	12	0
NULL	{"uri":null,"request_id":null,"id":"fdfdef74-0249-4c56-adeb-aab5834f8e3a","dt":"2023-01-12T00:25:17.876264Z","domain":"www.wikidata.org","stream":"rdf-streaming-updater.state-inconsistency"}	Q115608572	2023-01-12T00:25:17.875852Z	2023-01-12T00:22:40.864Z	1809645267	{"_schema":null,"meta":{"uri":null,"request_id":"089f327e-e5f5-4bb4-b5db-dd7163a0c3fc","id":"e0779cda-3936-4705-baa6-a7bb9e21122f","dt":"2023-01-12T00:22:40Z","domain":"www.wikidata.org","stream":"mediawiki.revision-create"},"dt":"2023-01-12T00:22:40Z"}	revision-create	1785111610	newer_revision_seen	1785111610	DELETED	true	{"project_class":"wikidata","project":"-","qualifiers":[],"tld":"org","project_family":"wikidata"}	codfw	2023	1	12	0
NULL	{"uri":null,"request_id":null,"id":"eb6219c5-9d08-4148-b046-9b829fb7840e","dt":"2023-01-12T00:21:02.235752Z","domain":"www.wikidata.org","stream":"rdf-streaming-updater.state-inconsistency"}	Q115608572	2023-01-12T00:21:02.235069Z	2023-01-12T00:18:54.174Z	1785104250	{"_schema":null,"meta":{"uri":null,"request_id":"ab00575f-7a12-43cb-9f73-a50bf7e98540","id":"372a8237-b3cc-43a1-bd53-9d81f8ed3a9c","dt":"2023-01-12T00:18:53Z","domain":"www.wikidata.org","stream":"mediawiki.revision-create"},"dt":"2023-01-12T00:18:53Z"}	revision-create	NULL	newer_revision_seen	1785111610	DELETED	true	{"project_class":"wikidata","project":"-","qualifiers":[],"tld":"org","project_family":"wikidata"}	codfw	2023	1	12	0
NULL	{"uri":null,"request_id":null,"id":"ca728ff4-7bbe-4d25-9862-56494aac1c2c","dt":"2023-01-12T00:35:24.918556Z","domain":"www.wikidata.org","stream":"rdf-streaming-updater.state-inconsistency"}	Q115608572	2023-01-12T00:35:24.918252Z	2023-01-12T00:33:23.599Z	1809650380	{"_schema":null,"meta":{"uri":null,"request_id":"708615f4-e67c-4cc0-85fe-0a8dfb73f1fc","id":"e546995d-0120-4b1f-a9d4-f82fb9af88e4","dt":"2023-01-12T00:33:23Z","domain":"www.wikidata.org","stream":"mediawiki.revision-create"},"dt":"2023-01-12T00:33:23Z"}	revision-create	1809649800	newer_revision_seen	1785111610	DELETED	true	{"project_class":"wikidata","project":"-","qualifiers":[],"tld":"org","project_family":"wikidata"}	codfw	2023	1	12	0
NULL	{"uri":null,"request_id":null,"id":"197fc474-7793-495b-b34b-b14df361fb4c","dt":"2023-01-12T00:38:51.435003Z","domain":"www.wikidata.org","stream":"rdf-streaming-updater.state-inconsistency"}	Q115608572	2023-01-12T00:38:51.434637Z	2023-01-12T00:36:45.418Z	1809652184	{"_schema":null,"meta":{"uri":null,"request_id":"8b2822eb-2c55-478e-bbb9-dbd70a2ebadb","id":"3f1a8d11-4dc2-49a3-b536-80a532d11d68","dt":"2023-01-12T00:36:44Z","domain":"www.wikidata.org","stream":"mediawiki.revision-create"},"dt":"2023-01-12T00:36:44Z"}	revision-create	1809650380	newer_revision_seen	1785111610	DELETED	true	{"project_class":"wikidata","project":"-","qualifiers":[],"tld":"org","project_family":"wikidata"}	codfw	2023	1	12	0
NULL	{"uri":null,"request_id":null,"id":"a1682210-27d0-43aa-b73a-dab169db6d09","dt":"2023-01-12T00:29:48.706526Z","domain":"www.wikidata.org","stream":"rdf-streaming-updater.state-inconsistency"}	Q115608572	2023-01-12T00:29:48.706271Z	2023-01-12T00:27:47.489Z	1809647736	{"_schema":null,"meta":{"uri":null,"request_id":"9189d7bf-6a8c-4076-a20e-d6f2b19223a1","id":"db3a7565-4f37-45cd-b8c7-995bb123c23c","dt":"2023-01-12T00:27:47Z","domain":"www.wikidata.org","stream":"mediawiki.revision-create"},"dt":"2023-01-12T00:27:47Z"}	revision-create	1809646313	newer_revision_seen	1785111610	DELETED	true	{"project_class":"wikidata","project":"-","qualifiers":[],"tld":"org","project_family":"wikidata"}	codfw	2023	1	12	0
NULL	{"uri":null,"request_id":null,"id":"e3f7a0ec-e592-466a-8cb0-4762db119ab5","dt":"2023-01-12T00:34:46.957749Z","domain":"www.wikidata.org","stream":"rdf-streaming-updater.state-inconsistency"}	Q115608572	2023-01-12T00:34:46.957575Z	2023-01-12T00:32:02.218Z	1809649627	{"_schema":null,"meta":{"uri":null,"request_id":"8ded7e5c-c8e0-47f3-a798-fb963434d36c","id":"d79048d7-ce32-44b8-ae3e-7112ef988b0a","dt":"2023-01-12T00:32:01Z","domain":"www.wikidata.org","stream":"mediawiki.revision-create"},"dt":"2023-01-12T00:32:01Z"}	revision-create	1809647736	newer_revision_seen	1785111610	DELETED	true	{"project_class":"wikidata","project":"-","qualifiers":[],"tld":"org","project_family":"wikidata"}	codfw	2023	1	12	0
NULL	{"uri":null,"request_id":null,"id":"194c0316-111c-4510-82c1-29b6002d88d7","dt":"2023-01-12T00:34:46.959180Z","domain":"www.wikidata.org","stream":"rdf-streaming-updater.state-inconsistency"}	Q115608572	2023-01-12T00:34:46.959053Z	2023-01-12T00:32:19.613Z	1809649800	{"_schema":null,"meta":{"uri":null,"request_id":"a154604e-a1f3-493f-8747-2ef96e508fbf","id":"89284866-1c87-4fed-9bf3-e3c3cf923997","dt":"2023-01-12T00:32:19Z","domain":"www.wikidata.org","stream":"mediawiki.revision-create"},"dt":"2023-01-12T00:32:19Z"}	revision-create	1809649627	newer_revision_seen	1785111610	DELETED	true	{"project_class":"wikidata","project":"-","qualifiers":[],"tld":"org","project_family":"wikidata"}	codfw	2023	1	12	0
NULL	{"uri":null,"request_id":null,"id":"a98d0811-9d1c-45b8-b139-0d39afc06d18","dt":"2023-01-12T00:42:09.402992Z","domain":"www.wikidata.org","stream":"rdf-streaming-updater.state-inconsistency"}	Q115608572	2023-01-12T00:42:09.402405Z	2023-01-12T00:39:52.536Z	1809653947	{"_schema":null,"meta":{"uri":null,"request_id":"d019b288-3a13-4024-a934-3d4db9b63852","id":"e876202c-be0d-4894-87d0-fab7fc1893a8","dt":"2023-01-12T00:39:52Z","domain":"www.wikidata.org","stream":"mediawiki.revision-create"},"dt":"2023-01-12T00:39:52Z"}	revision-create	1809653618	newer_revision_seen	1785111610	DELETED	true	{"project_class":"wikidata","project":"-","qualifiers":[],"tld":"org","project_family":"wikidata"}	codfw	2023	1	12	0
NULL	{"uri":null,"request_id":null,"id":"f0e648e0-59b1-4569-b62e-c111e75cc178","dt":"2023-01-12T00:42:09.406318Z","domain":"www.wikidata.org","stream":"rdf-streaming-updater.state-inconsistency"}	Q115608572	2023-01-12T00:42:09.406068Z	2023-01-12T00:39:56.528Z	1809653984	{"_schema":null,"meta":{"uri":null,"request_id":"313b92c9-500e-4b1d-8feb-913ed2ce7098","id":"6a746531-76a0-48ff-954f-33c118b6cdc9","dt":"2023-01-12T00:39:56Z","domain":"www.wikidata.org","stream":"mediawiki.revision-create"},"dt":"2023-01-12T00:39:56Z"}	revision-create	1809653947	newer_revision_seen	1785111610	DELETED	true	{"project_class":"wikidata","project":"-","qualifiers":[],"tld":"org","project_family":"wikidata"}	codfw	2023	1	12	0
NULL	{"uri":null,"request_id":null,"id":"130d0162-65ba-4bd7-be5d-4828ce278e09","dt":"2023-01-12T00:26:47.884125Z","domain":"www.wikidata.org","stream":"rdf-streaming-updater.state-inconsistency"}	Q115608572	2023-01-12T00:26:47.883671Z	2023-01-12T00:24:46.864Z	1809646313	{"_schema":null,"meta":{"uri":null,"request_id":"883d110b-61ba-43c1-b510-82a62b5c5cd4","id":"2052e478-bc42-44b3-870a-a276975b96ce","dt":"2023-01-12T00:24:46Z","domain":"www.wikidata.org","stream":"mediawiki.revision-create"},"dt":"2023-01-12T00:24:46Z"}	revision-create	1809645267	newer_revision_seen	1785111610	DELETED	true	{"project_class":"wikidata","project":"-","qualifiers":[],"tld":"org","project_family":"wikidata"}	codfw	2023	1	12	0
NULL	{"uri":null,"request_id":null,"id":"79900880-73f1-40cb-9691-e59d1cd1ef8d","dt":"2023-01-12T00:21:02.246341Z","domain":"www.wikidata.org","stream":"rdf-streaming-updater.state-inconsistency"}	Q115608572	2023-01-12T00:21:02.246120Z	2023-01-12T00:18:54.174Z	1785111610	{"_schema":null,"meta":{"uri":null,"request_id":"ab00575f-7a12-43cb-9f73-a50bf7e98540","id":"f2cee9ac-db27-4393-9978-985b14db0c29","dt":"2023-01-12T00:18:53Z","domain":"www.wikidata.org","stream":"mediawiki.revision-create"},"dt":"2023-01-12T00:18:53Z"}	revision-create	1785108184	newer_revision_seen	1785111610	DELETED	true	{"project_class":"wikidata","project":"-","qualifiers":[],"tld":"org","project_family":"wikidata"}	codfw	2023	1	12	0
NULL	{"uri":null,"request_id":null,"id":"242e1923-36a9-4bf1-8104-d9d737427082","dt":"2023-01-12T00:41:42.180087Z","domain":"www.wikidata.org","stream":"rdf-streaming-updater.state-inconsistency"}	Q115608572	2023-01-12T00:41:42.179762Z	2023-01-12T00:39:36.675Z	1809653618	{"_schema":null,"meta":{"uri":null,"request_id":"8426bbf9-8bae-4d03-8ab7-368c387b4fb3","id":"0fe708d9-21b3-436b-b8e8-59cdfa266abc","dt":"2023-01-12T00:39:36Z","domain":"www.wikidata.org","stream":"mediawiki.revision-create"},"dt":"2023-01-12T00:39:36Z"}	revision-create	1809653202	newer_revision_seen	1785111610	DELETED	true	{"project_class":"wikidata","project":"-","qualifiers":[],"tld":"org","project_family":"wikidata"}	codfw	2023	1	12	0
NULL	{"uri":null,"request_id":null,"id":"b1bab8e7-2941-4a50-b719-934d91eb6e0d","dt":"2023-01-12T00:40:42.860127Z","domain":"www.wikidata.org","stream":"rdf-streaming-updater.state-inconsistency"}	Q115608572	2023-01-12T00:40:42.859739Z	2023-01-12T00:38:38.662Z	1809653202	{"_schema":null,"meta":{"uri":null,"request_id":"e70af5d6-6fde-4de7-9438-c216d0ec4e11","id":"ff9c5ed9-4a98-4a90-b683-98c588539072","dt":"2023-01-12T00:38:38Z","domain":"www.wikidata.org","stream":"mediawiki.revision-create"},"dt":"2023-01-12T00:38:38Z"}	revision-create	1809652184	newer_revision_seen	1785111610	DELETED	true	{"project_class":"wikidata","project":"-","qualifiers":[],"tld":"org","project_family":"wikidata"}	codfw	2023	1	12	0
Time taken: 0.253 seconds, Fetched: 14 row(s)

So events are properly flowing in the rdf-streaming-updater.state-inconsistency stream.

Feb 7 2023, 5:21 PM · Discovery-Search (Current work), Wikidata, Wikidata-Query-Service
dcausse updated the task description for T329089: The rdf-streaming-updater does not reconcile missed page-undelete events.
Feb 7 2023, 5:05 PM · Discovery-Search (Current work), Wikidata, Wikidata-Query-Service
dcausse created T329089: The rdf-streaming-updater does not reconcile missed page-undelete events.
Feb 7 2023, 4:54 PM · Discovery-Search (Current work), Wikidata, Wikidata-Query-Service

Feb 6 2023

dcausse added a comment to T328478: Refactor parameterization of eventutilities-python and mediawiki-event-enrichment.

Interesting, and let the helm dict merging of e.g. config_files.my_app_config.content handle the creation of merged config files? Then the app doesn't do any fancy config merging itself?

Might be a simple solution. Too bad Java doesn't have a better config_file + CLI opt parser. I suppose Java Properties kind of work, but then setting them on CLI with -D'...' is a little annoying.

This does mean that all flink apps that use this need to know how to read a single config file, right?

Feb 6 2023, 5:07 PM · Event-Platform Value Stream (Sprint 11), Data-Engineering-Planning
dcausse moved T325273: EventRowTypeInfo should support schema evolution of rows seriliazed in flink application state from To Be Deployed to Needs Reporting on the Discovery-Search (Current work) board.
Feb 6 2023, 4:30 PM · Data-Engineering-Planning, Discovery-Search (Current work), CirrusSearch, Event-Platform Value Stream
dcausse moved T323690: Add support page deletion from To Be Deployed to Needs Reporting on the Discovery-Search (Current work) board.
Feb 6 2023, 4:28 PM · Discovery-Search (Current work), CirrusSearch
dcausse moved T323690: Add support page deletion from Needs review to To Be Deployed on the Discovery-Search (Current work) board.
Feb 6 2023, 4:27 PM · Discovery-Search (Current work), CirrusSearch
dcausse moved T325273: EventRowTypeInfo should support schema evolution of rows seriliazed in flink application state from Needs review to To Be Deployed on the Discovery-Search (Current work) board.
Feb 6 2023, 4:27 PM · Data-Engineering-Planning, Discovery-Search (Current work), CirrusSearch, Event-Platform Value Stream
dcausse added a comment to T328478: Refactor parameterization of eventutilities-python and mediawiki-event-enrichment.

I wish that the flink-app chart provided some tooling to help with that.

But yeah, I see what you are saying. Even if we had both config files and CLI opt overrides, Because the JobSpec args won't be merged in any way, there is going to repetition.

I think we could provide a flink-app helper template to make this easier. It would probably have to be opinionated about whatever common arg parser we'd like to use for Flink app code. This might be a good reason to rely on a JVM based one for python apps too (unfortunetly). Maybe we should try refinery-sources ConfigHelper? Oof, its scala though, so I dunno?

Feb 6 2023, 4:01 PM · Event-Platform Value Stream (Sprint 11), Data-Engineering-Planning

Feb 3 2023

dcausse added a comment to T328276: Add outlink topic model predictions to CirrusSearch indices.

@dcausse - does the current pipeline use the union of the predictions from articletopic and drafttopic model? or how does it aggregate the two predictions for the same article?

Currently the two set of predictions are kept and the user can use one or the other: searching for articletopic:biography vs searching for drafttopic:biography. As far I understood the drafttopic keyword was requested for searching by topic on the Draft (T249341) namespace for which the articletopic model does not work. If the outlink model is not enabled on the Draft namespace it might make to leave this problem out for now and not think too much about how to replace the ORES drafttopic model yet.

Feb 3 2023, 4:29 PM · Machine-Learning-Team, Discovery-Search, CirrusSearch
dcausse added a comment to T328478: Refactor parameterization of eventutilities-python and mediawiki-event-enrichment.

I think it's important for the flink-app chart to have a way to pass and/or populate a config_file, or have a dedicated feature for passing the script options, I was messing around with the args of the jobspec but I think it might be hard and messy to apply the various values-release.yaml -> values-$dc.yaml -> values.yaml overrides with a plain yaml array... I don't think also that you can define your own template once you're defining a helmfile.d service.

Not sure I follow.
Would this config file provide application only options, or also contain options for kafka/flink?

Feb 3 2023, 2:26 PM · Event-Platform Value Stream (Sprint 11), Data-Engineering-Planning

Feb 2 2023

dcausse added a comment to T293063: Write and adapt Runbooks and cookbooks related to the WDQS Streaming Updater and kubernetes.

Anyhow. AIUI this process will be more or less the same for flink deployments managed by the flink operator. It would be nice if you could verify this during your tests with the operator (I'm happy to help/pair ofc.) or if there maybe even is a better option in flink-operator world.

Feb 2 2023, 4:27 PM · Prod-Kubernetes, wdwb-tech, serviceops, Wikidata, Wikidata-Query-Service
dcausse added a comment to T293063: Write and adapt Runbooks and cookbooks related to the WDQS Streaming Updater and kubernetes.

Hey @dcausse, I'm reading this again because of the upcoming k8s 1.23 upgrade and was wondering:
In "To restore:" section of "Alternate actions (not fully untested):" - do we need to start the job somehow as well, specifying which jar file to use? Or is that information part of the configmaps/safepoint and the job can start automatically without submitting a jar?

Feb 2 2023, 4:13 PM · Prod-Kubernetes, wdwb-tech, serviceops, Wikidata, Wikidata-Query-Service
dcausse updated the task description for T293063: Write and adapt Runbooks and cookbooks related to the WDQS Streaming Updater and kubernetes.
Feb 2 2023, 4:10 PM · Prod-Kubernetes, wdwb-tech, serviceops, Wikidata, Wikidata-Query-Service
dcausse updated the task description for T326409: Migrate the wdqs streaming updater flink jobs to flink-k8s-operator deployment model.
Feb 2 2023, 3:08 PM · wdwb-tech, serviceops-radar, Discovery-Search (Current work), Wikidata, Wikidata-Query-Service
dcausse updated the task description for T326409: Migrate the wdqs streaming updater flink jobs to flink-k8s-operator deployment model.
Feb 2 2023, 3:05 PM · wdwb-tech, serviceops-radar, Discovery-Search (Current work), Wikidata, Wikidata-Query-Service
dcausse added a subtask for T326409: Migrate the wdqs streaming updater flink jobs to flink-k8s-operator deployment model: T328675: Create a dse-k8s service demonstrating how to run the rdf-streaming-updater using the flink-app chart.
Feb 2 2023, 3:04 PM · wdwb-tech, serviceops-radar, Discovery-Search (Current work), Wikidata, Wikidata-Query-Service
dcausse added a parent task for T328675: Create a dse-k8s service demonstrating how to run the rdf-streaming-updater using the flink-app chart: T326409: Migrate the wdqs streaming updater flink jobs to flink-k8s-operator deployment model.
Feb 2 2023, 3:04 PM · Patch-For-Review, Discovery-Search (Current work), Wikidata, Wikidata-Query-Service
dcausse created T328675: Create a dse-k8s service demonstrating how to run the rdf-streaming-updater using the flink-app chart.
Feb 2 2023, 3:03 PM · Patch-For-Review, Discovery-Search (Current work), Wikidata, Wikidata-Query-Service
dcausse added a comment to T328478: Refactor parameterization of eventutilities-python and mediawiki-event-enrichment.

From the point of view of a k8s and helm operator, are there best practices we should follow for structuring configs (e.g. config files vs CLI opts)?

I don't think so. Config files are nice because they are easier to read at a glance, but, for varying things, debugging, having to edit config files can be a pain. Being able to do both via ConfigHelper has been really really nice!

I think it's important for the flink-app chart to have a way to pass and/or populate a config_file, or have a dedicated feature for passing the script options, I was messing around with the args of the jobspec but I think it might be hard and messy to apply the various values-release.yaml -> values-$dc.yaml -> values.yaml overrides with a plain yaml array... I don't think also that you can define your own template once you're defining a helmfile.d service.

Feb 2 2023, 8:51 AM · Event-Platform Value Stream (Sprint 11), Data-Engineering-Planning
dcausse updated subscribers of T328478: Refactor parameterization of eventutilities-python and mediawiki-event-enrichment.

ParameterTool is nice but @pfischer has found a really nice abstraction on top of it, you define your options using a set of ConfigOption, the nice thing is that they're strongly typed and can be used with ParameterTools#getConfiguration().
ParameterTools can be constructed from argv or a properties file. Note it does seem to only support Map<String, String> so I'd avoid using yaml as the default format but rather use the simpler java property files format, they should be compatible tho in a way that you can easily use something like {{- toYaml .Values.app.config_file.options }} template function to generate it.

Feb 2 2023, 8:39 AM · Event-Platform Value Stream (Sprint 11), Data-Engineering-Planning

Feb 1 2023

dcausse updated subscribers of T328561: Flink Operations.
Feb 1 2023, 3:31 PM · Data-Engineering-Planning, Epic, Event-Platform Value Stream
dcausse awarded T328561: Flink Operations a Love token.
Feb 1 2023, 3:30 PM · Data-Engineering-Planning, Epic, Event-Platform Value Stream