@bking ack! I'll CC you in code review.

Hey @bking. Wikidata Platform is monitoring the search mailing list for WDQS alerts.

In T409781#11437042, @Pfps wrote:

It's well past 11/21. Has there been any progress?

This is great @bking !

Sounds reasonable to me.

In T410635#11395697, @Ifrahkhanyaree_WMDE wrote:

Ah wow okay, that helps give me more context. Would you have recommendations then, what might be a good enough heuristic to look for queries that time out?

In T409769#11373497, @Gehel wrote:

Do you already have an idea of the size of SSD that you need? Looking at current servers, I suspect you'll need at least 1TB, probably 2TB?

In T409769#11375265, @bking wrote:

However, would everyone be OK with not ordering hardware specifically for WDQS until we have more of a discussion? Here's my reasoning:

In T408216#11324303, @elukey wrote:

So you may be ready to go, lemme know if you are still blocked and I'll take a look!

instead of passing the date that would used in a directory name for the output file we would see it more clear to have a way to specify the entire directory/path to store the resulting dump under. This would make the location of the resulting dump even more explicit and predictable (/someplace/wikibase/wikidatawiki/DATE - there's more than a date here)

In T404161#11272392, @bking wrote:

@gmodena can you confirm access? I'm going to move to 'needs review', feel free to close out if everything is working OK.

I was able to log in and get started. Quick question: are there any constraints on which regions I can launch instances in?

We are looking into AWS as a possible way to bootstrap experiments while we wait for on prem hardware. Given that we are memory bound, we are considering these tow high-memory instances as a baseline:

Thanks for the context @dcausse. This is incredibly useful.

The WDQS service had an outage overnight (CEST) that has since been resolved.

My assumptions re fiddling with timeouts came from the link the logs point us too.
But I don't know this system well enough to meaningfully troubleshoot.

I manually cleared the stuck tasks, and the dag is now catching up.

In T374939#10176353, @Ottomata wrote:

Not sure this is helpful, but:

https://gerrit.wikimedia.org/r/plugins/gitiles/wikimedia-event-utilities/+/refs/heads/master/eventutilities-flink/src/main/java/org/wikimedia/eventutilities/flink/stream/SSESourceFunction.java

In T404340#11192631, @dcausse wrote:

In T404340#11191436, @gmodena wrote:

We need to spike readyness of eventutilities_python to adopt Java17. In _theory_ there should not be significant bytecode changes in our code paths. In practice I would not consider this transition entirely risk free.

Sure, added a bullet point regarding eventutilities_python, framed it as just "testing" that it can run with java17, could possibly be done by adding an extra CI step with a java17 image?

We need to spike readyness of eventutilities_python to adopt Java17. In _theory_ there should not be significant bytecode changes in our code paths. In practice I would not consider this transition entirely risk free.

The Wikidata Platform team will eventually onboard and refactor this application to make it orthogonal to blazegraph / other flink appw.

In T404073#11172541, @BTullis wrote:

Hi @gmodena - this all sounds fine, but I have a question about point 3:

A new Airflow DAGs monorepo setup for their pipelines.

All of our other Airflow instances share the same airflow-dags git repository, and we have a subdirectory per team.
Is there a reason why this shared repository model wouldn't work for the Wikidata Platform team?

We might be able to shave a few seconds off by tuning the query, but my suggestion would be to use APIs that better fit the access pattern.

In T396890#11164610, @Gehel wrote:

Let's see first if the Wikidata Platform team can understand why those queries are slow (slower than before graph split?). And if we need to do anything about it.

Reposting here for visibility:

In T402959#11132802, @CDanis wrote:

Hi @Lydia_Pintscher , SRE can make some exception here. It seems warranted given the status quo in the broader sparql ecosystem.

In T347282#10990516, @gmodena wrote:

Re-opening. We saw a data leak on mw-page-content-change-enrich that correlates with when the patch was deployed.

Regarding the alert linked in OP.

Here is a summary of a discussion we had on DPE's internal slack.

Re-opening to investigate memory alloc patterns / leaks that we encountered with mw-page-content-change-enrich post T347282: [Event Platform] eventutilites-python: improve consistency guarantees of async process functions deployment.

In T397330#10948116, @Ahoelzl wrote:

related to spikes observed here https://phabricator.wikimedia.org/T397854 ?

Re-opening. We saw a data leak on mw-page-content-change-enrich that correlates with when the patch was deployed.