Noticed a few of these today all from a single wiki. It's reproducible a GET to the URL in logstash.

Adding this as a train blocker as it's:

(meta note: tagged Parsoid as I'm not sure of the right tag for Content Transformation, please correct if I got it wrong)

Seeing the Query execution was interrupted increase (as expected, probably) in the logs since this rolled out.

I went through logstash to find a requestid from your logs, it looks like the error on the backend (for one of your errors anyway) is Error 1969: Query execution was interrupted (max_statement_time exceeded) (db1184) this may be related to this email sent to wikitech-l https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/thread/IPJNO75HYAQWIGTHI5LJHTDVLVOC4LJP/

Adding @Zabe and @Majavah as I see some recent updates to GlobalBlock (5000c74c8b0986967f3e927f67790d5ce126a5af and 9b92b5e93702fc53bbf53e5a9bb2bcde568f09a9) — no idea if they're related to this.

Saw a handful of these on commonswiki today, the stacktrace has shifted around a bit, pasting here in case it's helpful.

In T298659#7600064, @matmarex wrote:

The exception is probably caused by rELUA602cef87e06d: mw.title: Add pageLanguage property (https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Scribunto/+/346176). If there's no better fix, that can probably be reverted.

Ran a recheck on a core patch. This looks resolved.

I just did a recheck and it seems to catch it on contint2001, and I see zuul processing the queue in /var/log/zuul/debug.log and I see things merging in /var/log/zuul/merger-debug.log. Though there's not much running through it at the moment.

done !

In T297323#7565793, @MatthewVernon wrote:

@thcipriani are you OK to approve this request, please? [or suggest someone else in releng who might be appropriate to do so?]

1. Training completed!

In T297621#7569114, @MatthewVernon wrote:

@thcipriani I think you're the right person to approve additions to the restricted group; can you confirm you're happy for this to go ahead, please?

Documenting my understanding of this problem after reading this task (along with T297669 and T297667)).

noting T296098: 1.38.0-wmf.9 seems to have introduced a memory leak as possibly related

Hi @DMartin-WMF I've added you to the backport training on Thursday, December 16. See you then!

Declaring https://versions.toolforge.org/ to be a reasonable approximation of the acceptance criteria of this task.

Hi @sbassett

It seems we have all that's in the task description and more, now: https://gitlab.wikimedia.org/thcipriani/train-stats#train-stats

In T293953#7553328, @ssastry wrote:

In T293953#7551832, @thcipriani wrote:

If it's possible to mitigate this, that'd be ideal.

We are working to take Parsoid out of the risky patches set and should have a backport to vendor before the train rolls out. We decided to split the 2.3.0 -> 2.4.0 bump across 2 trains. That will eliminate the need for RESTBase purges altogether without needing to keep any eye on individual edits on those 3 wikis.

In T293953#7551761, @ssastry wrote:

All that said, if train operators prefer that we mitigate that for your sanity, I will happily work with my team to make those fixes.

I made a script that parses all Jenkins junit files and puts them in a database. The database (see here) has about 5 million test cases at this point.

In T294311#7507348, @sbassett wrote:

In T294311#7505539, @thcipriani wrote:

2. We have a node14 image; that's only meant for use in CI, currently

Ah, ok. Is there anything special about images under the releng/ namespace? i.e. does that signify they are only for CI?

In T293953#7550675, @ssastry wrote:

Just so this doesn't get lost in the comment above, if the train is rolled back after deploy, we will need to purge RESTBase content (See T296425) to ensure VE edits on metawiki and mediawiki on pages with <translate> tags don't break.

This is admittedly an edge case since VE editing on pages with <translate> isn't well-supported currently and so editors are unlikely to use VE on those pages. So, if a train is going to be rolled back and then rolled forward relatively quickly, there is probably some breathing room to let that be in a broken state for a while. But, if the train is going to be in a rolled back state for a long time, then, it is worth purging RESTBase storage via T296425.

1. Training completed!

In T296777#7538923, @Dzahn wrote:

@thcipriani Thoughts on this request?

The puppet failure on the second deployment host is:

In T296024#7516616, @JKieserman wrote:

Thank you! Would it be possible to attend the earlier session instead?

Unassigning now that this is rolled back.