In T265056#7904793, @Gehel wrote:

It looks like those dumps are very rarely accessed externally (1 hit over the last 30 days according to turnilo - webrequest_sampled_128).

@Hokwelum and I looked at this together. This is indeed another case of a bab blob address in the text table, as is clear just from looking at the logstash output for the specific request, i.e.

ExternalStoreDB::fetchBlob: primary DB fallback on cluster20/0@hrwiki

Hey @bd808, to have the dumps folks (plural!) on the task watching, you can either tag with the project tag or add also @Hokwelum so we both get notifications.

Uh... I don't know what you meant to change the time slot to, but in any case, I was there, you weren't, but no patches were scheduled either. 🤷

In T138208#7844298, @Ladsgroup wrote:

It's a bit hard to measure but it's probably fixed.

No training today because there were no patches, probably due to the gerrit maintenance window set for the same time. Let's reschedule this for next week.

Daniel did indeed show up, get a refresher training, and do the deploy. All is well!

Pinging @Atieno here for manager approval.

In T57503#7809889, @Kelson wrote:

@ArielGlenn Sorry, I meant T286588

In T57503#7809873, @Kelson wrote:

@ArielGlenn Hi, I already come back to you! T299993 is hidden to me and I have no visibility on it. Is that already implemented. If "no", in which timeline are we moving in?

Everything ran to completion, so this is fixed. Thanks for the help, folks!

Setting this to medium priority now (did I not set it to ubn initially? Oh well), since it is likely fixed.

The cherry pick for the third branch (.2) will go when the train for .2 runs, as that branch is not yet staged, deployment is not possible or needed.

Adding @Reedy for awareness.

This is caused by https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Flow/+/770072
I verfied this in testing in deployment-prep: as dumpsgen user on deployment-dnapshot03, run

/usr/bin/php7.2 /srv/mediawiki/multiversion/MWScript.php extensions/Flow/maintenance/dumpBackup.php --wiki=ruwiki --output=file:/tmp/junk.txt  https://phabricator.wikimedia.org/T304318

As @TheresNoTime mentioned, uk is the right language code. We have a handly list of the languages and their language codes here: https://meta.wikimedia.org/wiki/Special:SiteMatrix

All projects are available via the web and internally to WMCS projects; the manual download and rsync are complete. Fixes to the script will be coming in a few days.

I'm running a backfill manually from labstore1006, logged in as ariel, this is running as the dumpsgen user. I'll check in on it at the end of the day and again tomorrow to make sure it completes. After that we'll rsync it over. This will give some breathing room until I have time to see why the script didn't retry properly (or whether we need to sleep longer between retries, etc).

Hey jsut a note that we saw another failure:

Output of systemd timer for '/usr/local/bin/dumpwikibaserdf.sh -p wikidata -d truthy -f nt'

Thanks a lot, using it now :-) :-)

I've had a look at the image dujmps across all the wikis (in lieu of running expensive queries against the database for all of them) and have found a small (a few hundred_ number of files with the same problem, mostly pdfs. I'll look into it a little further to see if they have something in common, and post the results here. If there is some common bug that is responsible for this (for example, they were all uploaded aorund the same time), then it's possible someone could write a maintenance script to fix the entries for that period of time. I would definitely not be comfortable myself just sticking my hand in there and updating isolated fields manually.

Thanks for the report!

Followup: Training happened, got the full experience: CI broken, first patch failed and revert necessary. Lots of learning opportunities :-)

Followup: Training happened, got the full experience: CI broken, first patch failed and revert necessary. Lots of learning opportunities :-)

Flow jobs running properly this run, so I can close this out. At some ploint it would be nice to look at the actual memory issues in the Flow extension but since no one owns that...

/srv/mediawiki-staging/php-master on deployment-deploy03 looks up to date now when I compare to a local pull of mediawiki-core. Maybe this has been fixed by some kind soul or has fixed itself?

Hey Tyler and other folks, this is our co-dumps-maintainer in training, so to speak, so I'm getting her trained in all the things. She'll probably come to a few training sessions, though only one is I guess required. You know the drill, get trained early and often!

I am aware of and following this discussion but right now, my responsiveness on this task will be slow, most of my time needs to go to getting my teammate who will be dumps co-maintainer up to speed. Please bear with us.

So... what's happening with these? Do we have some sort of schedule?

The new run is complete for almost all wikis and the filenames look like they should, even if I don't love the duplicate "v2" in there either. So I'm going to close this out. We can look at how jobs and classes are named sometime in the glorious future when the whole infra for dumps gets rewritten :-)

Hannah is working with me on dumps and will need access to all the usual things. CCing @Atieno as direct manager for approval.

I would have preferred siteinvofv2-namespaces. But we can't do that, and at this point things are set. The problem is probably relying on the job name as part of the filename, in the specific way that we do, and it's definitely too late to change that.

Thanks for closing, and Hannah tested access today and it works like a charm :-)

Today's report:

<13>Feb 15 19:09:41 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20220214/enwiki-20220214-cirrussearch-content.json.gz

In T301876#7714786, @WDoranWMF wrote:

As Platform Engineering manager, I approve.

shell account name: hokwelum

I have put in place the flow dumps files for frwiki and wikidatawiki and the noop jobs are running now. Everything should show up sometime tomorrow on the public servers, fingers crossed.

https://gerrit.wikimedia.org/r/c/mediawiki/core/+/762448 merged by Pchelko (thanks!), looking good in deployment-prep for Flow and for all the other dump jobs too.

In T300760#7707663, @kostajh wrote:

@ArielGlenn could you please confirm if https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-deploy-2022.02.14?id=uhNt-H4Biq-bA-EUP2u_ is related to the errors described here? I see it's generated on a snapshot host, and looks similar, but has a different stack trace.

	from /srv/mediawiki/php-1.38.0-wmf.21/extensions/Flow/includes/Repository/UserNameBatch.php(75)
#0 [internal function]: MWExceptionHandler::handleFatalError()
#1 {main}

If it's not related, I can open a new task.