User Details
- User Since
- Oct 8 2014, 7:09 PM (396 w, 5 d)
- Availability
- Available
- IRC Nick
- apergos
- LDAP User
- ArielGlenn
- MediaWiki User
- ArielGlenn [ Global Accounts ]
Mon, May 9
Thu, May 5
@Hokwelum and I looked at this together. This is indeed another case of a bab blob address in the text table, as is clear just from looking at the logstash output for the specific request, i.e.
ExternalStoreDB::fetchBlob: primary DB fallback on cluster20/0@hrwiki
Tue, May 3
Mon, Apr 25
Wed, Apr 20
Apr 14 2022
Uh... I don't know what you meant to change the time slot to, but in any case, I was there, you weren't, but no patches were scheduled either. 🤷
Apr 11 2022
Apr 8 2022
Apr 7 2022
No training today because there were no patches, probably due to the gerrit maintenance window set for the same time. Let's reschedule this for next week.
Daniel did indeed show up, get a refresher training, and do the deploy. All is well!
Apr 5 2022
Pinging @Atieno here for manager approval.
Mar 28 2022
Mar 23 2022
Everything ran to completion, so this is fixed. Thanks for the help, folks!
Mar 21 2022
Setting this to medium priority now (did I not set it to ubn initially? Oh well), since it is likely fixed.
The cherry pick for the third branch (.2) will go when the train for .2 runs, as that branch is not yet staged, deployment is not possible or needed.
Adding @Reedy for awareness.
This is caused by https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Flow/+/770072
I verfied this in testing in deployment-prep: as dumpsgen user on deployment-dnapshot03, run
/usr/bin/php7.2 /srv/mediawiki/multiversion/MWScript.php extensions/Flow/maintenance/dumpBackup.php --wiki=ruwiki --output=file:/tmp/junk.txt https://phabricator.wikimedia.org/T304318
Mar 18 2022
Mar 11 2022
As @TheresNoTime mentioned, uk is the right language code. We have a handly list of the languages and their language codes here: https://meta.wikimedia.org/wiki/Special:SiteMatrix
Mar 4 2022
All projects are available via the web and internally to WMCS projects; the manual download and rsync are complete. Fixes to the script will be coming in a few days.
Mar 3 2022
I'm running a backfill manually from labstore1006, logged in as ariel, this is running as the dumpsgen user. I'll check in on it at the end of the day and again tomorrow to make sure it completes. After that we'll rsync it over. This will give some breathing room until I have time to see why the script didn't retry properly (or whether we need to sleep longer between retries, etc).
Hey jsut a note that we saw another failure:
Output of systemd timer for '/usr/local/bin/dumpwikibaserdf.sh -p wikidata -d truthy -f nt'
Mar 2 2022
Thanks a lot, using it now :-) :-)
Mar 1 2022
I've had a look at the image dujmps across all the wikis (in lieu of running expensive queries against the database for all of them) and have found a small (a few hundred_ number of files with the same problem, mostly pdfs. I'll look into it a little further to see if they have something in common, and post the results here. If there is some common bug that is responsible for this (for example, they were all uploaded aorund the same time), then it's possible someone could write a maintenance script to fix the entries for that period of time. I would definitely not be comfortable myself just sticking my hand in there and updating isolated fields manually.
Feb 28 2022
Thanks for the report!
Feb 24 2022
Followup: Training happened, got the full experience: CI broken, first patch failed and revert necessary. Lots of learning opportunities :-)
Followup: Training happened, got the full experience: CI broken, first patch failed and revert necessary. Lots of learning opportunities :-)
Feb 23 2022
Flow jobs running properly this run, so I can close this out. At some ploint it would be nice to look at the actual memory issues in the Flow extension but since no one owns that...
/srv/mediawiki-staging/php-master on deployment-deploy03 looks up to date now when I compare to a local pull of mediawiki-core. Maybe this has been fixed by some kind soul or has fixed itself?
Feb 22 2022
Hey Tyler and other folks, this is our co-dumps-maintainer in training, so to speak, so I'm getting her trained in all the things. She'll probably come to a few training sessions, though only one is I guess required. You know the drill, get trained early and often!
I am aware of and following this discussion but right now, my responsiveness on this task will be slow, most of my time needs to go to getting my teammate who will be dumps co-maintainer up to speed. Please bear with us.
So... what's happening with these? Do we have some sort of schedule?
The new run is complete for almost all wikis and the filenames look like they should, even if I don't love the duplicate "v2" in there either. So I'm going to close this out. We can look at how jobs and classes are named sometime in the glorious future when the whole infra for dumps gets rewritten :-)
Feb 21 2022
Hannah is working with me on dumps and will need access to all the usual things. CCing @Atieno as direct manager for approval.
I would have preferred siteinvofv2-namespaces. But we can't do that, and at this point things are set. The problem is probably relying on the job name as part of the filename, in the specific way that we do, and it's definitely too late to change that.
Feb 19 2022
Feb 18 2022
Thanks for closing, and Hannah tested access today and it works like a charm :-)
Today's report:
<13>Feb 15 19:09:41 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20220214/enwiki-20220214-cirrussearch-content.json.gz
Feb 16 2022
shell account name: hokwelum
Feb 15 2022
I have put in place the flow dumps files for frwiki and wikidatawiki and the noop jobs are running now. Everything should show up sometime tomorrow on the public servers, fingers crossed.
Feb 14 2022
https://gerrit.wikimedia.org/r/c/mediawiki/core/+/762448 merged by Pchelko (thanks!), looking good in deployment-prep for Flow and for all the other dump jobs too.