Page MenuHomePhabricator

ArielGlenn (ariel)
User

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Monday

  • Clear sailing ahead.

User Details

User Since
Oct 8 2014, 7:09 PM (467 w, 3 d)
Availability
Available
IRC Nick
apergos
LDAP User
ArielGlenn
MediaWiki User
ArielGlenn [ Global Accounts ]

Recent Activity

Tue, Sep 19

ArielGlenn added a comment to T346279: [Spike] Figure out what are good indicators for dumps data quality.

Just a note that sometimes size/page count/visible rev count might go down, if a large batch of pages are deleted for e.g. copyvio (more likely to occur on a small wiki).

Tue, Sep 19, 4:36 AM · Dumps 2.0, Data Products (Sprint 01)

Wed, Sep 13

ArielGlenn added a comment to T336573: PHP Warning: XMLReader::read(): Memory allocation failed : growing input buffer.

We could make sure that for commonswiki, the setting config "sevenzipprefetch" is 0. I'll need to check that this is one of the settings that can be overriden, and that the code will recognize 0 as a 'false' value. This should get done before next month's full run.

Wed, Sep 13, 7:32 AM · Unstewarded-production-error, Dumps-Generation, Wikimedia-production-error
ArielGlenn moved T341045: Get Data Engineering folks access to hosts and systems needed for maintenance of the existing dumps system from Active to Blocked/Stalled/Waiting for event on the Dumps-Generation board.
Wed, Sep 13, 7:28 AM · Data Engineering and Event Platform Team, Dumps-Generation
ArielGlenn added a comment to T341045: Get Data Engineering folks access to hosts and systems needed for maintenance of the existing dumps system.

There's one item in the checklist left before this task can be closed. And basically the holdup is just about getting the signoff from Tyler that the deployment trainings were completed; then we can get the rest of that item done.

Wed, Sep 13, 7:28 AM · Data Engineering and Event Platform Team, Dumps-Generation

Tue, Sep 12

ArielGlenn added a comment to T336573: PHP Warning: XMLReader::read(): Memory allocation failed : growing input buffer.

To expand on this a bit more: we saw the same error and stack trace on a slightly different page range, but with the identical symptoms. Logstash link here: https://logstash.wikimedia.org/goto/62b164dd91e2763a0a402d02087be836 Running the job hangs at the same point every time, even if nothing else is happening on the host; there aren't a particularly large number of revisions for the problem page, and their size isn't very large either. As before, using bz2 prfetch files permits the job to run to completion.

Tue, Sep 12, 9:31 AM · Unstewarded-production-error, Dumps-Generation, Wikimedia-production-error
ArielGlenn added a comment to T345874: XMLDumps broken on deployment-mwmaint02 due to Jade Extension related content.

So the patches went around and I checked that they are on snapshot03, but unfortunately I still see the error:

2023-09-12 05:20:33: enwiki (ID 14793) 683 pages (694.3|694.3/sec all|curr), 1000 revs (1016.6|1016.6/sec all|curr), ETA 2023-09-12 05:30:22 [max 600437]
MWUnknownContentModelException from line 192 of /srv/mediawiki/php-master/includes/content/ContentHandlerFactory.php: The content model 'JadeJudgment' is not registered on this wiki.
See https://www.mediawiki.org/wiki/Content_handlers to find out which extensions handle this content model.
#0 /srv/mediawiki/php-master/includes/content/ContentHandlerFactory.php(247): MediaWiki\Content\ContentHandlerFactory->validateContentHandler('JadeJudgment', NULL)
#1 /srv/mediawiki/php-master/includes/content/ContentHandlerFactory.php(181): MediaWiki\Content\ContentHandlerFactory->createContentHandlerFromHook('JadeJudgment')
#2 /srv/mediawiki/php-master/includes/content/ContentHandlerFactory.php(93): MediaWiki\Content\ContentHandlerFactory->createForModelID('JadeJudgment')
#3 /srv/mediawiki/php-master/includes/export/XmlDumpWriter.php(474): MediaWiki\Content\ContentHandlerFactory->getContentHandler('JadeJudgment')
#4 /srv/mediawiki/php-master/includes/export/XmlDumpWriter.php(402): XmlDumpWriter->writeSlot(Object(MediaWiki\Revision\SlotRecord), 1)
#5 /srv/mediawiki/php-master/includes/export/WikiExporter.php(554): XmlDumpWriter->writeRevision(Object(stdClass), Array)
#6 /srv/mediawiki/php-master/includes/export/WikiExporter.php(492): WikiExporter->outputPageStreamBatch(Object(Wikimedia\Rdbms\MysqliResultWrapper), Object(stdClass))
#7 /srv/mediawiki/php-master/includes/export/WikiExporter.php(316): WikiExporter->dumpPages('page_id >= 1900...', false)
#8 /srv/mediawiki/php-master/includes/export/WikiExporter.php(208): WikiExporter->dumpFrom('page_id >= 1900...', false)
#9 /srv/mediawiki/php-master/maintenance/includes/BackupDumper.php(355): WikiExporter->pagesByRange(190001, 195001, false)
#10 /srv/mediawiki/php-master/maintenance/dumpBackup.php(82): BackupDumper->dump(1, 1)
#11 /srv/mediawiki/php-master/maintenance/includes/MaintenanceRunner.php(685): DumpBackup->execute()
#12 /srv/mediawiki/php-master/maintenance/run.php(51): MediaWiki\Maintenance\MaintenanceRunner->run()
#13 /srv/mediawiki/multiversion/MWScript.php(159): require_once('/srv/mediawiki/...')
#14 {main}

Perhaps the override isn't being respected, or the usage isn't quite right?

Tue, Sep 12, 9:20 AM · Dumps-Generation, MediaWiki-ContentHandler, Beta-Cluster-Infrastructure

Fri, Sep 8

ArielGlenn added a comment to T345907: Alert for snapshot101[4567] not in mediawiki-installation dsh group.

the ops-dumps email alias ought to get notified about things like this; that way all the right people will see it.

Fri, Sep 8, 1:28 PM · Data-Platform-SRE, Dumps-Generation, Data-Engineering

Thu, Sep 7

ArielGlenn added a comment to T345874: XMLDumps broken on deployment-mwmaint02 due to Jade Extension related content.

Just a quick note that this breaks testing of dumps for enwiki in deployment-prep. We can work around it by testing only on other wikis, but it would be nice for this to be cleaned up.

Thu, Sep 7, 5:43 PM · Dumps-Generation, MediaWiki-ContentHandler, Beta-Cluster-Infrastructure
ArielGlenn added a project to T345874: XMLDumps broken on deployment-mwmaint02 due to Jade Extension related content: Dumps-Generation.
Thu, Sep 7, 5:42 PM · Dumps-Generation, MediaWiki-ContentHandler, Beta-Cluster-Infrastructure
ArielGlenn added a comment to T345186: Deployment training request for mabualruz.

We missed you today for the training. I'm guessing that something came up? Go ahead and reschedule, if you are still interested!

Thu, Sep 7, 7:42 AM · Release-Engineering-Team (Deployment Training Requests)

Thu, Aug 31

ArielGlenn moved T345176: {Investigation} Different file sizes for dumps from Backlog to Other teams on the Dumps-Generation board.
Thu, Aug 31, 10:01 AM · Wikimedia Enterprise, Dumps-Generation

Tue, Aug 29

ArielGlenn closed T344147: Puppet broken on mediawiki instances in deployment-prep as Resolved.

Thanks for the fix(es), everything is working as expected now.

Tue, Aug 29, 3:43 PM · User-ArielGlenn, Beta-Cluster-Infrastructure

Mon, Aug 28

ArielGlenn moved T143870: Some mw snapshot hosts are accessing main db servers from Blocked/Stalled/Waiting for event to Done on the Dumps-Generation board.
Mon, Aug 28, 9:57 AM · MW-1.35-notes (1.35.0-wmf.10; 2019-12-10), Dumps-Generation
ArielGlenn moved T295909: <Platform Initiative> Dumps + WME Gap Analysis from Blocked/Stalled/Waiting for event to Done on the Dumps-Generation board.
Mon, Aug 28, 9:57 AM · Platform Engineering, Dumps-Generation, Foundational Technology Requests
ArielGlenn moved T331129: cleanup_xmldumps is failing on dumpsdata1005 from Backlog to Done on the Dumps-Generation board.
Mon, Aug 28, 9:57 AM · Dumps-Generation
ArielGlenn moved T318849: analytics-dumps-fetch-unique_devices.service failing on dumps servers from Other teams to Done on the Dumps-Generation board.
Mon, Aug 28, 9:57 AM · Data-Engineering-Icebox, cloud-services-team, Analytics, Dumps-Generation
ArielGlenn moved T332562: Don't expose partial dumpfiles from Other teams to Done on the Dumps-Generation board.
Mon, Aug 28, 9:57 AM · Dumps-Generation, Wikimedia Enterprise
ArielGlenn moved T341058: Failures for Cirrus Search dumps for wikidatawiki, zhwiki, enwiki for the 20230703 run from Other teams to Done on the Dumps-Generation board.
Mon, Aug 28, 9:56 AM · Wikidata, MW-1.41-notes (1.41.0-wmf.19; 2023-07-25), Chinese-Sites, Dumps-Generation, Discovery-Search (Current work)
ArielGlenn added a comment to T336573: PHP Warning: XMLReader::read(): Memory allocation failed : growing input buffer.

Verified that with those same files from the above command the error is still present, nothing in the MW codebase has changed whatever the underlying issue is.

Mon, Aug 28, 9:10 AM · Unstewarded-production-error, Dumps-Generation, Wikimedia-production-error
ArielGlenn added a comment to T249310: TextPassDumper.php messages on retry of fetch of content have extra junk in them.

Related: T324463

Mon, Aug 28, 8:45 AM · Platform Team Workboards (Clinic Duty Team), Dumps-Generation
ArielGlenn added a comment to T306629: PHP Warning: failed to get text for revid [id] [Called from AbstractFilter::getText in /srv/mediawiki/php-1.39.0-wmf.7/extensions/ActiveAbstract/includes/AbstractFilter.php at line 195].

Who runs the findBadBlobs.php script in cases like this? It would be nice to get that done.

Mon, Aug 28, 8:32 AM · User-brennen, Dumps-Generation, Wikimedia-production-error
ArielGlenn closed T133547: set up automated HTML (restbase) dumps on francium as Declined.

Not doing this, since we now have WME (Enterprise) dumps in HTML format available for public download.

Mon, Aug 28, 8:26 AM · Platform Team Legacy (Watching / External), Services (watching), Datasets-Archiving, Dumps-Generation
ArielGlenn moved T186801: Flow page content dumps not resilient when database goes away from Backlog to Other teams on the Dumps-Generation board.
Mon, Aug 28, 8:24 AM · Growth-Team-Filtering, Growth-Team, StructuredDiscussions, Dumps-Generation
ArielGlenn closed T331129: cleanup_xmldumps is failing on dumpsdata1005 as Resolved.

Going ahead and closing this.

Mon, Aug 28, 8:20 AM · Dumps-Generation

Aug 24 2023

ArielGlenn added a comment to T344929: Maintenance script appservers running code from wmf.20 when version not live.

grep on mwmaint1002 for php, looking for long running stuff, gives me only

Jul11   0:00 /bin/bash /usr/local/bin/mwscript eval.php --wiki=commonswiki

The others are all Aug 22 or 23rd just fyi.

Aug 24 2023, 3:43 PM · Scap, Release-Engineering-Team, Wikimedia-production-error
ArielGlenn added a comment to T332562: Don't expose partial dumpfiles.

I"m presuming you didn't see any instances of this in the meantime, @awight ? Can we close this?

Aug 24 2023, 2:21 PM · Dumps-Generation, Wikimedia Enterprise
ArielGlenn added a comment to T320722: Deployment training request for Sergio Gimeno.

Hey @Sgs we missed you this morning at the deployment window for training. Or were you going to do the UTC late window this time?

Aug 24 2023, 7:31 AM · Release-Engineering-Team (Deployment Training Requests)

Aug 22 2023

ArielGlenn added a comment to T343325: Develop Dumps Triage Runbook.

I can try to dust off and restructure the troubleshooting guide on wikitech for the sql/xml dumps, if that would be helpful. This would by no means be a replacement for the runbook, but more of a minimal guide if people get stuck. Having a document specifically for dumps newcomers is great and I hope it will be expanded over time!

Aug 22 2023, 12:54 PM · Data Products (Sprint 00), Dumps 2.0, Data-Engineering

Aug 17 2023

ArielGlenn added a comment to T344409: Clarify whether config patches need reviews before being scheduled for deployment.

Counterpoint: knowing the config settings doesn't mean understanding the code activated by those changes or its possible impacts. At least, not for me. Some areas I know, and some I don't.

Aug 17 2023, 7:38 AM · Wikimedia-Site-requests, Release-Engineering-Team

Aug 16 2023

ArielGlenn updated the task description for T343882: Make wikidata dumps use two snapshot hosts to complete quicker.
Aug 16 2023, 11:13 AM · Patch-For-Review, Dumps-Generation

Aug 14 2023

ArielGlenn added a project to T344147: Puppet broken on mediawiki instances in deployment-prep: User-ArielGlenn.
Aug 14 2023, 12:01 PM · User-ArielGlenn, Beta-Cluster-Infrastructure
ArielGlenn added a comment to T344147: Puppet broken on mediawiki instances in deployment-prep.

Note that since dumps snapshot instances are sorta-kinda mediawiki instances, this affects them too.

Aug 14 2023, 11:58 AM · User-ArielGlenn, Beta-Cluster-Infrastructure
ArielGlenn triaged T344147: Puppet broken on mediawiki instances in deployment-prep as Medium priority.
Aug 14 2023, 11:58 AM · User-ArielGlenn, Beta-Cluster-Infrastructure

Aug 9 2023

ArielGlenn triaged T343882: Make wikidata dumps use two snapshot hosts to complete quicker as Medium priority.
Aug 9 2023, 12:44 PM · Patch-For-Review, Dumps-Generation

Aug 7 2023

ArielGlenn updated the task description for T341045: Get Data Engineering folks access to hosts and systems needed for maintenance of the existing dumps system.
Aug 7 2023, 12:40 PM · Data Engineering and Event Platform Team, Dumps-Generation

Aug 5 2023

ArielGlenn triaged T343621: Wikidata rdf lexemes dump failed for Fri Aug 4, due to db conn error as Medium priority.
Aug 5 2023, 6:48 AM · Wikidata, Wikidata Lexicographical data, wdwb-tech, Patch-For-Review, Dumps-Generation

Jul 27 2023

ArielGlenn added a comment to T341559: Deployment Training Request for jebe.

This training happened, though it was a lot less interactive and useful than it could have been because no patches were scheduled and no one showed up with a patch last minute, in spite of me begging :-D But we went through a description of all the steps, looked at all the relevant dashboards and hosts and commands, so there's that :-)

Jul 27 2023, 7:57 AM · Release-Engineering-Team (Deployment Training Requests)

Jul 19 2023

ArielGlenn added a comment to T325232: Migrate Dumpsdata and Htmldumper Hosts From Buster to Bullseye.

Dumpsdata1007, running bullseye, is now the fallback host for sql/xml and misc dumps. This means all hosts in production (not spares) are on bullseye now and this task can be closed after a day or so just to make sure things are stable.

Jul 19 2023, 3:41 PM · Patch-For-Review, Dumps-Generation

Jul 17 2023

ArielGlenn added a comment to T295909: <Platform Initiative> Dumps + WME Gap Analysis.

Sounds great to me, thanks!

Jul 17 2023, 12:09 PM · Platform Engineering, Dumps-Generation, Foundational Technology Requests
ArielGlenn updated the task description for T341045: Get Data Engineering folks access to hosts and systems needed for maintenance of the existing dumps system.
Jul 17 2023, 10:06 AM · Data Engineering and Event Platform Team, Dumps-Generation

Jul 13 2023

ArielGlenn added a comment to T341058: Failures for Cirrus Search dumps for wikidatawiki, zhwiki, enwiki for the 20230703 run.

Just for my understanding, it looks like the new patch would exception out in the case where there is a failure with the last connection of whatever sort. Am I reading that right? And if so, how does that help us in the current situation? Sorry for whatever I am missing here. Thanks!

Jul 13 2023, 11:15 AM · Wikidata, MW-1.41-notes (1.41.0-wmf.19; 2023-07-25), Chinese-Sites, Dumps-Generation, Discovery-Search (Current work)
ArielGlenn added a comment to T339929: custom partman recipe dumpsdata100X-no-data-format.cfg causes installer to hang at partitioning menu.

Hey @elukey (or anyone else watching who wants to chime in), I've got a recipe that might maybe possibly could work. (See patch above.) But I have questions. Some recipes in the repo deal with lvm partitions by "unknown ignore" instead of "lvmpv keep", and I wonder which is better. Some recipes without swap specify "d-i partman-basicfilesystems/no_swap boolean false" and some do not, and I wonder which is right. And last but not least, is it still the procedure for testing before merge, to announce "hey I'm testing on installX00Y and disabling puppet for awhile" in the channel and hoping no one speaks up? Thanks in advance!

Jul 13 2023, 10:50 AM · Patch-For-Review, Dumps-Generation

Jul 12 2023

ArielGlenn added a comment to T329491: ICU transition towards ICU 67.

A note that I did a test run of sql/xml dumps on deployment-prep with the new icu version and it looks fine to me, though I didn't check for any weird details of category sorting or whatever.

Jul 12 2023, 1:10 PM · Patch-For-Review, serviceops-radar, SRE

Jul 11 2023

ArielGlenn updated the task description for T341045: Get Data Engineering folks access to hosts and systems needed for maintenance of the existing dumps system.
Jul 11 2023, 2:52 PM · Data Engineering and Event Platform Team, Dumps-Generation
ArielGlenn added a comment to T341557: Grant Access to wmf for Jennifer Ebe.

Jennifer is already a member of wmf

https://ldap.toolforge.org/user/jebe

Double checked.

btullis@seaborgium:~$ ldapsearch -A -x member=uid=jebe,ou=people,dc=wikimedia,dc=org dn
# extended LDIF
#
# LDAPv3
# base <dc=wikimedia,dc=org> (default) with scope subtree
# filter: member=uid=jebe,ou=people,dc=wikimedia,dc=org
# requesting: dn 
#

# wmf, groups, wikimedia.org
dn: cn=wmf,ou=groups,dc=wikimedia,dc=org

# project-bastion, groups, wikimedia.org
dn: cn=project-bastion,ou=groups,dc=wikimedia,dc=org

# project-analytics, groups, wikimedia.org
dn: cn=project-analytics,ou=groups,dc=wikimedia,dc=org

# project-deployment-prep, groups, wikimedia.org
dn: cn=project-deployment-prep,ou=groups,dc=wikimedia,dc=org

# search result
search: 2
result: 0 Success

# numResponses: 5
# numEntries: 4
Jul 11 2023, 1:35 PM · SRE, LDAP-Access-Requests
ArielGlenn updated subscribers of T341557: Grant Access to wmf for Jennifer Ebe.

See also https://phabricator.wikimedia.org/T341045 for the context. @WDoranWMF please sign off just in case that's needed. Thanks!

Jul 11 2023, 11:02 AM · SRE, LDAP-Access-Requests

Jul 10 2023

ArielGlenn added a comment to T339929: custom partman recipe dumpsdata100X-no-data-format.cfg causes installer to hang at partitioning menu.

It sounds like the reuse-parts.cfg script is the way to go. Let me poke around and see how that's used elsewhere, and I'll come back if I get stuck. Thanks!

Jul 10 2023, 4:32 PM · Patch-For-Review, Dumps-Generation

Jul 9 2023

ArielGlenn added a comment to T341058: Failures for Cirrus Search dumps for wikidatawiki, zhwiki, enwiki for the 20230703 run.

One more to add:

Jul  4 09:16:50 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20230703/commonswiki-20230703-cirrussearch-content.json.gz
Jul 9 2023, 3:38 PM · Wikidata, MW-1.41-notes (1.41.0-wmf.19; 2023-07-25), Chinese-Sites, Dumps-Generation, Discovery-Search (Current work)
ArielGlenn added a comment to T341045: Get Data Engineering folks access to hosts and systems needed for maintenance of the existing dumps system.

@JEbe-WMF you will need to folllow the instructions here https://wikitech.wikimedia.org/wiki/SRE/Clinic_Duty/Access_requests#Checklist and create a task, feel free to add me as a subscriber and link this one to it. Make sure you ask for membership in the wmf ldap group. That will give you icinga/grafana/logstash access.

Jul 9 2023, 3:17 PM · Data Engineering and Event Platform Team, Dumps-Generation
ArielGlenn added a comment to T341045: Get Data Engineering folks access to hosts and systems needed for maintenance of the existing dumps system.

Dan and Xabriel already are members of the wmf group, giving access to grafana and icinga (though contact info might need to be added for executing commands on icinga). Jennifer is not yet in the group.

Jul 9 2023, 3:12 PM · Data Engineering and Event Platform Team, Dumps-Generation
ArielGlenn updated the task description for T341045: Get Data Engineering folks access to hosts and systems needed for maintenance of the existing dumps system.
Jul 9 2023, 3:07 PM · Data Engineering and Event Platform Team, Dumps-Generation
ArielGlenn updated the task description for T341045: Get Data Engineering folks access to hosts and systems needed for maintenance of the existing dumps system.
Jul 9 2023, 3:04 PM · Data Engineering and Event Platform Team, Dumps-Generation
ArielGlenn updated the task description for T341045: Get Data Engineering folks access to hosts and systems needed for maintenance of the existing dumps system.
Jul 9 2023, 3:03 PM · Data Engineering and Event Platform Team, Dumps-Generation
ArielGlenn added a comment to T325232: Migrate Dumpsdata and Htmldumper Hosts From Buster to Bullseye.

Swapped dumpsdata1003 in as the live nfs share for misc dumps.

Jul 9 2023, 2:51 PM · Patch-For-Review, Dumps-Generation

Jul 6 2023

ArielGlenn added a comment to T341045: Get Data Engineering folks access to hosts and systems needed for maintenance of the existing dumps system.

@JEbe-WMF and @xcollazo you should both sign up for MediaWiki deployment training here: https://phabricator.wikimedia.org/project/board/5265/ and get scheduled for that. Once that's done, we can add you to the deployers list in puppet. (Dan you are already a deployer so you're off the hook ;-) )

Jul 6 2023, 11:48 AM · Data Engineering and Event Platform Team, Dumps-Generation
ArielGlenn updated the task description for T341045: Get Data Engineering folks access to hosts and systems needed for maintenance of the existing dumps system.
Jul 6 2023, 11:42 AM · Data Engineering and Event Platform Team, Dumps-Generation
ArielGlenn added a comment to T341045: Get Data Engineering folks access to hosts and systems needed for maintenance of the existing dumps system.

@JEbe-WMF @Milimetric and @xcollazo would you please subscribe to https://lists.wikimedia.org/postorius/lists/xmldatadumps-l.lists.wikimedia.org/ and let me know which email addresses you used? I will add them as co-admins of the list. Thanks!

Jul 6 2023, 11:37 AM · Data Engineering and Event Platform Team, Dumps-Generation
ArielGlenn updated subscribers of T341045: Get Data Engineering folks access to hosts and systems needed for maintenance of the existing dumps system.

@RobH Would you be willing to add Milimetric, xcollazo and Jebe-WMF to the #acl*procurement-review acl so that they can view procurement tasks? I hope no new tasks will come up for some time but just in case, and it will let us look at psat ones and discuss. They will be working with me on the dumps now. Thanks!

Jul 6 2023, 11:35 AM · Data Engineering and Event Platform Team, Dumps-Generation
ArielGlenn updated the task description for T341045: Get Data Engineering folks access to hosts and systems needed for maintenance of the existing dumps system.
Jul 6 2023, 11:33 AM · Data Engineering and Event Platform Team, Dumps-Generation
ArielGlenn updated subscribers of T341045: Get Data Engineering folks access to hosts and systems needed for maintenance of the existing dumps system.

@WDoranWMF we will need your approval for this.

Jul 6 2023, 11:20 AM · Data Engineering and Event Platform Team, Dumps-Generation
ArielGlenn updated subscribers of T341045: Get Data Engineering folks access to hosts and systems needed for maintenance of the existing dumps system.

After a conversation with Will, Dan and others, the people who need the above access are @Milimetric , @xcollazo and @JEbe-WMF so now let me get started on that.

Jul 6 2023, 11:12 AM · Data Engineering and Event Platform Team, Dumps-Generation

Jul 5 2023

ArielGlenn added a comment to T341058: Failures for Cirrus Search dumps for wikidatawiki, zhwiki, enwiki for the 20230703 run.

Adding two more failures:

Jul  4 09:16:50 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20230703/mlwiki-20230703-cirrussearch-content.json.gz
Jul  4 09:16:50 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20230703/metawiki-20230703-cirrussearch-general.json.gz

These came from different db sections so the script running those finished later, and hence the error report was sent later.

Jul 5 2023, 12:19 PM · Wikidata, MW-1.41-notes (1.41.0-wmf.19; 2023-07-25), Chinese-Sites, Dumps-Generation, Discovery-Search (Current work)

Jul 4 2023

ArielGlenn added a comment to T273089: mediawiki scripts fail on new buster image in deployment-prep.

I've not spun up any more buster images, and the next one I create will likely be bullseye. Maybe someone else has done so though.

Jul 4 2023, 3:18 PM · Dumps-Generation, Beta-Cluster-Infrastructure
ArielGlenn moved T341045: Get Data Engineering folks access to hosts and systems needed for maintenance of the existing dumps system from Backlog to Active on the Dumps-Generation board.
Jul 4 2023, 12:38 PM · Data Engineering and Event Platform Team, Dumps-Generation
ArielGlenn moved T341058: Failures for Cirrus Search dumps for wikidatawiki, zhwiki, enwiki for the 20230703 run from Backlog to Other teams on the Dumps-Generation board.
Jul 4 2023, 12:37 PM · Wikidata, MW-1.41-notes (1.41.0-wmf.19; 2023-07-25), Chinese-Sites, Dumps-Generation, Discovery-Search (Current work)
ArielGlenn triaged T341058: Failures for Cirrus Search dumps for wikidatawiki, zhwiki, enwiki for the 20230703 run as High priority.
Jul 4 2023, 12:29 PM · Wikidata, MW-1.41-notes (1.41.0-wmf.19; 2023-07-25), Chinese-Sites, Dumps-Generation, Discovery-Search (Current work)
ArielGlenn moved T341045: Get Data Engineering folks access to hosts and systems needed for maintenance of the existing dumps system from Data Eng Backlog to Radar on the Data Engineering and Event Platform Team board.
Jul 4 2023, 10:18 AM · Data Engineering and Event Platform Team, Dumps-Generation
ArielGlenn added a project to T341045: Get Data Engineering folks access to hosts and systems needed for maintenance of the existing dumps system: Data Engineering and Event Platform Team.
Jul 4 2023, 10:17 AM · Data Engineering and Event Platform Team, Dumps-Generation
ArielGlenn updated the task description for T341045: Get Data Engineering folks access to hosts and systems needed for maintenance of the existing dumps system.
Jul 4 2023, 10:15 AM · Data Engineering and Event Platform Team, Dumps-Generation
ArielGlenn triaged T341045: Get Data Engineering folks access to hosts and systems needed for maintenance of the existing dumps system as High priority.
Jul 4 2023, 10:15 AM · Data Engineering and Event Platform Team, Dumps-Generation

Jun 22 2023

ArielGlenn created T340096: Build mwbzutils for bullseye.
Jun 22 2023, 8:14 AM · Patch-For-Review, Dumps-Generation
ArielGlenn added a comment to T325232: Migrate Dumpsdata and Htmldumper Hosts From Buster to Bullseye.

The above rsync is already complete.

Jun 22 2023, 8:08 AM · Patch-For-Review, Dumps-Generation
ArielGlenn added a comment to T325232: Migrate Dumpsdata and Htmldumper Hosts From Buster to Bullseye.

Started an rsync from dumpsdata1004 (fallback nfs share for both sql/xml and misc dumps) to dumpsdata1003, in a screen sessnio as ariel, bandwidth limited to 1G as has previously been requested by SRE folks. We'll do another one of these Friday evening and again just before the swap.

Jun 22 2023, 7:02 AM · Patch-For-Review, Dumps-Generation
ArielGlenn added a comment to T325232: Migrate Dumpsdata and Htmldumper Hosts From Buster to Bullseye.

After reimaging with bullseye, checked rpcinfo -p on dumpsdata1003 and the ports for mountd, nfs, nlockmgr are all correct, so once this host has the right data on it, it will be ready to go to be swapped in for dumpsdata1002, which can then be prepped for decommissioning.

Jun 22 2023, 6:43 AM · Patch-For-Review, Dumps-Generation
ArielGlenn updated the task description for T325232: Migrate Dumpsdata and Htmldumper Hosts From Buster to Bullseye.
Jun 22 2023, 6:35 AM · Patch-For-Review, Dumps-Generation
ArielGlenn moved T295909: <Platform Initiative> Dumps + WME Gap Analysis from Active to Blocked/Stalled/Waiting for event on the Dumps-Generation board.
Jun 22 2023, 5:40 AM · Platform Engineering, Dumps-Generation, Foundational Technology Requests
ArielGlenn moved T335761: Missing Enterprise Dumps in 2023-04-20, 2023-05-01 and 2023-05-20 runs from Active to Done on the Dumps-Generation board.
Jun 22 2023, 5:40 AM · Wikimedia Enterprise, Dumps-Generation
ArielGlenn moved T336742: Two page content jobs for wikidatawiki are taking days to complete. from Active to Done on the Dumps-Generation board.
Jun 22 2023, 5:40 AM · Dumps-Generation
ArielGlenn closed T335761: Missing Enterprise Dumps in 2023-04-20, 2023-05-01 and 2023-05-20 runs as Resolved.

I think this is resolved; the current run is already available on the public web server and on the nfs share for WMCS instances, with the same number of files as the download on the 1st of the month. Closing.

Jun 22 2023, 5:40 AM · Wikimedia Enterprise, Dumps-Generation
ArielGlenn closed T335761: Missing Enterprise Dumps in 2023-04-20, 2023-05-01 and 2023-05-20 runs, a subtask of T332032: Create baseline statistics for reference usage, as Resolved.
Jun 22 2023, 5:39 AM · WMDE-TechWish-Maintenance-2023, WMDE-TechWish-Sprint-2023-05-03, WMDE-TechWish-Sprint-2023-04-19, WMDE-TechWish-Sprint-2023-04-05, WMDE-TechWish-Sprint-2023-03-14, WMDE-References-FocusArea, Epic
ArielGlenn closed T336742: Two page content jobs for wikidatawiki are taking days to complete. as Resolved.

When I reran one of these jobs, it ran to completion in the usual period of time. Next time we see this behavior, we can try shooting the job and letting it rerun in a timely fashion, rather than blocking for days. Not exactly a resolution to whatever the underlying bug may have been, but it will have to do.

Jun 22 2023, 5:38 AM · Dumps-Generation
ArielGlenn added a comment to T281267: various weekly and daily dumps run from systemd timers are broken.

@fgiunchedi I notice that in some cases phab tasks are autocreated when systemd units fail. Is that true for systemd jobs on snapshot hosts? Could we get tagged on those (Dumps-Generation) or could we get emails from those (ops-dumps@wm.o)?

Jun 22 2023, 5:34 AM · User-jbond, wdwb-tech, Wikidata, SRE, observability, Dumps-Generation
ArielGlenn added a comment to T315902: New error "DB is set and has not been closed by the Load Balancer" for certain bad revisions during page content dumps.

Just a note that we still regularly see these errors on each dump run for a small selection of wikis.

Jun 22 2023, 5:28 AM · Platform Engineering, Dumps-Generation
ArielGlenn added a comment to T295909: <Platform Initiative> Dumps + WME Gap Analysis.

@WDoranWMF am I right to assume that this is long since moot, superceded by various other things? If not, can the task be updated to reflect the current work left to do and who will be taking that on? If you are not the right person to answer this, perhaps you can redirect me to the right person. Thanks!

Jun 22 2023, 5:26 AM · Platform Engineering, Dumps-Generation, Foundational Technology Requests
ArielGlenn moved T335130: The content model 'Json.JsonConfig' is not registered on this wiki(Collabwiki) from Active to Done on the Dumps-Generation board.
Jun 22 2023, 5:24 AM · Platform Engineering, Dumps-Generation, Wikimedia-production-error
ArielGlenn closed T335130: The content model 'Json.JsonConfig' is not registered on this wiki(Collabwiki) as Resolved.

Closing this task since the dumps do run to completion now.

Jun 22 2023, 5:24 AM · Platform Engineering, Dumps-Generation, Wikimedia-production-error
ArielGlenn moved T217549: bytemark dump mirror index.html file is out of date from Blocked/Stalled/Waiting for event to Done on the Dumps-Generation board.
Jun 22 2023, 5:23 AM · Dumps-Generation
ArielGlenn closed T217549: bytemark dump mirror index.html file is out of date as Resolved.

Nothing left to do here, closing.

Jun 22 2023, 5:23 AM · Dumps-Generation
ArielGlenn added a comment to T217549: bytemark dump mirror index.html file is out of date.

Never mind, it's already marked as inactive and it can stay that way. I'll make sure it's gone from our mirrors lists on various web pages too.

Jun 22 2023, 5:16 AM · Dumps-Generation
ArielGlenn added a comment to T217549: bytemark dump mirror index.html file is out of date.

Right, I'm going to make this mirror inactive, it's clearly not coming back. If someone over there changes their mind, we can reenable it.

Jun 22 2023, 5:12 AM · Dumps-Generation
ArielGlenn moved T220594: abstracts dumps for dewikiversity fail with MWUnknownContentModelException from ContentHandler.php from Blocked/Stalled/Waiting for event to Done on the Dumps-Generation board.
Jun 22 2023, 5:10 AM · MediaWiki-General, Dumps-Generation
ArielGlenn closed T220594: abstracts dumps for dewikiversity fail with MWUnknownContentModelException from ContentHandler.php as Resolved.

Long since fixed.

Jun 22 2023, 5:09 AM · MediaWiki-General, Dumps-Generation
ArielGlenn closed T220793: content still marked as flow-board on urwikibooks breaks abstract dumps as Resolved.

Long since fixed.

Jun 22 2023, 5:08 AM · MW-1.34-notes (1.34.0-wmf.1; 2019-04-16), MediaWiki-Maintenance-system, MediaWiki-General, Dumps-Generation
ArielGlenn moved T220793: content still marked as flow-board on urwikibooks breaks abstract dumps from Blocked/Stalled/Waiting for event to Done on the Dumps-Generation board.
Jun 22 2023, 5:08 AM · MW-1.34-notes (1.34.0-wmf.1; 2019-04-16), MediaWiki-Maintenance-system, MediaWiki-General, Dumps-Generation
ArielGlenn moved T263318: look into space issues on dumpsdata1001 and 1003 from Active to Done on the Dumps-Generation board.
Jun 22 2023, 5:08 AM · Dumps-Generation
ArielGlenn moved T226093: Capacity planning for Commons Structured Data from Blocked/Stalled/Waiting for event to Done on the Dumps-Generation board.
Jun 22 2023, 5:07 AM · wdwb-tech, Dumps-Generation, SDC General, Wikidata
ArielGlenn closed T226093: Capacity planning for Commons Structured Data, a subtask of T68108: [Epic] Store media information for files on Wikimedia Commons as structured data, as Resolved.
Jun 22 2023, 5:07 AM · GLAM-Tech, Multimedia, Commons, Epic, Wikidata, SDC General
ArielGlenn closed T226093: Capacity planning for Commons Structured Data as Resolved.

There's no point in having this open for a once a year check in, so I'll go ahead and close it. When capacity planning needs to be done for dbs in the regular course of things, this can be discussed.

Jun 22 2023, 5:06 AM · wdwb-tech, Dumps-Generation, SDC General, Wikidata
ArielGlenn added a comment to T143870: Some mw snapshot hosts are accessing main db servers.

Is this still an issue, after the changes to LBFactory and so on that remove depooled dbs from the list available for connections? @Ladsgroup you would know best I think.

Jun 22 2023, 5:05 AM · MW-1.35-notes (1.35.0-wmf.10; 2019-12-10), Dumps-Generation
ArielGlenn moved T221086: Apr 1 2019 and/or >=1.33-wmf.23 dump run issues from Blocked/Stalled/Waiting for event to Done on the Dumps-Generation board.
Jun 22 2023, 5:03 AM · Dumps-Generation