Page MenuHomePhabricator

ArielGlenn (ariel)
User

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Monday

  • Clear sailing ahead.

User Details

User Since
Oct 8 2014, 7:09 PM (496 w, 2 d)
Availability
Available
IRC Nick
apergos
LDAP User
ArielGlenn
MediaWiki User
ArielGlenn [ Global Accounts ]

Recent Activity

Tue, Apr 9

ArielGlenn added a comment to T316303: Check global rights during autocreation.

The above patch fails CI for three tests not in the Database group. All three call User::isAllowed() at some point, which now would require database access. Two tests are in ConfirmEdit and one is in core (DumpableObjectsTest). I could add al lthree to @group Database or I could mock out CentralAuthHooks and have the onUserGetRights method always return true in those tests, or maybe there's some better approach. Poking @Tgr for advice or a pointer.

Tue, Apr 9, 3:45 PM · Patch-For-Review, MediaWiki-Platform-Team, MediaWiki-extensions-CentralAuth, MediaWiki-Core-AuthManager

Mar 14 2024

ArielGlenn added a comment to T359957: Enroll in Chrome third-party cookies deprecation trial.

Note that for any of those issue links in the task description, I get "Access is denied to this issue". But the issue linked to in T359957#9625065 is visible at least.

Mar 14 2024, 5:33 AM · Patch-For-Review, WMF-General-or-Unknown, MediaWiki-extensions-CentralAuth, MediaWiki-Platform-Team

Mar 12 2024

ArielGlenn claimed T359947: Test cross-domain authentication with Federated Credentials Management API.
Mar 12 2024, 4:12 PM · MediaWiki-Platform-Team, MediaWiki-extensions-CentralAuth

Mar 6 2024

ArielGlenn added a watcher for MediaWiki-extensions-CentralAuth: ArielGlenn.
Mar 6 2024, 10:05 AM

Feb 27 2024

ArielGlenn added a comment to T358329: beta-update-databases-eqiad job times out / beta databases are having issues.

Yes I agree but that is inside a public git repository (from what I'm seeing) which means it'll be confusing will be potentially removed again. That's why I didn't put it there. It needs a better solution.

Feb 27 2024, 12:35 PM · Beta-Cluster-Infrastructure
ArielGlenn added a comment to T358329: beta-update-databases-eqiad job times out / beta databases are having issues.

I had to reclone it again from db11, it's usually better to set the db to read only before doing the cloning (read only if you cloning from master, "stop slave" if you're cloning from another replica.) even though binlog can be replayed, there are statements happening that wouldn't be written to binlogs. Regardless, that's fixed now.

Regarding password, it's just a user on master with that password, you can just change its password. I did exactly that, the password is in my home directory on db11 and only readable by root.

Feb 27 2024, 11:35 AM · Beta-Cluster-Infrastructure

Feb 26 2024

ArielGlenn added a comment to T358329: beta-update-databases-eqiad job times out / beta databases are having issues.

Hi 👋 I suggest stopping to touch this. I will take a look soon. Regarding databases, if you're not 100% sure what you're doing, you usually end up making things worse. Trust me, been there done that

Feb 26 2024, 5:28 PM · Beta-Cluster-Infrastructure
ArielGlenn added a comment to T358329: beta-update-databases-eqiad job times out / beta databases are having issues.

The cloning procedure is done for db14 but we are currently hunting around for the replication password, not where the docs ( https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/Databases#Starting_Replication ) say it should be, not anywhere in that repo ever, apparently.

Feb 26 2024, 5:20 PM · Beta-Cluster-Infrastructure
ArielGlenn added a comment to T358329: beta-update-databases-eqiad job times out / beta databases are having issues.

In the end @TheresNoTime figured it out: puppet was starting mariadb automatically when we didn't want it running and hence creating that file complained of in the error above. The cloning process looks like it's going ok at the moment.

Feb 26 2024, 4:32 PM · Beta-Cluster-Infrastructure
ArielGlenn added a comment to T358329: beta-update-databases-eqiad job times out / beta databases are having issues.

Just to explain to folks who might be following along, what's happening: the primary server (db13) will be cloned (via mariabackup --innobackupex) to the new replica; a new instance is bring created now for that. While that is happening, replication will be stopped and the primary will remain read-only. I am guessing that this will be a matter of some hours rather than a day. More updates as available.

Feb 26 2024, 2:56 PM · Beta-Cluster-Infrastructure

Feb 22 2024

ArielGlenn added a comment to T353787: Decom dumpsdata100[1-2].

1 and 2 are both role dumps::generation::server::spare and have been so since at least last July. See https://gerrit.wikimedia.org/r/c/operations/puppet/+/936379 and https://gerrit.wikimedia.org/r/c/operations/puppet/+/893265
While any nfs spare could in theory be swapped in for any production host, we have other newer spares with decent size filesystems for that; these are idle and can go at any time.

Feb 22 2024, 9:41 AM · Patch-For-Review, Data-Platform-SRE (2024.03.25 - 2024.04.14), Dumps-Generation

Feb 21 2024

ArielGlenn added a comment to T355281: Set up some beta cluster wikis with different registrable domain.

I'd prefer the beta be kept in the name, making it clear that these are wikis on the deployment cluster.

Feb 21 2024, 6:21 PM · MediaWiki-Platform-Team, Beta-Cluster-Infrastructure

Feb 15 2024

ArielGlenn added a comment to T343882: Make wikidata dumps use two snapshot hosts to complete quicker.

This is in progress still, notes so I don't forget:

  • we run 7zs after the bz2 history files, that job will remain untouched
  • need to adjust the script that does history backfil to not move files into place if there is a temp or bz2 file that appeared in the meantime (protection in case the fillin script and the main worker both reach the same files)
  • could do md5s of these files as we go but let's see how much we gain without, it would be safer as far as avoiding overlaps
Feb 15 2024, 9:00 PM · Patch-For-Review, Dumps-Generation

Feb 13 2024

ArielGlenn added a comment to T355281: Set up some beta cluster wikis with different registrable domain.

We may want to test the behaviour when going from logged in on a wiki on beta.wmflabs.org (let's say en.wikipedia) and then visiting some-language.some-wiki.beta.wmcloud.org which is not designated as the "representative wiki" for that wiki family, and see if the behaviour is different from visiting the "representative wiki" immediately after login on en.wp.beta.wmflabs.org. These scenarios behave differently for me in production.

Feb 13 2024, 3:59 PM · MediaWiki-Platform-Team, Beta-Cluster-Infrastructure

Feb 5 2024

ArielGlenn added a comment to T355281: Set up some beta cluster wikis with different registrable domain.

As I try to help plan this out, I have accumulated some questions.

Feb 5 2024, 5:25 PM · MediaWiki-Platform-Team, Beta-Cluster-Infrastructure

Jan 29 2024

ArielGlenn added a comment to T61702: Examine which extensions are installed on login.wikimedia.org (loginwiki) and vote.wikimedia.org (votewiki).

Listing more extensions not named above that appear in Special:Versions and seem to me to not be needed at loginwiki:

  • 3D
  • CodeEditor
  • CodeMirror
  • ElectronPdfService
  • Global Usage
  • Kartographer
  • MediaModeration
  • RevisionSlider
  • TemplateStyles
  • TextExtracts
  • TwoColConflict

and maybe others.

Jan 29 2024, 10:35 AM · Stewards-and-global-tools, Wikimedia-Site-requests

Jan 23 2024

ArielGlenn added a project to T355450: mediawiki support for one-click unsubscribe: User-ArielGlenn.
Jan 23 2024, 8:52 AM · MediaWiki-Platform-Team, User-ArielGlenn, MediaWiki-Engineering, Growth-Team, Notifications, MediaWiki-Email

Jan 17 2024

ArielGlenn added a comment to T345249: Mitigate phase-out of third-party cookies in CentralAuth.

Verified for Firefox: I logged out, deleted all *wik*org cookies except for non SUL sites (phab, etherpad and so on), set Enhanced Tracking Protection to "custom" and chose "All cross-site cookies" (see image below). After login at en.wikipedia, after the end of the redirect/session creation chain for commons.wikimedia.org, I have session cookies for commons with session id, UserId, UserName stored locally. When I go to visit commons,wm.o, these cookies are sent to the web server and I am logged in as a result.
Firefox version: 121.0, linux.

Screenshot from 2024-01-17 13-32-27.png (659×873 px, 55 KB)

Jan 17 2024, 11:43 AM · MediaWiki-Platform-Team, MediaWiki-extensions-CentralAuth

Jan 10 2024

ArielGlenn added a comment to T326937: Prepare CentralAuth extension for IP Masking.

While T17294 seems stalled, is there anything do be done for CentralAuth in the meantime?

Jan 10 2024, 10:28 AM · MediaWiki-Platform-Team, MediaWiki-extensions-CentralAuth, Anti-Harassment, Temporary accounts

Jan 4 2024

ArielGlenn added a comment to T354318: Next steps for MediaWiki deployment training program.

Sounds good to me, I'll lurk until input is called for :-)

Jan 4 2024, 4:51 PM · Release-Engineering-Team (Deployment Training Requests), User-ArielGlenn
brennen awarded T354318: Next steps for MediaWiki deployment training program a Love token.
Jan 4 2024, 4:46 PM · Release-Engineering-Team (Deployment Training Requests), User-ArielGlenn
ArielGlenn created T354318: Next steps for MediaWiki deployment training program.
Jan 4 2024, 8:07 AM · Release-Engineering-Team (Deployment Training Requests), User-ArielGlenn

Dec 18 2023

ArielGlenn closed T205992: Allusers query auprop=rights does not include global rights (and is possibly wrong in other ways) as Resolved.

The one thing I didn't think to check. Of course it works fine on other accounts. Closing!

Dec 18 2023, 5:21 PM · MW-1.42-notes (1.42.0-wmf.9; 2023-12-12), MediaWiki-Platform-Team, MediaWiki-extensions-CentralAuth, MediaWiki-Action-API
ArielGlenn added a comment to T205992: Allusers query auprop=rights does not include global rights (and is possibly wrong in other ways).

Seems to work fine for me.

Dec 18 2023, 4:54 PM · MW-1.42-notes (1.42.0-wmf.9; 2023-12-12), MediaWiki-Platform-Team, MediaWiki-extensions-CentralAuth, MediaWiki-Action-API
ArielGlenn added a comment to T205992: Allusers query auprop=rights does not include global rights (and is possibly wrong in other ways).

The above patch went out with last week's train, and is on all wikis (1.42.0-wmf.9) but the behaviour is unchanged so I'll need to look into this further.

Dec 18 2023, 4:41 PM · MW-1.42-notes (1.42.0-wmf.9; 2023-12-12), MediaWiki-Platform-Team, MediaWiki-extensions-CentralAuth, MediaWiki-Action-API

Dec 14 2023

ArielGlenn added a comment to T353383: Deployment training request for ssastry.

I wonder why I thought he was in mine? Wishful thinking maybe. Hope it all goes well!

Dec 14 2023, 9:51 AM · Release-Engineering-Team (Deployment Training Requests)
ArielGlenn added a comment to T353383: Deployment training request for ssastry.

We missed you, I'm guessing that you got notice of this too late to make the training? We can reschedule in any case.

Dec 14 2023, 8:38 AM · Release-Engineering-Team (Deployment Training Requests)

Dec 11 2023

ArielGlenn added a comment to T349761: Consider naming and usage conventions of new DatabaseVirtualDomains/VirtualDomainsMapping config.

I get confused by the name 'domain' every time. 'database' or 'dbname' is much clearer imo.

Dec 11 2023, 5:28 PM · API Platform, MediaWiki-Platform-Team (Radar), Documentation, DBA, MediaWiki-libs-Rdbms

Nov 14 2023

ArielGlenn added a comment to T349761: Consider naming and usage conventions of new DatabaseVirtualDomains/VirtualDomainsMapping config.

We ought to decide about the rest of the name too, i.e. "virtual-whatgoeshere?" I liked Tim's use of the extension name with case preserved (see https://gerrit.wikimedia.org/r/c/mediawiki/extensions/LoginNotify/+/968800) but maybe you all have other preferences.

Nov 14 2023, 10:19 AM · API Platform, MediaWiki-Platform-Team (Radar), Documentation, DBA, MediaWiki-libs-Rdbms
ArielGlenn updated subscribers of T348485: Migrate OAuth to use a virtual database domain.

Change 973857 had a related patch set uploaded (by ArielGlenn; author: ArielGlenn):

[mediawiki/extensions/OATHAuth@master] Use virtual domain for OATHAuth database

https://gerrit.wikimedia.org/r/973857

Nov 14 2023, 8:24 AM · MediaWiki-extensions-OAuth

Nov 13 2023

ArielGlenn added a comment to T268526: Use a dedicated mechanism to track page dependencies.

Native speaker chiming in to say I agree: dependency means literally that A depends on B, relation is vague and can mean all kinds of things. Not going to weigh in on other aspects of the name though, carry on :-)

Nov 13 2023, 3:19 PM · Schema-change, MediaWiki-Page-derived-data, Epic, MediaWiki-Parser
ArielGlenn added a comment to T205992: Allusers query auprop=rights does not include global rights (and is possibly wrong in other ways).

What do folks think of something like the above? (Untested)

Nov 13 2023, 10:39 AM · MW-1.42-notes (1.42.0-wmf.9; 2023-12-12), MediaWiki-Platform-Team, MediaWiki-extensions-CentralAuth, MediaWiki-Action-API

Nov 7 2023

ArielGlenn added a comment to T205992: Allusers query auprop=rights does not include global rights (and is possibly wrong in other ways).

I note that User::getRights() was deprecated in 1.34 and removed in 1.38. Apparently we are intended to use PermissionManager::getUserPermissions() instead.

Nov 7 2023, 2:51 PM · MW-1.42-notes (1.42.0-wmf.9; 2023-12-12), MediaWiki-Platform-Team, MediaWiki-extensions-CentralAuth, MediaWiki-Action-API

Nov 6 2023

ArielGlenn added a comment to T348486: Migrate CentralAuth to use a virtual database domain.

Took a stab at the GlobalBlocking change first, as it's smaller and simpler to my eyes. Not tested whatsoever.

Nov 6 2023, 2:59 PM · MW-1.42-notes (1.42.0-wmf.10; 2023-12-19), Patch-For-Review, MediaWiki-Platform-Team, MediaWiki-extensions-CentralAuth

Oct 5 2023

ArielGlenn added a comment to T347089: Deployment training request for dr0ptp4kt.

Um there is no Thurs Oct 8. There is Thurs Oct 5 (today) and Thurs Oct 12, 19, 26... wonder if you meant any of these?

Oct 5 2023, 6:03 AM · Release-Engineering-Team (Deployment Training Requests)

Oct 2 2023

ArielGlenn added a comment to T347347: Make "Quick" MW install a thing.

If this is partly for ease of installation by first-time patch submitters, we should bear in mind that the new developer also has to jump through the gerrit setup and wikitech account creation hoops, and setting up an ssh key etc. If one already has these things, then we can indeed get the install time down to something very short.

Oct 2 2023, 1:59 PM · MW-1.42-notes (1.42.0-wmf.12; 2024-01-02), User-zeljkofilipin, MediaWiki-Platform-Team, MediaWiki-Documentation

Sep 25 2023

ArielGlenn added a project to T319432: Migrate WMF production from PHP 7.4 to PHP 8.1: Dumps-Generation.
Sep 25 2023, 1:19 PM · Dumps-Generation, MediaWiki-Platform-Team, serviceops
ArielGlenn added a project to T281325: Text rows containing DB://cluster20/0, causing RevisionAccessException when the affected pages are viewed: User-ArielGlenn.
Sep 25 2023, 1:03 PM · User-ArielGlenn, MediaWiki-Platform-Team, Wikimedia-maintenance-script-run, Wikimedia-database-issue (Bad data)

Sep 19 2023

ArielGlenn added a comment to T346279: [Spike] Figure out what are good indicators for dumps data quality.

Just a note that sometimes size/page count/visible rev count might go down, if a large batch of pages are deleted for e.g. copyvio (more likely to occur on a small wiki).

Sep 19 2023, 4:36 AM · Data Products (Sprint 02), Dumps 2.0

Sep 13 2023

ArielGlenn added a comment to T336573: PHP Warning: XMLReader::read(): Memory allocation failed : growing input buffer.

We could make sure that for commonswiki, the setting config "sevenzipprefetch" is 0. I'll need to check that this is one of the settings that can be overriden, and that the code will recognize 0 as a 'false' value. This should get done before next month's full run.

Sep 13 2023, 7:32 AM · Unstewarded-production-error, Dumps-Generation, Wikimedia-production-error
ArielGlenn moved T341045: Get Data Engineering folks access to hosts and systems needed for maintenance of the existing dumps system from Active to Blocked/Stalled/Waiting for event on the Dumps-Generation board.
Sep 13 2023, 7:28 AM · Data-Engineering, Dumps-Generation
ArielGlenn added a comment to T341045: Get Data Engineering folks access to hosts and systems needed for maintenance of the existing dumps system.

There's one item in the checklist left before this task can be closed. And basically the holdup is just about getting the signoff from Tyler that the deployment trainings were completed; then we can get the rest of that item done.

Sep 13 2023, 7:28 AM · Data-Engineering, Dumps-Generation

Sep 12 2023

ArielGlenn added a comment to T336573: PHP Warning: XMLReader::read(): Memory allocation failed : growing input buffer.

To expand on this a bit more: we saw the same error and stack trace on a slightly different page range, but with the identical symptoms. Logstash link here: https://logstash.wikimedia.org/goto/62b164dd91e2763a0a402d02087be836 Running the job hangs at the same point every time, even if nothing else is happening on the host; there aren't a particularly large number of revisions for the problem page, and their size isn't very large either. As before, using bz2 prfetch files permits the job to run to completion.

Sep 12 2023, 9:31 AM · Unstewarded-production-error, Dumps-Generation, Wikimedia-production-error
ArielGlenn added a comment to T345874: XMLDumps broken on deployment-mwmaint02 due to Jade Extension related content.

So the patches went around and I checked that they are on snapshot03, but unfortunately I still see the error:

2023-09-12 05:20:33: enwiki (ID 14793) 683 pages (694.3|694.3/sec all|curr), 1000 revs (1016.6|1016.6/sec all|curr), ETA 2023-09-12 05:30:22 [max 600437]
MWUnknownContentModelException from line 192 of /srv/mediawiki/php-master/includes/content/ContentHandlerFactory.php: The content model 'JadeJudgment' is not registered on this wiki.
See https://www.mediawiki.org/wiki/Content_handlers to find out which extensions handle this content model.
#0 /srv/mediawiki/php-master/includes/content/ContentHandlerFactory.php(247): MediaWiki\Content\ContentHandlerFactory->validateContentHandler('JadeJudgment', NULL)
#1 /srv/mediawiki/php-master/includes/content/ContentHandlerFactory.php(181): MediaWiki\Content\ContentHandlerFactory->createContentHandlerFromHook('JadeJudgment')
#2 /srv/mediawiki/php-master/includes/content/ContentHandlerFactory.php(93): MediaWiki\Content\ContentHandlerFactory->createForModelID('JadeJudgment')
#3 /srv/mediawiki/php-master/includes/export/XmlDumpWriter.php(474): MediaWiki\Content\ContentHandlerFactory->getContentHandler('JadeJudgment')
#4 /srv/mediawiki/php-master/includes/export/XmlDumpWriter.php(402): XmlDumpWriter->writeSlot(Object(MediaWiki\Revision\SlotRecord), 1)
#5 /srv/mediawiki/php-master/includes/export/WikiExporter.php(554): XmlDumpWriter->writeRevision(Object(stdClass), Array)
#6 /srv/mediawiki/php-master/includes/export/WikiExporter.php(492): WikiExporter->outputPageStreamBatch(Object(Wikimedia\Rdbms\MysqliResultWrapper), Object(stdClass))
#7 /srv/mediawiki/php-master/includes/export/WikiExporter.php(316): WikiExporter->dumpPages('page_id >= 1900...', false)
#8 /srv/mediawiki/php-master/includes/export/WikiExporter.php(208): WikiExporter->dumpFrom('page_id >= 1900...', false)
#9 /srv/mediawiki/php-master/maintenance/includes/BackupDumper.php(355): WikiExporter->pagesByRange(190001, 195001, false)
#10 /srv/mediawiki/php-master/maintenance/dumpBackup.php(82): BackupDumper->dump(1, 1)
#11 /srv/mediawiki/php-master/maintenance/includes/MaintenanceRunner.php(685): DumpBackup->execute()
#12 /srv/mediawiki/php-master/maintenance/run.php(51): MediaWiki\Maintenance\MaintenanceRunner->run()
#13 /srv/mediawiki/multiversion/MWScript.php(159): require_once('/srv/mediawiki/...')
#14 {main}

Perhaps the override isn't being respected, or the usage isn't quite right?

Sep 12 2023, 9:20 AM · Dumps-Generation, MediaWiki-ContentHandler, Beta-Cluster-Infrastructure

Sep 8 2023

ArielGlenn added a comment to T345907: Alert for snapshot101[4567] not in mediawiki-installation dsh group.

the ops-dumps email alias ought to get notified about things like this; that way all the right people will see it.

Sep 8 2023, 1:28 PM · Data-Platform-SRE, Dumps-Generation, Data-Engineering

Sep 7 2023

ArielGlenn added a comment to T345874: XMLDumps broken on deployment-mwmaint02 due to Jade Extension related content.

Just a quick note that this breaks testing of dumps for enwiki in deployment-prep. We can work around it by testing only on other wikis, but it would be nice for this to be cleaned up.

Sep 7 2023, 5:43 PM · Dumps-Generation, MediaWiki-ContentHandler, Beta-Cluster-Infrastructure
ArielGlenn added a project to T345874: XMLDumps broken on deployment-mwmaint02 due to Jade Extension related content: Dumps-Generation.
Sep 7 2023, 5:42 PM · Dumps-Generation, MediaWiki-ContentHandler, Beta-Cluster-Infrastructure
ArielGlenn added a comment to T345186: Deployment training request for mabualruz.

We missed you today for the training. I'm guessing that something came up? Go ahead and reschedule, if you are still interested!

Sep 7 2023, 7:42 AM · Release-Engineering-Team (Deployment Training Requests)

Aug 31 2023

ArielGlenn moved T345176: {Investigation} Different file sizes for dumps from Backlog to Other teams on the Dumps-Generation board.
Aug 31 2023, 10:01 AM · Wikimedia Enterprise (sprint 53), Dumps-Generation

Aug 29 2023

ArielGlenn closed T344147: Puppet broken on mediawiki instances in deployment-prep as Resolved.

Thanks for the fix(es), everything is working as expected now.

Aug 29 2023, 3:43 PM · User-ArielGlenn, Beta-Cluster-Infrastructure

Aug 28 2023

ArielGlenn moved T143870: Some mw snapshot hosts are accessing main db servers from Blocked/Stalled/Waiting for event to Done on the Dumps-Generation board.
Aug 28 2023, 9:57 AM · MW-1.35-notes (1.35.0-wmf.10; 2019-12-10), Dumps-Generation
ArielGlenn moved T295909: <Platform Initiative> Dumps + WME Gap Analysis from Blocked/Stalled/Waiting for event to Done on the Dumps-Generation board.
Aug 28 2023, 9:57 AM · Platform Engineering, Dumps-Generation, Foundational Technology Requests
ArielGlenn moved T331129: cleanup_xmldumps is failing on dumpsdata1005 from Backlog to Done on the Dumps-Generation board.
Aug 28 2023, 9:57 AM · Dumps-Generation
ArielGlenn moved T318849: analytics-dumps-fetch-unique_devices.service failing on dumps servers from Other teams to Done on the Dumps-Generation board.
Aug 28 2023, 9:57 AM · Data-Engineering-Icebox, cloud-services-team, Analytics, Dumps-Generation
ArielGlenn moved T332562: Don't expose partial dumpfiles from Other teams to Done on the Dumps-Generation board.
Aug 28 2023, 9:57 AM · Dumps-Generation, Wikimedia Enterprise
ArielGlenn moved T341058: Failures for Cirrus Search dumps for wikidatawiki, zhwiki, enwiki for the 20230703 run from Other teams to Done on the Dumps-Generation board.
Aug 28 2023, 9:56 AM · Wikidata, MW-1.41-notes (1.41.0-wmf.19; 2023-07-25), Chinese-Sites, Dumps-Generation, Discovery-Search (Current work)
ArielGlenn added a comment to T336573: PHP Warning: XMLReader::read(): Memory allocation failed : growing input buffer.

Verified that with those same files from the above command the error is still present, nothing in the MW codebase has changed whatever the underlying issue is.

Aug 28 2023, 9:10 AM · Unstewarded-production-error, Dumps-Generation, Wikimedia-production-error
ArielGlenn added a comment to T249310: TextPassDumper.php messages on retry of fetch of content have extra junk in them.

Related: T324463

Aug 28 2023, 8:45 AM · Platform Team Workboards (Clinic Duty Team), Dumps-Generation
ArielGlenn added a comment to T306629: PHP Warning: failed to get text for revid [id] [Called from AbstractFilter::getText in /srv/mediawiki/php-1.39.0-wmf.7/extensions/ActiveAbstract/includes/AbstractFilter.php at line 195].

Who runs the findBadBlobs.php script in cases like this? It would be nice to get that done.

Aug 28 2023, 8:32 AM · User-brennen, Dumps-Generation, Wikimedia-production-error
ArielGlenn closed T133547: set up automated HTML (restbase) dumps on francium as Declined.

Not doing this, since we now have WME (Enterprise) dumps in HTML format available for public download.

Aug 28 2023, 8:26 AM · Platform Team Legacy (Watching / External), Services (watching), Datasets-Archiving, Dumps-Generation
ArielGlenn moved T186801: Flow page content dumps not resilient when database goes away from Backlog to Other teams on the Dumps-Generation board.
Aug 28 2023, 8:24 AM · Growth-Team-Filtering, Growth-Team, StructuredDiscussions, Dumps-Generation
ArielGlenn closed T331129: cleanup_xmldumps is failing on dumpsdata1005 as Resolved.

Going ahead and closing this.

Aug 28 2023, 8:20 AM · Dumps-Generation

Aug 24 2023

ArielGlenn added a comment to T344929: Maintenance script appservers running code from wmf.20 when version not live.

grep on mwmaint1002 for php, looking for long running stuff, gives me only

Jul11   0:00 /bin/bash /usr/local/bin/mwscript eval.php --wiki=commonswiki

The others are all Aug 22 or 23rd just fyi.

Aug 24 2023, 3:43 PM · Scap, Release-Engineering-Team, Wikimedia-production-error
ArielGlenn added a comment to T332562: Don't expose partial dumpfiles.

I"m presuming you didn't see any instances of this in the meantime, @awight ? Can we close this?

Aug 24 2023, 2:21 PM · Dumps-Generation, Wikimedia Enterprise
ArielGlenn added a comment to T320722: Deployment training request for Sergio Gimeno.

Hey @Sgs we missed you this morning at the deployment window for training. Or were you going to do the UTC late window this time?

Aug 24 2023, 7:31 AM · Release-Engineering-Team (Deployment Training Requests)

Aug 22 2023

ArielGlenn added a comment to T343325: Develop Dumps Triage Runbook.

I can try to dust off and restructure the troubleshooting guide on wikitech for the sql/xml dumps, if that would be helpful. This would by no means be a replacement for the runbook, but more of a minimal guide if people get stuck. Having a document specifically for dumps newcomers is great and I hope it will be expanded over time!

Aug 22 2023, 12:54 PM · Data Products (Sprint 00), Dumps 2.0, Data-Engineering

Aug 17 2023

ArielGlenn added a comment to T344409: Clarify whether config patches need reviews before being scheduled for deployment.

Counterpoint: knowing the config settings doesn't mean understanding the code activated by those changes or its possible impacts. At least, not for me. Some areas I know, and some I don't.

Aug 17 2023, 7:38 AM · Wikimedia-Site-requests, Release-Engineering-Team

Aug 16 2023

ArielGlenn updated the task description for T343882: Make wikidata dumps use two snapshot hosts to complete quicker.
Aug 16 2023, 11:13 AM · Patch-For-Review, Dumps-Generation

Aug 14 2023

ArielGlenn added a project to T344147: Puppet broken on mediawiki instances in deployment-prep: User-ArielGlenn.
Aug 14 2023, 12:01 PM · User-ArielGlenn, Beta-Cluster-Infrastructure
ArielGlenn added a comment to T344147: Puppet broken on mediawiki instances in deployment-prep.

Note that since dumps snapshot instances are sorta-kinda mediawiki instances, this affects them too.

Aug 14 2023, 11:58 AM · User-ArielGlenn, Beta-Cluster-Infrastructure
ArielGlenn triaged T344147: Puppet broken on mediawiki instances in deployment-prep as Medium priority.
Aug 14 2023, 11:58 AM · User-ArielGlenn, Beta-Cluster-Infrastructure

Aug 9 2023

ArielGlenn triaged T343882: Make wikidata dumps use two snapshot hosts to complete quicker as Medium priority.
Aug 9 2023, 12:44 PM · Patch-For-Review, Dumps-Generation

Aug 7 2023

ArielGlenn updated the task description for T341045: Get Data Engineering folks access to hosts and systems needed for maintenance of the existing dumps system.
Aug 7 2023, 12:40 PM · Data-Engineering, Dumps-Generation

Aug 5 2023

ArielGlenn triaged T343621: Wikidata rdf lexemes dump failed for Fri Aug 4, due to db conn error as Medium priority.
Aug 5 2023, 6:48 AM · Wikidata, [DEPRECATED] wdwb-tech, Wikidata Lexicographical data, Patch-For-Review, Dumps-Generation

Jul 27 2023

ArielGlenn added a comment to T341559: Deployment Training Request for jebe.

This training happened, though it was a lot less interactive and useful than it could have been because no patches were scheduled and no one showed up with a patch last minute, in spite of me begging :-D But we went through a description of all the steps, looked at all the relevant dashboards and hosts and commands, so there's that :-)

Jul 27 2023, 7:57 AM · Release-Engineering-Team (Deployment Training Requests)

Jul 19 2023

ArielGlenn added a comment to T325232: Migrate Dumpsdata and Htmldumper Hosts From Buster to Bullseye.

Dumpsdata1007, running bullseye, is now the fallback host for sql/xml and misc dumps. This means all hosts in production (not spares) are on bullseye now and this task can be closed after a day or so just to make sure things are stable.

Jul 19 2023, 3:41 PM · Data-Platform-SRE (2024.03.25 - 2024.04.14), Patch-For-Review, Dumps-Generation

Jul 17 2023

ArielGlenn added a comment to T295909: <Platform Initiative> Dumps + WME Gap Analysis.

Sounds great to me, thanks!

Jul 17 2023, 12:09 PM · Platform Engineering, Dumps-Generation, Foundational Technology Requests
ArielGlenn updated the task description for T341045: Get Data Engineering folks access to hosts and systems needed for maintenance of the existing dumps system.
Jul 17 2023, 10:06 AM · Data-Engineering, Dumps-Generation

Jul 13 2023

ArielGlenn added a comment to T341058: Failures for Cirrus Search dumps for wikidatawiki, zhwiki, enwiki for the 20230703 run.

Just for my understanding, it looks like the new patch would exception out in the case where there is a failure with the last connection of whatever sort. Am I reading that right? And if so, how does that help us in the current situation? Sorry for whatever I am missing here. Thanks!

Jul 13 2023, 11:15 AM · Wikidata, MW-1.41-notes (1.41.0-wmf.19; 2023-07-25), Chinese-Sites, Dumps-Generation, Discovery-Search (Current work)
ArielGlenn added a comment to T339929: custom partman recipe dumpsdata100X-no-data-format.cfg causes installer to hang at partitioning menu.

Hey @elukey (or anyone else watching who wants to chime in), I've got a recipe that might maybe possibly could work. (See patch above.) But I have questions. Some recipes in the repo deal with lvm partitions by "unknown ignore" instead of "lvmpv keep", and I wonder which is better. Some recipes without swap specify "d-i partman-basicfilesystems/no_swap boolean false" and some do not, and I wonder which is right. And last but not least, is it still the procedure for testing before merge, to announce "hey I'm testing on installX00Y and disabling puppet for awhile" in the channel and hoping no one speaks up? Thanks in advance!

Jul 13 2023, 10:50 AM · Patch-For-Review, Dumps-Generation

Jul 12 2023

ArielGlenn added a comment to T329491: ICU transition towards ICU 67.

A note that I did a test run of sql/xml dumps on deployment-prep with the new icu version and it looks fine to me, though I didn't check for any weird details of category sorting or whatever.

Jul 12 2023, 1:10 PM · serviceops-radar, SRE

Jul 11 2023

ArielGlenn updated the task description for T341045: Get Data Engineering folks access to hosts and systems needed for maintenance of the existing dumps system.
Jul 11 2023, 2:52 PM · Data-Engineering, Dumps-Generation
ArielGlenn added a comment to T341557: Grant Access to wmf for Jennifer Ebe.

Jennifer is already a member of wmf

https://ldap.toolforge.org/user/jebe

Double checked.

btullis@seaborgium:~$ ldapsearch -A -x member=uid=jebe,ou=people,dc=wikimedia,dc=org dn
# extended LDIF
#
# LDAPv3
# base <dc=wikimedia,dc=org> (default) with scope subtree
# filter: member=uid=jebe,ou=people,dc=wikimedia,dc=org
# requesting: dn 
#

# wmf, groups, wikimedia.org
dn: cn=wmf,ou=groups,dc=wikimedia,dc=org

# project-bastion, groups, wikimedia.org
dn: cn=project-bastion,ou=groups,dc=wikimedia,dc=org

# project-analytics, groups, wikimedia.org
dn: cn=project-analytics,ou=groups,dc=wikimedia,dc=org

# project-deployment-prep, groups, wikimedia.org
dn: cn=project-deployment-prep,ou=groups,dc=wikimedia,dc=org

# search result
search: 2
result: 0 Success

# numResponses: 5
# numEntries: 4
Jul 11 2023, 1:35 PM · SRE, LDAP-Access-Requests
ArielGlenn updated subscribers of T341557: Grant Access to wmf for Jennifer Ebe.

See also https://phabricator.wikimedia.org/T341045 for the context. @WDoranWMF please sign off just in case that's needed. Thanks!

Jul 11 2023, 11:02 AM · SRE, LDAP-Access-Requests

Jul 10 2023

ArielGlenn added a comment to T339929: custom partman recipe dumpsdata100X-no-data-format.cfg causes installer to hang at partitioning menu.

It sounds like the reuse-parts.cfg script is the way to go. Let me poke around and see how that's used elsewhere, and I'll come back if I get stuck. Thanks!

Jul 10 2023, 4:32 PM · Patch-For-Review, Dumps-Generation

Jul 9 2023

ArielGlenn added a comment to T341058: Failures for Cirrus Search dumps for wikidatawiki, zhwiki, enwiki for the 20230703 run.

One more to add:

Jul  4 09:16:50 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20230703/commonswiki-20230703-cirrussearch-content.json.gz
Jul 9 2023, 3:38 PM · Wikidata, MW-1.41-notes (1.41.0-wmf.19; 2023-07-25), Chinese-Sites, Dumps-Generation, Discovery-Search (Current work)
ArielGlenn added a comment to T341045: Get Data Engineering folks access to hosts and systems needed for maintenance of the existing dumps system.

@JEbe-WMF you will need to folllow the instructions here https://wikitech.wikimedia.org/wiki/SRE/Clinic_Duty/Access_requests#Checklist and create a task, feel free to add me as a subscriber and link this one to it. Make sure you ask for membership in the wmf ldap group. That will give you icinga/grafana/logstash access.

Jul 9 2023, 3:17 PM · Data-Engineering, Dumps-Generation
ArielGlenn added a comment to T341045: Get Data Engineering folks access to hosts and systems needed for maintenance of the existing dumps system.

Dan and Xabriel already are members of the wmf group, giving access to grafana and icinga (though contact info might need to be added for executing commands on icinga). Jennifer is not yet in the group.

Jul 9 2023, 3:12 PM · Data-Engineering, Dumps-Generation
ArielGlenn updated the task description for T341045: Get Data Engineering folks access to hosts and systems needed for maintenance of the existing dumps system.
Jul 9 2023, 3:07 PM · Data-Engineering, Dumps-Generation
ArielGlenn updated the task description for T341045: Get Data Engineering folks access to hosts and systems needed for maintenance of the existing dumps system.
Jul 9 2023, 3:04 PM · Data-Engineering, Dumps-Generation
ArielGlenn updated the task description for T341045: Get Data Engineering folks access to hosts and systems needed for maintenance of the existing dumps system.
Jul 9 2023, 3:03 PM · Data-Engineering, Dumps-Generation
ArielGlenn added a comment to T325232: Migrate Dumpsdata and Htmldumper Hosts From Buster to Bullseye.

Swapped dumpsdata1003 in as the live nfs share for misc dumps.

Jul 9 2023, 2:51 PM · Data-Platform-SRE (2024.03.25 - 2024.04.14), Patch-For-Review, Dumps-Generation

Jul 6 2023

ArielGlenn added a comment to T341045: Get Data Engineering folks access to hosts and systems needed for maintenance of the existing dumps system.

@JEbe-WMF and @xcollazo you should both sign up for MediaWiki deployment training here: https://phabricator.wikimedia.org/project/board/5265/ and get scheduled for that. Once that's done, we can add you to the deployers list in puppet. (Dan you are already a deployer so you're off the hook ;-) )

Jul 6 2023, 11:48 AM · Data-Engineering, Dumps-Generation
ArielGlenn updated the task description for T341045: Get Data Engineering folks access to hosts and systems needed for maintenance of the existing dumps system.
Jul 6 2023, 11:42 AM · Data-Engineering, Dumps-Generation
ArielGlenn added a comment to T341045: Get Data Engineering folks access to hosts and systems needed for maintenance of the existing dumps system.

@JEbe-WMF @Milimetric and @xcollazo would you please subscribe to https://lists.wikimedia.org/postorius/lists/xmldatadumps-l.lists.wikimedia.org/ and let me know which email addresses you used? I will add them as co-admins of the list. Thanks!

Jul 6 2023, 11:37 AM · Data-Engineering, Dumps-Generation
ArielGlenn updated subscribers of T341045: Get Data Engineering folks access to hosts and systems needed for maintenance of the existing dumps system.

@RobH Would you be willing to add Milimetric, xcollazo and Jebe-WMF to the #acl*procurement-review acl so that they can view procurement tasks? I hope no new tasks will come up for some time but just in case, and it will let us look at psat ones and discuss. They will be working with me on the dumps now. Thanks!

Jul 6 2023, 11:35 AM · Data-Engineering, Dumps-Generation
ArielGlenn updated the task description for T341045: Get Data Engineering folks access to hosts and systems needed for maintenance of the existing dumps system.
Jul 6 2023, 11:33 AM · Data-Engineering, Dumps-Generation
ArielGlenn updated subscribers of T341045: Get Data Engineering folks access to hosts and systems needed for maintenance of the existing dumps system.

@WDoranWMF we will need your approval for this.

Jul 6 2023, 11:20 AM · Data-Engineering, Dumps-Generation
ArielGlenn updated subscribers of T341045: Get Data Engineering folks access to hosts and systems needed for maintenance of the existing dumps system.

After a conversation with Will, Dan and others, the people who need the above access are @Milimetric , @xcollazo and @JEbe-WMF so now let me get started on that.

Jul 6 2023, 11:12 AM · Data-Engineering, Dumps-Generation

Jul 5 2023

ArielGlenn added a comment to T341058: Failures for Cirrus Search dumps for wikidatawiki, zhwiki, enwiki for the 20230703 run.

Adding two more failures:

Jul  4 09:16:50 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20230703/mlwiki-20230703-cirrussearch-content.json.gz
Jul  4 09:16:50 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20230703/metawiki-20230703-cirrussearch-general.json.gz

These came from different db sections so the script running those finished later, and hence the error report was sent later.

Jul 5 2023, 12:19 PM · Wikidata, MW-1.41-notes (1.41.0-wmf.19; 2023-07-25), Chinese-Sites, Dumps-Generation, Discovery-Search (Current work)