Page MenuHomePhabricator

ArielGlenn (ariel)
User

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Friday

  • Clear sailing ahead.

User Details

User Since
Oct 8 2014, 7:09 PM (441 w, 6 d)
Availability
Available
IRC Nick
apergos
LDAP User
ArielGlenn
MediaWiki User
ArielGlenn [ Global Accounts ]

Recent Activity

Mon, Mar 27

ArielGlenn added a comment to T333102: SystemdUnitDownForLong clouddumps1001:9100 Unit rsync_enterprise_htmldumps.service on node clouddumps1001 has been down for long..

Hey @Andrew I'm guessing this patch is related: https://gerrit.wikimedia.org/r/c/operations/puppet/+/902833

Mon, Mar 27, 11:55 AM · cloud-services-team
ArielGlenn updated the task description for T330165: eqiad row B switches upgrade.
Mon, Mar 27, 10:02 AM · Patch-For-Review, Data Pipelines, Data-Engineering-Planning, DBA, Discovery-Search (Current work), SRE, serviceops, cloud-services-team, Machine-Learning-Team, Platform Engineering, SRE Observability, Infrastructure-Foundations, serviceops-collab, Traffic

Thu, Mar 23

ArielGlenn added a comment to T332544: Provide an updated "latest" link for HTML dumps.

Hey @awight I see what you're getting at, and while that would be a cool feature, it's not in the cards for the current dumps. If they get a revamp at some point, that would be a nice thing to have, definitely. In the meantime I'm glad you have a workaround. One thing that would be nice in our glorious future would be some uniformity as to how various datasets for public download are published, but that's not coming any time soon...

Thu, Mar 23, 6:23 PM · Dumps-Generation, Wikimedia Enterprise
ArielGlenn added a project to T331761: Some dumps do not have checksums: Cloud-Services.
Thu, Mar 23, 1:33 PM · Data-Services, cloud-services-team, Datasets-General-or-Unknown, Community-Wishlist-Survey-2023, Dumps-Generation
ArielGlenn added a comment to T331761: Some dumps do not have checksums.

Summarizing from a conversation on IRC:

Thu, Mar 23, 1:27 PM · Data-Services, cloud-services-team, Datasets-General-or-Unknown, Community-Wishlist-Survey-2023, Dumps-Generation

Wed, Mar 22

ArielGlenn added a comment to T332544: Provide an updated "latest" link for HTML dumps.

I'm not sure I have a clear picture of what you want here, @awight Do you want a link to the directory once/if the last dump is downloaded successfully into it? So e.g. right now there could be link 'latest' at the same level as the date directories, pointing to 20230320 (asuming that's complete); is this what you are looking for, or something slightly different?

Wed, Mar 22, 3:16 PM · Dumps-Generation, Wikimedia Enterprise
ArielGlenn added a comment to T332562: Don't expose partial dumpfiles.

Hmm we already write to a temp file with the file name ending in ".tmp". So I wonder why the size is changing. The file is written into the same directory and moved with a rename so there should be no time period when the final file is growing.

Wed, Mar 22, 2:56 PM · Dumps-Generation, Wikimedia Enterprise
ArielGlenn added a comment to T332562: Don't expose partial dumpfiles.

We could download WME dumps to a name that ends with ".inprog" and move them once the download is complete; this is how generation of our sql/xml dumps works, so it would fit right in.

Wed, Mar 22, 1:11 PM · Dumps-Generation, Wikimedia Enterprise
ArielGlenn moved T332562: Don't expose partial dumpfiles from Backlog to Other teams on the Dumps-Generation board.
Wed, Mar 22, 1:07 PM · Dumps-Generation, Wikimedia Enterprise
ArielGlenn moved T332544: Provide an updated "latest" link for HTML dumps from Backlog to Other teams on the Dumps-Generation board.
Wed, Mar 22, 1:07 PM · Dumps-Generation, Wikimedia Enterprise

Sun, Mar 19

ArielGlenn added a comment to T330573: Resolve space issues on dumpsdata1001, 1003.

The swap has been done, verified via testing on snapshot1009 (testbed) for xml/sql dumps. Did not test adds-changes dumps but did sort out the ports for lockd so it should be ok. Wikitech page partially updated, will put the specs of the new hosts in later, ran out of oomph.

Sun, Mar 19, 4:51 PM · Dumps-Generation

Fri, Mar 3

ArielGlenn added a comment to T330573: Resolve space issues on dumpsdata1001, 1003.

rsync to dumpsdata1005 is about halfway done copying the old data from dumpsdata1001. we'll need to start updating it (with care) from the fresher data on dumpsdata1004, so that it's ready to swap in once the xml dumps run completes at around the 14th or 15th of the month.

Fri, Mar 3, 6:50 AM · Dumps-Generation

Thu, Mar 2

ArielGlenn triaged T330793: Address dataset frontrunning attack in dumps as Low priority.

We've discussed this a little here and randomising the order of retrieval is probably not the approach we would take, for performance reasons. But there are other possibilities. We would need to assess the risk and possible harm and weigh that against the work needed and resources available. For now we do have urgent scaling work and server replacements that need to take precedence.

Thu, Mar 2, 3:49 PM · Dumps-Generation
ArielGlenn moved T330936: Missing Cirrussearch dump (enwiki and wikidata) from Backlog to Other teams on the Dumps-Generation board.
Thu, Mar 2, 11:49 AM · Discovery-Search (Current work), CirrusSearch, Dumps-Generation
ArielGlenn added a comment to T330573: Resolve space issues on dumpsdata1001, 1003.

This graph shows the state of things over the past month on dumpsdata1003. We'll need to manually do cleanup of bz2 files from the previous full run for wikidatawiki as the new run progresses, in order to avoid space issues, and then swap in a new dumpsdata host once the run completes.

Στιγμιότυπο οθόνης από 2023-03-02 10-45-32.png (291×828 px, 25 KB)

Thu, Mar 2, 8:49 AM · Dumps-Generation
ArielGlenn added a comment to T330573: Resolve space issues on dumpsdata1001, 1003.

rsync to dumpsdata1005 has started, from dumpsdata1001, so we'll not have completely fresh data but we'll have the bulk of it. After that we'll do the same for dumpsdata1006 and 7,

Thu, Mar 2, 8:41 AM · Dumps-Generation

Wed, Mar 1

ArielGlenn moved T330573: Resolve space issues on dumpsdata1001, 1003 from Backlog to Active on the Dumps-Generation board.
Wed, Mar 1, 5:51 PM · Dumps-Generation
ArielGlenn added a comment to T330573: Resolve space issues on dumpsdata1001, 1003.

dumpsdata1006 /data partition resized, now all 4 new boxes have the correct size filesystem on /data.

Wed, Mar 1, 3:57 PM · Dumps-Generation
ArielGlenn added a comment to T330573: Resolve space issues on dumpsdata1001, 1003.

Besides watching today to make sure nothing is amiss, we should next:

Wed, Mar 1, 7:35 AM · Dumps-Generation
ArielGlenn added a comment to T299443: Q3: rack/setup/install dumpsdata100[67].

Wonderful, we have claimed them already :-) Thank you!

Wed, Mar 1, 7:27 AM · Dumps-Generation, SRE, ops-eqiad, DC-Ops

Mon, Feb 27

ArielGlenn moved T330686: Wikidata dumps wikidata-20230220-all.json.bzip2 file missing from 20230220 dump from Backlog to Other teams on the Dumps-Generation board.
Mon, Feb 27, 6:31 PM · Dumps-Generation
ArielGlenn added a comment to T299443: Q3: rack/setup/install dumpsdata100[67].

Awesome, I would have looked for you on irc in a few days if I hadn't heard anything, no worries. Happy to see this moving along!

Mon, Feb 27, 2:58 PM · Dumps-Generation, SRE, ops-eqiad, DC-Ops
ArielGlenn added a comment to T330573: Resolve space issues on dumpsdata1001, 1003.

Full speed rsync of xml dumps from dumpsdata1003 to dumpsdata1004 is underway, currently on commonswiki.

Mon, Feb 27, 1:09 PM · Dumps-Generation

Feb 25 2023

ArielGlenn added a comment to T330573: Resolve space issues on dumpsdata1001, 1003.

Note that dumpsdata1001 and 1002 are intended to be replaced and then decommed, so ideally we would swap in dumpsdata1004,5, (6 or 7) for all of 1001,2,3 increasing the size of each and freeing up those two boxes. We can then shuffle things around later (i.e. 1003 to be the 'other dumps' nfs filesystem, freeing up a larger box for use with xml dumps one way or another).

Feb 25 2023, 4:11 PM · Dumps-Generation
ArielGlenn triaged T330573: Resolve space issues on dumpsdata1001, 1003 as High priority.
Feb 25 2023, 4:00 PM · Dumps-Generation
ArielGlenn moved T325232: Migrate Dumpsdata and Htmldumper Hosts From Buster to Bullseye from Backlog to Active on the Dumps-Generation board.
Feb 25 2023, 3:55 PM · Dumps-Generation
ArielGlenn added a comment to T299443: Q3: rack/setup/install dumpsdata100[67].

Hey @RobH I am preemptively assuming that dumpsdata1007 is good to go for us to use, since it's got the role ("insetup::core_platform") and everything in the checklist is marked off. Basically we need these boxes, at least the one, tell me if it's not ready to be ours yet and I'll stop mucking about with it.

Feb 25 2023, 3:28 PM · Dumps-Generation, SRE, ops-eqiad, DC-Ops

Feb 24 2023

ArielGlenn added a comment to T330464: Import failed: XML error at line 1: Validation failed: no DTD found !.

Guess it's in the php innards then, somewhere in xmlreader and friends...

Feb 24 2023, 2:05 PM · Regression, MW-1.40-release, PHP 8.2 support, MediaWiki-Core-Snapshots
ArielGlenn added a comment to T330464: Import failed: XML error at line 1: Validation failed: no DTD found !.

@Reedy What version of ibxml2 is your PHP 8.2.16 using, compared to 8.1.16? Maybe something changed regarding entity loading between the two respective versions?

Feb 24 2023, 12:10 PM · Regression, MW-1.40-release, PHP 8.2 support, MediaWiki-Core-Snapshots

Feb 6 2023

ArielGlenn added a comment to T260223: Kiwix rsyncs not completing and stacking up on Clouddumps1001,2.

Fri-Sat-Sun: 16.15 run, error reported from clouddumps1002

@ ERROR: max connections (6) reached -- try again later

All other runs good.

Feb 6 2023, 9:24 AM · cloud-services-team, affects-Kiwix-and-openZIM, Dumps-Generation
ArielGlenn added a comment to T322156: "No server with index" and "Warning: Undefined index" from LoadBalancer::reconfigure.

is this still an issue?

Feb 6 2023, 9:01 AM · MW-1.40-notes (1.40.0-wmf.27; 2023-03-13), Performance-Team (Radar), Patch-For-Review, MediaWiki-libs-Rdbms, Wikimedia-production-error, DBA, Dumps-Generation

Feb 3 2023

ArielGlenn added a comment to T326278: Error: Cannot use object of type ContentTranslation\DTO\TranslationUnitDTO as array.

Just noting that we haven't seen this error for the last two weeks so it looks like the issue is resolved for us (Content Translation dumps) as well.

Feb 3 2023, 9:45 PM · MW-1.40-notes (1.40.0-wmf.18; 2023-01-09), Language-Team (Language-2023-January-March), ContentTranslation, Wikimedia-production-error
ArielGlenn added a project to T314697: Allow use of wmf's MW CLI scripts on snapshot hosts instead of bypassing: Dumps-Generation.
Feb 3 2023, 9:44 PM · Dumps-Generation, Growth-Team, Developer Productivity, serviceops, Datasets-General-or-Unknown
ArielGlenn added a comment to T328804: Make the Dumps Stats job faster.

Some options for the longer term might be:

  • keep track of uncompressed data sizes and write these numbers out along with the compressed bz2 files (would take a chunk f messing about with XMLWriter innards but would be very fast to generate stats afterwards)
  • see about the revision content in the data lake and how costly it would be to just sum the lengths up each month for enwiki plus $randomwiki
  • give an uppper bound using the size of the uncompressed data in the adds-changes dump each day (this would require choosing the random wiki at the beginning of the month and keeping a running total each day but it would be pretty fast, if less accurate)
Feb 3 2023, 7:23 PM · Dumps-Generation

Jan 30 2023

ArielGlenn added a comment to T260223: Kiwix rsyncs not completing and stacking up on Clouddumps1001,2.

Hey WMCS folks, see the above patch for the change of upstream hostname for kiwix downloads, I figure these are your boxes so you ought to give the thumbs up/merge. @Andrew you have been added as a reviewer but feel free to swap anyone else in as you see fit.

Jan 30 2023, 5:54 PM · cloud-services-team, affects-Kiwix-and-openZIM, Dumps-Generation

Jan 10 2023

ArielGlenn added a comment to T226093: Capacity planning for Commons Structured Data.

The task's original intent was to cover planning "over the next 3 years" starting in 2019. @ArielGlenn is the task still relevant, can it be closed, do we need a new one?

Jan 10 2023, 1:05 PM · wdwb-tech, Dumps-Generation, Wikidata, SDC General
ArielGlenn added a comment to T326278: Error: Cannot use object of type ContentTranslation\DTO\TranslationUnitDTO as array.

The weekly content translation dumps were also impacted by this, sample stack trace:

[e2d5d9d71060942e3def717f] [no req]   Error: Cannot use object of type ContentTranslation\DTO\TranslationUnitDTO as array
Backtrace:
from /srv/mediawiki/php-1.40.0-wmf.17/extensions/ContentTranslation/includes/CorporaLookup.php(87)
#0 /srv/mediawiki/php-1.40.0-wmf.17/extensions/ContentTranslation/includes/CorporaLookup.php(51): ContentTranslation\CorporaLookup::format(Wikimedia\Rdbms\MysqliResultWrapper)
#1 /srv/mediawiki/php-1.40.0-wmf.17/extensions/ContentTranslation/scripts/dump-corpora.php(140): ContentTranslation\CorporaLookup->getByTranslationId(integer)
#2 /srv/mediawiki/php-1.40.0-wmf.17/maintenance/includes/MaintenanceRunner.php(496): CXCorporaDump->execute()
#3 /srv/mediawiki/php-1.40.0-wmf.17/maintenance/doMaintenance.php(85): MediaWiki\Maintenance\MaintenanceRunner->run()
#4 /srv/mediawiki/php-1.40.0-wmf.17/extensions/ContentTranslation/scripts/dump-corpora.php(316): require_once(string)
#5 /srv/mediawiki/multiversion/MWScript.php(120): require_once(string)
#6 {main}
Jan 10 2023, 12:47 PM · MW-1.40-notes (1.40.0-wmf.18; 2023-01-09), Language-Team (Language-2023-January-March), ContentTranslation, Wikimedia-production-error
ArielGlenn added a comment to T265056: Make Cirrus Search dump script more resilient to failures (elasticsearch restarts).

Today's report:

<13>Jan  9 19:14:29 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20230109/wikidatawiki-20230109-cirrussearch-general.json.gz
Jan 10 2023, 10:37 AM · MW-1.40-notes (1.40.0-wmf.19; 2023-01-16), Patch-For-Review, Discovery-Search (Current work), CirrusSearch, Dumps-Generation

Dec 21 2022

ArielGlenn added a comment to T323837: Mirror Wikimedia Foundation, Wikipedia, wikimediatech, and MediaWiki Twitter accounts on Mastodon.

Adding this here, https://www.eff.org/deeplinks/2022/12/user-generated-content-and-fediverse-legal-primer for those of us not on the legal team and who may not be aware of the legal issues involved in hosting one's own instance of Mastodon. Some thought would need to go into managing trust and safety issues as well, in particular if the instance is open to use by community members and not just WMF employees.

Dec 21 2022, 9:27 AM · Developer-Advocacy, WMF-Communications

Dec 14 2022

ArielGlenn added a comment to T324954: Provide zstd compression algorithm for dumps.

See also T222985

Dec 14 2022, 12:00 PM · Dumps-Generation

Dec 13 2022

ArielGlenn added a comment to T324954: Provide zstd compression algorithm for dumps.

The reason we go with bz2 is that it is block-oriented, allowing easier parallel processing of input files for platforms that rely on massive parallelization such as Hadoop. It also allows us to recover if part of a file is written out and the dump process aborts for some reason. It's possble that other compression algorithms could be adopted in a future re-architected version of the dumps.

Dec 13 2022, 11:16 AM · Dumps-Generation

Dec 8 2022

ArielGlenn added a comment to T265056: Make Cirrus Search dump script more resilient to failures (elasticsearch restarts).

Yesterday's report:

<13>Nov 30 15:39:42 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20221123/frwiki-20221123-cirrussearch-general.json.gz
Dec 8 2022, 10:47 AM · MW-1.40-notes (1.40.0-wmf.19; 2023-01-16), Patch-For-Review, Discovery-Search (Current work), CirrusSearch, Dumps-Generation

Dec 5 2022

ArielGlenn added a comment to T324463: Dumps Exception email did not list full exception.

Note that because there's no explicit error or exception in the logs, this is just harmless noise, but it would be nice to understand where it comes from and decide what to do about it at some point (hence, low priority).

Dec 5 2022, 2:44 PM · Dumps-Generation

Nov 28 2022

ArielGlenn added a comment to T265056: Make Cirrus Search dump script more resilient to failures (elasticsearch restarts).

Last Wednesday's report:

<13>Nov 18 12:16:54 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20221115/cowiki-20221115-cirrussearch-content.json.gz
<13>Nov 23 12:57:44 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20221115/wikidatawiki-20221115-cirrussearch-general.json.gz
Nov 28 2022, 7:35 AM · MW-1.40-notes (1.40.0-wmf.19; 2023-01-16), Patch-For-Review, Discovery-Search (Current work), CirrusSearch, Dumps-Generation

Nov 23 2022

ArielGlenn added a comment to T322156: "No server with index" and "Warning: Undefined index" from LoadBalancer::reconfigure.

We saw some new errors of this sort during the abstracts dump job. Sample stacktrace:

[20221121172444]: InvalidArgumentException from line 2319 of /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/rdbms/loadbalancer/LoadBalancer.php: No server with index '4'
#0 /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1170): Wikimedia\Rdbms\LoadBalancer->getServerInfoStrict(4)
#1 /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1102): Wikimedia\Rdbms\LoadBalancer->reallyOpenConnection(4, Object(Wikimedia\Rdbms\DatabaseDomain), Array)
#2 /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/rdbms/loadbalancer/LoadBalancer.php(945): Wikimedia\Rdbms\LoadBalancer->reuseOrOpenConnectionForNewRef(4, Object(Wikimedia\Rdbms\DatabaseDomain), 3)
#3 /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/rdbms/loadmonitor/LoadMonitor.php(237): Wikimedia\Rdbms\LoadBalancer->getServerConnection(4, '', 3)
#4 /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/rdbms/loadmonitor/LoadMonitor.php(172): Wikimedia\Rdbms\LoadMonitor->computeServerStates(Array, Array)
#5 /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/objectcache/wancache/WANObjectCache.php(1755): Wikimedia\Rdbms\LoadMonitor->Wikimedia\Rdbms\{closure}(Array, 1, Array, 1669051415.1476, Array)
#6 /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/objectcache/wancache/WANObjectCache.php(1585): WANObjectCache->fetchOrRegenerate('global:rdbms-se...', 1, Object(Closure), Array, Array)
#7 /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/rdbms/loadmonitor/LoadMonitor.php(181): WANObjectCache->getWithSetCallback('global:rdbms-se...', 1, Object(Closure), Array)
#8 /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/rdbms/loadmonitor/LoadMonitor.php(104): Wikimedia\Rdbms\LoadMonitor->getServerStates(Array)
#9 /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/rdbms/loadbalancer/LoadBalancer.php(547): Wikimedia\Rdbms\LoadMonitor->scaleLoads(Array)
#10 /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/rdbms/loadbalancer/LoadBalancer.php(507): Wikimedia\Rdbms\LoadBalancer->getReaderIndex('dump')
#11 /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/rdbms/loadbalancer/LoadBalancer.php(920): Wikimedia\Rdbms\LoadBalancer->getConnectionIndex(-1, Array, 'cebwiki')
#12 /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/rdbms/database/DBConnRef.php(103): Wikimedia\Rdbms\LoadBalancer->getConnectionInternal(-1, Array, 'cebwiki', 0)
#13 /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/rdbms/database/DBConnRef.php(117): Wikimedia\Rdbms\DBConnRef->ensureConnection()
#14 /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/rdbms/database/DBConnRef.php(356): Wikimedia\Rdbms\DBConnRef->__call('selectRow', Array)
#15 /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/rdbms/querybuilder/SelectQueryBuilder.php(689): Wikimedia\Rdbms\DBConnRef->selectRow(Array, Array, Array, 'MediaWiki\\Page\\...', Array, Array)
#16 /srv/mediawiki/php-1.40.0-wmf.10/includes/page/PageStore.php(199): Wikimedia\Rdbms\SelectQueryBuilder->fetchRow()
#17 /srv/mediawiki/php-1.40.0-wmf.10/includes/cache/LinkCache.php(461): MediaWiki\Page\PageStore->MediaWiki\Page\{closure}(Object(Wikimedia\Rdbms\DBConnRef), 0, 'Oil_Manikin', Array)
#18 /srv/mediawiki/php-1.40.0-wmf.10/includes/cache/LinkCache.php(494): LinkCache->getGoodLinkRowInternal(Object(TitleValue), Object(Closure), 0)
#19 /srv/mediawiki/php-1.40.0-wmf.10/includes/page/PageStore.php(188): LinkCache->getGoodLinkRow(0, 'Oil_Manikin', Object(Closure), 0)
#20 /srv/mediawiki/php-1.40.0-wmf.10/includes/page/PageStore.php(154): MediaWiki\Page\PageStore->getPageByNameViaLinkCache(0, 'Oil_Manikin', 0)
#21 /srv/mediawiki/php-1.40.0-wmf.10/includes/page/PageStore.php(326): MediaWiki\Page\PageStore->getPageByName(0, 'Oil_Manikin', 0)
#22 /srv/mediawiki/php-1.40.0-wmf.10/includes/title/Title.php(4108): MediaWiki\Page\PageStore->getPageByReference(Object(Title), 0)
#23 /srv/mediawiki/php-1.40.0-wmf.10/includes/title/Title.php(1100): Title->getFieldFromPageStore('page_content_mo...', 0)
#24 /srv/mediawiki/php-1.40.0-wmf.10/extensions/ActiveAbstract/includes/AbstractFilter.php(131): Title->getContentModel()
#25 /srv/mediawiki/php-1.40.0-wmf.10/includes/export/DumpFilter.php(81): MediaWiki\Extension\ActiveAbstract\AbstractFilter->writeClosePage('  </page>\n')
#26 /srv/mediawiki/php-1.40.0-wmf.10/includes/export/ExportProgressFilter.php(44): DumpFilter->writeClosePage('  </page>\n')
#27 /srv/mediawiki/php-1.40.0-wmf.10/includes/export/WikiExporter.php(613): ExportProgressFilter->writeClosePage('  </page>\n')
#28 /srv/mediawiki/php-1.40.0-wmf.10/includes/export/WikiExporter.php(501): WikiExporter->finishPageStreamOutput(Object(stdClass))
#29 /srv/mediawiki/php-1.40.0-wmf.10/includes/export/WikiExporter.php(315): WikiExporter->dumpPages('page_id >= 3220...', false)
#30 /srv/mediawiki/php-1.40.0-wmf.10/includes/export/WikiExporter.php(198): WikiExporter->dumpFrom('page_id >= 3220...', false)
#31 /srv/mediawiki/php-1.40.0-wmf.10/maintenance/includes/BackupDumper.php(359): WikiExporter->pagesByRange(3220001, 3230001, false)
#32 /srv/mediawiki/php-1.40.0-wmf.10/maintenance/dumpBackup.php(84): BackupDumper->dump(2, 0)
#33 /srv/mediawiki/php-1.40.0-wmf.10/maintenance/includes/MaintenanceRunner.php(309): DumpBackup->execute()
#34 /srv/mediawiki/php-1.40.0-wmf.10/maintenance/doMaintenance.php(85): MediaWiki\Maintenance\MaintenanceRunner->run()
#35 /srv/mediawiki/php-1.40.0-wmf.10/maintenance/dumpBackup.php(144): require_once('/srv/mediawiki/...')
#36 /srv/mediawiki/multiversion/MWScript.php(120): require_once('/srv/mediawiki/...')
#37 {main}
Wikimedia\Rdbms\DBConnectionError from line 1341 of /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/rdbms/loadbalancer/LoadBalancer.php: Cannot access the database: could not connect to any replica DB server
#0 /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/rdbms/loadbalancer/LoadBalancer.php(514): Wikimedia\Rdbms\LoadBalancer->reportConnectionError('could not conne...')
#1 /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/rdbms/loadbalancer/LoadBalancer.php(920): Wikimedia\Rdbms\LoadBalancer->getConnectionIndex(-1, Array, 'cebwiki')
#2 /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/rdbms/database/DBConnRef.php(103): Wikimedia\Rdbms\LoadBalancer->getConnectionInternal(-1, Array, 'cebwiki', 0)
#3 /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/rdbms/database/DBConnRef.php(117): Wikimedia\Rdbms\DBConnRef->ensureConnection()
#4 /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/rdbms/database/DBConnRef.php(356): Wikimedia\Rdbms\DBConnRef->__call('selectRow', Array)
#5 /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/rdbms/querybuilder/SelectQueryBuilder.php(689): Wikimedia\Rdbms\DBConnRef->selectRow(Array, Array, Array, 'MediaWiki\\Page\\...', Array, Array)
#6 /srv/mediawiki/php-1.40.0-wmf.10/includes/page/PageStore.php(199): Wikimedia\Rdbms\SelectQueryBuilder->fetchRow()
#7 /srv/mediawiki/php-1.40.0-wmf.10/includes/cache/LinkCache.php(461): MediaWiki\Page\PageStore->MediaWiki\Page\{closure}(Object(Wikimedia\Rdbms\DBConnRef), 0, 'Tangudla_Gutta', Array)
#8 /srv/mediawiki/php-1.40.0-wmf.10/includes/cache/LinkCache.php(494): LinkCache->getGoodLinkRowInternal(Object(TitleValue), Object(Closure), 0)
#9 /srv/mediawiki/php-1.40.0-wmf.10/includes/page/PageStore.php(188): LinkCache->getGoodLinkRow(0, 'Tangudla_Gutta', Object(Closure), 0)
#10 /srv/mediawiki/php-1.40.0-wmf.10/includes/page/PageStore.php(154): MediaWiki\Page\PageStore->getPageByNameViaLinkCache(0, 'Tangudla_Gutta', 0)
#11 /srv/mediawiki/php-1.40.0-wmf.10/includes/page/PageStore.php(326): MediaWiki\Page\PageStore->getPageByName(0, 'Tangudla_Gutta', 0)
#12 /srv/mediawiki/php-1.40.0-wmf.10/includes/title/Title.php(4108): MediaWiki\Page\PageStore->getPageByReference(Object(Title), 0)
#13 /srv/mediawiki/php-1.40.0-wmf.10/includes/title/Title.php(1100): Title->getFieldFromPageStore('page_content_mo...', 0)
#14 /srv/mediawiki/php-1.40.0-wmf.10/extensions/ActiveAbstract/includes/AbstractFilter.php(131): Title->getContentModel()
#15 /srv/mediawiki/php-1.40.0-wmf.10/includes/export/DumpFilter.php(81): MediaWiki\Extension\ActiveAbstract\AbstractFilter->writeClosePage('  </page>\n')
#16 /srv/mediawiki/php-1.40.0-wmf.10/includes/export/ExportProgressFilter.php(44): DumpFilter->writeClosePage('  </page>\n')
#17 /srv/mediawiki/php-1.40.0-wmf.10/includes/export/WikiExporter.php(613): ExportProgressFilter->writeClosePage('  </page>\n')
#18 /srv/mediawiki/php-1.40.0-wmf.10/includes/export/WikiExporter.php(501): WikiExporter->finishPageStreamOutput(Object(stdClass))
#19 /srv/mediawiki/php-1.40.0-wmf.10/includes/export/WikiExporter.php(315): WikiExporter->dumpPages('page_id >= 5020...', false)
#20 /srv/mediawiki/php-1.40.0-wmf.10/includes/export/WikiExporter.php(198): WikiExporter->dumpFrom('page_id >= 5020...', false)
#21 /srv/mediawiki/php-1.40.0-wmf.10/maintenance/includes/BackupDumper.php(359): WikiExporter->pagesByRange(5020001, 5030001, false)
#22 /srv/mediawiki/php-1.40.0-wmf.10/maintenance/dumpBackup.php(84): BackupDumper->dump(2, 0)
#23 /srv/mediawiki/php-1.40.0-wmf.10/maintenance/includes/MaintenanceRunner.php(309): DumpBackup->execute()
#24 /srv/mediawiki/php-1.40.0-wmf.10/maintenance/doMaintenance.php(85): MediaWiki\Maintenance\MaintenanceRunner->run()
#25 /srv/mediawiki/php-1.40.0-wmf.10/maintenance/dumpBackup.php(144): require_once('/srv/mediawiki/...')
#26 /srv/mediawiki/multiversion/MWScript.php(120): require_once('/srv/mediawiki/...')
#27 {main}

from cebwiki, also seen on a handful of others. Note that the abstracts jobs did eventually retry and complete.

Nov 23 2022, 7:52 AM · MW-1.40-notes (1.40.0-wmf.27; 2023-03-13), Performance-Team (Radar), Patch-For-Review, MediaWiki-libs-Rdbms, Wikimedia-production-error, DBA, Dumps-Generation
ArielGlenn added a comment to T315902: New error "DB is set and has not been closed by the Load Balancer" for certain bad revisions during page content dumps.

Just a note that we are still seeing some of these with the most recent run.

Nov 23 2022, 7:47 AM · Platform Engineering, Dumps-Generation

Nov 10 2022

ArielGlenn added a comment to T191491: Adjust bandwidth/connection limits, memory settings on labstore1006,7 as appropriate.

I don't know, since I haven't been involved with these or similar hosts since WMCS took them over! But the question is wheher the rsync bandwidth cap can be loosened up somewhat without a negative impact on existing services.

Nov 10 2022, 2:34 PM · Data-Services, cloud-services-team, Datasets-Archiving, User-ArielGlenn, Datasets-General-or-Unknown

Nov 9 2022

ArielGlenn added a comment to T322367: WikiExporter::dumpPages() batching/paging logic is wrong in edge cases.

In general we try to order by revid since that's known to be unique, and lets us do nice things like re-use old pge content dumps for creating the new ones, searching a previous dump output file that is also ordered by page id and within each page by rev id. So if we were going to make a change, I'd want to go that direction.

Nov 9 2022, 3:26 PM · MediaWiki-Core-Snapshots

Nov 8 2022

ArielGlenn updated subscribers of T191491: Adjust bandwidth/connection limits, memory settings on labstore1006,7 as appropriate.

Hey @nskaggs I figure this task has been around long enough it may have gotten lost in the shuffle. Of course the right hosts are now clouddumps1001,2. Any thoughts?

Nov 8 2022, 10:15 AM · Data-Services, cloud-services-team, Datasets-Archiving, User-ArielGlenn, Datasets-General-or-Unknown
ArielGlenn closed T263318: look into space issues on dumpsdata1001 and 1003 as Resolved.

We have new dumpsdata servers ready to roll out once the last of the raid controller changes (for autocreation of tasks when a disk dies) is done. In the meantime we keep few enough dumps and clean up at an appropriate time to not have this be an issue for awhile.

Nov 8 2022, 9:39 AM · Dumps-Generation
ArielGlenn moved T302145: Get @Hokwelum access to all repos, hosts, services needed as a dumps co-maintainer from Active to Blocked/Stalled/Waiting for event on the Dumps-Generation board.
Nov 8 2022, 9:37 AM · Dumps-Generation
ArielGlenn moved T318423: PHP Notice: Trying to access array offset on value of type int from Backlog to Blocked/Stalled/Waiting for event on the Dumps-Generation board.
Nov 8 2022, 9:36 AM · Language-Team (Language-2022-July-September), MW-1.40-notes (1.40.0-wmf.3; 2022-09-26), PHP 7.4 support, ContentTranslation, Dumps-Generation, Wikimedia-production-error
ArielGlenn moved T322360: conf* hosts ran out of disk space due to log spam from Backlog to Other teams on the Dumps-Generation board.
Nov 8 2022, 9:36 AM · MW-1.39-notes, MW-1.40-notes (1.40.0-wmf.10; 2022-11-14), Wikimedia-Incident, Dumps-Generation, serviceops, SRE
ArielGlenn closed T318206: SQL/XML dumps multiple errors "Couldn't update index json file, Continuing anyways" from 09 21 2022 near midnight as Resolved.

No pending actions left, so I'm closing this.

Nov 8 2022, 9:36 AM · Dumps-Generation
ArielGlenn moved T322156: "No server with index" and "Warning: Undefined index" from LoadBalancer::reconfigure from Backlog to Blocked/Stalled/Waiting for event on the Dumps-Generation board.
Nov 8 2022, 9:34 AM · MW-1.40-notes (1.40.0-wmf.27; 2023-03-13), Performance-Team (Radar), Patch-For-Review, MediaWiki-libs-Rdbms, Wikimedia-production-error, DBA, Dumps-Generation
ArielGlenn closed T322363: Data dumps for November aborted, a subtask of T322360: conf* hosts ran out of disk space due to log spam, as Resolved.
Nov 8 2022, 9:34 AM · MW-1.39-notes, MW-1.40-notes (1.40.0-wmf.10; 2022-11-14), Wikimedia-Incident, Dumps-Generation, serviceops, SRE
ArielGlenn closed T322363: Data dumps for November aborted as Resolved.

I'll go ahead and close this then, since the issue seems to have been resolved.

Nov 8 2022, 9:34 AM · Dumps-Generation

Nov 7 2022

ArielGlenn added a comment to T322156: "No server with index" and "Warning: Undefined index" from LoadBalancer::reconfigure.

No more of these errors since then. I'd like to get through the entire run before closing out this task, though.

Nov 7 2022, 10:01 AM · MW-1.40-notes (1.40.0-wmf.27; 2023-03-13), Performance-Team (Radar), Patch-For-Review, MediaWiki-libs-Rdbms, Wikimedia-production-error, DBA, Dumps-Generation

Nov 4 2022

ArielGlenn added a comment to T322360: conf* hosts ran out of disk space due to log spam.

After the patch https://gerrit.wikimedia.org/r/c/mediawiki/core/+/852990 was backported and deployed, dumps resumed running about an hour later (there is a script that checks to see if they are running and if not, restarts them twice a day during the run period). As of this morning the stubs jobs seem to be completing normally. etcd looks good on the graphs too, as Amir noted, see below:

Στιγμιότυπο από 2022-11-04 12-41-54.png (788×1 px, 60 KB)

Nov 4 2022, 10:43 AM · MW-1.39-notes, MW-1.40-notes (1.40.0-wmf.10; 2022-11-14), Wikimedia-Incident, Dumps-Generation, serviceops, SRE
ArielGlenn added a comment to T322363: Data dumps for November aborted.

Thank you for your report.

Nov 4 2022, 6:01 AM · Dumps-Generation

Nov 3 2022

ArielGlenn added a comment to T322360: conf* hosts ran out of disk space due to log spam.

There was a config setting that turned it on for November. See https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/848201

Nov 3 2022, 9:09 PM · MW-1.39-notes, MW-1.40-notes (1.40.0-wmf.10; 2022-11-14), Wikimedia-Incident, Dumps-Generation, serviceops, SRE
ArielGlenn updated subscribers of T322328: The service unit analytics-dumps-fetch-pageview_complete_dumps.service is in failed status on host clouddumps1002..

Andrew asked if I might know someone who knows something about this. I've never touched the kerb or hdfs stuff, but @elukey worked with the kerberos stuff in the modules/dumps/manifests/web/fetches/stats.pp manifest at one point, and @BTullis built a bunch of related packages in T310643 and so maybe knows something too.

Nov 3 2022, 2:15 PM · cloud-services-team (Kanban), Dumps-Generation

Nov 2 2022

ArielGlenn added a comment to T322156: "No server with index" and "Warning: Undefined index" from LoadBalancer::reconfigure.

A lot more wikis in the latest exception email: arzwiki, bnwiki, cebwiki, cewiki, cswiki, cywiki, dawiki, dewiki, dewiktionary, elwiki, enwikisource, eowiki, eswiktionary, etwiki, euwiki, fiwiki, frwikisource, glwiki, hiwiki, hrwiki, hywiki, idwiki, ltwiki, mediawikiwiki, nowiki, plwiktionary, ruwikinews, ruwikisource, ruwiktionary, simplewiki, skwiki, specieswiki, thwiki, trwiki, warwiki, zhwiktionary

Nov 2 2022, 4:42 PM · MW-1.40-notes (1.40.0-wmf.27; 2023-03-13), Performance-Team (Radar), Patch-For-Review, MediaWiki-libs-Rdbms, Wikimedia-production-error, DBA, Dumps-Generation
ArielGlenn added a comment to T265056: Make Cirrus Search dump script more resilient to failures (elasticsearch restarts).

Weekly report for the record, without the latest fix:

<13>Nov  1 20:17:11 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20221021/wikidatawiki-20221021-cirrussearch-general.json.gz
Nov 2 2022, 4:23 PM · MW-1.40-notes (1.40.0-wmf.19; 2023-01-16), Patch-For-Review, Discovery-Search (Current work), CirrusSearch, Dumps-Generation
ArielGlenn updated subscribers of T322221: The service unit dumps-fetch-phabdumps.service is in failed status on host clouddumps1002.

phab1001 was recently replaced, looping in @Dzahn for follow-up, as I think he knows about that migration.

Nov 2 2022, 3:02 PM · Phabricator, serviceops-collab, cloud-services-team (Kanban), Dumps-Generation
ArielGlenn added a comment to T322156: "No server with index" and "Warning: Undefined index" from LoadBalancer::reconfigure.

For now I would say, let's keep it from breaking things, so the stub dumps or other jobs can run to completion, and I'm interested to follow along on the discussion as to LoadMonitor rewrites.

Nov 2 2022, 12:28 PM · MW-1.40-notes (1.40.0-wmf.27; 2023-03-13), Performance-Team (Radar), Patch-For-Review, MediaWiki-libs-Rdbms, Wikimedia-production-error, DBA, Dumps-Generation
ArielGlenn updated the task description for T322156: "No server with index" and "Warning: Undefined index" from LoadBalancer::reconfigure.
Nov 2 2022, 8:51 AM · MW-1.40-notes (1.40.0-wmf.27; 2023-03-13), Performance-Team (Radar), Patch-For-Review, MediaWiki-libs-Rdbms, Wikimedia-production-error, DBA, Dumps-Generation
ArielGlenn added a comment to T265056: Make Cirrus Search dump script more resilient to failures (elasticsearch restarts).

Hey @EBernhardson, so Hannah and I talked a little about the issue of the long run time of the dumps, which we can expect to get longer over time. One issue we have now is that we used to try to schedule any maintenance for the weekends, when all the misc dumps (including CirrusSearch) were complete. There is no longer any such window; in fact, we now would have to choose between breaking two search dumps and breaking various wikibase entity dumps; neither of those is a good choice.

Nov 2 2022, 8:51 AM · MW-1.40-notes (1.40.0-wmf.19; 2023-01-16), Patch-For-Review, Discovery-Search (Current work), CirrusSearch, Dumps-Generation
ArielGlenn updated the task description for T322156: "No server with index" and "Warning: Undefined index" from LoadBalancer::reconfigure.
Nov 2 2022, 5:58 AM · MW-1.40-notes (1.40.0-wmf.27; 2023-03-13), Performance-Team (Radar), Patch-For-Review, MediaWiki-libs-Rdbms, Wikimedia-production-error, DBA, Dumps-Generation

Nov 1 2022

ArielGlenn triaged T322156: "No server with index" and "Warning: Undefined index" from LoadBalancer::reconfigure as High priority.
Nov 1 2022, 5:26 PM · MW-1.40-notes (1.40.0-wmf.27; 2023-03-13), Performance-Team (Radar), Patch-For-Review, MediaWiki-libs-Rdbms, Wikimedia-production-error, DBA, Dumps-Generation

Oct 27 2022

ArielGlenn added a comment to T313812: Deployment training request for mfossati.

I verify that mfossati did complete an earlier training, see T302204#7735291

Oct 27 2022, 12:57 PM · Release-Engineering-Team (Deployment Training Requests)

Oct 25 2022

ArielGlenn added a comment to T260223: Kiwix rsyncs not completing and stacking up on Clouddumps1001,2.

Hannah and I talked about the issue and here's what we've come up with.

Oct 25 2022, 12:57 PM · cloud-services-team, affects-Kiwix-and-openZIM, Dumps-Generation

Oct 24 2022

ArielGlenn added a comment to T320722: Deployment training request for Sergio Gimeno.

(ignore, left a wrong comment due to not being able to read my calendar)

Oct 24 2022, 12:08 PM · Release-Engineering-Team (Deployment Training Requests)

Oct 21 2022

ArielGlenn closed T319269: Missing Enterprise Dumps from 2022-10-01 run as Resolved.

Pre-emptively closing this again, I expect no further issues.

Oct 21 2022, 7:38 AM · Dumps-Generation
ArielGlenn added a comment to T319269: Missing Enterprise Dumps from 2022-10-01 run.
Oct 21 2022, 6:30 AM · Dumps-Generation

Oct 20 2022

ArielGlenn reopened T319269: Missing Enterprise Dumps from 2022-10-01 run as "Open".

Not entirely perfect, there's a missing temp dir that needs to go into the manifests for the clouddumps or any new hosts, patch coming...

Oct 20 2022, 11:26 AM · Dumps-Generation
ArielGlenn moved T278031: Wikibase canonical JSON format is missing "modified" in Wikidata JSON dumps from Other teams to Done on the Dumps-Generation board.
Oct 20 2022, 11:24 AM · MW-1.39-notes (1.39.0-wmf.14; 2022-05-30), Dumps-Generation, wdwb-tech, Wikidata, Wikibase (3rd party installations)
ArielGlenn moved T280631: Make Wikimedia Enterprise Daily Exports and Diffs available to WMCS community from Other teams to Done on the Dumps-Generation board.
Oct 20 2022, 11:23 AM · Wikimedia Enterprise (Okapi Wikimedia Enterprise), Dumps-Generation
ArielGlenn moved T302412: Request creation of wmf-dumps-playground VPS project from Backlog to Done on the Dumps-Generation board.
Oct 20 2022, 11:23 AM · User-dcaro, Dumps-Generation, Cloud-VPS (Project-requests)
ArielGlenn moved T303581: Include uawiki in dumps from Backlog to Done on the Dumps-Generation board.
Oct 20 2022, 11:23 AM · Dumps-Generation
ArielGlenn moved T304318: Flow dumps are broken on all wikis due to MediaWIki update from Backlog to Done on the Dumps-Generation board.
Oct 20 2022, 11:23 AM · Dumps-Generation
ArielGlenn moved T310075: Point Wikimedia Enterprise HTML Dumps to trial API features from Backlog to Done on the Dumps-Generation board.
Oct 20 2022, 11:23 AM · Wikimedia Enterprise, Dumps-Generation
ArielGlenn moved T315263: change enwiki database dump intervals (15th day of month instead of 20th) from Backlog to Done on the Dumps-Generation board.
Oct 20 2022, 11:22 AM · Dumps-Generation
ArielGlenn moved T307055: Remaining data engineering host security restarts from Other teams to Done on the Dumps-Generation board.
Oct 20 2022, 11:22 AM · Dumps-Generation, Infrastructure-Foundations (FY2021/2022-Q4), SRE, Security
ArielGlenn moved T309757: Public Phabricator dump includes restricted project columns from Other teams to Done on the Dumps-Generation board.
Oct 20 2022, 11:22 AM · Release-Engineering-Team (Doing), SecTeam-Processed, Vuln-Infoleak, Dumps-Generation, Phabricator, Security
ArielGlenn moved T311441: Missing Enterprise Dumps from 2022-06-20 run from Other teams to Done on the Dumps-Generation board.
Oct 20 2022, 11:21 AM · Dumps-Generation
ArielGlenn moved T315276: Latest English Wikipedia Wikimedia Enterprise HTML dumps do not seem to be updated from Other teams to Done on the Dumps-Generation board.
Oct 20 2022, 11:21 AM · Dumps-Generation, Wikimedia Enterprise Engineering, Wikimedia Enterprise
ArielGlenn moved T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium from Active to Done on the Dumps-Generation board.
Oct 20 2022, 11:21 AM · DBA, MW-1.38-notes (1.38.0-wmf.18; 2022-01-17), wdwb-tech, Wikidata, Dumps-Generation
ArielGlenn moved T271736: Migrate WMF production from PHP 7.2 to PHP 7.4 from Active to Done on the Dumps-Generation board.
Oct 20 2022, 11:21 AM · Dumps-Generation, Patch-For-Review, Performance-Team (Radar), serviceops
ArielGlenn moved T302930: Missing projects on Enterprise HTML dumps from Active to Done on the Dumps-Generation board.
Oct 20 2022, 11:21 AM · Dumps-Generation
ArielGlenn moved T297755: Move dumps generation to the new JWT based authentication process from Blocked/Stalled/Waiting for event to Done on the Dumps-Generation board.
Oct 20 2022, 11:20 AM · Dumps-Generation, Wikimedia Enterprise
ArielGlenn moved T318894: Remove php 7.2 from production from Backlog to Done on the Dumps-Generation board.
Oct 20 2022, 11:20 AM · Patch-For-Review, Dumps-Generation, Performance-Team (Radar), serviceops
ArielGlenn added a comment to T320722: Deployment training request for Sergio Gimeno.

We missed you at today's training. I guess something came up? Please go ahead and reschedule, and in case you cna't make it, let us know (if possible). See you soon!

Oct 20 2022, 7:50 AM · Release-Engineering-Team (Deployment Training Requests)

Oct 19 2022

ArielGlenn added a comment to T320722: Deployment training request for Sergio Gimeno.

Hey @Sgs don't forget to add your patch to the Deployment Calendar for the window Also note that we are now using scap backport url-of-gerrit-patch instead of all the steps you learned in the previous training. See you on the 20th!

Oct 19 2022, 10:03 PM · Release-Engineering-Team (Deployment Training Requests)

Oct 17 2022

ArielGlenn added a comment to T298485: MW scripts should reload the database config.

Change 828577 merged by jenkins-bot:

[mediawiki/core@master] rdbms: Instead of reconfiguring all of LB, just remove depooled db

https://gerrit.wikimedia.org/r/828577

Oct 17 2022, 10:59 AM · MW-1.40-notes (1.40.0-wmf.6; 2022-10-17), MW-1.39-notes (1.39.0-wmf.25; 2022-08-15), Patch-For-Review, Performance-Team (Radar), MediaWiki-Maintenance-system, User-Ladsgroup, DBA

Oct 12 2022

ArielGlenn added a comment to T260223: Kiwix rsyncs not completing and stacking up on Clouddumps1001,2.

I last saw this on September 17th. Other than new hardware, I don't know of any other changes. I'm hopeful this may not recur.

Oct 12 2022, 12:03 PM · cloud-services-team, affects-Kiwix-and-openZIM, Dumps-Generation

Oct 10 2022

ArielGlenn added a comment to T309346: Replace labstore100[67] with clouddumps100[12].

Given the new machines much larger capacity, I believe any pending requests for more space can now be reconsidered. @ArielGlenn do you know of any pending needs? Shall we discuss on a separate ticket any potential use cases for the extra space?

Oct 10 2022, 8:40 PM · Patch-For-Review, cloud-services-team (Kanban), Infrastructure-Foundations, SRE

Oct 5 2022

ArielGlenn added a comment to T309346: Replace labstore100[67] with clouddumps100[12].

Note that labstore1006 has some html dumps that didn't make it around to the other boxes, so please don't reimage until I get that sorted. Thanks!

Oct 5 2022, 5:01 PM · Patch-For-Review, cloud-services-team (Kanban), Infrastructure-Foundations, SRE

Oct 4 2022

ArielGlenn added a comment to T319269: Missing Enterprise Dumps from 2022-10-01 run.

The dumps are there on labstore1006. We'll make sure they get rsynced around tomorrow. This is probably a side effect of the transition to the new clouddumps hosts.

Oct 4 2022, 5:55 PM · Dumps-Generation

Sep 30 2022

ArielGlenn added a comment to T318423: PHP Notice: Trying to access array offset on value of type int.

Looks like this is fixed, after looking at logstash today once the same job finished on snapshot1008.

Sep 30 2022, 2:41 PM · Language-Team (Language-2022-July-September), MW-1.40-notes (1.40.0-wmf.3; 2022-09-26), PHP 7.4 support, ContentTranslation, Dumps-Generation, Wikimedia-production-error