Hey @Andrew I'm guessing this patch is related: https://gerrit.wikimedia.org/r/c/operations/puppet/+/902833

Hey @awight I see what you're getting at, and while that would be a cool feature, it's not in the cards for the current dumps. If they get a revamp at some point, that would be a nice thing to have, definitely. In the meantime I'm glad you have a workaround. One thing that would be nice in our glorious future would be some uniformity as to how various datasets for public download are published, but that's not coming any time soon...

Summarizing from a conversation on IRC:

I'm not sure I have a clear picture of what you want here, @awight Do you want a link to the directory once/if the last dump is downloaded successfully into it? So e.g. right now there could be link 'latest' at the same level as the date directories, pointing to 20230320 (asuming that's complete); is this what you are looking for, or something slightly different?

Hmm we already write to a temp file with the file name ending in ".tmp". So I wonder why the size is changing. The file is written into the same directory and moved with a rename so there should be no time period when the final file is growing.

We could download WME dumps to a name that ends with ".inprog" and move them once the download is complete; this is how generation of our sql/xml dumps works, so it would fit right in.

The swap has been done, verified via testing on snapshot1009 (testbed) for xml/sql dumps. Did not test adds-changes dumps but did sort out the ports for lockd so it should be ok. Wikitech page partially updated, will put the specs of the new hosts in later, ran out of oomph.

rsync to dumpsdata1005 is about halfway done copying the old data from dumpsdata1001. we'll need to start updating it (with care) from the fresher data on dumpsdata1004, so that it's ready to swap in once the xml dumps run completes at around the 14th or 15th of the month.

We've discussed this a little here and randomising the order of retrieval is probably not the approach we would take, for performance reasons. But there are other possibilities. We would need to assess the risk and possible harm and weigh that against the work needed and resources available. For now we do have urgent scaling work and server replacements that need to take precedence.

This graph shows the state of things over the past month on dumpsdata1003. We'll need to manually do cleanup of bz2 files from the previous full run for wikidatawiki as the new run progresses, in order to avoid space issues, and then swap in a new dumpsdata host once the run completes.

rsync to dumpsdata1005 has started, from dumpsdata1001, so we'll not have completely fresh data but we'll have the bulk of it. After that we'll do the same for dumpsdata1006 and 7,

dumpsdata1006 /data partition resized, now all 4 new boxes have the correct size filesystem on /data.

Besides watching today to make sure nothing is amiss, we should next:

Wonderful, we have claimed them already :-) Thank you!

Awesome, I would have looked for you on irc in a few days if I hadn't heard anything, no worries. Happy to see this moving along!

Full speed rsync of xml dumps from dumpsdata1003 to dumpsdata1004 is underway, currently on commonswiki.

Note that dumpsdata1001 and 1002 are intended to be replaced and then decommed, so ideally we would swap in dumpsdata1004,5, (6 or 7) for all of 1001,2,3 increasing the size of each and freeing up those two boxes. We can then shuffle things around later (i.e. 1003 to be the 'other dumps' nfs filesystem, freeing up a larger box for use with xml dumps one way or another).

Hey @RobH I am preemptively assuming that dumpsdata1007 is good to go for us to use, since it's got the role ("insetup::core_platform") and everything in the checklist is marked off. Basically we need these boxes, at least the one, tell me if it's not ready to be ours yet and I'll stop mucking about with it.

Guess it's in the php innards then, somewhere in xmlreader and friends...

@Reedy What version of ibxml2 is your PHP 8.2.16 using, compared to 8.1.16? Maybe something changed regarding entity loading between the two respective versions?

Fri-Sat-Sun: 16.15 run, error reported from clouddumps1002

@ ERROR: max connections (6) reached -- try again later

All other runs good.

In T322156#8588664, @Marostegui wrote:

is this still an issue?

Just noting that we haven't seen this error for the last two weeks so it looks like the issue is resolved for us (Content Translation dumps) as well.

Some options for the longer term might be:

• keep track of uncompressed data sizes and write these numbers out along with the compressed bz2 files (would take a chunk f messing about with XMLWriter innards but would be very fast to generate stats afterwards)
• see about the revision content in the data lake and how costly it would be to just sum the lengths up each month for enwiki plus $randomwiki
• give an uppper bound using the size of the uncompressed data in the adds-changes dump each day (this would require choosing the random wiki at the beginning of the month and keeping a running total each day but it would be pretty fast, if less accurate)

Hey WMCS folks, see the above patch for the change of upstream hostname for kiwix downloads, I figure these are your boxes so you ought to give the thumbs up/merge. @Andrew you have been added as a reviewer but feel free to swap anyone else in as you see fit.

In T226093#8512308, @LSobanski wrote:

The task's original intent was to cover planning "over the next 3 years" starting in 2019. @ArielGlenn is the task still relevant, can it be closed, do we need a new one?

The weekly content translation dumps were also impacted by this, sample stack trace:

[e2d5d9d71060942e3def717f] [no req]   Error: Cannot use object of type ContentTranslation\DTO\TranslationUnitDTO as array
Backtrace:
from /srv/mediawiki/php-1.40.0-wmf.17/extensions/ContentTranslation/includes/CorporaLookup.php(87)
#0 /srv/mediawiki/php-1.40.0-wmf.17/extensions/ContentTranslation/includes/CorporaLookup.php(51): ContentTranslation\CorporaLookup::format(Wikimedia\Rdbms\MysqliResultWrapper)
#1 /srv/mediawiki/php-1.40.0-wmf.17/extensions/ContentTranslation/scripts/dump-corpora.php(140): ContentTranslation\CorporaLookup->getByTranslationId(integer)
#2 /srv/mediawiki/php-1.40.0-wmf.17/maintenance/includes/MaintenanceRunner.php(496): CXCorporaDump->execute()
#3 /srv/mediawiki/php-1.40.0-wmf.17/maintenance/doMaintenance.php(85): MediaWiki\Maintenance\MaintenanceRunner->run()
#4 /srv/mediawiki/php-1.40.0-wmf.17/extensions/ContentTranslation/scripts/dump-corpora.php(316): require_once(string)
#5 /srv/mediawiki/multiversion/MWScript.php(120): require_once(string)
#6 {main}

Today's report:

<13>Jan  9 19:14:29 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20230109/wikidatawiki-20230109-cirrussearch-general.json.gz

Adding this here, https://www.eff.org/deeplinks/2022/12/user-generated-content-and-fediverse-legal-primer for those of us not on the legal team and who may not be aware of the legal issues involved in hosting one's own instance of Mastodon. Some thought would need to go into managing trust and safety issues as well, in particular if the instance is open to use by community members and not just WMF employees.

See also T222985

The reason we go with bz2 is that it is block-oriented, allowing easier parallel processing of input files for platforms that rely on massive parallelization such as Hadoop. It also allows us to recover if part of a file is written out and the dump process aborts for some reason. It's possble that other compression algorithms could be adopted in a future re-architected version of the dumps.

Yesterday's report:

<13>Nov 30 15:39:42 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20221123/frwiki-20221123-cirrussearch-general.json.gz

Note that because there's no explicit error or exception in the logs, this is just harmless noise, but it would be nice to understand where it comes from and decide what to do about it at some point (hence, low priority).

Last Wednesday's report:

<13>Nov 18 12:16:54 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20221115/cowiki-20221115-cirrussearch-content.json.gz
<13>Nov 23 12:57:44 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20221115/wikidatawiki-20221115-cirrussearch-general.json.gz

We saw some new errors of this sort during the abstracts dump job. Sample stacktrace:

[20221121172444]: InvalidArgumentException from line 2319 of /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/rdbms/loadbalancer/LoadBalancer.php: No server with index '4'
#0 /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1170): Wikimedia\Rdbms\LoadBalancer->getServerInfoStrict(4)
#1 /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1102): Wikimedia\Rdbms\LoadBalancer->reallyOpenConnection(4, Object(Wikimedia\Rdbms\DatabaseDomain), Array)
#2 /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/rdbms/loadbalancer/LoadBalancer.php(945): Wikimedia\Rdbms\LoadBalancer->reuseOrOpenConnectionForNewRef(4, Object(Wikimedia\Rdbms\DatabaseDomain), 3)
#3 /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/rdbms/loadmonitor/LoadMonitor.php(237): Wikimedia\Rdbms\LoadBalancer->getServerConnection(4, '', 3)
#4 /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/rdbms/loadmonitor/LoadMonitor.php(172): Wikimedia\Rdbms\LoadMonitor->computeServerStates(Array, Array)
#5 /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/objectcache/wancache/WANObjectCache.php(1755): Wikimedia\Rdbms\LoadMonitor->Wikimedia\Rdbms\{closure}(Array, 1, Array, 1669051415.1476, Array)
#6 /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/objectcache/wancache/WANObjectCache.php(1585): WANObjectCache->fetchOrRegenerate('global:rdbms-se...', 1, Object(Closure), Array, Array)
#7 /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/rdbms/loadmonitor/LoadMonitor.php(181): WANObjectCache->getWithSetCallback('global:rdbms-se...', 1, Object(Closure), Array)
#8 /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/rdbms/loadmonitor/LoadMonitor.php(104): Wikimedia\Rdbms\LoadMonitor->getServerStates(Array)
#9 /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/rdbms/loadbalancer/LoadBalancer.php(547): Wikimedia\Rdbms\LoadMonitor->scaleLoads(Array)
#10 /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/rdbms/loadbalancer/LoadBalancer.php(507): Wikimedia\Rdbms\LoadBalancer->getReaderIndex('dump')
#11 /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/rdbms/loadbalancer/LoadBalancer.php(920): Wikimedia\Rdbms\LoadBalancer->getConnectionIndex(-1, Array, 'cebwiki')
#12 /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/rdbms/database/DBConnRef.php(103): Wikimedia\Rdbms\LoadBalancer->getConnectionInternal(-1, Array, 'cebwiki', 0)
#13 /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/rdbms/database/DBConnRef.php(117): Wikimedia\Rdbms\DBConnRef->ensureConnection()
#14 /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/rdbms/database/DBConnRef.php(356): Wikimedia\Rdbms\DBConnRef->__call('selectRow', Array)
#15 /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/rdbms/querybuilder/SelectQueryBuilder.php(689): Wikimedia\Rdbms\DBConnRef->selectRow(Array, Array, Array, 'MediaWiki\\Page\\...', Array, Array)
#16 /srv/mediawiki/php-1.40.0-wmf.10/includes/page/PageStore.php(199): Wikimedia\Rdbms\SelectQueryBuilder->fetchRow()
#17 /srv/mediawiki/php-1.40.0-wmf.10/includes/cache/LinkCache.php(461): MediaWiki\Page\PageStore->MediaWiki\Page\{closure}(Object(Wikimedia\Rdbms\DBConnRef), 0, 'Oil_Manikin', Array)
#18 /srv/mediawiki/php-1.40.0-wmf.10/includes/cache/LinkCache.php(494): LinkCache->getGoodLinkRowInternal(Object(TitleValue), Object(Closure), 0)
#19 /srv/mediawiki/php-1.40.0-wmf.10/includes/page/PageStore.php(188): LinkCache->getGoodLinkRow(0, 'Oil_Manikin', Object(Closure), 0)
#20 /srv/mediawiki/php-1.40.0-wmf.10/includes/page/PageStore.php(154): MediaWiki\Page\PageStore->getPageByNameViaLinkCache(0, 'Oil_Manikin', 0)
#21 /srv/mediawiki/php-1.40.0-wmf.10/includes/page/PageStore.php(326): MediaWiki\Page\PageStore->getPageByName(0, 'Oil_Manikin', 0)
#22 /srv/mediawiki/php-1.40.0-wmf.10/includes/title/Title.php(4108): MediaWiki\Page\PageStore->getPageByReference(Object(Title), 0)
#23 /srv/mediawiki/php-1.40.0-wmf.10/includes/title/Title.php(1100): Title->getFieldFromPageStore('page_content_mo...', 0)
#24 /srv/mediawiki/php-1.40.0-wmf.10/extensions/ActiveAbstract/includes/AbstractFilter.php(131): Title->getContentModel()
#25 /srv/mediawiki/php-1.40.0-wmf.10/includes/export/DumpFilter.php(81): MediaWiki\Extension\ActiveAbstract\AbstractFilter->writeClosePage('  </page>\n')
#26 /srv/mediawiki/php-1.40.0-wmf.10/includes/export/ExportProgressFilter.php(44): DumpFilter->writeClosePage('  </page>\n')
#27 /srv/mediawiki/php-1.40.0-wmf.10/includes/export/WikiExporter.php(613): ExportProgressFilter->writeClosePage('  </page>\n')
#28 /srv/mediawiki/php-1.40.0-wmf.10/includes/export/WikiExporter.php(501): WikiExporter->finishPageStreamOutput(Object(stdClass))
#29 /srv/mediawiki/php-1.40.0-wmf.10/includes/export/WikiExporter.php(315): WikiExporter->dumpPages('page_id >= 3220...', false)
#30 /srv/mediawiki/php-1.40.0-wmf.10/includes/export/WikiExporter.php(198): WikiExporter->dumpFrom('page_id >= 3220...', false)
#31 /srv/mediawiki/php-1.40.0-wmf.10/maintenance/includes/BackupDumper.php(359): WikiExporter->pagesByRange(3220001, 3230001, false)
#32 /srv/mediawiki/php-1.40.0-wmf.10/maintenance/dumpBackup.php(84): BackupDumper->dump(2, 0)
#33 /srv/mediawiki/php-1.40.0-wmf.10/maintenance/includes/MaintenanceRunner.php(309): DumpBackup->execute()
#34 /srv/mediawiki/php-1.40.0-wmf.10/maintenance/doMaintenance.php(85): MediaWiki\Maintenance\MaintenanceRunner->run()
#35 /srv/mediawiki/php-1.40.0-wmf.10/maintenance/dumpBackup.php(144): require_once('/srv/mediawiki/...')
#36 /srv/mediawiki/multiversion/MWScript.php(120): require_once('/srv/mediawiki/...')
#37 {main}
Wikimedia\Rdbms\DBConnectionError from line 1341 of /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/rdbms/loadbalancer/LoadBalancer.php: Cannot access the database: could not connect to any replica DB server
#0 /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/rdbms/loadbalancer/LoadBalancer.php(514): Wikimedia\Rdbms\LoadBalancer->reportConnectionError('could not conne...')
#1 /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/rdbms/loadbalancer/LoadBalancer.php(920): Wikimedia\Rdbms\LoadBalancer->getConnectionIndex(-1, Array, 'cebwiki')
#2 /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/rdbms/database/DBConnRef.php(103): Wikimedia\Rdbms\LoadBalancer->getConnectionInternal(-1, Array, 'cebwiki', 0)
#3 /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/rdbms/database/DBConnRef.php(117): Wikimedia\Rdbms\DBConnRef->ensureConnection()
#4 /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/rdbms/database/DBConnRef.php(356): Wikimedia\Rdbms\DBConnRef->__call('selectRow', Array)
#5 /srv/mediawiki/php-1.40.0-wmf.10/includes/libs/rdbms/querybuilder/SelectQueryBuilder.php(689): Wikimedia\Rdbms\DBConnRef->selectRow(Array, Array, Array, 'MediaWiki\\Page\\...', Array, Array)
#6 /srv/mediawiki/php-1.40.0-wmf.10/includes/page/PageStore.php(199): Wikimedia\Rdbms\SelectQueryBuilder->fetchRow()
#7 /srv/mediawiki/php-1.40.0-wmf.10/includes/cache/LinkCache.php(461): MediaWiki\Page\PageStore->MediaWiki\Page\{closure}(Object(Wikimedia\Rdbms\DBConnRef), 0, 'Tangudla_Gutta', Array)
#8 /srv/mediawiki/php-1.40.0-wmf.10/includes/cache/LinkCache.php(494): LinkCache->getGoodLinkRowInternal(Object(TitleValue), Object(Closure), 0)
#9 /srv/mediawiki/php-1.40.0-wmf.10/includes/page/PageStore.php(188): LinkCache->getGoodLinkRow(0, 'Tangudla_Gutta', Object(Closure), 0)
#10 /srv/mediawiki/php-1.40.0-wmf.10/includes/page/PageStore.php(154): MediaWiki\Page\PageStore->getPageByNameViaLinkCache(0, 'Tangudla_Gutta', 0)
#11 /srv/mediawiki/php-1.40.0-wmf.10/includes/page/PageStore.php(326): MediaWiki\Page\PageStore->getPageByName(0, 'Tangudla_Gutta', 0)
#12 /srv/mediawiki/php-1.40.0-wmf.10/includes/title/Title.php(4108): MediaWiki\Page\PageStore->getPageByReference(Object(Title), 0)
#13 /srv/mediawiki/php-1.40.0-wmf.10/includes/title/Title.php(1100): Title->getFieldFromPageStore('page_content_mo...', 0)
#14 /srv/mediawiki/php-1.40.0-wmf.10/extensions/ActiveAbstract/includes/AbstractFilter.php(131): Title->getContentModel()
#15 /srv/mediawiki/php-1.40.0-wmf.10/includes/export/DumpFilter.php(81): MediaWiki\Extension\ActiveAbstract\AbstractFilter->writeClosePage('  </page>\n')
#16 /srv/mediawiki/php-1.40.0-wmf.10/includes/export/ExportProgressFilter.php(44): DumpFilter->writeClosePage('  </page>\n')
#17 /srv/mediawiki/php-1.40.0-wmf.10/includes/export/WikiExporter.php(613): ExportProgressFilter->writeClosePage('  </page>\n')
#18 /srv/mediawiki/php-1.40.0-wmf.10/includes/export/WikiExporter.php(501): WikiExporter->finishPageStreamOutput(Object(stdClass))
#19 /srv/mediawiki/php-1.40.0-wmf.10/includes/export/WikiExporter.php(315): WikiExporter->dumpPages('page_id >= 5020...', false)
#20 /srv/mediawiki/php-1.40.0-wmf.10/includes/export/WikiExporter.php(198): WikiExporter->dumpFrom('page_id >= 5020...', false)
#21 /srv/mediawiki/php-1.40.0-wmf.10/maintenance/includes/BackupDumper.php(359): WikiExporter->pagesByRange(5020001, 5030001, false)
#22 /srv/mediawiki/php-1.40.0-wmf.10/maintenance/dumpBackup.php(84): BackupDumper->dump(2, 0)
#23 /srv/mediawiki/php-1.40.0-wmf.10/maintenance/includes/MaintenanceRunner.php(309): DumpBackup->execute()
#24 /srv/mediawiki/php-1.40.0-wmf.10/maintenance/doMaintenance.php(85): MediaWiki\Maintenance\MaintenanceRunner->run()
#25 /srv/mediawiki/php-1.40.0-wmf.10/maintenance/dumpBackup.php(144): require_once('/srv/mediawiki/...')
#26 /srv/mediawiki/multiversion/MWScript.php(120): require_once('/srv/mediawiki/...')
#27 {main}

from cebwiki, also seen on a handful of others. Note that the abstracts jobs did eventually retry and complete.

Just a note that we are still seeing some of these with the most recent run.

I don't know, since I haven't been involved with these or similar hosts since WMCS took them over! But the question is wheher the rsync bandwidth cap can be loosened up somewhat without a negative impact on existing services.

In general we try to order by revid since that's known to be unique, and lets us do nice things like re-use old pge content dumps for creating the new ones, searching a previous dump output file that is also ordered by page id and within each page by rev id. So if we were going to make a change, I'd want to go that direction.

Hey @nskaggs I figure this task has been around long enough it may have gotten lost in the shuffle. Of course the right hosts are now clouddumps1001,2. Any thoughts?

We have new dumpsdata servers ready to roll out once the last of the raid controller changes (for autocreation of tasks when a disk dies) is done. In the meantime we keep few enough dumps and clean up at an appropriate time to not have this be an issue for awhile.

No pending actions left, so I'm closing this.

I'll go ahead and close this then, since the issue seems to have been resolved.

No more of these errors since then. I'd like to get through the entire run before closing out this task, though.

After the patch https://gerrit.wikimedia.org/r/c/mediawiki/core/+/852990 was backported and deployed, dumps resumed running about an hour later (there is a script that checks to see if they are running and if not, restarts them twice a day during the run period). As of this morning the stubs jobs seem to be completing normally. etcd looks good on the graphs too, as Amir noted, see below:

Thank you for your report.

There was a config setting that turned it on for November. See https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/848201

Andrew asked if I might know someone who knows something about this. I've never touched the kerb or hdfs stuff, but @elukey worked with the kerberos stuff in the modules/dumps/manifests/web/fetches/stats.pp manifest at one point, and @BTullis built a bunch of related packages in T310643 and so maybe knows something too.

A lot more wikis in the latest exception email: arzwiki, bnwiki, cebwiki, cewiki, cswiki, cywiki, dawiki, dewiki, dewiktionary, elwiki, enwikisource, eowiki, eswiktionary, etwiki, euwiki, fiwiki, frwikisource, glwiki, hiwiki, hrwiki, hywiki, idwiki, ltwiki, mediawikiwiki, nowiki, plwiktionary, ruwikinews, ruwikisource, ruwiktionary, simplewiki, skwiki, specieswiki, thwiki, trwiki, warwiki, zhwiktionary

Weekly report for the record, without the latest fix:

<13>Nov  1 20:17:11 dumpsgen: extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20221021/wikidatawiki-20221021-cirrussearch-general.json.gz

phab1001 was recently replaced, looping in @Dzahn for follow-up, as I think he knows about that migration.

For now I would say, let's keep it from breaking things, so the stub dumps or other jobs can run to completion, and I'm interested to follow along on the discussion as to LoadMonitor rewrites.

Hey @EBernhardson, so Hannah and I talked a little about the issue of the long run time of the dumps, which we can expect to get longer over time. One issue we have now is that we used to try to schedule any maintenance for the weekends, when all the misc dumps (including CirrusSearch) were complete. There is no longer any such window; in fact, we now would have to choose between breaking two search dumps and breaking various wikibase entity dumps; neither of those is a good choice.

I verify that mfossati did complete an earlier training, see T302204#7735291

Hannah and I talked about the issue and here's what we've come up with.

(ignore, left a wrong comment due to not being able to read my calendar)

Pre-emptively closing this again, I expect no further issues.

In T319269#8332506, @Andrew wrote:

does that explain https://phabricator.wikimedia.org/T321280 ?

Not entirely perfect, there's a missing temp dir that needs to go into the manifests for the clouddumps or any new hosts, patch coming...

We missed you at today's training. I guess something came up? Please go ahead and reschedule, and in case you cna't make it, let us know (if possible). See you soon!

Hey @Sgs don't forget to add your patch to the Deployment Calendar for the window Also note that we are now using scap backport url-of-gerrit-patch instead of all the steps you learned in the previous training. See you on the 20th!

In T298485#8320648, @gerritbot wrote:

Change 828577 merged by jenkins-bot:

[mediawiki/core@master] rdbms: Instead of reconfiguring all of LB, just remove depooled db

https://gerrit.wikimedia.org/r/828577

In T260223#8300609, @nskaggs wrote:

I last saw this on September 17th. Other than new hardware, I don't know of any other changes. I'm hopeful this may not recur.

In T309346#8300585, @nskaggs wrote:

Given the new machines much larger capacity, I believe any pending requests for more space can now be reconsidered. @ArielGlenn do you know of any pending needs? Shall we discuss on a separate ticket any potential use cases for the extra space?

Note that labstore1006 has some html dumps that didn't make it around to the other boxes, so please don't reimage until I get that sorted. Thanks!

The dumps are there on labstore1006. We'll make sure they get rsynced around tomorrow. This is probably a side effect of the transition to the new clouddumps hosts.

Looks like this is fixed, after looking at logstash today once the same job finished on snapshot1008.