Page MenuHomePhabricator

Run the maintenance script linter extension migrateNamespace.php on all wikis
Closed, ResolvedPublic

Description

The linter table has been enhanced with a new linter_namespace column which will allow the linter reports to avoid making queries against the page table, greatly reducing the load on the database. For several weeks new linter errors have been writing the namespace ID info into this table, but older linter records have this field set to NULL.

This maintenance script updates the new column with the namespace ID from the page table using batching and waiting on replication to avoid causing performance problems while updating the linter table.

This script has gone through extensive review and full testing on Beta cluster and writes log entries as it executes to allow monitoring.

Event Timeline

Umherirrender renamed this task from Run the maintenance script linter extension migrateNamspace.php on all wikis to Run the maintenance script linter extension migrateNamespace.php on all wikis.Feb 15 2023, 8:14 PM

I ran it on testwiki:

ladsgroup@mwmaint1002:/srv/mediawiki/php-1.40.0-wmf.22$ time mwscript extensions/Linter/maintenance/migrateNamespace.php --wiki=testwiki --sleep 1

*******************************************************************************
NOTE: Do not run maintenance scripts directly, use maintenance/run.php instead!
      Running scripts directly has been deprecated in MediaWiki 1.40.
      It may not work for some (or any) scripts in the future.
*******************************************************************************

Running linter migrate namespace function, this may take a while
Migrating the page table page_namespace field to the linter table...
Completed migration of page_namespace data to the linter table, 7086 rows updated.

real	0m16.234s
user	0m0.853s
sys	0m0.158s

No major issue, replication didn't break not any large replag was built up (checked through https://grafana.wikimedia.org/d/000000278/mysql-aggregated?orgId=1&from=now-1h&to=now&var-site=eqiad&var-group=core&var-shard=s3&var-role=All)

One suggestion, the script should output something after each batch to show progress, it's a bit scary you start something and it's not showing anything.

Mentioned in SAL (#wikimedia-operations) [2023-02-15T23:01:55Z] <Amir1> running linter migrate namespace on all wikis (T329764)

There is logging in both scripts:

		$logger = LoggerFactory::getInstance( 'MigrateNamespaceChannel' );

. . .

			$logger->info( 'Migrated ' . $updated . " page IDs\n" );

		} while ( $linterPagesLength > 0 );

and

		$logger = LoggerFactory::getInstance( 'MigrateTagAndTemplateChannel' );

. . .

			$logger->info( 'Migrated ' . $updated . " linter IDs\n" );

		} while ( $linterBatchLength > 0 );

so those should be visible to the logging dashboard on a per batch basis

The logger is not that useful, first, if the channel is not explicitly added, the whole thing goes to blackhole (which this thing does, it's better to simply use 'Linter' as the channel here). Second, info-level logs might also go to blackhole so it probably needs something like warning which doesn't make much sense. And last, I would want it on the screen I'm running it on to get feedback on the action I'm running.

Ah sorry, the logger interface was recommended. I wish I had known about its quirks and limitations. Live and learn. Next time I will just output to console.