Page MenuHomePhabricator

RfC: Create a proper command-line runner for MediaWiki maintenance tasks
Closed, ResolvedPublic

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

@daniel , we are adding this as a future initiative.

CCicalese_WMF triaged this task as Medium priority.

I'm an outreachy applicant. Can I take up this task? How should I get started with this?

@WDoranWMF @kchapman @daniel Would you all be interested in promoting/mentoring this project via Google Summer of Code 2020 or Outreachy Round 20?

@WDoranWMF @kchapman @daniel Would you all be interested in promoting/mentoring this project via Google Summer of Code 2020 or Outreachy Round 20?

Generally yes - which time periods are we talking about exactly?

Hi! This project seems interesting to me and I would like to contribute to it via GSoC'20. But before that I have some queries:

  • Is the command runner we are planning is similar to the way we use Git? Like we enter git on the terminal or command prompt and we see all the commands that can be run with one line description about each command.
  • Can you please link the source of maintenance scripts we are planning to include?!

Thanks in advance!

@Soumyaa1804 Thanks for your interest! FYI, we are not planning to promote this project via GSoC and might end up promoting via Outreachy. We will finalize in the next few days.

Okay! I would be interested to work on this even in Outreachy if it gets promoted there and my initial application gets selected. :)

srishakatux changed the visibility from "Public (No Login Required)" to "acl*outreachy-mentors (Project)".
srishakatux moved this task from Backlog to Featured projects on the Outreachy (Round 20) board.

(as per the program rules need to restrict access to this project task to #Outreachy-mentors group until the application period begins)

srishakatux changed the visibility from "acl*outreachy-mentors (Project)" to "Public (No Login Required)".Mar 5 2020, 6:16 PM

Hi! @daniel I would like to contribute, can you please assign me some task to get started with?

@Aklapper Yes, that's a good point; I've added the link from the Outreachy/Participants page.

srishakatux removed a project: Outreachy (Round 20).

We are unlisting this project from Outreachy (Round 20). If you a potential intern, please explore other projects here https://www.mediawiki.org/wiki/Outreachy/Round_20#Ideas_for_projects.

Change 693134 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/core@master] WIP: Symfony Console based CLI for maintenance scripts

https://gerrit.wikimedia.org/r/693134

kostajh added a subscriber: kostajh.

I'll be doing some more exploration around the patch above during the hackathon, if anyone's interested to code / critique / test, help is welcome :)

I noticed today that one of the first commits in the Maintenance system actually had a class-based registry, and while we lost that at some point, it connected the dots for me to something we still do today: The maintenance scripts are in the autoloader and each script has logic to avoid execution if loaded outside their own entry point. This was I believe designed to be run via a wrapper one day.

From the original commit:

$wgMaintenanceScripts = array();

Register their maintenance scripts [class name] with the system

$wgMaintenanceScripts for extensions to add their scripts to the default list.

public static function getMaintenanceScripts() {
      global $wgMaintenanceScripts;
      return $wgMaintenanceScripts + self::getCoreScripts();
  }

http://mediawiki.org/wiki/Special:Code/MediaWiki/54225
https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+/a1c51e18af85a9ac464c5b555921e58ec422cd11%5E%21/

If someone is willing to make small patches doing this step-by-step, I'd be happy to review it.

Last time I tried, I got lost in the way this is doing setup and loading config. Perhaps with SettingsBuilder, it can now be done in a saner way...

@Ladsgroup, @Krinkle, @kostajh: I have an experimantal chain of patches up, starting at https://gerrit.wikimedia.org/r/c/mediawiki/core/+/782569. It still needs some polishing, testing, and bikeshedding about names, but it seems to work. I'd love to hear your thoughts on the approach I'm takeing there, before I invest more time.

It sounds like an awesome hackathon project!

Change 818574 had a related patch set uploaded (by Daniel Kinzler; author: Daniel Kinzler):

[mediawiki/core@master] ExtensionRegistry: split exportExtractedData

https://gerrit.wikimedia.org/r/818574

Change 783428 had a related patch set uploaded (by Daniel Kinzler; author: Daniel Kinzler):

[mediawiki/core@master] Introduce runner entry point

https://gerrit.wikimedia.org/r/783428

I noticed today that one of the first commits in the Maintenance system actually had a class-based registry, and while we lost that at some point, it connected the dots for me to something we still do today: The maintenance scripts are in the autoloader and each script has logic to avoid execution if loaded outside their own entry point. This was I believe designed to be run via a wrapper one day.

From the original commit:

$wgMaintenanceScripts = array();

Register their maintenance scripts [class name] with the system

$wgMaintenanceScripts for extensions to add their scripts to the default list.

public static function getMaintenanceScripts() {
      global $wgMaintenanceScripts;
      return $wgMaintenanceScripts + self::getCoreScripts();
  }

http://mediawiki.org/wiki/Special:Code/MediaWiki/54225
https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+/a1c51e18af85a9ac464c5b555921e58ec422cd11%5E%21/

That was indeed the original idea over a decade ago :p

I think this is mostly done, there is one last push needed to get the last piece over the line.

Change 818574 abandoned by Daniel Kinzler:

[mediawiki/core@master] ExtensionRegistry: split exportExtractedData

Reason:

Going with I6f8f9f3f7252f0024282d7b005671f28a5b3acc3 instead

https://gerrit.wikimedia.org/r/818574

Change 783428 merged by jenkins-bot:

[mediawiki/core@master] Introduce run.php for running maintenance scripts

https://gerrit.wikimedia.org/r/783428

Ladsgroup assigned this task to daniel.
amir@amir-ThinkPad-P1-Gen-3:/var/lib/mediawiki$ php maintenance/run.php extensions/AbuseFilter/maintenance/UpdateVarDumps.php --force
...the abuse_filter_log table is empty.

amir@amir-ThinkPad-P1-Gen-3:/var/lib/mediawiki$ php maintenance/run.php MediaWiki.Extension.AbuseFilter.Maintenance.UpdateVarDumps --force
...the abuse_filter_log table is empty.
Tgr added a subscriber: Tgr.

This is great, but the new warning introduced in rMW4912fe27a8e0: Maintenance scripts: show a warning when run directly. shown for old-style scripts is likely to mess up any automation and should be reverted IMO (or maybe it should try to detect whether it is writing to an interactive terminal).
Vagrant for example breaks quite spectacularly: T326522: Vagrant mediawiki::maintenance resource broken by new maintenance script output

In T99268#8507896, @Tgr wrote:

This is great, but the new warning shown for old-style scripts is likely to mess up any automation and should be reverted IMO (or maybe it should try to detect whether it is writing to an interactive terminal).
Vagrant for example breaks quite spectacularly: T326522: Vagrant mediawiki::maintenance resource broken by new maintenance script output

Would https://gerrit.wikimedia.org/r/c/mediawiki/core/+/876395 help?

Probably not, stderr can also cause issues, sometimes it's taken as a sign that the script failed, sometimes it's piped together with normal output etc.
It might be better than stdout, I'm not sure, but either way warnings should be limited to interactive mode. Or use the deprecation warning system, etc.

Change 876792 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/core@master] Do not add random text to maint script output when not in TTY mode

https://gerrit.wikimedia.org/r/876792

Change 876792 merged by jenkins-bot:

[mediawiki/core@master] maintenance: Do not output run.php warning when not in TTY mode

https://gerrit.wikimedia.org/r/876792

Change 693134 abandoned by Kosta Harlan:

[mediawiki/core@master] WIP: Symfony Console based CLI for maintenance scripts

Reason:

https://gerrit.wikimedia.org/r/693134