I've merged T57689: Implement -save option on replace.py core here because this is a more general task (not just -save).

• -save was implemented in https://phabricator.wikimedia.org/rSVNP8700
• -saveexc was implemented in https://phabricator.wikimedia.org/rSVNP9918

• there is a draft patch for -save in core at https://gerrit.wikimedia.org/r/220771, but there are some issues with it.

Questions:

• How much of this workflow can be supported by using listpages.py -grep ?
• Do we need both -save and -savenew, or can we just output the list to stdout, and let the user redirect using > or >>?
• Same for saveexc.
• -save does not seem to save page titles if -always is passed. Is that on purpose?

I don't know listpages as I always used compat.

-save/savenew (with flush() after every save) works reliably even for very slow, long search tasks. (I mean up to several days!) Even if there is a crush somewhere, or I have to disconnect my computer from network, the result is there, and I can continue later with -xmlstart from the last saved title (or whatever the script suggested me when I press ctrl c).
I suppose, this is not the case with > (and it still my be operating system dependent).

More to say, if it lasts too long, I can work parallel. When the bot saved 300 titles and still works on the next 3000, I can process the saved titles from another command window. This is much more flexibility than that of > / >>.

I extensively use both save and savenew.

Saveexc is different, it is created during the work. For using > you should create two output streams (one for console output, one for exceptions), and all data might be lost upon unexpected termination.

In compat -save works fine for me either with -always or without it.

In T144692#2607590, @binbot wrote:

I don't know listpages as I always used compat.

Then could you please set up core and test whether listpages solves this use case?

-save/savenew (with flush() after every save) works reliably even for very slow, long search tasks. (I mean up to several days!) Even if there is a crush somewhere, or I have to disconnect my computer from network, the result is there, and I can continue later with -xmlstart from the last saved title (or whatever the script suggested me when I press ctrl c).
I suppose, this is not the case with > (and it still my be operating system dependent).

Why would this not be possible with >?

More to say, if it lasts too long, I can work parallel. When the bot saved 300 titles and still works on the next 3000, I can process the saved titles from another command window. This is much more flexibility than that of > / >>.

stdout can also be flushed.

Saveexc is different, it is created during the work. For using > you should create two output streams (one for console output, one for exceptions), and all data might be lost upon unexpected termination.

We already split output (user output goes to stderr, pipe-able output goes to stdout), so I'm unsure why this would not work, nor why there would be data loss in some cases?

In compat -save works fine for me either with -always or without it.

Never mind, I misread the code. It's indeed saved.

I will set up core but that takes more. For the moment I visited https://www.mediawiki.org/wiki/Manual:Pywikibot/Scripts which states: "Compat equivalent: pagegenerators.py and get.py." I also looked into the code and it does not have the complexity of fixes. I don't need a simple pagegenerator for this task, I need replace.py with -save that uses the same fixes and exceptions (included T144693) that the real replacements use. My fixes are rather sophisticated and I don't see any sign in listpages.py that it could provide this level.

I don't believe that listpages.py can handle this funcionality: https://hu.wikipedia.org/wiki/Szerkeszt%C5%91:Bin%C3%A1ris/Fixes_and_functions_HOWTO

I also don't know how much experience you have with frustrating console encoding problems, but I have suffered from them for years. While saved fixes always worked well, it was not the same with command line arguments. I had really, really many problems with that. Now, -save opens a relaiable UTF-8 file using codecs. That is proved. I don't feel the energy for trying new methods and experimenting again once something had already worked fine.

Or maybe listpages functionality should be extended instead.