I've been looking at this, assuming it would be OK to enter a directory name in the "file" field, and that directory contains the xml files. I think it's doable, scanning the files one at a time, but the assumption that only one file is being scanned per scan is woven into the code in various places. Also handling error conditions would be a pain. As I'm about to go on a 5-week trip I doubt I could get this done before mid-November. I'd be delighted for someone else to take it of course.

ETA: Or it might be easier to write a standalone app (or AWB procedure) to merge the separate files into one humungous file. That would double the disk space required, but I wouldn't be surprised if someone has already done it.

Imagine a "pre parse mode" for the Database Scanner: read the 19 separate files, or perhaps the "one humungous file", apply any namespace / title / regex / whatever filters that have been configured, and instead of copying the page name to the list box, write that XML record out to an output file. That might get the output file back down to 100 GB.

I take your point, that a generic merge-everything operation would be an expensive step. But you are describing a fair amount of code already in the AWB scanner, so it may be quicker to cut out the middle man and write the filtered deserialized information straight into the list box. In either case there's some code refactoring needed.

I have a solution, checked by John, but it needs some cleanup and error handling. It involves an extra dialog, but the alternative of changing the existing UI to ask for a directory full of XML files instead of a single file might be too disruptive. Do we know how many users do database scanning?

Technical: create a Stream subclass that wraps all the files in the list, together with a canned enclosing element. This stream is read by XmlTextReader instead of the "normal" FileStream.

I'll try to get this finished because the old style will no longer be exported from Jan 2027. There are other changes afoot (including adding 7z to bz2).
https://wikitech.wikimedia.org/wiki/Legacy_content_dumps_sunset