Stalled until making more progress on T114703, where I'll be figuring out exactly what data I need to include in the dump in order to be able to import it.

Change 242569 merged by jenkins-bot:
Dump Flow data

https://gerrit.wikimedia.org/r/242569

Tested with the following. Main thing I didn't cover is the ID from/to, but I'm sure @Etonkovidova can see what I didn't think of.

full-wiki

• Destroy
• Rebuild
• Setup wikis. Make sure there are:
  • Multiple pages
    • Enabled board 1 (S:EF/Special:EnableFlow)
    • Wiki:Enabled board (S:EF)
    • User talk:Admin (Beta)
    • Flow test talk:Flow namespace 1
  • Headers
    • Flow namespace 1 without a header
  • Multiple topics
  • A few topic summaries
  • IP post
• Some history
  • Edited posts
  • Edited summary
  • Edited header
  • Edited topic title
  • Delete whole board and record title and Flow board ID (Flow test talk:Deleting_whole_board and ...)
  • Delete a couple posts
  • Delete a couple topics
  • Suppress a couple posts
  • Suppress a couple topics
  • Restore one deleted post
  • Restore one deleted topic
  • Restore one suppressed post
  • Restore one suppressed topic
  • Delete, suppress, unsuppress post (unsuppress apparently goes directly to visible)
• Export full wiki with all history
  • cd /vagrant/mediawiki/extensions/Flow; mwscript extensions/Flow/maintenance/dumpBackup.php --full 2>&1 | tee /vagrant/Flow-dump-full-wiki-all-history-$(git rev-parse --short HEAD).xml
• Destroy
• Rebuild
• Full wiki import
  • cd /vagrant/mediawiki/extensions/Flow; mwscript maintenance/importDump.php < /vagrant/Flow-dump-full-wiki-all-history-$(git rev-parse --short HEAD).xml 2>&1 | tee "/vagrant/Flow-dump-full-wiki-import-$(date).log"
  • Check debug log file (https://www.mediawiki.org/wiki/Manual:$wgDebugLogFile)

pages with current

• Add custom user (who won't exist at destination)
• Post as that user (done, to Flow_test_talk:Flow_namespace_1)
• Export specific boards with current (User talk:Admin, Wiki:Enabled board, Flow_test_talk:Flow_namespace_1)
  • cd /vagrant/mediawiki/extensions/Flow; mwscript extensions/Flow/maintenance/dumpBackup.php --current --pagelist=/vagrant/pagelist-2015-11-24.txt 2>&1 | tee /vagrant/Flow-dump-pagelist-current-$(git rev-parse --short HEAD).xml
• Destroy
• Rebuild
• Create other boards
• Import those boards
  • cd /vagrant/mediawiki/extensions/Flow; mwscript maintenance/importDump.php < /vagrant/Flow-dump-pagelist-current-$(git rev-parse --short HEAD).xml 2>&1 | tee "/vagrant/Flow-dump-pagelist-current-import-$(date).log"

Where on https://dumps.wikimedia.org/backup-index.html do I see the dumps?

Ah, I see, reversed dependencies.

I've got a gerrit changeset which needs testing (https://gerrit.wikimedia.org/r/#/c/282883/).

I expect to be able to pass --output:bz2:/mnt/data/xmldatadumps/..../mediawikiwiki-MMDDYY_flow.xml.bz to Flow/maintenance/dumpBackup.php, and indeed I can but it doesn't actually write to the specified output file like other dump jobs do. Can someone have a look?

Change 283117 had a related patch set uploaded (by Mattflaschen):
Fix sink-handling

https://gerrit.wikimedia.org/r/283117

In T89398#2198392, @ArielGlenn wrote:

I expect to be able to pass --output:bz2:/mnt/data/xmldatadumps/..../mediawikiwiki-MMDDYY_flow.xml.bz to Flow/maintenance/dumpBackup.php, and indeed I can but it doesn't actually write to the specified output file like other dump jobs do. Can someone have a look?

There was a bug on our side, fixed in https://gerrit.wikimedia.org/r/283117 .

For those testing, it's:

--output=bzip2:/mnt/data/xmldatadumps/..../mediawikiwiki-MMDDYY_flow.xml.bz

Note equal sign and bzip2, not bz2. It is correct in that Gerrit, but I haven't reviewed that otherwise.

Yep sorry, I guarantee I got the option right when doing my manual tests, shoulda copypasted here.

datasets@snapshot1005:~$ php /srv/mediawiki/multiversion/MWScript.php extensions/Flow/maintenance/dumpBackup.php --wiki mediawikiwiki --current --output=bzip2:/home/datasets/testing/blot.bz2

Currently producing output as expected in the specified file in bz2 format, just waiting to make sure it completes properly before I give the total thumbs up. Looks good though.

(edited) Finished and looks good.

Because both current and full dumps can be produced for Flow pages according to the options for the script at any rate, do you have a means for full dumps to be produced by reading the old fulls and polling the db only for new/changed revisions? Or would we have to go to the db for each revision each time?

We do this for the regular xml dumps but it requires a 2-stage process, involving dumping the metadata for each page/revision first and using that to decide which revisions to grab from the old fulls, in case some pages/revisions have been deleted or oversighted in the meantime. I don't know if it is possible to delete or oversight Flow pages; can you give me a bit of an overview or point me to something to read? Thanks.

Change 283117 merged by jenkins-bot:
Fix sink-handling

https://gerrit.wikimedia.org/r/283117

@ArielGlenn: Where does that 2-stage process happen? I took a quick look at core's dumpBackup.php but that one didn't seem to take an existing dump file to work with.

It is possible to delete/oversight Flow pages: https://www.mediawiki.org/wiki/Extension:Flow/Moderation

Do you think being able to dump a range of revisions is useful at all?
If yes, how should that work? Current code just checks if the last change in a topic is within that range and includes the entire topic in such case
But we should probably only include the specific revisions in that range. Which means that some history (earlier edits of a post/topic) & context (post may have been in that range, but topic was created earlier) may be missing.
If we were to include only specific revisions, an import would change because of that missing context/history. Should we:

• assume that said history/context is already in place (e.g. through earlier imports) and just let it fail if it isn't
• product dumps as if there was no earlier history & with additional context where needed (so these partial dumps can be imported standalone)
• just get rid of revrange because it doesn't make much sense in Flow's context as it is :)

These dumps should make it possible for someone to set up a mirror (right to fork). That means that all data available to the public should be provided. IMHO:

"Current" ought to dump topics currently visible on a page.
"Full" ought to dump the history (whatever that means), i.e. all topics that were ever on the page, that are not deleted/made private.
"Revision range" sounds pretty useless for your model.

Does that make sense so far?

In T89398#2204827, @ArielGlenn wrote:

"Current" ought to dump topics currently visible on a page.

It does. Also, it's the latest version of everything (topic titles, posts, board-level descriptions, and topic-level summaries can all be edited and are versioned).

"Full" ought to dump the history (whatever that means), i.e. all topics that were ever on the page, that are not deleted/made private.

Yes, this has all the versions (every rename of a renamed topic title, every edit of a post, summary, header, etc.) of the versioned content. Topics can not be disassociated from the board, so otherwise this is mostly the same as current.

In both cases (current and full), private data (defined as "not visible to an anonymous user") is removed. In the case of "full", that requires reconstructing the history to make it importable (it makes some internal changes like updating the previous revision pointer).

"Revision range" sounds pretty useless for your model.

In theory, it could maybe be useful to someone, but right now it's implemented but broken (and beyond the superficial breakage there are some more complicated issues as Matthias raised). So if it doesn't seem useful, I think I'll just rip it out for now.

Change 283328 had a related patch set uploaded (by Mattflaschen):
Remove revision range

https://gerrit.wikimedia.org/r/283328

Change 283328 merged by jenkins-bot:
Remove revision range

https://gerrit.wikimedia.org/r/283328

When will https://gerrit.wikimedia.org/r/283117 land in a deployed branch? So far it seems only to be available in master.

In T89398#2230914, @ArielGlenn wrote:

When will https://gerrit.wikimedia.org/r/283117 land in a deployed branch? So far it seems only to be available in master.

It should be in 1.27.0-wmf.22 this Tue., April 26th. The branch cuts were paused a week due to the data center switchover: https://www.mediawiki.org/wiki/MediaWiki_1.27/Roadmap .

Ok, I see from https://wikitech.wikimedia.org/wiki/Deployments that barring any issues it will be deployed to all wikis Thursday evening (my Friday morning). I'm off on Friday so that means we probably won't make the May run for these dumps, but I can certainly test and commit the code so it's ready for the following month.

Sounds good, thanks.

Change 282883 had a related patch set uploaded (by ArielGlenn):
add dumps for flow pages for those wikis which have Flow enabled

https://gerrit.wikimedia.org/r/282883

Change 282883 merged by ArielGlenn:
add dumps for flow pages for those wikis which have Flow enabled

https://gerrit.wikimedia.org/r/282883

This has been tested in a production environment and is ready to go. It will be deployed before the next run. I'll keep this ticket open til then as a reminder.

The new dump run shows a lot of failures for flow; these are on dbs that don't in fact have flow active. The code currently tries to run over wikis that are private or closed, insofar as they are not explicitly excluded in the nonflowdb list. That's an error; T137360 will clear that up.

Checking a couple of wikis where flow is in fact enabled, officewiki (private) and mediawikiwiki (regular wiki) both show xml dump files produced for flow pages successfully, as part of the month's dump run.

In T89398#2366101, @ArielGlenn wrote:

Checking a couple of wikis where flow is in fact enabled, officewiki (private) and mediawikiwiki (regular wiki) both show xml dump files produced for flow pages successfully, as part of the month's dump run.

When can this task be closed? After all wikis with a Flow page have had their first Flow dump generated?

I can close it after the current run completes, I push out a fix to get the right list of flow-enabled wikis, and I run a noop on all wikis that claim failure (they simply don't have flow enabled). That should be in about 5-6 days.

T119511: Publish recurring Flow dumps at http://dumps.wikimedia.org/ is also now a blocker for this, so they should be set to auto-recur before these two tasks are closed.

There's nothing needed for autorecurrence; the flow job is part of the regular dump runs.

This looks good, except for two things:

• There are only current dumps. We need full too (--full).
• Small issues with which wikis, below.

In T89398#2376524, @ArielGlenn wrote:

I can close it after the current run completes, I push out a fix to get the right list of flow-enabled wikis, and I run a noop on all wikis that claim failure (they simply don't have flow enabled). That should be in about 5-6 days.

As Roan mentioned at T137360: Handle 'dynamic' dblists, you can use flow.dblist, which is now not dynamic.

After that, let's check that Wikidata, etc. are present, and there are no errors for the non-Flow wikis, then we can move this to QA.

Thanks.

Non-flow wikis now do not get dumped (see the current https://dumps.wikimedia.org/backup-index.html page).

Wikidatawiki flow pages have just been dumped.

I have yet to add a job for --full. (Is there code in MW to support that?)

Change 295587 had a related patch set uploaded (by ArielGlenn):
add job that dumps history of flow pages

https://gerrit.wikimedia.org/r/295587

I'll test the above changeset manually in the next couple of days and see if we get results that differ from dumps of the "current" flow content.

Now testing on mediawiki via manual run.

SOofar it looks ok. I am now running flow history dumps across all wikis with flow enabled.

All completed except for mediawiki, which is taking rather a long time (days). Anyone have a clue why it should be so slow? Is there something different about the tables, or are there that many more flow entries?

Woops, missed ptwikibooks too, due to a bad grep on my part.

I looked at the number of entries in the flow_workflows tables for mediawiki and for ptwikibooks: 31k vs 26k. I'm running the flow history dumps for ptwikibooks now manually and we'll see how long it takes. There's no reason for so few rows to take days to dump; can retrieval and formatting of the revisions really take that long? Some investigation is needed. I'll report the length of time needed for ptwikibooks, that may shed some light on things.

Change 295587 merged by ArielGlenn:
add job that dumps history of flow pages

https://gerrit.wikimedia.org/r/295587

Another thing for the Flow extension folks to look at is the failures on testwiki. I get this exception:

command /usr/bin/php5 -q /srv/mediawiki/multiversion/MWScript.php extensions/Flow/maintenance/dumpBackup.php --wiki=testwiki --current --report=1000 --output=bzip2:/mnt/data/xmldatadumps/public/testwiki/20160601/testwiki-20160601-flow.xml.bz2 (21805) started...
[effb0c25c66162949d5acb27] [no req]   Flow\Exception\InvalidDataException from line 366 of /srv/mediawiki/php-1.28.0-wmf.7/extensions/Flow/includes/Model/AbstractRevision.php: Failed to load the content
Backtrace:
#0 /srv/mediawiki/php-1.28.0-wmf.7/extensions/Flow/includes/Dump/Exporter.php(395): Flow\Mode\\AbstractRevision->getContent(string)
#1 /srv/mediawiki/php-1.28.0-wmf.7/extensions/Flow/includes/Dump/Exporter.php(360): Flow\Dump\Exporter->formatRevision(Flow\Model\PostRevision)
#2 /srv/mediawiki/php-1.28.0-wmf.7/extensions/Flow/includes/Dump/Exporter.php(255): Flow\Dump\Exporter->formatRevisions(Flow\Model\PostRevision)
#3 /srv/mediawiki/php-1.28.0-wmf.7/extensions/Flow/includes/Dump/Exporter.php(213): Flow\Dump\Exporter->formatPost(Flow\Model\PostRevision)
#4 /srv/mediawiki/php-1.28.0-wmf.7/extensions/Flow/includes/Dump/Exporter.php(196): Flow\Dump\Exporter->formatTopic(Flow\Model\PostRevision)
#5 /srv/mediawiki/php-1.28.0-wmf.7/extensions/Flow/includes/Dump/Exporter.php(174): Flow\Dump\Exporter->formatWorkflow(Flow\Model\Workflow, Flow\Search\Iterators\HeaderIterator, Flow\Search\Iterators\TopicIterator)
#6 /srv/mediawiki/php-1.28.0-wmf.7/extensions/Flow/maintenance/dumpBackup.php(85): Flow\Dump\Exporter->dump(BatchRowIterator)
#7 /srv/mediawiki/php-1.28.0-wmf.7/extensions/Flow/maintenance/dumpBackup.php(58): FlowDumpBackup->dump(integer)
#8 /srv/mediawiki/php-1.28.0-wmf.7/maintenance/doMaintenance.php(103): FlowDumpBackup->execute()
#9 /srv/mediawiki/php-1.28.0-wmf.7/extensions/Flow/maintenance/dumpBackup.php(124): require_once(string)
#10 /srv/mediawiki/multiversion/MWScript.php(97): require_once(string)
#11 {main}

The ptwikibooks flow history dump took 62 minutes. Compare this to the mediawiki flow history dump which took 2.5 days. I would guess there's a query in there someplace that needs to be optimized, probably querying against the whole revision table. Can anyone check?

I don't know of T113682: Finish conversion from LQT to Flow on pt.wikibooks may have an impact on that task.

Interesting, thanks for the heads up.

I've tried dumping --full on my machine (with enwiki dataset) and on tin, and consistently get around 70-80 seconds.

mlitn@tin:~$ time mwscript extensions/Flow/maintenance/dumpBackup.php --wiki=enwiki --full --output=bzip2:/tmp/full.xml.bz2

real	1m14.007s
user	0m53.187s
sys	0m3.157s

What server are you running the dump on, and with which arguments? Is there any way we can profile it there?

Enwiki isn't a problem, mediawikiwiki is. Can you try that one?

Oh god, I misread. Yes, I'll try that one!

This is now waiting on the completion of this month's run, or at least some small wikis to run the flow history dumps, at which point I will consider it done. The mediawiki issues can get moved to a new ticket at that time.

In T89398#2409081, @ArielGlenn wrote:

Another thing for the Flow extension folks to look at is the failures on testwiki. I get this exception:

This is probably because of T95580: Flow data missing on Wikimedia production wikis. I've filed a follow up task (T139791: Flow: Handle missing content when dumping).

There may be problems dumping the topics listed at T138306: Broken topics on mediawiki.org and test wiki that the API doesnt believe exists.

May I close this task then as done, and the followup work can go on the two tasks you linked above?