Maniphest T209942

FileImporter should add entries to the import log
Closed, ResolvedPublic8 Story Points
Actions

Description

When the FileImporter successfully finishes an import it should not only add a NullRevision to inform about the import step in the revision history but should also add an entry to the upload log and one to the import log. These should all be associated with the importing user and be timestamped with the import date.

Having the NullRevision and the upload log entry is the default for uploads [1] [2] (although it leads to "doubled" entries on the history page). Since we, apart from the upload, do an import, having an import log entry is the right thing to do.

[1] https://commons.wikimedia.org/w/index.php?title=File:Usenet_Services_Mind_Map.svg&action=history
[2] https://commons.wikimedia.org/w/index.php?title=Special:Log&page=File%3AUsenet+Services+Mind+Map.svg

Acceptance Criteria

Add an entry to the import log associated with the null revision of the import

Notes
Most probably we will still just have one entry (the null revision) in the page history.

Dev notes
This includes tests

Details

Related Gerrit Patches:

mediawiki/extensions/FileImporter : master	Integration test for import log entry
mediawiki/extensions/FileImporter : master	When importing a file add an import log entry.
mediawiki/extensions/FileImporter : master	Avoid using the real User class in tests, where possible

Related Objects

Mentioned In: T210755: Create upload log entries for all imported file revisions
T198136: Insert a log entry on import
rEFLI677688d45e9d: Integration test for import log entry
rEFLIfc322f6e8a63: Integration test for import log entry
rEFLI82a29d1e462e: Avoid using the real User class in tests, where possible
rEFLIb5c58110e042: Avoid using the real User class in tests, where possible
rEFLIfda27ad5ab06: When importing a file add an import log entry.
rEFLIa4ba9f2c078a: Avoid using the real User class in tests, where possible
rEFLIcfcd2e267f53: When importing a file add an import log entry.
rEFLI119050db8fe8: When importing a file add an import log entry.
Mentioned Here: T198324: Set a tag so that files imported with FileImporter can be filtered in Recent Changes
T210755: Create upload log entries for all imported file revisions

Event Timeline

WMDE-Fisch created this task.Nov 20 2018, 1:14 PM

thiemowmde triaged this task as Medium priority.Nov 20 2018, 1:27 PM

thiemowmde moved this task from Incoming to Move files to Commons on the TCB-Team board.

thiemowmde added subscribers: JStrodt_WMDE, thiemowmde, • Lea_WMDE.

Just as first comment here, when working on this it might help to have a look at LocalFile::recordUpload2()and ImportReporter::reportPage in core.

WMDE-Fisch renamed this task from [PLACEHOLDER] FileImporter logging to FileImporter should add entries to the import and upload log.Nov 23 2018, 4:08 PM

WMDE-Fisch updated the task description. (Show Details)

JStrodt_WMDE awarded a token.Nov 23 2018, 4:09 PM

Raymond added a subscriber: Raymond.Nov 23 2018, 4:30 PM

Andrew-WMDE claimed this task.Nov 28 2018, 12:00 PM

Andrew-WMDE moved this task from Sprint Backlog to Doing on the WMDE-QWERTY-Sprint-2018-11-20 board.

Change 476270 had a related patch set uploaded (by Andrew-WMDE; owner: Andrew-WMDE):
[mediawiki/extensions/FileImporter@master] When importing a file add an import log entry.

https://gerrit.wikimedia.org/r/476270

gerritbot added a project: Patch-For-Review.Nov 28 2018, 2:33 PM

Andrew-WMDE mentioned this in rEFLI119050db8fe8: When importing a file add an import log entry..Nov 28 2018, 2:34 PM

Andrew-WMDE mentioned this in rEFLIcfcd2e267f53: When importing a file add an import log entry..Nov 28 2018, 3:05 PM

Andrew-WMDE moved this task from Doing to Review on the WMDE-QWERTY-Sprint-2018-11-20 board.Nov 28 2018, 3:32 PM

• Lea_WMDE updated the task description. (Show Details)Nov 28 2018, 4:52 PM

• Lea_WMDE set the point value for this task to 8.

• Lea_WMDE moved this task from Backlog to Tickets in sprint on the Move-Files-To-Commons board.

Is new type of import that would allow filtering imports where file versions were imported along with page revisions in import log also planned?

As for handling upload log entries, I note that currently only the latest upload log entry is copied from (or generated after) source wiki if there were reuploads, e.g. compare this and this. I suppose that if upload log entries are copied then it'd be consistent to copy all of them along with all file versions that are imported.

Change 476523 had a related patch set uploaded (by Thiemo Kreuz (WMDE); owner: Thiemo Kreuz (WMDE)):
[mediawiki/extensions/FileImporter@master] Avoid using the real User class in tests, where possible

https://gerrit.wikimedia.org/r/476523

thiemowmde mentioned this in rEFLIa4ba9f2c078a: Avoid using the real User class in tests, where possible.Nov 29 2018, 3:10 PM

WMDE-Fisch claimed this task.Nov 29 2018, 4:44 PM

WMDE-Fisch added a subscriber: Andrew-WMDE.

WMDE-Fisch mentioned this in rEFLIfda27ad5ab06: When importing a file add an import log entry..Nov 29 2018, 4:44 PM

WMDE-Fisch mentioned this in rEFLIb5c58110e042: Avoid using the real User class in tests, where possible.Nov 29 2018, 4:47 PM

WMDE-Fisch mentioned this in rEFLI82a29d1e462e: Avoid using the real User class in tests, where possible.Nov 29 2018, 4:50 PM

Change 476523 merged by jenkins-bot:
[mediawiki/extensions/FileImporter@master] Avoid using the real User class in tests, where possible

https://gerrit.wikimedia.org/r/476523

ReleaseTaggerBot added a project: MW-1.33-notes (1.33.0-wmf.8; 2018-12-11).Nov 29 2018, 5:00 PM

In T209942#4784293, @Pikne wrote:

Is new type of import that would allow filtering imports where file versions were imported along with page revisions in import log also planned?
As for handling upload log entries, I note that currently only the latest upload log entry is copied from (or generated after) source wiki if there were reuploads, e.g. compare this and this. I suppose that if upload log entries are copied then it'd be consistent to copy all of them along with all file versions that are imported.

Interesting point, I assumed that this already working. - I created a separate ticket for that T210755

Change 476270 merged by jenkins-bot:
[mediawiki/extensions/FileImporter@master] When importing a file add an import log entry.

https://gerrit.wikimedia.org/r/476270

Change 476851 had a related patch set uploaded (by WMDE-Fisch; owner: WMDE-Fisch):
[mediawiki/extensions/FileImporter@master] Integration test for import log entry

https://gerrit.wikimedia.org/r/476851

WMDE-Fisch mentioned this in rEFLIfc322f6e8a63: Integration test for import log entry.Nov 30 2018, 1:37 PM

WMDE-Fisch mentioned this in rEFLI677688d45e9d: Integration test for import log entry.Nov 30 2018, 3:30 PM

WMDE-Fisch updated the task description. (Show Details)Dec 3 2018, 11:21 AM

In T209942#4784293, @Pikne wrote:

Is new type of import that would allow filtering imports where file versions were imported along with page revisions in import log also planned?

We are going to use "transwiki import" as sub type in the import log. This type already exists. Would it make more sense for you if there was another specific type? We thought that this is pretty much a transwiki activity :)

In T209942#4793870, @Lea_WMDE wrote:

We are going to use "transwiki import" as sub type in the import log. This type already exists. Would it make more sense for you if there was another specific type? We thought that this is pretty much a transwiki activity :)

I thought that new subtype would be useful as it'd allow filtering file imports. But indeed, then you'd probably have to rename "transwiki import" to make distinction between imports with and without file versions clear. So I see that adding a change tag (T198324) for this purpose might be less complicated and sufficient as well (though it's less convenient to retrieve revisions older than 30 days not present in recent changes).

• Lea_WMDE renamed this task from FileImporter should add entries to the import and upload log to FileImporter should add entries to the import log.Dec 4 2018, 1:17 PM

• Lea_WMDE updated the task description. (Show Details)

• Lea_WMDE moved this task from Review to Demo on the WMDE-QWERTY-Sprint-2018-11-20 board.

• Lea_WMDE mentioned this in T198136: Insert a log entry on import.Dec 4 2018, 4:02 PM

• Lea_WMDE merged a task: T198136: Insert a log entry on import.

• Lea_WMDE added a subscriber: • MarcoAurelio.

Pikne mentioned this in T210755: Create upload log entries for all imported file revisions.Dec 5 2018, 11:13 AM

Change 476851 merged by jenkins-bot:
[mediawiki/extensions/FileImporter@master] Integration test for import log entry