Page MenuHomePhabricator

reflinks should fetch only 1Mb of each linked document
Open, HighPublic

Description

T111300: Convert reflinks to requests regressed performance slightly, as the previous code would only request ~1Mb. This fetch limit should be re-added to reflinks.

Event Timeline

jayvdb created this task.Jan 20 2016, 5:33 AM
jayvdb raised the priority of this task from to High.
jayvdb updated the task description. (Show Details)
Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald TranscriptJan 20 2016, 5:33 AM
MtDu claimed this task.Jan 22 2016, 5:30 PM
MtDu added a subscriber: MtDu.

I'll fix this.

Change 265769 had a related patch set uploaded (by MtDu):
Make reflinks only fetch 1 Mb of each linked document

https://gerrit.wikimedia.org/r/265769

MtDu added a subscriber: jayvdb.Jul 12 2016, 4:29 AM

@jayvdb
Could you take a look at the patch and offer some comments?

@jayvdb
Could you take a look at the patch and offer some comments?

Done.

MtDu added a comment.Jul 29 2016, 12:24 AM

@jayvdb
Thanks for reviewing the panoramiopicker patch. I recently uploaded a new patch here. Could you please give some insight?
Thanks,
MtDu

jayvdb added a comment.Jan 7 2018, 2:42 AM

sorry I missed this; could you rebase the patch and submit to travis so we can see if it doesnt break anything (reflinks has good tests, so that is sufficient to verify nothing is broken). Then make sure that one of the tests tries to fetch a 2 Mb PDF, and emit a warning to indicate that the document was trimmed.

D3r1ck01 moved this task from Backlog to Needs Review on the Pywikibot board.Nov 5 2018, 11:33 AM