Page MenuHomePhabricator
Create Task
Maniphest T47237

LinkSearch uses numeric offset paging instead of paging by last entry returned
Open, MediumPublic
Actions

Description

I noticed today, that paging on special:linksearch uses a numeric offset. For example: http://commons.wikimedia.org/w/index.php?title=Special:LinkSearch&limit=500&offset=5000&target=http%3A%2F%2Fwww.wga.hu%2Fart%2F

For efficiency reasons it should probably use an offset of the last entry displayed on the page. (For example: offset=http://www.wga.hu/art/c/crespi/daniele/pieta.jpg or something)

Note: I say this without looking how it is implemented/the db queries. So I may be misunderstanding the issues involved.

Version: 1.21.x
Severity: normal
URL: http://commons.wikimedia.org/w/index.php?title=Special:LinkSearch&limit=500&offset=5000&target=http%3A%2F%2Fwww.wga.hu%2Fart%2F

Details

Reference
bz45237
SubjectRepoBranchLines +/-
[WIP] SpecialLinkSearch: Switch to ordering based on el_idmediawiki/coremaster+78 -2
ExternalLinks: Make order by and continue only rely on el_id in READ NEWmediawiki/corewmf/1.41.0-wmf.16+18 -3
ExternalLinks: Make order by and continue only rely on el_id in READ NEWmediawiki/corewmf/1.41.0-wmf.15+18 -3
ExternalLinks: Make order by and continue only rely on el_id in READ NEWmediawiki/coremaster+18 -3
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedBawolffT130058 Special:LinkSearch fails over 10,000
 OpenNoneT47237 LinkSearch uses numeric offset paging instead of paging by last entry returned
 Resolved• LadsgroupT312666 Remove duplication in externallinks table
 Resolved• LadsgroupT318604 Add new externallinks fields
 Resolved• LadsgroupT318605 Deploy new externallinks fields to production
 Resolved• LadsgroupT330056 Switchover s8 master (db2165 -> db2161)
 Resolved• LadsgroupT318606 Write code for write both on externallinks
 Resolved• LadsgroupT321662 Enable write both for externallinks in beta and production
 Resolved• LadsgroupT321663 Write maintaince script backfilling externallinks data
 Duplicate• LadsgroupT321664 Enable write both for externallinks in beta cluster
 Resolved• LadsgroupT326251 Write code for read new fields of externallinks
 Resolved• LadsgroupT326314 Run maint script backfilling new external links fields
 Resolved• LadsgroupT335343 Set externallinks migration stage to read new on beta and production
 Resolved• LadsgroupT335819 Deprecate and remove protocol-relative external links searching
 ResolvedAntoine_QuhenT335917 Update Sqoop for externallinks table changes
 Resolved• LadsgroupT341828 Prepare MediaWiki for drop of externallinks
 Resolved• LadsgroupT342617 Make old columns of externallinks nullable
 Resolved• LadsgroupT342683 Stop writing to the old externallinks columns in beta cluster and production
 Resolved• LadsgroupT343718 Drop old columns of externallinks
 ResolvedBUG REPORT• LadsgroupT344866 Queries to externallinks table fail following schema changes

Event Timeline

• bzimport raised the priority of this task from to Medium.Nov 22 2014, 1:14 AM
• bzimport added a project: MediaWiki-Special-pages.
• bzimport set Reference to bz45237.
• bzimport added a subscriber: Unknown Object (MLST).
Bawolff created this task.Feb 21 2013, 4:28 PM
duplicatebug added a comment.Mar 16 2013, 6:02 PM

API uses also offset, because there is no primary key you can use for pagination. That makes it also hard to reverse the order.

Umherirrender added a comment.Nov 18 2013, 6:22 PM

primary key was added with gerrit 51675, maybe that helps

duplicatebug unsubscribed.Dec 13 2014, 11:50 AM
• Reguyla moved this task from To triage to Special:LinkSearch on the MediaWiki-Special-pages board.May 26 2015, 2:08 AM
Umherirrender mentioned this in T100807: Database error when cold-querying Special:LinkSearch with a high offset.May 29 2015, 4:36 PM
Umherirrender unsubscribed.Sep 29 2015, 3:05 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 29 2015, 3:05 PM
matmarex merged tasks: T130058: Special:LinkSearch fails over 10,000, T100807: Database error when cold-querying Special:LinkSearch with a high offset.Apr 6 2016, 4:30 PM
matmarex added subscribers: • DarTar, Sadads, Legoktm and 7 others.
matmarex subscribed.
Sadads added a comment.Apr 6 2016, 5:58 PM

@matmarex Why was T130058 merged? That appears to be tied to a recent other bug fix.

matmarex added a comment.Apr 6 2016, 8:04 PM

I think this is the only way to solve T130058. The artificial limit of 10 000 could be removed, I guess, but then you'd hit the real limit around 50 000 or so when the query time exceeds the timeout. Scanning 50 000 rows is just too slow.

Sadads added a comment.Apr 12 2016, 7:07 PM

@matmarex Sorry for not seeing this sooner: 50,000 fixes most of The Wikipedia Library concern for the short term (though not all -- JSTOR, one of our longest running partnerships, has over 70,000 links).

The magic bullet for the large number problem (since it mostly effects English), might be the ability to generate reports by namespace -- that way you could distinguish between Article space, and project space and User page links. The bulk of the use cases for Special:LinkSearch have functional difference between how links are used in Articles and how links are used in User space, etc.

matmarex mentioned this in T130058: Special:LinkSearch fails over 10,000.Apr 14 2016, 7:51 PM
matmarex added a parent task: T130058: Special:LinkSearch fails over 10,000.
• gpaumier unsubscribed.Jul 18 2018, 5:57 PM
Legoktm mentioned this in T312666: Remove duplication in externallinks table.Aug 12 2022, 2:28 AM
Legoktm added a subtask: T312666: Remove duplication in externallinks table.Aug 16 2022, 1:34 AM
gerritbot added a comment.Jul 4 2023, 2:40 PM

Change 935445 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@master] ExternalLinks: Make order by and continue only rely on el_id in READ NEW

https://gerrit.wikimedia.org/r/935445

gerritbot added a project: Patch-For-Review.Jul 4 2023, 2:40 PM
• Ladsgroup subscribed.Jul 4 2023, 2:49 PM

This patch doesn't solve the issue but it would make it easier to solve.

gerritbot added a comment.Jul 5 2023, 5:35 PM

Change 935445 merged by jenkins-bot:

[mediawiki/core@master] ExternalLinks: Make order by and continue only rely on el_id in READ NEW

https://gerrit.wikimedia.org/r/935445

ReleaseTaggerBot added a project: MW-1.41-notes (1.41.0-wmf.17; 2023-07-11).Jul 5 2023, 6:00 PM
Maintenance_bot removed a project: Patch-For-Review.Jul 5 2023, 6:10 PM
gerritbot added a comment.Jul 6 2023, 1:33 PM

Change 935856 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@wmf/1.41.0-wmf.16] ExternalLinks: Make order by and continue only rely on el_id in READ NEW

https://gerrit.wikimedia.org/r/935856

gerritbot added a project: Patch-For-Review.Jul 6 2023, 1:33 PM

Change 935857 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@wmf/1.41.0-wmf.15] ExternalLinks: Make order by and continue only rely on el_id in READ NEW

https://gerrit.wikimedia.org/r/935857

gerritbot added a comment.Jul 7 2023, 11:18 AM

Change 935857 abandoned by Ladsgroup:

[mediawiki/core@wmf/1.41.0-wmf.15] ExternalLinks: Make order by and continue only rely on el_id in READ NEW

Reason:

https://gerrit.wikimedia.org/r/935857

gerritbot added a comment.Jul 10 2023, 1:34 PM

Change 935856 merged by jenkins-bot:

[mediawiki/core@wmf/1.41.0-wmf.16] ExternalLinks: Make order by and continue only rely on el_id in READ NEW

https://gerrit.wikimedia.org/r/935856

Stashbot added a comment.Jul 10 2023, 1:35 PM

Mentioned in SAL (#wikimedia-operations) [2023-07-10T13:35:09Z] <ladsgroup@deploy1002> Started scap: Backport for [[gerrit:935856|ExternalLinks: Make order by and continue only rely on el_id in READ NEW (T341000 T47237)]]

Stashbot added a comment.Jul 10 2023, 1:36 PM

Mentioned in SAL (#wikimedia-operations) [2023-07-10T13:36:38Z] <ladsgroup@deploy1002> ladsgroup: Backport for [[gerrit:935856|ExternalLinks: Make order by and continue only rely on el_id in READ NEW (T341000 T47237)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet

Stashbot added a comment.Jul 10 2023, 1:46 PM

Mentioned in SAL (#wikimedia-operations) [2023-07-10T13:46:12Z] <ladsgroup@deploy1002> Finished scap: Backport for [[gerrit:935856|ExternalLinks: Make order by and continue only rely on el_id in READ NEW (T341000 T47237)]] (duration: 11m 03s)

ReleaseTaggerBot edited projects, added MW-1.41-notes (1.41.0-wmf.16; 2023-07-04); removed MW-1.41-notes (1.41.0-wmf.17; 2023-07-11).Jul 10 2023, 2:00 PM
Maintenance_bot removed a project: Patch-For-Review.Jul 10 2023, 2:11 PM
gerritbot added a comment.Jul 18 2023, 5:13 PM

Change 939354 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@master] [WIP] SpecialLinkSearch: Switch to ordering based on el_id

https://gerrit.wikimedia.org/r/939354

gerritbot added a project: Patch-For-Review.Jul 18 2023, 5:13 PM
Jdforrester-WMF closed subtask T312666: Remove duplication in externallinks table as Resolved.Sep 5 2023, 3:05 PM
Pppery edited projects, added Patch-Needs-Improvement; removed Patch-For-Review.Sat, Aug 17, 5:03 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits