⚓ T177681 Page URL should update on redirect

Subject	Repo	Branch	Lines +/-
Fix: gracefully handle redirects to external sites	marvin	master	+177 -78
Chore: add fetch wrapper tests	marvin	master	+73 -8
Update: report server URL redirect statuses	marvin	master	+103 -14
Chore: expose HTTP response statuses from data clients	marvin	master	+74 -37

Status	Assigned	Task
Declined	ovasileva	T176958 Display article content within article namespace
Declined	None	T173317 Initial version of article rendering
Declined	None	T177681 Page URL should update on redirect
Declined	• bearND	T179600 Add canonical title to mobile-sections lead

• Niedzielski created this task.Oct 6 2017, 11:59 PM

• Niedzielski mentioned this in rMARVfe0e870cafb9: New: add random page and summary endpoints.Oct 11 2017, 8:02 PM

• Niedzielski updated the task description. (Show Details)Oct 12 2017, 4:09 PM

Some AC

Server side render of a redirect, responds with 301 to canonical url
Client side render of a redirect, replace the url with the canonical one

• Niedzielski mentioned this in rMARV979d10864f54: New: add random page and summary endpoints.Oct 17 2017, 1:49 PM

• Niedzielski updated the task description. (Show Details)Oct 17 2017, 6:18 PM

• Jhernandez added a parent task: T173317: Initial version of article rendering.Oct 18 2017, 11:13 AM

• Niedzielski mentioned this in rMARVaef0c7499355: New: add random page and summary endpoints.Oct 18 2017, 1:27 PM

Change 384985 had a related patch set uploaded (by Niedzielski; owner: Sniedzielski):
[marvin@master] Chore: expose HTTP response statuses from data clients

https://gerrit.wikimedia.org/r/384985

gerritbot added a project: Patch-For-Review.Oct 18 2017, 2:17 PM

• Niedzielski mentioned this in rMARVe82f31d78266: Chore: expose HTTP response statuses from data clients.Oct 18 2017, 2:18 PM

• Niedzielski mentioned this in rMARVc6df8c639972: Chore: expose HTTP response statuses from data clients.Oct 18 2017, 8:48 PM

• Niedzielski mentioned this in rMARV97cc198dbcc7: Chore: expose HTTP response statuses from data clients.Oct 19 2017, 1:32 PM

• Niedzielski mentioned this in rMARV6225dc28f24a: Chore: expose HTTP response statuses from data clients.Oct 19 2017, 3:49 PM

Change 384985 merged by jenkins-bot:
[marvin@master] Chore: expose HTTP response statuses from data clients

https://gerrit.wikimedia.org/r/384985

• Niedzielski claimed this task.Oct 30 2017, 7:03 PM

Change 387328 had a related patch set uploaded (by Niedzielski; owner: Sniedzielski):
[marvin@master] Update: report server URL redirect statuses

https://gerrit.wikimedia.org/r/387328

• Niedzielski mentioned this in rMARV541a730aff88: Update: report server URL redirect statuses.Oct 30 2017, 7:11 PM

• Niedzielski mentioned this in rMARVcd4ec6950db4: Update: report server URL redirect statuses.Oct 30 2017, 7:29 PM

• Niedzielski mentioned this in rMARVd43268c89a90: Update: report server URL redirect statuses.Oct 30 2017, 7:49 PM

• Niedzielski moved this task from Needs Triage to Backlog (Priority Descending) on the Marvin board.Oct 31 2017, 3:38 PM

• Niedzielski mentioned this in rMARV2fb92dad4e5c: Update: report server URL redirect statuses.Oct 31 2017, 6:27 PM

• Niedzielski added a subtask: T179600: Add canonical title to mobile-sections lead.Nov 2 2017, 5:52 PM

Denormalized / canonical page title information is available in the page summary endpoint's titles.title and titles.namespace_name properties. The client can use this to update the Marvin URL for summaries. e.g:

http://appservice.wmflabs.org/en.wikipedia.org/v1/page/summary/Dog
{
  "type": "standard",
  "title": "Dog",
  "displaytitle": "Dog",
  "titles": {
    "title": "Dog",
    "normalized_title": "Dog",
    "display_title": "Dog",
    "namespace_id": 0,
    "namespace_name": ""
  },
  "pageid": 4269567,
  "thumbnail": {
    "source": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/d9/Collage_of_Nine_Dogs.jpg/320px-Collage_of_Nine_Dogs.jpg",
    "width": 320,
    "height": 281,
    "original": "https://upload.wikimedia.org/wikipedia/commons/d/d9/Collage_of_Nine_Dogs.jpg"
  },
  "originalimage": {
    "source": "https://upload.wikimedia.org/wikipedia/commons/d/d9/Collage_of_Nine_Dogs.jpg",
    "width": 1665,
    "height": 1463
  },
  "lang": "en",
  "dir": "ltr",
  "revision": "806438597",
  "timestamp": "2017-10-22T01:20:54Z",
  "description": "domestic animal",
  "extract": "The domestic dog is a member of genus Canis (canines) that forms part of the wolf-like canids, and is the most widely abundant carnivore. The dog and the extant gray wolf are sister taxa, with modern wolves not closely related to the wolves that were first domesticated, which implies that the direct ancestor of the dog is extinct. The dog was the first domesticated species and has been selectively bred over millennia for various behaviors, sensory capabilities, and physical attributes.",
  "extract_html": "<p>The <b>domestic dog</b> is a member of genus <i><span>Canis</span></i> (canines) that forms part of the <span>wolf-like canids</span>, and is the most widely abundant <span>carnivore</span>. The dog and the extant <span>gray wolf</span> are sister <span class=\"mw-redirect\">taxa</span>, with modern wolves not closely related to the wolves that were first domesticated, which implies that the direct ancestor of the dog is extinct. The dog was the first <span>domesticated</span> species and has been <span>selectively bred</span> over millennia for various behaviors, sensory capabilities, and physical attributes.</p>"
}

References:

These changes are yet deployed but will be.

For the mobile-sections-lead endpoint, @bearND made a nice ticket here to add a titles.title property of some kind. We can use this property with the existing ns_text property to get the resolved canonical URL. Note that redirects with fragments are not yet supported by RESTBase (T148645).

• Niedzielski mentioned this in rMARV511d06d7fd73: Update: report server URL redirect statuses.Nov 7 2017, 7:09 PM

Change 387328 merged by jenkins-bot:
[marvin@master] Update: report server URL redirect statuses

https://gerrit.wikimedia.org/r/387328

• Niedzielski updated the task description. (Show Details)Nov 7 2017, 7:30 PM

@bearND has taken the trouble to add the title property to the lead response in T179600 (thanks!). The change has been merged and appears to be available on appservice.wmflabs.org. We're waiting for it to appear on prod but I believe this task can be worked on in parallel.

The sections endpoint is deprecated so these changes have been rolled back per discussion offline.

• Niedzielski updated the task description. (Show Details)Nov 14 2017, 11:50 PM

Change 392573 had a related patch set uploaded (by Niedzielski; owner: Sniedzielski):
[marvin@master] Fix: gracefully handle redirects to external sites

https://gerrit.wikimedia.org/r/392573

• Niedzielski mentioned this in rMARVe997c18d1e84: Fix: gracefully handle redirects to external sites.Nov 21 2017, 4:13 AM

• Niedzielski mentioned this in rMARVfced8a7fc83a: Fix: gracefully handle redirects to external sites.Nov 22 2017, 4:34 PM

• Niedzielski mentioned this in rMARVf3a9f32d9c31: Fix: gracefully handle redirects to external sites.Nov 22 2017, 5:06 PM

• Niedzielski mentioned this in rMARV5e72fdadd223: Fix: gracefully handle redirects to external sites.Nov 30 2017, 1:41 PM

Change 394476 had a related patch set uploaded (by Niedzielski; owner: Sniedzielski):
[marvin@master] Chore: add fetch wrapper tests

https://gerrit.wikimedia.org/r/394476

• Niedzielski mentioned this in rMARVf8fdbf43e0c3: Chore: add fetch wrapper tests.Nov 30 2017, 10:57 PM

• Niedzielski mentioned this in rMARVd7181bcf3309: Fix: gracefully handle redirects to external sites.

• Niedzielski mentioned this in rMARVb87dc9b0e78d: Chore: add fetch wrapper tests.Nov 30 2017, 11:17 PM

• Niedzielski mentioned this in rMARVe03e52e614f7: Fix: gracefully handle redirects to external sites.

• Niedzielski mentioned this in rMARV5c6472b1b109: Chore: add fetch wrapper tests.Dec 1 2017, 12:02 PM

• Niedzielski mentioned this in rMARVe6a5dcbd72e5: Chore: add fetch wrapper tests.Dec 1 2017, 6:14 PM

• Niedzielski mentioned this in rMARVb7c70d2e59bc: Fix: gracefully handle redirects to external sites.

This should also include a "redirected from" notice, cf. https://en.m.wikipedia.org/wiki/Einstein and T53736#3382857 .

Also, regarding section redirects,
http://marvin.wmflabs.org/wiki/Economy_of_Christmas_Island should redirect to http://marvin.wmflabs.org/wiki/Christmas_Island#Economy (@Krinkle's example from T53736#3389432 .)

More generally, is this already implementing the RfC outcome on implementing proper HTTP redirects (T53736)?

• Niedzielski mentioned this in rMARV1e94296af657: Chore: add fetch wrapper tests.Dec 2 2017, 9:02 PM

• Niedzielski mentioned this in rMARV525d2044865c: Fix: gracefully handle redirects to external sites.

This should also include a "redirected from" notice, cf. https://en.m.wikipedia.org/wiki/Einstein and T53736#3382857 .

@Tbayer, thanks! Now tracked in T182120.

I think that section redirects are blocked by T148645 too.

is this already implementing the RfC outcome on implementing proper HTTP redirects

Regardless of redirects, Marvin needs the final resolved page title which ideally would be provided in the response body. (This data is already present in the summary staging endpoint and is planned for PCS.) We're currently handling redirects manually to obtain it for the client. On the server, we must handle the redirect to propagate the appropriate HTTP response. This is necessary even with HTTP redirects.

• Niedzielski mentioned this in rMARVc25ccbc00a9a: Chore: add fetch wrapper tests.Dec 5 2017, 10:36 PM

• Niedzielski mentioned this in rMARV3da812f23237: Chore: add fetch wrapper tests.

• Niedzielski mentioned this in rMARVc1b21aa6b891: Fix: gracefully handle redirects to external sites.

• Niedzielski mentioned this in rMARVb847779302b3: Chore: add fetch wrapper tests.Dec 6 2017, 6:48 PM

• Niedzielski mentioned this in rMARV441cad1115f4: Chore: add fetch wrapper tests.Dec 6 2017, 7:24 PM

• Niedzielski mentioned this in rMARV9f948324548b: Chore: add fetch wrapper tests.Dec 6 2017, 7:31 PM

• Niedzielski mentioned this in rMARV805ff7bc344a: Fix: gracefully handle redirects to external sites.Dec 6 2017, 7:55 PM

Change 394476 merged by jenkins-bot:
[marvin@master] Chore: add fetch wrapper tests

https://gerrit.wikimedia.org/r/394476

• Jhernandez mentioned this in rMARV09251fef92ea: Fix: gracefully handle redirects to external sites.Dec 7 2017, 12:37 PM

• Niedzielski mentioned this in rMARV9ef1fc11e963: Fix: gracefully handle redirects to external sites.Dec 7 2017, 1:05 PM

• Niedzielski mentioned this in rMARVa87dfd443144: Fix: gracefully handle redirects to external sites.Dec 7 2017, 1:54 PM