Page MenuHomePhabricator

Transwiki import not working in production
Closed, ResolvedPublic

Description

When I was trying to import a page to betawikiversity, this happened:

Please help.

Event Timeline

Cosine02 created this task.Jul 13 2016, 3:08 AM
Restricted Application added subscribers: Zppix, JEumerus, Matanya and 2 others. · View Herald TranscriptJul 13 2016, 3:08 AM
TTO added a comment.Jul 13 2016, 3:15 AM

Could you please provide the name of the page in text form?

TTO added a comment.Jul 13 2016, 3:20 AM

Okay, I was worried that that page might have had a massive history, but that doesn't seem to be the case.

Can you import other pages from Chinese Wikipedia? Can you import this page with the "copy all revisions" option turned off?

Okay, I was worried that that page might have had a massive history, but that doesn't seem to be the case.
Can you import other pages from Chinese Wikipedia? Can you import this page with the "copy all revisions" option turned off?

Both are not ok: They display "Import failed: Could not open import file". But it's quite strange that some days ago, I successfully imported a page for zhwiki. So whats the problem?

TTO renamed this task from Import problem in betawikiversity to Transwiki import not working in production.Jul 13 2016, 3:31 AM
TTO triaged this task as High priority.

OK, seems to be a global issue.

OK, seems to be a global issue.

Confirmed, attempted Meta->MwWiki

Change 298693 had a related patch set uploaded (by TTO):
Force HTTPS for the interwiki map

https://gerrit.wikimedia.org/r/298693

Change 298693 merged by jenkins-bot:
Force HTTPS for the interwiki map

https://gerrit.wikimedia.org/r/298693

Change 298694 had a related patch set uploaded (by Legoktm):
Force HTTPS for the interwiki map

https://gerrit.wikimedia.org/r/298694

Change 298694 merged by jenkins-bot:
Force HTTPS for the interwiki map

https://gerrit.wikimedia.org/r/298694

TTO closed this task as Resolved.EditedJul 13 2016, 4:24 AM
TTO claimed this task.

The import process was still fetching the pages over HTTP, and after recent changes requiring clients to use HTTPS, the internal import requests were being denied with HTTP 403 Forbidden errors.

To fix this, the interwiki map has been updated so that all interwiki links now use https: URLs instead of protocol-relative URLs (which are always expanded to http: URLs by the import process).

Thanks to @Legoktm for looking into this so quickly.

TTO added a subscriber: Legoktm.Jul 13 2016, 4:24 AM
Legoktm added a subscriber: BBlack.Jul 13 2016, 4:25 AM

This was caused by the change to break HTTP POST requests to the API, we just didn't notice that transwiki import was still going over HTTP. Since it used the interwiki map, which had proto-rel URLs, it was assuming HTTP. We made the interwiki map HTTPS, so that transwiki import will use HTTPS as well. Confirmed by importing an article from enwp to testwp.

CC @BBlack because I'm not sure why we didn't notice this in the logs he was looking through for bots...

We only reported on logged-in account access during the final phase, not anonymous access.

TTO added a comment.Jul 16 2016, 9:39 AM

I don't have import rights on enwiki, so I wonder if @Legoktm has time to look at this?

nostalgiawiki is still protcol-relative on https://meta.wikimedia.org/wiki/Interwiki_map and there are others like mediawiki.org, oldwikisource, etc. that also need to be converted...a meta admin, GEI, or steward will need to do that and then I can re-sync it.

(Also I'm afk most of today so if someone else wants to take the lead on this, that would be great. Otherwise I'll get to it in ~12 hours).

Dereckson closed this task as Resolved.Jul 16 2016, 7:29 PM

Okay, I'm going to find someone with the correct edit rights on meta, then prepare the config change.

Dereckson reopened this task as Open.Jul 16 2016, 7:29 PM
Dereckson claimed this task.

Change 299299 had a related patch set uploaded (by Dereckson):
Update interwiki map

https://gerrit.wikimedia.org/r/299299

Change 299299 merged by jenkins-bot:
Update interwiki map

https://gerrit.wikimedia.org/r/299299

Mentioned in SAL [2016-07-16T20:41:15Z] <twentyafterfour> deploying interwiki https config change https://gerrit.wikimedia.org/r/#/c/299299 refs T140206

Dereckson closed this task as Resolved.Jul 16 2016, 8:47 PM

New interwiki map is live.

TTO added a comment.Jul 16 2016, 11:31 PM

Funny thing is, I actually preempted this, but didn't put two and two together when nostalgiawiki was mentioned (I forgot it was in the meta map). Thanks to those who got onto fixing this.