Page MenuHomePhabricator

All wikis under wikimedia.org (Meta, Commons, loginwiki, others) are using wikimedia.org VHost, including /wiki/ -> wikimediafoundation.org redirect
Closed, ResolvedPublic

Tokens
"The World Burns" token, awarded by Liuxinyu970226."The World Burns" token, awarded by Ladsgroup."The World Burns" token, awarded by Ricordisamoa."The World Burns" token, awarded by MartinK."The World Burns" token, awarded by Ata."The World Burns" token, awarded by Scoopfinder."The World Burns" token, awarded by Pcoombe."The World Burns" token, awarded by Luke081515."The World Burns" token, awarded by Steinsplitter."The World Burns" token, awarded by I_JethroBT."The World Burns" token, awarded by Didym."The World Burns" token, awarded by Thibaut120094."The World Burns" token, awarded by MarcoAurelio.
Assigned To
Authored By
Vituzzu, Jan 26 2016

Description

Visiting commons.wikimedia.org or meta.wikimedia.org instead directs traffic to the main Wikimedia portal at wikimedia.org.

Original task description:

Here's what I got trying loggin in at meta There is no user by the name "Vituzzu". Check your spelling.

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
matmarex renamed this task from Meta, Commons, Wikispecies seem to redirect to wikimediafoundation.org to All wikis under wikimedia.org (Meta, Commons, others) are redirecting to wikimediafoundation.org.Jan 26 2016, 6:54 PM
matmarex removed a subscriber: Sargoth.
Dzahn added a subscriber: Dzahn.Jan 26 2016, 6:55 PM

The remaining issues are because a tagged puppet run is now executed on all appservers, which will take a little while, estimate is 15 to 20 minutes. People are also working on purging cached broken redirects from the varnish layer.

Office.wikimedia.org is affected too, just for the record.

Dzahn added a comment.Jan 26 2016, 6:56 PM

Office.wikimedia.org is affected too, just for the record.

everything under .wikimedia.org is affected but not wikipedia.org or other projects.

This issue is definitely going to require incident documentation (https://wikitech.wikimedia.org/wiki/Incident_documentation). A major action item should be implementing better testing before deploying Apache configuration changes.

Izno added a subscriber: Izno.Jan 26 2016, 6:58 PM

everything under .wikimedia.org is affected but not wikipedia.org or other projects.

I actually had the problem pop up on en.wp, so that's not correct. Seems like the problem is soon-fixed anyway.

everything under .wikimedia.org is affected but not wikipedia.org or other projects.

or Phabricator. ;-)

Izno removed a subscriber: Izno.Jan 26 2016, 6:58 PM
Elitre added a subscriber: Elitre.Jan 26 2016, 6:59 PM
Dzahn added a comment.Jan 26 2016, 7:00 PM

everything under .wikimedia.org is affected but not wikipedia.org or other projects.

or Phabricator. ;-)

yes, correct. "everything that is in wikimedia.org AND in the main cluster Apache configuration and not a misc service with a separate Apache"

Didym awarded a token.Jan 26 2016, 7:00 PM
Didym added a subscriber: Didym.

This kind of outage should probably appear on http://status.wikimedia.org/ ... (unless there's another status board that I'm not aware of?)

Pine added a comment.Jan 26 2016, 7:08 PM

Update: Commons is working now, but not Meta.

This kind of outage should probably appear on http://status.wikimedia.org/ ... (unless there's another status board that I'm not aware of?)

Well, the sites are not down… they're just redirecting :P

Update: Commons is working now, but not Meta.

Neither is working for me. It depends on your geographical location and luck. The original issue is fixed now (I think), but some of the redirects got cached in various layers. Ops are working to purge them.

Pine added a comment.Jan 26 2016, 7:12 PM

Commons is down again for me.

"There is no user by the name "Vituzzu". Check your spelling." again at meta.

Commons is down again for me.

Please see T124804#1966709: It depends on your geographical location and luck. Hence: Patience, please.

matmarex renamed this task from All wikis under wikimedia.org (Meta, Commons, others) are redirecting to wikimediafoundation.org to All wikis under wikimedia.org (Meta, Commons, loginwiki, others) are redirecting to wikimediafoundation.org.Jan 26 2016, 7:15 PM
Dzahn added a comment.Jan 26 2016, 7:15 PM

This kind of outage should probably appear on http://status.wikimedia.org/ ... (unless there's another status board that I'm not aware of?)

I made T124812 for this.

RobH added a subscriber: RobH.Jan 26 2016, 7:16 PM

Operations is still working on this issue. At this time the underlying issue has been reverted, and now is working to clear our caching of the bad redirection.

KTC added a subscriber: KTC.Jan 26 2016, 7:16 PM
Sargoth removed a subscriber: Sargoth.Jan 26 2016, 7:18 PM

At 7:36 PM, for reasons operations team has not yet investigated, a wrong configuration for the Apaches was set in effect, afecting all wikis with the wikimedia.org domain, including Commons, Meta, Wikispecies, Office and many others.

At present, that configuration has already been reverted, however, due to the heavy cache usage, the problem is still present. Varnishes are being purged right now to solve the problem. I will keep you updated, but expect the issue to be solved, very gradualy and in an inconsistent way for some time.

@Mike_Peel That panel is not handled by the Operations team, AFAIK, but Communications team was communicated within 5 minutes of the problems appearing.

@Mike_Peel That panel is not handled by the Operations team, AFAIK, but Communications team was communicated within 5 minutes of the problems appearing.

The webpage showing the status of operations isn't handled by the operations team? That doesn't sound right.

I made T124812 for this.

Thanks!

Correction, it was 19:23 UTC.

Ata awarded a token.Jan 26 2016, 7:24 PM
jeblad added a subscriber: jeblad.Jan 26 2016, 7:24 PM

...and from Oslo, 10 points for well-done cleanup! :)

The webpage showing the status of operations isn't handled by the operations team? That doesn't sound right.

It is long to explain, use #wikimedia-operations topic for the latest status.

He7d3r added a subscriber: He7d3r.Jan 26 2016, 7:25 PM

"There is no user by the name "Vituzzu". Check your spelling." again at meta.

Are you sure you're seeing this error at meta.wikimedia.org and not wikimediafoundation.org?

"There is no user by the name "Vituzzu". Check your spelling." again at meta.

Are you sure you're seeing this error at meta.wikimedia.org and not wikimediafoundation.org?

You're right :|

Update: while we believe most issues have been solved now, the caching purge has finished. Followup will be on this ticket and an outage report will be generated on Wikitech when the causes will be investigated to be avoided in the future.

If issues continue for some users, make sure you refresh your browser' cache and then report it here.

jayvdb added a subscriber: jayvdb.Jan 26 2016, 7:39 PM
Harej updated the task description. (Show Details)Jan 26 2016, 7:39 PM
Harej removed a subscriber: jayvdb.
Harej added a subscriber: jayvdb.Jan 26 2016, 7:40 PM
cscott added a subscriber: cscott.Jan 26 2016, 7:40 PM

Followup will be on this ticket and an outage report will be generated on Wikitech when the causes will be investigated to be avoided in the future.

https://wikitech.wikimedia.org/wiki/Incident_documentation/20160126-WikimediaDomainRedirection

I think this task can be marked resolved.

RobH closed this task as Resolved.Jan 26 2016, 10:10 PM
RobH claimed this task.

resolving as I've sent the outage notification to the ops list and logged it on wikitech as @MZMcBride points out.

TheDJ added a subscriber: TheDJ.Jan 26 2016, 11:25 PM

The webpage showing the status of operations isn't handled by the operations team? That doesn't sound right.

It is long to explain, use #wikimedia-operations topic for the latest status.

Eh, this also has me flabbergasted a bit. Is there a ticket for this mind fu*k ?

Ricordisamoa added a subscriber: Ricordisamoa.
Ladsgroup added a subscriber: Ladsgroup.

The webpage showing the status of operations isn't handled by the operations team?

Eh, this also has me flabbergasted a bit. Is there a ticket for this mind fu*k ?

Not that I knew. Please feel free to create one.

Krenair renamed this task from All wikis under wikimedia.org (Meta, Commons, loginwiki, others) are redirecting to wikimediafoundation.org to All wikis under wikimedia.org (Meta, Commons, loginwiki, others) are redirecting to wikimedia.org.Jan 27 2016, 7:57 PM
Krenair renamed this task from All wikis under wikimedia.org (Meta, Commons, loginwiki, others) are redirecting to wikimedia.org to All wikis under wikimedia.org (Meta, Commons, loginwiki, others) are using wikimedia.org VHost, including /wiki/ -> wikimediafoundation.org redirect.Jan 27 2016, 8:03 PM
Dzahn added a comment.Jan 27 2016, 8:48 PM

status.wikimedia.org is the product that used to be called "watchmouse" and is nowadays the "Nimsoft Cloud User Experience Monitor "

https://wikimediafoundation.org/wiki/Press_releases/February_2011_Wikimedia_selects_Watchmouse_for_monitoring_services

https://wikitech.wikimedia.org/wiki/Monitoring_package_survey#Nimsoft_Cloud_Monitor_.28formerly_Watchmouse.29

http://www.nimserve.com/CA-Nimsoft-Cloud-Monitor.asp

status.wikimedia.org is an alias for status.watchmouse.com.
status.watchmouse.com is an alias for dualstack.lb-1710199131.us-east-1.elb.amazonaws.com.

Change 266908 had a related patch set uploaded (by Mattflaschen):
Don't accidentally parse topic title on log pages

https://gerrit.wikimedia.org/r/266908

Whoops, sorry, I copied the wrong bug number.

Pine added a comment.Feb 1 2016, 7:25 PM

Is there an incident report for this that we can see, or a link to the place where it will be posted when it's published?

Pine added a comment.Feb 1 2016, 7:31 PM

Thank you!