The remaining issues are because a tagged puppet run is now executed on all appservers, which will take a little while, estimate is 15 to 20 minutes. People are also working on purging cached broken redirects from the varnish layer.

Office.wikimedia.org is affected too, just for the record.

In T124804#1966593, @Tbayer wrote:

Office.wikimedia.org is affected too, just for the record.

everything under .wikimedia.org is affected but not wikipedia.org or other projects.

This issue is definitely going to require incident documentation (https://wikitech.wikimedia.org/wiki/Incident_documentation). A major action item should be implementing better testing before deploying Apache configuration changes.

In T124804#1966597, @Dzahn wrote:

everything under .wikimedia.org is affected but not wikipedia.org or other projects.

I actually had the problem pop up on en.wp, so that's not correct. Seems like the problem is soon-fixed anyway.

In T124804#1966597, @Dzahn wrote:

everything under .wikimedia.org is affected but not wikipedia.org or other projects.

or Phabricator. ;-)

In T124804#1966618, @Mike_Peel wrote:

In T124804#1966597, @Dzahn wrote:

everything under .wikimedia.org is affected but not wikipedia.org or other projects.

or Phabricator. ;-)

yes, correct. "everything that is in wikimedia.org AND in the main cluster Apache configuration and not a misc service with a separate Apache"

This kind of outage should probably appear on http://status.wikimedia.org/ ... (unless there's another status board that I'm not aware of?)

@Mike_Peel agreed.

Update: Commons is working now, but not Meta.

In T124804#1966668, @Mike_Peel wrote:

This kind of outage should probably appear on http://status.wikimedia.org/ ... (unless there's another status board that I'm not aware of?)

Well, the sites are not down… they're just redirecting :P

In T124804#1966694, @Pine wrote:

Update: Commons is working now, but not Meta.

Neither is working for me. It depends on your geographical location and luck. The original issue is fixed now (I think), but some of the redirects got cached in various layers. Ops are working to purge them.

Commons is down again for me.

"There is no user by the name "Vituzzu". Check your spelling." again at meta.

In T124804#1966713, @Pine wrote:

Commons is down again for me.

Please see T124804#1966709: It depends on your geographical location and luck. Hence: Patience, please.

In T124804#1966668, @Mike_Peel wrote:

This kind of outage should probably appear on http://status.wikimedia.org/ ... (unless there's another status board that I'm not aware of?)

I made T124812 for this.

Operations is still working on this issue. At this time the underlying issue has been reverted, and now is working to clear our caching of the bad redirection.

@Dzahn @RobH thank you.

At 7:36 PM, for reasons operations team has not yet investigated, a wrong configuration for the Apaches was set in effect, afecting all wikis with the wikimedia.org domain, including Commons, Meta, Wikispecies, Office and many others.

At present, that configuration has already been reverted, however, due to the heavy cache usage, the problem is still present. Varnishes are being purged right now to solve the problem. I will keep you updated, but expect the issue to be solved, very gradualy and in an inconsistent way for some time.

@Mike_Peel That panel is not handled by the Operations team, AFAIK, but Communications team was communicated within 5 minutes of the problems appearing.

In T124804#1966757, @jcrespo wrote:

@Mike_Peel That panel is not handled by the Operations team, AFAIK, but Communications team was communicated within 5 minutes of the problems appearing.

The webpage showing the status of operations isn't handled by the operations team? That doesn't sound right.

In T124804#1966738, @Dzahn wrote:

I made T124812 for this.

Thanks!

Correction, it was 19:23 UTC.

...and from Oslo, 10 points for well-done cleanup! :)

The webpage showing the status of operations isn't handled by the operations team? That doesn't sound right.

It is long to explain, use #wikimedia-operations topic for the latest status.

In T124804#1966722, @Vituzzu wrote:

"There is no user by the name "Vituzzu". Check your spelling." again at meta.

Are you sure you're seeing this error at meta.wikimedia.org and not wikimediafoundation.org?

In T124804#1966833, @MZMcBride wrote:

In T124804#1966722, @Vituzzu wrote:

"There is no user by the name "Vituzzu". Check your spelling." again at meta.

Are you sure you're seeing this error at meta.wikimedia.org and not wikimediafoundation.org?

You're right :|

Update: while we believe most issues have been solved now, the caching purge has finished. Followup will be on this ticket and an outage report will be generated on Wikitech when the causes will be investigated to be avoided in the future.

If issues continue for some users, make sure you refresh your browser' cache and then report it here.

In T124804#1966836, @jcrespo wrote:

Followup will be on this ticket and an outage report will be generated on Wikitech when the causes will be investigated to be avoided in the future.

https://wikitech.wikimedia.org/wiki/Incident_documentation/20160126-WikimediaDomainRedirection

I think this task can be marked resolved.

In T124804#1966782, @jcrespo wrote:

The webpage showing the status of operations isn't handled by the operations team? That doesn't sound right.

It is long to explain, use #wikimedia-operations topic for the latest status.

Eh, this also has me flabbergasted a bit. Is there a ticket for this mind fu*k ?

In T124804#1968040, @TheDJ wrote:

The webpage showing the status of operations isn't handled by the operations team?

Eh, this also has me flabbergasted a bit. Is there a ticket for this mind fu*k ?

Not that I knew. Please feel free to create one.

status.wikimedia.org is the product that used to be called "watchmouse" and is nowadays the "Nimsoft Cloud User Experience Monitor "

https://wikimediafoundation.org/wiki/Press_releases/February_2011_Wikimedia_selects_Watchmouse_for_monitoring_services

https://wikitech.wikimedia.org/wiki/Monitoring_package_survey#Nimsoft_Cloud_Monitor_.28formerly_Watchmouse.29

http://www.nimserve.com/CA-Nimsoft-Cloud-Monitor.asp

status.wikimedia.org is an alias for status.watchmouse.com.
status.watchmouse.com is an alias for dualstack.lb-1710199131.us-east-1.elb.amazonaws.com.

Change 266908 had a related patch set uploaded (by Mattflaschen):
Don't accidentally parse topic title on log pages

https://gerrit.wikimedia.org/r/266908

Whoops, sorry, I copied the wrong bug number.

Is there an incident report for this that we can see, or a link to the place where it will be posted when it's published?

Yes, was linked above: https://wikitech.wikimedia.org/wiki/Incident_documentation/20160126-WikimediaDomainRedirection

In T124804#1967560, @MZMcBride wrote:

https://wikitech.wikimedia.org/wiki/Incident_documentation/20160126-WikimediaDomainRedirection

Thank you!