Page MenuHomePhabricator

September 2019 DoS attacks [Public]
Closed, ResolvedPublic

Description

From 2019-09-06 17:50 UTC to 02:40 the next day, Wikimedia websites were affected by a denial-of-service attack.

The attack targeted different datacenters at different times. The Amsterdam datacenter was the most heavily affected, so users in Europe were the most likely to experience problems.

The attack saturated some network links, and thus caused some users to see timeouts or slow service across all hosted wikis.

Throughout the attack, the SRE team were working hard to restore service. Reports from individual affected users are not required -- monitoring systems recorded the effects of the attack at all times.

Blog post: https://wikimediafoundation.org/news/2019/09/07/malicious-attack-on-wikipedia-what-we-know-and-what-were-doing/


How to: Report a connectivity issue

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Looks like attackers have stopped for now, but may be back tomorrow.

I've had a good response from NCSC they have also contacted the Dutch equivalent.

As to what we know? Attackers are pretty open about what they are doing

Nope, reflection sucks. Anyone who goes down to amplification attacks is pretty much asking for it. it's also pretty much useless now since a lot of upstreams employ good ACL and firewall rules. IOT is where it's at right now.

Not a 0day. Just some new devices we loaded off of an old PoC, but like you mentioned, it's a meme how all EU traffic is directed to a 20G AMS-IX link. Like 20G in 2019. Pretty funny.

I've contacted the UK National Cyber Security Centre about this and had a response. Is there an official contact I should use.

It looks like the perpetrator https://twitter.com/UKDrillas has moved on from wikipedia to targeting Twitch streams.

Please forward to legal@wikimedia.org if you haven't already.

Feel free to delete the comment if my trouble has nothing to do with the incident, but a hour ago Ī̲ found myself unable to hear any TCP reply from text-lb.eqiad.wikimedia.org[208.80.154.224] querying it from exactly one IP.

Replied by email.

@Habitator_terrae: Please file a separate task and be more specific in that new task what "doesn't work" means. See https://www.mediawiki.org/wiki/How_to_report_a_bug - thanks a lot! :)

@Aklapper: If I (German IP) for example want to see the Pageviews in the German Wikipedia (for example the at the German Mainpage linked https://de.wikipedia.org/wiki/Wikipedia:Hauptseite#footer-info-copyright-stats pageviews https://tools.wmflabs.org/pageviews?pages=Wikipedia:Hauptseite&project=de.wikipedia.org ) there I only see an "Page not found" error. It seems to be the same problem as this, because it is also a Wikimedia website which isn't aviable.

How certain is that the attack—if it was noticeable at all—was the root cause of the esams outage? My analysis of Grafana plots and personal experience led to following conclusions:

  1. About 17:44 some network hardware in Amsterdam failed, presumably a router. Varnish and backends stayed idle.
  2. About 18:04 a change in the dyna.wikimedia.org resolving threw Old-World users to eqiad, and the event is reflected in a strong activity surge recorded in Ashburn.
  3. About 18:24 European admins restarted the crashed device and the Foundation switched dyna.wikimedia.org back.
  4. The problematical device suffered more failures, but dyna.wikimedia.org remained at text-lb.esams.wikimedia.org.

It was definitely the attack, not a device failure. We won't generally release fine-grained details about an attack publicly, at least not this early and while threats and mitigations continue to be an ongoing concern. While attempting to investigate and mitigate various phases and variants of the attack during various windows of time yesterday, we did take various network engineering steps which shifted global traffic around between our edges, some of which can lead to the confusing analysis results above.

How certain is that the attack—if it was noticeable at all—was the root cause of the esams outage? My analysis of Grafana plots and personal experience led to following conclusions:

  1. About 17:44 some network hardware in Amsterdam failed, presumably a router. Varnish and backends stayed idle.
  2. About 18:04 a change in the dyna.wikimedia.org resolving threw Old-World users to eqiad, and the event is reflected in a strong activity surge recorded in Ashburn.
  3. About 18:24 European admins restarted the crashed device and the Foundation switched dyna.wikimedia.org back.
  4. The problematical device suffered more failures, but dyna.wikimedia.org remained at text-lb.esams.wikimedia.org.

From the attackers twitter

We've stopped hitting Wikipedia's AMS-IX link to prove authenticity.

We'll resume hitting at 22:45 BST+0

Stay tuned =)

A couple of other times they stopped and restarted. You can see spikes in server logs which correspond to their announced activity.

When attack started, I was able to access Wikipedia from time to time.
But now only possibility for me to access it is using Tor (my actual location is Ukraine).
If you banned my addresses as security measure, please unban them.

@Vort You probably need to send more information but on what wiki? What's the exact error?

@Aklapper here it is: T232254. Please hide it since it contains private data.

CDanis closed subtask Restricted Task as Resolved.Sep 7 2019, 3:06 PM
RhinosF1 closed subtask Restricted Task as Resolved.Sep 7 2019, 3:34 PM

Is there any idea on when the Wikimedia websites will act normally again? Oddly enough I only had this issue with Microsoft Edge and not with the Ecosia browser while using them at the same time, could this have a technical reason?

Is there any idea on when the Wikimedia websites will act normally again? Oddly enough I only had this issue with Microsoft Edge and not with the Ecosia browser while using them at the same time, could this have a technical reason?

Most people have had no issues for a while now

Is there any idea on when the Wikimedia websites will act normally again? Oddly enough I only had this issue with Microsoft Edge and not with the Ecosia browser while using them at the same time, could this have a technical reason?

Can you provide more information about the issues you are experiencing by following: https://wikitech.wikimedia.org/wiki/Reporting_a_connectivity_issue
Thank you!

Message received from UK National Cyber Security Centre. Is there any info we could send to them.

Hi Richard,
Are you aware of any further incidents against Wikipedia or degradation of service since the original reported incident?

Also do you know if there are any logs available regarding the incident or any IoC’s that can be sent to us?

Kind regards,

Nick U

NCSC Incident Management Team

Message received from UK National Cyber Security Centre. Is there any info we could send to them.

Hi Richard,
Are you aware of any further incidents against Wikipedia or degradation of service since the original reported incident?

Also do you know if there are any logs available regarding the incident or any IoC’s that can be sent to us?

Kind regards,

Nick U

NCSC Incident Management Team

WMF-Legal are in communication with them. You don't need to respond on their behalf. Thanks

Just forwarded some information to legal/ca@ regarding UKDrillas' next host (now the twitter fun has ended)

RhinosF1 raised the priority of this task from High to Unbreak Now!.Sep 8 2019, 11:37 PM

Attacker has confirmed on twitter he's back, ops reporting connectivity issues and multiple reports of issues on twitter

bd808 lowered the priority of this task from Unbreak Now! to High.Sep 8 2019, 11:38 PM
bd808 subscribed.

Down grading from UBN! to High. The actions that we can take are being taken.

Note we don't actually use phabricator for the actual incident response on something like this. There's no need to mess with priorities or send notifications here :)

Krinkle renamed this task from September 2019 DoS attack to September 2019 DoS attack [Public].Sep 8 2019, 11:56 PM
Krinkle added a subtask: Restricted Task.
CDanis closed subtask Restricted Task as Resolved.Sep 9 2019, 2:02 PM
Patriccck renamed this task from September 2019 DoS attack [Public] to September 2019 DoS attacks [Public].Sep 12 2019, 12:34 PM

Any chance of an Incident report?

Aklapper lowered the priority of this task from High to Low.Nov 1 2019, 9:54 AM

Any chance of an Incident report?

Rather unlikely, I'm afraid, as this touches legal territories.

Any chance of an Incident report?

Rather unlikely, I'm afraid, as this touches legal territories.

See wikitech-l, we were and have been promised something

See wikitech-l, we were and have been promised something

Do you have a link?

See wikitech-l, we were and have been promised something

Do you have a link?

Wrong mailing list but https://lists.wikimedia.org/pipermail/wikimedia-l/2019-October/093773.html is the most recent

If Heather says so then I guess that WMF Communications might publish something... However that likely will not be an "incident report" (in its technical meaning).

(Not sure why this task was moved to "Follow-up/Actionables" as I don't see any open followup tasks left here. I think this task could be closed.)

(Not sure why this task was moved to "Follow-up/Actionables" as I don't see any open followup tasks left here. I think this task could be closed.)

It probably could be, my thinking was any incident report/statement could be classed as followup

I am going to close this as resolved.
If someone feels this needs to stay open, please reopen.