In T90115#1055938, @csteipp wrote:

@Manybubbles, what the timeline around this?

Still unclear. We haven't chosen it as the backend, but assuming that we do it'll be at least two months before we have it in production.

Its a Java server application similar to Elasticsearch but in this case we'd like to be able to expose its query language with very little in the way. The tentative plan is to write a plugin that uses their query optimizer facilities to catch dangerous queries and rewrite or reject them. Its against the abstract syntax tree of the query rather than text so its going to be quite accurate.

We should also look into Apache River from a security perspective. Apache River is an important component of BlazeGraph's HA mechanism.

Disconnected from the decision finalization - the decision has been finalized. We did some initial security investigations and it'll need work but nothing that is beyond us.

It's a little concerning that systap doesn't have any security updates for blazegraph that I can find. Their engineers must write flawless code ;)

Can we schedule some hangout time to talk through how your planning to set this up, and where we need to have different controls?

In T90115#1132377, @csteipp wrote:

It's a little concerning that systap doesn't have any security updates for blazegraph that I can find. Their engineers must write flawless code ;)

You mean like dot releases that mention that they fix security bugs? I see 1.4 didn't have any dot releases but 1.3 and below had a few. One thing that makes this weird is that it used to be called BigData, which is totally un-google-able.

Can we schedule some hangout time to talk through how your planning to set this up, and where we need to have different controls?

Invite sent.

Talked with Nik today about running this. We're planning to expose sending raw queries into our cluster.

The biggest threats are a malicious users causes data corruption or resource consumption DoS, or an attacker is able to compromise the Blazegraph server and pivot to the rest of our cluster. The data in Blazegraph is all public (assuming we work out removing deleted/suppressed items), so authorization within Blazegraph isn't a big concern.

Mitigating those threats:

• We want to make sure we are aware of security patches to Blazegraph, and ops applies those in an appropriate timeframe. @Beebs.systap, is there a special mailing list we need to be on to get notified? I haven't seen any CVE's issued for Blazegraph, so I want to make sure we're watching the right places.
• Since we know that we're running a more risky environment than most Blazegraph users, it would be nice if we could ensure that if it's compromised, the attacker can't start attacking the cluster. @Joe, I know ops isn't too fond of creating many new subnets for our services, but since we're starting from scratch, is this a case where we can put the boxes on a dedicated subnet and make sure the other mediawiki infrastructure isn't directly routable from there?
• In blazegraph, @Manybubbles is looking into what options need to be disabled to prevent queries from,
  • modify existing data
  • opening external or internal resources (it sounds like there might be capabilities to cause Blazegraph to query an external db, or load local files)
• At the application (proxy?) layer, we'll setup some per-ip/user throttles, and make sure we set appropriate timeouts
• We'll make sure revision deletion is working correctly so we don't leak suppressed items

it sounds like there might be capabilities to cause Blazegraph to query an external db, or load local files

There's definitely a LOAD command that loads local files.

We'll make sure revision deletion is working correctly so we don't leak suppressed items

We will delete the item data if we get delete update (TBD as all updates now) so I don't think we'd need any special handling for any reason why it's deleted. We have no plans for storing revisions, so if it gone in latest revision, it's gone.

In T90115#1143393, @csteipp wrote:

Talked with Nik today about running this. We're planning to expose sending raw queries into our cluster.

The biggest threats are a malicious users causes data corruption or resource consumption DoS, or an attacker is able to compromise the Blazegraph server and pivot to the rest of our cluster. The data in Blazegraph is all public (assuming we work out removing deleted/suppressed items), so authorization within Blazegraph isn't a big concern.

Mitigating those threats:

• We want to make sure we are aware of security patches to Blazegraph, and ops applies those in an appropriate timeframe. @Beebs.systap, is there a special mailing list we need to be on to get notified? I haven't seen any CVE's issued for Blazegraph, so I want to make sure we're watching the right places.

BB: We have not had a CVE issued directly against the Blazegraph (or predecessor) software. We have had a number of patches applied to versions of the libraries that had CVE on the included library. The commons-fileupload in release 1.3.2 was the most recent example. See http://trac.bigdata.com/ticket/1010. We distribute these via the users mailing list and our internally maintained mailing lists.

• Since we know that we're running a more risky environment than most Blazegraph users, it would be nice if we could ensure that if it's compromised, the attacker can't start attacking the cluster. @Joe, I know ops isn't too fond of creating many new subnets for our services, but since we're starting from scratch, is this a case where we can put the boxes on a dedicated subnet and make sure the other mediawiki infrastructure isn't directly routable from there?
• In blazegraph, @Manybubbles is looking into what options need to be disabled to prevent queries from,
  • modify existing data
  • opening external or internal resources (it sounds like there might be capabilities to cause Blazegraph to query an external db, or load local files)
• At the application (proxy?) layer, we'll setup some per-ip/user throttles, and make sure we set appropriate timeouts
• We'll make sure revision deletion is working correctly so we don't leak suppressed items

At the application (proxy?) layer, we'll setup some per-ip/user throttles, and make sure we set appropriate timeouts

We should probably not expose anything that takes longer than a minute to compute. Ideally, we'd limit it further to 20-30 seconds. Which is still a long time.

Individual queries should not be allowed to take up an amount of resources that would affect other user's response times significantly.

is this a case where we can put the boxes on a dedicated subnet and make sure the other mediawiki infrastructure isn't directly routable from there?

A separate VLAN would be great.

Memory may be a bigger issue than time. I've experienced some queries that OOM the VM and bring the whole server to unstable state.

The analytic query mode does offer some ability to bound memory but it is not 100% across all queries. For example, quads mode queries do not currently put the distinct triple pattern filter on the native heap. ORDER BY is not currently on the native heap, property paths are not currently on the native heap (though they now support incremental eviction and are executed much more efficiently). Also the native DISTINCT solutions operator can not be used with queries that must preserve order (e.g., where LIMIT or ORDER BY are in use). Finally, we do not buffer the input queues for the operators on the native heap in the analytic query mode. These things all limit our ability to strictly bound native memory usage.

You probably also want to place a limit on the #of solutions emerging from an unconstrained join (full Cartesian cross product). The ability to do this is built into a least the solution set hash join operators.

It sounds like we should define an interface with a variety of query complexity and similar security options and then expose a means to configure that interface so we can coordinate the enabling and disabling of relevant features as well as collocate the set of features that could be of concern. I suggest creating a ticket at trac.bigdata.com for this and cross linking it with your issue tracker. We can then work to define the appropriate interface and ensure that the desired restrictions can be enforced.

The HTTP interface can be made read-only via a web.xml configuration. See SparqlEndpointConfig and web.xml. However, this does not restrict your ability to have a plugin for the database that listens to the update feed and applies mutations. Thus you can make it read-only to the world but still get updates.

if (getConfig(servletContext).readOnly) {
    
    buildAndCommitResponse(resp, HTTP_METHOD_NOT_ALLOWED, MIME_TEXT_PLAIN,
            "Not writable.");

    // Not writable.  Response has been committed.
    return false;
    
}

Another possibility is using CAS counters (striped atomic counters) to track resources associated with a query and use that to bound memory for queries running on the java managed heap (in addition to bounding the memory associated with the native heap, input queue capacity, etc.).

So we can close this ticket, can we confirm / finish the following?

• @Beebs.systap, is there a special mailing list we need to be on to get notified of security issues? Is someone from Ops subscribed?
• @Joe, can put the boxes on a dedicated subnet and make sure the other mediawiki infrastructure isn't directly routable from there?
• Someone needs to check that revision deletion is working correctly.
• @Manybubbles, I'm assuming you worked out all of the settings we need to set, and (maybe) implemented throttles. Can you confirm?

If any of that's already happened, let me know. Thanks!

Query runtime caps are in T97095. Not sure what is the status with memory and if runaway query can break that. Separating read from write queries should also be easy - all reads go through GET, all writes go through POST, so if we limit POST access via some kind of proxy, we should be fine.

Since we do not save any revision but the last one, there should not be problems with deleted revisions - as soon as the change happens, the data is updated to match the current state of affairs, and the old state is forgotten.

In T90115#1259141, @Smalyshev wrote:

Since we do not save any revision but the last one, there should not be problems with deleted revisions - as soon as the change happens, the data is updated to match the current state of affairs, and the old state is forgotten.

Cool. Do you store metadata like username / edit summary?

@csteipp: no, the only thing stored in last revision ID and time.
You can see example here: https://www.wikidata.org/wiki/Special:EntityData/Q1.ttl?flavor=dump

data:Q1 schema:version "214221701"^^xsd:integer ;
	schema:dateModified "2015-05-04T05:54:17Z"^^xsd:dateTime ;

This is the metadata that is stored for entity Q1. The rest is current entity data.

In T90115#1259101, @csteipp wrote:

So we can close this ticket, can we confirm / finish the following?

• @Beebs.systap, is there a special mailing list we need to be on to get notified of security issues? Is someone from Ops subscribed?

Is there an OPS email alias that should be added? We generally do announce to the developers list, but do push out specific notices directly in some cases.

If any of that's already happened, let me know. Thanks!

In T90115#1259754, @Beebs.systap wrote:

• @Beebs.systap, is there a special mailing list we need to be on to get notified of security issues? Is someone from Ops subscribed?

Is there an OPS email alias that should be added? We generally do announce to the developers list, but do push out specific notices directly in some cases.

@MoritzMuehlenhoff does ops have an email address for upgrade notifications?

@Beebs.systap: For the Blazegraph security announcements, please add moritz@wikimedia.org to the distribution list if you push out security releases etc, there's no specific ops-security mailing at this point (and the general purpose ops lists reach quite a few people).

@csteipp: I still have it on my TODO list to reorg the security groups in Phabricator (and associated mail exploders) a bit, I'll send you a separate mail later.

Only thing left here is,

Since we know that we're running a more risky environment than most Blazegraph users, it would be nice if we could ensure that if it's compromised, the attacker can't start attacking the cluster. @Joe, I know ops isn't too fond of creating many new subnets for our services, but since we're starting from scratch, is this a case where we can put the boxes on a dedicated subnet and make sure the other mediawiki infrastructure isn't directly routable from there?

@Joe / @MoritzMuehlenhoff, is this possible? Or are you comfortable with the setup we have now?

In T90115#1143411, @Smalyshev wrote:

it sounds like there might be capabilities to cause Blazegraph to query an external db, or load local files

There's definitely a LOAD command that loads local files.

We'll make sure revision deletion is working correctly so we don't leak suppressed items

We will delete the item data if we get delete update (TBD as all updates now) so I don't think we'd need any special handling for any reason why it's deleted. We have no plans for storing revisions, so if it gone in latest revision, it's gone.

What about oversight-deleted items? I recently stumbled about a property which got deleted on Wikidata on June 3rd but still was referenced in the SPARQL query service. query and link to property Oversight-deleted items are just gone completely, there is no way to see that they got removed.

@Bene All deleted items are deleted from the database too, unless there has been some issue with the update (which is possible of course). However, the update is guided by recent changes, so if oversight-delete does not create recent changes entry, we will need some other way to deliver the news to the DB - otherwise there is no way for WDQS to know something changed. There is a way to update the entity outside of the recent changes stream, but it still needs to be initiated somehow.

In T90115#1413173, @Bene wrote:

What about oversight-deleted items? I recently stumbled about a property which got deleted on Wikidata on June 3rd but still was referenced in the SPARQL query service. query and [[ link to property | https://www.wikidata.org/wiki/Property:P1926 ]] Oversight-deleted items are just gone completely, there is no way to see that they got removed.

That one is not oversight-deleted, just normally deleted so I created T104355 for it.

In our discussion yesterday it was suggested to instruct the people who oversight to never do oversight-deletes but instead add an empty revision and then normally delete. The only reason to oversight-delete is to to remove the title, which for Wikibase entities is an incremented ID. If someone manages again to convince courts to require us to delete such a number we can still give them some butterflies ( https://xkcd.com/378/ ) or reload all our WDQS instances.

Another suggestion was to change Mediawiki to publish a log with the page-ids that were oversight-deleted and store the page-ids to know what to delete. (Other projects might need this or a similar solution. Please don't create solutions that only work with special access.)

@csteipp: A Discovery Q1 goal is to deploy WDQS on production hardware in a test mode. Should we use this phabricator ticket to track that, or would a different/another ticket be better (since it includes code beyond just blazegraph itself)?

@ksmith, there should be a separate task for that, depending on this. Any other code to be deployed should have a separate security review request.

@Joe / @MoritzMuehlenhoff, ping again on this-- are you guys comfortable that we can detect/contain Blazegraph if it gets exploited?

In T90115#1512306, @csteipp wrote:

@Joe / @MoritzMuehlenhoff, ping again on this-- are you guys comfortable that we can detect/contain Blazegraph if it gets exploited?

Various countermeasures are mentioned, have they been implemented for the planned test deployment?

• Limit memory usage of queries
• Limit maximum duration of queries
• Parsing the AST of the query to filter malicious queries

It also seems useful to limit the number of parallel queries per IP.

We'll be able to detect attacks against availability by our usual monitoring (availability would be my biggest concern for this scenario, I doubt there are many/other installations with a setup like this)

We can implement a number of countermeasures to contain an attacker leveraging code injection inside BlazeGraph or through the JVM with the privileges of the BlazeGraph process.

We need to make sure that Linux kernel updates on the BlazeGraph servers are possible without major impact/downtime to be able to quickly deploy fixes for Linux vulnerabilitiies allowing privilege escalation.

@MoritzMuehlenhoff kernel upgrades should be doable relatively easily. If we want to be stricter about what the blazegraph user can do on the system, we can look into it too. I don't think network segregation is what we need here either.

Alas, I don't see a general way of protecting us against malicious users creating DDOS-like queries, if not using a query killer (a thing we already sort-of do if the query takes more than N seconds, if I got it right).

Limit memory usage of queries

yes

Limit maximum duration of queries

yes

Parsing the AST of the query to filter malicious queries

No. We have a capability of doing AST filter (we actually have one already implementing label service) but we don't have any query filter. The main reason is we don't know which queries would be "malicious" - i.e. right now we don't have any data on what we need to filter out. In theory, we know some queries make no sense and would only cause load, but in practice, we don't have (yet) any real examples of queries that might do harm. If we identify such queries, we can probably add filtering for them.

It also seems useful to limit the number of parallel queries per IP.

I think as it gets to the servers, we don't have much info on the originating IP, as we will be talking to Varnish frontends. So it may be needed to be implemented on Varnish side? Or Varnish passes us some info on that?

We can implement a number of countermeasures to contain an attacker leveraging code injection inside BlazeGraph or through the JVM with the privileges

What kind of measures do you propose?

@MoritzMuehlenhoff (and others): Please keep in mind that our current target is a "beta" deployment, and thus does not have specific uptime requirements. We do not plan to widely publicize this service, and DoS attacks (intentional or otherwise) at this point would be acceptable. That is, assuming they only brought the WDQS down, with no effect on other systems.

I don't have any concerns/objections about setting this up, I mostly wanted to know the status of the various countermeasures mentioned in this task.

What kind of measures do you propose?

systemd supports various features to restrict running processes, e.g. for restricting filesystem access or through disallowing potentially harmful syscalls using seccomp-bpf. This doesn't need to be present in the initial deployment, but it would be good to add in a followup step.

This doesn't need to be present in the initial deployment, but it would be good to add in a followup step.

Stas, how much time is Discovery going to be dedicating to this in Q2? I was under the impression that once it's in production, Discovery was planning to move on to other projects, but are you comfortable making sure this work is scheduled in?

@csteipp that doesn't look like much work, and probably mostly won't be done by me - as such, I have little experience with seccomp-bpf, so it's best to have somebody who knows what it is to do it, especially when we're talking about security. So I don't think it would be a problem to help with this if needed with what I can, but I don't think Discovery schedule would be a problem here. Unless I'm missing something.

@csteipp: Discovery plans to deploy this in beta status, and then (based on my understanding), we plan to shift to other priorities while we wait for feedback to come in. Our level of effort after that will depend in part on that feedback.

It will be up to @Deskana to prioritize any work in Q2. Presumably if you said "this must happen", we would find a way to make it happen.

@ksmith's summary is spot on. Thanks!

So it looks like the only remaining issue is mitigating T105427, which @Smalyshev has a warning message for (and process to involve an ops person if someone accidentally does a suppressed delete). Once we're sure that is going to get added to wikidata, I feel ok closing this.

@csteipp: Discovery plans to deploy this in beta status, and then (based on my understanding), we plan to shift to other priorities while we wait for feedback to come in. Our level of effort after that will depend in part on that feedback.

It will be up to @Deskana to prioritize any work in Q2. Presumably if you said "this must happen", we would find a way to make it happen.

Thanks! I added T108410 so it doesn't get lost. But we won't make it a blocker for this.

In T90115#1514604, @Smalyshev wrote:

It also seems useful to limit the number of parallel queries per IP.

I think as it gets to the servers, we don't have much info on the originating IP, as we will be talking to Varnish frontends. So it may be needed to be implemented on Varnish side? Or Varnish passes us some info on that?

This is incorrect, the client IP is surely present at the nginx level within the X-Forwarded-For HTTP header, maybe even in another one (but I have to check).

If we have X-Forwarded-For with real IP on the nginx level, we could make rate limiting per IP via http://nginx.org/en/docs/http/ngx_http_limit_conn_module.html

I have created T108488 to track this.

Note for oversighters posted to https://www.wikidata.org/wiki/Wikidata:Oversight

In T90115#1532455, @Smalyshev wrote:

Note for oversighters posted to https://www.wikidata.org/wiki/Wikidata:Oversight

@csteipp: In light of this, can we consider the security review completed and that we have signoff?

@Deskana, waiting for the patch on T108101 to get merged

In T90115#1532552, @csteipp wrote:

@Deskana, waiting for the patch on T108101 to get merged

Thanks for the update!

@csteipp T108101 is done now. Can we resolve this?

@csteipp Thank you!