Page MenuHomePhabricator

Add user blocking in WCQS
Closed, ResolvedPublic3 Estimated Story Points

Description

As a WCQS maintainer I want to be able to block specific users so that I can have a quick way of blocking actions that threaten the stability of WCQS.

We implemented authentication for WCQS, but to better leverage it, we should at least be able to ban users, since now we can identify them. Since blocking will happen during WCQS issues in most cases, we don't really need a dynamic mechanism for this - we can restart the service if necessary.

AC:

  • There is a configurable list of banned users
  • WCQS blocks traffic from specific users

Event Timeline

Change 734411 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[wikidata/query/rdf@master] mw-oauth-proxy: Store usernames and support bans

https://gerrit.wikimedia.org/r/734411

Change 735445 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[schemas/event/secondary@master] Add user_identifier field to sparql/query

https://gerrit.wikimedia.org/r/735445

Change 735454 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[wikidata/query/rdf@master] blazegraph: Optionally include user identity in query events

https://gerrit.wikimedia.org/r/735454

With these patches in I think user blocking should be possible. The expected process will be as follows, does this seem reasonable? We could put the ban list in puppet, but since it would have to flow through wikidata/query/deploy anyways and more of us have deploy rights there it seems sensible. If this is appropriate where does it get documented?

  1. Operator identifies problematic query pattern in sparql/query event stream
  2. Operator submits a patch to wikidata/query/deploy to change runBlazegraph.sh to include: -D${OAUTH_PROPS_PREFIX}.bannedUsernamesCsv=user_one,user_two
  3. Operator deploys updated configuration to wcqs environment, which restarts instances and loads the ban list.

Change 734411 merged by jenkins-bot:

[wikidata/query/rdf@master] mw-oauth-proxy: Store usernames and support bans

https://gerrit.wikimedia.org/r/734411

Change 735454 merged by jenkins-bot:

[wikidata/query/rdf@master] blazegraph: Optionally include user identity in query events

https://gerrit.wikimedia.org/r/735454

Patches are merged, we could deploy but we can't test the deploy so I'm holding off. Before we can test the deploy we need to bring the wcqs servers back into lvs production state.

Change 735445 merged by jenkins-bot:

[schemas/event/secondary@master] Add performer field to sparql/query

https://gerrit.wikimedia.org/r/735445

Change 757526 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[wikidata/query/rdf@master] Update sparql/query event schema to 1.2.0

https://gerrit.wikimedia.org/r/757526

Change 757526 merged by jenkins-bot:

[wikidata/query/rdf@master] Update sparql/query event schema to 1.2.0

https://gerrit.wikimedia.org/r/757526

  • Currently no usernames show up in the event logs, schema version was too low. Once we run another build/deploy will need to re-check if usernames are coming through the events.
  • Tested banning and unbanning myself by providing an appropriate line to java. Works but is not particularly easy. The difficulty is that usernames have spaces in them, and we are passing this all through shell scripts. It's entirely doable but error prone with the existing runBlazegraph.sh script. Will work up some patch to make this more reasonable.

Change 757993 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[wikidata/query/rdf@master] oauth: Read banned usernames from a file

https://gerrit.wikimedia.org/r/757993

Change 757993 merged by jenkins-bot:

[wikidata/query/rdf@master] oauth: Read banned usernames from a file

https://gerrit.wikimedia.org/r/757993

Change 761072 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[wikidata/query/rdf@master] blazegraph: Simplify conf of jwt-identity-filter

https://gerrit.wikimedia.org/r/761072

Change 761075 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[operations/puppet@production] Provide jwt secret to blazegraph for logging

https://gerrit.wikimedia.org/r/761075

Change 761075 abandoned by Ebernhardson:

[operations/puppet@production] Provide jwt secret to blazegraph for logging

Reason:

Rather than defining more one-off environment vars, prefer to add jvm args to extra_jvm_args

https://gerrit.wikimedia.org/r/761075

Change 761072 merged by jenkins-bot:

[wikidata/query/rdf@master] blazegraph: Simplify conf of jwt-identity-filter

https://gerrit.wikimedia.org/r/761072

Change 765667 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[operations/puppet@production] query_service: pass cookies on to blazegraph

https://gerrit.wikimedia.org/r/765667

I manually applied the fixes in the latest patch, to pass cookies on to blazegraph, and my username came through into the request logs. These fixes are of course undone by puppet shortly after. Hoping this will be resovled once the above is merged.

Change 765667 merged by Ryan Kemper:

[operations/puppet@production] query_service: pass cookies on to blazegraph

https://gerrit.wikimedia.org/r/765667