Page MenuHomePhabricator
Create Task
Maniphest T142073

Improve user management for AQS Cassandra
Open, LowPublic5 Estimated Story Points
Actions

Description

After a chat with @JAllemandou and @Eevans we agreed that the AQS clusters should migrate to a better user management scheme. We are currently using the admin 'cassandra' user for Restbase reads and also for writes, that has multiple downsides:

  1. requires QUORUM during user authentication, not really great for performances (as opposed to local one for simple users);
  2. does not protect the system.auth table properly.

The migration procedure should be something like:

  • Set an new application_username and application_password (to create $CASSANDRA_CONF/user_aqs.cql on nodes)
  • Run cqlsh -u cassandra -f $CASSANDRA_CONF/user_aqs.cql $HOSTNAME (creates user)
  • Change restbase::cassandra_user (configuring AQS service to use the aqs user)
  • Set a new super_password for the Cassandra username (in private.git) for templating of $CASSANDRA_CONF/cqlshrc
  • Update the super password in Cassandra to match $CASSANDRA_CONF/cqlshrc
  • Set an new application_username and application_password (to create $CASSANDRA_CONF/user_aqsloader.cql on nodes)

We definitely want to do it for aqs100[456] but aqs100[123] will need extra care because live.

Details

SubjectRepoBranchLines +/-
Switch the AQS restbase use from 'cassandra' to 'aqs'operations/puppetproduction+6 -15
Change the AQS restbase user from 'cassandra' to 'aqs'operations/puppetproduction+15 -0
Move the include of cassandra/aqs passwords up to solve a priority issueoperations/puppetproduction+2 -2
Include the password::aqs namespace in the AQS roleoperations/puppetproduction+2 -0
Add the configuration needed to prepare a new AQS Cassandra user creationoperations/puppetproduction+7 -0
Add fake aqs Cassandra user's passwordlabs/privatemaster+5 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Restricted Task
 OpenNoneT142073 Improve user management for AQS Cassandra

Event Timeline

elukey created this task.Aug 4 2016, 8:05 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 4 2016, 8:05 AM
Nuria changed the point value for this task from 0 to 5.Aug 4 2016, 5:19 PM
elukey moved this task from Next Up to In Progress on the Analytics-Kanban board.Aug 9 2016, 7:36 AM
elukey added a comment.Aug 9 2016, 9:15 AM

Interesting reading: https://issues.apache.org/jira/browse/CASSANDRA-5310

QUORUM is only used for default superuser ('cassandra'), for other users ONE is used. You are not supposed to use 'cassandra' user directly, except to create another superuser and use that one from that point on.

This is probably why the password caching increase on aqs100[123] gave so much performance improvements.

gerritbot added a subscriber: gerritbot.Aug 9 2016, 9:25 AM

Change 303772 had a related patch set uploaded (by Elukey):
Add fake aqs Cassandra user's password

https://gerrit.wikimedia.org/r/303772

gerritbot added a comment.Aug 9 2016, 9:26 AM

Change 303772 merged by Elukey:
Add fake aqs Cassandra user's password

https://gerrit.wikimedia.org/r/303772

gerritbot added a comment.Aug 9 2016, 9:32 AM

Change 303774 had a related patch set uploaded (by Elukey):
Add the configuration needed to prepare a new AQS Cassandra user creation

https://gerrit.wikimedia.org/r/303774

elukey mentioned this in rOPUPdce7f94133bc: Add the configuration needed to prepare a new AQS Cassandra user creation.Aug 9 2016, 9:38 AM
elukey mentioned this in rOPUPa8001e916994: Add the configuration needed to prepare a new AQS Cassandra user creation.Aug 9 2016, 9:59 AM
gerritbot added a comment.Aug 9 2016, 10:00 AM

Change 303774 merged by Elukey:
Add the configuration needed to prepare a new AQS Cassandra user creation

https://gerrit.wikimedia.org/r/303774

elukey mentioned this in rOPUPddb1056b535d: Add the configuration needed to prepare a new AQS Cassandra user creation.Aug 9 2016, 10:04 AM
gerritbot added a comment.Aug 9 2016, 10:31 AM

Change 303783 had a related patch set uploaded (by Elukey):
Include the password::aqs namespace in the AQS role

https://gerrit.wikimedia.org/r/303783

gerritbot added a comment.Aug 9 2016, 10:34 AM

Change 303783 merged by Elukey:
Include the password::aqs namespace in the AQS role

https://gerrit.wikimedia.org/r/303783

elukey mentioned this in rOPUP03a4928294fd: Include the password::aqs namespace in the AQS role.Aug 9 2016, 10:35 AM
gerritbot added a comment.Aug 9 2016, 11:13 AM

Change 303786 had a related patch set uploaded (by Elukey):
Move the include of cassandra/aqs passwords up to solve a priority issue

https://gerrit.wikimedia.org/r/303786

gerritbot added a comment.Aug 9 2016, 11:16 AM

Change 303786 merged by Elukey:
Move the include of cassandra/aqs passwords up to solve a priority issue

https://gerrit.wikimedia.org/r/303786

elukey mentioned this in rOPUP0b82a6e2571e: Move the include of cassandra/aqs passwords up to solve a priority issue.Aug 9 2016, 11:16 AM
elukey added a comment.Aug 9 2016, 12:10 PM

aqs user added to aqs100[456] and verified that it returns data on each instance with the following query:

elukey@aqs1004:~$ cat showdata.cql
select project from "local_group_default_T_pageviews_per_article_flat".data limit 10;
elukey@aqs1004:~$ cqlsh -u aqs -f showdata.cql aqs1004-a.eqiad.wmnet
gerritbot added a comment.Aug 9 2016, 12:19 PM

Change 303792 had a related patch set uploaded (by Elukey):
Change the AQS restbase user from 'cassandra' to 'aqs'

https://gerrit.wikimedia.org/r/303792

elukey mentioned this in rOPUPbff852462f70: Change the AQS restbase user from 'cassandra' to 'aqs'.Aug 9 2016, 12:23 PM
elukey mentioned this in rOPUPb9fa5a84d597: Change the AQS restbase user from 'cassandra' to 'aqs'.Aug 9 2016, 12:28 PM
gerritbot added a comment.Aug 9 2016, 12:34 PM

Change 303792 merged by Elukey:
Change the AQS restbase user from 'cassandra' to 'aqs'

https://gerrit.wikimedia.org/r/303792

elukey added a comment.Aug 9 2016, 1:15 PM

New cluster switched, installed the new user in the current one (aqs100[123]) and tested:

elukey@aqs1001:~$ cqlsh -u aqs -f showdata.cql aqs1001.eqiad.wmnet
Password:

 project
---------------
 en.wikisource
  ja.wikipedia
  hu.wikipedia
  fr.wikipedia
  fr.wikipedia
  fr.wikipedia
  fr.wikipedia
 bg.wiktionary
  en.wikipedia
  en.wikipedia

(10 rows)
elukey@aqs1001:~$ cqlsh -u aqs -f showdata.cql aqs1002.eqiad.wmnet
Password:

 project
---------------
 en.wikisource
  ja.wikipedia
  hu.wikipedia
  fr.wikipedia
  fr.wikipedia
  fr.wikipedia
  fr.wikipedia
 bg.wiktionary
  en.wikipedia
  en.wikipedia

(10 rows)
elukey@aqs1001:~$ cqlsh -u aqs -f showdata.cql aqs1003.eqiad.wmnet
Password:

 project
---------------
 en.wikisource
  ja.wikipedia
  hu.wikipedia
  fr.wikipedia
  fr.wikipedia
  fr.wikipedia
  fr.wikipedia
 bg.wiktionary
  en.wikipedia
  en.wikipedia

(10 rows)
gerritbot added a comment.Aug 9 2016, 1:19 PM

Change 303798 had a related patch set uploaded (by Elukey):
Switch the AQS restbase use from 'cassandra' to aqs

https://gerrit.wikimedia.org/r/303798

elukey mentioned this in rOPUPbbf442ddcc69: Switch the AQS restbase use from 'cassandra' to aqs.Aug 9 2016, 1:23 PM
elukey mentioned this in rOPUPe32b73bdc0ae: Switch the AQS restbase use from 'cassandra' to 'aqs'.Aug 9 2016, 1:28 PM
elukey mentioned this in rOPUP6a10793fbe1f: Switch the AQS restbase use from 'cassandra' to 'aqs'.Aug 9 2016, 2:14 PM
Stashbot added a subscriber: Stashbot.Aug 9 2016, 3:59 PM

Mentioned in SAL [2016-08-09T15:59:50Z] <elukey> switching restbase/cassandra user on aqs100[123] to aqs (T142073) - https://gerrit.wikimedia.org/r/303798 will be applied to one node at the time with depool/pool

gerritbot added a comment.Aug 9 2016, 4:01 PM

Change 303798 merged by Elukey:
Switch the AQS restbase use from 'cassandra' to 'aqs'

https://gerrit.wikimedia.org/r/303798

elukey mentioned this in rOPUP8b5e29dbaba1: Switch the AQS restbase use from 'cassandra' to 'aqs'.Aug 9 2016, 4:05 PM
elukey added a comment.Aug 9 2016, 4:36 PM

Remaining steps:

  1. establish how to distribute the new user/password credentials to oozie;
  2. move oozie away from the 'cassandra' user, either using the newly created 'aqs' user or creating a new one;
  3. replace the current cassandra admin password with another one.
elukey moved this task from In Progress to Paused on the Analytics-Kanban board.Aug 29 2016, 3:02 PM
elukey mentioned this in T124314: Better response times on AQS (Pageview API mostly) {melc}.Sep 6 2016, 8:23 AM
Milimetric edited projects, added Analytics; removed Analytics-Kanban.Sep 15 2016, 3:46 PM
Milimetric moved this task from Incoming to Backlog (Later) on the Analytics board.
Nuria added a project: Pageviews-API.Oct 10 2016, 4:00 PM
Milimetric removed a parent task: T124314: Better response times on AQS (Pageview API mostly) {melc}.Nov 7 2016, 4:57 PM
Nuria moved this task from Backlog (Later) to Wikistats on the Analytics board.Nov 21 2016, 4:49 PM
elukey added a project: User-Elukey.Dec 14 2016, 11:04 AM
elukey moved this task from Backlog to Analytics Backlog on the User-Elukey board.Dec 14 2016, 5:43 PM
Nuria moved this task from Wikistats to Dashiki on the Analytics board.May 29 2017, 3:57 PM
Nuria moved this task from Dashiki to Backlog (Later) on the Analytics board.Jul 6 2017, 4:48 PM
elukey moved this task from Analytics Backlog to Backlog on the User-Elukey board.Aug 4 2017, 3:10 PM
elukey moved this task from Backlog to Analytics Backlog on the User-Elukey board.Aug 9 2017, 10:35 AM
• fdans moved this task from Backlog (Later) to Wikistats on the Analytics board.Oct 26 2017, 4:17 PM
elukey moved this task from Analytics Backlog to Backlog on the User-Elukey board.Feb 16 2018, 12:01 PM
• fdans moved this task from Wikistats to Operational Excellence on the Analytics board.Mar 8 2018, 6:31 PM
elukey moved this task from Backlog to Analytics Backlog on the User-Elukey board.Mar 23 2018, 3:55 PM
mforns lowered the priority of this task from Medium to Low.Apr 16 2018, 4:25 PM
elukey added a parent task: Restricted Task.Jul 25 2018, 8:58 AM
• chasemp added a subscriber: • chasemp.Aug 29 2018, 1:44 PM
Milimetric renamed this task from Improve user management for AQS to Improve user management for AQS Cassandra.Oct 22 2018, 3:43 PM
elukey moved this task from Analytics Backlog to Backlog on the User-Elukey board.Dec 7 2018, 2:53 PM
elukey removed elukey as the assignee of this task.May 29 2020, 6:18 AM
odimitrijevic added a project: Data-Engineering.Jan 6 2022, 1:21 AM
odimitrijevic moved this task from Incoming (new tickets) to Security & Governance on the Data-Engineering board.
odimitrijevic removed a project: Analytics.Jan 11 2022, 11:22 PM
Restricted Application added a project: Analytics. · View Herald TranscriptJan 11 2022, 11:22 PM
odimitrijevic added a subscriber: odimitrijevic.Jun 6 2022, 5:46 PM

@Eevans Is this request still relevant given the latest AQS plans?

Eevans added a comment.Jun 6 2022, 6:18 PM
In T142073#7983633, @odimitrijevic wrote:

@Eevans Is this request still relevant given the latest AQS plans?

It is, yeah.

It looks like it is partially complete. RESTBase (aka "The AQS Service") is using its own (non-privileged) dedicated user now, but the superuser password still needs to be changed. Additionally, what was originally done with adduser.cql mentioned above, now has a new convention, and the aqsloader user needs to avail itself of that as well (that user existed when this ticket was created; I think that was just an oversight).

I will update the ticket description to reflect this, and put this ticket on the Cassandra board so it isn't forgotten about.

Eevans updated the task description. (Show Details)Jun 6 2022, 6:27 PM
Eevans added a project: Cassandra.
JArguello-WMF removed a project: Analytics.Jun 8 2022, 11:22 PM
JArguello-WMF moved this task from Security & Governance to Incoming (new tickets) on the Data-Engineering board.Jun 29 2023, 11:44 PM
JArguello-WMF moved this task from Incoming (new tickets) to Radar (External Teams) on the Data-Engineering board.Jun 29 2023, 11:51 PM
BTullis added a project: Data-Platform-SRE.Jul 14 2023, 11:28 PM
lbowmaker moved this task from Radar (External Teams) to Icebox (not considered in current quarter) on the Data-Engineering board.Nov 10 2023, 1:23 PM
Gehel moved this task from Incoming to Toil / Automation on the Data-Platform-SRE board.Wed, Dec 6, 1:52 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL