Page MenuHomePhabricator
Create Task
Maniphest T356400

User aqsloader hasn't MODIFY permissions on image_suggestions.* Cassandra tables anymore
Closed, ResolvedPublic
Actions

Description

T355917: Create a puppet template for aqsloader Cassandra user led to https://gerrit.wikimedia.org/r/c/operations/puppet/+/993102, which doesn't include the image_suggestions keyspace.
As a result, the Airflow DAG tasks that populate those tables are failing with messages like User aqsloader has no MODIFY permission on <table image_suggestions.title_cache> or any of its parents, which break the production system.
@Eevans, since this is rather urgent, can you please re-grant those permissions as a hotfix? We can then discuss about switching to the image_suggestions user as the one responsible for populating those tables.

See also Slack thread

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
cassandra: break-fix image suggestions Cassandra accessEevansoperations/puppetproduction+5 -0
Customize query in gerrit

Related Objects

Event Timeline

mfossati created this task.Feb 1 2024, 12:36 PM
Restricted Application added a subscriber: • Aklapper. · View Herald TranscriptFeb 1 2024, 12:36 PM
mfossati mentioned this in T356030: Search dag image_suggestions_weekly failed waiting for analytics_platform_eng.image_suggestions_search_index_delta/snapshot=2024-01-15.Feb 1 2024, 12:37 PM
mfossati added a project: Structured-Data-Backlog (Current Work).Feb 1 2024, 12:40 PM
mfossati moved this task from Incoming to Verify on Production on the Structured-Data-Backlog (Current Work) board.
dcausse added a project: Discovery-Search (Current work).Feb 1 2024, 1:04 PM
dcausse moved this task from Incoming to Blocked/Waiting on the Discovery-Search (Current work) board.
dcausse subscribed.Feb 1 2024, 1:17 PM

The airflow dag (search) image_suggestions_weekly has been paused today

mfossati added a comment.Feb 1 2024, 2:44 PM

Same for platform_eng's image_suggestions one.

JAllemandou subscribed.Feb 1 2024, 5:16 PM

Data engineering team has written some code for our cassandra-loading jobs to be able to read a password from a file on HDFS:
https://github.com/wikimedia/analytics-refinery-source/blob/master/refinery-spark/src/main/scala/org/wikimedia/analytics/refinery/spark/utils/WmfCassandraAuthConfFactory.scala

To make this code work, the spark job should be launched with options:

--conf spark.cassandra.auth.conf.factory=org.wikimedia.analytics.refinery.spark.utils.WmfCassandraAuthConfFactory --conf spark.cassandra.auth.username=CASSANDRA_USER_NAME --conf spark.cassandra.auth.passwordfilepath=hdfs:///PATH/TO/FILE

This code is not yet running for other prod jobs though :(

We already have instances of MySQL password being made able in files on HDFS:
https://github.com/wikimedia/operations-puppet/blob/3ada81b99516aef466f33b6816ff26f34a568a84/modules/profile/manifests/analytics/cluster/secrets.pp#L31.

This ticket feels like a great opportunity to start moving in the right direction for those cassandra passwords!

gerritbot added a comment.Feb 1 2024, 5:34 PM

Change 995064 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] cassandra: break-fix image suggestions Cassandra access

https://gerrit.wikimedia.org/r/995064

gerritbot added a project: Patch-For-Review.Feb 1 2024, 5:34 PM
gerritbot added a comment.Feb 1 2024, 5:40 PM

Change 995064 merged by Eevans:

[operations/puppet@production] cassandra: break-fix image suggestions Cassandra access

https://gerrit.wikimedia.org/r/995064

Eevans added a comment.Feb 1 2024, 5:49 PM

GRANTs (SELECT & MODIFY) for image_suggestions have been re-added.

cassandra@cqlsh> LIST ALL PERMISSIONS OF aqsloader;

 role      | username  | resource                                                    | permission
-----------+-----------+-------------------------------------------------------------+------------
 aqsloader | aqsloader |                                              <keyspace aqs> |     SELECT
 aqsloader | aqsloader |                                              <keyspace aqs> |     MODIFY
 aqsloader | aqsloader |                                <keyspace image_suggestions> |     SELECT
 aqsloader | aqsloader |                                <keyspace image_suggestions> |     MODIFY
 aqsloader | aqsloader |          <keyspace local_group_default_T_editors_bycountry> |     SELECT
 aqsloader | aqsloader |          <keyspace local_group_default_T_editors_bycountry> |     MODIFY
 aqsloader | aqsloader |  <keyspace local_group_default_T_knowledge_gap_by_category> |     SELECT
 aqsloader | aqsloader |  <keyspace local_group_default_T_knowledge_gap_by_category> |     MODIFY
 aqsloader | aqsloader | <keyspace local_group_default_T_lgc_pagecounts_per_project> |     SELECT
 aqsloader | aqsloader | <keyspace local_group_default_T_lgc_pagecounts_per_project> |     MODIFY
 aqsloader | aqsloader |      <keyspace local_group_default_T_mediarequest_per_file> |     SELECT
 aqsloader | aqsloader |      <keyspace local_group_default_T_mediarequest_per_file> |     MODIFY
 aqsloader | aqsloader |   <keyspace local_group_default_T_mediarequest_per_referer> |     SELECT
 aqsloader | aqsloader |   <keyspace local_group_default_T_mediarequest_per_referer> |     MODIFY
 aqsloader | aqsloader |     <keyspace local_group_default_T_mediarequest_top_files> |     SELECT
 aqsloader | aqsloader |     <keyspace local_group_default_T_mediarequest_top_files> |     MODIFY
 aqsloader | aqsloader | <keyspace local_group_default_T_pageviews_per_article_flat> |     SELECT
 aqsloader | aqsloader | <keyspace local_group_default_T_pageviews_per_article_flat> |     MODIFY
 aqsloader | aqsloader |   <keyspace local_group_default_T_pageviews_per_project_v2> |     SELECT
 aqsloader | aqsloader |   <keyspace local_group_default_T_pageviews_per_project_v2> |     MODIFY
 aqsloader | aqsloader |              <keyspace local_group_default_T_top_bycountry> |     SELECT
 aqsloader | aqsloader |              <keyspace local_group_default_T_top_bycountry> |     MODIFY
 aqsloader | aqsloader |              <keyspace local_group_default_T_top_pageviews> |     SELECT
 aqsloader | aqsloader |              <keyspace local_group_default_T_top_pageviews> |     MODIFY
 aqsloader | aqsloader |             <keyspace local_group_default_T_top_percountry> |     SELECT
 aqsloader | aqsloader |             <keyspace local_group_default_T_top_percountry> |     MODIFY
 aqsloader | aqsloader |             <keyspace local_group_default_T_unique_devices> |     SELECT
 aqsloader | aqsloader |             <keyspace local_group_default_T_unique_devices> |     MODIFY

(28 rows)
cassandra@cqlsh>
Eevans claimed this task.Feb 1 2024, 5:49 PM
Eevans triaged this task as High priority.
xcollazo subscribed.Feb 1 2024, 6:01 PM
Eevans mentioned this in T355917: Create a puppet template for aqsloader Cassandra user.Feb 1 2024, 8:19 PM
KStoller-WMF subscribed.Feb 1 2024, 11:55 PM
Eevans closed this task as Resolved.EditedFeb 2 2024, 4:06 PM

By way of a closing summary (and for posterity sake): The image suggestions DAG should not be using the aqsloader user. The only reason this ever worked is because that user had excessive permissions. Fixing those permissions is what created surfaced the breakage seen here. As stated above, I've restored those grants as a temporary fix, and opened T356446 to followup with the permanent solution.

mfossati awarded a token.Feb 2 2024, 4:11 PM
mfossati mentioned this in T356446: image suggestions DAG should not use aqsloader Cassandra role.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits