Approved

Hi @dr0ptp4kt, could you help me understand what kind of access you are after (i.e. what hosts/service/commands) ? As far as I can see analytics-privatedata-users is the group to be in for wikireplicas access in production

Hi @fgiunchedi, using the config at https://wikitech.wikimedia.org/wiki/SRE/Production_access#Setting_up_your_SSH_config (after pinning the confirmed fingerprint obtained via bast1003) I'm seeing the following:

$ ssh clouddb1017.eqiad.wmnet
Enter passphrase for key '<path_to_key>': 
dr0ptp4kt@clouddb1017.eqiad.wmnet: Permission denied (publickey,keyboard-interactive).

Any pointers?

As far as commands, generally the ones listed in https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Wiki_Replicas (besides surely heavy use of read utilities; I imagine FS perms generally allow for reading but in some cases I also expect FS perms will require elevation) . I anticipate needing to modify, or at least copy-and-modify then in narrow execution invoke with flags certain scripts and commands so that I can run them on copies of elements safely (e.g., distinct safe-to-start/stop/restart haproxy service instances). The systemctl, mysql, maintain-* (because of their internals), and puppet agent commands seem likely.

Thank you for the context @dr0ptp4kt . From reading https://wikitech.wikimedia.org/wiki/Analytics/Data_access#Access_Groups it seems wiki replicas access is granted by virtue of being in analytics-privatedata-users (which you are already), though given the cloud prefix on the host you are trying I think this a wmcs thing. I'm looping in Cloud-Services to assist with this request

Thanks @fgiunchedi and thanks for working through this with me! Should I Phab-mention anyone or tag the ticket, or do the WMCS folks normally catch stuff with the project mention here in Phabricator? Now that I've looked more at the Puppet repo I see the wmcs-admin affiliation for a decent portion of the Wiki Replicas related nodes (including the clouddb* ones). That access seems like it could get me a good part of the way there.

I'll add the following context, mainly for helping remind myself or others following along some day.

What I heard in some conversations the past week was that the cloud prefix on those hostnames is overloaded. The clouddb* servers are said to be part of the prod realm, which seems to mirror the "Service architecture by layers" diagram.

The other analytics replicas (the ones most likely to be used by NDA'd data people) referenced in https://wikitech.wikimedia.org/wiki/Analytics/Data_access are the dbstore* servers, described a bit more at https://wikitech.wikimedia.org/wiki/Analytics/Systems/MariaDB#Database_setup (I can't SSH jump to those, either, although the analytics-privatedata-users access does allow the analytics-mysql wrapper script access via the mysql CLI and the SRV mappings; this is fine for now and probably won't matter, at least not yet).

The WMCS realm "replicas" are fronting proxies on the lefthand side of the "Service architecture by layers", and their naming convention is described at https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#Naming_conventions - those ones have the .cloud suffix. I'm going to Meet with Bryan and Andrew to learn a bit more about WMCS-hosted nodes after attending to some physical world obligations over the next days.

For the righthand side proxies, dbproxy1018 and dbproxy1019, I may need to look at the Puppet repo a bit more, but those look like they probably need special access (they don't seem to have a specifically defined admin group), but I can start with the other stuff first.

I'll be interested in accessing the various Linux nodes across the architecture, but starting from one clump of servers is a good start.

In T343039#9060092, @dr0ptp4kt wrote:

As far as commands, generally the ones listed in https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Wiki_Replicas (besides surely heavy use of read utilities; I imagine FS perms generally allow for reading but in some cases I also expect FS perms will require elevation) . I anticipate needing to modify, or at least copy-and-modify then in narrow execution invoke with flags certain scripts and commands so that I can run them on copies of elements safely (e.g., distinct safe-to-start/stop/restart haproxy service instances). The systemctl, mysql, maintain-* (because of their internals), and puppet agent commands seem likely.

Admin actions on the Wiki Replica hosts generally require full root rights. The wmcs-admin role is fundamentally root on the WMCS infrastructure hosts, but this role has been denied having meaningful sudo rights on the Wiki Replicas out of an abundance of caution from the DBA team. See also:

• T166310: Grant root access for Bryan Davis on labstore* and admin for maintain scripts for labsdb*
• T337848: WMCS-roots wiki replica access

We really need to come up with a way to be able to grant root access to clouddb* hosts that doesn't imply root on all the production databases, because that is really overkill. I don't know if this is something the Infrastructure-Foundations team could help with? cc @joanna_borun

I'm going to open a separate task for the wmcs-admin membership, but leave this task here open so we can continue to explore the additional matter of being

able to grant root access to clouddb* hosts that doesn't imply root on all the production databases

I'll submit a patch on the other task.

In T343039#9068357, @Marostegui wrote:

We really need to come up with a way to be able to grant root access to clouddb* hosts that doesn't imply root on all the production databases, because that is really overkill. I don't know if this is something the Infrastructure-Foundations team could help with? cc @joanna_borun

FYI we are trying to come up with better solutions for this in T337848 but we may need some dba support, ill ping you on that task ;)