I think that we can use the airflow-deployers group, as defined here, to ensure that those users with enhanced privileges in Airflow are able to manage the artifact files on HDFS.

That looks great, thanks for chasing that group down.

In T400902#11056857, @BTullis wrote:

...

the topic at hand is to somehow reduce the privileges that blunderbuss has today on HDFS, because we noted a privilege escalation to analytics. We want to blunderbuss to have just the right amount of privileges to accomplish its current use case of landing files into the Airflow caches.

I am not sure whether that conversation has resulted in a ticket (or tickets) yet, but ultimately we probably want the files in /wmf/cache/artifacts/airflow to be owned by the blunderbuss system user, but readable by the airflow system users.

Actually, this suggestions from Ben sounds like the best approach to me. If the blunderbuss system user owns the files, their provenance is clear. And Airflow system users just happen to be consumers, and not owners, of those files. +1 to this approach.

Change #1180113 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Add a system user and group for blunderbuss

https://gerrit.wikimedia.org/r/1180113

Change #1180113 abandoned by Btullis:

[operations/puppet@production] Add a system user and group for blunderbuss

Reason:

Using a different approach

https://gerrit.wikimedia.org/r/1180113

Change #1180113 restored by Btullis:

[operations/puppet@production] Add a system user and group for blunderbuss

https://gerrit.wikimedia.org/r/1180113

Change #1180113 abandoned by Btullis:

[operations/puppet@production] Add a system user and group for blunderbuss

Reason:

We don't need this posix uid/gid after all.

https://gerrit.wikimedia.org/r/1180113

I started by thinking that we needed a POSIX uid/gid for blunderbuss, so I created https://gerrit.wikimedia.org/r/1180113 and reserved a pair.
However, after doing a bit more research, I don't think that we need this after all. The blunderbuss user and group, only need to exist on the HDFS file system, which doesn't use numeric values.

All we need to do is to make sure that the correct kerberos principal is used to authenticate.

The principal already exists.

root@krb1002:~# sudo manage_principals.py list *blunderbuss*
HTTP/blunderbuss.discovery.wmnet@WIKIMEDIA
analytics/blunderbuss.discovery.wmnet@WIKIMEDIA
blunderbuss/blunderbuss.discovery.wmnet@WIKIMEDIA

We can see that it's in the keytab that has been provided to blunderbuss.

btullis@deploy1003:~$ kube-env blunderbuss-deploy dse-k8s-eqiad
btullis@deploy1003:~$ kubectl exec -it blunderbuss-production-5bdc8c9cbc-2vbnl -c blunderbuss-production-renew-kerberos-token -- bash

blunderbuss@blunderbuss-production-5bdc8c9cbc-2vbnl:/srv/app$ klist -k /etc/kerberos/keytabs/blunderbuss.keytab 
Keytab name: FILE:/etc/kerberos/keytabs/blunderbuss.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   1 analytics/blunderbuss.discovery.wmnet@WIKIMEDIA
   1 blunderbuss/blunderbuss.discovery.wmnet@WIKIMEDIA
   1 HTTP/blunderbuss.discovery.wmnet@WIKIMEDIA
blunderbuss@blunderbuss-production-5bdc8c9cbc-2vbnl:/srv/app$

However, it's not the first principal in the keytab and therefore not the default.

blunderbuss@blunderbuss-production-5bdc8c9cbc-2vbnl:/srv/app$ klist
Ticket cache: FILE:/etc/kerberos/tgt/tkt
Default principal: analytics/blunderbuss.discovery.wmnet@WIKIMEDIA

Valid starting     Expires            Service principal
08/19/25 13:24:49  08/19/25 23:24:49  krbtgt/WIKIMEDIA@WIKIMEDIA
	renew until 08/26/25 13:24:49

There are a couple of ways that we could handle this.

1. We could change the keytab, so that only the blunderbuss/blunderbuss.discovery.wmnet@WIKIMEDIA principal is present.
2. We could change the way that k5start is executed in the blunderbuss-production-renew-kerberos-token container.

Currently we execute k5start with the -U option, which causes it to...

use the first principal in the keytab as the client principal and don't look for a principal on the command line

I think that it's probably fine for us to go with option 1. We only have one deployment of blunderbuss and I think that we only need this one principal. I'm not sure why the HTTP/blunderbuss.discovery.wmnet@WIKIMEDIA principal was created and added to the keytab, but I will try removing that.

I have created this new keytab.

root@krb1002:~# mkdir /srv/kerberos/keytabs/blunderbuss.discovery.wmnet
root@krb1002:~# mkdir /srv/kerberos/keytabs/blunderbuss.discovery.wmnet/blunderbuss

root@krb1002:~# kadmin.local ktadd -norandkey -k /srv/kerberos/keytabs/blunderbuss.discovery.wmnet/blunderbuss/blunderbuss.keytab blunderbuss/blunderbuss.discovery.wmnet@WIKIMEDIA
Entry for principal blunderbuss/blunderbuss.discovery.wmnet@WIKIMEDIA with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/srv/kerberos/keytabs/blunderbuss.discovery.wmnet/blunderbuss/blunderbuss.keytab.

root@krb1002:~# klist -k /srv/kerberos/keytabs/blunderbuss.discovery.wmnet/blunderbuss/blunderbuss.keytab
Keytab name: FILE:/srv/kerberos/keytabs/blunderbuss.discovery.wmnet/blunderbuss/blunderbuss.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   1 blunderbuss/blunderbuss.discovery.wmnet@WIKIMEDIA

Now I will base64 encode it and update the value in the puppet private repository.

Before changing how it works, I will make sure that all of the existing files are group and world readable.

Set the files to mode 644

hdfs@an-launcher1002:~$ hdfs dfs -chmod 644  $(hdfs dfs -ls -R /wmf/cache/artifacts/airflow | grep "^-" | awk '{print $8}')

Set the directoriers to mode 755

hdfs@an-launcher1002:~$ hdfs dfs -chmod 755  $(hdfs dfs -ls -R /wmf/cache/artifacts/airflow | grep "^d" | awk '{print $8}')

Change the ownership of all files and directories, recursively.

hdfs@an-launcher1002:~$ hdfs dfs -chown -R blunderbuss:blunderbuss /wmf/cache/artifacts/airflow

Looks good. It has the principal blunderbuss/blunderbuss.discovery.wmnet@WIKIMEDIA now.

btullis@deploy1003:/srv/deployment-charts/helmfile.d/dse-k8s-services/blunderbuss$ kubectl exec -it blunderbuss-production-7fd4c999dd-4m9bb -- bash
Defaulted container "blunderbuss-production" out of: blunderbuss-production, blunderbuss-production-tls-proxy, statsd-exporter, blunderbuss-production-renew-kerberos-token
blunderbuss@blunderbuss-production-7fd4c999dd-4m9bb:/srv/app$ klist
Ticket cache: FILE:/etc/kerberos/tgt/tkt
Default principal: blunderbuss/blunderbuss.discovery.wmnet@WIKIMEDIA

Valid starting     Expires            Service principal
08/19/25 14:52:44  08/20/25 00:52:44  krbtgt/WIKIMEDIA@WIKIMEDIA
	renew until 08/26/25 14:52:44
blunderbuss@blunderbuss-production-7fd4c999dd-4m9bb:/srv/app$

btullis opened https://gitlab.wikimedia.org/repos/data-engineering/blunderbuss/-/merge_requests/12

Update the umask with which files are written to HDFS

btullis merged https://gitlab.wikimedia.org/repos/data-engineering/blunderbuss/-/merge_requests/12

Update the umask with which files are written to HDFS

Change #1180184 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Use a specific image version for blunderbuss

https://gerrit.wikimedia.org/r/1180184

Change #1180184 merged by jenkins-bot:

[operations/deployment-charts@master] Use a specific image version for blunderbuss

https://gerrit.wikimedia.org/r/1180184