When running toolforge tools on Kubernetes, the pod should have a symlink /public/dumps/public pointing to a mounted NFS directory. Currently, the target NFS directory seems to be getting mounted into the pod, but at a slightly different mount point than the symlink’s target location.
Steps to reproduce:
$ ssh sascha@dev.toolforge.org $ become qrank $ cat nfs-test.yaml apiVersion: batch/v1 kind: Job metadata: name: qrank.nfs-test namespace: tool-qrank labels: name: qrank.nfs-test toolforge: tool spec: template: metadata: labels: name: qrank.nfs-test toolforge: tool spec: restartPolicy: Never containers: - name: test workingDir: /data/project/qrank image: docker-registry.tools.wmflabs.org/toolforge-python37-sssd-base:latest command: [ "ls", "-l", "/mnt/nfs", "/public/dumps/public" ] $ kubectl apply --validate=true -f nfs-test.yaml job.batch/qrank.nfs-test created $ kubectl logs jobs/qrank.nfs-test -f lrwxrwxrwx 1 root root 42 Mar 13 2020 /public/dumps/public -> /mnt/nfs/dumps-labstore1007.wikimedia.org/ /mnt/nfs: total 72 drwxr-xr-x 1005 400 400 36864 Feb 22 10:30 dumps-labstore1006.wikimedia.org drwxr-xr-x 1005 400 400 36864 Feb 22 10:31 dumps-labstore1007.wikimedia.orgs
Expected: /public/dumps/public should be a working symlink.
Observed: /public/dumps/public is a broken symlink because the mount point in /mnt/nfs on the pod is dumps-labstore1007.wikimedia.orgs instead of dumps-labstore1007.wikimedia.org. Note the final s in the directory name.
Temporary workaround: Configure custom tools to read dumps from /mnt/nfs/dumps-labstore1007.wikimedia.orgs/ (with final s) instead of /public/dumps/public.