Page MenuHomePhabricator

Toolforge NFS tracing misses some dumps events
Closed, ResolvedPublic

Description

I accessed a dumps file over NFS from a tool shell pod:

[tools.majavah-test@shell-1769005612 ~] $ cat /public/dumps/public/enwiki/latest/enwiki-latest-md5sums.txt | head
10754ee2ff42845e44c5a50ee2aa6372  enwiki-20260101-site_stats.sql.gz
06be099469550601557bd02a8c1ce8bf  enwiki-20260101-image.sql.gz
b20cf5bf318cc78e52e9aa3cfc87ef3f  enwiki-20260101-pagelinks.sql.gz
122f7eef64feb443f3fe4ca9fedc9e2b  enwiki-20260101-categorylinks.sql.gz
0f6135c33a1aa8d51b0e7e475d9d388f  enwiki-20260101-imagelinks.sql.gz
729da6aef4fa179b386c1ea250fc17dc  enwiki-20260101-templatelinks.sql.gz
e246fd0593bba08ea03688297c0266b4  enwiki-20260101-linktarget.sql.gz
1aa2b96a8786e2e897723347b8650673  enwiki-20260101-externallinks.sql.gz
929e2c4b731e7321bcb7399d3c8faa2e  enwiki-20260101-langlinks.sql.gz
b22753be8c3bab46ba6664e3b24aac2d  enwiki-20260101-user_groups.sql.gz

This is not visible in the tracing dashboard like I would expect it to be:

image.png (1,430×659 px, 86 KB)

strace shows the openat() call not resolving the symlink beforehand, which I presume is the issue:

openat(AT_FDCWD, "/public/dumps/public/enwiki/latest/enwiki-latest-md5sums.txt", O_RDONLY) = 3

Event Timeline

fnegri triaged this task as Medium priority.Jan 21 2026, 4:10 PM

Thanks for the report, that's why I was asking if we knew of tools using for example dumps. So it looks like the pre-existing code had an optimization to exclude anything that doesn't start with /mnt/nfs and I naively assumed that it was already checked that the data at that point was already a resolved path, but it's not.
It looks like we could move the hook from the open/openat syscall to LSM_PROBE (I've quickly checked and we seem to have support for it, but I'll check more extensively on the whole fleet of workers to be sure) that should gets hooked after the kernel has resolved the path.
I'm working on a patch to test the above assumptions.

Change #1231034 had a related patch set uploaded (by Volans; author: Volans):

[operations/puppet@production] wmcs: fix infra-tracing-nfs

https://gerrit.wikimedia.org/r/1231034

After quite few testing the route of LSM_PROBE wasn't feasible because of some missing feature in the kernel that AFAICT can't just be enabled but requires recompilation compared to the standard Debian kernel. I've tried various hooks including file_open and vfs_open without success. I've come up with a solution using the current approach and sent the above patch for it.

Change #1231034 merged by Volans:

[operations/puppet@production] wmcs: fix infra-tracing-nfs

https://gerrit.wikimedia.org/r/1231034

Change #1237251 had a related patch set uploaded (by Volans; author: Volans):

[operations/puppet@production] wmcs: infra-tracing-nfs bail out earlier if root

https://gerrit.wikimedia.org/r/1237251

Change #1237251 merged by Volans:

[operations/puppet@production] wmcs: infra-tracing-nfs bail out earlier if root

https://gerrit.wikimedia.org/r/1237251

Change #1239689 had a related patch set uploaded (by Volans; author: Volans):

[operations/puppet@production] wmcs: infra-tracing-nfs support non-k8s nodes

https://gerrit.wikimedia.org/r/1239689

Change #1239689 merged by Volans:

[operations/puppet@production] wmcs: infra-tracing-nfs support non-k8s nodes

https://gerrit.wikimedia.org/r/1239689

Change #1239905 had a related patch set uploaded (by Volans; author: Volans):

[operations/puppet@production] wmcs: infra-tracing-nfs fix variable

https://gerrit.wikimedia.org/r/1239905

Change #1239905 merged by Volans:

[operations/puppet@production] wmcs: infra-tracing-nfs fix variable

https://gerrit.wikimedia.org/r/1239905

I think we should be able to call this resolved as it should now track pretty much all accesses.
The only ones known to be not tracked are small corner cases where an absolute path is used but has some double slash // in the very first directory of the path or plays with the .. to go to parent directories.
If we want to be sure also for those corner cases we'll be forced to pass all absolute paths from ebpf to python to be able to resolve them before deciding.
From one side it will make the ebpf code simpler, from the other it will add more work on the python (slow) side of the tracing.
Thoughts?

For the paths with .., we are still catching the ones that have it after the mount prefix right? (like /mnt/nfs/home/data/public/something/../../somethingelse)

The issue is with the ones that have it in the prefix itself that we don't catch right? (like /usr/../mnt/home/data/public/..)

If so I think though that it might be enough for our current needs as is, specially as we provide the path with an envvar, most cases would just have the full prefix.

We could try enabling all in one worker, to asses how many we are actually missing if we want more certainty.

For the paths with .., we are still catching the ones that have it after the mount prefix right? (like /mnt/nfs/home/data/public/something/../../somethingelse)

The issue is with the ones that have it in the prefix itself that we don't catch right? (like /usr/../mnt/home/data/public/..)

Correct, those and things like //data/project/... or /data//project/...

We could try enabling all in one worker, to asses how many we are actually missing if we want more certainty.

Sure, that's always an option.