pacct)
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Ladsgroup
	Jan 5 2026, 4:54 AM

Description

I'm seeing A LOT of files like:

./log/account/pacct.0.4.gz.2.gz.2.gz.1.2.gz.1.1.1.1.1.2.gz.1.1.2.gz.1.1.1.1.1
./log/account/pacct.0.4.gz.2.gz.2.gz.1.2.gz.1.1.1.1.1.2.gz.2.gz.1.1.2.gz.2.gz
./log/account/pacct.0.4.gz.2.gz.2.gz.1.2.gz.1.1.1.1.2.gz.1.1.1.1.2.gz.1.1.1
./log/account/pacct.0.4.gz.2.gz.2.gz.1.2.gz.1.1.1.1.2.gz.1.1.2.gz.2.gz
./log/account/pacct.0.4.gz.2.gz.2.gz.1.2.gz.1.1.1.1.2.gz.1.2.gz.1.1.1.2.gz.1.1.1
./log/account/pacct.0.4.gz.2.gz.2.gz.1.2.gz.1.1.1.1.2.gz.1.2.gz.1.1.2.gz.2.gz.1
./log/account/pacct.0.4.gz.2.gz.2.gz.1.2.gz.1.1.1.1.2.gz.1.2.gz.1.2.gz.1.1.1.1
./log/account/pacct.0.4.gz.2.gz.2.gz.1.2.gz.1.1.1.1.2.gz.1.2.gz.1.2.gz.1.2.gz.1.1
./log/account/pacct.0.4.gz.2.gz.2.gz.1.2.gz.1.1.1.1.2.gz.1.2.gz.2.gz.1.2.gz.1
./log/account/pacct.0.4.gz.2.gz.2.gz.1.2.gz.1.1.1.1.2.gz.2.gz.1.2.gz.2.gz.1.1
./log/account/pacct.0.4.gz.2.gz.2.gz.1.2.gz.1.1.1.2.gz.1.1.1.2.gz.2.gz
./log/account/pacct.0.4.gz.2.gz.2.gz.1.2.gz.1.1.1.2.gz.1.2.gz.2.gz.1.1.1.1
./log/account/pacct.0.4.gz.2.gz.2.gz.1.2.gz.1.1.1.2.gz.1.2.gz.2.gz.1.1.1.2.gz.1
./log/account/pacct.0.4.gz.2.gz.2.gz.1.2.gz.1.1.1.2.gz.2.gz.1.2.gz.1.1.1.1.1.1
./log/account/pacct.0.4.gz.2.gz.2.gz.1.2.gz.1.1.1.2.gz.2.gz.1.2.gz.1.4.gz.1
./log/account/pacct.0.4.gz.2.gz.2.gz.1.2.gz.1.1.2.gz.1.1.1.2.gz.1.1.1.4.gz.1
./log/account/pacct.0.4.gz.2.gz.2.gz.1.2.gz.1.1.2.gz.1.1.1.2.gz.1.1.2.gz.2.gz
./log/account/pacct.0.4.gz.2.gz.2.gz.1.2.gz.1.1.2.gz.1.2.gz.1.1.2.gz.1.1.2.gz.1
./log/account/pacct.0.4.gz.2.gz.2.gz.1.2.gz.1.1.2.gz.1.2.gz.1.2.gz.1.1
./log/account/pacct.0.4.gz.2.gz.2.gz.1.2.gz.1.1.2.gz.2.gz.1.1.1.1.2.gz.1.2.gz.1
./log/account/pacct.0.4.gz.2.gz.2.gz.1.2.gz.1.1.2.gz.2.gz.1.1.1.2.gz.1.1.1
./log/account/pacct.0.4.gz.2.gz.2.gz.1.2.gz.1.1.2.gz.2.gz.1.1.2.gz.1.1.1.1.1.1
./log/account/pacct.0.4.gz.2.gz.2.gz.1.2.gz.1.1.2.gz.2.gz.1.1.2.gz.1.4.gz.1
./log/account/pacct.0.4.gz.2.gz.2.gz.1.2.gz.1.1.2.gz.2.gz.1.2.gz.1.2.gz.3.gz.1
./log/account/pacct.0.4.gz.2.gz.2.gz.1.2.gz.1.1.2.gz.2.gz.2.gz.1.2.gz.1.3.gz.1
./log/account/pacct.0.4.gz.2.gz.2.gz.1.2.gz.1.1.2.gz.2.gz.2.gz.2.gz.1
./log/account/pacct.0.4.gz.2.gz.2.gz.1.2.gz.1.2.gz.1.1.1.1.1.1.2.gz.1.1.1.1.1
./log/account/pacct.0.4.gz.2.gz.2.gz.1.2.gz.1.2.gz.1.1.1.1.1.2.gz.2.gz.1.2.gz
./log/account/pacct.0.4.gz.2.gz.2.gz.1.2.gz.1.2.gz.1.1.1.1.1.2.gz.2.gz.1.1
./log/account/pacct.0.4.gz.2.gz.2.gz.1.2.gz.1.2.gz.1.1.1.1.2.gz.1.2.gz.1.1.1.1
./log/account/pacct.0.4.gz.2.gz.2.gz.1.2.gz.1.2.gz.1.1.1.2.gz.1.2.gz.1.2.gz.2.gz
./log/account/pacct.0.4.gz.2.gz.2.gz.1.2.gz.1.2.gz.1.1.1.2.gz.2.gz.1.2.gz

Around 1M of them. I had to delete them to unlock the system

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	codesearch: fix recursive logrotate config	operations/puppet	production	+2 -1

Customize query in gerrit

Related Objects

Mentioned In: T417397: Codesearch is regularly partially or fully unavailable
Mentioned Here: T417397: Codesearch is regularly partially or fully unavailable

Event Timeline

Ladsgroup created this task.Jan 5 2026, 4:54 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 5 2026, 4:54 AM

We already added logrotate for this, but since that is not enough, just turned it off:

sudo /usr/sbin/accton off
Turning off process accounting.

weird file names though. maybe the logrotate config had a problem? I think we can just keep it off though.

cat /etc/logrotate.d/pacct 
# SPDX-License-Identifier: Apache-2.0
/var/log/account/* {
    daily
    missingok
    compress
    delaycompress
    notifempty
    copytruncate
}

Krinkle mentioned this in T417397: Codesearch is regularly partially or fully unavailable.Fri, Feb 13, 9:35 PM

Mentioned in SAL (#wikimedia-cloud) [2026-02-13T21:39:05Z] <Krinkle> Manually pruning codesearch9:/var/log/account/pacct, ref T417397 T413739

In T417397#11616797, @Krinkle wrote:

Ah, the curse of inodes again. This time on the root drive instead of the /srv volume.

Grafana dashboard:

In T417397#11616949, @Krinkle wrote:
krinkle@codesearch9:/var/log$ sudo find /var/log/account/ -type f -name "pacct*" -delete
[…] It finished after 20 minutes.
$ df -i  
Filesystem       Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1       1302528  120553 1181975   10% /
…
/dev/sdb       10485760 5526733 4959027   53% /srv
…
The backends are up again at https://codesearch.wmcloud.org/_health/ […]

I'm leaving this task open because […], I expect this to happen again. […]

In T413739#11498877, @Dzahn wrote:

weird file names though. maybe the logrotate config had a problem? I think we can just keep it off though.

Yeah, it was the same issue again. Here's a sample of some files I deleted with rm -fv instead of find -delete.

$ sudo rm -rvf account/pacct.0.4.gz.2.gz.2* 
removed 'account/pacct.0.4.gz.2.gz.2.gz'
removed 'account/pacct.0.4.gz.2.gz.2.gz.1'
removed 'account/pacct.0.4.gz.2.gz.2.gz.1.1'
removed 'account/pacct.0.4.gz.2.gz.2.gz.1.1.1'
removed 'account/pacct.0.4.gz.2.gz.2.gz.1.1.1.1'
removed 'account/pacct.0.4.gz.2.gz.2.gz.1.1.1.1.1'
removed 'account/pacct.0.4.gz.2.gz.2.gz.1.1.1.1.1.1'
removed 'account/pacct.0.4.gz.2.gz.2.gz.1.1.1.1.1.1.1'
removed 'account/pacct.0.4.gz.2.gz.2.gz.1.1.1.1.1.1.1.1'
removed 'account/pacct.0.4.gz.2.gz.2.gz.1.1.1.1.1.1.1.1.1'
removed 'account/pacct.0.4.gz.2.gz.2.gz.1.1.1.1.1.1.1.1.1.1'
removed 'account/pacct.0.4.gz.2.gz.2.gz.1.1.1.1.1.1.1.1.1.1.1'
removed 'account/pacct.0.4.gz.2.gz.2.gz.1.1.1.1.1.1.1.1.1.1.1.1'
removed 'account/pacct.0.4.gz.2.gz.2.gz.1.1.1.1.1.1.1.1.1.1.1.1.1'
removed 'account/pacct.0.4.gz.2.gz.2.gz.1.1.1.1.1.1.1.1.1.1.1.1.1.1'
removed 'account/pacct.0.4.gz.2.gz.2.gz.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1'
removed 'account/pacct.0.4.gz.2.gz.2.gz.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1'
removed 'account/pacct.0.4.gz.2.gz.2.gz.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1'
removed 'account/pacct.0.4.gz.2.gz.2.gz.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1'
removed 'account/pacct.0.4.gz.2.gz.2.gz.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1'
removed 'account/pacct.0.4.gz.2.gz.2.gz.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1'
removed 'account/pacct.0.4.gz.2.gz.2.gz.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.gz'
removed 'account/pacct.0.4.gz.2.gz.2.gz.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.gz'
removed 'account/pacct.0.4.gz.2.gz.2.gz.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.gz.1'
removed 'account/pacct.0.4.gz.2.gz.2.gz.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.gz'
removed 'account/pacct.0.4.gz.2.gz.2.gz.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.gz.1'
removed 'account/pacct.0.4.gz.2.gz.2.gz.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.gz.1.1'
removed 'account/pacct.0.4.gz.2.gz.2.gz.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.gz'
removed 'account/pacct.0.4.gz.2.gz.2.gz.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.gz'
…
removed 'account/pacct.0.4.gz.2.gz.1.2.gz.2.gz.2.gz.2.gz.2.gz.2.gz.2.gz.1.1'
removed 'account/pacct.0.4.gz.2.gz.1.2.gz.2.gz.2.gz.2.gz.2.gz.2.gz.3.gz'
removed 'account/pacct.0.4.gz.2.gz.1.2.gz.2.gz.2.gz.2.gz.2.gz.3.gz'
removed 'account/pacct.0.4.gz.2.gz.1.2.gz.2.gz.2.gz.2.gz.2.gz.4.gz'
removed 'account/pacct.0.4.gz.2.gz.1.2.gz.2.gz.2.gz.2.gz.3.gz'
removed 'account/pacct.0.4.gz.2.gz.1.2.gz.2.gz.2.gz.2.gz.4.gz'
removed 'account/pacct.0.4.gz.2.gz.1.2.gz.2.gz.2.gz.3.gz'
removed 'account/pacct.0.4.gz.2.gz.1.2.gz.2.gz.2.gz.4.gz'
removed 'account/pacct.0.4.gz.2.gz.1.2.gz.2.gz.3.gz'
removed 'account/pacct.0.4.gz.2.gz.1.2.gz.2.gz.4.gz'
removed 'account/pacct.0.4.gz.2.gz.1.2.gz.3.gz'
removed 'account/pacct.0.4.gz.2.gz.1.2.gz.4.gz'
removed 'account/pacct.0.4.gz.2.gz.1.3.gz'
removed 'account/pacct.0.4.gz.2.gz.1.4.gz'
removed 'account/pacct.0.4.gz.2.gz.3.gz'
removed 'account/pacct.0.4.gz.2.gz.4.gz'
$ sudo rm -rvf account/pacct.0.4.gz*
removed 'account/pacct.0.4.gz'
removed 'account/pacct.0.4.gz.1'
removed 'account/pacct.0.4.gz.3.gz'
removed 'account/pacct.0.4.gz.4.gz'
$ sudo rm -rvf account/pacct.0.*
-bash: /usr/bin/sudo: Argument list too long
$ sudo rm -rvf account/pacct.0.3.gz.2.*
-bash: /usr/bin/sudo: Argument list too long

I think Daniel shut down pacct, it gets re-enabled again. Probably a puppet patch is needed.

Change #1239441 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] codesearch: fix recursive logrotate config

https://gerrit.wikimedia.org/r/1239441

gerritbot added a project: Patch-For-Review.Fri, Feb 13, 11:32 PM

Change #1239441 merged by Dzahn:

[operations/puppet@production] codesearch: fix recursive logrotate config

https://gerrit.wikimedia.org/r/1239441

removed 'account/pacct.0.4.gz.2.gz.2.gz.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1'

I think we see these weird file names because of a bug in the logrotate config. It uses a wildcard /var/log/account/* which makes it rotate already rotated logs and recurses.

Attempted fix merged on prod puppetmaster.

Mentioned in SAL (#wikimedia-cloud) [2026-02-14T00:01:01Z] <mutante> codesearch9: systemctl restart logrotate with new config without * - T413739

Logrotate config changed and restarted. I hope this is fixed now. Currently /var/log/account is empty.

Maintenance_bot removed a project: Patch-For-Review.Sat, Feb 14, 12:30 AM

Krinkle closed this task as Resolved.Sat, Feb 14, 1:42 AM

Krinkle assigned this task to Dzahn.

Krinkle added a project: collaboration-services.

"no space left on device" for codesearch9 root disk (out of inodes due to /var/log/account/pacct)Closed, ResolvedPublicActions

Description

Details

Related Objects

Event Timeline

"no space left on device" for codesearch9 root disk (out of inodes due to /var/log/account/pacct)
Closed, ResolvedPublic
Actions