Shellbox is broken on wikitech-static due to disk fullness
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	RLazarus
	Jun 8 2023, 9:24 PM

Description

@Perryprog pointed out that https://wikitech-static.wikimedia.org/wiki/Reporting_a_connectivity_issue is broken on wikitech:

[65d8539dde7bff1e028dffdc] /wiki/Reporting_a_connectivity_issue Shellbox\ShellboxError: Error creating directory shellbox-57fe4b1591a44cff

Backtrace:

from /srv/mediawiki/w/vendor/wikimedia/shellbox/src/FileUtils.php(114)
#0 /srv/mediawiki/w/vendor/wikimedia/shellbox/src/TempDirManager.php(164): Shellbox\FileUtils::mkdir()
[...]

Full stack trace:

65d8539dde7bff1e028dffdc.txt4 KBDownload

It looks like this affects any page on wikitech-static that uses a <syntaxhighlight> tag, but it's particularly unfortunate for the "Reporting a connectivity issue" page -- we regularly send users to -static for that page, so that they can help us troubleshoot when they can't reach our data centers. As a temporary workaround, if we can't fix shellbox-syntaxhighlight on wikitech-static quickly, we could just remove syntax highlighting from that page.

Event Timeline

RLazarus triaged this task as High priority.Jun 8 2023, 9:24 PM

RLazarus created this task.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 8 2023, 9:24 PM

Looks like disk fullness:

root@wikitech-static:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            979M     0  979M   0% /dev
tmpfs           199M   21M  179M  11% /run
/dev/xvda1       79G   76G     0 100% /
tmpfs           995M     0  995M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           995M     0  995M   0% /sys/fs/cgroup
tmpfs           199M     0  199M   0% /run/user/0

@Andrew Is this something you can take a look at?

RLazarus renamed this task from Shellbox is broken on wikitech-static to Shellbox is broken on wikitech-static due to disk fullness.Jun 8 2023, 11:48 PM

I will look if I can get an ssh connection. Worst case we can resize the instance and increase our monthly bill be a few bucks. Thanks for noticing!

RLazarus assigned this task to Andrew.Jun 9 2023, 1:18 AM

A ton of files in /srv/mediawiki/images/wikitech/archive but deleteArchivedFiles.php --delete says there's nothing to delete. It's tempting to just rm that directory anyway but it would be nice to know what's happening first...

Mentioned in SAL (#wikimedia-operations) [2023-06-09T15:30:39Z] <andrewbogott> wikitech-static: deleted everything in /srv/mediawiki/images/wikitech/archive for T338520

root@wikitech-static:/srv/mediawiki/images/wikitech/archive# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            979M     0  979M   0% /dev
tmpfs           199M   21M  179M  11% /run
/dev/xvda1       79G   36G   40G  47% /
tmpfs           995M     0  995M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           995M     0  995M   0% /sys/fs/cgroup
tmpfs           199M     0  199M   0% /run/user/0

No let's see if that was actually the problem...

No let's see if that was actually the problem...

It was!

Thank you!

Shellbox is broken on wikitech-static due to disk fullnessClosed, ResolvedPublicActions

Description

Event Timeline

Shellbox is broken on wikitech-static due to disk fullness
Closed, ResolvedPublic
Actions