arclamp hosts ran out of space: big excimer daily/hourly logs
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	fgiunchedi
	Apr 28 2025, 7:12 AM

Project Tags

Referenced Files

	F59502123: host-overview.png
	Apr 28 2025, 7:12 AM

Subscribers

Description

Starting around the 22nd both hosts grow their filesystem usage at a rate where space fills up: https://grafana.wikimedia.org/goto/6fd6KRbNR?orgId=1

Related Objects

Mentioned Here: T374230: Frequent 500 Errors and Timeouts When Adding Statements to New Item or Lexeme-typed Properties
T391516: https://performance.wikimedia.org/php-profiling/ leads to 404 for all listed sources

Event Timeline

fgiunchedi created this task.Apr 28 2025, 7:12 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 28 2025, 7:12 AM

Due to the timing I suspect this is a side effect of T391516 where things now have started to actually working. Something to note is also that newer logs don't seem to get compressed

It was also pointed out that T374230: Frequent 500 Errors and Timeouts When Adding Statements to New Item or Lexeme-typed Properties and infinite recursion might be the cause for size explosion of logs

@andrea.denisse cleaned up files on both hosts and we're back

thank you @andrea.denisse and @fgiunchedi

arclamp hosts ran out of space: big excimer daily/hourly logsClosed, ResolvedPublicActions

Description

Related Objects

Event Timeline

arclamp hosts ran out of space: big excimer daily/hourly logs
Closed, ResolvedPublic
Actions