Toolforge outage: toolsdb out of space
Closed, ResolvedPublicBUG REPORT
Actions

Assigned To

Authored By

	Bamyers99
	Nov 5 2025, 12:58 AM

Description

Original title: Could not connect to database: User s51080 already has more than 'max_user_connections' active connections

Toolforge project: checkwiki
host: tools.db.svc.wikimedia.cloud
user: s51080
database: s51080__checkwiki_p
web server url: Check Wikipedia

No software changes for weeks.
To try and fix: stopped web server, and continuous service (cw-live-scan) for 5 minutes. Then restarted and receiving the same error from both services.

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		fnegri	T409287 [toolsdb] Destroy tools-db-4 and create new host
		Resolved	BUG REPORT	fnegri	T409244 Toolforge outage: toolsdb out of space

Event Timeline

Bamyers99 created this task.Nov 5 2025, 12:58 AM

Restricted Application added a project: cloud-services-team. · View Herald TranscriptNov 5 2025, 12:58 AM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

It seems that toolsdb is generally down. Here is ticket for montage db connection [https://github.com/hatnote/montage/issues/353#issuecomment-3488646411 timeouting].

Also affects: fountain, PetScan

I triaged this high as this is still going on and impacts to multiple tools.

Betty2407 subscribed.Nov 5 2025, 6:57 AM

Alex_Cohn subscribed.Nov 5 2025, 7:03 AM

tools-db-4 storage volume is out of space, I'll use this task for tracking

disk space free trend for tools-db-4 over the last 30d

2025-11-05-082453_813x632_scrot.png (632×813 px, 45 KB)

and over the last couple of days

2025-11-05-082559_807x619_scrot.png (619×807 px, 43 KB)

HakanIST subscribed.Nov 5 2025, 7:29 AM

jeremyb subscribed.Nov 5 2025, 7:34 AM

Doing a comparison with the replica on tools-db-6, there's ~800G free there:

root@tools-db-6:/srv/labsdb/data# df -h /srv/labsdb
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb        3.9T  2.8T  885G  77% /srv/labsdb

And most/all of the discrepancy in space seems to come from ibdata1

root@tools-db-4:~# du -cs /srv/labsdb/data/* | sort -nr | head -10
3098616996	total
1032818748	/srv/labsdb/data/ibdata1
294125620	/srv/labsdb/data/s53220__quickstatements_p
258368168	/srv/labsdb/data/s51434__mixnmatch_p
191399380	/srv/labsdb/data/s53685__editgroups
98341700	/srv/labsdb/data/s51698__yetkin
86048076	/srv/labsdb/data/s51412__data
72501176	/srv/labsdb/data/s51114__enwp10
60054592	/srv/labsdb/data/s53952__freebase_p
58990172	/srv/labsdb/data/s51499__wikiminiatlas

root@tools-db-6:~# du -cs /srv/labsdb/data/* | sort -nr | head -10
2174533936	total
294125620	/srv/labsdb/data/s53220__quickstatements_p
258122072	/srv/labsdb/data/s51434__mixnmatch_p
191378900	/srv/labsdb/data/s53685__editgroups
120139788	/srv/labsdb/data/ibdata1
97220368	/srv/labsdb/data/s51698__yetkin
86057432	/srv/labsdb/data/s51412__data
72501176	/srv/labsdb/data/s51114__enwp10
60054592	/srv/labsdb/data/s53952__freebase_p
58990160	/srv/labsdb/data/s51499__wikiminiatlas

Edgars2007 subscribed.Nov 5 2025, 7:56 AM

Chlod renamed this task from Could not connect to database: User s51080 already has more than 'max_user_connections' active connections to Toolforge outage: toolsdb out of space.Nov 5 2025, 8:23 AM

Chlod updated the task description. (Show Details)

Albertoleoncio subscribed.Nov 5 2025, 8:24 AM

MPGuy2824 subscribed.Nov 5 2025, 8:38 AM

DreamRimmer subscribed.Nov 5 2025, 9:36 AM

Novem_Linguae subscribed.Nov 5 2025, 9:39 AM

Rachmat04 subscribed.Nov 5 2025, 9:49 AM

We added some disk space to tools-db-4 and toolsdb is working again.

We are still investigating what caused the big increase in size for ibdata1.

Thank you very much !

This also impacted ClueBot NG, alert for report interface being down was sent at 23:55 UTC (alerting rule is after 2min), then bot not editing at 00:39 UTC (alerting after 1hour, 5min). Recovery at 08:45 UTC. Manually checked around 1am (CET) and was reporting max connections used per the above.

fnegri mentioned this in T409287: [toolsdb] Destroy tools-db-4 and create new host.Nov 5 2025, 1:25 PM