| vaughnwalters | |
| Jan 20 2026, 10:25 PM |
| F71579234: Screenshot 2026-01-20 at 4.24.40 PM.png | |
| Jan 20 2026, 10:25 PM |
Steps to replicate the issue (include links if applicable):
Visit https://releng-data.wmcloud.org/
What happens?:
Error 500
attempt to write a readonly database
What should have happened instead?:
Should display the site
Other information (browser name/version, screenshots, etc.):
So was the root cause here that external crawlers were causing an overload of our ability to handle traffic, and the root cause was blocking those IPs? Just trying to learn here.
datasette, the project that powers releng-data was getting a bunch of traffic from one thing. Since this lives on Cloud VPS, I can't actually see IP addresses, but I spotted some patterns in the logs.
thcipriani@stats:~$ sudo zgrep -ci '<bad guy identifier>' /var/log/nginx/access.log 20080 thcipriani@stats:~$ sudo zgrep -civ '<bad guy identifier>' /var/log/nginx/access.log 11752
So 63% of traffic was from one entity. I blocked that entity and everything returned to normal. I'll say the entity was generating a lot of csv files, which might have clogged up the tubes.
This one entity slipped through despite putting anubis out front of this project :(
I doubt we would have noticed it, but the job to update flaky tests was running, too, which meant the server was under some load outside of being squeezed to death. This entity has been hammering it for a while—at least a week.