I cannot currently access the beta cluster due to the above error. I just noticed this happening now.
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Restricted Task | |||||
| Declined | BUG REPORT | None | T398450 End user reported "Error: 503, Backend fetch failed" contacting an unspecified Beta Cluster hosted wiki |
Event Timeline
Comment Actions
https://meta.wikimedia.beta.wmflabs.org/wiki/Main_Page and https://simple.wikipedia.beta.wmflabs.org/wiki/Main_Page are currently responding to my requests.
I am not seeing any automated monitoring alerts related to Beta Cluster around the time of this report.
There are a number of connection errors logged today by the trafficserver instance on deployment-cache-text08. I haven't ever monitored this log stream before to have a good instinct about what is "normal".
There was a big spike in load on deployment-mediawiki14 from about 13:10 to 13:40 and other smaller spiked following. Load was climbing around the time this bug report was filed.
Comment Actions
root@deployment-cache-text08:/var/log/trafficserver# grep '20250702' error.log | cut -c 10-12 | uniq -c 8 00h 23 01h 282 02h 29 03h 58 04h 20 05h 17 06h 16 07h 359 08h 70 09h 2088 10h 154 11h 2093 12h 15031 13h 15637 14h 9 15h root@deployment-cache-text08:/var/log/trafficserver# grep '20250702.14h' error.log | cut -d\ -f1 | cut -d. -f2 | cut -c 4-6 | uniq -c 815 00m 598 01m 323 02m 635 03m 536 04m 759 05m 755 06m 780 07m 712 08m 611 09m 603 10m 492 11m 29 12m 24 13m 163 14m 366 15m 225 16m 3 17m 1 18m 3 20m 3 21m 1 22m 3 23m 4 24m 25 25m 210 26m 551 27m 442 28m 579 29m 653 30m 310 31m 477 32m 351 33m 219 34m 324 35m 332 36m 285 37m 34 38m 28 39m 1 40m 4 42m 5 43m 6 44m 27 45m 201 46m 634 47m 704 48m 583 49m 208 50m
Comment Actions
I don't think there is much we can do at this point other than being aware this happened.