I'm currently getting incredibly long load times at https://wikipedialibrary.wmflabs.org/, leading either to a server error or an unstyled page.
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T402053 Replace deprecated Bullseye VMs in Cloud VPS | |||
| Resolved | None | T429442 Migrate Moderator Tools Cloud VPS tools from Debian Bullseye | |||
| Resolved | jsn.sherman | T402054 twl: Replace deprecated Bullseye VMs in Cloud VPS | |||
| Resolved | jsn.sherman | T432600 The Wikipedia Library is not loading |
Event Timeline
Comment Actions
Looks related to our move to docker compose from docker swarm. I landed a couple of hotfixes to address this:
ip addresses could shuffle at deploy time since compose doesn't have an extra layer of internal service endpoints and we don't pin things:
https://github.com/WikipediaLibrary/TWLight/commit/7ef863e0cb364849b8531e242fe0ac3ac986ce29
the image pruning could race the image pull and make all containers sad. swarm prevented such races:
https://github.com/WikipediaLibrary/TWLight/commit/df5ca455347339d7f7ac73a0f881259c91d9604d
Comment Actions
noting that pushing the hotfixes did in fact take the site back down since the hotfixes were deployed with the prehotifx setup. Leaving this open in QA
Comment Actions
verified that the nginx change + the pruning change guard against the relevant behavior differences from docker swarm; deploys are going through happily now.