I just noticed these on Cumin and assumed someone had forgotten to drop their puppet node entries, so I went and deactivated/cleaned them there, then realised I had been running cumin without the puppetdb backend so these actually existing according to nova. I also see mediawiki-config.git still has references to them. However cumin is showing Permission denied (publickey). and I cannot log in normally or as root. Something is wrong here.
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T53494 Use Beta cluster as a true canary for code deployments (epic) | |||
| Declined | None | T87220 Minimize infrastructure differences between Beta Cluster and production | |||
| Open | None | T196662 Set up LVS in beta like prod | |||
| Resolved | • bd808 | T166396 Program 1 Outcome 4: VPS hosting | |||
| Resolved | None | T167293 Nova-network to Neutron migration | |||
| Resolved | Krenair | T208101 Migrate deployment-prep to eqiad1 | |||
| Resolved | dduvall | T210301 Status of deployment-redis0[56]? |
Event Timeline
Comment Actions
I'm also seeing MW fatals resulting from the inoperable redis servers and the mediawiki-config for redis_lock that still references them.
Comment Actions
I just merged the final patch related to T210030: RedisBagOStuff is broken on beta that removed the stale redis server entries from the labs related mediawiki-config, and the instances are terminated.