All of the Beta Cluster is unavailable (along with much of Toollabs)
This was caused by the restart of all WMF Labs servers due to the libc security vulnerability that was released today.
All of the Beta Cluster is unavailable (along with much of Toollabs)
This was caused by the restart of all WMF Labs servers due to the libc security vulnerability that was released today.
@Andrew / @yuvipanda: I assume this is due to the restart of Labs infra due to the security release this morning, yes?
Seems it's likely because nginx isn't running/won't start
root@deployment-cache-text02:/home/reedy# service nginx status * nginx is not running Starting nginx: nginx: [emerg] SSL_CTX_use_PrivateKey_file("/etc/ssl/private/star.wmflabs.org.key") failed (SSL: error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch) nginx: configuration file /etc/nginx/nginx.conf test failed
Getting this error now:
(Cannot access the database: Can't connect to MySQL server on '10.68.16.193' (111) (10.68.16.193))
Its not working yet though, I am still getting a different kind of error:
Error loading data from server:0: parsoidserver-http:HTTP:0, would you like to retry when I try to load VE.Should I raise it as separate issue?
@Ryasmeen the parsoid cache machine was in virt1009 which was down. Is back up now, can you check / verify if it still works?