https://discuss-space.wmflabs.org/ is offline:
Oops
The software powering this discussion forum encountered an unexpected problem. We apologize for the inconvenience.
Detailed information about the error was logged, and an automatic notification generated. We'll take a look at it.
No further action is necessary. However, if the error condition persists, you can provide additional detail, including steps to reproduce the error, by posting a discussion topic in the site's feedback category.
I forgot that I have access to this new instance.
I ran sudo ./launcher restart app and the site is back up now. The recent log messages (before restarting) were:
run-parts: executing /etc/runit/1.d/00-ensure-links run-parts: executing /etc/runit/1.d/00-fix-var-logs run-parts: executing /etc/runit/1.d/anacron run-parts: executing /etc/runit/1.d/cleanup-pids Cleaning stale PID files run-parts: executing /etc/runit/1.d/copy-env Started runsvdir, PID is 38 ok: run: redis: (pid 52) 0s ok: run: postgres: (pid 49) 0s rsyslogd: command 'KLogPermitNonKernelFacility' is currently not permitted - did you already set it via a RainerScript command (v6+ config)? [v8.16.0 try http://www.rsyslog.com/e/2222 ] rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Operation not permitted. rsyslogd: activation of module imklog failed [v8.16.0 try http://www.rsyslog.com/e/2145 ] rsyslogd: Could not open output pipe '/dev/xconsole':: No such file or directory [v8.16.0 try http://www.rsyslog.com/e/2039 ] supervisor pid: 47 unicorn pid: 77 (47) Stopping Sidekiq (47) Reloading unicorn (77) (47) Waiting for new unicorn master pid... (47) Waiting for new unicorn master pid... (47) Waiting for new unicorn master pid...
Thank you @Samwilson !
Next time please ping me (or maybe we should have a place to hang around and react in situations like these). OTOH we should have filed a task immediately but well, it was my midnight and I was happy when I left it running before going to sleep. This is the difference between proper SRE and, well, myself. :/
Looking at the time, there is a chance that you restarted when I was upgrading. My first attempt (from the web UI) ended in a "Upgrading..." stall. Then I went to the CLI and rebuild from there.
This is what was going on: https://meta.discourse.org/t/events-plugin-calendar/69776/483
Sorry! I hope I didn't stuff anything up! I opened this ticket and figured it'd go to you and whoever else is helping, and then didn't hear anything for 6 hours so figured you were all asleep and that I might be able to help (after Lego reminded me that I had access to do so). I hoped I wouldn't be stepping on anyone's toes, but I think probably just when I was thinking that was when you were doing the upgrade.
Is there a chat channel for Space? IRC? Riot?