I recently got an email;
"systemd killed by Wheel of Misfortune on tools bastion"
It would be good if the actual host could be identified. There's at least 4 machines I know of which could be used as a Toolforge bastion (two of which I use on a regular basis). Identifying the full hostname would make this notification more useful.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Andrew | T289883 cloud-vps: actively deprecate/phase out use of Debian Stretch | |||
| Resolved | • taavi | T306099 Cloud VPS "tools" project Stretch deprecation | |||
| Resolved | • taavi | T306100 Cloud VPS "toolsbeta" project Stretch deprecation | |||
| Resolved | • taavi | T275864 Toolforge: migrate to Debian Buster or later | |||
| Resolved | aborrero | T275865 Toolforge: migrate bastions to Debian Buster | |||
| Resolved | • Bstorm | T266300 Establish a systemd timer to remove long-running processes on the bastion in a random and somewhat friendly way | |||
| Resolved | • taavi | T282949 Wheel of Misfortune should provide bastion host details |
If it's helpful, this specific case was on tools-sgebastion-11 / dev-buster.toolforge.org.
How was systemd even killed? The script is configured to not touch it at all.
Got another one today:
Your process `bash` has been killed by the Wheel of Misfortune script.
just playing around with psutil, I see it lists '/usr/bin/bash', but SHELLS in that script includes '/bin/bash', not ''/usr/bin/bash'.
Is this happening only on the buster bastions? psutil changed a lot over time, and the stretch version is very old. I had to develop directly on the bastion to work with it.
That said, it should not be hard to include the hostname, which would help answering my question.
Change 693485 had a related patch set uploaded (by Bstorm; author: Bstorm):
[operations/puppet@production] toolforge bastion: do not run wheelofmisfortune on buster yet
Change 693487 had a related patch set uploaded (by Bstorm; author: Bstorm):
[operations/puppet@production] wheel_of_misfortune: include the hostname in the email
Change 693487 merged by Bstorm:
[operations/puppet@production] wheel_of_misfortune: include the hostname in the email
@RoySmith Have you got any examples with the host in them so far? If not, I can go ahead and deliberately create test cases.
No, I've been well-behaved lately. But, I'll be happy to nail a tmux session up on one of the bastions for as long as it takes to generate an example if you like :-)
Just got one:
Your process bash has been killed on tools-sgebastion-11 by the Wheel of
Misfortune script.
Change 693485 merged by Bstorm:
[operations/puppet@production] toolforge bastion: do not run wheelofmisfortune on buster yet
Since that confirms the suspicion that the problem is that the problem is that buster has a much later library for the python that is using, I've merged my patch to be selective about it for now, and I can try to to port the script to buster. I also manually yanked the service out of systemd on the buster bastions.
Honestly, on buster, it will probably work more like my laptop does, making it easier. On stretch I had to develop in a VM because the versions were so old.
Change 729593 had a related patch set uploaded (by Majavah; author: Majavah):
[operations/puppet@production] toolforge: wheel of misfortune: dry run on buster
Change 729593 merged by Bstorm:
[operations/puppet@production] toolforge: wheel of misfortune: dry run on buster