Whenever a job is completed, the Gearman plugin properly put the node offline.But Nodepool doesn't notice it.
The reason is that the Nodepool / Jenkins ZeroMQ bus is dead. Jenkins no more listen on port 8888 and Nodepool kindly ignore the fact it can't receive any ZMQ events.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | hashar | T120668 Nodepool does not delete nodes in a timely manner (ZeroMQ dead) | |||
| Resolved | hashar | T120669 Monitor Jenkins master listens on TCP port 8888 (ZeroMQ) |
That is probably because when restarting Jenkins there were duplicate process. Thus on the second instance ZMQ would have failed because TCP port 8888 was already bound.