A "Requires=ircd.service" in the ircecho unit would probably fix this.

yeah I think it would, we should test ircd restarts while ircecho is running

yep, i'll test it in labs. (we know setup works there now. i have added fake private data in labs/private, Krenair has tested it and fixed more. so that should not be hard now :)

(but don't restart ircd in prod, users hate it :)

thanks @Dzahn @Krenair ! I think ircecho should react better to exceptions, generally except Exception is a bad sign in python, possibly just log the exception and exit and have systemd restart it or even better reconnect

for the record: created labs project "ircd" with instance "udpmx-01", added puppet group / role class

before reboot, things all work now in labs

root@udpmx-01:/etc/systemd/system# systemctl status ircd
● ircd.service - IRCd for Mediawiki RecentChanges feed
   Loaded: loaded (/etc/systemd/system/ircd.service; disabled)
   Active: active (running) since Tue 2016-05-10 21:58:50 UTC; 1 day 1h ago
 Main PID: 1474 (ircd)
   CGroup: /system.slice/ircd.service
           └─1474 /usr/bin/ircd -foreground

May 10 21:58:50 udpmx-01 systemd[1]: Started IRCd for Mediawiki RecentChanges feed.

root@udpmx-01:/etc/systemd/system# systemctl status ircecho
● ircecho.service - IRC bot for the MW RC IRCD
   Loaded: loaded (/etc/systemd/system/ircecho.service; disabled)
   Active: active (running) since Tue 2016-05-10 21:58:48 UTC; 1 day 1h ago
 Main PID: 1177 (python)
   CGroup: /system.slice/ircecho.service
           └─1177 python /usr/local/bin/udpmxircecho.py

May 10 21:58:48 udpmx-01 systemd[1]: Started IRC bot for the MW RC IRCD.

Info: Caching catalog for udpmx-01.ircd.eqiad.wmflabs
Info: Applying configuration version '1463008283'
Notice: Finished catalog run in 7.02 seconds

..but the bot is not on the irc server. after restarting the service it is connected. checked with irssi /whois rc-pmtpa, so it's as reported.

then added the Requires=ircd.service into the ircecho unit file, disabled puppet, rebooted instance..

but now both services are " Active: inactive (dead)" ...:p looking ...

I tried a couple different combinations with Requires= , After=, Before= etc in both unit files but i haven't found a combination yet that results in both services being up after reboot AND the bot being logged in without a manual service restart. also tried letting udpmxircecho.py die if it can't connect/ join channels but it just sits there and still tries to connect even when ircd is not running.

Change 290588 had a related patch set uploaded (by Dzahn):
irecho: add systemd require/after to start after ircd

https://gerrit.wikimedia.org/r/290588

Change 290588 merged by Dzahn:
ircecho: add systemd require/after to start after ircd

https://gerrit.wikimedia.org/r/290588

so this solved it in so far that after a reboot the next puppet run will start both the IRC server and the ircbot and the bot will be logged in ! tested in labs on udpmx-01 instance

May 25 01:34:15 udpmx-01 systemd[1]: Starting IRCd for Mediawiki RecentChanges feed...
May 25 01:34:15 udpmx-01 systemd[1]: Started IRCd for Mediawiki RecentChanges feed.
May 25 01:34:15 udpmx-01 systemd[1]: Starting IRC bot for the MW RC IRCD...
May 25 01:34:15 udpmx-01 systemd[1]: Started IRC bot for the MW RC IRCD.

puppet just starts the ircecho service and then both are up:

Notice: /Stage[main]/Mw_rc_irc::Irc_echo/Service[ircecho]/ensure: ensure changed 'stopped' to 'running'
Info: /Stage[main]/Mw_rc_irc::Irc_echo/Service[ircecho]: Unscheduling refresh on Service[ircecho]
Notice: Finished catalog run in 12.31 seconds

● ircd.service - IRCd for Mediawiki RecentChanges feed

Loaded: loaded (/etc/systemd/system/ircd.service; disabled)
Active: active (running) since Wed 2016-05-25 01:39:58 UTC; 21s ago

● ircecho.service - IRC bot for the MW RC IRCD

Loaded: loaded (/etc/systemd/system/ircecho.service; disabled)
Active: active (running) since Wed 2016-05-25 01:39:58 UTC; 28s ago

@fgiunchedi btw since we once talked about this on IRC. meanwhile i added you to the labs project for this.

https://wikitech.wikimedia.org/wiki/Nova_Resource:Ircd