So i started implementing this as an Icinga check. Unless you specifically meant for it _not_ to be an Icinga check and just run in cron and send email if it detects them.

Alex said on Gerrit: "This patch makes it possible for a host to not be in our icinga installation configured. Which is not what T151632 originally asked for. I am not sure we want to have "ghost" hosts in our infrastructure (we 've had that in the past) that are there but are not monitored." hmm..

done now: cerium xenon, praseodymium (cassandra test cluster)

status is: in puppet as role::spare, in DHCP and DNS. remnants in mysql grants (https://gerrit.wikimedia.org/r/369832) , all else is removed

assigning it from me to pool. it can now be finalized and iridium can be shutdown and wiped. I can't do all the non-interruptable steps myself due to lack of switch access.

And here are the things you can do as root:

Hi @Addshore you have been added to the group now, as requested. i merged it on puppetmaster.

^ Merged but had to revert it as well. After merging the service would fail to start (tested on gerrit2001 before doing it on cobalt). So this is _not_ applied currently.

removed exim alias in private repo.

This is where this has been used:

@bcampbell Thank you, i'll just remove it then (unless Winifred tells me otherwise).

@bcampbell or @bbogaert Can we move legal-tm-vio@wikimedia.org to Google please? We did almost everything else but this is one of the few left. Cheers, Daniel

@bbogaert maybe we can give it another try now that the Wikimania travel season is mostly over? I don't know why it failed but there must be something that makes these aliases different from all the others we moved without such issues.

@bcampbell or @bbogaert Could you check for me on the Google side if you have any group/alias/account called studentclubs@wikimedia.org nowadays? Thanks!

Thanks Aklapper. I mailed Ariley and Winifred (blindly trying the old @wikimedia.org addresses).

@demon Last call before we are actually killing iridium and wiping the disk?

This removed the HTTPS and LE cert check for netmon2001 (on einsteinium) based on netmon1002 being set as the netmon_server in ./hieradata/common.yaml.

If you have an Android phone you can install the Google Authenticator app from https://play.google.com/store/apps/details?id=com.google.android.apps.authenticator2&hl=en to scan the barcode on the page and then use it to enter the second factor at login.

There is some cronspam from stat1006:

The host is down again (since 1d 8h 34m 8s)

It's back but with "Difference in size anonymous" vs. "Difference in size authenticated" this time.

@DarTar Robh was out today, i think we can get this done tomorrow though! thanks for your patience.

Thank you very much chasemp. The goal is just to silence the cron job one way or another then.

@chasemp ^ In https://github.com/wikimedia/phabricator-tools/blob/master/wmfphablib/rtlib.py can't find rtppl in "from rtppl import ppl as users" . Do you know where rtppl is?

this also appears here: https://www.mediawiki.org/wiki/Phabricator/Trello_to_Phabricator/Research_import#Debug_output

In T127524#3528169, @Aklapper wrote:

(Also see T172558: Phlogiston data not updated since Aug 2 for recent issues with the dump file.)

In T127524#2087788, @ArielGlenn wrote:

I see cron output today:
NOTICE: rtppl not found!

Is this an issue? If not, please go ahead and close this again.

@Platonides No, it can't be closed. It's still unknown if these are used. If studentclubs@ is a Google group, the easiest solution would be if OIT can just add studentgroups@ as an alias there. Then we could remove it on our side and close the ticket.

@HakanIST and @Freddy2001 joined the workshop and we talked about git-review and Gerrit.

Great! Thanks for confirming. Closing as resolved then.

I think since we already said yes for -operations, -tech shouldn't be a problem is kind of implied.

I had to follow the commands described on https://wikitech.wikimedia.org/wiki/Varnish#How_to_execute_a_ban_across_a_cluster_with_cumin with adjusted req.url ^/resources. Otherwise the 404 was cached, even though the redirect was already working.

This is working now, after:

and https://gerrit.wikimedia.org/r/#/c/370123/

reason:

https://gerrit.wikimedia.org/r/370518 by paladox merged as well, to fix another cron that sends the stats mail to admins

@JbuattiWMF i'm in in middle of travel now @RobH could you help out by any chance? would be great, thank you!

In T137928#3505993, @jcrespo wrote:

I think we added m3-master.codfw.wmnet not a long time ago (maybe it was another misc server)

nrpe::monitor_service has parameter "timeout" for this.

@Antoine2711 Is this working for you? Any issues? I will be in travel to Wikimania soon, let me know if i can help with anything else.

Thanks! @jcrespo. Unfortunately i was wrong, there would have been cross-dc queries when you accessed the URL https://phabricator-new.wikimedia.org/diffusion/ ,even without phd service. Jjust the / URL wasn't working. I stopped Apache and disabled puppet to prevent this from happening again. I will follow-up with a patch to make sure Apache is always stopped on the passive server.

Just to make sure, I checked with tcpdump port 3306 and there was nothing, even when Apache is up. (on phab1001 there is constant activity).

Yea, so the service isn't up, but the config is indeed like this:

The answer should be done, as the phd service isn't running there and nothing changed about phab2001 recently.

@Jayprakash12345 Hi, That's a different project, run by different people. That would be the "Analytics" open checkbox on T168765.

Both servers are the same model, same number of CPUs, same RAM, same puppet role.. just that phab2001 is a lot LESS busy...

In T137928#3502839, @Paladox wrote:

Is codfw ment to be slow?

< mutante> !log phab2001 - changing UID/GID for phd user from 997:997 to 498:498 to make it match phab1001, to fix rsync breaking permissions. (rsync forces --numeric-ids when fetching from and rsyncd configured with chroot=yes). chown -R phw:www-data /srv/repos/
< mutante> !log "reserved" UID 498 for phd on https://wikitech.wikimedia.org/wiki/UID | phab2001: find -exec chown to fix all the files , restart cron
< mutante> Notice: /Stage[main]/Profile::Phabricator::Main/Rsync::Quickdatacopy[srv-repos]/Cron[rsync-srv-repos]/ensure: created
< mutante> the change is fine.. there is something that stops sync though
< mutante> !log phab2001 - installing various package upgrades, apt-get autoremove old kernel images
< mutante> !log phab2001 rebooting
< mutante> !log phab2001 - removed outdated /etc/hosts entries, that fixed rsync, syncing /srv/repos/ from phab1001

please keep stalled for a few more days like this and don't shut down. we are a spare but don't want the disk wiped just yet, and phab-admins still have shell.. just in case

monitoring added:

http://wikistats.wmflabs.org/display.php?t=wx and click on description column to sort alpha

There is also https://status.wikimedia.org on this. It had the same issue, the cert was about to expire, but for that one we didn't have monitoring?

added to root's crontab:

I renewed the certificate with this command:

T152129 has been resolved

00:21 < mutante> twentyafterfour: try again
00:22 < twentyafterfour> mutante: works!!!