Thank you @herron ! Yes, confirmed. No more duplicates right now.

In T251726#6109023, @Vgutierrez wrote:

IMHO 7 / 3 is not enough for the unified cert even when LE is the issuer considering our anti clock skew measures and that acme-chief should issue the new cert 30 days before the valid one expires

To SRE doing clinic duty: This is an easy one. Really just need to move the existing user from group "restricted" to group "deployment" and the other steps are all done from the previous request.

percentage of jessie systems left as of today (because we were asked): 4.2%

mw2187, mw2188 are new canary appservers, replacing mw2271, mw2272

I'm afraid this is one of those tickets that you can never really close because at any given time there will be 1 or more feeds with errors. Some will be fixed later, some will go away, some will change URLs due to new software... It is just a constant maintenance and one has to check the logs every once in a while. I am not sure I want to keep a ticket open forever though.. .so at some point it will have to be "good enough for now".

The 2 boxes not checked on this ticket are meanwhile working again. A good example why deleting all feeds with an occasional 404 too quickly is not the best idea.

Technically resolved because we made more than enough room for the 5 (not 15 anymore, 10 were used for T252185) servers.

23 servers from rack C3 have been decom'ed. mw2150 through mw2172. (lower part of the rack)

@Papaul 20 servers from rack C3 have been decom'ed. mw2150 through mw2169. (lower part of the rack)

reopening because i am decom'ing servers in T247018 and that included some canaries.

These were changed in https://gerrit.wikimedia.org/r/c/operations/dns/+/595959/2/templates/wmnet

As disussed on IRC, i'd rather not use /srv/docroot and would prefer instead if we can clean out the repo called "docroot" to actually just contain files for the docroot.

Is there a specific thing we are waiting for?

@Mjohnson_WMF I temporarily added an alias for grants@ to myself on the mail servers and then hit the password recovery form again. I see it has now delivered an email to you. Please check if you received it and you are now unblocked.

It seems like the previously existing grants@wikimedia.org Google group has been deleted as part of T191881 or otherwise.

@Mjohnson_WMF @bd808 I just tested sending the password recovery mail to you while watching log files on the server and I found:

wmcs team reported not being able to do installs from the cloudvirt VLAN (cloudnet1004)

please also see T246945 which might conflict with this effort and introduce yet another place for documentation rather than having a single place for developers to go to

In T246945#5945175, @CCicalese_WMF wrote:

This has happened and an announcement has been sent to ops and wikitech-l lists.

@QChris @hashar Do you see a new for a DNS name like "gerrit-new.wikimedia.org" or someting similar which then can be used for the baseurl in this example and other things?

@lmata Welcome! This is done. You should now be able to log into Icinga, Grafana etc.

unassigning from me because i got the "cookie-licking" warning from T228575

I think option 3) is the easiest of all, has no risk to break existing checks and we already have many different check_commands using check_http in different ways so another one should not hurt us.

Giving it back to the pool and setting to stalled because of ongoing discussion whether this should be on a dedicated VM or on mwmaint.

Thanks for taking care of this @RLazarus I can confirm Max's user exists on the Phabricator prod server, is in the new group and that group has the sudo privileges to run the requested comment.

from the log files on bast1002 and cumin1001 I can see there are 2 different keys involved.

In T249506#6152449, @Nintendofan885 wrote:

In T249506#6152442, @Dzahn wrote:

added to Wikidata: https://www.wikidata.org/wiki/Q94952969

I already created an item on it a couple of days ago so I've just merged it https://www.wikidata.org/wiki/Q94700087

In T191182#6153918, @bd808 wrote:

@Dzahn's explanation ignores all of T191182#4935787 and the literal fact that there are nearly 350 Diffusion repos for Toolforge tools which have been created through Striker. My declining moving 1 of these 350 repos to gerrit was not the root cause of the blockage here.

added to Wikidata: https://www.wikidata.org/wiki/Q94952969

Added to wikistats.wmflabs.org

https://www.wikidata.org/wiki/Q94694371

https://www.wikidata.org/wiki/Q94952446

re-added now that the wiki has been created.

Thanks for reporting. Likely it will be similar to:

+---[RSA 2048]----+ +---[ECDSA 256]---+ +--[ED25519 256]--+

+----[SHA256]-----+ +----[SHA256]-----+ +----[SHA256]-----+

A VM called malmok.wikimedia.org has been created and can be used now. Currently it has the "insetup" role in site.pp.

@ssingh The VM has been created (now with public IP).

In T252703#6152113, @soworu wrote:

SOworu worked. Now in Turnilo. Thanks.

@soworu I found the following comment in the Apache config of superset:

@soworu Looks like it works now, i think i just saw you login in the log files, am i right?

@soworu Try again now, please. Looks like i forgot to add the -01 part when adding you the right group. Sorry about that.

Please try the following variations of the username including the capitalization:

In T252703#6152022, @soworu wrote:

In T252703#6142940, @Dzahn wrote:

@soworu Use the same user/password you used on https://wikitech.wikimedia.org when you created your account there.

I still can't login into Superset and Turnilo using my Wikitech login details. Can you help?

In T165348#6144076, @jcrespo wrote:

I would like to separate better "things that can wait" from "outages or potential outages" on different dashboards.

This is apparently stalled because T252910 has been declined with a reason that there were only few commits in a specific repo. That seems to cause a dependency cycle though which i find pretty unfortunate because what EddieGP describes here is totally legit and really the worst possible outcome.

These 4 hosts have been reimaged and now have RAID5 instead of RAID1 after gerrit:597261

Handing back over for the next "init" command steps you have mentioned are needed next.

@akosiaris All of these hosts have RAID5 now:

Enabled remote IPMI on these machines which was disabled but is needed. (wikitech how to

@RobH @Cmjohnson I noticed by chance there are more ganeti machines beyond ganeti1018. ganeti1019-ganeti1022 are in netbox but i don't see a racking ticket for them. Should there be one?

@RobH Remote IPMI was disabled on these hosts which popped up when i tried to run the reimage cookbook (to change software RAID level from 1 to 5) and it failed.

In T224591#6150943, @hashar wrote:

@Dzahn would you be available at the beginning of next week (Monday/Tuesday)? (we can sync up over irc to find a good time)

In T252932#6150527, @MBeat33 wrote:

In T205361#6150835, @Krinkle wrote:

Also, is it MediaWiki/1.html or MediaWiki/rev/1.html. I've seen both versions. It seems we're back to the former?

@QChris I added you to the nda group. You should now be able to login to Icinga.

Deleted it again for now because adding it before the wiki has been created cause more issues like the one linked above and then T253115.

@Xqt I have deleted "awa" from the wikistats db again. I will reopen the task to re-add it once it has actually been created.

I think the best solution is probably that i don't add sites before they are created and simply delete it for now.

done! the module has been removed

I did the latter and set total= and good= to 0 in the db. Did that fix it, @Xqt?

In T252869#6140065, @Aklapper wrote:

(If this requires creating the wiki then wiki creation is a subtask not a parent task, as the subtask first needs to be solved)

I could also just set "total" to 0 manually for now so that it has a number.

I did not even know "wikistats_tests" is a thing.

Is the requested hostname "homer" a copy/paste error?

This has happened. The module is gone now.

I think both Commons and Youtube, depending on licenses, would be good places to store the actual video. The wikiworkshop site could then link to them or embed them directly.

I fixed this by creating "role::simplelamp2" which uses mariadb instead of mysql. This has been applied on glampipe instead of the old simplelamp role and puppet runs fine again.

This is done.

Ok, resolving. Note: The thresholds are currently set to 240 hours (10 days) for WARN and 480 hours (20 days) for CRIT.

I think "how to handle Icinga warnings" is not something specific to this task about monitoring screens.