In T254480#6197961, @ArielGlenn wrote:

I agree. Example: the first attempt at ./modules/dumps/templates/web/fetches/analytics/job/rsync_script.sh.erb was here https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/594773/2/modules/dumps/manifests/web/fetches/analytics/job.pp so you can see why erb was the natural approach in the end. Another file in there (modules/snapshot/templates/set_dump_dirs.sh.erb) literally just sets some vars to be picked up by the 'real' shell scripts that use it; I favour allowing scripts like these and strongly encouraging folks with long scripts as erb files to refactor them accordingly.

In T254480#6197931, @Volans wrote:

@CDanis yeah, sorry I noticed that after replying.
Totally agree with this approach, would be nice to have the CI check but we have a bunch of few liners that would become just more complex and probably not gaining a lot, see for example the first part of this list:

for scripts that are just one line i think it reasonable to just build the content in puppet, this is a bit of a cheat but for simple one liners i think its a fine compromise. I took a quick look through the files other then the oneliners, some of them don't actually need to be templates as there is no erb and a few others look like they are already config files so we could possibly just remove the shebang . however there are of course some that may be a bit awkward to fit into this model. Perhaps we should just whitelist them. It would at least mean going forward we don't have more violations added.

Another option is we could just ban templated scripts and add CI to reject any erb file with a shebang in it. This would mean updating any current templated scripts to read a new templated config. i.e. move all the dynamic bits to a template

Originally i wanted to do this checking in the Rake checks as that feels like the right place for them. however i cant think of a reasonable way to compile the erb files as we dont know the values of variables that would ordinarily be bound by the parent Puppet manifest. And different variables/hosts/roles etc could produce different outputs. As such im now leaning more to putting this in PCC, however i wonder if this should be in standard and in the main PCC code or if we should just have some helper scripts in utils. Regardless ill probably work on the latter to see how it looks but ideas welcome

i have made a first pass at the incident report. The main sections which could use some more input are

• impact
• detection
• documentation
• actionable

this is a bit better

quick glance suggest most are SC2086 & SC2006

This is a great idea, i think we may be able to do it in a rake task by adding something to task gen, the biggest issues with adding stuff like this is fixing all of the current issues. the wmf_style checks dose avoid this by only alerting on new errors but i have not looked at it. We should also add shellcheck for sh files in general

We could resolve this by having mcrouter talk directly to memcache in the other DC however this requires 1.5.13 which is not currently in buster.

1.6.6 is available in testing and seems to build correctly on buster.

I have configured memcache and mcrouter for CAS however there is currently an error. If CAS talks directly to memcache then all works fine. however when CAS talks to mcrouter we get and amplification storm before a timeout is sent to CAS. After speaking with @elukey we think the following may be happening

Is this a duplicate of https://phabricator.wikimedia.org/T251340 ?

@CDanis you still have the example text for the "Where did we get lucky?" and "Links to relevant documentation" sections. Also still need tasks for

Reviewed this looks good to publish, what is the procedure for that @CDanis ?

Resolve as per Moritz comment

when i checked stat1008 already had a AAAA record, not sure if someone fixed it or some issue in the script?

Currently mongodb is not listening on IPv6 however[[ https://phabricator.wikimedia.org/T180761 | mongodb is going away ]] so we should wait until that work has been completed

In T233947#6172089, @MoritzMuehlenhoff wrote:

Puppet is now deployed as a deb.

Awesome thanks

I think I'm leaning towards a few stable anchors in similar geographic locations to our PoPs. Maybe also a few root servers as well even though they're less apples-to-apples.

mcrouter 0.4.1 is now available in buster-wikimedia

I had a look at this on the CAS side and i think it would be doable to add some level of 2FA with an account in ldap. however i think most scripted users would only need to login to one service and not all services as such im not sure it makes sense to give scripted accounts a full SSO account as we would end up complicating things by applying additional ACL's for. Again we can restrict which services a specific account has access to on the CAS side however it seems to me that we are just adding complexity , both on CAS but also on the monitoring scripts which would need some type of session management.

Currently blocked waiting on mcrouter for buster

A new skin is now in place

This has been discussed and documented . Please reopen if further investigation is required

This is now in place, resolving

This is now in place, resolving

I asked @Zbyszko to look at this and they where unable to get to the root cause however as we now use an external tomcat instance this issue no longer affects us. As such im resolving.

Thanks for raising this, i have now added the unstable src repos to the build hosts.

Have added yamllint checking to the private repo

Which measurements to you plan to scrap?

• all measurements the anchors are performing outbound?
• the anchoring measurements directed at thes anchor.

I noticed that the ns0.openstack.codfw1dev.wikimediacloud.org. servers are configured as name servers for the 57.15.185.in-addr.arpa. zone. however you actually need to configure the 0/29.57.15.185.in-addr.arpa. zone

That's exactly what i have used, docker_entry.sh is just a cleaned up version of the FB packaging scripts

In T251574#6148741, @MoritzMuehlenhoff wrote:

(I tried to rebuild our current package for Buster, but 0.37.0 is incompatible with OpenSSL 1.1, this is likely fixed in current releases.)

Correct the 0.41 version in the docker process builds correctly

LGMT

@Andrew I have actully made a bit of progress on this here