In T261693#6663005, @Ottomata wrote:

Interesting, thank you! It's good to have that on the profile class.

unfortunately i had to revert the other change in the chain :(

however note that that spec test is dependent on a general refactor of the profile spect test which is in relation chain

In T229397#6650923, @jbond wrote:

I think we could probably include all devices under something like

netbox::network_devices:
  fqdn:
    * => $metadata

In T268802#6662467, @ayounsi wrote:

Another useful thing would be to duplicate the production "networking" puppet fact, to have LLDP, IPs, etc. exposed there as well.

In T268974#6662092, @jcrespo wrote:

The previous patch did not have an effect- backups continue to be executed on eqiad, but codfw ones (cumin2001) do not run at all- they are not even attempted, according to journalctl.

Is there a way to "kick"/reload a timer (daemon-reload seems to have no effect).

Seems like the errors in the console are somewhat expected. however I wonder if we can make this better somewhere in the apache/mod_auth_cas configuration

in fact observability is already tagged, @fgiunchedi wodner if this could be a more general issue?

Do you get this error on all expressions, a specific expression or spasmodically? have also tagged observability in case there is something other then CORS in play

Looking at the code i think it tries CORS, retries a few times and then falls back to no-cors which means that it eventually gets its self in a working state. This seems to be similar to what i observe as i don't ever get logged out or see an error in the GUI but i do see issues like the one reported above in the console

This looks like the issue we are hitting https://github.com/prymitive/karma/issues/1157

I have not been able to recreate this, is this still causing an issue?

The main issue here is that both puppet-lint and puppet syntax validate are too light weight to pick up issues like this as they are designed to work on individual files and as BooleanXXXXX* (e.g type BooleanXXXXX = Boolean) could be defined else where in the manifest its not obvious to theses tools if that's an error. I would say the standard way to pick up issues like this is to write a spec test for the class, see the example here, which does pick up the issue. however note that that spec test is dependent on a general refactor of the profile spect test which is in relation chain

This looks like it was an artefact of the python2 -> python3 migration. I have pushed the PS that drops all the hacks added to support both python versions and this is running correctly now.

Another use case for netbox data in puppet is exposing the network devices so they can be used in configuration such as icinga parent mapping and turnilo data augmentation. @ayounsi perhaps there are others in this space as well.

i have applied a fix from upstream and created a new debian package (1.2.0+git20160825.89.7fb22c8-3+deb10u2) i have installed this to pki2001 and now see AKI's in the database will test further tomorrow

No Ideas however i did notice that i got a 405 earlier today before failing over idp to idp2001. idp1001 was on an older software version so its possible there could have been a config mismatch. That aside i think the next thing to try would be to call the JS code manually via the console

im nt that familure with chromes error logging net::ERR_FAILED looks like it may be a more generic network error then specificity with CORS which should have given a clear HTTP/1.1 403 Forbidden. I think the next thing to try is to trigger the pre-flight check from manually from the JavaScript console as it seems to work fine with curl*

This is working as expected thanks

@Marostegui I had to add a firewall rule but all looks good now, can be closed from my end. thanks

In T268327#6644232, @Marostegui wrote:

@jbond we already have a cas user that has access to cas_staging database. Do you want to re-use that user/password or use a different one for the new casdatabase?

In T268104#6636631, @jcrespo wrote:

I realized I closed this ticket because the part I reported (backup failures) is fixed for me. Feel free to reopen and edit the task to add the RFO/root cause (what I usually do is edit the ticket title with a "(was: <original title>)". Up to you.

In T268327#6636565, @LSobanski wrote:

@jbond What is your preferred delivery date for this?

In T268329#6636560, @LSobanski wrote:

@jbond What is your preferred delivery date for this?

for this one ~1 week would be nice however it wont really become a blocker for me untill we approch the end of the Q ~4weeks

In T268233#6633781, @Kormat wrote:

Just reproduced this in chrome, and got this message:

Access to XMLHttpRequest at 'https://idp.wikimedia.org/login?service=https%3a%2f%2fthanos.wikimedia.org%2fapi%2fv1%2fquery%3fquery%3dsum_over_time(mysql_exporter_last_scrape_error%255B5m%255D)%2520%253E%25201%26dedup%3dtrue%26partial_response%3dtrue%26time%3d1605799406.496%26_%3d1605795670420' (redirected from 'https://thanos.wikimedia.org/api/v1/query?query=sum_over_time(mysql_exporter_last_scrape_error%5B5m%5D)%20%3E%201&dedup=true&partial_response=true&time=1605799406.496&_=1605795670420') from origin 'https://thanos.wikimedia.org' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.

The last open question is how much is worth to invest time in icinga-related stuff based on the current plans for alertmanager.

I spoke with Filippo and its expected that scheduled downtimes will continue in icinga for some time as such it probably is worth fixing. I have made a quick [incomplete] PS to add this, let me know what you think if you get a sec

could be related to https://phabricator.wikimedia.org/T267186

In considering this task more I think one complication is an SSL certificate for CN=puppet which is copied to every server so that it can be used to provide SSL Client authentication to the https://puppet:8140/ endpoint. Note this is not the Root certificate which has a CN of Puppet CA: palladium.eqiad.wmnet.

In T267186#6633298, @fgiunchedi wrote:

Different error message this time in the browser's console, but something has definitely changed!

I have just pushed another change which i think should fix this are you able to test again

I think the best way forward here would be to run the script from one of the cumin hosts. Then we could use spicerack to query the icinga downtime status of a host. however Saying that i just took a quick look at the spicerack API and i couldn't see a function to get the current downtime status only functions to add and remove dowtime. tagging @Volans in case i missed something but also to ask if its worth adding the functions from this script to spicerack. My gut feeling is no this script is a bit to specific in the way it queries the puppetdb backend but perhaps not.

however i wonder if we should instead base the idp_primary/idp_failover parameters on a DNS lookup for dig CNAME idp.iwkimeida.org @ns0.wikimedia.org so there is only one source of truth?

We should definetly avoid qurying the auth server directly, otherwise puppet could fail over the system before all DNS cache's have expired. Querying a cache also has a race condition, one we can probably game to ensure we win, although its still not ideal

This has now been deployed to production and at the pre-flight test seems to be working. Are you able to test if the underlining issue is resolved

In T256454#6630672, @jcrespo wrote:

Answering the question I made on the previous patch, my suspicion is that it should avoid future issues like T268104 after deploy, but I think there is more not-explicit dependencies on install.

In T268104#6629682, @jcrespo wrote:

I also noted the following warning on seaborgium, serpens AND sretest1002. Related?

WARNING: Failed to apply catalog, zero resources tracked by Puppet. Could be an interrupted request or a dependency cycle.

I have done some initial testing an i think we should just drop the ssl config from the puppetmaster backend servers and let it use the default. the backends dont ,ever do CA operations so they dont need to worry about that and the frontends have the CA dir rsynced

checking the following one the backends shows that the keys are all different which points to the puppet master process generating theses keys when it first needs them as appose to reciving them from the configuered puppet master

Tagging https://gerrit.wikimedia.org/r/c/operations/puppet/+/386666 as although its slightly different it seems to be around the same bit of code

https://github.com/github/octocatalog-diff/pull/226

in relation to deployment-cache-upload06 puppet runs successfully but the following command fails on each execution @Vgutierrez may be able to quickly spot the issue

In T267006#6624104, @jbond wrote:

deployment-wdqs01 should be fixed with https://gerrit.wikimedia.org/r/641171 however there is now a conflict wit a local commit which needs deleting:
c503964991 [LOCAL HACK] Fill placeholder for etcd::autogen_pwd_seed from root@deployment-puppetmaster04:/var/lib/git/operations/puppet
`

This has been corrected also needed a few extra yaml settings

to fix deployment-logstash03:

• rename role::logstash::apifeatureusage to profile::logstash::apifeatureusage in horizon
• rename role::logstash::collector to profile::logstash::collector in horizon
• add modules/secret/secrets/certificates/kafka_logstash-eqiad_broker/truststore.jks to private repo
• add the following to the kafka_cluters global hiera value in the deployment-prep puppet project config

deployment-wdqs01 should be fixed with https://gerrit.wikimedia.org/r/641171 however there is now a conflict wit a local commit which needs deleting:
c503964991 [LOCAL HACK] Fill placeholder for etcd::autogen_pwd_seed from root@deployment-puppetmaster04:/var/lib/git/operations/puppet
`

deployment-mdb01 fixed

deployment-kafka-* servers where all working fine when tested

Adding the following to match production has fixed deployment-ores01

I have added the following to the puppet config in horizon, this equates to the config which is in the current exim.conf file

the pre-flight check is now working on idp-test, ill now enable on production

In T267439#6609779, @thcipriani wrote:

Noticed that puppet stopped running on beta's deployment-cache-text06 due to c871b021a44108bed44cd044f5177b075ecb322c

After attempting a fix (live hack on deployment-puppetmaster04 changing:

modules/confd/manifests/init.pp

-    Stdlib::Fqdn     $srv_dns       = $facts['domain'],
+    String           $srv_dns       = $::domain,

I'm curious if this change made any difference? as i dont think it should (but there could be something with horizons backed which im not familiar with)