Spent some time on IRC with @jbond reproducing this and indeed puppetdb-populate will fail repeatedly for new hosts until performing a run using an empty manifest to populate facts. Then subsequent runs succeed.

In my testing simply removing instances of "index_options":"docs" from the logstash template addresses the issue, please see https://gerrit.wikimedia.org/r/583112

Hello! The multimedia-team list has been renamed to structured-data-team, and redirects/forwarding have been put into place. I'll transition this to resolved as a soft close, but please re-open if any follow up is needed. Thanks!

In T247538#5980045, @MoritzMuehlenhoff wrote:

Another low-hanging fruit is to reduce the SSH check for the mgmts I think: It currently runs every minute, but the non-avail of the mgmt sshd has no end-user impact, so checking them hourly should be good enough? That would slash the 30087 checks from above by a lot.

https://gerrit.wikimedia.org/r/579329 seems like low hanging fruit that could help reduce load

Thanks @Cmjohnson! Will resolve this and track service setup in T247376

In T240882#5937405, @Papaul wrote:

@herron is it possible to create another task to track this down and close the racking and setup task?
thanks.

@EBernhardson thanks!

In T247014#5946926, @EBernhardson wrote:

I hope you don't mind, i started up a test on logstash-next with logstash-2020.02.20 recreated as ebernhardson-size-test where i think i've adjusted all text fields to have index_options: docs and copy_to: all, along with the all field defined as only a text field with standard indexing. A reindex is running to copy all the docs over and we can see how the sizes differ with a little less guessing on my part. It hasn't finished yet, but with 2M docs indexed it looks like the change might only be from ~800 bytes/doc to ~950 bytes/doc.

Thank you @Gehel @EBernhardson @dcausse @colewhite for looking at this!

Additionally, before moving on to max_clause_count I had experimented with settings like "default_field": "*" and "default_field": "message" in Kibana config query:queryString:options but the errors persisted.

In testing I was able to work around this by increasing indices.query.bool.max_clause_count to a value greater than the number of fields matched. This comes with some tradeoffs wrt resource utilization, but it does resolve the issue.

Thank you!

In T240881#5912321, @wiki_willy wrote:

@herron - is there a specific date that you need these by? We can adjust our priorities and the need by date of this task to meet that.

Hi @wiki_willy, do you know what the ETA is for these hosts?

Looking a bit closer I think this is happening because the nodes in labs are assigned their roles/profiles/etc via the external node classifier in horizon, which isn't making the call to role() as we do in prod and so $::_role isn't set in the process.

Similar to recent issue T245725

I've cherry picked https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/571239/ on deployment-puppetmaster04.deployment-prep.eqiad.wmflabs (and made a minor change in patchset 9, since logstash was complaining about the quotes). The config loads ok in logstash.

Learned today that T243226 is tracking the current beta cluster puppetmaster issues

Turning off debug logging on puppetmaster04 (I set logdest = /dev/null in /etc/puppet/puppet.conf) has helped with the disk usage, and sluggishness issues. But sadly puppet runs are currently failing with:

Fwiw I do see logs flowing into logstash-beta generally, but puppet was broken in the beta cluster because the master filled its disk. Puppet master on deployment-puppetmaster04.deployment-prep.eqiad.wmflabs seems to be logging at debug level, making puppet runs super slow and rapidly filling the disk. I don't have time at the moment, but if still broken in the morning I'll take a closer look.

Hey @jijiki, usually to test/validate filters like this I'll cherry pick or live-hack the logstash config on the beta cluster and generate the desired traffic there to see how logstash behaves. There are some details at https://wikitech.wikimedia.org/wiki/Logstash#Beta_Cluster_Logstash

In T239732#5807275, @Papaul wrote:

@herron thanks in that case you can just add the server to site.pp with the role ( spare::system) and assign the task to @jbond

Hey @Papaul, I don't think there is any specific urgency to this and it can wait until he's back, but if it needs to go sooner I could work on it.

This should be fixed now.

In T240906#5774687, @herron wrote:

I'd like to rule out possible hardware issues by migrating this VM to another Ganeti host, and seeing if that makes any improvement.

The row_A ganeti group is running low on memory capacity (please see T239151#5707691) . Should we allocate a few of these new hosts to expand the existing row_A ganeti group?

The eqsin ganeti cluster is now up and running, and a first VM netflow5001 has been created.

ganeti-test.wikimedia.org VM has been created on row_C, and I've uploaded a patch to assign it role::gerrit with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/562587/

In T240906#5762100, @jcrespo wrote:

There has been multiple of mx1001 issues lately (even if that is unreliable, it is worth noting). My suggestion would be, at least initially, to detect the same issue, if real, on icinga.

Hi @Nuria, a friendly ping/bump for approval on this. Happy new year!

I'd like to edit the form but don't currently have permission. Primarily I'd like to add the clinic duty checklist and clarify a few prerequisites for the requestor to complete. These are things that we currently do manually via back-and-forth comments. Adding them to the template should save time on every request. I'd like to update the template like so:

Thanks for the update @Kris_Litson_WMDE

Sounds good @jcrespo, please pass back to me when you've received the export and uploaded it to the mailman host and I'll see what I can do to import. Thanks!

Hello! Looping in @RStallman-legalteam to coordinate getting your NDA on file.

Removing the SRE-Access-Requests project tag for now. Please update and re-add if/when any further action is needed. Thanks!

esams and ulsfo are online now, and eqsin should be shortly. Not sure if it's best to do all at once, or per-site, but wanted to get a task created to keep tabs on it.

The esams ganeti cluster is now up and running, and netflow3001 has been created there as a first VM.

These hosts have been reimaged with buster, certs created, and patches uploaded to enable ganeti.

Hey @RobH, T229243 is encouraging. How are these hosts looking now?

Actually since netflow4001 is not yet puppetized the instance has been shut down. https://gerrit.wikimedia.org/r/559330 should unblock the first puppet run, and can re-start the instance after its merged.

For sure, but its a work in progress currently. Basically I'd like a sanity check that the manual steps make sense and aren't already automated, or are better handled, in a way that I'm not aware of.

The ulsfo buster ganeti cluster is up and running now, and netflow4001 has been created there as a first VM.

Looked into these alerts a bit, and pulled the source IP addresses for these checks from watchmouse, but I don't see these IPs appearing in the mx logs. I think it is because the exim mx logs are not currently detailed enough. So I'll make the logs a bit more verbose and review again after more log information has been gathered.

Yes, we will need a second logstash stretch instance, and to migrate the Kafka broker ID from deployment-logstash2 to the new host.

@elukey hey, yes that's been fixed by making a newer version of curator available to the new clusters. Haven't seen cron errors from these since Dec 5. Thanks for cleaning up the "config does not exist" entries!

Looking more closely the problem was due to a Broker: Leader not available issue in the deployment-prep kafka logging cluster. After starting deployment-logstash2 back up (the instance had been stopped) logs are flowing again. Longer term we'll likely need another logstash stretch instance and to migrate over the broker id from deployment-logstash2 to the new instance.

In T233134#5713956, @Krinkle wrote:

@colewhite wrote at https://gerrit.wikimedia.org/r/554599:

seeing rsyslog complaining about "omkafka: kafka delivery FAIL" on deployment-prep hosts.

Ok, for my own edification, how would the private only LVS model work if we wanted to stand up a public facing non HTTP(S) service in a VM at one+ of these sites?

Will this Ganeti cluster use vlan tagged interfaces, or will separate physical interfaces connect to both public and private vlans? If tagging, are the switchports configured for that yet?

Friendly ping to @Volans about @fgiunchedi question above

Thanks for the ping, I missed the question. Sure, being added to the Gerrit Manager that would work for me!

https://gerrit.wikimedia.org/r/551270 should do the trick for source/dest ports. I don't recall why these weren't parsed out in the first place. While we're at it would any of the other parts the ulogd/iptables events be useful as fields?

In T235891#5659716, @fgiunchedi wrote:

re: bridging the gap with non-kafka inputs, my current thinking is to output all logs with deprecated-input tag back into kafka-logging on a separate topic and consume that from the new cluster. cc @herron @colewhite

In T230236#5627261, @BBlack wrote:

https://grafana.wikimedia.org/d/B9JpocKZz/ipsec-tunnel-status probably needs some cleanup (some of the graphs are empty, there's a note there to ignore icinga errors, etc). Also fix missing doc link on the alert?

In T236497#5644446, @BBlack wrote:

Sorry I missed that you already had a patch! But in any case, we only need commenting from cache::nodes to fix up this case (there's no good reason to e.g. churn it out of conftool or the various iptables rules defined from the other stuff).

Since it looks like cp3056 might be down for some time could we remove it from the config until fixed? It would be good to let the ipsec checks in icinga return to green.

In terms of “what” should be escalated, so far we discussed