Hey @Papaul, I added a raid10-gpt-srv-lvm-ext4-8disks.cfg for the initial installs on these.

In T221969#5182836, @hashar wrote:

I thought about this task a little bit. The current instances have 4 vCPUs. The operations-puppet-catalog-compiler-test job runs the compiler with NUM_THREADS=2

I would suggest:

• to use x1.large instances (8 vCPUS / 16G RAM / 160G disk). The RAM / disk is a bit overkill since the compiler is mostly CPU bound iirc.
• Set the jobs to use NUM_THREADS=6 (or 7? so we at least have one CPU available for the rest)
• add a third instance to the pool

Good point! And if we number from 200[1-5] it should simplify mapping of broker IDs between old and new hosts too. I updated the description to reflect this, but if you think its best to keep the 200[4-8] suffix happy to go that route instead.

In T221969#5176406, @hashar wrote:

I have deployed it on May 6th and thus puppet compile jobs should be hopefully equally split between the compiler1001 and compiler1002. I don't know how to proof check that though :-(

In T187147#5172711, @Krinkle wrote:

@herron Yes, I can do that to help avoid this specific instance of the problem. The problem I'd like to solve in this task, however, is to be able to detect it. That is, if there is a significant influx of errors that happen to be too large, this really should show up under type:mediawiki in some kind of channel (e.g. syslog_truncated) with a severity of "ERROR", so that they still get counted and immediately trigger the necessary alarms during a MediaWiki deployment.

For that it's totally find if the json is no parsed and only stored as raw message text. It would still be picked up at least with a timestamp, type and a bit of context (e.g. which MW server it came from), and the raw text will have to suffice for a MW developer to figure out where it came from and either to fix the problem that caused the error to be reported, or to make the error message less big.

But the immediate issue is to be able to at least index them and detect the problem.

yup!

After further testing I'm seeing these messages are arriving to rsyslog with @cee formatting, but truncated. Meaning the msg field does not contain valid json, specifically within the json-in-json-in-json field msg.fatal_exception.trace. The msg field comes from rsyslog extraction of the json payload prefixed by the @cee cookie in the syslog message.

Comparing (in beta) a working mediawiki log message and a log message failing with max_bytes_length_exceeded_exception I'm noticing differences in json formatting as well. For example

@hashar while on the topic, is it possible for Jenkins to more evenly dispatch PCC jobs across the workers? Currently compiler1002 receives the bulk of the work and currently is at 95% disk full, while compiler1001 is at only 50% disk full.

In T221288#5167063, @GedHaywood wrote:

Do those IPv6 addresses actually send any mail?

Ready to resolve afaict!

Seeing errors like this from logstash that appear related. This one specifically originated from logstash1007 /var/log/logstash/logstash-plain.log

On paper this use case also would lend itself to a filesystem with transparent compression. Maybe btrfs with compression. The data stored on disk is non-critical, and there are multiple worker nodes should issues arise with one filesystem.

In T222072#5150744, @Dzahn wrote:

I think it can be even simpler. How about https://gerrit.wikimedia.org/r/c/operations/puppet/+/507623/2/modules/profile/manifests/puppet_compiler.pp

Looking better after merging the above. From a password reminder mail:

uid=sukhe,ou=people,dc=wikimedia,dc=org has been added to the NDA group. Please re-open if any follow up is needed. Thanks!

In T189434#5128665, @TerraCodes wrote:

(eg, it says it's to security@tools.wmflabs.org but I somehow got the email).

Looking at cumin1001 I noticed that the log prefix at the end of the input chan is "fw-out-drop" and the output chain is empty with an accept policy. Is "out" indeed the direction in this case? Or would dropped packets logged by the input chain be considered "in"?

Since we're approaching two weeks on this request I've proposed the above patch to move forward using the existing deployment group and trust that caution will be exercised. Happy to see another approach implemented, but at the same time would like to unblock this individual access request.

Hello, I am not seeing an existing account with username Progresslabs. Could you please confirm that the account has already been created, and this is indeed the username? If you know what email was used I could try searching for that.

The Kibana lvs has been updated to use the source hash scheduler

Is there anything left to do before closing this?

I think it's safe to resolve this now since we're on logstash 5.6.15, and have disabled the logstash persistent queue.

jfishback has been added to the wmf ldap group. If any follow-up is needed please don't hesitate to re-open. Thanks!

Resolving as checklist in description has been completed

Looks like this is complete, but if any follow up is needed please don't hesitate to re-open!

Here are related puppet agent and puppetmaster1001 apache logs from a sampling of hosts

Deleted the remaining instances

Today we discussed desired hardware configs and expansion strategies during a meeting with @elukey @mobrovac @Ottomata and myself. Here are the outcomes:

That's interesting, based on the headers in T221290#5123805 it looks like this issue goes back as far as 2015.

In T221290#5123622, @faidon wrote:

How did it work until now?

Indeed I'm able to produce a DKIM issue as well with wiki-mail. Here's an example (seen in headers of message triggered by account preferences change):

In T141324#5122806, @Gehel wrote:

• structured logging from log4j can be exposed in a number of ways. The easier is probably to continue logging to file, use syslog for transport, but have messages in a structured format. A json layout could be used for that (here is one from Jetbrains, which I haven't tested, but I tend to trust Jetbrains). This would allow to output all logging context and not just a few basic information.

To give a few real-world examples

I'm for erring on the side of simplicity. Since these logs are useful on the command line of an individual host, on centrallog, and in Kibana, it makes sense to me to stream the ulogd syslogs formatted as shown in the description to logstash and parse them with a grok pattern.

In T217359#5032835, @elukey wrote:

In the SRE spreadsheet I can see that the suggested replacement FY is 20/21, not the upcoming one.. Just adding the info, not sure if these servers are eligible or not for refresh before the 5y of usage.

The intention is that ulogd logs from all servers will be sent to kafaka as such it would seem to make senses to move ::profile::rsyslog::kafka_shipper to the ::standard class

In T220387#5093826, @elukey wrote:

One thing that we didn't discuss for this goal is Zookeeper.

Localhost is seen in the mail headers because the Phabricator application relays mail to a local Exim MTA via SMTP. The local MTA provides message queueing and outbound SMTP server failover. This was done intentionally to improve the reliability of email delivery from Phabricator.

The below SPF record is now active

In T220412#5095320, @Jgreen wrote:

As far as I know, fundraising does not send mail using this domain but only from wikimedia.org, so I don't think our mass mail contractor needs to be listed.

Since today we have a mix of package and require_package this would be very nice indeed. Does it need to be homegrown? Seems worthwhile to weigh the pros/cons of using native Puppet ordering as well.