Hey @Papaul, I added a raid10-gpt-srv-lvm-ext4-8disks.cfg for the initial installs on these.
Tue, May 21
Fri, May 17
Good point! And if we number from 200[1-5] it should simplify mapping of broker IDs between old and new hosts too. I updated the description to reflect this, but if you think its best to keep the 200[4-8] suffix happy to go that route instead.
Thu, May 16
Tue, May 14
Mon, May 13
Fri, May 10
Thu, May 9
After further testing I'm seeing these messages are arriving to rsyslog with @cee formatting, but truncated. Meaning the msg field does not contain valid json, specifically within the json-in-json-in-json field msg.fatal_exception.trace. The msg field comes from rsyslog extraction of the json payload prefixed by the @cee cookie in the syslog message.
Wed, May 8
Comparing (in beta) a working mediawiki log message and a log message failing with max_bytes_length_exceeded_exception I'm noticing differences in json formatting as well. For example
@hashar while on the topic, is it possible for Jenkins to more evenly dispatch PCC jobs across the workers? Currently compiler1002 receives the bulk of the work and currently is at 95% disk full, while compiler1001 is at only 50% disk full.
Ready to resolve afaict!
Tue, May 7
Seeing errors like this from logstash that appear related. This one specifically originated from logstash1007 /var/log/logstash/logstash-plain.log
Wed, May 1
On paper this use case also would lend itself to a filesystem with transparent compression. Maybe btrfs with compression. The data stored on disk is non-critical, and there are multiple worker nodes should issues arise with one filesystem.
Looking better after merging the above. From a password reminder mail:
Tue, Apr 30
Mon, Apr 29
uid=sukhe,ou=people,dc=wikimedia,dc=org has been added to the NDA group. Please re-open if any follow up is needed. Thanks!
Fri, Apr 26
Thu, Apr 25
Looking at cumin1001 I noticed that the log prefix at the end of the input chan is "fw-out-drop" and the output chain is empty with an accept policy. Is "out" indeed the direction in this case? Or would dropped packets logged by the input chain be considered "in"?
Since we're approaching two weeks on this request I've proposed the above patch to move forward using the existing deployment group and trust that caution will be exercised. Happy to see another approach implemented, but at the same time would like to unblock this individual access request.
Hello, I am not seeing an existing account with username Progresslabs. Could you please confirm that the account has already been created, and this is indeed the username? If you know what email was used I could try searching for that.
Wed, Apr 24
The Kibana lvs has been updated to use the source hash scheduler
Is there anything left to do before closing this?
I think it's safe to resolve this now since we're on logstash 5.6.15, and have disabled the logstash persistent queue.
Apr 23 2019
jfishback has been added to the wmf ldap group. If any follow-up is needed please don't hesitate to re-open. Thanks!
Resolving as checklist in description has been completed
Looks like this is complete, but if any follow up is needed please don't hesitate to re-open!
Here are related puppet agent and puppetmaster1001 apache logs from a sampling of hosts
Deleted the remaining instances
Apr 19 2019
Apr 18 2019
That's interesting, based on the headers in T221290#5123805 it looks like this issue goes back as far as 2015.
Indeed I'm able to produce a DKIM issue as well with wiki-mail. Here's an example (seen in headers of message triggered by account preferences change):
Apr 17 2019
To give a few real-world examples
I'm for erring on the side of simplicity. Since these logs are useful on the command line of an individual host, on centrallog, and in Kibana, it makes sense to me to stream the ulogd syslogs formatted as shown in the description to logstash and parse them with a grok pattern.
Apr 16 2019
Apr 15 2019
The intention is that ulogd logs from all servers will be sent to kafaka as such it would seem to make senses to move ::profile::rsyslog::kafka_shipper to the ::standard class
Apr 11 2019
Localhost is seen in the mail headers because the Phabricator application relays mail to a local Exim MTA via SMTP. The local MTA provides message queueing and outbound SMTP server failover. This was done intentionally to improve the reliability of email delivery from Phabricator.
The below SPF record is now active
Apr 9 2019
Apr 8 2019
Since today we have a mix of package and require_package this would be very nice indeed. Does it need to be homegrown? Seems worthwhile to weigh the pros/cons of using native Puppet ordering as well.