A simple option: if puppet-merge.sh is given a treeish, it *only* does the ops repo or the labsprivate repo (depending on what flag was passed).

Believe this has been worked around for now.

boldly re-opening this, now that the POPs have Ganeti clusters available.

Nothing in racadm getsel or racadm lclog view (latter just has me logging in over SSH).

The steps outlined in Filippo's comment happened, with the difference that I chose to use the netmon* machines for this role.

I'll keep an eye on this and close if there's no other noise.

Thanks! I had been confused by Attack protocol: tcp in the reports.

BTW I'm curious about both 1) the current need (or lack thereof) for cross-DC Mediawiki DB traffic, and also about 2) the future need. As it would be fairly trivial to make dbctl output this if/when we do need it -- but enabling hostsByName from etcd now is even easier.

+1 to doing #1 and revisiting if it becomes a problem again.

Seems to be working well.

I tested this on mwdebug1001 by manually installing my patch there.

In T239900#5715511, @Anomie wrote:

At the MediaWiki level, that would look like assigning a non-zero value in $wgLBFactoryConf['sectionLoads'] and/or $wgLBFactoryConf['groupLoadsBySection'] and/or $wgLBFactoryConf['groupLoadsByDB']. I don't know how that translates into dbctl.

We also have two probes on Comcast's network constantly performing pings towards our RIPE Atlas anchor in ulsfo. Their network performance looks relatively stable over the past 24h: https://w.wiki/DiS

I did some ICMP pings and TCP port 443 traceroutes from RIPE Atlas probes on Comcast's network that had IPv6 enabled. There were a few that couldn't reach Phabricator, including one in the Kirkland area (so probably reasonably close in network-space to @brion ), but it's hard to be sure what this means -- there's always going to be some ambient background number of probes that are malfunctioning in some way. (And most of the probes that failed weren't on the west coast, but in the midwest or on the east coast, which will be a different part of Comcast's network that shows as congested in your mtr.)

Looking at some data in grafana explore, this would have solved most cases of noise in the past few months. So calling it resolved for now.

another instance of P9808

In T229686#5704377, @Marostegui wrote:

Any rough ETA on when externalLoads will be able to be handled by dbctl?

In T234450#5698503, @sbassett wrote:

I've reapplied and deployed the "500 limit patch" from T234450#5595510 to wmf.5 as a (hopefully) very temporary measure while we continue to further troubleshoot this issue and tweak the PoolCounter solution.

In T224888#5693928, @ayounsi wrote:

That looks good! We might want to create a specific LibreNMS alert for the transit/peering links only, but can start with the existing ones.

In T224888#5692672, @fgiunchedi wrote:

Sounds great to me! I am assuming on the icinga side it'll be only one alert at least to start with, for e.g. silencing purposes which I think will work fine for now.

Grafana 6.4.4 is now in use at https://grafana.wikimedia.org.

I've a proposal for doing this:

At ~18:36 there was another spike in long-tail latency, but then, latency seemed to return to 'normal':
https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red-dashboard?orgId=1&from=1574443410296&to=1574450056679

In T238833#5684837, @Ottomata wrote:

Could we make the cergen script itself modify the permissions after it creates the files? It won't ensure things are right ongoing, but at least they'd be right from the start.

I've heard no complaints, and can verify from the logs that it's seen at least some testing by others. Planning to do a final snapshot and move traffic over on Monday afternoon my time.

Sure, I can create the configuration patch.

I grepped through both the swiftrepl logs on ms-fe1005 and also
the aggregated Swift mutation-operation logs on centrallog1001 and
found no mention of the file.

Tim suggested 2 as a concurrency limit. I think we can start less conservative than that, though -- let's say 10? It feels pretty hard for that to hurt normal user traffic, while it should still prevent excessive usage.

upgraded the pie chart plugin to a recent version that actually works with 6.x:

❌cdanis@grafana1002.eqiad.wmnet ~ 🕕🍺 sudo http_proxy=http://webproxy.eqiad.wmnet:8080 grafana-cli plugins install grafana-piechart-panel

+1 to @tstarling's proposal.

Generated a list of all $hosts_allow arguments from rsync::server::module invocations across all of Puppet: P9544

Another thing that just came up: not all users of rsync::server::module are actually passing an array to the $hosts_allow argument: https://gerrit.wikimedia.org/r/c/operations/puppet/+/549142
Need to go through PuppetDB and look for this.