@greg kinda...:-P
I might have abused a bit this task for things that are related to make it a proper release, PyPi ready and some metadata/testing polishing.

The implemented proposal (CR will be sent later today) is pretty much what listed as (2) in the task description:

• have an optional default_backend parameter in the configuration that will cause:
  • if set: try to parse the query with the default backend first, if it fails parsing try with the multi-query global grammar
  • if not set: run directly with the multi-query global grammar
• have aliases only in the global grammar, they need to use the global multi-query grammar and can be specified as A:alias_name
• aliases are recursively replaced with their value, so an alias can be a composition of other aliases
• Each query block can be aggregated with the others with the boolean operators: and, or, and not, xor

After few days without incident seems that we can call it resolved! \o/

@Pnorman the issue on Icinga was solved around 12:10 UTC, but, as you can see in the detailed explanation in T164206#3435220, there was a loss of downtimes and acknowledgments, and the cleanup of this was done a bit later after the fix, from the logs I can see that at 13:43 UTC the maps-tests alarms where acknowledged again.

@Pnorman when was that?
I'm asking because the Icinga issue is T164206 and since today around 12:10 UTC it should be fixed, see T164206#3435220 for the full explanation.
And the alarm I mentioned above was already re-acked after the fix and I can see the ack is still there.

So after a lot of digging and live debugging between @akosiaris and me, we *think* to have an explanation and to have fixed the issue.

In T170394#3430118, @faidon wrote:

For option 2, how do you resolve ambiguities? Would letters be unique to backends (i.e. F/R are claimed by PuppetDB and can't be claimed by another backend, period) or would they be claimed on a first-come first-served basis, in which case the order of backend inclusion matters? I'd strongly prefer the former, but this means that letters need to be reserved across/above backends, right?

@Gehel I've ACK'ed the Icinga alert for kartotherian endpoints health on maps-test* hosts, given that they are in alarm since 4 days my guess is that they were downtimed but today seems that Icinga lost downtime again. Just FYI.

db1053.mgmt.eqiad.wmnet seems to work now, I can both ssh and get an chassis status from neodymium. Transient issue?

I've try to fix the 5 hosts with the remote wrong remote config:

sudo cumin -b 1 "bast3002.wikimedia.org,cp4021.ulsfo.wmnet,db2082.codfw.wmnet,gerrit2001.wikimedia.org,naos.codfw.wmnet" "ipmi-config --category=core --key-pair="Lan_Channel:Volatile_Access_Mode=Always_Available" --key-pair="Lan_Channel:Non_Volatile_Access_Mode=Always_Available" --commit"

I've run the audit again with a small script on neodymium in my home using cumin to grab the list of hostnames. The only requirement is to have exported IPMI_PASSWORD with the right password in the current environment (I use a space when setting it so that it doesn't get saved at all into the bash history because I'm using HISTCONTROL=ignoreboth).

Regarding sodium it seems to me that puppet runs get stuck because it try to execute ipmi-config while loading facts:

@MoritzMuehlenhoff the broken disk was known: T169959

Closing as duplicate of T169693 . The disk went to a failed state when removed and is now rebuilding.

In T169765#3410598, @elukey wrote:

Another thing that would be nice is the possibility to specify more than one conf host in profile::pybal::config_host: conf2001.codfw.wmnet, and allow pybal to connect to more hosts in case of connection failures.

False positive, I'll update the list of patterns to skip.

@fgiunchedi FYI Doing some cleanup with @ArielGlenn we discovered that most of the load was generated by the prometheus-node-exporter:

3007 ?        Dsl   44:58 /usr/bin/prometheus-node-exporter -collector.diskstats.ignored-devices=^(ram|loop|fd)\\d+$ -collector.textfile.directory=/var/lib/prometheus/node.d -collectors.enabled=diskstats,filefd,filesystem,hwmon,loadavg,mdadm,meminfo,netdev,netstat,sockstat,stat,tcpstat,textfile,time,uname

You could try to force unmount the NFS partition from the hosts that mount it, it should abort the outstanding I/O they are waiting for, also processes that are in TASK_KILLABLE state should be killable.

@hashar is there still any reason to keep tox pinned to this old version? Any chance we could upgrade it to a newer one? Jessie backports has 2.5.0 and latest from pip is 2.7.0.

Opened T169564 for the mdadm configuration.

@mmodell thanks for cleaning those too, but I don't think that this method can be applied in general. There are ex-legit users that might be suspended (we have some of those) and their uploads would be deleted too although legit.

@Aklapper @mmodell the cleaning effort is clearly not working! After my last full cleaning of recent files, there are already 134 files uploaded during the last few days by users that are disabled now. And the ~3k previously reported ones are still there.

@Framawiki @Mainframe98 thanks for letting us know. I've disabled the two users and removed their files. I didn't touched the task T169502, as it's harmless at this point, I'll leave it to Phabricator admins.

And of course that was not enough, I had to also add an exit 0 to /etc/cron.daily/mdadm to prevent it from running, without the MAILADDR setting the report check refuses to run and generates cronspam.

And of course that was not enough, I had to also add an exit 0 to /etc/cron.daily/mdadm to prevent it from running, without the MAILADDR setting the report check refuses to run and generates cronspam.

I've commented out the MAILADDR line to avoid to get one email per day. Given that we have also the Icinga check we could consider to comment it out broadly across the fleet. The file is currently not managed by puppet.

I've commented out the MAILADDR line to avoid to get one email per day. Given that we have also the Icinga check we could consider to comment it out broadly across the fleet. The file is currently not managed by puppet.

Rebuild completed, RAID back to optimal. There are 2 disks with predictive failure that might fail sooner or later

FYI: This is s1 master! Adding @Marostegui @jcrespo directly too for visibility.
At least the other 2 disks with predictive failure are in different spans.

@mmodell thanks for the additional info.

@fgiunchedi the old names needs to be cleaned from puppetdb too, I guess we need to add this step in the related documentations too.

@Aklapper, yes AFAIK that field in the DB respect the query filter Upload Source in the file query page. But from what I'm seeing, that determines only from where the file was uploaded (I guess if from the file upload page or indirectly from a drag and drop in task comment, a Paste, etc...).

@Aklapper not really, same order of magnitude, see below. Also what is exactly the difference? I've opened some of the isExplicitUpload = 0 and are usually images that could be spam as well.

512 sounds reasonable and seems the only pertinent one in that list, +1!

Related to T166965, please do not close until the parent is fixed as well because it will be re-opened again if Icinga loose the downtimes/acknowledges or the alarm flaps for any reason.

Today @MaxSem got another report of an abuse file in Phabricator. After checking it I found that was uploaded last week during the incident and I coudn't find it in Phabricator UI in the file search, but I could confirm it was there in the DB.

@elukey thanks for the update. Which one is the final change to be reviewed then?

As per the obvious concerns about www-data being able to ssh as the Phabricator daemon_user, that seems to be a hard requirement on Phabricator side in order to enable the clustering unfortunately. I'd like @MoritzMuehlenhoff to comment on this too, in the end is a trade off between security and high availability.
Is not fully clear to me right now in the single host configuration what kind of access has www-data to the repositories data. The sudo line seems limited to git-upload-pack and git-receive-pack.

Following @Dzahn last message on https://gerrit.wikimedia.org/r/#/c/324841/3 , continuing here.

We should also investigate other available tools in the container space, for example one recently released is https://github.com/puppetlabs/lumogon or from CoreOS https://github.com/coreos/clair (thanks @Joe for this one). Disclaimer: I've not yet done an extensive search for other available tools ;)

What @jcrespo said, see also my comment on https://gerrit.wikimedia.org/r/#/c/356383/12/modules/role/files/mariadb/eventlogging_cleaner.py@206 regarding the addition of an ORDER BY.

Sorry for the late reply, partially because I was too busy cleaning stuff around to reply here (thanks Reedy for the help) and partially to not give too much of a realtime feedback to the abusers.
Thanks everyone here that helped notifying us and limiting the impact whenever possible.

There is an ETA for a permanent fix? It seems to me that we've already delayed this too much given the frequency at which it's happening lately.

@Paladox FYI I'm still getting Invalid SSH Key when trying to add my key

@akosiaris yes we were aware of it and I spoke with @Joe last week about the requirements for the Docker part, sorry to not have mentioned/referenced it here too. The idea is to have a single tool at this point that can work for both physical hosts and Docker images, so it should overlap fully with the requirements of T167269.