Jun 4 2019
Pretty sure everything is done here, I will close once the archive sync runs successfully.
ef920db allow loggers to ssh to monitoring servers for archive collection
a968e43d enable grafana backups
May 20 2019
May 8 2019
@RLewis restarting the browser is worth a try. If that doesn't work could you open the menu described here in browser prefs and see if you see the certificate listed?
@RLewis your cert has 135 days left until expiration, did this just start happening? Are you using a different computer?
Apr 30 2019
@Papaul thanks! Working :)
Apr 29 2019
commit 173b34cd01bdc0aa5998af347406d9775755656f (HEAD -> master, origin/master, origin/HEAD) Author: Casey Dentinger <email@example.com> Date: Mon Apr 29 17:07:06 2019 +0000
commit 5e6951f6082627a1f61f38d27532a48197eada4f Author: Casey Dentinger <firstname.lastname@example.org> Date: Mon Apr 29 17:03:13 2019 +0000
Apr 26 2019
Apr 23 2019
commit 606f45371334528bbbd51a4daa17805f1fddd7e4 (HEAD -> master, origin/master, origin/HEAD)
Author: Casey Dentinger <email@example.com> Date: Tue Apr 23 16:23:20 2019 +0000
Apr 20 2019
@ayounsi - the new policies are at 1555726449, let me know if you need anything else thanks
Apr 19 2019
Apr 18 2019
Apr 16 2019
@CCogdill_WMF you're right that part is just a comment that is fine to leave off, the key is also shorter than your old one, but that can be normal if the new computer has different default ciphers. Did you try to log in?
That ssh key looks funny, like it got cut off early, but maybe it's just shorter, let me know if you can log in.
@CCogdill_WMF actually one more minute
@CCogdill_WMF that's fine and sorry for not reading the task! You should be good now.
@CCogdill_WMF please post the output of the yubikey press to your officewiki profile and I'll get it updated
Apr 8 2019
@ayounsi there is a new config at -1554758904, removing prod prometheus and grafana access to pay-lvs servers.
Apr 5 2019
Apr 4 2019
@fgiunchedi @colewhite actually we have a private grafana/prometheus instance up and running in fundraising now so we can disconnect this and remove the fundraising boards. I'd like to export some FR metrics some day but for the time being this will be easier in terms of PCI compliance and data safety.
Mar 26 2019
Mar 15 2019
Mar 14 2019
I looked back and rsyslog was set to install from backports like 6 months ago. What I don't understand is why it is not installed from backports right now. I updated them manually. I'll reopen if this comes up again.
$ apt-cache policy rsyslog rsyslog: Installed: 8.24.0-1 Candidate: 8.1901.0-1~bpo9+1 Version table: 8.1901.0-1~bpo9+1 1001 -10 http://ftp.debian.org/debian stretch-backports/main amd64 Packages 8.38.0-1~bpo9+1wmf1 500 500 http://apt.wikimedia.org/wikimedia stretch-wikimedia/main amd64 Packages *** 8.24.0-1 500 500 http://ftp.us.debian.org/debian stretch/main amd64 Packages 100 /var/lib/dpkg/status 8.4.2-1+deb8u2 -10 -10 http://ftp.us.debian.org/debian jessie/main amd64 Packages
@mepps - Did we decide which of the two described approaches we want to take? Once we do that I'll start on some subtasks for the 'advanced prep' section.
Mar 13 2019
@CDanis thanks! I will definitely use that trick.
Mar 6 2019
Mar 5 2019
@mepps I think it would be more usefully discussed on a task where it can be seen by everyone, because my essential worry is there haven't been enough eyes on this change, and it is a likely source of unintended consequences.
Mar 1 2019
Feb 28 2019
Feb 27 2019
@DStrine wondered if it would be possible to roll the CSP header before the PHP upgrade. According to https://www.mediawiki.org/wiki/Compatibility we could upgrade PHP before MW. However I think there may be problems with ResourceLoader image inlining.
Feb 19 2019
Feb 13 2019
Unlike 1004 which is working properly, Apache on on 2003 is returning "(52) Empty reply from server"
Feb 6 2019
Working, thanks @ayounsi !
Feb 5 2019
This is done, incidentally the dumps are now big enough to cause disk alerts about /srv
Feb 4 2019
Just kicked this off, will update when it finishes, probably tomorrow.
Jan 30 2019
@Cmjohnson yep, sorry forgot about this ticket! Thanks for your help.
Jan 17 2019
@Pcoombe yes indeed, I made a subtask for that for tracking and to remind myself.
@Pcoombe unfortunately the available versions on Jessie (current OS) and Stretch (upcoming OS) are both old (0.14 and 0.18 respectively). Buster (2 OS versions out) has 1.0 but that's a ways off. So we'd want to look at packaging it ourselves which is doable, unless someone around here (analytics?) has already done that which would be easier.
Jan 16 2019
Jan 9 2019
@ayounsi thanks for the help, this looks good
Jan 8 2019
@ayounsi the new rules are at 1546987554
Jan 7 2019
@CDanis all good here, go ahead and remove the old service at your convenience.
@ayounsi thanks! Fundraising grafana is now fixed. i pushed up 1546890827 which removes krypton.
@ayounsi the updated config for this is at 1546888529
Deployed iptables change:
@Pythoncoder thanks for the report, I shuffled the tags around so this gets seen by the right people.
Jan 3 2019
@Eileenmcnaughton sure, looking at the grants, would that be just select on dev_*?
Jan 2 2019
Dec 19 2018
Dec 18 2018
While currently tech can see the alerts in IRC, Jeff and I get SMS about it, which means dropping what you are doing, getting out of bed, etc. So it would be good to keep that alert stream to things that are actionable by ops. Another icinga channel might be the answer, but there is the open question of who pays for our phones.
There are a couple related tickets here: T202419
Does this mean fr-tech wants to get paged by icinga?
@Cstone has logged in to everything
c917cff add cstone mysql grants