Wed, Jan 12
Tue, Jan 11
@RhinosF1 I meant to ask earlier, where are you receiving these alerts? I thought icinga was configured such that FR alerts weren't going to anyone outside fr-tech.
This has not moved in ~6 years, clearly it's not going to happen.
We've done some additional tuning.
Mon, Jan 10
Fri, Jan 7
Thu, Jan 6
Dec 16 2021
Dec 13 2021
Dec 12 2021
I added sns.us-east-1.amazonaws.com to ipset, please let me know if this doesn't fix the timeouts.
Dec 10 2021
Dec 9 2021
A third change made:
Stealing this task fair and square.
Removing myself as owner, this is normally handled by SRE.
Dec 8 2021
Two changes were made to the system:
As far as I can tell Dallas's monitoring adjustments have reigned in the noise as much as is feasible for the moment. Software fixes are in the pipeline which should help with the replication lag.
Dec 7 2021
@Eileenmcnaughton testing on frdev1001 is definitely feasible but we need to make some configuration changes to insure messages can't leave the server. I'll look at this today.
Nov 30 2021
No longer necessary, we've decommissioned dash. See T276394
Based on yesterday's fr-tech & fr-tech-ops meeting, it sounds like it is desireable to take Dash offline and use Superset, since everything in Dash can be reported there and having two different reporting schemes/sites causes confusion. I removed the Dash frontend today, see above for what needs to be done.
Nov 29 2021
Nov 24 2021
Nov 22 2021
Nov 17 14:53:46 frdev1001 selective_db_restorer: starting
Nov 20 00:09:46 frdev1001 selective_db_restorer: finished
Nov 17 2021
started Wed 17 Nov 2021 02:54:00 PM UTC
Nov 16 2021
@ayounsi I think it would be fine to do the codfw pfw's this year. Please ping me on IRC when you have some time to discuss.
Woot, it works!
Nov 12 2021
Nov 10 2021
Apparently you can use notes_url set the service definition, something to this effect:
Nov 9 2021
I tested the install automation in virtualbox and everything went smoothly.
Nov 3 2021
Oct 22 2021
Oct 5 2021
Sorry, I should have moved this out of FR-Ops when I removed the fr-tech-ops tag! Moving now so it will get re-triaged.
We missed this task because it didn't get the fr-tech-ops tag, we're good to go however!