Fri, Nov 15
Closing this, there's no longer a push for this specific direction.
From an Ops/SRE perspective we're thinking the architecture should be a separate application server (like T237442) vs database server (like T237437).
Thu, Nov 14
Wed, Nov 13
Thu, Nov 7
Note an enterprise license is required for iDRAC, afaict this was included in recent server orders.
Reassigning to @Dwisehaupt since he's done the heavy lifting on the reimage/recommissions.
Wed, Nov 6
Wed, Oct 30
In addition to the repair, we're looking at adding another db system to the cluster for capacity/redundancy expansion. See T236920
Tue, Oct 29
Got api.paypal.com working with client certs on payments and civi. Leaving the ingenico check as a pass with 404, since it at least it shows us the endpoint is online.
Mon, Oct 28
done when bringing frdb1001 back from RAID system crash
Tue, Oct 22
Oct 18 2019
Used compresscmd to encrpyt/compress at log rotation, and a separate cron job to sweep those to /srv/archive/logs where they're picked up a day later by archive_sync and stored on the logger/archive hosts.
Oct 17 2019
Closing this task, what remains is mostly user-side stuff
Oct 11 2019
cert and password sent
Received authorization from Lisa Gruwell - Date: Thu, 10 Oct 2019 14:52:52 -0700
- bonded ethernet configuration done
- redis replication appears to be working now that firewall policy is deployed
- added to icinga
Oct 10 2019
@Ejegg this is mostly done, with some notes:
I'm going to backup and repurpose dev_analytics for this, which hasn't been touched since 2017.
Oct 8 2019
There's not really anything to be done here. We've archived the v1 historical data, and could, in theory, spin up a v1 instance to feed it to grafana, but it's just not worth the effort. It's a good reminder not to think of prometheus as a store for historical data, and to consider alternatives.
Oct 7 2019
@NNichols circling back on this task, did you ever receive your yubikey?
Working correctly now.
Oct 4 2019
Ah HA! Both scripts indeed suffered the same bug and were thus colliding. Both are fixed, leaving the task open until we're sure they're running successfully.
Oct 2 2019
Oct 1 2019
Sep 30 2019
Sep 24 2019
Flipping this to "Unbreak Now!" since it's a timely issue, and service outage interfering with the donation pipeline. We do have some donation activity at the moment.
Sep 20 2019
I moved these files to a temporary directory /srv/purge_after_20191031 for now, which is out of the way of log collection, nfs, and log processing. We'll do a final purge sometime after that date.
This is done, please reopen if there's any issue with the new mysql user.
@Ejegg is this something we still need? I know we're collecting a lot of metrics on civi1001 and am not sure if these queues are already included?
Sticking with simple backup of the live host for now...
Cross-host snapshot is set to run daily to back up /srv/prometheus.
Sep 19 2019
Author: Jeff Green <email@example.com>
Date: Thu Sep 19 15:48:07 2019 +0000
Sep 13 2019
Caitlin Cogdill sent an access authorization request to Lisa Gruwell earlier this week, we're waiting to hear back on that.
Sep 12 2019
Yubikey requested from OIT...