| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Dwisehaupt | T243110 OKR 2019-2020 Q3: Increase visibility and awareness of Fundraising system health and wellness | |||
| Resolved | Dwisehaupt | T246119 Upgrade fundraising grafana to v6.6 | |||
| Declined | Dwisehaupt | T250441 Move from phantomJS to grafana-image-renderer for frack grafana |
Event Timeline
Pushed the change to add in the repos so we can pull from the main grafana repo instead of repackaging on our own.
[frack::puppet] 5c32e91e Add grafana repo to valid repos to pull from
Reverified this commit that grafana should not get automatic upgrades. Just in case:
[frack::puppet] ef22862c Lock the grafana package to keep it from getting auto updates
Pushed the following commit to fix up some repo settings found in testing in the vb setup.
[frack::puppet] 42f201f9 Update grafana repo to use http and stable dist
After testing in the vb land, I pushed these two commits to ensure the packages for prometheus and grafana are properly set, pinned, and clean for puppet runs without accidental upgrading.
[frack::puppet] d5e7a12e Add pin for libjs_moment_timezone for stretch [frack::puppet] e65881dc Enable the grafana repo for the grafana package
Right now, grafana is exempted from automatic updates so it will stay on our frack version. a 'risky package update' will force and upgrade and bring it to the latest released version (6.7 at this point in time).
A backup and upgrade will be scheduled for off hours this week.
I had to copy over the sqlite db from frmon1001 as the db was out of date and didn't have my user account. After doing that, I was able to test via ssh tunnel. I verified functionality before the update.
Package updates were then run on frmon2001. This updated grafana to 6.7.2 and there was a prometheus update to 2.7.1. Functionality was tested after the updates and all dashboards rendered as the would on the old version. Additional functionality of Explore is also present and tested.
Will let this bake for a day just to see if anything odd shakes out and then I'll run the update on frmon1001.
Everything looked good and clean overnight. Rebooted frmon2001 just to verify it was good after a clean restart and grafana and prometheus were all clean and functioning correctly.
Went ahead and ran the upgrade on frmon1001. Logs look clean, authentication worked fine, and dashboards and new features look good. Leaving this open until Monday in case there are issues to track.