Create Task
Maniphest T211712

moving from krypton to grafana1001 broke fundraising dashboards
Closed, ResolvedPublic
Actions

Description

It appears likely that grafana1001.eqiad.wmnet is missing some firewall rules somewhere (I am yet to figure out where) that make it able to talk to pay-lvs1001.frack.eqiad.wmnet and pay-lvs2001.frack.eqiad.wmnet.

These are datasources used in a few consoles e.g. https://grafana.wikimedia.org/d/000000403/fundraising-database?orgId=1 which are now broken.

@cwdent indicated that it's preferred to not make firewall changes while the fundraiser is running, so instead, for the interim, I will set up a new endpoint https://grafana-old.wikimedia.org/ and then take it down once this issue is fixed properly.

Related Objects

CDanis created this task.Dec 11 2018, 7:01 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 11 2018, 7:01 PM
gerritbot added a subscriber: gerritbot.Dec 11 2018, 7:11 PM

Change 479028 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/puppet@production] add hiera for grafana-old.w.o pointing to krypton

https://gerrit.wikimedia.org/r/479028

gerritbot added a project: Patch-For-Review.Dec 11 2018, 7:11 PM

Change 479028 merged by CDanis:
[operations/puppet@production] add hiera for grafana-old.w.o pointing to krypton

https://gerrit.wikimedia.org/r/479028

gerritbot added a comment.Dec 11 2018, 7:15 PM

Change 479029 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/dns@master] grafana-old.wikimedia.org DNS points to text caches

https://gerrit.wikimedia.org/r/479029

gerritbot added a comment.Dec 11 2018, 7:15 PM

Change 479029 merged by CDanis:
[operations/dns@master] grafana-old.wikimedia.org DNS points to text caches

https://gerrit.wikimedia.org/r/479029

CDanis added a comment.EditedDec 11 2018, 7:19 PM

krypton.eqiad.wmnet is serving again: https://grafana-old.wikimedia.org/dashboard/db/fundraising-database?orgId=1

Presently grafana-old's database is read-only, so edits to dashboards can't be made. This was done as part of the switchover, to give a clean window where we would fail on dashboard edits instead of silently dropping them. If read-only is a problem I can adjust that, just do note that any changes made on grafana-old won't be migrated anywhere afterwards.

I'll take grafana-old back down once we fix the firewall rules.

CDanis moved this task from Backlog to Blocked on others on the User-CDanis board.Dec 11 2018, 7:19 PM
CDanis updated the task description. (Show Details)
CDanis reassigned this task from CDanis to cwdent.Dec 11 2018, 8:07 PM

When you're ready to make firewall changes, I'm happy to help :)

cwdent added a comment.Mon, Jan 7, 6:11 PM

Deployed iptables change:

7c8dce3 new grafana server, grafana1001.eqiad.wmnet

@CDanis thanks so much for the temporary solution. We have separate firewalls for the fundraising cluster where I'll make the corresponding config change shortly, and defer to netops to deploy.

CDanis added a comment.Mon, Jan 7, 6:12 PM

Great, thanks! Please reassign back to me once the changes are deployed, to track tearing down the temporary solution.

cwdent added a subscriber: ayounsi.Mon, Jan 7, 7:18 PM

@ayounsi the updated config for this is at 1546888529

Stashbot added a subscriber: Stashbot.Mon, Jan 7, 7:42 PM

Mentioned in SAL (#wikimedia-operations) [2019-01-07T19:42:56Z] <XioNoX> push firewall change to pfw3-codfw/eqiad - T211712

cwdent added a comment.Mon, Jan 7, 7:56 PM

@ayounsi thanks! Fundraising grafana is now fixed. i pushed up 1546890827 which removes krypton.

cwdent reassigned this task from cwdent to CDanis.Mon, Jan 7, 8:06 PM

@CDanis all good here, go ahead and remove the old service at your convenience.

gerritbot added a comment.Tue, Jan 8, 3:10 PM

Change 482818 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/dns@master] Revert "grafana-old.wikimedia.org DNS points to text caches"

https://gerrit.wikimedia.org/r/482818

gerritbot added a comment.Tue, Jan 8, 3:11 PM

Change 482818 merged by CDanis:
[operations/dns@master] Revert "grafana-old.wikimedia.org DNS points to text caches"

https://gerrit.wikimedia.org/r/482818

gerritbot added a comment.Tue, Jan 8, 3:14 PM

Change 482819 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/puppet@production] Revert "add hiera for grafana-old.w.o pointing to krypton"

https://gerrit.wikimedia.org/r/482819

gerritbot added a comment.Tue, Jan 8, 3:15 PM

Change 482819 merged by CDanis:
[operations/puppet@production] Revert "add hiera for grafana-old.w.o pointing to krypton"

https://gerrit.wikimedia.org/r/482819

CDanis added a comment.Tue, Jan 8, 3:16 PM

grafana-old.wikimedia.org is no more.

CDanis closed this task as Resolved.Tue, Jan 8, 3:16 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL