produce_canary_events currently looks up stream config from https://meta.wikimedia.org/w/api.php?format=json&action=streamconfigs&all_settings=true. To contact this URL, it must use webproxy (I think due to Analytics VLAN firewall).
We could open up a hole in the Analytics VLAN to this endpoint, but the more correct thing to do would be to use api-ro.discovery.wmnet with a Host header set to meta.wikimedia.org. We still might need a VLAN hole to api-ro.discovery.wmnet.
Oh, this is a bit more of a problem than just canary events. Camus is using webproxy to get to meta.wm.org API to discover topics to import.
We should open this hole in the vlan asap.
Ok, @akosiaris has webproxy turned back on for now. We need to do 2 things:
Change 665415 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[wikimedia-event-utilities@master] Use api-ro.discovery.wmnet as default Wikimedia EventStreamConfig URI
@razzi, @elukey, Analytics VLAN should be able to access api-ro.discovery.wmnet port 80, which resolves to 10.2.2.22 for me in both eqiad and codfw. I'm not 100% sure that is the right address, but it seems to be?
It might be handy if we could also access meta.wikimedia.org too, which resolves to 208.80.154.224 for me (dyna.wikimedia.org?) but I'm not sure if that is possible or correct.
Could one of you open do this? Thank you!
elukey@puppetmaster1001:~$ sudo -i confctl --quiet --object-type discovery select 'dnsdisc=api-ro' get
{"codfw": {"pooled": false, "references": [], "ttl": 300}, "tags": "dnsdisc=api-ro"}
{"eqiad": {"pooled": true, "references": [], "ttl": 300}, "tags": "dnsdisc=api-ro"}I would add also the codfw IP just to be sure, if dns-disc settings are changed we'll have trouble, going to send a code review asap. In theory meta.wikimedia.org should be able to be accessible via app/api-servers (with Host: meta.wikimedia.org), otherwise if we want to use the external IP we should use the webproxy (with the caveat that if it has troubles we have them as well). I am going to send a code review and then we'll discuss in there :)
Change 665814 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/homer/public@master] Add a mediawiki-api term to the analytics-in4 filter
Change 665814 merged by Elukey:
[operations/homer/public@master] Add a mediawiki-api term to the analytics-in4 filter
Change 668858 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[wikimedia-event-utilities@master] [WIP] Simplify code needed to make URL based requests
Change 668858 abandoned by Ottomata:
[wikimedia-event-utilities@master] [WIP] Simplify code needed to make URL based requests
Reason:
Ifcaa2f3b60ef763d7bdfe032cd9a0d1e72a1e88e better.
Change 665415 abandoned by Ottomata:
[wikimedia-event-utilities@master] Use api-ro.discovery.wmnet as default Wikimedia EventStreamConfig URI
Reason:
in favor of I296ff3c1c771d7e66e631cc85b2652458998c725
Change 677287 had a related patch set uploaded (by Ottomata; author: Ottomata):
[wikimedia-event-utilities@master] Fix for BasicHttpClient so that a non 2xx response will fail when used with ResourceLoader
Change 677287 merged by Ottomata:
[wikimedia-event-utilities@master] Fix for BasicHttpClient so that a non 2xx response will fail when used with ResourceLoader
Change 678600 had a related patch set uploaded (by Ottomata; author: Ottomata):
[operations/puppet@production] produce_canary_events - No longer use http proxies and use api-ro
Change 678600 merged by Ottomata:
[operations/puppet@production] produce_canary_events - No longer use http proxies and use api-ro
Mentioned in SAL (#wikimedia-analytics) [2021-04-12T14:21:42Z] <ottomata> stop using http proxies for produce_canary_events_job - T274951
Change 678919 had a related patch set uploaded (by Ottomata; author: Ottomata):
[analytics/refinery/source@master] ProduceCanaryEvents - include httpRequest body in failure message
Change 678919 merged by Ottomata:
[analytics/refinery/source@master] ProduceCanaryEvents - include httpRequest body in failure message