Page MenuHomePhabricator

WikimediaEventUtilities and produce_canary_events job should use api-ro.discovery.wmnet instead of meta.wikimedia.,org to get stream config
Open, Needs TriagePublic


produce_canary_events currently looks up stream config from To contact this URL, it must use webproxy (I think due to Analytics VLAN firewall).

We could open up a hole in the Analytics VLAN to this endpoint, but the more correct thing to do would be to use api-ro.discovery.wmnet with a Host header set to We still might need a VLAN hole to api-ro.discovery.wmnet.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptTue, Feb 16, 10:06 PM

Oh, this is a bit more of a problem than just canary events. Camus is using webproxy to get to API to discover topics to import.

We should open this hole in the vlan asap.

Ok, @akosiaris has webproxy turned back on for now. We need to do 2 things:

  • Make WikimediaEventUtilities take a headers parameter for its stream_config_uri, so we can properly use api-ro.discovery.wmnet with Host:
  • Open a hole in the Analytics VLAN firewall api-ro.discovery.wmnet

Change 665415 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[wikimedia-event-utilities@master] Use api-ro.discovery.wmnet as default Wikimedia EventStreamConfig URI

Ottomata added a comment.EditedFri, Feb 19, 10:25 PM

@razzi, @elukey, Analytics VLAN should be able to access api-ro.discovery.wmnet port 80, which resolves to for me in both eqiad and codfw. I'm not 100% sure that is the right address, but it seems to be?

It might be handy if we could also access too, which resolves to for me ( but I'm not sure if that is possible or correct.

Could one of you open do this? Thank you!

elukey added a comment.EditedMon, Feb 22, 7:19 AM
elukey@puppetmaster1001:~$ sudo -i confctl --quiet --object-type discovery select 'dnsdisc=api-ro' get
{"codfw": {"pooled": false, "references": [], "ttl": 300}, "tags": "dnsdisc=api-ro"}
{"eqiad": {"pooled": true, "references": [], "ttl": 300}, "tags": "dnsdisc=api-ro"}

I would add also the codfw IP just to be sure, if dns-disc settings are changed we'll have trouble, going to send a code review asap. In theory should be able to be accessible via app/api-servers (with Host:, otherwise if we want to use the external IP we should use the webproxy (with the caveat that if it has troubles we have them as well). I am going to send a code review and then we'll discuss in there :)

Change 665814 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/homer/public@master] Add a mediawiki-api term to the analytics-in4 filter

Change 665814 merged by Elukey:
[operations/homer/public@master] Add a mediawiki-api term to the analytics-in4 filter