See explanation of the issue in the following email thread:

This issue seems to be the same as mentioned in the email alerts with the similar name of last week.
See Andrew's comments for those below:

Hm, very strange. These seem to be occasionally failing because eventgate-analytics-external in codfw fails to look up stream config ? Like:

• produce_canary_events runs on an-launcher1002
• looks up all streams to produce canary events for
• iterates over them and produces a canary event one at a time for each stream to eventgate in both eqiad and codfw
• eventgate in codfw receives the produce request, and looks up stream config for the event
• This usually always succeeds, but for 5 times in the last 2 days, eventgate got error when fetching stream config: HTTPError: getaddrinfo ENOTFOUND api-ro.discovery.wmnet api-ro.discovery.wmnet:443"

When this has happened, it only happened for 1 out of the 70+ requests produce_canary_events makes, and only when POSTing to codfw. It doesn't seem to be for any particular stream, although I do see a few repeats.

Anyway, lets keep watching this, if it is still happening next week I'll file a task and investigate more.

FYI, to see these errors, run this on an-launcher1002:

cat /var/log/refinery/produce_canary_events/produce_canary_events.log | grep -A 2 'ERROR ProduceCanaryEvents Some canary events failed to be produced'

I executed this convenient log grep (thanks!) and the error message is a bit different from last week:

HTTPError: connect EHOSTUNREACH 10.2.1.22:443

The IP 10.2.1.22 belongs to MediaWiki API cluster (https://github.com/wikimedia/puppet/blob/production/hieradata/common/service.yaml#L87)
Maybe the MediaWiki API is throttling EventGate? Maybe the EventGate cache for inactive schemas' configs is expiring at the same time for all, and then it tries to get them at the same time, because all canaries are sent together? Or maybe just the MediaWiki API is unavailable for a couple seconds, after looking at EventGate code I think we are not retrying when getting the StreamConfig, but I might be wrong. If so, we could add a retry?

Also, there's a couple errors of another kind:

Cannot read property 'eventlogging_TemplateDataEditor' of undefined
Cannot read property 'eventlogging_TranslationRecommendationUserAction' of undefined

Not sure what those are.
I think my EventGate ops troubleshooting abilities end here.
Since the alert has recovered, and canaries seem fine, I will create a Phab task for this,
and retake on Monday with Andrew's help :].