Page MenuHomePhabricator

wdqs-proxy on k8s: Enable egress to eventgate to post events
Open, Needs TriagePublic5 Estimated Story Points

Description

Posting to eventgate doesn't currently work. From the proxy logs:

{"timestamp":"2026-07-28T13:26:53.028968918Z","sequence":612,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.wikimedia.wdqs.events.QueryEventFilters","level":"WARN","message":"Event posting failure: connection timed out after 15000 ms: eventgate-analytics.discovery.wmnet/10.2.2.42:4592","threadName":"vert.x-eventloop-thread-0","threadId":24,"mdc":{},"ndc":"","hostName":"wdqs-main-external-proxy-79b7f66d87-7cbc2","processName":"/usr/lib/jvm/java-25-openjdk-amd64/bin/java","processId":1}

This is because egress to eventgate isn't enabled in our Kubernetes config.
@BTullis: "Eventgate access can be enabled in one of two different ways, either through external-services, or through the service mesh that uses envoy. We can look into the pros and cons of both options." We are going with the second option.

Our current understanding of the requirements is that we need to

  • enable egress at all for the service mesh
  • add a discovery listener on eventgate-analytics
  • change the eventgate posting host from http://eventgate-analytics.discovery.wmnet:4592 to http://localhost:6004 to go through the mesh

Additionally we should treat this as an opportunity to learn about the service mesh and network policies work so we understand it better for our future work as a platform team. We are therefore upgrading the complexity of this task to account for the time spent learning and knowledge sharing.

AC:

  • update the chart to allow service mesh egress
  • update the helmfile to add the discovery listener and update the event posting URL
  • knowledge share about the service mesh - how does this work?

Event Timeline

I locally edited the network policy in wdqs-next to try to imitate what was done here. For mysterious reasons I could not actually apply an updated helmfile, but I could edit the network policy directly. With an updated network policy like this:

...
spec:
  egress:
  - ports:
    - port: 4592
      protocol: TCP
    to:
    - ipBlock:
        cidr: 10.2.2.42/32
  ingress:
  - {}
...

I ended up with this error:

{"timestamp":"2026-07-28T21:02:28.050622323Z","sequence":99,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.wikimedia.wdqs.events.QueryEventFilters","level":"WARN","message":"Event posting failure: Connection was closed","threadName":"vert.x-eventloop-thread-1","threadId":24,"mdc":{},"ndc":"","hostName":"wdqs-main-external-proxy-cbfd9d4f-x62bs","processName":"/usr/lib/jvm/java-25-openjdk-amd64/bin/java","processId":1}

So, it still doesn't work, but at least the connection was made (and then immediately closed).

Change #1326893 had a related patch set uploaded (by Lerickson; author: Lerickson):

[operations/deployment-charts@master] Enable posting to eventgate from the proxy.

https://gerrit.wikimedia.org/r/1326893

lerickson set the point value for this task to 5.

Change #1327186 had a related patch set uploaded (by Lerickson; author: Lerickson):

[operations/deployment-charts@master] Add egress to the service mesh.

https://gerrit.wikimedia.org/r/1327186

Change #1327186 merged by jenkins-bot:

[operations/deployment-charts@master] Add egress to the service mesh.

https://gerrit.wikimedia.org/r/1327186

Change #1326893 merged by jenkins-bot:

[operations/deployment-charts@master] Enable posting to eventgate from the proxy.

https://gerrit.wikimedia.org/r/1326893

Change #1331764 had a related patch set uploaded (by Lerickson; author: Lerickson):

[operations/deployment-charts@master] Update default Eventgate URL value in the WDQS chart.

https://gerrit.wikimedia.org/r/1331764

Change #1331764 merged by jenkins-bot:

[operations/deployment-charts@master] Update default Eventgate URL value in the WDQS chart.

https://gerrit.wikimedia.org/r/1331764