Page MenuHomePhabricator
Create Task
Maniphest T348763

Make eventstreams-internal available to WMF staff without an ssh tunnel
Closed, ResolvedPublic
Actions

Description

eventstreams-internal is a non-public deployment of EventStreams that has access to all Event Platform streams. Currently can only be accessed by users with a WMF production ssh account via ssh tunneling.

Exposing this at a public domain with proper auth would allow Data Platform users to explore stream documentation and schemas using OpenAPI docs (e.g. this), as well as view live stream data in their browsers.

Original ticket description from Luca:

While deploying the new version of eventstreams, I noticed that the internal endpoint seems not used in ages:

https://grafana.wikimedia.org/d/znIuUcsWz/eventstreams?orgId=1&refresh=1m&var-dc=codfw%20prometheus%2Fk8s&var-service=eventstreams-internal&from=now-6M&to=now

From the logs on logstash (both eqiad and codfw) I don't see anything relevant either, maybe I am missing something but I am wondering if we should undeploy to simplify the maintenance.

Let me know!

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Ottomata moved this task from Stream Data Products to Components on the Event-Platform board.Oct 25 2024, 1:28 PM
Ottomata renamed this task from Traffic for eventstreams-internal seems to be zero for the past months to Make eventstreams-internal available to WMF staff without an ssh tunnel.Oct 25 2024, 1:43 PM
Ottomata updated the task description. (Show Details)
Ottomata mentioned this in T376841: Render human-readable schemas on schema.wikimedia.org.Oct 25 2024, 1:45 PM
CDanis subscribed.Oct 25 2024, 1:48 PM
BTullis added a subscriber: Sfaci.Oct 25 2024, 1:50 PM
In T348763#10262402, @Ottomata wrote:

@BTullis do you think it would be possible to add authentication and a public domain to this service?

Yes, I think that would be quite feasible. We could use the CAS/SSO implementation and authenticate to it using the OIDC protocol, as we are with Superset and Airflow and DataHub and MPIC.
There would need to be an LDAP group to whom the rights would be given, equivalent to the analytics-privatedata-users POSIX group, I suppose. We are already configuring a number of new LDAP groups with cross-checking in T375729, so I can't see a big problem with this part.

Then the next question would be where to do the authentcation.
We could either:

...or

  • add an authenticating reverse proxy using envoy or some other kind of service.

At first glance, the envoy based solution looks pretty neat and tidy, given that we already have envoy installed in every pod.

CDanis added a comment.Oct 25 2024, 2:38 PM
In T348763#10262531, @BTullis wrote:
  • add an authenticating reverse proxy using envoy or some other kind of service.

At first glance, the envoy based solution looks pretty neat and tidy, given that we already have envoy installed in every pod.

Envoy Gateway is a different thing from Envoy Proxy -- the former is a management layer around the latter. We only run Envoy Proxy.

But we're using oauth2-proxy on k8s in the aux cluster to front trace.wikimedia.org and it's been working fine for that -- link to our config. Other teams also use it on bare metal for access to Thanos and a few other pieces of infra.

BTullis added a comment.Oct 25 2024, 3:55 PM
In T348763#10262664, @CDanis wrote:

But we're using oauth2-proxy on k8s in the aux cluster to front trace.wikimedia.org and it's been working fine for that -- link to our config. Other teams also use it on bare metal for access to Thanos and a few other pieces of infra.

Nice! Thanks for that info.

phuedx added a comment.Oct 28 2024, 10:02 AM

Would it be valuable to move MPIC to using oauth2-proxy for consistency with these other systems?

• VirginiaPoundstone added a project: Data-Engineering-Radar.Jan 6 2025, 9:12 PM
Ottomata added a project: Data-Platform-SRE.Mar 6 2025, 2:52 AM
Gehel triaged this task as Medium priority.Mar 7 2025, 8:42 AM
Gehel moved this task from Incoming to Misc on the Data-Platform-SRE board.
kostajh subscribed.Jul 15 2025, 4:22 PM
Ottomata mentioned this in T401453: FY25-26 SDS2.1.2 Data Reliability - Debugging event loss.Aug 26 2025, 2:02 PM
JMeybohm edited projects, added: ServiceOps new, ServiceOps-Services-Oids; removed: serviceops-deprecated.Feb 11 2026, 3:44 PM
JMeybohm moved this task from Inbox to Radar (Awareness) on the ServiceOps new board.
JMeybohm subscribed.

Moving this to radar on ServiceOps new side sine we won't be driving an implementation here. Let us know if we can help with something.

JAllemandou claimed this task.Mar 9 2026, 8:40 AM
JAllemandou edited projects, added: Data-Platform-SRE (2026-03-06 - 2026-03-27); removed: Data-Platform-SRE.
JAllemandou moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2026-03-06 - 2026-03-27) board.
JAllemandou edited projects, added: Data-Engineering (Q3 FY25/26 January 1st - March 31th); removed: Data-Engineering.Mar 9 2026, 2:41 PM
JAllemandou moved this task from Next Up to In progress on the Data-Engineering (Q3 FY25/26 January 1st - March 31th) board.
bking subscribed.Mar 17 2026, 4:11 PM
JAllemandou moved this task from In Progress to Blocked/Waiting on the Data-Platform-SRE (2026-03-06 - 2026-03-27) board.Mar 18 2026, 8:59 AM
Gehel edited projects, added: Data-Platform-SRE (2026-03-27 - 2026-04-17); removed: Data-Platform-SRE (2026-03-06 - 2026-03-27).Mar 27 2026, 9:52 AM
Gehel moved this task from Backlog - project to Blocked/Waiting on the Data-Platform-SRE (2026-03-27 - 2026-04-17) board.
Ahoelzl edited projects, added: Data-Engineering (Q4 FS25/26 April 1st - June 30st); removed: Data-Engineering (Q3 FY25/26 January 1st - March 31th).Apr 3 2026, 10:24 PM
Ahoelzl moved this task from Next Up to In progress on the Data-Engineering (Q4 FS25/26 April 1st - June 30st) board.
Gehel edited projects, added: Data-Platform-SRE (2026-04-24 - 2026-05-15); removed: Data-Platform-SRE (2026-03-27 - 2026-04-17).Apr 24 2026, 9:15 AM
Gehel moved this task from Backlog - project to Blocked/Waiting on the Data-Platform-SRE (2026-04-24 - 2026-05-15) board.
atsuko subscribed.May 5 2026, 1:32 PM
atsuko claimed this task.May 6 2026, 2:16 PM
atsuko added a subscriber: JAllemandou.
atsuko added a comment.EditedMay 8 2026, 10:06 AM

Plan:

  1. Take @JAllemandou diff for httpd_cas modularisation and finish it. For test, applying it and compare with functional deployment.
  2. Adapt httpd_cas for eventstreams chart and deploy next version of chart.
/usr/bin/helmfile \
  --file /home/atsuko/src/operations/deployment-charts/helmfile.d/dse-k8s-services/turnilo/helmfile.yaml \
  --chart /home/atsuko/src/operations/deployment-charts/charts/turnilo \
  --environment dse-k8s-eqiad diff --color --detailed-exitcode --context 5
atsuko changed the task status from Open to In Progress.May 8 2026, 1:41 PM
atsuko moved this task from Blocked/Waiting to In Progress on the Data-Platform-SRE (2026-04-24 - 2026-05-15) board.
gerritbot added a comment.May 11 2026, 9:52 AM

Change #1283791 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/deployment-charts@master] Add auth_proxy.httpd_cas module

https://gerrit.wikimedia.org/r/1283791

gerritbot added a project: Patch-For-Review.May 11 2026, 9:52 AM

Change #1285739 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/deployment-charts@master] Migrated turnilo to auth_proxy.httpd_cas module

https://gerrit.wikimedia.org/r/1285739

atsuko added a comment.May 11 2026, 10:08 AM

Result output is https://phabricator.wikimedia.org/P92446

gerritbot added a comment.May 11 2026, 2:09 PM

Change #1283791 merged by jenkins-bot:

[operations/deployment-charts@master] Add auth_proxy.httpd_cas module

https://gerrit.wikimedia.org/r/1283791

gerritbot added a comment.May 11 2026, 2:09 PM

Change #1285739 merged by jenkins-bot:

[operations/deployment-charts@master] Migrated turnilo to auth_proxy.httpd_cas module

https://gerrit.wikimedia.org/r/1285739

atsuko added a comment.May 11 2026, 2:17 PM

Going to push updates for turnilo and start converting eventstreams-internal

Maintenance_bot removed a project: Patch-For-Review.May 11 2026, 2:31 PM
atsuko added a comment.May 11 2026, 2:56 PM

Plan

  1. update openstream chart so it will have external httpd_cas port that is exposed to 30443
  2. we create all DNS records (public and private)
  3. we deploy the app, which sets up ingress
  4. we check that https://<service>.discovery.wmnet:30443 works
  5. we enable the ATS proxy and edge caching configuration
gerritbot added a comment.Mon, May 18, 10:35 AM

Change #1288502 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/puppet@production] deployment_server: add eventstreams-internal to dse-k8s-eqiad

https://gerrit.wikimedia.org/r/1288502

gerritbot added a project: Patch-For-Review.Mon, May 18, 10:35 AM
gerritbot added a comment.Mon, May 18, 12:43 PM

Change #1288832 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/deployment-charts@master] admin_ng/dse-k8s: add eventstreams-internal to dse-k8s-eqiad

https://gerrit.wikimedia.org/r/1288832

gerritbot added a comment.Mon, May 18, 12:52 PM

Change #1288502 merged by Atsuko:

[operations/puppet@production] deployment_server: add eventstreams-internal to dse-k8s-eqiad

https://gerrit.wikimedia.org/r/1288502

gerritbot added a comment.Mon, May 18, 1:01 PM

Change #1288832 merged by jenkins-bot:

[operations/deployment-charts@master] admin_ng/dse-k8s: add eventstreams-internal to dse-k8s-eqiad

https://gerrit.wikimedia.org/r/1288832

Maintenance_bot removed a project: Patch-For-Review.Mon, May 18, 1:31 PM
atsuko added a comment.EditedMon, May 18, 1:54 PM

Created the namespaces in dse-k8s-eqiad, unblocked testing of the new chart/configuration.

gerritbot added a comment.Tue, May 19, 2:10 PM

Change #1289357 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/deployment-charts@master] draft: upgrade eventstream chart to ingress

https://gerrit.wikimedia.org/r/1289357

gerritbot added a project: Patch-For-Review.Tue, May 19, 2:10 PM
gerritbot added a comment.Wed, May 20, 1:50 PM

Change #1289978 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/deployment-charts@master] eventstreams: convert configs

https://gerrit.wikimedia.org/r/1289978

gerritbot added a comment.Wed, May 20, 1:50 PM

Change #1289979 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/deployment-charts@master] eventstreams: copy eventstreams-internal to dse

https://gerrit.wikimedia.org/r/1289979

gerritbot added a comment.Wed, May 20, 2:44 PM

Change #1289978 merged by jenkins-bot:

[operations/deployment-charts@master] eventstreams: convert configs

https://gerrit.wikimedia.org/r/1289978

atsuko added a comment.EditedWed, May 20, 5:44 PM

Got service working on dpe-k8s-eqiad, need to do:

  1. Register service with idp (Application Not Authorized to Use CAS)
  2. Figure production certificate (for now it generated internal one)
  3. Finish diffs massaging and review, so far I
    • edited some of the vendor templates
    • didn't backport debug functionality yet

but otherwise it is matter of a few hours of work.

gerritbot added a comment.Tue, May 26, 9:25 AM

Change #1293663 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/puppet@production] idp: adding stream-internal.w.o to allowed services

https://gerrit.wikimedia.org/r/1293663

gerritbot added a comment.Tue, May 26, 9:51 AM

Change #1293663 merged by Atsuko:

[operations/puppet@production] idp: adding stream-internal.w.o to allowed services

https://gerrit.wikimedia.org/r/1293663

gerritbot added a comment.Wed, May 27, 11:30 AM

Change #1294257 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/deployment-charts@master] httpd-cas: config option to disable httpd-cas

https://gerrit.wikimedia.org/r/1294257

gerritbot added a comment.Wed, May 27, 2:46 PM

Change #1294257 merged by jenkins-bot:

[operations/deployment-charts@master] httpd-cas: config option to disable httpd-cas

https://gerrit.wikimedia.org/r/1294257

gerritbot added a comment.Wed, May 27, 2:47 PM

Change #1289357 merged by jenkins-bot:

[operations/deployment-charts@master] eventstreams: upgrade chart to ingress and idp

https://gerrit.wikimedia.org/r/1289357

gerritbot added a comment.Wed, May 27, 3:00 PM

Change #1289979 merged by jenkins-bot:

[operations/deployment-charts@master] eventstreams: copy eventstreams-internal to dse

https://gerrit.wikimedia.org/r/1289979

gerritbot added a comment.Wed, May 27, 3:28 PM

Change #1294326 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/dns@master] Provision stream-internal.w.o

https://gerrit.wikimedia.org/r/1294326

gerritbot added a comment.Wed, May 27, 3:30 PM

Change #1294327 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/puppet@production] trafficserver: enable stream-internal.w.o

https://gerrit.wikimedia.org/r/1294327

atsuko added a comment.EditedWed, May 27, 3:40 PM

https://stream-internal.wikimedia.org is available

gerritbot added a comment.Wed, May 27, 4:51 PM

Change #1294326 merged by Atsuko:

[operations/dns@master] Provision stream-internal.w.o

https://gerrit.wikimedia.org/r/1294326

gerritbot added a comment.Wed, May 27, 4:59 PM

Change #1294327 merged by Atsuko:

[operations/puppet@production] trafficserver: enable stream-internal.w.o

https://gerrit.wikimedia.org/r/1294327

atsuko moved this task from In Progress to Done on the Data-Platform-SRE (2026-04-24 - 2026-05-15) board.Wed, May 27, 5:16 PM
Maintenance_bot removed a project: Patch-For-Review.Wed, May 27, 5:31 PM
atsuko added a comment.EditedWed, May 27, 5:41 PM

Cleanup:

  1. Check monitorings
  2. Remove eventstream-internal from main k8s
  3. Remove extra configs from turnilo and eventstream
  4. Downgrade eventstreams-internal.discovery.wmnet from lvs to ingress lb
Gehel moved this task from Done to Reported on the Data-Platform-SRE (2026-04-24 - 2026-05-15) board.Fri, May 29, 7:09 AM
Gehel changed the task status from In Progress to Open.Fri, May 29, 8:40 AM
Gehel moved this task from Reported to In Progress on the Data-Platform-SRE (2026-04-24 - 2026-05-15) board.
gerritbot added a comment.Fri, May 29, 9:00 AM

Change #1295405 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/deployment-charts@master] Cleanup old values for turnilo and eventstreams

https://gerrit.wikimedia.org/r/1295405

gerritbot added a project: Patch-For-Review.Fri, May 29, 9:00 AM
gerritbot added a comment.Fri, May 29, 9:32 AM

Change #1295406 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/puppet@production] Cleanup eventstream-internal

https://gerrit.wikimedia.org/r/1295406

gerritbot added a comment.Fri, May 29, 9:32 AM

Change #1295409 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/puppet@production] service: move eventstreams-internal to lvs_setup

https://gerrit.wikimedia.org/r/1295409

gerritbot added a comment.Fri, May 29, 9:32 AM

Change #1295410 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/puppet@production] service: move eventstreams-internal to service_setup

https://gerrit.wikimedia.org/r/1295410

atsuko added a comment.EditedFri, May 29, 9:58 AM
  1. Slience id 7d0bff61-73e8-4e1f-b324-c107b5b54adc
  2. DNS removed
  3. lvs_setup, A:dnsbox OK
  4. lvs removed
  5. removed eventstreams-internal from staging, eqiad, codfw k8s
  6. merged https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1295405, applied admin-ng
  7. merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/1295406, applied
gerritbot added a comment.Fri, May 29, 10:18 AM

Change #1295409 merged by Atsuko:

[operations/puppet@production] service: move eventstreams-internal to lvs_setup

https://gerrit.wikimedia.org/r/1295409

gerritbot added a comment.Mon, Jun 1, 12:57 PM

Change #1295410 merged by Atsuko:

[operations/puppet@production] service: move eventstreams-internal to service_setup

https://gerrit.wikimedia.org/r/1295410

gerritbot added a comment.Mon, Jun 1, 2:36 PM

Change #1295405 merged by jenkins-bot:

[operations/deployment-charts@master] Cleanup old values for turnilo and eventstreams

https://gerrit.wikimedia.org/r/1295405

gerritbot added a comment.Mon, Jun 1, 3:18 PM

Change #1295406 merged by Atsuko:

[operations/puppet@production] Cleanup eventstream-internal

https://gerrit.wikimedia.org/r/1295406

Maintenance_bot removed a project: Patch-For-Review.Mon, Jun 1, 3:31 PM
atsuko closed this task as Resolved.Mon, Jun 1, 3:49 PM
atsuko added a subtask: T427839: 502/503 for mediawiki.page_change.v1 stream.Mon, Jun 1, 9:15 PM
atsuko changed the status of subtask T427839: 502/503 for mediawiki.page_change.v1 stream from Open to In Progress.Mon, Jun 1, 9:23 PM
atsuko closed subtask T427839: 502/503 for mediawiki.page_change.v1 stream as Resolved.Mon, Jun 1, 9:49 PM
Gehel moved this task from In Progress to Done on the Data-Platform-SRE (2026-04-24 - 2026-05-15) board.Fri, Jun 5, 8:05 AM
Gehel moved this task from Done to Reported on the Data-Platform-SRE (2026-04-24 - 2026-05-15) board.Fri, Jun 5, 8:07 AM
Ottomata added a comment.Tue, Jun 9, 1:02 PM

Thank you so so so so much everyone! This has already been useful to me many times!

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits