Page MenuHomePhabricator
Create Task
Maniphest T269160

Set up internal eventstreams instance exposing all streams declared in stream config (and in kafka jumbo)
Closed, ResolvedPublic
Actions

Description

stream.wikimedia.org is a public eventstreams instance, and as such only exposes an explicitly declared list of streams that do not have PII.

We recently added a nice browser UI to eventstreams. https://stream.wikimedia.org/v2/ui. It'd be handy to have an internal instance deployed to make viewing and debugging streams easier. We'd have to ssh tunnel to that instance (unless we put it behind SSO?), but even that would be nice.

Since an eventstreams instance can only be connected to a single Kafka cluster, we can only expose streams that exist in our 'aggregate' kafka cluster: jumbo-eqiad (which is not cross DC).

Details

Show related patches Customize query in gerrit

Related Objects

Event Timeline

Ottomata created this task.Dec 1 2020, 7:48 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 1 2020, 7:48 PM
Ottomata updated the task description. (Show Details)Dec 1 2020, 7:49 PM
Ottomata updated the task description. (Show Details)
Ottomata updated the task description. (Show Details)
gerritbot added a comment.Dec 1 2020, 8:29 PM

Change 644612 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/deployment-charts@master] Add new service eventstreams-internal

https://gerrit.wikimedia.org/r/644612

gerritbot added a project: Patch-For-Review.Dec 1 2020, 8:29 PM
Ottomata added projects: serviceops, Service-deployment-requests.Dec 1 2020, 8:31 PM
Ottomata added a subscriber: akosiaris.
Restricted Application added projects: SRE, Services. · View Herald TranscriptDec 1 2020, 8:31 PM
Ottomata added a subscriber: JMeybohm.Dec 2 2020, 8:39 PM
• fdans triaged this task as Medium priority.Dec 10 2020, 5:53 PM
• fdans moved this task from Incoming to Event Platform on the Analytics board.
• fdans added a project: Analytics-Kanban.
Ottomata updated the task description. (Show Details)Jan 5 2021, 3:00 PM
elukey subscribed.Jan 13 2021, 11:11 AM

Reserved port 4992 in https://wikitech.wikimedia.org/wiki/Service_ports

gerritbot added a comment.Jan 13 2021, 11:20 AM

Change 655879 had a related patch set uploaded (by Elukey; owner: Elukey):
[labs/private@master] Add eventstreams-internal k8s service dummy token config

https://gerrit.wikimedia.org/r/655879

elukey added a comment.Jan 13 2021, 11:35 AM

I am following https://wikitech.wikimedia.org/wiki/Kubernetes#Add_a_new_service, will file the other changes for puppet private/public repos and then I'll follow up on the helm changes :)

gerritbot added a comment.Jan 13 2021, 2:55 PM

Change 655879 merged by Elukey:
[labs/private@master] Add eventstreams-internal k8s service dummy token config

https://gerrit.wikimedia.org/r/655879

elukey mentioned this in rLPRI840c0e2c8e43: Add eventstreams-internal k8s service dummy token config.Jan 13 2021, 3:25 PM
gerritbot added a comment.Jan 14 2021, 11:23 AM

Change 656129 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Add deployment config for a new k8s service - eventstreams-internal

https://gerrit.wikimedia.org/r/656129

gerritbot added a comment.Jan 14 2021, 11:28 AM

Change 656129 merged by Elukey:
[operations/puppet@production] Add deployment config for a new k8s service - eventstreams-internal

https://gerrit.wikimedia.org/r/656129

Ottomata moved this task from Next Up to In Code Review on the Analytics-Kanban board.Jan 14 2021, 3:04 PM
Ottomata moved this task from Backlog to In Progress Before Value Streams Kickoff (August 15th) on the Event-Platform board.Jan 14 2021, 7:34 PM
gerritbot added a comment.Jan 20 2021, 2:38 PM

Change 644612 merged by Elukey:
[operations/deployment-charts@master] Add new service eventstreams-internal

https://gerrit.wikimedia.org/r/644612

elukey added a comment.Jan 20 2021, 5:29 PM

Get up to deploying the service in staging, it seems working! Updated all the steps in https://wikitech.wikimedia.org/wiki/Kubernetes#Add_a_new_service

elukey claimed this task.Jan 21 2021, 6:09 PM
elukey moved this task from In Code Review to In Progress on the Analytics-Kanban board.
Ottomata added a comment.Jan 21 2021, 6:26 PM

Oo we'll also want eventstreams-internal.svc.* LVS set up too.

Ottomata added a comment.Jan 21 2021, 9:47 PM

@elukey it works! I realized that since this service is not proxied via varnish/ATS, we don't have an X-Client-IP header set, which external eventstreams requires and uses for throttling the user. I'll make a fix to the repo for this. You can proceed with deployment to eqiad and codfw clusters.

BTW, we don't need 2 replicas in staging, so we should add a values-staging.yaml and set replicas: 1 there.

gerritbot added a comment.Jan 22 2021, 3:11 PM

Change 657852 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/deployment-charts@master] eventstreams - move client_ip_connection_limit setting to helmfile values

https://gerrit.wikimedia.org/r/657852

gerritbot added a comment.Jan 22 2021, 3:23 PM

Change 657852 merged by Ottomata:
[operations/deployment-charts@master] eventstreams - move client_ip_connection_limit setting to helmfile values

https://gerrit.wikimedia.org/r/657852

gerritbot added a comment.Jan 22 2021, 3:43 PM

Change 657860 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/deployment-charts@master] eventstreams-internal - only 1 replica needed in staging

https://gerrit.wikimedia.org/r/657860

gerritbot added a comment.Jan 22 2021, 3:46 PM

Change 657860 merged by Ottomata:
[operations/deployment-charts@master] eventstreams-internal - only 1 replica needed in staging

https://gerrit.wikimedia.org/r/657860

elukey added a comment.EditedJan 26 2021, 3:29 PM

Created all the TLS certs and configs as described in https://wikitech.wikimedia.org/wiki/Enable_TLS_for_Kubernetes_deployments#Create_and_place_certificates and ran puppet on deploy1001. In theory if everything is fine I should be now able to proceed with:

cd /srv/deployment-charts/helmfile.d/services/eventstreams-internal; helmfile -e production -i apply on deploy1001

Waiting for @JMeybohm's greenlight before proceeding :)

• Mholloway subscribed.Jan 26 2021, 8:26 PM
JMeybohm added a comment.Jan 27 2021, 3:44 PM
In T269160#6777382, @elukey wrote:

Waiting for @JMeybohm's greenlight before proceeding :)

Looks good to me, go ahead! (But it's "helmfile -e [codfw|eqiad] ..." 🙂 )

elukey added a comment.Jan 27 2021, 3:47 PM
In T269160#6780685, @JMeybohm wrote:
In T269160#6777382, @elukey wrote:

Waiting for @JMeybohm's greenlight before proceeding :)

Looks good to me, go ahead! (But it's "helmfile -e [codfw|eqiad] ..." 🙂 )

I was definitely going for eqiad/codfw but I wanted to know if you were paying attention! thanks :)

elukey added a comment.Jan 27 2021, 3:56 PM
Error: pods is forbidden: User "eventstreams-internal" cannot list resource "pods" in API group "" in the namespace "eventstreams-internal"

This is the nice error msg that I got after running helmfile, going to check what the problem is.

JMeybohm added a comment.EditedJan 27 2021, 4:01 PM

You probably have not yet deployed the admin part (the new namespace etc.) to codfw/eqiad.

elukey added a comment.EditedJan 27 2021, 4:04 PM
In T269160#6780761, @JMeybohm wrote:

You probably have not yet deployed the admin part (the new namespace etc.) to codfw/eqiad.

Yes I realized it after adding the comment, I deployed the admin part only for staging of course. So I should proceed with:

ssh deploy1001
sudo -i; cd /srv/deployment-charts/helmfile.d/admin/codfw/; kube_env admin codfw; ./cluster-helmfile.sh -i apply
sudo -i; cd /srv/deployment-charts/helmfile.d/admin/eqiad/; kube_env admin eqiad; ./cluster-helmfile.sh -i apply

Then retry with the deploy. Does it sound ok?

JMeybohm added a comment.Jan 27 2021, 4:06 PM

Apart from you testing my attention again (kube_env admin [codfw|eqiad]), this looks good, yes :-)

elukey added a comment.Jan 27 2021, 4:56 PM

es-internal deployed in both eqiad and codfw, next steps are:

  • test locally on kubernetes nodes with @Ottomata
  • add LVS VIPs in front of the services
Ottomata added a comment.Jan 27 2021, 5:00 PM

@elukey app logs look normal; it'll be easier to test once LVS is up! :D Let's go!

gerritbot added a comment.Feb 2 2021, 10:10 AM

Change 661067 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Add conftool data for eventstreams-internal (new VIP)

https://gerrit.wikimedia.org/r/661067

gerritbot added a comment.Feb 2 2021, 10:19 AM

Change 661071 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Add eventstreams-internal to service_catalog

https://gerrit.wikimedia.org/r/661071

gerritbot added a comment.Feb 2 2021, 10:22 AM

Change 661072 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::kubernetes::worker: add empty stanza for eventstreams-internal

https://gerrit.wikimedia.org/r/661072

elukey added a comment.Feb 2 2021, 10:23 AM

I have followed https://wikitech.wikimedia.org/wiki/LVS#Add_a_new_load_balanced_service, first steps have code reviews opened, going to wait for reviewers before proceeding :)

gerritbot added a comment.Feb 3 2021, 3:24 PM

Change 661386 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/dns@master] Add eventstreams-internal VIP DNS config

https://gerrit.wikimedia.org/r/661386

gerritbot added a comment.Feb 3 2021, 3:37 PM

Change 661067 merged by Elukey:
[operations/puppet@production] Add conftool data for eventstreams-internal (new VIP)

https://gerrit.wikimedia.org/r/661067

gerritbot added a comment.Feb 3 2021, 4:05 PM

Change 661386 merged by Elukey:
[operations/dns@master] Add eventstreams-internal VIP DNS config

https://gerrit.wikimedia.org/r/661386

gerritbot added a comment.Feb 3 2021, 4:33 PM

Change 661071 merged by Elukey:
[operations/puppet@production] Add eventstreams-internal to service_catalog

https://gerrit.wikimedia.org/r/661071

gerritbot added a comment.Feb 3 2021, 4:36 PM

Change 661072 merged by Elukey:
[operations/puppet@production] role::kubernetes::worker: add empty stanza for eventstreams-internal

https://gerrit.wikimedia.org/r/661072

elukey added a comment.Feb 3 2021, 4:53 PM

Next step is https://wikitech.wikimedia.org/wiki/LVS#Configure_the_load_balancers, going to do it tomorrow with Valentin since we'll need to restart some Pybals :)

gerritbot added a comment.Feb 4 2021, 9:39 AM

Change 661687 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Set the lvs_setup flag for eventstreams-internal

https://gerrit.wikimedia.org/r/661687

gerritbot added a comment.Feb 4 2021, 9:53 AM

Change 661687 merged by Elukey:
[operations/puppet@production] Set the lvs_setup flag for eventstreams-internal

https://gerrit.wikimedia.org/r/661687

Stashbot added a comment.Feb 4 2021, 10:05 AM

Mentioned in SAL (#wikimedia-operations) [2021-02-04T10:05:47Z] <elukey> restart pybal on lvs2010 (low-traffic standby) to pick up new changes for eventstreams-internal (new VIP) - T269160

Stashbot added a comment.Feb 4 2021, 10:08 AM

Mentioned in SAL (#wikimedia-operations) [2021-02-04T10:08:53Z] <elukey> restart pybal on lvs1016 (low-traffic standby) to pick up new changes for eventstreams-internal (new VIP) - T269160

Stashbot added a comment.Feb 4 2021, 10:13 AM

Mentioned in SAL (#wikimedia-operations) [2021-02-04T10:13:31Z] <elukey> restart pybal on lvs2009 (low-traffic active) to pick up new changes for eventstreams-internal (new VIP) - T269160

Stashbot added a comment.Feb 4 2021, 10:15 AM

Mentioned in SAL (#wikimedia-operations) [2021-02-04T10:15:07Z] <elukey> restart pybal on lvs1015 (low-traffic active) to pick up new changes for eventstreams-internal (new VIP) - T269160

gerritbot added a comment.Feb 4 2021, 10:24 AM

Change 661695 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Set the monitoring_setup flag for eventstreams-internal

https://gerrit.wikimedia.org/r/661695

gerritbot added a comment.Feb 4 2021, 10:26 AM

Change 661695 merged by Elukey:
[operations/puppet@production] Set the monitoring_setup flag for eventstreams-internal

https://gerrit.wikimedia.org/r/661695

gerritbot added a comment.Feb 4 2021, 10:47 AM

Change 661697 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Set the production flag for eventstreams-internal

https://gerrit.wikimedia.org/r/661697

gerritbot added a comment.Feb 4 2021, 10:48 AM

Change 661697 merged by Elukey:
[operations/puppet@production] Set the production flag for eventstreams-internal

https://gerrit.wikimedia.org/r/661697

gerritbot added a comment.Feb 4 2021, 11:09 AM

Change 661702 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Set desired state for eventstreams-internal dns-disc

https://gerrit.wikimedia.org/r/661702

gerritbot added a comment.Feb 4 2021, 11:16 AM

Change 661702 merged by Elukey:
[operations/puppet@production] Set desired state for eventstreams-internal dns-disc

https://gerrit.wikimedia.org/r/661702

elukey added a comment.Feb 4 2021, 1:25 PM

Tested with ssh -L 4992:eventstreams-internal.discovery.wmnet:4992 -N mwmaint1002.eqiad.wmnet + https://localhost:4992 on the browser. I can see the UI and the streams!

@Ottomata to confirm that we are done :)

Ottomata added a comment.Feb 4 2021, 9:28 PM

Added some docs here:
https://wikitech.wikimedia.org/wiki/Event_Platform/Instrumentation_How_To#In_production

Ottomata added a comment.Feb 4 2021, 9:28 PM

Yeehaw thank you Luca!

elukey closed this task as Resolved.Feb 5 2021, 7:03 AM
JMeybohm awarded a token.Feb 5 2021, 8:59 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL