Flink is an implementation detail so maybe namespace: stream_enrichment_poc or something? We still don't know what our deployment model will be. Will we run a Flink Session cluster and submit multiple jobs to it? Or, will each job need its own Flink cluster and namespace?

In T321682#8345590, @Ottomata wrote:

Flink is an implementation detail so maybe namespace: stream_enrichment_poc or something? We still don't know what our deployment model will be. Will we run a Flink Session cluster and submit multiple jobs to it? Or, will each job need its own Flink cluster and namespace?

Good questions Andrew. I'd still be quite keen on the flink-operator approach, which is more similar to what we're trying for spark: T321686: Create namespaces and kubernetes users for spark-operator and for spark jobs

In that case we are looking to have a namespace for the spark-operator itself, then the operator watches other namespaces and deploys spark driver and executor pods in those pods, upon receipt of suitable requests.

I would have thought that a similar kind of setup for flink would be the most preferable solution too, but I'm happy to be guided by you if you have any particular suggestions.

For now, what about a simple approach for now of?

• user: stream_enrichment
• namespace: stream_enrichment

I'd prefer not to have the _poc suffix just in case it stays there forever (as can tend to happen).

Is this OK with everyone? Even if we experiment with the filnk-operator, it could all be done in one namespace I suppose, at this stage.

Sorry, you weren't a subscriber @gmodena - Is the suggestion above for a username and namespace acceptable?

I'd still be quite keen on the flink-operator approach

We are too!

I'd prefer not to have the _poc suffix just in case it stays there forever (as can tend to happen).

I'd prefer to have something that indicated that this was for development/experiment/testing/PoC, so that it is clear that it not for production use.

I'd prefer not to have the _poc suffix just in case it stays there forever (as can tend to happen).

I'd prefer to have something that indicated that this was for development/experiment/testing/PoC, so that it is clear that it not for production use.

OK. 👍

• user: stream_enrichment_poc
• namespace: stream_enrichment_poc

In T321682#8356153, @BTullis wrote:

• user: stream_enrichment_poc
• namespace: stream_enrichment_poc

Works for me. Can this namespace be shared with @bking and the search team? Our use cases / exploration needs overlap.

Change 851063 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Add a namespace for the stream-enrichment-poc on dse-k8s

https://gerrit.wikimedia.org/r/851063

Slight correction; it has to be hyphens and not underscores, for consistency.

I've added the service accounts: stream-enrichment-poc and stream-enrichment-poc-deploy
I have a CR for the namespace: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/851063

Now I'm setting up the kubectl config files on the deployment server, but I'm double-checking who should be permitted to deploy to this namespace.

@Ottomata and @bking will already have permission with the default settings, by virtue of being in the ops group. Is this sufficient, or do we need to permit more users (e.g. @gmodena ?) to deploy to this namespace? If so, is there a suitable group already in the data.yaml file or would we need to create one?

or do we need to permit more users (e.g. @gmodena ?) to deploy to this namespace?

Yes please, and @tchin and @Milimetric.

If so, is there a suitable group already in the data.yaml file or would we need to create one?

If we didn't mind adding some people that weren't going to use this atm, we could use analytics-admins and platform-engineering. Or, we could create a new one.

Folks one suggestion - we should aim to use the Deployment Pipeline (https://wikitech.wikimedia.org/wiki/Deployment_pipeline) and GitOps as much as possible, because it is really easy to kubectl apply things and forget to remove them. I am fully aware that we need to test first, and I support it 100%, but let's be organized and have few people at the time operating via kubectl. I'd also suggest to have clear rollback procedures in place, and a way to track what was applied and what wasn't.

In T321682#8356594, @elukey wrote:

Folks one suggestion - we should aim to use the Deployment Pipeline (https://wikitech.wikimedia.org/wiki/Deployment_pipeline) and GitOps as much as possible, because it is really easy to kubectl apply things and forget to remove them. I am fully aware that we need to test first, and I support it 100%, but let's be organized and have few people at the time operating via kubectl. I'd also suggest to have clear rollback procedures in place, and a way to track what was applied and what wasn't.

Thanks @elukey - I agree. As far as I'm aware, everything within this namespace is intended to use the deployment pipeline and helmfile for deployment. So I don't think that anyone is expecting to run kubectl manually for anything.
Please do correct me (anyone) if you feel that I've got this wrong.

In T321682#8356624, @BTullis wrote:

In T321682#8356594, @elukey wrote:

Folks one suggestion - we should aim to use the Deployment Pipeline (https://wikitech.wikimedia.org/wiki/Deployment_pipeline) and GitOps as much as possible, because it is really easy to kubectl apply things and forget to remove them. I am fully aware that we need to test first, and I support it 100%, but let's be organized and have few people at the time operating via kubectl. I'd also suggest to have clear rollback procedures in place, and a way to track what was applied and what wasn't.

+1.

Thanks @elukey - I agree. As far as I'm aware, everything within this namespace is intended to use the deployment pipeline and helmfile for deployment. So I don't think that anyone is expecting to run kubectl manually for anything.
Please do correct me (anyone) if you feel that I've got this wrong.

+1. Interaction with the cluster was expected to happen via Deployment Pipeline, and GitOps and not interactively.

Change 851063 merged by jenkins-bot:

[operations/deployment-charts@master] Add a namespace for the stream-enrichment-poc on dse-k8s

https://gerrit.wikimedia.org/r/851063

Change 854007 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Add kubectl files for stream-enrichment-poc on dse-k8s

https://gerrit.wikimedia.org/r/854007

Change 854007 merged by Btullis:

[operations/puppet@production] Add kubectl files for stream-enrichment-poc on dse-k8s

https://gerrit.wikimedia.org/r/854007

I have now deployed the changes required to:

• create this namespace on the dse-k8s cluster
• configure kube_env to allow kubectl and helmfile to work.

As we can see from the test below, the namespace exists with no resources present yet.

btullis@deploy1002:~$ kube_env stream-enrichment-poc dse-k8s-eqiad
btullis@deploy1002:~$ kubectl get pods
No resources found in stream-enrichment-poc namespace.

You can now go ahead and create the helm chart and helmfile deployment for this service.

In T321682#8374985, @BTullis wrote:

[...]

You can now go ahead and create the helm chart and helmfile deployment for this service.

Terrific! My plan is to pick this up in our current sprint. Do I understand correctly that deployments to DSE will require merging helm/helmfile changes to deployment-charts main branch? Or is it an option to trigger deployments from a working branch?

Do I understand correctly that deployments to DSE will require merging helm/helmfile changes to deployment-charts main branch?

Yes, I believe so. I think we would probably like to stick to deployments from the master branch, if possible.
Although I can see the appeal of using a working branch, the safer option is definitely to get it reviewed and approved first.
Feel free to add me to helm/helmfile reviews and let me know if you have something time-critical. I'll do my best not to impede your work.

In T321682#8379638, @BTullis wrote:

[...]

Although I can see the appeal of using a working branch, the safer option is definitely to get it reviewed and approved first.

Thanks for the ack.

Feel free to add me to helm/helmfile reviews and let me know if you have something time-critical. I'll do my best not to impede your work.

Thanks! I'll ping you for coordination as soon as I'll resume T320812.