Create backwards compatible changes to catalyst (deploy_env.py ) to be able to accept other charts besides wikifunctions (currently wikifunctions chart is hardcoded

This is already done! See here. I am in the middle of updating Wikilambda tests to use this. I'll cc you on the changes when I send them to gerrit.

Create a new mwext-catalyst-daily-api-key (or something like that) in jenkins that can be shared by every extension that will be added under the catalyst-daily-extensions

Each token has a limit of 10 envs running at a time to avoid overwhelming our cluster. We could start out with one API key and increase if needed.

In T424411#11863366, @jeena wrote:

Create backwards compatible changes to catalyst (deploy_env.py ) to be able to accept other charts besides wikifunctions (currently wikifunctions chart is hardcoded

This is already done! See here. I am in the middle of updating Wikilambda tests to use this. I'll cc you on the changes when I send them to gerrit.

Oh, awesome!! Is there a phab task I can link to this one? Yes please do CC me on the changes when they land on gerrit.

Create a new mwext-catalyst-daily-api-key (or something like that) in jenkins that can be shared by every extension that will be added under the catalyst-daily-extensions

Each token has a limit of 10 envs running at a time to avoid overwhelming our cluster. We could start out with one API key and increase if needed.

Okay great, good to know

@jeena do you think this in the AC sounds correct for a next step or way of thinking about this after you update the wikilambda tests in gerrit?
create new file in jjb to run daily visual editor tests (and soon others) against ephemeral catalyst environments

Oh, awesome!! Is there a phab task I can link to this one? Yes please do CC me on the changes when they land on gerrit.

This is the task: T423330
I just uploaded it to gerrit and added you

@jeena do you think this in the AC sounds correct for a next step or way of thinking about this after you update the wikilambda tests in gerrit?
create new file in jjb to run daily visual editor tests (and soon others) against ephemeral catalyst environments

I'm not sure on whether you need a new file or just a new job definition, like in jjb/mediawiki-extensions.yaml?

In T424411#11863441, @jeena wrote:

Oh, awesome!! Is there a phab task I can link to this one? Yes please do CC me on the changes when they land on gerrit.

This is the task: T423330
I just uploaded it to gerrit and added you

@jeena do you think this in the AC sounds correct for a next step or way of thinking about this after you update the wikilambda tests in gerrit?
create new file in jjb to run daily visual editor tests (and soon others) against ephemeral catalyst environments

I'm not sure on whether you need a new file or just a new job definition, like in jjb/mediawiki-extensions.yaml?

Yeah, I'm not sure either I was thinking about something like jjb/mediawiki-catalyst-daily.yaml so it's easy to see what runs on catalyst daily as more extensions onboard onto this and it might make following the patterns easier. But adding into mediawiki-extensions.yaml works too, will do whatever you and the catalyst team think is best. I want to add Visual Editor and then shortly after Campaign Events and just watch them for a while.

In T424411#11863468, @vaughnwalters wrote:

In T424411#11863441, @jeena wrote:

Oh, awesome!! Is there a phab task I can link to this one? Yes please do CC me on the changes when they land on gerrit.

This is the task: T423330
I just uploaded it to gerrit and added you

@jeena do you think this in the AC sounds correct for a next step or way of thinking about this after you update the wikilambda tests in gerrit?
create new file in jjb to run daily visual editor tests (and soon others) against ephemeral catalyst environments

I'm not sure on whether you need a new file or just a new job definition, like in jjb/mediawiki-extensions.yaml?

Yeah, I'm not sure either I was thinking about something like jjb/mediawiki-catalyst-daily.yaml so it's easy to see what runs on catalyst daily as more extensions onboard onto this and it might make following the patterns easier. But adding into mediawiki-extensions.yaml works too, will do whatever you and the catalyst team think is best. I want to add Visual Editor and then shortly after Campaign Events and just watch them for a while.

I don't think we have an opinion on that part. Your idea sounds nice to me.

In T424411#11863486, @jeena wrote:

In T424411#11863468, @vaughnwalters wrote:

In T424411#11863441, @jeena wrote:

@jeena do you think this in the AC sounds correct for a next step or way of thinking about this after you update the wikilambda tests in gerrit?
create new file in jjb to run daily visual editor tests (and soon others) against ephemeral catalyst environments

I'm not sure on whether you need a new file or just a new job definition, like in jjb/mediawiki-extensions.yaml?

Yeah, I'm not sure either I was thinking about something like jjb/mediawiki-catalyst-daily.yaml so it's easy to see what runs on catalyst daily as more extensions onboard onto this and it might make following the patterns easier. But adding into mediawiki-extensions.yaml works too, will do whatever you and the catalyst team think is best. I want to add Visual Editor and then shortly after Campaign Events and just watch them for a while.

I don't think we have an opinion on that part. Your idea sounds nice to me.

Okay great, I'll wait for 1277759 to land and then see if i can build out jjb/mediawiki-catalyst-daily.yaml for VisualEditor and others.

@vaughnwalters 👋 hi there. Does the new chart already exist somewhere?

I wanted to mention early that right now the main resource bottleneck that we have in the cluster is pods. We would want to make sure that the new chart uses a single pod if possible; it can be composed of multiple containers it it needs them.

On a related note, we keep talking about refactoring the wikifunctions precisely for the same reason. Currently it creates 5 pods and we will need to reduce that at some point if we want to keep Catalyst-based CI viable.

In T424411#11866234, @jnuche wrote:

@vaughnwalters 👋 hi there. Does the new chart already exist somewhere?

I wanted to mention early that right now the main resource bottleneck that we have in the cluster is pods. We would want to make sure that the new chart uses a single pod if possible; it can be composed of multiple containers it it needs them.

On a related note, we keep talking about refactoring the wikifunctions precisely for the same reason. Currently it creates 5 pods and we will need to reduce that at some point if we want to keep Catalyst-based CI viable.

Heya Jaime, it won't be a new chart, planning on just using what is already in ci-charts for mediawiki and then jjb/mediawiki-catalyst-daily.yaml (doesn't exist yet) will reference that.

So, the only one I have working now is visual editor which I believe has just one pod https://ve-test-042426.catalyst.wmcloud.org ("id":5960). Other extensions running on a daily job would need their own pod I think, but the timing could be staggered on that. And the ones that I am looking at right now (VisualEditor, CampaignEvents) don't have any services. But it would need to be a fresh install for each extension being tested, once per day like the wikifunctions daily run.

If any of this seems wrong please let me know!

Heya Jaime, it won't be a new chart, planning on just using what is already in ci-charts for mediawiki and then jjb/mediawiki-catalyst-daily.yaml (doesn't exist yet) will reference that.

So, the only one I have working now is visual editor which I believe has just one pod https://ve-test-042426.catalyst.wmcloud.org ("id":5960). Other extensions running on a daily job would need their own pod I think, but the timing could be staggered on that. And the ones that I am looking at right now (VisualEditor, CampaignEvents) don't have any services. But it would need to be a fresh install for each extension being tested, once per day like the wikifunctions daily run.

If any of this seems wrong please let me know!

Gotcha, you wanted the change to deploy_env.py because the wikifunctions chart used to be hardcoded.

But one pod per CI job in the future is perfect (re: services we can cross that bridge when we get there). Sorry for the noise!

In T424411#11869951, @jnuche wrote:

Heya Jaime, it won't be a new chart, planning on just using what is already in ci-charts for mediawiki and then jjb/mediawiki-catalyst-daily.yaml (doesn't exist yet) will reference that.

So, the only one I have working now is visual editor which I believe has just one pod https://ve-test-042426.catalyst.wmcloud.org ("id":5960). Other extensions running on a daily job would need their own pod I think, but the timing could be staggered on that. And the ones that I am looking at right now (VisualEditor, CampaignEvents) don't have any services. But it would need to be a fresh install for each extension being tested, once per day like the wikifunctions daily run.

If any of this seems wrong please let me know!

Gotcha, you wanted the change to deploy_env.py because the wikifunctions chart used to be hardcoded.

Yep, exactly

But one pod per CI job in the future is perfect (re: services we can cross that bridge when we get there). Sorry for the noise!

Oh, it is not noise at all! I am a n00b to this so I really appreciate any thoughts and feedback you have! And please if you have any more insight to it or suggestions, let me know! Thx Jaime!

Change #1285491 had a related patch set uploaded (by Vaughn Walters; author: Vaughn Walters):

[integration/config@master] jjb: Add catalyst-daily-VisualEditor

https://gerrit.wikimedia.org/r/1285491

Change #1285491 merged by jenkins-bot:

[integration/config@master] jjb: Add catalyst-daily-VisualEditor

https://gerrit.wikimedia.org/r/1285491

Change #1285895 had a related patch set uploaded (by Vaughn Walters; author: Vaughn Walters):

[integration/config@master] jjb: Adding in Zuul paramters

https://gerrit.wikimedia.org/r/1285895

vwalters opened https://gitlab.wikimedia.org/repos/test-platform/catalyst/catalyst-ci-client/-/merge_requests/20

Adding mediawikiCore.ingress for the mediawiki chart

vwalters closed https://gitlab.wikimedia.org/repos/test-platform/catalyst/catalyst-ci-client/-/merge_requests/20

Adding mediawikiCore.ingress for the mediawiki chart

Change #1285895 merged by jenkins-bot:

[integration/config@master] jjb: [catalyst-daily-VisualEditor] Add in Zuul parameters

https://gerrit.wikimedia.org/r/1285895

Tests are green for visual editor running against catalyst env now https://integration.wikimedia.org/ci/view/selenium-daily/job/catalyst-daily-VisualEditor/6/

Change #1287941 had a related patch set uploaded (by Vaughn Walters; author: Vaughn Walters):

[integration/config@master] jjb: [catalyst-daily-CampaignEvents] Add CampaignEvents job

https://gerrit.wikimedia.org/r/1287941

Change #1294369 had a related patch set uploaded (by Vaughn Walters; author: Vaughn Walters):

[integration/config@master] jjb: [wikilambda-catalyst-end-to-end-daily] Email on daily failure only

https://gerrit.wikimedia.org/r/1294369

Change #1294369 merged by jenkins-bot:

[integration/config@master] jjb: [wikilambda-catalyst-end-to-end-daily] Email on daily failure/unstable only

https://gerrit.wikimedia.org/r/1294369

Of note, Central Notice is the first extension that needs a seed to pass (see T427006#12015674) but I don't anticipate it being the last. Side quest to figure out where the best place to create seed data.