What was T130404: Setup ORES service in beta cluster for then?

In T141825#2513758, @greg wrote:

What was T130404: Setup ORES service in beta cluster for then?

When I built the service in the beta cluster my idea was to test the service itself. I mean if the service is broken, the review tool should not be broken. I have no strong preferences.

Change 302406 had a related patch set uploaded (by Ladsgroup):
Beta: move from ores.wikimedia.org to ores-beta.wmflabs.org

https://gerrit.wikimedia.org/r/302406

In T141825#2514785, @Ladsgroup wrote:

In T141825#2513758, @greg wrote:

What was T130404: Setup ORES service in beta cluster for then?

When I built the service in the beta cluster my idea was to test the service itself. I mean if the service is broken, the review tool should not be broken. I have no strong preferences.

If the service that lives in beta is broken then the extension in beta should also be broken. IOW: the setup in beta should be similar to production and contained within beta. That was my understanding of that original bug that I linked, which is why "we couldn't catch this in beta" surprised me greatly yesterday.

In T141825#2515793, @greg wrote:

That was my understanding of that original bug that I linked

... when I first read it back when it was created. (to clarify)

There were two bugs in the yesterday deployment. We couldn't catch the first one because the redis instance in the beta cluster is not password-secured unlike the production cluster (which has its own task in phab now), the second bug which made us to revert back was actually could be found if the beta cluster was using the ores in beta service.

@greg, I'm not sure what you're looking for here. We obviously agree with you that beta should look as much like prod as possible. As soon as we realized that it didn't and that was why these bugs slipped through, we filed these bugs and prioritized them "high".

If you are asking how we accidentally got to this point in the first place, I'm not sure that is a productive use of time beyond, "it was a mistake, we'll work to not repeat". OK?

In T141825#2515821, @Halfak wrote:

If you are asking how we accidentally got to this point in the first place, I'm not sure that is a productive use of time beyond, "it was a mistake, we'll work to not repeat". OK?

Actually, it's that. Because I don't want anyone else to fall through the cracks as well. I haven't really "owned" the deployment of new services in production in a strong way (much of that process/recommendations is owned by the Services Team) but every time I do see a new service going to production I make sure to tell the team/people to set it up in Beta Cluster first to catch issues like (one of the two of) yesterday's.

So, I'm trying to figure out how I can make it clear that for a service to go to production it needs to be setup in Beta Cluster as it would be in production first. (see eg my comments just now on T102512#2515942 ) I think a separate task about that (review of new service process) is warranted and I'll file one now.

In T141825#2515946, @greg wrote:

I think a separate task about that (review of new service process) is warranted and I'll file one now.

T141897: Review new service 'pre-deployment to production' checklist

Change 302406 merged by jenkins-bot:
Beta: move from ores.wikimedia.org to ores-beta.wmflabs.org

https://gerrit.wikimedia.org/r/302406

Well, I merged it. But I don't think we're still at the right level of beta integration...is ores-beta.wmflabs.org using the beta labs wikis as the source for the edits? or is using production wikis?

Basically I think we need to get rid of $wgOresWikiId = 'testwiki'; so we're not using the junk values generated by the fake testwiki option, but actual values calculated by ores.

In T141825#2514785, @Ladsgroup wrote:

When I built the service in the beta cluster my idea was to test the service itself. I mean if the service is broken, the review tool should not be broken. I have no strong preferences.

Beta is supposed to be super end-to-end integration testing, so if the service is broken, the wiki should be just as broken as it would be in production basically.

In T141825#2517811, @Legoktm wrote:

Well, I merged it. But I don't think we're still at the right level of beta integration...is ores-beta.wmflabs.org using the beta labs wikis as the source for the edits? or is using production wikis?

It uses production wikis

Basically I think we need to get rid of $wgOresWikiId = 'testwiki'; so we're not using the junk values generated by the fake testwiki option, but actual values calculated by ores.

Hmm, yeah. We might make a model for a wiki in beta. I will look into the possibility.