Given ongoing discussion on the Headless VE service and its path to production, I'm parking this for the moment in Needs Info / Blocked for the moment.

+cc @jhathaway as DE1 SRE ambassador (FYI only)

In T431497#12102323, @Scott_French wrote:

Given ongoing discussion on the Headless VE service and its path to production, I'm parking this for the moment in Needs Info / Blocked for the moment.

NOTE: This task will likely be superseded by the SCROLL entry-point task. See https://wikitech.wikimedia.org/wiki/SCROLL for relevant trail heads and https://wikitech.wikimedia.org/wiki/SCROLL/Docs/Guide for an overview of the process.

+cc @jhathaway as DE1 SRE ambassador (FYI only)

@Scott_French, hi! An update and a resulting question for you (I think)...

Update
Last week, DPE, Editing, and SRE (via @jhathaway) aligned on moving forward with a technical pilot (T432286) to evaluate/estimate what resources Headless VE requires under realistic traffic.

Resulting question
What (if any) remaining questions/risks do you think need to be addressed before you can move forward with the request this ticket is describing?

In T431497#12151902, @ppelberg wrote:

@Scott_French, hi! An update and a resulting question for you (I think)...

Update
Last week, DPE, Editing, and SRE (via @jhathaway) aligned on moving forward with a technical pilot (T432286) to evaluate/estimate what resources Headless VE requires under realistic traffic.

Great, thanks for the update on where things stand, and thanks @jhathaway for representing SRE here.

Resulting question
What (if any) remaining questions/risks do you think need to be addressed before you can move forward with the request this ticket is describing?

Could you expand on what you mean when you say "... before you can move forward with the request this ticket is describing"? (i.e., clarify what is the request)

Context: My understanding is that this was opened by way of the Service-deployment-requests New Service Request template, e.g., as linked from New service request. There are two key ways in which reality has drifted from that process (aside from, e.g., out of date references to tasks such as "machine allocation"):

1. Review of the production readiness of a new service has been subsumed by the SCROLL process.
2. New-service setup is no longer primarily request-based, and is mostly focused on consulting engagements where we (ServiceOps) guide service owners through that process (i.e., self-service). For example, this is one of the main motivations for the bi-weekly Kubernetes Office Hours.

So! IIUC, this is a bit of a chicken and egg problem! Yes, Editing would like to deploy Headless VE. But, ServiceOps is hesitant for reasons. The agreed next step is to do a 'technical pilot' to evaluate the ues of a headless browser for this. T432286: [MILESTONE] Run technical pilot to evaluate Headless VE resource requirements.

See also T432046: Propose parameters for headless VE pilot and T433024: Implement sampling requirements for Headless VE technical pilot.

clarify what is the request

IIUC, there are 2 requests then:

1. How can we deploy Headless VE now so we can evaluate it as part of the technical pilot. Who does what part? Who write the helm chart? (related: T428174: Standard helm chart for simple service-utils nodejs apps). We'd like to do this ASAP.

2. What are SRE ServiceOps's criteria for which the tech pilot would be considered accepted for a regular production service.

Review of the production readiness of a new service has been subsumed by the SCROLL process.

BTW, We've asked about SCROLL before, and IIUC, we were told that we need to understand if the service SHOULD be rolled about before we engage with the SCROLL process.

Thanks for the follow-up @Ottomata.

In T431497#12167389, @Ottomata wrote:

IIUC, there are 2 requests then:

1. How can we deploy Headless VE now so we can evaluate it as part of the technical pilot. Who does what part? Who write the helm chart? (related: T428174: Standard helm chart for simple service-utils nodejs apps). We'd like to do this ASAP.

2. What are SRE ServiceOps's criteria for which the tech pilot would be considered accepted for a regular production service.

For #1, I think this goes back to what was mentioned in T431497#12161367 - i.e., ServiceOps can provide 1:1 support to service owners who are setting up their service. It sounds like there's a decision that the system under evaluation for the pilot (I do not know the details) will exist in production in some form, so there's an emphasis on front-loading service setup work. That's helpful to know.

For #2, I don't have an answer for you, and would defer to folks who have been involved in those discussions. While SCROLL gives a pretty detailed definition of, e.g., soft-launch entry requirements, if the pilot is specifically focused on resource requirements, then presumably there would need to be a decision threshold on projected resource usage (i.e., what is maintainable or not).

In any case, I'll flag this within the team (to the extent that certain aspects are ServiceOps-specific, while others are broader SRE concerns), and hopefully we can get this moving!

Review of the production readiness of a new service has been subsumed by the SCROLL process.

BTW, We've asked about SCROLL before, and IIUC, we were told that we need to understand if the service SHOULD be rolled about before we engage with the SCROLL process.

Not quite, or perhaps not quite in the way you mean here.

The idea is that SCROLL begins with engagement at the design stage, which gives us all (Developers and SREs) a chance to discuss the design and its suitability for production use, including iterating on / mitigating potential architectural concerns. Thus, "understand if the service SHOULD be rolled out" has two meanings:

1. If you mean "should" in the sense of "we believe this is the correct course of action, as a product decision" (i.e., "we should build something that fulfills this need"), then yes, absolutely that should happen first.
2. If you mean "should" in the sense of "a binary decision on a specific realization of a design," then no, since agreement is a consequence of that design-phase discussion / iteration.

Hope that helps clarify.

Thank you!

If you mean "should" in the sense of "we believe this is the correct course of action, as a product decision" (i.e., "we should build something that fulfills this need"), then yes, absolutely that should happen first.

Well, I suppose the decision in on this point is "we believe this is a good product but we need to run it so we can experiment and A/B test it to know for sure.". But, yes, the decided hypothesis is this is something we should build.

If you mean "should" in the sense of "a binary decision on a specific realization of a design," then no, since agreement is a consequence of that design-phase discussion / iteration.

Great, and I think the way they want to make this decision (as part of the design-phase) is to try it and get approval (or alternatives).

I think...maybe something that is causing some chicken/egg red tape friction is:

The idea is that SCROLL begins with engagement at the design stage

This sounds great! But also SCROLL is brand new and this project has already been going through both

• https://wikitech.wikimedia.org/wiki/Prep_Pantry
• https://wikitech.wikimedia.org/wiki/SRE/Data_Persistence/Design_Review

Which are also "please engage with us at the design phase" processes. Notably, neither of those processes directly address what SCROLL is trying to cover. Prep Pantry is about derived data architecture and pipelines, DP Design Review is about storage for serving, and SCROLL is about online services for product features running with SRE ServiceOps support (IIUC).

(Maybe a learning for all of these processes (they all have been KR hypothesis at some point) is that one day they should be collected into a single process somehow.)

Anyway, what I'm kind of hearing is that we should convert this task into a SCROLL task and follow the SCROLL process. Does that sound right?

We do need to have the conversations about harmonizing Prep Pantry and SCROLL in the general case. @MLechvien-WMF have chatted briefly about it and should catch up again, now that we have this ticket as a real data point :) I'd say that discussion is out of scope for this ticket.

Anyway, what I'm kind of hearing is that we should convert this task into a SCROLL task and follow the SCROLL process. Does that sound right?

Hmm, from what I understand, no. Based on T431497#12161367 and T431497#12169461, I propose:

1. Editing should follow the self-service path for setting up Headless VE in Kubernetes, consulting with ServiceOps per their current process.
2. With that done, we can run the Headless VE pilot and subsequent A/B test, which will help assess resource utilization and whether the overall product intervention is worth investing in.
3. Depending on the outcomes of those:
   • Editing has also already agreed to, depending on the above outcomes, throw away this prototype and revisit the design, including Headless VE.
   • With a candidate architecture in mind post the pilot, Editing / DPE will go through SCROLL (in general Prep Pantry should direct teams toward SCROLL!)

If that sounds right, this ticket can be closed as there's no need to engage ServiceOps via a ticket to set up the deployment.

If that sounds right, this ticket can be closed as there's no need to engage ServiceOps via a ticket to set up the deployment.

Can we just reuse the task for our own purposes to deploy this for the tech pilot then? I've already renamed the task; we can update the description too.

Sounds good to me, and we should assign to a point of contact in Editing to drive this.