without triggering a compute job on a cache miss

Hm, IIUC, I think what folks have asked for is not to avoid triggering a compute job, but to avoid waiting for a compute job. They want the result to be computed and cached so the next time they fetch it will be available.

In T433607#12238260, @Ottomata wrote:

without triggering a compute job on a cache miss

Hm, IIUC, I think what folks have asked for is not to avoid triggering a compute job, but to avoid waiting for a compute job. They want the result to be computed and cached so the next time they fetch it will be available.

My understanding was that we were looking for best effort: Return a value if cached, or nothing otherwise. The context (as I understood it) was that of a user request that raced the pre-warm event (i.e. presumably the cache would be warmed, it just wasn't warmed in time for the first request). At any rate, this is the behavior consistent with Cache-control: only-if-cached semantics, so I think we should try to hew to that.

To get the behavior you're talking about, I think you could either: a) run the request in a thread, without waiting past {duration} for slow responses (presumably in the service layer). The full LAC request would continue and result in a cached value, even if a response to the client had already been sent. Or b) enqueue a job to pre-warm the cache on a miss.

With all of that said though, I think we should avoid prematurely optimizing here. If misses are common, then either there is a reliability issue with the pre-warm events, or we're consistently hitting the cache before the pre-warm can populate it (presumably the case where a read that quickly follows a save/new revision races the pre-warm). If it's a case of the latter, do we really want to incur the write amplification? Would it even help? Maybe instead, we can focus on what information we'd need to capture to understand how often this happens, and/or to understand why when it does.

Hm, good points.

However, in the case of missed events/jobs, if clients always use Cache-control: only-if-cached, then nothing will ever trigger a compute for the record (until the next edit happens). Doesn't this bypass one of the big advantages for LAC vs LAS: self-healing?

you could either: a) run the request in a thread, without waiting past {duration} for slow responses (presumably in the service layer)

Indeed, I think this is the other option.

Or b) enqueue a job to pre-warm the cache on a miss.

I don't think read clients should get involved in submitting cache warm jobs :). Let's keep that separate.

In T433607#12277457, @Ottomata wrote:

Hm, good points.

However, in the case of missed events/jobs, if clients always use Cache-control: only-if-cached, then nothing will ever trigger a compute for the record (until the next edit happens). Doesn't this bypass one of the big advantages for LAC vs LAS: self-healing?

It does if that's the behavior the client is choosing, yes. How common do you expect a missed event to be?

How common do you expect a missed event to be?

Probably not common!

It does if that's the behavior the client is choosing, yes.

So, the client is expected to choose if they want future requests to result in a cache-hit? Not really a big deal, but we should document this so they understand.

eevans opened https://gitlab.wikimedia.org/repos/sre/hoarde/-/merge_requests/13

Draft: Implement Cache-Control: only-if-cached semantics

In T436709#12335852, @Eevans wrote:

if the client hangs up before the LAC response (i.e. because its timeout is short), you should assume that the artifact will not be cached. In this case, LAC will propagate an error to the lambda, so that it does not continue working

I'm replying here to keep the discussion together.

Let's consider the following scenario if LAC works in this way.

Scenario:

• a lambda is expensive, computation could take 5+ seconds.
• a client use case can only tolerate 500ms latency, so they set a client timeout of 500ms
• an edit creates new revision_id: 101, but its corresponding precompute job fails for some reason (e.g. missed job event). Artifact for revision_id: 101 is not cached in LAC.
• client requests LAC for revision_id: 101, but it is not cached in LAC, so they hang up after 500ms.
• Because LAC propagates the error to the lambda so it stops working, the artifact for revision_id: 101 will never be cached.
• Every subsequent client request for revision_id: 101 will timeout after 500ms.

I think this negates one of the main pros of LAC over event driven materialized-view: LAC is self-healing.

Clients (LAC clients) should harmonize their timeouts with it. If they are unable to block for requests that take longer than some threshold, then they can background the request, or use Cache-Control: only-if-cached after that lands

I don't think it should be the responsibility of a client request to manage the responses for future client requests. There may be many different clients using LAC endpoints, which one should be the one that is responsible for backgrounding async requests for self healing?

Additionally, asking e.g. MW PHP (probably the main client of LAC) to background the async requests is quite cumbersome. They could maybe re-use some of the code we write for T436695: Linked Artifact Cache - Automated MediaWiki JobQueue precompute for this (in a Job or a DeferredUpdate), but it does it seem right to require LAC clients to manage what happens behind LAC (in the lambda)?

Thank Andrew for the scenario. Thats pretty much T436709 use-case! Also re-iterating, our initial thought was client does 500ms as they require. LAC and lambda work together to ensure the result is cached. Meaning LAC would have a high timeout, therefore allowing lambda to finish the work and cache. But lets discuss!

In T436709#12370166 @Eevans wrote:

My assumption is that delivery of the cache warm event is reliable. Obviously nothing is entirely reliable, but missed updates should be quite exceptional. If that's not the case, I think we should be talking about what is wrong there, and how to make it right. Additionally, if it is unreliable, then surely so are stream-processing produced artifacts (meaning: what is the alternative?).

If they are exceptional, then the problem is one of timing, a racing read that lands before the warm event has succeeded, in which case use only-if-cached and know that follow-up reads will succeed (at least those that arrive after computation has had time to complete; we cannot serve a result sooner than it can be generated).

TL;DR It sounds like we're looking for an engineering solution to a subset of high-latency lambda requests that overlap with a lost cache warm event. How frequently do we expect that to happen?

Additionally, if it is unreliable, then surely so are stream-processing produced artifacts (meaning: what is the alternative?).

Indeed! This was one of the arguments you used to convince me that the cache is better than a materialized view.

My assumption is that delivery of the cache warm event is reliable. Obviously nothing is entirely reliable, but missed updates should be quite exceptional.

MW JobQueue does not have an SLO. We may be asking for one (at least for our LAC precompute jobs) soon, but I'm not sure. The point is more that

• during regular option we know that while rare, events can be lost
• during bugs/outages, many events can be lost

As an example, here's a quick phab search for tickets about failing to enqueue jobqueue events: https://phabricator.wikimedia.org/search/query/MaDnnUROUQnv/#R

LAC's (cache) self-healing capability is one of its biggest advantages. If we advise reader clients to not tolerate cache misses, via client side timeouts or only-if-cached, and nothing ever 'heals' the cache in cases of missed precomputes, then LAC is not actually self-healing.

Let's step back a bit.

I think it's one thing if your client-facing service has an SLO that requires 100ms response times for 99p of requests, and your lambda's 99p response time is 150ms. Precomputation here seems like a reasonable optimization. A miss pushes you 50ms out of bounds, which may not be the experience you are after, but if rare, probably not the end of the world. It's another thing if you absolutely cannot tolerate more than 50ms of client latency against a lambda with a latency in the range of seconds to minutes, and you need a near 100% hit rate. I think this latter scenario is where you need to reevaluate the feasibility of your approach (and that is regardless of any implementation specifics under discussion here).

That said: I've made up all of the numbers above to be illustrative, so let's plug in the real values (even if we need to SWAG them). What is the distribution of latencies for some the more problematic lambda's under consideration? For those use-cases, what is the maximum acceptable latency of client requests (what will the timeout in use be)? What SLO is required? What is the expected hit rate? And, should we be including any requests that race pregeneration (because we cannot solve for those)?

I don't think it should be the responsibility of a client request to manage the responses for future client requests.

I agree with this actually, but how is it different for Hoarde to be doing this? It is the client you are asking to background the request, in this instance. The reason I suggested having callers do this¹, is that it pushes the burden up the chain. The resources have to be allocated upstream, rather than spawning a bunch of asynchronous work for Hoarde to do, and then hanging up.

¹: To be clear, I was hoping anyone that did this would merely run that request asynchronously, and populate the results in their UI best-effort after it completed.

so let's plug in the real values (even if we need to SWAG them)

How about just WAG? :)

What is the distribution of latencies for some the more problematic lambda's under consideration?

p50 - 1 second
p99 - 10 seconds

what is the maximum acceptable latency of client requests (what will the timeout in use be)?

500ms

What SLO is required?

I'm not sure how to WAG this

What is the expected hit rate?

Since the client's timeout is greater than the p50 lambda latency, this would directly correspond with client request success rate. So instead lets' WAG:

Expected client request success rate

99%.

This implies a reader client 99% hit rate.

should we be including any requests that race pregeneration (because we cannot solve for those)

I'm not sure if I'm fully following here, but I think you mean that LAC doesn't differentiate between precompute and client read requests, and I agree. So, an effective reader hit / success rate has to be measured on the reader client side, not the LAC side.

I don't think it should be the responsibility of a client request to manage the responses for future client requests.

I agree with this actually, but how is it different for Hoarde to be doing this? It is the client you are asking to background the request

I guess I mean the clients of LAC in general. This would be a feature of LAC that clients could make use of. But, I get your point about the benefits pushing and distributing the async complexity up to LAC reader clients. I just think that it adds unfair complexity to LAC reader clients. If we can support this at the platform/server level we have a better story for client adoption.

¹: To be clear, I was hoping anyone that did this would merely run that request asynchronously, and populate the results in their UI best-effort after it completed.

So the suggestion is to ask clients to always fetch async with no http client timeout, yes?

This might work in some cases, but I'm expecting that most of these LAC clients will be MW PHP.

E.g. MW Attribution API will fetch from LAC. There is no late-update UI feature here, it's a JSON HTTP API response that includes data from a lot of different sources. They can't block their Attribution API response on a high latency cache miss, so they'll have to have LAC request timeout.

I'd expect some LAC uses to be used on page load, and to bake the LAC result in to the page load -- no UI to late update here either. (I'm not sure what Server Side Suggestions plans to do.)

and you need a near 100% hit rate

BTW, I don't think anyone is hoping for this, but I would expect folks to hope that a missed precompute event does not mean that a new revision's LinkedArtifact will never be fetchable. So yes, something (I was expecting it to be Hoarde) will have to wait for a cache miss to resolve.

Otherwise, indeed, we need to explicitly document that clients are expected to async wait for the LAC response at least for the maximum (or upper percentile) of their known lambda's latency.

In T433607#12370877, @Ottomata wrote:

[ ... ]

Expected client request success rate

99%.

This implies a reader client 99% hit rate.

should we be including any requests that race pregeneration (because we cannot solve for those)

I'm not sure if I'm fully following here, but I think you mean that LAC doesn't differentiate between precompute and client read requests, and I agree. So, an effective reader hit / success rate has to be measured on the reader client side, not the LAC side.

Look at a wiki edit; You edit the content of a page, and in so doing you create a new revision of wikitext. After saving the editor expects to see the new HTML output of the page (an artifact!); A request for the artifact immediately follows the creation of a new revision. Or someone loads tone issues for a revision, edits to correct one or more, saves (thus creating a new revision), and expects to see the outcome reflected in the list of issues for the new revision. I assume that this synchronous pattern will be quite common. If that request races the prewarm (which is not hard to imagine for something taking seconds or minutes), then that is going to be a miss (even if the prewarm later succeeds). Do those misses count? You cannot solve for them without lifting one or more constraints (i.e. make the lambda quicker, or wait longer for it to complete).

[ ... ]

I guess I mean the clients of LAC in general. This would be a feature of LAC that clients could make use of. But, I get your point about the benefits pushing and distributing the async complexity up to LAC reader clients. I just think that it adds unfair complexity to LAC reader clients. If we can support this at the platform/server level we have a better story for client adoption.

How does this work in terms of resource management? It's one thing to have these requests coming from a queue, where we can establish constraints to avoid overwhelming hoarde and/or the lambdas. We can regulate concurrency if nothing else. For regular client requests (via MediaWiki, or otherwise), there are at least implicit limits on concurrency, everything making a request has to hold up its end of the exchange, or timeout and free up resources all the way down the chain (hoarde & lambda). If the client can fire off a request and disconnect while leaving hoarde (and the underlying lambdas) to carry out a disproportionate amount of work, what's to stop them from effectively DoS'ing us? How do we manage that? How do we apply back-pressure, or circuit-break and degrade gracefully?

Again, I think what this amounts to is having the client request (on a miss) do the job that the prewarm should —and may yet— do/have done. I'm not sure it's an actual problem, and if it is, I think a better solution would be to make the prewarm more reliable.

If the client can fire off a request and disconnect while leaving hoarde (and the underlying lambdas) to carry out a disproportionate amount of work, what's to stop them from effectively DoS'ing us?

This is a very convincing point. I did not consider this.

Let's document this and recommend to clients that if they want self-healing, they need to async wait for a cache miss to finish.

Alternatively, I suppose could do like https://wikitech.wikimedia.org/wiki/Search/CirrusStreamingUpdater#Saneitizer_(background_repair_process) and slowly iterate through all page_ids (and latest rev_ids) to manually ensure artifacts are available. But premature for now.

a better solution would be to make the prewarm more reliable.

I would love to. T120242. ;) (please don't respond to this, I am semi-trolling now)

eevans merged https://gitlab.wikimedia.org/repos/sre/hoarde/-/merge_requests/13

Implement Cache-Control: only-if-cached semantics

Hoarde v1.5.2 (supporting Cache-control: only-if-cached) has been deployed to staging.

Change #1352538 had a related patch set uploaded (by Ottomata; author: Ottomata):

[mediawiki/extensions/EventBus@master] LinkedArtifacts: Rename Fetcher to Client, add only-if-cached

https://gerrit.wikimedia.org/r/1352538

Change #1352538 merged by jenkins-bot:

[mediawiki/extensions/EventBus@master] LinkedArtifacts: Rename Fetcher to Client, add only-if-cached

https://gerrit.wikimedia.org/r/1352538