In T437080#12294568, @JAllemandou wrote:

Even if we decide to have an Iceberg only metastore, I think the new data platform should also be capable of reading datasets as folders, allowing us to ingest files in folders in the same was we do now, without having to handle the Iceberg metadata and schema management aspect.

In case someone could review it, this is the patch that will write into MainStash and Memcached:

Ok, I'll work on that tomorrow, thanks!

For the record, I've done a test where our Flink application produces updates for Mediawiki, based on page_id. It is one update per page_id trending status change. Meaning that if a page changes from trending to not trending, or the opposite, an event is emitted.
The result is that we never reach 1k events on the 5 minutes window:

Yes, the regular output doesn't change, we create a new output eqiad.mediawiki.job.pageviewTrendingRelativeUpdate with the new format per wiki_id.

We have changed the design of the storage, as described in https://phabricator.wikimedia.org/T432352#12280536 and following recommendations, we'll produce an event per wiki_id with an array of trending page_ids on it. The event will be produced only every 5 minutes to avoid doing too many requests. Every event overrides the previous event, if a page_id is not in the list, it means it is not trending.

We had a meeting yesterday to talk about this topic, following @Ladsgroup recommendations, we are going to change the behavior to write per wiki_id rather than per page_id. Writing per page_id could cause too many writes, and it doesn't solve the future use case about getting the whole list of trending pages.
Writing by wiki_id, we can store an array with all trending page_ids inside. To avoid a lot of noise, we'll emit only one event per wiki_id every 5 minutes. By doing so, we can store the objects in Object Cache (MainStash) and we can also save them in Memcache. Every new object invalidates the old one, the list of trending pages will always be the full list, if a page isn't inside, it means the page is not trending.

The results from the test are these:

That's correct, we don't have yet exact numbers, but on some tests there were around 7k ~ 15k trending pages over all wikis.

We are preparing this data for the Attribution API and their team, which I believe will be consumed by Enterprise. We are only the producers, and they will be the readers.

Abut Read rate:

@Ottomata added some numbers here: https://phabricator.wikimedia.org/T432352#12231296

1. Using Object Cache

A staging application of mw-page-html-feature-counts-change-enrich-next has been deployed in Jumbo. After a week we'll see the differences between this topic and the production one.

Looking at how to enable this, the main blocker for it is Gobblin. Gobblin uses kafka-clients 1.1.1 but zstd was introduced in Kafka 2.1.0.

In T432352#12214749, @Eevans wrote:

Let's try to start from First Principles: What we have here is a single dataset. Our dataset (ostensibly) contains an entry for every page, and records whether or not the page is trending, with a timestamp to indicate when that trending status was last updated. We require that this dataset be indexed according to it's trending status, so that we can query it for a list of pages where is_trending=true.

I think that describes the problem, someone correct me though if I've gotten something wrong.

This describes the situation correctly, and the problem of the second endpoint (second table). The first endpoint doesn't need to query pages where is_trending=true, it only needs to query page_id=xxxxx to know if it's trending or not.

If the first table is released, we can enable one of the 2 endpoints (GET /public/analytics/pageview_trending_relative_per_page/:wiki_id/:page_id) for the Attribution API, and if I recall correctly, this was the most important of the two, so my assumption is that it will at least allow one of the use cases.
We can work on improving the second table if needed meanwhile.

I believe the only missing piece is to merge this patch that removes old stuff:

• https://gerrit.wikimedia.org/r/c/operations/puppet/+/1321974

Tables requested in this ticket: https://phabricator.wikimedia.org/T432352

Compaction cleanup.policy=compact,delete was added to pageview.trending.relative.v1 but it is causing other issues. There is an alert GobblinKafkaRecordsExtractedNotEqualRecordsExpected that gets confused as the number of offsets doesn't match the number of records. We also think that enabling compaction removes some valuable data for the Datalake, so we can to disable compaction now that the topic is controlled.

Hi @Eevans, thanks for looking into it!

I changed a couple of configs. https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/merge_requests/157

I believe zstd compression cannot be enabled if we don't upgrade the Gobblin Kafka client: https://gitlab.wikimedia.org/repos/data-engineering/gobblin-wmf/-/blob/main/pom.xml#L472-476

We could reduce the size of events on pageview.v1 by removing many fields. I don't think many of the fields are used for Trending right now, but pageview.v1 looks like a nice reusable stream, it could be used for other use cases, specially as it's using the same schema as the batch process.
Maybe @Ottomata you have some ideas about this, I'm not sure how we can measure the importance of the topic versus how much storage it uses.

Thanks! I'll start the pageview.trending.relative.v1 application now and monitorize it during the day. Let's see if it grows that fast or not. Meanwhile I'm looking at some of the other options to reduce the size.

I think adding cleanup.policy = compact,delete to pageview.trending.relative.v1 is a quick win. I think it will reduce the data stored drastically. Do you think we could do it @brouberol?