Offsite chat suggests that there's value in actually storing the raw "root" data sources that we build the feature tree from. Let's estimate the storage requirements for doing that.

I'm backing away from the scope of this—going to implement the feature for this task in the quick way, and save refactoring for a rainier day.

https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/JADE/+/476771/

Okay, thanks for the feedback. I like your suggestion and will try to extract all IP calculations into a third class, ParallelHostLimiter or something of the sort.

@Marostegui Hello! I've added a few summary columns and indexes to the link tables, and the resulting DDL would look like this:
https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/JADE/+/4644f1f0f9fa723062123df947af30b084d3b1f8/sql/

In T209731#4779180, @JAllemandou wrote:

Another comment about folders that I hadn't thought before having read your update in the description: I actually think that the chosen is not the most efficient.
In term of data-retrieval, using /wmf/data/ores/score/wiki=enwiki/model=editquality/model_version=2.1/ or /wmf/data/ores/score/model=editquality/model_version=2.1/wiki=enwiki/ is very similar.
However it is different at deletion step: the latter is a lot easier, as it involves only a single delete for the whole model.

@Halfak We need your feedback here. All I remember from IRC was that you wanted "not null" validation, but don't know what your reasoning is.

Thanks, this has been a helpful tangent!

In T200297#4778966, @daniel wrote:

I think we can support filtering by adding an index on the summary data? I was planning to do this unless there's a technical reason not to.

This would be a significant change to the schema and usage patterns as discussed in the RFC. It's actually a completely new usage pattern. It will need at least another feedback round with DBAs, and kind of invalidates the RFC discussion we had.

An index on that field would allow prefix matches. Denormalized summaries for display are conceptually quite different from filter criteria, though. This should be explored with a clearer view on the use cases .

Again: adding filter functionality is not a minor change, it really turns the entire proposals on its head.

In T200297#4778731, @daniel wrote:

In T200297#4778378, @awight wrote:

In T200297#4777873, @Halfak wrote:

Harej and I chatted about this yesterday, and I've started working on code to populate link table summary data. The tentative agreement is that we'll go ahead and implement this summary data for the damaging and goodfaith schemas. The reason this is happening now is that ALTERs to the link table will become prohibitively expensive, and might take a few months to perform live.

But the summary denormalization is still for augmentation (display), not filtering, no?

In T181542#4776713, @Ladsgroup wrote:

So the plan is to have the data in logstash which is already behind NDA.

In T200297#4777873, @Halfak wrote:

The second bullet point ("filter RC for edits which are ORES-nondamaging and JADE-damaging") seems like a product proposal. I think this one will be up to @Harej to prioritize.

The details are vague to me at this point, but I think the issue was that we would be spinning up several docker containers and they have to be able to find each other. If we're getting the URLs configured some other way, that's fine!

I checked on ores1001 and locally, and "unicode-escape" is a valid encoding. Let's call this "invalid" until it happens again.