@Pchelolo we looked at this and the request id and id were different as well. Any idea where the duplication occurs? I'd be curious to help debug this if you're not sure right away where it is.

Moving to radar and ping to @Pchelolo that tags changed events look to be sent twice.

@Milimetric The entry point for generating the tags change events is located at https://github.com/wikimedia/mediawiki-extensions-EventBus/blob/master/includes/EventBusHooks.php#L533 if you wanna debug it yourself

I will look into it soon.

I haven't looked at the data, but are meta.id and meta.dt different? If so, these are likely just different events. IIRC revision tags can change after the fact (right?). If so, each time they do would result in a different event (with the same rev id) but different tags.

I came across this issue when looking to use mediawiki_revision_tags_change to count tagged edits for the Growth Team and chatting with the Product Analytics team about it. Based on that discussion, and as @Ottomata mentions above, multiple rows for the same revision ID are expected because tags can change.

Digging into the last year's worth of data suggests that something isn't the way we'd expect it to be. Consistently from 2019-01 through 2020-01, I can find a few hundred thousand revisions per month (some millions in July–September) that have the same values for the revision ID and meta.dt columns. However, if we also include the meta.id column, none of these are duplicates, meaning that EventBus does regard these as separate events. I take this to mean that there are multiple parts of the MW code that call the entry point on a single edit, resulting in duplicate rows with the same timestamp, rather than combine all these changes into a single event. One of the examples I did find when looking at the Growth Team's data is that the second row added the "visualeditor" tag.

Below are a couple of the queries and results from running them. I use Python's format() call to insert the year and month. Let me know if I've missed something.

Query to get duplicates by rev ID and timestamp:

SELECT "{year}-{month}" AS log_month, SUM(IF(num_rows > 1, 1, 0)) AS num_duplicated_ts
FROM (
    SELECT rev_id, meta.dt, count(*) AS num_rows
    FROM event.mediawiki_revision_tags_change
    WHERE year = {year}
    AND month = {month}
    GROUP BY rev_id, meta.dt
) AS mrtc

Query to get duplicates by rev ID, timestamp, and event ID:

SELECT "{year}-{month}" AS log_month, SUM(IF(num_rows > 1, 1, 0)) AS num_duplicated_ids
FROM (
    SELECT rev_id, meta.dt, meta.id, count(*) AS num_rows
    FROM event.mediawiki_revision_tags_change
    WHERE year = {year}
    AND month = {month}
    GROUP BY rev_id, meta.dt, meta.id
) AS mrtc

Resulting data from 2019:

The two duplicate count columns corresponds to the two queries above.

Seems like any fix here requires digging into mediawiki code to see where those multiple events are being generated. CC-ing @WDoranWMF for prioritization on their end.

Hm yeah if the events have different meta.ids, that indicates that the MW hook that generates these events is being fired multiple times.

In T218246#5941239, @Ottomata wrote:

Hm yeah if the events have different meta.ids, that indicates that the MW hook that generates these events is being fired multiple times.

Hey are the meta.ids and meta.dts being filled in at send time by the MW hook or are they blank and filled in by EventBus/EventGate at receive/processing time?

EventBus is the MW extension that registers the hook handler to send the event to EventGate. meta.id and meta.dt are set by EventBus in MW.

I finally got to look into this. How and where are you running these queries? I would need some examples of events that are considered duplicates to figure out what's happening here.

Queries can be run in hadoop (ssh to 1107) , data is on event database. Remember to do >knit to run queries as kerberos authentication is now required.

Here's an example of a duplicated events pair:

As you can see, these events are not really duplicates - first, some android app edit tags are added, and then, almost instantly, apps-suggested-edits tag is added. The latter tag is added by the WikimediaEditorTasks extension via an edit hook. So, basically what is happening is that if there's 2 or more extensions adding tags to a certain edit, we emit 2 or more events about this edit, since the tags changes are not merged together between the extensions.

To further proof that this is what's happening, the following query

SELECT * FROM ( SELECT rev_id, meta.dt, count(*) AS num_rows FROM event.mediawiki_revision_tags_change WHERE year = 2020 AND month = 03 GROUP BY rev_id, meta.dt ) as rows sort by rows.num_rows desc limit 5;

Gives the result:

401981919	2020-03-07T01:18:18Z	2
401477581	2020-03-04T13:35:25Z	2
402529653	2020-03-08T14:32:52Z	2
403044128	2020-03-10T02:43:00Z	2
404756992	2020-03-17T10:16:28Z	2

Thus we at most have 2 tags-change events in the same revision/timestamp.

Trying to group it by tags as well:

SELECT * FROM ( SELECT rev_id, meta.dt, count(*) AS num_rows FROM event.mediawiki_revision_tags_change WHERE year = 2020 AND month = 03 GROUP BY rev_id, meta.dt, tags ) as rows sort by rows.num_rows desc limit 5;

Gives the result:

404532630	2020-03-16T08:13:12Z	2
405022725	2020-03-18T11:35:21Z	2
404532643	2020-03-16T08:13:17Z	2
404532677	2020-03-16T08:13:31Z	2
404509786	2020-03-16T05:04:56Z	2

Which means that true duplicates do exist. Poking deeper.

So, I have established that 'true' duplicates do exist in the previous comments.

Running:

SELECT SUM(rows.num_rows) FROM ( SELECT count(*) AS num_rows FROM event.mediawiki_revision_tags_change WHERE year = 2019 AND month = 12 GROUP BY rev_id, meta.dt, tags ) as rows WHERE rows.num_rows > 1;

We get 2900 - which is much lower and nicer then 253137 reported in a comment above.

Running

SELECT * FROM ( SELECT rev_id, tags, count(*) AS num_rows FROM event.mediawiki_revision_tags_change WHERE year = 2019 AND month = 12 GROUP BY rev_id, meta.dt, tags ) as rows WHERE rows.num_rows > 1;

shows that all the duplicates are for the 'fileimporter' tag.