Technically nothing prevents us from just enabling the new JobQueue for private wikis.

Note that ChangeProp and RESTBase update use-case doesn't support private wikis, but it's a matter of simple config flag to only submit jobs and not normal events.

The only concern I have is that the jobs are replicated to Hadoop and for private wikis they might contain some private information. @Nuria and @Ottomata - is that ok that jobs from private wikis will be replicate to hadoop or should we add some additional logic to filter them out?

Hm, currently the data we import into Hadoop is readable by anyone with a
Hadoop account (not just analytics-privatedata-users) (but overlap of
Hadoop by non privatedata-users is very small). We could change the
permissions on the event database (or even possibly just a few job topic
tables) to be readable only by analytics-privatedata-users.

In general I think this is ok, as we delete this data within the 90 days.

Hm, however, we’re trying to make internal ‘private’ cross DC data all go
over TLS. If we do this, we would want to have TLS enabled for main-eqiad
<-> main-codfw MirrorMaker instance. To do that we need to upgrade main
Kafkas first.

Hm, however, we’re trying to make internal ‘private’ cross DC data all go over TLS. If we do this, we would want to have TLS enabled for main-eqiad <-> main-codfw MirrorMaker instance. To do that we need to upgrade main Kafkas first.

Hm, that would significantly complicate and delay things. This is a part of our Q4 goal, what's your timeline on upgrading main kafka and enabling TLS for mirror-maker?

Some workarounds are possible, like not mirroring the private wiki events by sending them to special topics with some suffix, but all what comes to mind is pretty ugly.

Timeline for upgrading main is Q4, but MirrorMaker +TLS wasn't in the plan. I don't think we should block your work though (private cross DC data has happened for a long time, we jut have to get incrementally better).

Ok, in that case, are you ok with enabling it in very near future and adding a mental note that this increases the priority of MirrorMaker + Kafka TLS work?

Change 425591 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[mediawiki/extensions/EventBus@master] Support per-event-type EventBus enabling configuration.

https://gerrit.wikimedia.org/r/425591

For posterity, in today's JobQueue biweekly meeting we agreed that having unencrypted mirroring of private wikis' data is not acceptable. Given that TLS work for EventBus' MirrorMaker is scheduled for next quarter, we have decided to temporarily switch mirroring off until the TLS work is done. In this way, we can proceed with enabling support for private wikis and start decommissioning the Redis machines tied to the old JobQueue transport mechanism.

Change 425601 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[operations/mediawiki-config@master] Enable EventBus for job events for all the wikis.

https://gerrit.wikimedia.org/r/425601

+1 :) To clarify: we haven't officially scheduled any MirrorMaker TLS work, but after we upgrade main Kafka clusters, it should be relatively easy to set this up.

Change 425591 merged by jenkins-bot:
[mediawiki/extensions/EventBus@master] Support per-event-type EventBus enabling configuration.

https://gerrit.wikimedia.org/r/425591

Change 425888 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[mediawiki/extensions/EventBus@wmf/1.31.0-wmf.29] Support per-event-type EventBus enabling configuration.

https://gerrit.wikimedia.org/r/425888

Change 425888 merged by Mobrovac:
[mediawiki/extensions/EventBus@wmf/1.31.0-wmf.29] Support per-event-type EventBus enabling configuration.

https://gerrit.wikimedia.org/r/425888

Mentioned in SAL (#wikimedia-operations) [2018-04-17T13:42:52Z] <mobrovac@tin> Synchronized php-1.31.0-wmf.29/extensions/EventBus/extension.json: Support per-event dispatch of events, file 1/3 - T191464 (duration: 03m 07s)

Mentioned in SAL (#wikimedia-operations) [2018-04-17T14:16:11Z] <mobrovac@tin> Synchronized php-1.31.0-wmf.29/extensions/EventBus/extension.json: Support per-event dispatch of events, file 1/3 - T191464 (duration: 03m 00s)

Mentioned in SAL (#wikimedia-operations) [2018-04-17T14:22:01Z] <mobrovac@tin> Synchronized php-1.31.0-wmf.29/extensions/EventBus/includes/EventBus.php: Support per-event dispatch of events, file 2/3 - T191464 (duration: 03m 06s)

Mentioned in SAL (#wikimedia-operations) [2018-04-17T14:25:38Z] <mobrovac@tin> Synchronized php-1.31.0-wmf.29/extensions/EventBus/includes/JobQueueEventBus.php: Support per-event dispatch of events, file 3/3 - T191464 (duration: 03m 07s)

Change 425601 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable EventBus for job events for all but wikitech wikis

https://gerrit.wikimedia.org/r/425601

Mentioned in SAL (#wikimedia-operations) [2018-04-24T20:03:09Z] <mobrovac@tin> Synchronized wmf-config/InitialiseSettings.php: Use EventBus for all wikis but wikitech - T191464 (duration: 01m 26s)