Change 469378 had a related patch set uploaded (by Smalyshev; owner: Smalyshev):
[operations/puppet@production] Disable Kafka update due to breakage

https://gerrit.wikimedia.org/r/469378

Change 469378 merged by Elukey:
[operations/puppet@production] Disable Kafka update due to breakage

https://gerrit.wikimedia.org/r/469378

The first error is at Oct 24 00:28:30 which matches Mediawiki deployment. Pinging @20after4 as the deployer.

Another problem on this - for some reason, when Updater fails, systemd keeps restarting in a cycle every 3 seconds. This creates unnecessary load on the machine, while not helping anything. We should fix that too.

Also, I can not access most hosts by ssh. Whatever updater problem is there, it should not block ssh, but except for 1003 and 2003, all others are unresponsive over ssh.

< _joe_> as to why it's impossible to ssh into those hosts, I guess it has to do with journald and some interactions with ssh

The message that comes from mediawiki.org looks like:

{"comment": "Stabl", "database": "mediawikiwiki", "meta": {"domain": "www.mediawiki.org", "dt": "2018-10-24T00:28:24.162300+00:00", "id": "b7aa095a-d723-11e8-be6d-1418776134a1", "request_id": "23e8fa7b-8c07-4ffc-ba43-d30e025b65cb", "schema_uri": "mediawiki/revision/create/3", "topic": "mediawiki.revision-create", "uri": "https://www.mediawiki.org/wiki/Extension:EditSubpages"}, "page_id": 22458, "page_is_redirect": false, "page_namespace": 102, "page_title": "Extension:EditSubpages", "parsedcomment": "Stabl", "performer": {"user_groups": ["*"], "user_is_bot": false, "user_text": "2001:56A:76EC:E800:3C6D:9381:428E:ACE5"}, "rev_content_changed": true, "rev_content_format": "text/x-wiki", "rev_content_model": "wikitext", "rev_id": 2930951, "rev_len": 6582, "rev_minor_edit": false, "rev_parent_id": 2930950, "rev_sha1": "7sk6l1w8g56i7aiff8kuursjb4p7goj", "rev_timestamp": "2018-10-24T00:28:06Z"}

Note the different format for "dt" timestamp. Timestamps from all other sources are normal, all timestamps from mediawiki.org are in weird format.

Mentioned in SAL (#wikimedia-operations) [2018-10-24T07:33:26Z] <gehel> powercycling wdqs1010 - T207817

Do we have a patch or should we roll back group0?

In T207817#4691569, @mmodell wrote:

Do we have a patch or should we roll back group0?

We have a workaround on the WDQS side (switching back to recent changes instead of kafka events). But the root cause isn't fixed, and it is unclear to me what change caused that issue. IMHO, this should be treated as a contract change and either needs to be communicated accordingly as a breaking change or rolled back.

Ping @Pchelolo.

This is happening because of https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/EventBus/+/468482/ as part of a fix for T207329: Clear watchlist on enwiki only removes 50 items at a time.

Interesting! I checked Jodatime stuff to make sure one of our Java based pipeline handled the timestamp format change, I'm surprised that Jackson can't parse this!

We could rollback https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/EventBus/+/468482/ - we can workaround it in change-prop and the issue fixed by that is much rarer, but eventually, we'd want to include millisecond resolution into the dt of the events.

@Pchelolo I can look into how to make Jackson parse it, it's probably possible, but I'd appreciate rolling it back for now until that is done.

Maybe jackson just can't parse this microsecond stuff? Maybe milliseconds are fine?

Change 469449 had a related patch set uploaded (by 20after4; owner: 20after4):
[mediawiki/extensions/EventBus@master] Revert "Set event datetime with microsecond resolution."

https://gerrit.wikimedia.org/r/469449

In T207817#4691885, @Ottomata wrote:

Interesting! I checked Jodatime stuff to make sure one of our Java based pipeline handled the timestamp format change, I'm surprised that Jackson can't parse this!

We can definitely parse that format if we wanted to, but we do have a fixed format configured. We could enable some magic, but it seems very bad form to autodetect a time format. There are too many ways to get it wrong.

Side note, technically, this change has introduced a timezone, which should change the semantic from a LocalDate to an Instance in Joda time terminology :) so either it was technically wrong before or it is now :)

Change 469451 had a related patch set uploaded (by 20after4; owner: 20after4):
[mediawiki/extensions/EventBus@wmf/1.33.0-wmf.1] Revert "Set event datetime with microsecond resolution."

https://gerrit.wikimedia.org/r/469451

Maybe jackson just can't parse this microsecond stuff? Maybe milliseconds are fine?

Judging by the error message, it cripples on the . after seconds.

@Smalyshev Also please note we're only including microseconds and not milliseconds because of the HHVM incompatibility with PHP and the need to support HHVM for now. For all our purposes milliseconds should be enough, but seems like Jackson will not like them either out of the box.

ok anyone care to give the ol' +2 to revert this on the branch? https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/EventBus/+/469451/

Change 469451 merged by jenkins-bot:
[mediawiki/extensions/EventBus@wmf/1.33.0-wmf.1] Revert "Set event datetime with microsecond resolution."

https://gerrit.wikimedia.org/r/469451

Mentioned in SAL (#wikimedia-operations) [2018-10-24T16:00:33Z] <twentyafterfour> 15:59:06 Synchronized php-1.33.0-wmf.1/extensions/EventBus/: revert "Set event datetime with microsecond resolution." on 1.33.0-wmf.1 refs T207817 (duration: 00m 56s)

Judging by the quick skim of the Jackson code it supports DateTimeFormatter.ISO_INSTANT format that should support fractional seconds parsing.

However, this incident raises a question whether the change applied to EventBus was indeed backward-compatible. We've checked the known consumers (one we apparently forgot to check) that they at least don't cripple on parsing fractional seconds in ISO 8601, but maybe there exist more things introducing the fractional seconds into the dt will break. Maybe our initial approach was incorrect @Ottomata ?

@Pchelolo I think a public notification on Kafka format change would be a great idea. Since various tools may use Kafka streams (and streams derived from them), and some tools may be sensitive to exact formats.

The spec at https://github.com/wikimedia/mediawiki-event-schemas/blob/master/jsonschema/resource_change/1.yaml defines "dt" as:

format: date-time
description: the time stamp of the event, in ISO8601 format

But I'm not sure what "ISO8601 format" specifically means - ISO 8601 allows many format variations. Maybe we should specify it more strictly.

Change 469449 merged by jenkins-bot:
[mediawiki/extensions/EventBus@master] Revert "Set event datetime with microsecond resolution."

https://gerrit.wikimedia.org/r/469449

With T207873 deployed (planned next Monday) Updater should be able to parse such timestamps, but I'd appreciate announcement upfront, so I could test it. If it can wait till Nov 6 when I am back from vacation, even better.

Thanks @Smalyshev, I think you are write that changes like this should be announced a bit better. We'll try to do better next time.

I wonder...where should we announce changes like this? engineering@? wikitech-l?

engineering@? wikitech-l?

I think engineering is good, and probably wikitech too since the data can get into EventStreams as I understand and that is public.

(I'm guessing this is more of a "follow-up" task vs an "active situation" task, yes?)

I'm guessing this is more of a "follow-up" task vs an "active situation" task, yes?

Yes, by now it's definitely a follow-up tracking task.