Page MenuHomePhabricator
Create Task
Maniphest T390140

Eventstreams 'assignments' logstash field type
Closed, ResolvedPublic
Actions

Description

logstash / opensearch has been complaining about type mismatch for field assignments. Investigating further it does indeed look like eventstreams changes the field type depending on the operation.

For example this has been indexed id 6h_n1pUBuXzFNByTSDF_ (logstash)

assignments: eqiad.mediawiki.recentchange, codfw.mediawiki.recentchange
msg: Bulding assignments from passed in assignments

While for msg"=>"Final resolved Kafka assignments" (logstash dlq) the assignments field is an object:

"assignments"=>[{"topic"=>"eqiad.mediawiki.recentchange", "offset"=>-1, "partition"=>0}, {"topic"=>"codfw.mediawiki.recentchange", "offset"=>-1, "partition"=>0}],

cc @Ottomata

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
[eventstreams] Bump version 0.18.0operations/deployment-chartsmaster+2 -2
[eventstreams] Bump version 0.17.0operations/deployment-chartsmaster+2 -2
[eventstreams] Bump version 0.16.0operations/deployment-chartsmaster+2 -2
logstash: stringify 'assignments' from eventstreamsoperations/puppetproduction+37 -1
Customize query in gerrit
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Bump kafka-sse and conform logs to ecsrepos/data-engineering/eventstreams!25tchinkafkasse-0.5.3master
Nest assignment log under labelrepos/data-engineering/kafkasse!5tchinassignment-labelmaster
Bump KafkaSSE to use Winstonrepos/data-engineering/eventstreams!21tchinbump-kafka-ssemaster
Switch from bunyan to winstonrepos/data-engineering/kafkasse!2tchinuse-winstonmaster
Customize query in GitLab

Event Timeline

fgiunchedi created this task.Mar 27 2025, 9:26 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 27 2025, 9:26 AM
Stashbot added a comment.Mar 27 2025, 9:29 AM

Mentioned in SAL (#wikimedia-operations) [2025-03-27T09:29:34Z] <godog> silence LogstashKafkaConsumerLag and LogstashIndexingFailures for today for 1d - T390140

fgiunchedi updated the task description. (Show Details)Mar 27 2025, 9:32 AM
gerritbot added a comment.Mar 27 2025, 11:17 AM

Change #1131672 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] logstash: stringify 'assignments' from eventstreams

https://gerrit.wikimedia.org/r/1131672

gerritbot added a project: Patch-For-Review.Mar 27 2025, 11:18 AM
Ottomata added a subscriber: tchin.Mar 27 2025, 1:11 PM

Curious! I wonder if this changed recently as @tchin replaced service-runner with service-utils and (started?) producing ECS logs.

Either way, makes sense. Hm. We could just log assignments as an object in the first case too, setting topic but not the other details.

Or, perhaps better: don't set assignments in log data for the first message. It can be read out of the message string just fine.

Ottomata renamed this task from Eventstreams 'assignments' field type to Eventstreams 'assignments' logstash field type.Mar 27 2025, 1:11 PM
tchin added a comment.Mar 27 2025, 1:46 PM

Ah, the issue is in KafkaSEE, which still assumes bunyan. I'll have to standardize the logging inside that lib

tchin claimed this task.Mar 27 2025, 1:46 PM
tchin added a project: Data-Engineering (Q4 2025 April 1st - June 30th).
tchin added a comment.Mar 27 2025, 1:51 PM

Or a real quick way is to not pass in the new logger from eventstreams to KafkaSSE, that way it creates a bunyan logger and the logs won't appear as an error anymore in logstash

Ottomata added a comment.Mar 27 2025, 1:57 PM

Hm, this message doesn't look like it should be an error? Pretty normal operation?

Standardizing the logger in KafkaSSE too sounds better :)

not pass in the new logger from eventstreams to KafkaSSE

Would this mean that logs emitted from KafkaSSE would not associated with eventstreams service? Or would the fact that they come from the same service container be enough?

Whatever you decide is best will be fine! TY!

gerritbot added a comment.Mar 28 2025, 10:50 PM

Change #1131672 merged by Cwhite:

[operations/puppet@production] logstash: stringify 'assignments' from eventstreams

https://gerrit.wikimedia.org/r/1131672

colewhite subscribed.Mar 28 2025, 11:22 PM

We've rolled out a logstash filter to check for name KafkaSSE and to cast the assignments field into a string. This can be undone when it is no longer needed.

Maintenance_bot removed a project: Patch-For-Review.Mar 28 2025, 11:30 PM
• lmata moved this task from Inbox to Radar on the SRE Observability board.Apr 2 2025, 2:31 PM
CodeReviewBot added a project: Patch-For-Review.Jun 22 2025, 8:06 PM

tchin opened https://gitlab.wikimedia.org/repos/data-engineering/kafkasse/-/merge_requests/2

Draft: Switch from bunyan to winston

CodeReviewBot added a comment.Jun 30 2025, 2:21 PM

tchin merged https://gitlab.wikimedia.org/repos/data-engineering/kafkasse/-/merge_requests/2

Switch from bunyan to winston

Maintenance_bot removed a project: Patch-For-Review.Jun 30 2025, 2:31 PM
tchin moved this task from Next Up to In progress on the Data-Engineering (Q4 2025 April 1st - June 30th) board.Jun 30 2025, 4:44 PM
mforns moved this task from In progress to Ready to Deploy on the Data-Engineering (Q4 2025 April 1st - June 30th) board.Jul 7 2025, 4:30 PM
gmodena added a project: Event-Platform.Jul 8 2025, 8:42 AM
gmodena subscribed.
CodeReviewBot added a project: Patch-For-Review.Jul 8 2025, 3:11 PM

tchin opened https://gitlab.wikimedia.org/repos/data-engineering/eventstreams/-/merge_requests/21

Bump KafkaSSE to use Winston

CodeReviewBot added a comment.Jul 9 2025, 1:16 PM

tchin merged https://gitlab.wikimedia.org/repos/data-engineering/eventstreams/-/merge_requests/21

Bump KafkaSSE to use Winston

Maintenance_bot removed a project: Patch-For-Review.Jul 9 2025, 1:43 PM
gerritbot added a comment.Jul 9 2025, 2:17 PM

Change #1167638 had a related patch set uploaded (by TChin; author: TChin):

[operations/deployment-charts@master] [eventstreams] Bump version 0.16.0

https://gerrit.wikimedia.org/r/1167638

gerritbot added a project: Patch-For-Review.Jul 9 2025, 2:17 PM
gerritbot added a comment.Jul 9 2025, 7:19 PM

Change #1167638 merged by jenkins-bot:

[operations/deployment-charts@master] [eventstreams] Bump version 0.16.0

https://gerrit.wikimedia.org/r/1167638

Maintenance_bot removed a project: Patch-For-Review.Jul 9 2025, 7:30 PM
Ottomata moved this task from Backlog to Components on the Event-Platform board.Jul 21 2025, 8:27 PM
gerritbot added a comment.Jul 22 2025, 2:19 PM

Change #1171588 had a related patch set uploaded (by TChin; author: TChin):

[operations/deployment-charts@master] [eventstreams] Bump version 0.17.0

https://gerrit.wikimedia.org/r/1171588

gerritbot added a project: Patch-For-Review.Jul 22 2025, 2:19 PM
gerritbot added a comment.Jul 22 2025, 2:25 PM

Change #1171588 merged by jenkins-bot:

[operations/deployment-charts@master] [eventstreams] Bump version 0.17.0

https://gerrit.wikimedia.org/r/1171588

Maintenance_bot removed a project: Patch-For-Review.Jul 22 2025, 2:31 PM
Ahoelzl edited projects, added Data-Engineering (Q1 FY25/26 July 1st - September 30th); removed Data-Engineering (Q4 2025 April 1st - June 30th).Jul 25 2025, 10:58 PM
Ahoelzl moved this task from Next Up to Ready to Deploy on the Data-Engineering (Q1 FY25/26 July 1st - September 30th) board.
tchin added a comment.Jul 28 2025, 3:30 PM

assignments is now stringified in KafkaSSE, but in the logs I see that assignments is in normalized.dropped.no_such_field. Is there something I'm missing? @colewhite

colewhite added a comment.Jul 29 2025, 3:20 PM
In T390140#11039766, @tchin wrote:

assignments is now stringified in KafkaSSE, but in the logs I see that assignments is in normalized.dropped.no_such_field. Is there something I'm missing? @colewhite

When this task was filed, eventstreams used the legacy logstash format. Now that eventstreams is writing ECS, the field is reaped because the field does not exist in the schema.

Within ECS, we can either rely on event.original, amend the schema, or use another field.

Ottomata added a comment.Jul 29 2025, 4:20 PM

another field: https://doc.wikimedia.org/ecs/#field-labels ?

label.assignments: <json assignments string> ?

CodeReviewBot added a project: Patch-For-Review.Aug 5 2025, 3:17 PM

tchin opened https://gitlab.wikimedia.org/repos/data-engineering/kafkasse/-/merge_requests/5

Nest assignment log under label

tchin added a comment.Aug 5 2025, 3:19 PM
In T390140#11043685, @colewhite wrote:
In T390140#11039766, @tchin wrote:

assignments is now stringified in KafkaSSE, but in the logs I see that assignments is in normalized.dropped.no_such_field. Is there something I'm missing? @colewhite

When this task was filed, eventstreams used the legacy logstash format. Now that eventstreams is writing ECS, the field is reaped because the field does not exist in the schema.

Within ECS, we can either rely on event.original, amend the schema, or use another field.

Was this a recent change? In the ecs docs it says:

If your events have additional data that cannot be mapped to ECS, you can simply add them to your events, using custom field names.

colewhite added a comment.EditedAug 5 2025, 3:36 PM
In T390140#11061304, @tchin wrote:

Was this a recent change? In the ecs docs it says:

If your events have additional data that cannot be mapped to ECS, you can simply add them to your events, using custom field names.

Wow, good catch! That may be true for upstream, but it is not true for our usage. c.f. wikitech for more info.

I've updated the docs to reflect this. Thanks!

CodeReviewBot added a comment.Aug 6 2025, 1:42 AM

tchin merged https://gitlab.wikimedia.org/repos/data-engineering/kafkasse/-/merge_requests/5

Nest assignment log under label

Maintenance_bot removed a project: Patch-For-Review.Aug 6 2025, 2:30 AM
CodeReviewBot added a project: Patch-For-Review.Aug 11 2025, 2:11 AM

tchin opened https://gitlab.wikimedia.org/repos/data-engineering/eventstreams/-/merge_requests/25

Bump kafka-sse and conform logs to ecs

CodeReviewBot added a comment.Aug 12 2025, 1:52 PM

tchin merged https://gitlab.wikimedia.org/repos/data-engineering/eventstreams/-/merge_requests/25

Bump kafka-sse and conform logs to ecs

Maintenance_bot removed a project: Patch-For-Review.Aug 12 2025, 2:31 PM
gerritbot added a comment.Aug 13 2025, 1:50 PM

Change #1178550 had a related patch set uploaded (by TChin; author: TChin):

[operations/deployment-charts@master] [eventstreams] Bump version 0.18.0

https://gerrit.wikimedia.org/r/1178550

gerritbot added a project: Patch-For-Review.Aug 13 2025, 1:50 PM
gerritbot added a comment.Aug 18 2025, 2:13 PM

Change #1178550 merged by jenkins-bot:

[operations/deployment-charts@master] [eventstreams] Bump version 0.18.0

https://gerrit.wikimedia.org/r/1178550

Maintenance_bot removed a project: Patch-For-Review.Aug 18 2025, 2:30 PM
mforns moved this task from Ready to Deploy to Done on the Data-Engineering (Q1 FY25/26 July 1st - September 30th) board.Aug 25 2025, 3:39 PM
Ahoelzl closed this task as Resolved.Sep 24 2025, 9:37 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits