Page MenuHomePhabricator

Fix eventutillites_python stream_manager error_sink configuration
Closed, ResolvedPublic

Description

In T328478: Refactor parameterization of eventutilities-python and mediawiki-event-enrichment, we added some really nice auto param and config parsing to eventutilities_python stream_manager.

This was done using jsonsargparse's add_class_arguments to allow for auto instantiation of our ConnectorDesctriptor dataclasses from configs. This works great, except for the fact that the error_sink param is a Union type, which causes jsonargparse not know how to auto instantiate it (the same problem happens if this param is Optional).

As of 2023-04, the 'auto error sink' handling added in T326536: Streaming services errors should be routed to an error event topic. works fine, as for that we either pass in a bool or a str. But, providing an error_sink dict config as you might for the sink param does not currently work.

Additionally, it would be nice if the Namespace config returned by the load_config function would have the instantiated error sink in it, whether or not if it was directly from config or via the auto error sink handling. This would make testing error output easier, as you could know the resolved file uri that a file error sink will write to. As is, if an auto error sink is used, it won't be in the config; it will be constructed by stream_manager later.

Done is:

  • error_sink can be configured from config just like sink
  • auto error sink handling still works
  • The Namespace that load_config returns should have e.g. config.stream_manager.error_sink as an instiated SinkDescriptor for both cases of configured and auto error sink.

Or, if ^ is not possible / too difficult:

  • error_sink can be configured from config just like sink
  • The default for error_sink will still cause no error side output & sink to be used.

Details

TitleReferenceAuthorSource BranchDest Branch
Remove support for auto error sinkrepos/data-engineering/eventutilities-python!72ottoT335591_error_sinkmain
Customize query in GitLab

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Ottomata edited projects, added Event-Platform (Sprint 14 B); removed Event-Platform.
Ottomata updated the task description. (Show Details)
Ottomata moved this task from Next Up to In Review on the Event-Platform (Sprint 14 B) board.