Improve Event Platform and MediaWiki Event Enrichment wikitech documentation
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	lbowmaker
	Feb 14 2023, 2:28 PM

Description

User Story

As an event platform engineer, I need to document how to deploy a Flink job to k8s, where and how various eventplatform enrichment components live, and how to use Event Platform Value Stream work within the last year.

Why?

So other engineers can easily follow steps to deploy a new or modified job
So other engineers can more easily work with the components we've built.

Done is:

MediaWiki Event Enrichment page is published on wikitech and reviewed by team: https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment
Event Utilities documentation is updated on wikitech and reviewed by team - https://wikitech.wikimedia.org/wiki/Event_Platform/Event_Utilities
flink-app kubernetes deployment is documented on wikitech - https://wikitech.wikimedia.org/wiki/Event_Platform/Stream_Processing/Flink
Flink Event Catalog docs are moved to wikitech and reviewed by team - https://wikitech.wikimedia.org/wiki/Event_Platform/Stream_Processing/Flink_Catalog
Event Platform decision log is moved to Wikitech - https://wikitech.wikimedia.org/wiki/Event_Platform/Decision_Log
Document how to rename streams in stream config - https://wikitech.wikimedia.org/wiki/Event_Platform/Stream_Configuration#Stream_Configuration_deployment

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		lbowmaker	T329628 Event Platform Value Stream Documentation Tasks
		Resolved		Ottomata	T329629 Improve Event Platform and MediaWiki Event Enrichment wikitech documentation

Event Timeline

lbowmaker created this task.Feb 14 2023, 2:28 PM

lbowmaker moved this task from Backlog to To be Estimated/To be discussed on the Event-Platform board.

lbowmaker moved this task from To be Estimated/To be discussed to Sprint 09 on the Event-Platform board.Feb 16 2023, 2:12 PM

lbowmaker edited projects, added Event-Platform (Sprint 09); removed Event-Platform.

lbowmaker edited projects, added Data-Engineering-Planning; removed Data-Engineering.Feb 17 2023, 6:23 PM

lbowmaker moved this task from Backlog to Event Platform on the Data-Engineering-Planning board.

lbowmaker edited projects, added Event-Platform; removed Event-Platform (Sprint 09).

lbowmaker moved this task from Backlog to Sprint 09 on the Event-Platform board.Mar 6 2023, 2:12 PM

lbowmaker edited projects, added Event-Platform (Sprint 09); removed Event-Platform.

JArguello-WMF edited projects, added Event-Platform (Sprint 10); removed Event-Platform (Sprint 09).Mar 13 2023, 2:17 PM

JArguello-WMF edited projects, added Event-Platform (Sprint 11); removed Event-Platform (Sprint 10).Mar 31 2023, 2:38 PM

JArguello-WMF edited projects, added Event-Platform; removed Event-Platform (Sprint 11).Apr 3 2023, 3:17 PM

lbowmaker updated the task description. (Show Details)May 10 2023, 1:28 PM

lbowmaker moved this task from Backlog to Sprint 14 A on the Event-Platform board.

lbowmaker edited projects, added Event-Platform (Sprint 14 A); removed Event-Platform.

There is some deployment documentation (dse-k8s) at https://www.mediawiki.org/wiki/Platform_Engineering_Team/Event_Platform_Value_Stream/Pyflink_Enrichment_Service_Deployment.

This should be updated to reflect charts changes, and the fact we are moving to wikikube.

As for other docs to move: probably all links at
https://www.mediawiki.org/wiki/Platform_Engineering_Team/Event_Platform_Value_Stream (expect maybe for the RFCs on Use Cases?).

We also need to finalize and publish the design doc for eventutilities-python (https://phabricator.wikimedia.org/T336488).

Ottomata moved this task from Next Up to In progress on the Event-Platform (Sprint 14 A) board.May 24 2023, 7:21 PM

I think we don't want to move most of that stuff, the value stream and POC stuff can stay there. We mostly want organized user manual docs about the things we are deploying.

Started https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment with your docs from https://www.mediawiki.org/wiki/Platform_Engineering_Team/Event_Platform_Value_Stream/Pyflink_Enrichment_Service_Deployment, and left a redirect there.

I updated the Event Utilities doc on wikitech.

@tchin for catalog stuff, should we just move https://www.mediawiki.org/wiki/Platform_Engineering_Team/Event_Platform_Value_Stream/Event_Catalog as is, or are there any major changes we should make? And/or should we wait for T333795: Event Catalog: Standardize Options Handling ?

JArguello-WMF assigned this task to Ottomata.May 25 2023, 1:14 PM

Ottomata updated the task description. (Show Details)May 25 2023, 5:41 PM

Today I moved [[ https://wikitech.wikimedia.org/wiki/Event_Platform/Stream_Processing/Use_cases

Use Cases ]] and Stream Processing Evaluation pages to wikitech.

I also moved the Flink on Kubernetes page under Event Platform, and have begun updating it.

Updated https://wikitech.wikimedia.org/wiki/Event_Platform/Stream_Processing/Flink more today, reorged around and focused on flink-kubernetes-operator deployments.

JArguello-WMF edited projects, added Event-Platform (Sprint 14 B); removed Event-Platform (Sprint 14 A).Jun 5 2023, 11:43 AM

JArguello-WMF moved this task from Next Up to In progress on the Event-Platform (Sprint 14 B) board.

Added a Flaws page
Added a page describing the event hadoop ingestion lifecycle, as well as canary events.

Added docs about the mediawiki/state schema models, and removed docs about the now outdated mediawiki/**/common stuff.

Ottomata updated the task description. (Show Details)Jun 21 2023, 5:46 PM

Ottomata updated the task description. (Show Details)Jun 21 2023, 6:01 PM

Some https://wikitech.wikimedia.org/wiki/Template:Navigation_Event_Platform updates today.

Also added some Event Platform Links in https://wikitech.wikimedia.org/wiki/Template:Navigation_Wikimedia_infrastructure

Ottomata updated the task description. (Show Details)Jun 21 2023, 6:22 PM

FYI Am trying to collect various things I do for 'parental leave transition' docs at https://wikitech.wikimedia.org/wiki/User:Ottomata/Notes

Ottomata moved this task from In progress to In Review on the Event-Platform (Sprint 14 B) board.Jun 26 2023, 3:55 PM

Ottomata updated the task description. (Show Details)Jun 27 2023, 1:34 PM

Ottomata updated the task description. (Show Details)Jun 27 2023, 1:39 PM

Ottomata updated the task description. (Show Details)Jun 27 2023, 2:43 PM

Moved decision log to wikitech: https://wikitech.wikimedia.org/wiki/Event_Platform/Decision_Log
Added better docs for stream config deployment: https://wikitech.wikimedia.org/wiki/Event_Platform/Stream_Configuration#Stream_Configuration_deployment
Added docs on how to rename streams: https://wikitech.wikimedia.org/wiki/Event_Platform/Stream_Configuration#Renaming_streams

JArguello-WMF moved this task from In Review to Done on the Event-Platform (Sprint 14 B) board.Jun 28 2023, 1:02 PM

JArguello-WMF closed this task as Resolved.Jun 29 2023, 1:08 PM

Improve Event Platform and MediaWiki Event Enrichment wikitech documentationClosed, ResolvedPublicActions