lbowmaker renamed T360969: [SPIKE] Evaluate and document solutions for table-management tooling from [SPIKE] Investigate OpenHouse as a data lake management tool to [SPIKE] Evaluate and document solutions for table-management tooling.

Mon, Apr 15, 2:44 PM · Data-Engineering (Q4 2024 April 1st - June 30th), Spike

Tue, Apr 9

lbowmaker assigned T361242: Unique devices tables have missing or incorrect data for January and February 2024 to JAllemandou.

Tue, Apr 9, 1:53 PM · Data-Engineering, Movement-Insights, Patch-For-Review, Data-Platform

lbowmaker moved T361242: Unique devices tables have missing or incorrect data for January and February 2024 from Next Up to In progress on the Data-Engineering (Q4 2024 April 1st - June 30th) board.

Tue, Apr 9, 1:52 PM · Data-Engineering, Movement-Insights, Patch-For-Review, Data-Platform

Mon, Apr 8

lbowmaker moved T361962: Update Sample Rates for Metrics Platform Events from Incoming (new tickets) to Radar (External Teams) on the Data-Engineering board.

Mon, Apr 8, 12:21 PM · FY2023-24-WE 2.1 Typography and palette customizations, Web-Team-Backlog (FY2023-24 Q4 Sprint 2), Data Products, Patch-For-Review, Data-Engineering

lbowmaker added a project to T361962: Update Sample Rates for Metrics Platform Events: Data Products.

Mon, Apr 8, 12:20 PM · FY2023-24-WE 2.1 Typography and palette customizations, Web-Team-Backlog (FY2023-24 Q4 Sprint 2), Data Products, Patch-For-Review, Data-Engineering

Fri, Apr 5

lbowmaker added a comment to T357468: [Dataset Config Store] Setup initial CI checks.

I think if we organized files by dataset we discussed at one point requiring things like config for deleting data after a certain time (for example).

Fri, Apr 5, 4:46 PM · Data-Engineering (Q4 2024 April 1st - June 30th)

Thu, Apr 4

lbowmaker added a comment to T346046: [Search Update Pipeline] Source streams for private wikis.

@Ahoelzl - what implementation is the 13 point estimate based on?

Thu, Apr 4, 1:15 AM · Discovery-Search (Current work), Data-Engineering, CirrusSearch

Wed, Apr 3

lbowmaker updated the task description for T305688: Make HTML Dumps available in hadoop.

Wed, Apr 3, 12:51 PM · Data-Engineering (Q4 2024 April 1st - June 30th), Structured-Data-Backlog

lbowmaker moved T361214: Public dashboard process from Incoming (new tickets) to Radar (External Teams) on the Data-Engineering board.

Wed, Apr 3, 12:16 PM · Epic, Data Products (Epics Timeline), Data-Engineering-Dashiki, Data-Engineering

lbowmaker added projects to T361684: Create per-wiki user preference metrics: Data Products, Metrics Platform Backlog.

Wed, Apr 3, 12:16 PM · Product-Analytics, Data Products

lbowmaker moved T361684: Create per-wiki user preference metrics from Incoming (new tickets) to Radar (External Teams) on the Data-Engineering board.

Wed, Apr 3, 12:16 PM · Product-Analytics, Data Products

Mon, Apr 1

lbowmaker created Data-Engineering (Q1 2024 July 1st - September 30th).

Mon, Apr 1, 8:51 PM

lbowmaker moved T357434: [Dataset Config Store] Deploy poc to dse-k8s from Next Up to In progress on the Data-Engineering (Q4 2024 April 1st - June 30th) board.

Mon, Apr 1, 2:44 PM · Data-Engineering (Q4 2024 April 1st - June 30th), Patch-For-Review

lbowmaker moved T360924: Replace service runner with a simplified library to better support metrics and debugging from Next Up to In progress on the Data-Engineering (Q4 2024 April 1st - June 30th) board.

Mon, Apr 1, 2:13 PM · Data-Engineering (Q4 2024 April 1st - June 30th)

lbowmaker moved T347690: [Iceberg Migration] Migrate pageview tables to Iceberg from Q4 2024 April 1st - June 30th to Estimated (To be planned) on the Data-Engineering board.

Mon, Apr 1, 12:42 PM · Data-Engineering

lbowmaker moved T347690: [Iceberg Migration] Migrate pageview tables to Iceberg from Estimated (To be planned) to Q4 2024 April 1st - June 30th on the Data-Engineering board.

Mon, Apr 1, 12:31 PM · Data-Engineering

lbowmaker moved T344730: Migrate Data Engineering Pipelinelib repos to GitLab from Estimated (To be planned) to Q4 2024 April 1st - June 30th on the Data-Engineering board.

Mon, Apr 1, 12:31 PM · Data-Engineering (Q4 2024 April 1st - June 30th), GitLab (Pipeline Services Migration🐤), Event-Platform

lbowmaker moved T361017: [SPIKE] Can we express Event Platform configs in config store? from To be estimated/discussed to Q4 2024 April 1st - June 30th on the Data-Engineering board.

Mon, Apr 1, 12:23 PM · Data-Engineering (Q4 2024 April 1st - June 30th), Spike, Event-Platform

lbowmaker set the point value for T361017: [SPIKE] Can we express Event Platform configs in config store? to 5.

Mon, Apr 1, 12:22 PM · Data-Engineering (Q4 2024 April 1st - June 30th), Spike, Event-Platform

lbowmaker edited projects for T306896: Spike: Integrate Spark with DataHub, added: Data-Engineering (Q4 2024 April 1st - June 30th); removed Data-Engineering.

Mon, Apr 1, 12:21 PM · Data-Engineering (Q4 2024 April 1st - June 30th), Data-Catalog, Data Pipelines

lbowmaker moved T335528: [datahub] Implement automatic deletion of datasets with deleted data sources from To be estimated/discussed to Estimated (To be planned) on the Data-Engineering board.

Mon, Apr 1, 12:21 PM · Data-Engineering, Data Pipelines

lbowmaker set the point value for T335528: [datahub] Implement automatic deletion of datasets with deleted data sources to 8.

Mon, Apr 1, 12:20 PM · Data-Engineering, Data Pipelines

lbowmaker closed T359435: [Airflow] SparkSqlOperator fails when executing via Skein with master=local as Invalid.

Based on @JAllemandou comment I will close. @mforns let us know if you want us to investigate further.

Mon, Apr 1, 12:19 PM · Data-Engineering

lbowmaker moved T360210: Document instructions for deleting an event stream and its usages from To be estimated/discussed to Q4 2024 April 1st - June 30th on the Data-Engineering board.

Mon, Apr 1, 12:18 PM · Data-Engineering (Q4 2024 April 1st - June 30th), Metrics Platform Backlog, Event-Platform

lbowmaker set the point value for T360210: Document instructions for deleting an event stream and its usages to 1.

Mon, Apr 1, 12:18 PM · Data-Engineering (Q4 2024 April 1st - June 30th), Metrics Platform Backlog, Event-Platform

lbowmaker moved T360794: Implement stream of HTML content on mw.page_change event from To be estimated/discussed to Q4 2024 April 1st - June 30th on the Data-Engineering board.

Mon, Apr 1, 12:17 PM · Data-Engineering (Q4 2024 April 1st - June 30th), Event-Platform

lbowmaker edited projects for T360794: Implement stream of HTML content on mw.page_change event, added: Data-Engineering (Q4 2024 April 1st - June 30th); removed Data-Engineering.

Mon, Apr 1, 12:16 PM · Data-Engineering (Q4 2024 April 1st - June 30th), Event-Platform

lbowmaker edited projects for T360896: [SPIKE] [Dataset Config Store] - Design how config store feeds DataHub, added: Data-Engineering (Q4 2024 April 1st - June 30th); removed Data-Engineering.

Mon, Apr 1, 12:15 PM · Data-Engineering (Q4 2024 April 1st - June 30th), Spike

lbowmaker edited projects for T360922: [Status Store] [SPIKE] Investigate and document approach for Iceberg Sensors, added: Data-Engineering (Q4 2024 April 1st - June 30th); removed Data-Engineering.

Mon, Apr 1, 12:15 PM · Data-Engineering (Q4 2024 April 1st - June 30th), Spike

lbowmaker edited projects for T360967: [Developer Experience] Implement CI hql Linting, added: Data-Engineering (Q4 2024 April 1st - June 30th); removed Data-Engineering.

Mon, Apr 1, 12:14 PM · Data-Engineering (Q4 2024 April 1st - June 30th)

lbowmaker edited projects for T360968: [Developer Experience] [SPIKE] Investigate process to automate deployment of folders and artifacts to HDFS, added: Data-Engineering (Q4 2024 April 1st - June 30th); removed Data-Engineering.

Mon, Apr 1, 12:14 PM · Data-Engineering (Q4 2024 April 1st - June 30th), Spike

lbowmaker edited projects for T360969: [SPIKE] Evaluate and document solutions for table-management tooling, added: Data-Engineering (Q4 2024 April 1st - June 30th); removed Data-Engineering.

Mon, Apr 1, 12:13 PM · Data-Engineering (Q4 2024 April 1st - June 30th), Spike

lbowmaker edited projects for T361014: [Data Quality] Migrate the anomaly detection job to DeeQu checks, added: Data-Engineering (Q4 2024 April 1st - June 30th); removed Data-Engineering.

Mon, Apr 1, 12:13 PM · Data-Engineering (Q4 2024 April 1st - June 30th)

lbowmaker moved T361016: [Data Quality] Migrate MWHistoryChecker to DeeQu checks from To be estimated/discussed to Q4 2024 April 1st - June 30th on the Data-Engineering board.

Mon, Apr 1, 12:12 PM · Data-Engineering (Q4 2024 April 1st - June 30th)

lbowmaker set the point value for T361016: [Data Quality] Migrate MWHistoryChecker to DeeQu checks to 8.

Mon, Apr 1, 12:11 PM · Data-Engineering (Q4 2024 April 1st - June 30th)

lbowmaker edited projects for T338792: [Event Platform] Implement PoC Event-Driven Data Pipeline for Revert Risk Model Scores using Event Platform Capabilities, added: Data-Engineering (Q4 2024 April 1st - June 30th); removed Data-Engineering (Sprint 9).

Mon, Apr 1, 12:09 PM · Data-Engineering, Epic, Wikimedia Enterprise, Machine-Learning-Team, Event-Platform

lbowmaker set the point value for T360924: Replace service runner with a simplified library to better support metrics and debugging to 13.

Mon, Apr 1, 12:09 PM · Data-Engineering (Q4 2024 April 1st - June 30th)

lbowmaker edited projects for T353940: We should provide DQ integration with Python, added: Data-Engineering (Q4 2024 April 1st - June 30th); removed Data-Engineering (Sprint 9).

Mon, Apr 1, 12:09 PM · Data-Engineering (Q4 2024 April 1st - June 30th)

lbowmaker archived Data-Engineering (Sprint 9).

Mon, Apr 1, 12:09 PM

lbowmaker edited projects for T354692: [Data Quality] Implement basic data quality metrics for MW history, added: Data-Engineering (Q4 2024 April 1st - June 30th); removed Data-Engineering (Sprint 9).

Mon, Apr 1, 12:09 PM · Data-Engineering (Q4 2024 April 1st - June 30th)

lbowmaker edited projects for T266641: [Data Platform] Test Alluxio as cache layer for Presto, added: Data-Engineering (Q4 2024 April 1st - June 30th); removed Data-Engineering (Sprint 9).

Mon, Apr 1, 12:08 PM · Data-Engineering (Q4 2024 April 1st - June 30th), Data-Platform-SRE, Patch-For-Review

lbowmaker edited projects for T350180: Upgrade prom-client in NodeJS service-runner and enable collectDefaultMetrics, added: Data-Engineering (Q4 2024 April 1st - June 30th); removed Data-Engineering (Sprint 9).

Mon, Apr 1, 12:08 PM · Data-Engineering, observability, ChangeProp, Event-Platform, service-runner

lbowmaker edited projects for T356363: [Refine Refactoring] Refactor refinery code for compatibility with Airflow integration, added: Data-Engineering (Q4 2024 April 1st - June 30th); removed Data-Engineering (Sprint 9).

Mon, Apr 1, 12:08 PM · Data-Engineering (Q4 2024 April 1st - June 30th)

lbowmaker edited projects for T345914: [Event Platform] Define Flink k8s operator SLO, added: Data-Engineering (Q4 2024 April 1st - June 30th); removed Data-Engineering (Sprint 9).

Mon, Apr 1, 12:08 PM · Data-Engineering, Data-Platform-SRE, Event-Platform

lbowmaker edited projects for T357005: eventstreams regularly uses more than 95% of its memory limit, added: Data-Engineering (Q4 2024 April 1st - June 30th); removed Data-Engineering (Sprint 9).

Mon, Apr 1, 12:08 PM · Data-Engineering, Event-Platform, EventStreams, serviceops, Prod-Kubernetes, Kubernetes

lbowmaker edited projects for T338796: Rewrite all Airflow sensors that use datacenter prepartitions to depend on both datacenters, added: Data-Engineering (Q4 2024 April 1st - June 30th); removed Data-Engineering (Sprint 9).

Mon, Apr 1, 12:07 PM · Data-Engineering (Q4 2024 April 1st - June 30th), Data Products (Data Products Sprint 05), serviceops-radar

lbowmaker edited projects for T347561: [Maintenance] Set up deletion jobs for Structured Data's data pipelines, added: Data-Engineering (Q4 2024 April 1st - June 30th); removed Data-Engineering (Sprint 9).

Mon, Apr 1, 12:07 PM · Data-Engineering (Q4 2024 April 1st - June 30th), Data Products, Structured-Data-Backlog

lbowmaker edited projects for T347430: [Data Platform] Install a Prometheus connector for Presto, pointed at thanos-query, added: Data-Engineering (Q4 2024 April 1st - June 30th); removed Data-Engineering (Sprint 9).

Mon, Apr 1, 12:07 PM · Data-Engineering (Q4 2024 April 1st - June 30th), Data-Platform-SRE, SRE Observability

lbowmaker edited projects for T354552: [Maintenance] Migrate ReportUpdater browser queries to Airflow, added: Data-Engineering (Q4 2024 April 1st - June 30th); removed Data-Engineering (Sprint 9).

Mon, Apr 1, 12:06 PM · Data-Engineering (Q4 2024 April 1st - June 30th)

lbowmaker edited projects for T357372: [Maintenance] Migrate pingback to Airflow, added: Data-Engineering (Q4 2024 April 1st - June 30th); removed Data-Engineering (Sprint 9).

Mon, Apr 1, 12:06 PM · Data-Engineering (Q4 2024 April 1st - June 30th)

lbowmaker edited projects for T356362: [Refine Refactoring] [Spike] Define a concept and provide a PoC for dynamic DAG execution in Airflow, added: Data-Engineering (Q4 2024 April 1st - June 30th); removed Data-Engineering (Sprint 9).

Mon, Apr 1, 12:06 PM · Data-Engineering (Q4 2024 April 1st - June 30th)

lbowmaker edited projects for T357430: Airflow mapped tasks UI & metrics, added: Data-Engineering (Q4 2024 April 1st - June 30th); removed Data-Engineering (Sprint 9).

Mon, Apr 1, 12:05 PM · Data-Engineering (Q4 2024 April 1st - June 30th)

lbowmaker edited projects for T354694: [Maintenance] Safeguard VarnishKafka to HAProxy analytics transition, added: Data-Engineering (Q4 2024 April 1st - June 30th); removed Data-Engineering (Sprint 9).

Mon, Apr 1, 12:05 PM · Data-Engineering (Q4 2024 April 1st - June 30th)

lbowmaker moved T357434: [Dataset Config Store] Deploy poc to dse-k8s from Sprint 9 to Q4 2024 April 1st - June 30th on the Data-Engineering board.

Mon, Apr 1, 12:05 PM · Data-Engineering (Q4 2024 April 1st - June 30th), Patch-For-Review

lbowmaker closed T357472: Add movement insights group/users to MWH denormalize job alerts as Resolved.

Mon, Apr 1, 12:05 PM · Data-Engineering, Data-Platform

lbowmaker closed T357472: Add movement insights group/users to MWH denormalize job alerts, a subtask of T357462: Enable notifications for completion of Hive table snapshots, as Resolved.

Mon, Apr 1, 12:05 PM · Movement-Insights, Data-Engineering, Data-Platform

lbowmaker closed T341229: ProduceCanaryEvents job should be scheduled by Airflow and/or a k8s service as Resolved.

Mon, Apr 1, 12:05 PM · Data-Engineering (Sprint 9), Event-Platform

lbowmaker closed T356424: [Maintenance] Migrate cx ReportUpdater job as Resolved.

Mon, Apr 1, 12:05 PM · Data-Engineering (Sprint 9)

lbowmaker closed T351093: [Data Quality] Define concept for Alerting in coordination with SRE as Resolved.

Mon, Apr 1, 12:05 PM · Data-Engineering (Sprint 9)

lbowmaker closed T356360: [Refine Refactoring] Orchestrate Airflow execution of navigationtiming from config store as Resolved.

Mon, Apr 1, 12:05 PM · Data-Engineering (Sprint 9)

lbowmaker closed T359051: eventstreams: change default num_workers to 0 as Resolved.

Mon, Apr 1, 12:05 PM · Data-Engineering (Sprint 9)

lbowmaker closed T356424: [Maintenance] Migrate cx ReportUpdater job, a subtask of T307540: [Airflow Migration] Migrate 1+ reportupdater jobs, as Resolved.

Mon, Apr 1, 12:05 PM · Data-Engineering, Data Pipelines

lbowmaker moved T356866: [Data Quality] Update data_quality schemas to be compatible with Iceberg tables from Sprint 9 to Q4 2024 April 1st - June 30th on the Data-Engineering board.

Mon, Apr 1, 12:05 PM · Data-Engineering (Q4 2024 April 1st - June 30th)

lbowmaker moved T358210: Delete reportupdater jobs data/puppet-code from Sprint 9 to Q4 2024 April 1st - June 30th on the Data-Engineering board.

Mon, Apr 1, 12:04 PM · Data-Engineering (Q4 2024 April 1st - June 30th)

lbowmaker moved T360924: Replace service runner with a simplified library to better support metrics and debugging from Sprint 9 to Q4 2024 April 1st - June 30th on the Data-Engineering board.

Mon, Apr 1, 12:04 PM · Data-Engineering (Q4 2024 April 1st - June 30th)

lbowmaker closed T333011: [Maintenance] Define Migration/Deprecation Plan for Hue, a subtask of T341895: Deprecate Hue and stop the services, as Resolved.

Mon, Apr 1, 12:03 PM · Patch-For-Review, Data-Platform-SRE (2024.03.25 - 2024.04.14), Data-Engineering

lbowmaker closed T333011: [Maintenance] Define Migration/Deprecation Plan for Hue as Resolved.

Mon, Apr 1, 12:03 PM · Data-Engineering, Data Pipelines (Sprint 14)

lbowmaker set the point value for T358210: Delete reportupdater jobs data/puppet-code to 1.

Mon, Apr 1, 12:03 PM · Data-Engineering (Q4 2024 April 1st - June 30th)

lbowmaker moved T357472: Add movement insights group/users to MWH denormalize job alerts from Next Up to Done on the Data-Engineering (Sprint 9) board.

Mon, Apr 1, 12:02 PM · Data-Engineering, Data-Platform

lbowmaker moved T357938: [Maintenance] Migrate wmcs to Airflow from Sprint 9 to Q4 2024 April 1st - June 30th on the Data-Engineering board.

Mon, Apr 1, 12:02 PM · Data-Engineering (Q4 2024 April 1st - June 30th)

Thu, Mar 28

lbowmaker moved T361210: FYI: Changes to the cuc_agent column in the cu_changes table from Incoming (new tickets) to Radar (External Teams) on the Data-Engineering board.

Thu, Mar 28, 1:01 PM · Data-Engineering, CheckUser

Tue, Mar 26

lbowmaker moved T361024: NEW BUG REPORT SSL certificate verification error when using internal API endpoints from conda-analytics and Jupyter on stat host from Backlog to SRE on the Data-Platform board.

Tue, Mar 26, 4:00 PM · Data-Platform-SRE, Data-Platform

lbowmaker added a comment to T341895: Deprecate Hue and stop the services.

In T341895#9529415, @lbowmaker wrote:

The new version of Superset allows nested data types to be visualized and we have spoken with users about using Hive cli to update their datasets instead of using Hue. Conversation in slack

I think we can turn off Hue now when SRE has availability.

Tue, Mar 26, 3:54 PM · Patch-For-Review, Data-Platform-SRE (2024.03.25 - 2024.04.14), Data-Engineering

lbowmaker moved T361016: [Data Quality] Migrate MWHistoryChecker to DeeQu checks from Incoming (new tickets) to To be estimated/discussed on the Data-Engineering board.

Tue, Mar 26, 1:53 PM · Data-Engineering (Q4 2024 April 1st - June 30th)

lbowmaker created T361016: [Data Quality] Migrate MWHistoryChecker to DeeQu checks.

Tue, Mar 26, 1:53 PM · Data-Engineering (Q4 2024 April 1st - June 30th)

lbowmaker renamed T361014: [Data Quality] Migrate the anomaly detection job to DeeQu checks from Migrate the anomaly detection job to DeeQu checks to [Data Quality] Migrate the anomaly detection job to DeeQu checks.

Tue, Mar 26, 1:50 PM · Data-Engineering (Q4 2024 April 1st - June 30th)

lbowmaker moved T361014: [Data Quality] Migrate the anomaly detection job to DeeQu checks from Incoming (new tickets) to To be estimated/discussed on the Data-Engineering board.

Tue, Mar 26, 1:46 PM · Data-Engineering (Q4 2024 April 1st - June 30th)

lbowmaker created T361014: [Data Quality] Migrate the anomaly detection job to DeeQu checks.

Tue, Mar 26, 1:45 PM · Data-Engineering (Q4 2024 April 1st - June 30th)

lbowmaker created T360969: [SPIKE] Evaluate and document solutions for table-management tooling.

Tue, Mar 26, 12:28 AM · Data-Engineering (Q4 2024 April 1st - June 30th), Spike

lbowmaker created T360968: [Developer Experience] [SPIKE] Investigate process to automate deployment of folders and artifacts to HDFS.

Tue, Mar 26, 12:23 AM · Data-Engineering (Q4 2024 April 1st - June 30th), Spike

lbowmaker created T360967: [Developer Experience] Implement CI hql Linting.

Tue, Mar 26, 12:20 AM · Data-Engineering (Q4 2024 April 1st - June 30th)

lbowmaker moved T335528: [datahub] Implement automatic deletion of datasets with deleted data sources from DataHub to To be estimated/discussed on the Data-Engineering board.