Page MenuHomePhabricator
Feed Advanced Search

Wed, Dec 4

AndyRussG added a comment to T237605: Create kerberos principals for users.

Hi! Here's my request for the new creds for stat100* and notebook100*, please. Username: andyrussg. Thanks so much for working on this!!!!! :)

Wed, Dec 4, 5:02 PM · Analytics-Kanban, Analytics
AndyRussG changed the visibility for F31453782: T236834_log_differences_20191105.html.
Wed, Dec 4, 6:17 AM
AndyRussG changed the visibility for F31453782: T236834_log_differences_20191105.html.
Wed, Dec 4, 6:16 AM

Mon, Dec 2

AndyRussG added a comment to T239570: Investigate options for dropped CN EventLogging events for new pipeline.

I think it's blocking on the URL path /beacon/event. See https://easylist.to/easylist/easyprivacy.txt and T220627#5638168.

Mon, Dec 2, 6:05 PM · Fundraising-Backlog
AndyRussG moved T198752: Queries and maybe scripts to verify equivalence of data in new-Kafka-pipeline-testing and pgehres production databases from Backlog to Review on the Fundraising Sprint X-rays board.
Mon, Dec 2, 7:50 AM · Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint Visual Basic Instinct, Fundraising Sprint Usual Subscripts, Fundraising Sprint Trojan Horse Wisperer, Fundraising Sprint Sysadmin Kane, Fundraising Sprint Rocky Horror Presentation Layer, Fundraising Sprint Quick and the Deadlocked, Fundraising Sprint Men In Slack, Fundraising Sprint Land before Timeouts, Fundraising Sprint Bert and Ernie's Excellent Adventure, Fundraising Sprint A series of unfortunate event handlers, Fundraising Sprint XML ate my homework, Fundraising Sprint Window dressing is mostly olive oil, Fundraising Sprint Vestigial tails shoot from the hip, Fundraising Sprint USB stands for underhanded socket bureaucracy, Fundraising Sprint They Live, Fundraising Sprint Sasquatches can't find us either, Fundraising Sprint Raw data can give you salmonella, Patch-For-Review, Fundraising Sprint Queue is pronounced GJif, Fundraising Sprint Pluto is still a planet, Fundraising Sprint Owls, Fundraising-Backlog
AndyRussG added a subtask for T183978: [Epic] Fundraising kafkatee changes: T239570: Investigate options for dropped CN EventLogging events for new pipeline.
Mon, Dec 2, 7:44 AM · Epic, Fundraising Sprint Cottage Cheese isn't Made of Cottages, Fundraising Sprint Bermuda Rhombus (where things disappear then reappear), Fundraising Sprint Asymmetrical Earth Theory, Fundraising-Backlog
AndyRussG added a parent task for T239570: Investigate options for dropped CN EventLogging events for new pipeline: T183978: [Epic] Fundraising kafkatee changes.
Mon, Dec 2, 7:44 AM · Fundraising-Backlog
AndyRussG updated the task description for T239570: Investigate options for dropped CN EventLogging events for new pipeline.
Mon, Dec 2, 7:44 AM · Fundraising-Backlog
AndyRussG created T239570: Investigate options for dropped CN EventLogging events for new pipeline.
Mon, Dec 2, 7:44 AM · Fundraising-Backlog
AndyRussG added a comment to T220627: QuickSurveys EventLogging missing ~10% of interactions.

Just to note, we have the same problem for the new CentralNotice data pipeline, which uses EventLogging, as compared to the old pipeline, which uses a custom call to beacon/impression not blocked by AdBlock. In case it's useful: see T236834#5696044 (and the two comments after that).

Mon, Dec 2, 7:33 AM · MW-1.35-notes (1.35.0-wmf.3; 2019-10-22), Patch-For-Review, Readers-Web-Backlog (Kanbanana-2019-20-Q2), Analytics, Analytics-EventLogging, QuickSurveys
AndyRussG added a comment to T236835: FRUEC: Debug minor discrepancy in landing page data between old and new pipelines.

I'm not sure if this is at all pertinent, but we spent some time trying to debug a situation where we were missing about 10% of EventLogging data from people taking a survey served via the QuickSurveys tool. In essence, for 10% of readers, the tool was displaying the surveys correctly and we knew they had taken the survey but we never get the EventLogging that we should have. We ultimately decided that it was a mixture of adblock (which has settings that allow the javascript etc. to show the survey but blocks EventLogging) and, in our case, people could right-click off the page to take the survey and that wasn't triggered EventLogging as expected.
T220627#5641946

Mon, Dec 2, 7:27 AM · Fundraising Sprint YAMLton, the Musical, Fundraising Sprint X-rays, Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint Visual Basic Instinct, Fundraising-Backlog
AndyRussG updated the task description for T239564: Monitor and investigate possible event dropping by Kafkatee.
Mon, Dec 2, 7:00 AM · fundraising-tech-ops, Fundraising-Backlog
AndyRussG moved T236834: FRUEC: Detailed comparison of events in old and new log files for banner impression pipeline from Doing to Review on the Fundraising Sprint X-rays board.
Mon, Dec 2, 6:58 AM · Fundraising Sprint YAMLton, the Musical, Fundraising Sprint X-rays, Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint Visual Basic Instinct, Fundraising-Backlog
AndyRussG updated subscribers of T236834: FRUEC: Detailed comparison of events in old and new log files for banner impression pipeline.

Windows for log samples:

Mon, Dec 2, 6:57 AM · Fundraising Sprint YAMLton, the Musical, Fundraising Sprint X-rays, Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint Visual Basic Instinct, Fundraising-Backlog
AndyRussG created T239564: Monitor and investigate possible event dropping by Kafkatee.
Mon, Dec 2, 4:58 AM · fundraising-tech-ops, Fundraising-Backlog

Wed, Nov 27

AndyRussG moved T196563: Write a specification for mapping banner/landing page impression event properties -> database schema from Backlog to Pending Deployment on the Fundraising Sprint X-rays board.
Wed, Nov 27, 5:35 PM · Fundraising Sprint YAMLton, the Musical, Fundraising Sprint X-rays, Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint Visual Basic Instinct, Fundraising Sprint Usual Subscripts, Fundraising Sprint Trojan Horse Wisperer, Fundraising Sprint Sysadmin Kane, Fundraising Sprint Rocky Horror Presentation Layer, Fundraising Sprint Quick and the Deadlocked, Fundraising Sprint Princess Mongodb, Fundraising Sprint Office  , Fundraising Sprint Never Ending Query, Fundraising Sprint Men In Slack, Fundraising Sprint Land before Timeouts, Fundraising Sprint Vestigial tails shoot from the hip, Fundraising Sprint USB stands for underhanded socket bureaucracy, Fundraising Sprint They Live, Fundraising Sprint Sasquatches can't find us either, Fundraising Sprint Raw data can give you salmonella, Fundraising Sprint Queue is pronounced GJif, Fundraising Sprint Pluto is still a planet, Fundraising Sprint Owls, Fundraising Sprint Naming Sprints Is Not Important, Fundraising Sprint Matt Damon to head up Space Force, Fundraising Sprint Lactose is unusually tolerant, Fundraising-Backlog
AndyRussG moved T235284: FRUEC: Debug large discrepancy in data in initial test run. from Backlog to Deployed on the Fundraising Sprint X-rays board.
Wed, Nov 27, 5:34 PM · Fundraising Sprint X-rays, Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint Visual Basic Instinct, Fundraising Sprint Usual Subscripts, Fundraising Sprint Trojan Horse Wisperer, Fundraising-Backlog
AndyRussG moved T236835: FRUEC: Debug minor discrepancy in landing page data between old and new pipelines from Backlog to Review on the Fundraising Sprint X-rays board.
Wed, Nov 27, 5:34 PM · Fundraising Sprint YAMLton, the Musical, Fundraising Sprint X-rays, Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint Visual Basic Instinct, Fundraising-Backlog
AndyRussG moved T236834: FRUEC: Detailed comparison of events in old and new log files for banner impression pipeline from Backlog to Doing on the Fundraising Sprint X-rays board.
Wed, Nov 27, 5:34 PM · Fundraising Sprint YAMLton, the Musical, Fundraising Sprint X-rays, Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint Visual Basic Instinct, Fundraising-Backlog
AndyRussG moved T237736: FRUEC: Raise error when timestamp not found in filename from Backlog to Review on the Fundraising Sprint X-rays board.
Wed, Nov 27, 5:34 PM · Fundraising Sprint YAMLton, the Musical, Fundraising Sprint X-rays, Fundraising Sprint A Wrinkle in Timezones, Patch-For-Review, Fundraising Sprint Visual Basic Instinct, Fundraising-Backlog
AndyRussG added a comment to T236834: FRUEC: Detailed comparison of events in old and new log files for banner impression pipeline.

Differences found in orphaned old pipeline events:

  • 28% orphaned GET requests vs. 10% overall GET requests
  • 63% orphaned Windows requests vs. 39% overall Windows requests
Wed, Nov 27, 5:08 PM · Fundraising Sprint YAMLton, the Musical, Fundraising Sprint X-rays, Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint Visual Basic Instinct, Fundraising-Backlog
AndyRussG added a comment to T236834: FRUEC: Detailed comparison of events in old and new log files for banner impression pipeline.

Just did one-to-one merges using web request logs in Hive, in both directions, using fairly large samples in both cases.

Wed, Nov 27, 5:47 AM · Fundraising Sprint YAMLton, the Musical, Fundraising Sprint X-rays, Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint Visual Basic Instinct, Fundraising-Backlog
AndyRussG created P9757 T236834_merge_new_old_logs_1.hql.
Wed, Nov 27, 5:40 AM
AndyRussG created P9756 T236834_merge_old_new_logs_1.hql.
Wed, Nov 27, 5:32 AM

Mon, Nov 25

AndyRussG added a comment to T238560: Doubts and questions about Kerberos and Hadoop.

@elukey @Nuria Thanks so much!!!!!!!!!!!!!!!!!!!!!!!!

Mon, Nov 25, 8:26 PM · Analytics
AndyRussG added a comment to T236834: FRUEC: Detailed comparison of events in old and new log files for banner impression pipeline.

Here are some results:

Mon, Nov 25, 6:00 AM · Fundraising Sprint YAMLton, the Musical, Fundraising Sprint X-rays, Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint Visual Basic Instinct, Fundraising-Backlog

Thu, Nov 21

AndyRussG added a comment to T238560: Doubts and questions about Kerberos and Hadoop.

could you give some examples of issues you expect to see and troubleshoot (maybe some tickets from the past?)?

Thu, Nov 21, 8:07 PM · Analytics

Wed, Nov 20

AndyRussG renamed T236834: FRUEC: Detailed comparison of events in old and new log files for banner impression pipeline from FRUEC: Debug minor discrepancy in banner impression data between old and new pipelines to FRUEC: Detailed comparison of events in old and new log files for banner impression pipeline.
Wed, Nov 20, 3:58 PM · Fundraising Sprint YAMLton, the Musical, Fundraising Sprint X-rays, Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint Visual Basic Instinct, Fundraising-Backlog
AndyRussG added a comment to T235447: 2019 English campaign fundraising in apps.

Hi all! Congrats to all for your work on this...

Wed, Nov 20, 3:21 PM · Wikipedia-Android-App-Backlog, Wikipedia-iOS-App-Backlog, Android-app-feature-Feeds, iOS-app-feature-Feed

Tue, Nov 19

AndyRussG moved T236834: FRUEC: Detailed comparison of events in old and new log files for banner impression pipeline from Backlog to Doing on the Fundraising Sprint A Wrinkle in Timezones board.
Tue, Nov 19, 7:55 PM · Fundraising Sprint YAMLton, the Musical, Fundraising Sprint X-rays, Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint Visual Basic Instinct, Fundraising-Backlog
AndyRussG moved T235461: FRUEC: For CentralNotice impression counts, take into account client-side sample rate from Doing to Backlog on the Fundraising Sprint A Wrinkle in Timezones board.
Tue, Nov 19, 7:55 PM · Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint Visual Basic Instinct, Fundraising Sprint Usual Subscripts, Fundraising-Backlog

Mon, Nov 18

AndyRussG created T238594: FRUEC: Adapt comparison scripts to focus on specific campaigns or other event properties with expected specific values.
Mon, Nov 18, 8:55 PM · Fundraising-Backlog
AndyRussG added a subtask for T183978: [Epic] Fundraising kafkatee changes: T238592: Ask other teams for input on extra entries in new pipeline landing page logs .
Mon, Nov 18, 8:45 PM · Epic, Fundraising Sprint Cottage Cheese isn't Made of Cottages, Fundraising Sprint Bermuda Rhombus (where things disappear then reappear), Fundraising Sprint Asymmetrical Earth Theory, Fundraising-Backlog
AndyRussG added a parent task for T238592: Ask other teams for input on extra entries in new pipeline landing page logs : T183978: [Epic] Fundraising kafkatee changes.
Mon, Nov 18, 8:45 PM · Fundraising-Backlog
AndyRussG created T238592: Ask other teams for input on extra entries in new pipeline landing page logs .
Mon, Nov 18, 8:44 PM · Fundraising-Backlog
AndyRussG moved T235461: FRUEC: For CentralNotice impression counts, take into account client-side sample rate from Backlog to Doing on the Fundraising Sprint A Wrinkle in Timezones board.
Mon, Nov 18, 11:30 AM · Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint Visual Basic Instinct, Fundraising Sprint Usual Subscripts, Fundraising-Backlog
AndyRussG moved T236835: FRUEC: Debug minor discrepancy in landing page data between old and new pipelines from Doing to Review on the Fundraising Sprint A Wrinkle in Timezones board.
Mon, Nov 18, 11:30 AM · Fundraising Sprint YAMLton, the Musical, Fundraising Sprint X-rays, Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint Visual Basic Instinct, Fundraising-Backlog
AndyRussG changed the point value for T236835: FRUEC: Debug minor discrepancy in landing page data between old and new pipelines from 2 to 4.
Mon, Nov 18, 11:30 AM · Fundraising Sprint YAMLton, the Musical, Fundraising Sprint X-rays, Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint Visual Basic Instinct, Fundraising-Backlog
AndyRussG added a comment to T236835: FRUEC: Debug minor discrepancy in landing page data between old and new pipelines.

Windows for log samples:

Mon, Nov 18, 11:29 AM · Fundraising Sprint YAMLton, the Musical, Fundraising Sprint X-rays, Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint Visual Basic Instinct, Fundraising-Backlog

Thu, Nov 14

AndyRussG moved T235461: FRUEC: For CentralNotice impression counts, take into account client-side sample rate from Deployed to Backlog on the Fundraising Sprint A Wrinkle in Timezones board.
Thu, Nov 14, 5:51 PM · Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint Visual Basic Instinct, Fundraising Sprint Usual Subscripts, Fundraising-Backlog
AndyRussG moved T235284: FRUEC: Debug large discrepancy in data in initial test run. from Backlog to Deployed on the Fundraising Sprint A Wrinkle in Timezones board.
Thu, Nov 14, 5:51 PM · Fundraising Sprint X-rays, Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint Visual Basic Instinct, Fundraising Sprint Usual Subscripts, Fundraising Sprint Trojan Horse Wisperer, Fundraising-Backlog
AndyRussG moved T236835: FRUEC: Debug minor discrepancy in landing page data between old and new pipelines from Backlog to Doing on the Fundraising Sprint A Wrinkle in Timezones board.
Thu, Nov 14, 5:51 PM · Fundraising Sprint YAMLton, the Musical, Fundraising Sprint X-rays, Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint Visual Basic Instinct, Fundraising-Backlog

Tue, Nov 12

AndyRussG closed T236627: CentralNotice: Adapt impression event schema for campaign fallback, a subtask of T183978: [Epic] Fundraising kafkatee changes, as Resolved.
Tue, Nov 12, 9:12 PM · Epic, Fundraising Sprint Cottage Cheese isn't Made of Cottages, Fundraising Sprint Bermuda Rhombus (where things disappear then reappear), Fundraising Sprint Asymmetrical Earth Theory, Fundraising-Backlog
AndyRussG closed T236627: CentralNotice: Adapt impression event schema for campaign fallback as Resolved.
Tue, Nov 12, 9:12 PM · Fundraising Sprint Visual Basic Instinct, MediaWiki-extensions-CentralNotice, Fundraising-Backlog
AndyRussG added a comment to T236835: FRUEC: Debug minor discrepancy in landing page data between old and new pipelines.

Here's a summary of the situation regarding discrepancies between the new and old pipelines for Landing Pages (includes also measures obtained in T235284).

Tue, Nov 12, 5:34 AM · Fundraising Sprint YAMLton, the Musical, Fundraising Sprint X-rays, Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint Visual Basic Instinct, Fundraising-Backlog

Mon, Nov 11

AndyRussG renamed T238023: FREUC: Check that empty or missing uselang or language URL parameter should default to 'en' language for Landing Pages from FREUC: Check that empty uselang or missing URL parameter should default to 'en' language for Landing Pages to FREUC: Check that empty or missing uselang or language URL parameter should default to 'en' language for Landing Pages.
Mon, Nov 11, 11:25 PM · Fundraising-Backlog
AndyRussG renamed T238023: FREUC: Check that empty or missing uselang or language URL parameter should default to 'en' language for Landing Pages from FREUC: Check that empty uselang URL parameter should default to 'en' language for Landing Pages to FREUC: Check that empty uselang or missing URL parameter should default to 'en' language for Landing Pages.
Mon, Nov 11, 8:56 PM · Fundraising-Backlog
AndyRussG renamed T237997: FRUEC: For legacy compatibility, empty language property should default to 'en' for LandingPage impressions from FRUEC: Empty language property should default to 'en' for LandingPage impressions to FRUEC: For legacy compatibility, empty language property should default to 'en' for LandingPage impressions.
Mon, Nov 11, 8:53 PM · Fundraising-Backlog
AndyRussG added a comment to T237997: FRUEC: For legacy compatibility, empty language property should default to 'en' for LandingPage impressions.

See also T237997: FRUEC: For legacy compatibility, empty language property should default to 'en' for LandingPage impressions.

Mon, Nov 11, 8:51 PM · Fundraising-Backlog
AndyRussG added a comment to T237997: FRUEC: For legacy compatibility, empty language property should default to 'en' for LandingPage impressions.

See also T238023: FREUC: Check that empty or missing uselang or language URL parameter should default to 'en' language for Landing Pages.

Mon, Nov 11, 8:50 PM · Fundraising-Backlog
AndyRussG created T238023: FREUC: Check that empty or missing uselang or language URL parameter should default to 'en' language for Landing Pages.
Mon, Nov 11, 8:50 PM · Fundraising-Backlog
AndyRussG added a comment to T237997: FRUEC: For legacy compatibility, empty language property should default to 'en' for LandingPage impressions.

FRUEC currently accepts LandingPage events with no language property in the JSON input. However, if the property exists and its value is an empty string, the event is marked invalid and not counted. However, in such a case, the legacy system (DjangoBannerStats) defaults the language to 'en'.

Mon, Nov 11, 8:40 PM · Fundraising-Backlog
AndyRussG added a subtask for T183978: [Epic] Fundraising kafkatee changes: T237997: FRUEC: For legacy compatibility, empty language property should default to 'en' for LandingPage impressions.
Mon, Nov 11, 3:54 PM · Epic, Fundraising Sprint Cottage Cheese isn't Made of Cottages, Fundraising Sprint Bermuda Rhombus (where things disappear then reappear), Fundraising Sprint Asymmetrical Earth Theory, Fundraising-Backlog
AndyRussG added a parent task for T237997: FRUEC: For legacy compatibility, empty language property should default to 'en' for LandingPage impressions: T183978: [Epic] Fundraising kafkatee changes.
Mon, Nov 11, 3:54 PM · Fundraising-Backlog
AndyRussG created T237997: FRUEC: For legacy compatibility, empty language property should default to 'en' for LandingPage impressions.
Mon, Nov 11, 3:54 PM · Fundraising-Backlog

Nov 8 2019

AndyRussG moved T237736: FRUEC: Raise error when timestamp not found in filename from Backlog to Review on the Fundraising Sprint Visual Basic Instinct board.
Nov 8 2019, 3:48 PM · Fundraising Sprint YAMLton, the Musical, Fundraising Sprint X-rays, Fundraising Sprint A Wrinkle in Timezones, Patch-For-Review, Fundraising Sprint Visual Basic Instinct, Fundraising-Backlog
AndyRussG renamed T237736: FRUEC: Raise error when timestamp not found in filename from FRUEC: Raise error when no timestamp in filename to FRUEC: Raise error when timestamp not found in filename.
Nov 8 2019, 3:47 PM · Fundraising Sprint YAMLton, the Musical, Fundraising Sprint X-rays, Fundraising Sprint A Wrinkle in Timezones, Patch-For-Review, Fundraising Sprint Visual Basic Instinct, Fundraising-Backlog
AndyRussG set the point value for T237736: FRUEC: Raise error when timestamp not found in filename to 1.
Nov 8 2019, 3:47 PM · Fundraising Sprint YAMLton, the Musical, Fundraising Sprint X-rays, Fundraising Sprint A Wrinkle in Timezones, Patch-For-Review, Fundraising Sprint Visual Basic Instinct, Fundraising-Backlog
AndyRussG created T237736: FRUEC: Raise error when timestamp not found in filename.
Nov 8 2019, 3:47 PM · Fundraising Sprint YAMLton, the Musical, Fundraising Sprint X-rays, Fundraising Sprint A Wrinkle in Timezones, Patch-For-Review, Fundraising Sprint Visual Basic Instinct, Fundraising-Backlog

Nov 6 2019

AndyRussG added a subtask for T183978: [Epic] Fundraising kafkatee changes: T237553: FRUEC: Discuss with stakeholders, Analytics and fr-tech implications and options for new Landing Page pipeline.
Nov 6 2019, 5:48 PM · Epic, Fundraising Sprint Cottage Cheese isn't Made of Cottages, Fundraising Sprint Bermuda Rhombus (where things disappear then reappear), Fundraising Sprint Asymmetrical Earth Theory, Fundraising-Backlog
AndyRussG added a parent task for T237553: FRUEC: Discuss with stakeholders, Analytics and fr-tech implications and options for new Landing Page pipeline: T183978: [Epic] Fundraising kafkatee changes.
Nov 6 2019, 5:48 PM · Fundraising-Backlog
AndyRussG updated the task description for T237553: FRUEC: Discuss with stakeholders, Analytics and fr-tech implications and options for new Landing Page pipeline.
Nov 6 2019, 5:47 PM · Fundraising-Backlog
AndyRussG created T237553: FRUEC: Discuss with stakeholders, Analytics and fr-tech implications and options for new Landing Page pipeline.
Nov 6 2019, 5:31 PM · Fundraising-Backlog

Nov 4 2019

AndyRussG added a comment to T236835: FRUEC: Debug minor discrepancy in landing page data between old and new pipelines.

Matching on landing page, country, language, and other event fields, with old log timestamps always earlier than the new log timestamps, by at most 30 seconds, we get:

Nov 4 2019, 6:23 PM · Fundraising Sprint YAMLton, the Musical, Fundraising Sprint X-rays, Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint Visual Basic Instinct, Fundraising-Backlog
AndyRussG added a comment to T236835: FRUEC: Debug minor discrepancy in landing page data between old and new pipelines.

Trying different options for better matching... Just a small improvement by allowing new log timestamps that are closest to the new old log ones, that is, removing the requirement that the new log event be after the old log one. With this method, we get 136 unmatched events in new log, and 510 in the old one.

Nov 4 2019, 5:56 PM · Fundraising Sprint YAMLton, the Musical, Fundraising Sprint X-rays, Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint Visual Basic Instinct, Fundraising-Backlog
AndyRussG added a comment to T236835: FRUEC: Debug minor discrepancy in landing page data between old and new pipelines.

I've dug into this some more, improving filtering compared to what was done for T235284. I've got some more clarity about what the differences are, but it's still pretty ugly.

Nov 4 2019, 8:03 AM · Fundraising Sprint YAMLton, the Musical, Fundraising Sprint X-rays, Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint Visual Basic Instinct, Fundraising-Backlog
AndyRussG claimed T236835: FRUEC: Debug minor discrepancy in landing page data between old and new pipelines.
Nov 4 2019, 7:49 AM · Fundraising Sprint YAMLton, the Musical, Fundraising Sprint X-rays, Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint Visual Basic Instinct, Fundraising-Backlog

Nov 1 2019

AndyRussG moved T236627: CentralNotice: Adapt impression event schema for campaign fallback from Review to Deployed on the Fundraising Sprint Visual Basic Instinct board.
Nov 1 2019, 4:35 PM · Fundraising Sprint Visual Basic Instinct, MediaWiki-extensions-CentralNotice, Fundraising-Backlog
AndyRussG added a comment to T235845: Implement banner design for WMDEs autum new editor recruitment campaign.

Okay, great, thank you for the update! @GoranSMilovanovic @Tim_WMDE Does it work now?

Nov 1 2019, 3:30 PM · Analytics, WMDE-Analytics-Engineering, WMDE-FUN-Funban-2019, WMDE-FUN-Sprint-2019-10-14, WMDE-New-Editors-Banner-Campaigns (Banner Campaign Autumn 2019)

Oct 31 2019

AndyRussG added a comment to T235845: Implement banner design for WMDEs autum new editor recruitment campaign.

Scheduled to deploy in a few minutes...

Oct 31 2019, 6:00 PM · Analytics, WMDE-Analytics-Engineering, WMDE-FUN-Funban-2019, WMDE-FUN-Sprint-2019-10-14, WMDE-New-Editors-Banner-Campaigns (Banner Campaign Autumn 2019)
AndyRussG added a comment to T235845: Implement banner design for WMDEs autum new editor recruitment campaign.

Great, thank you everyone! @GoranSMilovanovic Can you now get the impression data or do you need anything else?

Oct 31 2019, 4:13 PM · Analytics, WMDE-Analytics-Engineering, WMDE-FUN-Funban-2019, WMDE-FUN-Sprint-2019-10-14, WMDE-New-Editors-Banner-Campaigns (Banner Campaign Autumn 2019)
AndyRussG added a comment to T235845: Implement banner design for WMDEs autum new editor recruitment campaign.

Thanks so much for flagging this, and many apologies for the noise!! There's now a patch in review for T236627: CentralNotice: Adapt impression event schema for campaign fallback.

Oct 31 2019, 3:54 PM · Analytics, WMDE-Analytics-Engineering, WMDE-FUN-Funban-2019, WMDE-FUN-Sprint-2019-10-14, WMDE-New-Editors-Banner-Campaigns (Banner Campaign Autumn 2019)
AndyRussG moved T236627: CentralNotice: Adapt impression event schema for campaign fallback from Backlog to Review on the Fundraising Sprint Visual Basic Instinct board.
Oct 31 2019, 3:51 PM · Fundraising Sprint Visual Basic Instinct, MediaWiki-extensions-CentralNotice, Fundraising-Backlog
AndyRussG added a project to T236627: CentralNotice: Adapt impression event schema for campaign fallback: Fundraising Sprint Visual Basic Instinct.
Oct 31 2019, 3:51 PM · Fundraising Sprint Visual Basic Instinct, MediaWiki-extensions-CentralNotice, Fundraising-Backlog
AndyRussG added a comment to T236627: CentralNotice: Adapt impression event schema for campaign fallback.

Here's a patch!!! Apologies for the trouble...!

Oct 31 2019, 3:51 PM · Fundraising Sprint Visual Basic Instinct, MediaWiki-extensions-CentralNotice, Fundraising-Backlog
AndyRussG moved T235284: FRUEC: Debug large discrepancy in data in initial test run. from Review to Pending Deployment on the Fundraising Sprint Visual Basic Instinct board.
Oct 31 2019, 1:36 AM · Fundraising Sprint X-rays, Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint Visual Basic Instinct, Fundraising Sprint Usual Subscripts, Fundraising Sprint Trojan Horse Wisperer, Fundraising-Backlog

Oct 30 2019

AndyRussG moved T196563: Write a specification for mapping banner/landing page impression event properties -> database schema from Backlog to Pending Deployment on the Fundraising Sprint Visual Basic Instinct board.
Oct 30 2019, 3:30 PM · Fundraising Sprint YAMLton, the Musical, Fundraising Sprint X-rays, Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint Visual Basic Instinct, Fundraising Sprint Usual Subscripts, Fundraising Sprint Trojan Horse Wisperer, Fundraising Sprint Sysadmin Kane, Fundraising Sprint Rocky Horror Presentation Layer, Fundraising Sprint Quick and the Deadlocked, Fundraising Sprint Princess Mongodb, Fundraising Sprint Office  , Fundraising Sprint Never Ending Query, Fundraising Sprint Men In Slack, Fundraising Sprint Land before Timeouts, Fundraising Sprint Vestigial tails shoot from the hip, Fundraising Sprint USB stands for underhanded socket bureaucracy, Fundraising Sprint They Live, Fundraising Sprint Sasquatches can't find us either, Fundraising Sprint Raw data can give you salmonella, Fundraising Sprint Queue is pronounced GJif, Fundraising Sprint Pluto is still a planet, Fundraising Sprint Owls, Fundraising Sprint Naming Sprints Is Not Important, Fundraising Sprint Matt Damon to head up Space Force, Fundraising Sprint Lactose is unusually tolerant, Fundraising-Backlog
AndyRussG moved T236835: FRUEC: Debug minor discrepancy in landing page data between old and new pipelines from Backlog to Doing on the Fundraising Sprint Visual Basic Instinct board.
Oct 30 2019, 3:29 PM · Fundraising Sprint YAMLton, the Musical, Fundraising Sprint X-rays, Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint Visual Basic Instinct, Fundraising-Backlog

Oct 29 2019

AndyRussG moved T235284: FRUEC: Debug large discrepancy in data in initial test run. from Backlog to Review on the Fundraising Sprint Visual Basic Instinct board.
Oct 29 2019, 9:17 PM · Fundraising Sprint X-rays, Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint Visual Basic Instinct, Fundraising Sprint Usual Subscripts, Fundraising Sprint Trojan Horse Wisperer, Fundraising-Backlog
AndyRussG added a comment to T234248: More small fixes needed for Campaign fallback.

Note: the second bullet point from the task description has been spun out as T236845.

Oct 29 2019, 8:55 PM · Fundraising Sprint Usual Subscripts, Fundraising Sprint Trojan Horse Wisperer, MediaWiki-extensions-CentralNotice, Fundraising-Backlog
AndyRussG renamed T236845: Still one more small fix needed for Campaign fallback: local campaign variable. from Still one more small fixe needed for Campaign fallback: local campaign variable. to Still one more small fix needed for Campaign fallback: local campaign variable..
Oct 29 2019, 8:54 PM · Fundraising-Backlog
AndyRussG created T236845: Still one more small fix needed for Campaign fallback: local campaign variable..
Oct 29 2019, 8:53 PM · Fundraising-Backlog
AndyRussG moved T235284: FRUEC: Debug large discrepancy in data in initial test run. from Doing to Review on the Fundraising Sprint Usual Subscripts board.
Oct 29 2019, 8:43 PM · Fundraising Sprint X-rays, Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint Visual Basic Instinct, Fundraising Sprint Usual Subscripts, Fundraising Sprint Trojan Horse Wisperer, Fundraising-Backlog
AndyRussG added a comment to T235284: FRUEC: Debug large discrepancy in data in initial test run..

This task was about the large-scale discrepancy, which I think we can consider to be solved. There are still smaller unexplained differences between data we're getting from the old and new pipelines. I've created new tasks to investigate that: T236835 and T236834.

Oct 29 2019, 7:28 PM · Fundraising Sprint X-rays, Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint Visual Basic Instinct, Fundraising Sprint Usual Subscripts, Fundraising Sprint Trojan Horse Wisperer, Fundraising-Backlog
AndyRussG created T236835: FRUEC: Debug minor discrepancy in landing page data between old and new pipelines.
Oct 29 2019, 7:26 PM · Fundraising Sprint YAMLton, the Musical, Fundraising Sprint X-rays, Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint Visual Basic Instinct, Fundraising-Backlog
AndyRussG created T236834: FRUEC: Detailed comparison of events in old and new log files for banner impression pipeline.
Oct 29 2019, 7:25 PM · Fundraising Sprint YAMLton, the Musical, Fundraising Sprint X-rays, Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint Visual Basic Instinct, Fundraising-Backlog
AndyRussG moved T234728: Andy to finish estimating old tasks from Backlog to Deployed on the Fundraising Sprint Usual Subscripts board.
Oct 29 2019, 7:21 PM · Fundraising Sprint Usual Subscripts, Fundraising-Backlog
AndyRussG set the point value for T213915: wmf_deploy updated but not merged and deployed into .12 to 1.
Oct 29 2019, 7:20 PM · Fundraising Sprint Bert and Ernie's Excellent Adventure, MW-1.33-notes (1.33.0-wmf.12; 2019-01-08), Fundraising Sprint A series of unfortunate event handlers, Fundraising-Backlog, MediaWiki-extensions-CentralNotice
AndyRussG set the point value for T203925: Save times for changes to translation variable text in centralnotice paralysingly slow to 4.
Oct 29 2019, 7:16 PM · Core Platform Team Workboards (Done with CPT), Performance-Team-publish, Fundraising Sprint Vestigial tails shoot from the hip, Fundraising Sprint USB stands for underhanded socket bureaucracy, Fundraising Sprint They Live, Core Platform Team (Needs Cleaning - Security, stability, performance, and scalability (TEC1)), Performance-Team, Fundraising Sprint Sasquatches can't find us either, Language-Team, Fundraising Sprint Raw data can give you salmonella, MediaWiki-extensions-Translate, Fundraising-Backlog, MediaWiki-extensions-CentralNotice
AndyRussG set the point value for T195276: CentralNotice: Some URL params break EventLogging impression schema to 1.
Oct 29 2019, 7:14 PM · Fundraising Sprint Karma chameleons hide amongst us, Fundraising Sprint Junebugs prefer July, MediaWiki-extensions-CentralNotice, Fundraising-Backlog
AndyRussG set the point value for T185816: CentralNotice: Prevent autofill of multiple selection fields by browsers to 2.
Oct 29 2019, 7:13 PM · Fundraising Sprint Dinosaur Cookies co-existed with Gingerbread People, Fundraising Sprint Cottage Cheese isn't Made of Cottages, Fundraising Sprint Bermuda Rhombus (where things disappear then reappear), MediaWiki-extensions-CentralNotice, Fundraising-Backlog

Oct 28 2019

AndyRussG created T236734: Adapt Druid storage of CentralNotice data following campaign fallback.
Oct 28 2019, 7:46 PM · Fundraising-Backlog
AndyRussG added a comment to T235284: FRUEC: Debug large discrepancy in data in initial test run..

I've dug deeper into the Landing Page discrepancy, comparing sequences of log entries from the same IPs in both old and new pipelines.

Oct 28 2019, 8:49 AM · Fundraising Sprint X-rays, Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint Visual Basic Instinct, Fundraising Sprint Usual Subscripts, Fundraising Sprint Trojan Horse Wisperer, Fundraising-Backlog

Oct 27 2019

AndyRussG added a subtask for T183978: [Epic] Fundraising kafkatee changes: T236627: CentralNotice: Adapt impression event schema for campaign fallback.
Oct 27 2019, 6:08 PM · Epic, Fundraising Sprint Cottage Cheese isn't Made of Cottages, Fundraising Sprint Bermuda Rhombus (where things disappear then reappear), Fundraising Sprint Asymmetrical Earth Theory, Fundraising-Backlog
AndyRussG added a parent task for T236627: CentralNotice: Adapt impression event schema for campaign fallback: T183978: [Epic] Fundraising kafkatee changes.
Oct 27 2019, 6:08 PM · Fundraising Sprint Visual Basic Instinct, MediaWiki-extensions-CentralNotice, Fundraising-Backlog
AndyRussG created T236627: CentralNotice: Adapt impression event schema for campaign fallback.
Oct 27 2019, 6:07 PM · Fundraising Sprint Visual Basic Instinct, MediaWiki-extensions-CentralNotice, Fundraising-Backlog

Oct 25 2019

AndyRussG added a comment to T235284: FRUEC: Debug large discrepancy in data in initial test run..

So actually it seems the problem is not duplicate entries in the old logs, but rather a few IP addresses hammering on the site using some sort of script, which doesn't run client-side code, so it doesn't generate any client-side events.

Oct 25 2019, 3:06 PM · Fundraising Sprint X-rays, Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint Visual Basic Instinct, Fundraising Sprint Usual Subscripts, Fundraising Sprint Trojan Horse Wisperer, Fundraising-Backlog
AndyRussG added a comment to T235284: FRUEC: Debug large discrepancy in data in initial test run..

Made some progress on figuring out what the difference is between old and new logs. It looks like there are a lot of duplicate entries in the old log files.

Oct 25 2019, 6:00 AM · Fundraising Sprint X-rays, Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint Visual Basic Instinct, Fundraising Sprint Usual Subscripts, Fundraising Sprint Trojan Horse Wisperer, Fundraising-Backlog

Oct 23 2019

AndyRussG set the point value for T235284: FRUEC: Debug large discrepancy in data in initial test run. to 4.
Oct 23 2019, 4:45 PM · Fundraising Sprint X-rays, Fundraising Sprint A Wrinkle in Timezones, Fundraising Sprint Visual Basic Instinct, Fundraising Sprint Usual Subscripts, Fundraising Sprint Trojan Horse Wisperer, Fundraising-Backlog
AndyRussG created T236285: CentralNotice: Adapt banner history for campaign fallback.
Oct 23 2019, 3:29 PM · MediaWiki-extensions-CentralNotice, Fundraising-Backlog