Page MenuHomePhabricator
Create Task
Maniphest T344499

Move behaviors/proxies/tunnels processing into an independent script
Closed, ResolvedPublic
Actions

Description

From https://gitlab.wikimedia.org/repos/mediawiki/services/ipoid/-/merge_requests/20#note_45767:

The script should be able to be run independently and generate a sql file that can be run.

Details

TitleReferenceAuthorSource BranchDest Branch
Remove updates to behaviors, proxies and tunnels from output-sql.jsrepos/mediawiki/services/ipoid!63tchandersremove-insert-propertiesmain
Add a script for popuplating behaviors, proxies and tunnels tablesrepos/mediawiki/services/ipoid!48tchandersimport-propertiesmain
Customize query in GitLab

Related Objects
Search...

Event Timeline

STran created this task.Aug 18 2023, 11:10 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 18 2023, 11:10 AM
Tchanders added a parent task: T341122: Implement daily data update routine.Aug 23 2023, 2:10 PM
Tchanders edited projects, added Anti-Harassment (AHaT Sprint 32 - Baseball Cap); removed Anti-Harassment.
Tchanders subscribed.EditedAug 23 2023, 3:47 PM

I think this is worth doing for a couple of reasons:

  • Checking whether to insert known values for up to 25 million actors every day is a lot of unnecessary work (see analysis below)
  • Inserting behaviors/proxies/tunnels just before inserting rows in the map tables that use them creates dependencies which make batching more complicated (see T344272)

Profiling

I did some profiling of output-diff.js and found that inserting the behaviors/proxies/tunnels during the update takes up a large proportion of the running time, and uses a large proportion of memory.

Here's where we do it: https://gitlab.wikimedia.org/repos/mediawiki/services/ipoid/-/blob/main/output-diff.js#L261

We gradually build arrays of behaviors, proxies and tunnels, potentially appending data from the latest actor to each, then ensuring there are no duplicates. We spend a lot of time in the isUnique function:

image.png (1×3 px, 201 KB)

And of course those arrays and the objects whose properties they are references from use up more memory:

Without arraysWith arrays
image.png (671×1 px, 182 KB)
image.png (686×1 px, 202 KB)

Time to run script on full dataset with the arrays: 13m38.731s
Time to run script on full dataset without the arrays: 5m28.828s

Tchanders mentioned this in T344921: Ensure dependent SQL statements are all run in the same batch.Aug 24 2023, 1:42 PM
Tchanders mentioned this in T344939: Log when the lookup fails on attempting to INSERT to actor_data (because the proxy/behaviour doesn't exist).Aug 24 2023, 4:59 PM
Tchanders updated the task description. (Show Details)
Tchanders updated the task description. (Show Details)Aug 24 2023, 5:05 PM

The script should be able to be run independently and generate a sql file that can be run.

The dataset is so small that we can just store it as a JSON

Tchanders mentioned this in T344942: [S] Ensure that tunnel rows are updated with the correct data during actor data import and update.Aug 24 2023, 5:10 PM
CodeReviewBot added a project: Patch-For-Review.Aug 24 2023, 5:14 PM

tchanders opened https://gitlab.wikimedia.org/repos/mediawiki/services/ipoid/-/merge_requests/48

Add a script for popuplating behaviors, proxies and tunnels tables

Tchanders claimed this task.Aug 24 2023, 5:14 PM
Tchanders moved this task from Ready 🎬 (ONLY IF YOU HAVE NO MORE CODE TO REVIEW) to Code Review 🔍 on the Anti-Harassment (AHaT Sprint 32 - Baseball Cap) board.
CodeReviewBot added a comment.Aug 28 2023, 8:49 AM

stran merged https://gitlab.wikimedia.org/repos/mediawiki/services/ipoid/-/merge_requests/48

Add a script for popuplating behaviors, proxies and tunnels tables

Maintenance_bot removed a project: Patch-For-Review.Aug 28 2023, 9:10 AM
Tchanders moved this task from Code Review 🔍 to Done Q1 2023-2024 on the Anti-Harassment (AHaT Sprint 32 - Baseball Cap) board.Aug 28 2023, 9:38 AM
Tchanders moved this task from Done Q1 2023-2024 to In Progress 💪 on the Anti-Harassment (AHaT Sprint 32 - Baseball Cap) board.Sep 6 2023, 5:04 PM

We also need to make sure the other scripts don't try to populate these tables.

CodeReviewBot added a project: Patch-For-Review.Sep 7 2023, 5:39 PM

tchanders opened https://gitlab.wikimedia.org/repos/mediawiki/services/ipoid/-/merge_requests/63

Add 'import' mode to the script that outputs the .sql file

Tchanders moved this task from In Progress 💪 to Code Review 🔍 on the Anti-Harassment (AHaT Sprint 32 - Baseball Cap) board.Sep 7 2023, 5:40 PM
CodeReviewBot added a comment.Sep 7 2023, 5:50 PM

tchanders updated https://gitlab.wikimedia.org/repos/mediawiki/services/ipoid/-/merge_requests/63

Remove updates to behaviors, proxies and tunnels from output-sql.js

Maintenance_bot removed a project: Patch-For-Review.Sep 7 2023, 6:11 PM
CodeReviewBot added a comment.Sep 11 2023, 10:30 AM

stran merged https://gitlab.wikimedia.org/repos/mediawiki/services/ipoid/-/merge_requests/63

Remove updates to behaviors, proxies and tunnels from output-sql.js

Tchanders closed this task as Resolved.Sep 11 2023, 4:21 PM
Tchanders moved this task from Code Review 🔍 to Done Q1 2023-2024 on the Anti-Harassment (AHaT Sprint 32 - Baseball Cap) board.

Will be tested as part of T341122

STran moved this task from Backlog to Done on the iPoid-Service board.Oct 4 2023, 6:27 AM
Tchanders closed subtask T344939: Log when the lookup fails on attempting to INSERT to actor_data (because the proxy/behaviour doesn't exist) as Declined.Feb 7 2024, 5:24 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL