Maniphest T344272

Use batching for daily data import
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Tchanders
	Aug 15 2023, 4:56 PM

Tags

Referenced Files

None

Subscribers

Description

Following T341122: Implement daily data update routine, the daily data import process is as follows:

Run diff.sh, which creates a .sql file for the database updates that need to be performed
Run update-db.js to perform these database updates

This task is for ensuring that step 2 happens in batches. This is necessary because several millions of rows will be updated, and we need to pace this process so as not to choke up DB resources.

Details

	Title	Reference	Author	Source Branch	Dest Branch
	Implement batched import of db updates	repos/mediawiki/services/ipoid!45	stran	batch-import	main

Customize query in GitLab

Related Objects
Search...

		Status	Subtype	Assigned	Task
		In Progress		Niharika	T324492 Temporary accounts - MVP
		Open		None	T340895 [Epic] IP Info accommodations for temporary accounts
		Open		STran	T341395 Display Spur data on IPInfo infobox
		Resolved		kostajh	T339284 Deploy ipoid
		Resolved		kostajh	T340984 Fix problems with data import
		Resolved		STran	T344749 Refactor import-db and update-db into one script
		Resolved		STran	T341122 Implement daily data update routine
		Resolved		STran	T344272 Use batching for daily data import

Event Timeline

Tchanders created this task.Aug 15 2023, 4:56 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 15 2023, 4:56 PM

@STran Please feel free to improve the task description!

Tchanders assigned this task to STran.Aug 22 2023, 5:01 PM

Tchanders mentioned this in T341660: Investigate non-deterministic behaviour of import.Aug 22 2023, 5:13 PM

Tchanders moved this task from Ready 🎬 (ONLY IF YOU HAVE NO MORE CODE TO REVIEW) to In Progress 💪 on the Anti-Harassment (AHaT Sprint 32 - Baseball Cap) board.Aug 23 2023, 9:34 AM

stran opened https://gitlab.wikimedia.org/repos/mediawiki/services/ipoid/-/merge_requests/45

Implement batched import of db updates

Tchanders added a parent task: T341122: Implement daily data update routine.Aug 23 2023, 1:54 PM

Tchanders mentioned this in T344499: Move behaviors/proxies/tunnels processing into an independent script.Aug 23 2023, 3:47 PM

Tchanders moved this task from In Progress 💪 to Code Review 🔍 on the Anti-Harassment (AHaT Sprint 32 - Baseball Cap) board.Aug 24 2023, 1:18 PM

Tchanders mentioned this in T344921: Ensure dependent SQL statements are all run in the same batch.Aug 24 2023, 1:29 PM

tchanders merged https://gitlab.wikimedia.org/repos/mediawiki/services/ipoid/-/merge_requests/45

Implement batched import of db updates

Maintenance_bot removed a project: Patch-For-Review.Aug 24 2023, 3:10 PM

I'm proposing to bypass QA for now, so that update can be tested in one go, once these small fixes are ironed out. Does that sound OK @STran ?

kostajh moved this task from Backlog to Done on the iPoid-Service board.Aug 29 2023, 12:38 PM

Tchanders closed this task as Resolved.Sep 6 2023, 5:01 PM