Page MenuHomePhabricator

mwdumper should use bulk load optimizations
Open, Needs TriagePublic

Description

We can improve MySQL load times by using SET autocommit = 0 and disabling key checks. It might be worth benchmarking LOAD DATA for our use case. I thought I remembered a mysql-fast-import utility, but nothing immediately jumps out on the Internet.

PostgreSQL can bulk load using the COPY command, reading from a CSV file or CSV-formatted rows inlined in the SQL file.

It would be awesome if we could make our CSVs compatible between both mysql and pgsql, and bundle static SQL scripts for each backend, but I'm not sure off-hand whether that's possible.

Event Timeline

awight created this task.Mar 28 2019, 12:17 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 28 2019, 12:17 AM