Change Details

Roundtripping through python pandas seems to require too much memory to be done all at once. Either 1. change script to reduce memory between tables (one- table at a time). 2. Use mysql's bulk insert LOAD DATA INFILE function but generate additional (needs proof of concept) 3. a way to skip items that have validation issues like born in the year 18260723 ``` sqlalchemy.exc.DataError: (MySQLdb._exceptions.DataError) (1264, "Out of range value for column 'year_of_birth' at row 1") [SQL: INSERT INTO human (fill_id, qid, year_of_birth, gender, sitelink_count) VALUES (%s, %s, %s, %s, %s)] [parameters: ( (8, 55226613.0, 18260723.0, 6581097.0, 0.0))] ```

Roundtripping through python pandas seems to require too much memory to be done all at once. Either 1. change script to reduce memory between tables (one- table at a time). 2. Use mysql's bulk insert LOAD DATA INFILE function but generate additional (needs proof of concept) 3. a way to skip items that have validation issues like born in the year 18260723 ``` sqlalchemy.exc.DataError: (MySQLdb._exceptions.DataError) (1264, "Out of range value for column 'year_of_birth' at row 1") [SQL: INSERT INTO human (fill_id, qid, year_of_birth, gender, sitelink_count) VALUES (%s, %s, %s, %s, %s)] [parameters: ( (8, 55226613.0, 18260723.0, 6581097.0, 0.0))] ``` [ ] label only needs to be inserted once, or insert ignore

Roundtripping through python pandas seems to require too much memory to be done all at once. Either 1. change script to reduce memory between tables (one- table at a time). 2. Use mysql's bulk insert LOAD DATA INFILE function but generate additional (needs proof of concept) 3. a way to skip items that have validation issues like born in the year 18260723 ``` sqlalchemy.exc.DataError: (MySQLdb._exceptions.DataError) (1264, "Out of range value for column 'year_of_birth' at row 1") [SQL: INSERT INTO human (fill_id, qid, year_of_birth, gender, sitelink_count) VALUES (%s, %s, %s, %s, %s)] [parameters: ( (8, 55226613.0, 18260723.0, 6581097.0, 0.0))] ``` [ ] label only needs to be inserted once, or insert ignore