Page MenuHomePhabricator
Create Task
Maniphest T340984

Fix problems with data import
Closed, ResolvedPublic
Actions

Description

Various errors have been found with the data import script, which we should fix.

See subtasks for more details.

Related Objects
Search...

StatusSubtypeAssignedTask
 In ProgressNiharikaT324492 Temporary accounts - MVP
 OpenNoneT340895 [Epic] IP Info accommodations for temporary accounts
 OpenSTranT341395 Display Spur data on IPInfo infobox
 ResolvedkostajhT339284 Deploy ipoid
 ResolvedkostajhT340984 Fix problems with data import
 ResolvedBUG REPORT• AGueyteT340178 "Column 'anonymous' cannot be null"
 DeclinedBUG REPORT• TThoabalaT340176 "Data too long for column 'behavior' at row 1"
 DeclinedBUG REPORTNoneT339837 "Data too long for column 'anonymous' at row 1"
 ResolvedBUG REPORT• TThoabalaT339359 'Parameter at position 2 is undefined\n' if tunnels.type is not set
 ResolvedBUG REPORT• AGueyteT339345 "Duplicate entry '0' for key 'behavior'"
 ResolvedBUG REPORT• TThoabalaT339343 "Column 'type' cannot be null"
 DeclinedBUG REPORT• TThoabalaT341605 Ipoid: ER_DATA_TOO_LONG
 DeclinedBUG REPORT• AGueyteT339325 "Duplicate entry '0' for key 'proxy'"
 InvalidBUG REPORTNoneT339324 "Data too long for column 'type' at row 1"
 InvalidNoneT345107 Update Tunnels Type schema VARBINARY value
 ResolvedBUG REPORT• TThoabalaT339073 Density imported incorrectly
 ResolvedSTranT344749 Refactor import-db and update-db into one script
 ResolvedSTranT341122 Implement daily data update routine
 ResolvedTchandersT344517 Fix statements generated for updated IP data
 ResolvedTchandersT344520 Remove temporary files to avoid taking up unnecessary disk space
 ResolvedSTranT344272 Use batching for daily data import
 ResolvedSTranT325630 Implement call to data vendor
 ResolvedTchandersT344499 Move behaviors/proxies/tunnels processing into an independent script
 DeclinedNoneT344939 Log when the lookup fails on attempting to INSERT to actor_data (because the proxy/behaviour doesn't exist)
 ResolvedTchandersT344921 Ensure dependent SQL statements are all run in the same batch
 ResolvedTchandersT344981 Map tables should not have null for any of their values
 ResolvedSTranT345757 Create orchestration script for daily data update
 ResolvedNoneT345761 Turn scripts into modules that export
 ResolvedSTranT345684 Implement batch tracking for imports
 OpenBUG REPORTTchandersT341660 Investigate non-deterministic behaviour of import

Event Timeline

Tchanders created this task.Jul 3 2023, 1:03 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 3 2023, 1:03 PM
Tchanders added subtasks: T340178: "Column 'anonymous' cannot be null", T340176: "Data too long for column 'behavior' at row 1", T339837: "Data too long for column 'anonymous' at row 1", T339359: 'Parameter at position 2 is undefined\n' if tunnels.type is not set, T339345: "Duplicate entry '0' for key 'behavior'", T339343: "Column 'type' cannot be null", T339325: "Duplicate entry '0' for key 'proxy'", T339324: "Data too long for column 'type' at row 1", T339073: Density imported incorrectly.Jul 3 2023, 1:06 PM
Tchanders added a parent task: T339284: Deploy ipoid.Jul 3 2023, 1:09 PM
• TThoabala closed subtask T339837: "Data too long for column 'anonymous' at row 1" as Declined.Jul 21 2023, 2:47 PM
• TThoabala closed subtask T340176: "Data too long for column 'behavior' at row 1" as Declined.Jul 24 2023, 1:59 PM
STran closed subtask T340178: "Column 'anonymous' cannot be null" as Resolved.Jul 25 2023, 2:23 AM
STran closed subtask T339345: "Duplicate entry '0' for key 'behavior'" as Resolved.Jul 25 2023, 2:29 AM
Tchanders closed subtask T339343: "Column 'type' cannot be null" as Resolved.Aug 14 2023, 10:20 AM
Tchanders closed subtask T339359: 'Parameter at position 2 is undefined\n' if tunnels.type is not set as Resolved.Aug 14 2023, 10:23 AM
Tchanders closed subtask T339073: Density imported incorrectly as Resolved.Aug 14 2023, 11:09 AM
Tchanders added a subtask: T344749: Refactor import-db and update-db into one script.Aug 23 2023, 1:51 PM
Tchanders closed subtask T344749: Refactor import-db and update-db into one script as Resolved.Sep 11 2023, 6:25 PM
STran closed subtask T339325: "Duplicate entry '0' for key 'proxy'" as Declined.Oct 4 2023, 6:31 AM
STran closed subtask T339324: "Data too long for column 'type' at row 1" as Invalid.Oct 17 2023, 8:10 AM
kostajh closed this task as Resolved.Jan 29 2024, 1:40 PM
kostajh claimed this task.
kostajh subscribed.

Closing this one, we can track the remaining two open subtasks individually.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL