Steps to replicate the issue (include links if applicable):
- Save the JSON from "Reproduction data" above as a .gz file (e.g. type_too_big.json.gz) into the ipoid/tmp directory
- If necessary, start up docker in the ipoid directory (e.g. docker compose up -d)
- Initialise the database: docker compose exec web node init-db.js
- Run: docker compose exec -e FEED_PATH=./tmp/type_too_big.json.gz web node import-data.js
- The first time you run this you might see the exception from T339325 instead, so you might need to run the command again
- If you still don't see the exception from the description above, keep running the command until you do
What happens?:
When getting the reproduction data from https://phabricator.wikimedia.org/T339324. I am not getting the error of Data too long but now code: 'ER_TRUNCATED_WRONG_VALUE_FOR_FIELD'
/srv/service/node_modules/mariadb/lib/misc/errors.js:61
return new SqlError(msg, sql, fatal, info, sqlState, errno, additionalStack, addHeader);
^
SqlError: (conn=65, no: 1366, SQLState: 22007) Incorrect integer value: '?' for column `ipoid`.`tunnels`.`anonymous` at row 1
sql: INSERT INTO
tunnels (operator, type, anonymous)
VALUES
(?, ?, ?) - parameters:['¿;¸
ÂÈÞ𢳎','UNKNOWN','']
at Object.module.exports.createError (/srv/service/node_modules/mariadb/lib/misc/errors.js:61:10)
at PacketNodeEncoded.readError (/srv/service/node_modules/mariadb/lib/io/packet.js:511:19)
at Query.readResponsePacket (/srv/service/node_modules/mariadb/lib/cmd/resultset.js:46:28)
at PacketInputStream.receivePacketBasic (/srv/service/node_modules/mariadb/lib/io/packet-input-stream.js:104:9)
at PacketInputStream.onData (/srv/service/node_modules/mariadb/lib/io/packet-input-stream.js:169:20)
at Socket.emit (node:events:513:28)
at addChunk (node:internal/streams/readable:315:12)
at readableAddChunk (node:internal/streams/readable:289:9)
at Socket.Readable.push (node:internal/streams/readable:228:10)
at TCP.onStreamRead (node:internal/stream_base_commons:190:23) {
text: "Incorrect integer value: '?' for column `ipoid`.`tunnels`.`anonymous` at row 1",
sql: 'INSERT INTO\n' +
' tunnels (operator, type, anonymous)\n' +
' VALUES\n' +
" (?, ?, ?) - parameters:['¿;¸\x0F\x0BÂÈÞ𢳎','UNKNOWN','']",
fatal: false,
errno: 1366,
sqlState: '22007',
code: 'ER_TRUNCATED_WRONG_VALUE_FOR_FIELD'What should have happened instead?:
Request non-integer value to be filtered out from being saved in tunnels.anonymous table, and add a question of whether we want to filter for such cases if we believe Spur would be delivering such data in the import files.
Other information (browser name/version, screenshots, etc.):
ipoid commit: 7e0b97c89f535f3813dd58ab076dfdd04e354001
also related to:
https://phabricator.wikimedia.org/T339324
https://phabricator.wikimedia.org/T345107