I have also seen this error for a legitimate proxy entry. The exception only appears intermittently (for 10 tries I got it 7 times):

/srv/service/node_modules/mariadb/lib/misc/errors.js:61
  return new SqlError(msg, sql, fatal, info, sqlState, errno, additionalStack, addHeader);
         ^

SqlError: (conn=1902, no: 1062, SQLState: 23000) Duplicate entry 'LUMINATI_PROXY' for key 'proxy'
sql: INSERT INTO proxies (proxy) VALUES (?); - parameters:['LUMINATI_PROXY']
    at Object.module.exports.createError (/srv/service/node_modules/mariadb/lib/misc/errors.js:61:10)
    at PacketNodeEncoded.readError (/srv/service/node_modules/mariadb/lib/io/packet.js:511:19)
    at Query.readResponsePacket (/srv/service/node_modules/mariadb/lib/cmd/resultset.js:46:28)
    at PacketInputStream.receivePacketBasic (/srv/service/node_modules/mariadb/lib/io/packet-input-stream.js:104:9)
    at PacketInputStream.onData (/srv/service/node_modules/mariadb/lib/io/packet-input-stream.js:169:20)
    at Socket.emit (node:events:513:28)
    at addChunk (node:internal/streams/readable:315:12)
    at readableAddChunk (node:internal/streams/readable:289:9)
    at Socket.Readable.push (node:internal/streams/readable:228:10)
    at TCP.onStreamRead (node:internal/stream_base_commons:190:23) {
  text: "Duplicate entry 'LUMINATI_PROXY' for key 'proxy'",
  sql: "INSERT INTO proxies (proxy) VALUES (?); - parameters:['LUMINATI_PROXY']",
  fatal: false,
  errno: 1062,
  sqlState: '23000',
  code: 'ER_DUP_ENTRY'
}

Reproduction data: ask @dom_walden

Been testing our reprod data for different bugs lately (T339324, T339345, T340178, T341660) and haven't come across these specific issues with the Proxy table.

Looking at the Proxy table in the latest version of SchemaFile.js

CREATE TABLE IF NOT EXISTS proxies (
   pkid BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
   proxy VARBINARY(128) UNIQUE,
   PRIMARY KEY (pkid)
);

By importing the same files over and over again, the values simply do not duplicate into the existing DB. No errors are being raised. ImportData.js covers the cases of duplicate entries and increments the proxies pkid.

The issue am I getting from this reprod data is for the Tunnel table.

Incorrect integer value: '?' for column `test`.`tunnels`.`anonymous` at row 1
sql: INSERT INTO tunnels (operator, type, anonymous) VALUES (?, ?, ?) - parameters:['¿;¸
                                                    ÂÈÞ𢳎','«','񀯚']

  code: 'ER_TRUNCATED_WRONG_VALUE_FOR_FIELD'

More reprod data, if it helps:

In T339325#9048106, @dom_walden wrote:

More reprod data, if it helps:

random_test_data_valid_5000_18.json.gz289 KBDownload

Thanks for that, @dom_walden, I was able to replicate the error with the above test data.

1. I get a duplicated key error on the first import.
2. I also see new data added the more I import the same file, over and over again, randomly
3. I get an "out of range" error when I quickly re-initiate and re-import the same file after the first try.

Reproducing the error

1. When I first import the file, the bug seems to happen after the 195th row (I log each proxy before being added into DB).

"Duplicate entry 'r' for key 'proxy'".

The log in importData.js shows it has stopped for pkid 195 and proxy "ImFhy" as seen below, and while replicating the same steps, it has also stopped at row 189th. In the DB, I do have +200 entries from that first import, and pkid ordered by ASC, shows up to pkid 204.
We do see the 195th pkid being "ImFhy" and a lot more into DB that didn't get log from the import.

2. I re-import the same file, not being blocked this time. I get +100 new entries. I would re-import, sometimes I would have new entries, and sometimes nothing.

Re-initiating the DB, importing again, and getting the same block at row 195th.

"Duplicate entry 'r' for key 'proxy'".

3. Reinitiate (quickly) the DB, re-import (quickly after), and get a new error at row 132nd:

"Duplicate entry 'hjLkoKK' for key 'proxy'"

The proxyPkid isn't being created (or at least, not logged), yet, still exists in the DB. I also got an error of "Duplicate entry 'PEydL' for key 'proxy'" at row 131st.

But what I also notice it's the second error always stops due to an "out of range" error.

Routes for investigation:

• There's a lag between the import and what is actually being saved into DB. Is it coming from the speed of our system?
• There are a lot of errors happening in the import that might contribute to failing the import and creating lag:

{"name":"data-import","hostname":"fd12dff0eb3d","pid":2269,"level":50,"err":**{"message":"Cannot read properties of undefined (reading 'country')**","name":"TypeError","stack":"TypeError: Cannot read properties of undefined (reading 'country')\n    at Transform.transform [as _transform] (/srv/service/import-data.js:271:55)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)"},"ip":"7ab3d130-8221-49bf-90b8-66e282a00eae","msg":"Cannot read properties of undefined (reading 'country')","time":"2023-07-28T13:43:38.241Z","v":0}

{"name":"data-import","hostname":"fd12dff0eb3d","pid":2365,"level":50,"err":{"message":"(conn=553, no: 1264, SQLState: 22003) Out of range value for column 'conc_density' at row 1\nsql: REPLACE INTO\n            actor_data (ip,org,client_count,types,conc_geohash,conc_city,conc_state,conc_country,conc_skew,conc_density,countries,location_country,risks)\n        VALUES\n            (?,?,?,?,?,?,?,?,?,?,?,?,?) - parameters:['4e006d00-98dd-4a46-...]"

In T339325#9048106, @dom_walden wrote:

More reprod data, if it helps:

random_test_data_valid_5000_18.json.gz289 KBDownload

Further testing with the same testing files. In this testing file, there are around 5000 entries. The "type": "R" value is indeed present 9 times in the file. 4 times in small caps and 5 times in capital letters.

"type": "R"
"type": "r"

Results:
Import with less than 1000 rows does not raise duplicate import error or any other type of error despite having multiple occurrences of the value "R" in the file. Yet, no error is raised but the value is also not saved into DB.
It seems that it's not the values themselves or the way we import that is causing issues but the capacity of our system in taking in so many errors, entries to be saved into DB, and logging errors, at the same time.

Test 1: Small file - No error - No value "r" added
Running: docker compose exec web node init-db.js
Running: docker compose exec web node import-data.js ./tmp/test-1.json.gz
Importing the 533 first rows, with two occurrences of "type": "R", one in small-cap and one in capital letter.
No error when importing.
Running: select * from proxies ORDER by pkid ASC;
In the DB, 4 rows were added, and no entry of value "r" or "R" was added.

Test 2: 2500 rows - Duplicate error - One value 'r' added in DB
Importing the 2500 first rows, with 9 occurrences of "type": "R"
ERROR: Duplicate entry 'r' for key 'proxy'. Import stopped at pkid 189.
In the DB, 204 entries are in the proxies table. Only one entry of value "r" was added with pkid 182

Test 3: 2300 rows - Duplicate error - One value 'r' added in DB
Importing the 2300 first rows, with 8 occurrences of "type": "R"
ERROR: Duplicate entry 'r' for key 'proxy'. Import stopped at pkid 189.
In the DB, 204 entries are in the proxies table. Only one entry of value "r" was added with pkid 182.

Test 4: 5000 rows - Duplicate error - One value 'r' added in DB
Importing the the 5000+ rows, but moving the 9 occurrences of "type": "R" within the first 800 rows.
ERROR: Duplicate entry 'r' for key 'proxy'. Import stopped at pkid 189.
In the DB, 204 entries are in the proxies table. Only one entry of value "r" was added with pkid 182.

Test 5: 1000 rows - No error - No value "r" added
Importing the 1000 first rows, with the 10 occurrences of "type": "R" within the first 800 rows.
No import error
In the DB, 47 entries are in the proxies table. No entry with value "r" or "R".

Test 6: 5000 rows - Duplicate error - One value 'r' added in DB
Importing the the 5000+ rows, but moving the 9 occurrences of "type": "R" within the first 20 rows.
ERROR: Duplicate entry 'r' for key 'proxy'. Import stopped at pkid 189.
In the DB, 204 entries are in the proxies table. Only one entry of value "r" was added with pkid 182.

Test 7: 2500 rows - Duplicate error - One value 'r' added in DB
Importing the the 2500 rows, but moving the 9 occurrences of "type": "R" within the first 20 rows.
ERROR: Duplicate entry 'r' for key 'proxy'. Import stopped at pkid 191.
In the DB, 204 entries are in the proxies table. Only one entry of value "r" was added with pkid 182.

Test 8: 1000 rows - No error - No value "r" added
Importing the the 1000 rows, but moving the 9 occurrences of "type": "R" within the first 20 rows.
No error
In the DB, 47 entries are in the proxies table. No entry with value "r" or "R".

@AGueyte Just wanted to check what's changed on this task since we discussed it in Engineering? Thanks

In T339325#9089722, @Tchanders wrote:

@AGueyte Just wanted to check what's changed on this task since we discussed it in Engineering? Thanks

Hi, nothing since we talked about it in Engineering meeting.
If I recall correctly, we decided to close this ticket as the issue won't be happening with T341122.

Thanks @AGueyte - happy to let @STran decide what to do with this task.