Having an issue with the encoding of the externalllinks dumps, which seem to be latin1 instead of utf-8, which I think is what is expected.
Steps to reproduce
Using standard python3 installation on stat1004 machine. As far as we can tell, this applies to other dates / languages too.
import gzip
# this doesn't work (utf-8 encoding)
with gzip.open('/mnt/data/xmldatadumps/public/enwiki/latest/enwiki-latest-externallinks.sql.gz', 'rt') as fin:
for line in fin:
continue
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
File "/usr/lib/python3.7/codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd1 in position 406: invalid continuation byte
# this works just fine:
with gzip.open('/mnt/data/xmldatadumps/public/enwiki/latest/enwiki-latest-externallinks.sql.gz', 'rt', encoding='latin1') as fin:
for line in fin:
continue