Page MenuHomePhabricator

writeuptopageid failing to split svwiki dump
Closed, ResolvedPublic

Description

Using this throaway script to generate the command: P8201

I get:

writeuptopageid -fspec svwiki-01.bz2:1:1000000,svwiki-02.bz2:1000000:2000000,svwiki-03.bz2:2000000:3000000,svwiki-04.bz2:3000000:4000000,svwiki-05.bz2:4000000:5000000,svwiki-06.bz2:5000000:6000000,svwiki-07.bz2:6000000:7000000,svwiki-08.bz2:7000000:8000000,svwiki-09.bz2:8000000:8274856 -i /mnt/data/xmldatadumps/public/svwiki/20190301/svwiki-20190301-pages-meta-history.xml.bz2 -odir .

Which, when ran on stat1007, errors:

*** Error in `writeuptopageid': double free or corruption (fasttop): 0x00005605dffcd230 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x70bfb)[0x7f78c85abbfb]
/lib/x86_64-linux-gnu/libc.so.6(+0x76fc6)[0x7f78c85b1fc6]
/lib/x86_64-linux-gnu/libc.so.6(+0x7780e)[0x7f78c85b280e]
writeuptopageid(+0x2c83)[0x5605df100c83]
writeuptopageid(+0x2eb5)[0x5605df100eb5]
writeuptopageid(+0x2fe0)[0x5605df100fe0]
writeuptopageid(+0x13f9)[0x5605df0ff3f9]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f78c855b2e1]
writeuptopageid(+0x144a)[0x5605df0ff44a]
======= Memory map: ========
5605df0fe000-5605df103000 r-xp 00000000 fd:00 194840216                  /srv/home/gilles/mwbzutils/xmldumps-backup/mwbzutils/writeuptopageid
5605df303000-5605df304000 r--p 00005000 fd:00 194840216                  /srv/home/gilles/mwbzutils/xmldumps-backup/mwbzutils/writeuptopageid
5605df304000-5605df305000 rw-p 00006000 fd:00 194840216                  /srv/home/gilles/mwbzutils/xmldumps-backup/mwbzutils/writeuptopageid
5605dffac000-5605dffee000 rw-p 00000000 00:00 0                          [heap]
7f78c4000000-7f78c4021000 rw-p 00000000 00:00 0 
7f78c4021000-7f78c8000000 ---p 00000000 00:00 0 
7f78c8324000-7f78c833a000 r-xp 00000000 09:00 1310734                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7f78c833a000-7f78c8539000 ---p 00016000 09:00 1310734                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7f78c8539000-7f78c853a000 r--p 00015000 09:00 1310734                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7f78c853a000-7f78c853b000 rw-p 00016000 09:00 1310734                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7f78c853b000-7f78c86d0000 r-xp 00000000 09:00 1310741                    /lib/x86_64-linux-gnu/libc-2.24.so
7f78c86d0000-7f78c88d0000 ---p 00195000 09:00 1310741                    /lib/x86_64-linux-gnu/libc-2.24.so
7f78c88d0000-7f78c88d4000 r--p 00195000 09:00 1310741                    /lib/x86_64-linux-gnu/libc-2.24.so
7f78c88d4000-7f78c88d6000 rw-p 00199000 09:00 1310741                    /lib/x86_64-linux-gnu/libc-2.24.so
7f78c88d6000-7f78c88da000 rw-p 00000000 00:00 0 
7f78c88da000-7f78c88f3000 r-xp 00000000 09:00 1310881                    /lib/x86_64-linux-gnu/libz.so.1.2.8
7f78c88f3000-7f78c8af2000 ---p 00019000 09:00 1310881                    /lib/x86_64-linux-gnu/libz.so.1.2.8
7f78c8af2000-7f78c8af3000 r--p 00018000 09:00 1310881                    /lib/x86_64-linux-gnu/libz.so.1.2.8
7f78c8af3000-7f78c8af4000 rw-p 00019000 09:00 1310881                    /lib/x86_64-linux-gnu/libz.so.1.2.8
7f78c8af4000-7f78c8b03000 r-xp 00000000 09:00 1310727                    /lib/x86_64-linux-gnu/libbz2.so.1.0.4
7f78c8b03000-7f78c8d02000 ---p 0000f000 09:00 1310727                    /lib/x86_64-linux-gnu/libbz2.so.1.0.4
7f78c8d02000-7f78c8d03000 r--p 0000e000 09:00 1310727                    /lib/x86_64-linux-gnu/libbz2.so.1.0.4
7f78c8d03000-7f78c8d04000 rw-p 0000f000 09:00 1310727                    /lib/x86_64-linux-gnu/libbz2.so.1.0.4
7f78c8d04000-7f78c8d27000 r-xp 00000000 09:00 1310735                    /lib/x86_64-linux-gnu/ld-2.24.so
7f78c8f13000-7f78c8f15000 rw-p 00000000 00:00 0 
7f78c8f23000-7f78c8f27000 rw-p 00000000 00:00 0 
7f78c8f27000-7f78c8f28000 r--p 00023000 09:00 1310735                    /lib/x86_64-linux-gnu/ld-2.24.so
7f78c8f28000-7f78c8f29000 rw-p 00024000 09:00 1310735                    /lib/x86_64-linux-gnu/ld-2.24.so
7f78c8f29000-7f78c8f2a000 rw-p 00000000 00:00 0 
7ffd83241000-7ffd83345000 rw-p 00000000 00:00 0                          [stack]
7ffd833af000-7ffd833b1000 r--p 00000000 00:00 0                          [vvar]
7ffd833b1000-7ffd833b3000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Aborted

Event Timeline

The fspec wants a semicolon (;) separator instead of a comma. The usage message is right about that. However you should get shown a usage message instead of the core dump of death. I'll fix that up, thanks!

Change 497263 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/dumps/mwbzutils@master] die with usage message on bad filespec arg

https://gerrit.wikimedia.org/r/497263

Change 497263 merged by ArielGlenn:
[operations/dumps/mwbzutils@master] die with usage message on bad filespec arg

https://gerrit.wikimedia.org/r/497263

This will be incorporated with the next build, probably in a few weeks. I'll leave this open until that happens.

This was deployed ... at some point. We're at least two versions past that now so I'm closing this task.