I have the recipe dumpsdata100X-no-data-format.cfg which does less than it should (but at least doesn't format the array). I'd love a fully functional solution.

@dcausse what's your time frame?

Unless folks want to keep it open to work on speeding it up in the future?

I have tested the above patch on snapshot1010 and the exception is suppressed.

I guess that we need to wrap $page->getRedirectTarget() with invokeLenient() and make sure that no other parts of the code can later trip up on this exception.

More things needed:

I see that we're no longer blocked. Does this mean that we're good to go for weekly runs?

In T238199#6135018, @Ladsgroup wrote:

...

Anyway, Lydia said it's fine to do it tomorrow when it gets announced by our communication manager. Does that work for you?

Can we do this temporarily while the query is being fixed up? It looks like it had to be killed in Nov, Feb, Apr, May, so I'd rather temp disable than require folks to shoot it (and anything else hung as a side effect).

Can we just skip the updateSpecialPages.php wikidatawiki --override --only=Fewestrevisions script altogether, instead of shooting it every month?

As I understand it the long running query comes from a monthly cron job.

The basic idea is that the scripts will run as they have done up to now, but after a host running 'small' or 'big' (but not huge) wikis completes, it will check for 'batches' for wikidatawiki and run any o those available, at the same time that the main server for the wikidatawiki is going about its business writing out page meta history files, also in batches.

One more question: are these changes going to be applied in deployment-prep first? Because I'd like to test there just to be extra sure.

When these new fields rev_actor and rev_comment_id are added, what populates them?

In T238966#6118136, @Marostegui wrote:

In T238966#6118001, @Bstorm wrote:

Should I get a ticket open and get communicating with the community ASAP on dropping rev_text_id, rev_content_model, rev_content_format, ar_text_id, ar_content_model, and ar_content_format?

+1
I will also start the change on a s6 (frwiki, jawiki, ruwiki) codfw slave first, leave it for a few days, and then on an s6 eqiad slave and leave it for a few days as well to make sure we are good to go.

Also ping @ArielGlenn as this will be a massive schema change, pinging in case there's stuff that needs to change within the dumps world!

No internal links @Reedy, sorry ;-) The above can go as soon as someone gives the final thumbs up.

Let's add that to the 'other' index.html page too, or no one will know it's there. Can someone supply a phrase describing the contents, for downloaders?

On which host(s) are you running the above queries?

Making the command decision to block this task on T218446 / T35334.

See also T35334 for another bug on removing that field from core.

Sounds good to me, though we probably want that dicussion on the other task.

The HTML dump can be in a tarball for download, sure. But that is separate from what was requested in T243056 i.e. actually serving a static copy for browsing. I don't think the labstore boxes should be doing that.

I don't know the extension either. Hey @brion :-D

See also: T218446

This has been broken a long time. I need to check a little bit more of the history, but I can verify that exports use the contents of the obsolete page_restrictions field from the page table instead of the cooresponding entries in the page_restrictions table. Proof of this: enwiki page 'Authur Schoepnhauer' with page id 700 has the following row entry:

wikiadmin@10.64.32.76(enwiki)> select * from page where page_id = 700;
+---------+----------------+---------------------+-------------------+------------------+-------------+--------------------+----------------+--------------------+-------------+----------+--------------------+-----------+
| page_id | page_namespace | page_title          | page_restrictions | page_is_redirect | page_is_new | page_random        | page_touched   | page_links_updated | page_latest | page_len | page_content_model | page_lang |
+---------+----------------+---------------------+-------------------+------------------+-------------+--------------------+----------------+--------------------+-------------+----------+--------------------+-----------+
|     700 |              0 | Arthur_Schopenhauer | move=:edit=       |                0 |           0 | 0.8153719695428131 | 20200430073836 | 20200426170144     |   952937594 |   155341 | wikitext           | NULL      |
+---------+----------------+---------------------+-------------------+------------------+-------------+--------------------+----------------+--------------------+-------------+----------+--------------------+-----------+

and no entries in the page_restrictions table. But the contents of the xml element in the pages-articles bz2 file are "move=:edit="

I've been looking at the en wiki dump files for part 18 (containing this page) and there are no entries for page restrictions in either the 2019-02 dumps or the 2018-02 dumps. So this bug seems to have been around for awhile. I'm checking older dumps but it will take a while to download and decompress them.

In T205361#6093612, @Dzahn wrote:

Maybe talk to @ArielGlenn about getting it on the official dumps servers (dumps.wikimedia.org) under "misc". That would be more stable than the people VM.

labstore1006.wikimedia.org and labstore1007.wikimedia.org in /srv/dumps/xmldatadumps/public/other let's make the subdirectory codereview with a subdirectory for the date (20200428 I guess) and drop the gz files in there. I guess you can make the directories and files owner and group dumpsgen:dumpsgen.

So to be very clear, this report is only about importing the current revision of a page if it is the same as the existing top revision on the local wiki. I am unsure as to whether skipping such revisions ought to be the default, but it could at least be a checkbox.

Done.

This is complete and tables have been dumped using the new code.

Keeping this open until the code path gets used, probably during the May run.

Closing this since all files are now available and patch has been deployed and used.

Just a check-in to see where this is on people's radar.

Sent: https://lists.wikimedia.org/pipermail/xmldatadumps-l/2020-April/001531.html so closing.

Hi, just checking in: any progress on invetigating the 'extra' dumps content?

Closing this, as all links have been fixed up.

Can you clarify the specific circumstances in which you would want a revision for import to be skipped? What would the person by trying to do, import the current revision of a page from elsewhere or potentially multiple revisions?

The noop for wikidata is running now.

Running

dumpsgen@snapshot1009:/srv/deployment/dumps/dumps/xmldumps-backup$ bash fixup_scripts/do_dumptextpass_jobs.sh --wiki wikidatawiki --config /etc/dumps/confs/wikidump.conf.dumps:wd --date 20200401 --jobinfo 27:70360538:88436737 --skiplock --numjobs 27

in a screen session. I'll need to intervene at some point to stop the current run, catch up the 7zs, and perhaps do a noop.

This is done now. Feel free to announce it wherever you like. After I have sent mail to the xmldatadumps-l list, I will close this task unless there is something else you need.

This is deployed, dumps have run with it, so it can be closed at last.

@Mholloway care to give me a one line description that I can use for the index.html page mentioned above? See the existing page for examples. I could announce it on the xmldatadumps-l mailing list afterwards, unless you care to do the honours.

Enwiki is complete, run with the merged patch. Wikidata wikis 7zs are being generated by manual runs a couple of times a day as the bz2 page content files are generated. I'll leave this task open until that's done.

I am manually running 7zs on wikidatawiki for te bz2 files that have been produced, to make sure we'll have them at the end of the run. Screen session on snapshot1009. 7zs for enwiki are proceeding in screen session on snapshot1010. The regular wikidatawiki run is continuing as usual on snapshot1006.

Done: arwiki commonswiki kowiki eswiki hewiki huwiki ukwiki metawiki dewiki jawiki zhwiki nlwiki plwiki viwiki ptwiki itwiki svwiki
In progress: ptwiki itwiki svwiki enwiki
Remaining: enwiki wikidatawiki

The dumps ran on Saturday as expected and are now available, but we should add an entry to https://dumps.wikimedia.org/other/ describing these. See https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/production/modules/dumps/files/web/html/other_index.html

I now understand the issue fully and have revised the patchset accordingly. Catchup will likely take a day or so on the wikis to fix.

I am running the 7z job on the above wikis with the patchset on the testbed host snapshot1010 in a screen session, three wikis at a time. If I get very impatient I may do another 3 at a time on another host.

This ws a consequence of the file lister refactoring done recently, and although I have a fix I d o not fully understand what caused the coed to break. Still investigating.