Change 361466 had a related patch set uploaded (by BearND; owner: BearND):
[mediawiki/services/mobileapps@master] Add scripts to measure payloads

https://gerrit.wikimedia.org/r/361466

Here are some preliminary results (before stripping of references is implemented) with a small sample of test pages (some were taken from the most-read results from 6/22/2017):
If I don't strip any HTML and just add some of the markers and other changes needed MCS read-html adds around 1.9% to the payload.
If stripping of unneeded tags is included MCS read-html reduces the payload from 23% to 47% (avg. around 37%).

Very interesting. Seems in line with previous research 👍

The bulk of the space-saving transformations comes from https://github.com/wikimedia/mediawiki-services-mobileapps/blob/master/lib/transforms.js#L193-L268.

To add to this we want to

• get the top 1000 page views of the month from PageViews AP automatically and
• run this against zhwiki, too.

Here's a spreadsheet with the results against the top monthly titles in enwiki and zhwiki.

Here's a summary of the payload sizes I measured. The spreadsheet compares the gzipped (-6) payloads: plain Parsoid against various MCS/PCS read-html variants: no stripping, just stripping of unneeded markup, just stripping of reference lists, stripping both.

enwiki:

• no stripping: adds about 2% to the payload
• stripping of unneeded markup: saves around 36%
• stripping of reference lists: saves around 36%
• stripping of both unneeded markup and reference lists: saves around 53%

The savings of stripping reflists varies more than the savings of unneeded markup. Adding reflist stripping to already stripping unneeded markup only save almost around 17% points of the original payload.

zhwiki:

• no stripping: adds about 1.2% to the payload
• stripping of unneeded markup: saves around 36%
• stripping of reference lists: saves around 15%
• stripping of both unneeded markup and reference lists: saves around 42%

Adding reflist stripping to already stripping unneeded markup only save almost around 7% points of the original payload.

This leads to the question: is ref stripping really worth the hassle?

Screen shot of the averages…

@bearND looking over it seems like both striping reflists and removing unneeded markup both give a significant savings (Both about 36% on en). But as you noted: when combined both isn't as drastic as you would have thought (About 53% on en). Having said that, an extra 16% savings isn't insignificant.

zh seems to be not as heavily referenced for sure. This may be a cultural thing.

As far as next steps are concerned, I think we have enough data to proceed with the parsoid markup stripping in the PCS.

For the references, lets keep pushing on the structured data exploration and see how that turns out and then make decisions after we determine if we can reliably parse that data.

Sound like a plan to you?

Yes, sounds like a plan. I was just a bit surprised to see that the savings don't get close to adding up. My explanation for this is that the transforms for stripping the unneeded markup heavily strip the reference list content. There's still one included which I'll have to take out: stripping of ref back links. For the Android app we remove them but for the web case I think we would want to preserve them.

I looked a bit into the effect of stripping specific elements, focusing on those that aren't essential for editing. We plan to move data-mw out in any case (see T78676), so I started with that. All ratios are relative to the original Parsoid HTML, all using gzip -6.

So I think it is quite clear that moving out data-mw & reducing the number of ids would already realize a significant portion of the gains you saw with the more aggressive removal. IDs are currently used to anchor metadata, but we could replace this with a lighter weight scheme using path selectors (nth-child etc).

About attributes are heavily used to associate references with their reflist entry. They are somewhat redundant with the href. Both about & href could be removed if we inlined reference content.

The vast majority of rel attributes are mw:Wikilink or Extlink. This information is already implicit in the link target, so could just be removed without loss of information or editing support. However, this does not make a huge difference as they compress away mostly.

Quick & dirty script used to generate the numbers above:

#!/bin/bash

if [ -z $1 ];then
    echo "Usage: $0 <title>"
    exit 1
fi

file="$1.html"
curl -s "https://en.wikipedia.org/api/rest_v1/page/html/$1" | gzip -6 > "$file"
size_raw=$(wc -c < "$file")
size_data_mw=$(zcat $file | sed -e 's/ data-mw="[^"]\+"//g' -e "s/ data-mw='[^']\+'//g" | gzip -6 | wc -c)
ratio_data_mw=$(bc <<< "scale=2; $size_data_mw / $size_raw")

size_data_mw_ids=$(zcat $file | sed -e 's/ data-mw="[^"]\+"//g' -e "s/ data-mw='[^']\+'//g" -e 's/ id="[^"]\+"//g' | gzip -6 | wc -c)
ratio_data_mw_ids=$(bc <<< "scale=2; $size_data_mw_ids / $size_raw")


size_data_mw_ids_about=$(zcat $file | sed -e 's/ data-mw="[^"]\+"//g' -e "s/ data-mw='[^']\+'//g" -e 's/ \(id\|about\)="[^"]\+"//g' | gzip -6 | wc -c)
ratio_data_mw_ids_about=$(bc <<< "scale=2; $size_data_mw_ids_about / $size_raw")

size_data_mw_ids_about_rel=$(zcat $file | sed -e 's/ data-mw="[^"]\+"//g' -e "s/ data-mw='[^']\+'//g" -e 's/ \(id\|about\|rel\)="[^"]\+"//g' | gzip -6 | wc -c)
ratio_data_mw_ids_about_rel=$(bc <<< "scale=2; $size_data_mw_ids_about_rel / $size_raw")

echo "|$1|$ratio_data_mw|$ratio_data_mw_ids|$ratio_data_mw_ids_about|$ratio_data_mw_ids_about_rel|"

I called this on a list of titles using

cat titles.txt | while read t; do ./checkSizes.sh "$t"; done

Thank you for this. For some of the pages I measured this looks quite close to the more aggressive markup stripping MCS is doing (usually < 15% points). I agree that the rel attributes don't make much difference.

In T164033#3444391, @bearND wrote:

Yes, sounds like a plan. I was just a bit surprised to see that the savings don't get close to adding up. My explanation for this is that the transforms for stripping the unneeded markup heavily strip the reference list content.

I noticed the same when doing research in the past. My hypothesis was that one as well. I realized trying to make the pie charts add up to 100%, it doesn't work because some transforms overlap with others.

Change 361466 merged by jenkins-bot:
[mediawiki/services/mobileapps@master] Add scripts to measure payloads

https://gerrit.wikimedia.org/r/361466

Mentioned in SAL (#wikimedia-releng) [2017-09-06T21:32:14Z] <bearND> Update mobileapps to 2cb6281 (T168848 T169277 T169274 T162179 T164033 T167921 T174698 T168848 T174808)

Mentioned in SAL (#wikimedia-operations) [2017-09-06T22:06:32Z] <bsitzmann@tin> Started deploy [mobileapps/deploy@507a479]: Update mobileapps to 2cb6281 (T168848 T169277 T169274 T162179 T164033 T167921 T174698 T168848 T174808)

Mentioned in SAL (#wikimedia-operations) [2017-09-06T22:11:25Z] <bsitzmann@tin> Finished deploy [mobileapps/deploy@507a479]: Update mobileapps to 2cb6281 (T168848 T169277 T169274 T162179 T164033 T167921 T174698 T168848 T174808) (duration: 04m 53s)