What is the bytesize payload of the summaries (it's estimated to be about 10-15mb, is that true?

I simulated the output size of the summaries by creating a folder with 10,000 files. Each file is given the name of the article, and the contents of each file is the first 100 words of the original intro text.

data source: https://gitlab.wikimedia.org/repos/web/simple-summary-server/-/blob/main/output.csv?ref_type=heads

On disk, on a Mac, it looks like the amount of summary data is closer to 40mb. This is about twice as large as the original CSV file. probably due to the overhead of filesystem metadata associated with individual files.
Regardless, this is too big! Adding 40mb to the browser extension is too much. It can affect download as well as update times.
> IF It turns out the suggested approach is not viable, propose an alternative strategy,
I just uploaded the test summaries to a WMF static file server called people.wikimedia.org:
e.g: https://people.wikimedia.org/~jdrewniak/test-summaries/0
The open question now is: are there CORS restrictions preventing us from accessing the summaries from people.wikimedia.org on a Wikipedia page? If so, there a different or better place to host these summaries?

EDIT:

After further investigation, it turns out that, although a directory full of summaries is 40Mb, a single JSON file with that same data is only 6Mb, and, the directory size gzipped is only 3Mb. The reason for this large initial size is still unknown. I'm linking both the JSON file and the directory for comparison:

• https://people.wikimedia.org/~jdrewniak/test-summaries.tar.gz
• https://people.wikimedia.org/~jdrewniak/summary-output.json

I think for the purpose of this experiment, 6Mb of JSON is acceptable to load into memory, so I think we can get away using a single file as a data store for the extension.

I created a patch that integrates 10,000 article demo summaries into the extension here:
https://gitlab.wikimedia.org/repos/web/web-experiments-extension/-/commit/17c1ef2e777f5946cf17fe77d01ff9f057ca06cd#ca642a22686971b261b60de81d8ff554fdd91e2e
This adds 6.2mb to the extension, but the summaries are only loaded as necessary based on the article title.

The patch introduces a new content script file, summaryDispatcher.js this file is loaded as a content script in the 'isolated' context, which means it's only loaded when someone visits certain domains like https://*.wikipedia.org. The script initiates a window.postMessage handler that responds to postMessage requests in the form of:

// code to request an article summary
window.postMessage( { action: 'SUMMARY_REQUEST', title: 'Cleopatra' } );

The summaryDispatcher.js script reads the extension specific summary files and returns a postMessage response with the summary content.

The on-page script can the retrieve a summary with the following code (testable in the browser console):

// add a listener to capture responses
window.addEventListener( 'message', ( message ) => {
	if ( message.data.action === 'SUMMARY_RESPONSE' ) {
		console.log( 'got the summary:', message.data.content );
	}
} );
// send requests for summaries based on title
window.postMessage( { action: 'SUMMARY_REQUEST', title: 'Cleopatra' } );
window.postMessage( { action: 'SUMMARY_REQUEST', title: 'Andes' } );

next steps here: sync with @bwang to review this approach and see how we can integrate the summary source with the UI component.

jdrewniak opened https://gitlab.wikimedia.org/repos/web/web-experiments-extension/-/merge_requests/15

Add Pre-generated summaries & delivery mechanism

bwang merged https://gitlab.wikimedia.org/repos/web/web-experiments-extension/-/merge_requests/15

Add Pre-generated summaries & delivery mechanism

Can this be QAed since this is a spike?