@jijiki can you expand on what you mean, please? This task is currently too broad...

I can't speak for Effie, but my imagined reimplementation of wikitech-static would be to produce an OCI container image daily containing Wikitech's content along with a MediaWiki deployment to serve it up. This image could be published in a well-known 3rd party registry and pulled by any interested party to produce a browseable copy of Wikitech. One usage could be hosting wikitech-static.wikimedia.org for anyone to visit, but it could also be pulled and run locally by anyone needing access to the content.

In T376400#10287901, @MatthewVernon wrote:

@jijiki can you expand on what you mean, please? This task is currently too broad...

For the time being the task is deliberately left without a description. It is something that must being taken care of, however it has not been officially assigned/claimed by anyone, thus there are no actual plans in place. I will update the task with the really basic prep work for this.

Sorry for the confusion

@MatthewVernon updated description

While this is being discussed, who is taking care of maintaining the current installation of wikitech-static?

This alert has been firing for more than 1 month:

MWVERSION WARNING - wikitech-static.wikimedia.org is running MediaWiki 1.42.1, latest is MediaWiki 1.42.3 Consult https://wikitech.wikimedia.org/wiki/Wikitech-static for details.

The alert is currently routed to team=wmcs in alertmanager (the routing was added in this patch), but I'm not sure if WMCS is the best team to handle this. I'm happy to help if somebody can show me the upgrade procedure. :)

Before I went on sabbatical I spent a while trying to decide if we can make wikitech-static into an actual static site. A static site would need much less maintenance, and syncing content should be pretty trivial (at the very worst, it could just use actual public https page loads from prod wikitech.)

What advantages do we get from running a static site on mediawiki? So far the only advantage I can think of that mediawiki has over a static site is that for a static site we'd have to figure out some kind of add-on for search -- so far I haven't turned up a good drop-in search solution for a static site, to my surprise.

The advantages of a static site seem obvious: it's the tech stack that's the least likely to let us down when we need it most, and doing a recursive sync/copy/download of wikitech should be pretty trivial. It would completely do away with our current need to keep MW versions in sync.

I worked on this a bit over the break. I'm pretty happy with the static site that httrack produces. It was generated like this:

httrack --update \
    --update \
    --keep-alive \
    --search-index \
    -#L1000000 \
    --connection-per-second=5000 \
    --sockets=99 \
    --max-rate=1000000000 \
    --disable-security-limits \
    -v \
    -wikitech.wikimedia.org/wiki/Special:* \
    -wikitech.wikimedia.org/wiki/Nova_Resource:* \
    -wikitech.wikimedia.org/w/index.php?title=Special \
    -*Special:* \
    -*Nova_Resource:* \
    -*oldid=* \
    -*action=* \
    -*index.php=* \
    https://wikitech.wikimedia.org

We could probably prune out even more for a smaller copy.

I'm still nowhere with search. I've followed various lunr guides to produce an index and a search bar but haven't seen anything work yet. What I /have/ seen is that generating an index takes an impractical amount of RAM and slows page loading by quite a lot; it might be possible to tune away much of that but I wouldn't want to have every page of the site include a full static search index; might have to add an additional hop to a search-only page.

https://wts.wmcloud.org/wiki/Main_Page.html now has a search bar (just on that one page) which is VERY ugly but which does provide useful (to my eyes, at least) search results.

There are lots of things I can do to make that page look better and to organize the mirroring process, but first I'd like to know if anyone likes this idea.

On @CDanis's suggestion, 'static wikitech-static' can now be built in a docker container using https://gitlab.wikimedia.org/repos/sre/wikitech-static-docker

The build takes around 4 hours, which hopefully is slow enough to not upset any wikitech monitoring. Since the copy is made via scraping a public website, I've now assigned quay.io a build task; let's see if it's patient enough to let it finish. https://quay.io/organization/wikitechstatic

Change #1117575 had a related patch set uploaded (by FNegri; author: FNegri):

[operations/puppet@production] icinga_exporter: don't route wikitech-static to wmcs

https://gerrit.wikimedia.org/r/1117575

I've worked on this a bit more; here's what we have today:

The good

• http://ec2-54-81-201-239.compute-1.amazonaws.com
• That site will update itself with the latest container build available on quay.io.
• Each page in the site has a search bar that relies on a pre-built search index + lunr
• Daily container rebuilds are automatically run on gitlab (gitlab.wikimedia.org:repos/sre/wikitech-static-docker.git)
• The VM can be trivially rebuilt using the user_data.sh script in the git repo without need for ssh
• The latest build of the site can also be downloaded and run locally with

docker run --name wikitechstatic -d -p 80:80  quay.io/wikitechstatic/static

• The site is 100% static content; running it doesn't rely on mediawiki in any way.

The bad

• The container is built by running a full scrape of wikitech.wikimedia.org once per day
• The aws site is http only; no https.

The assumptions

• I am assuming that a daily throttled scrape of the site is OK, and not a significant traffic load since it's just wikitech. Is that right?
• I am also assuming that adding a cert for https is not worth the trouble; since it's for emergency use the simplicity seems like an advantage.

As far as I'm concerned this site is ready for prime time and we should start working on getting the other wikitech-static services moved into the same approximate pipeline. I'd like at least one other person to agree with that though.

@Andrew thanks for setting this up. I did a quick tour and found some issues:

1. The first page is very very slow to load, I think it's because it downloads lunr_index.js that is 49.5MB and takes quite some time to download (order of 10~20 seconds)
2. While the JS is loading the whole page is blank beside the logo and the header menu on the left, giving the false impression that the site is broken/empty
3. If you are in a nested page, say /wiki/DNS/Netbox.html and search for something, say Cumin, the resulting links in the search results are broken because don't remove the nested path, pointing to say /wiki/DNS/Cumin.html instead of /wiki/Cumin.html and all return 404s
4. If a page has aliases it will result in the result page multiple times, pointing to the same page with different urls. For example if you search for ipmi the first 3 results are:
   • http://ec2-52-23-161-9.compute-1.amazonaws.com/wiki/IPMI-2.html: opens http://ec2-52-23-161-9.compute-1.amazonaws.com/wiki/Management_Interfaces with the text (Redirected from IPMI ) and if I reload the page it gives 404
   • http://ec2-52-23-161-9.compute-1.amazonaws.com/wiki/Ipmi.html opens http://ec2-52-23-161-9.compute-1.amazonaws.com/wiki/Management_Interfaces with the text (Redirected from Ipmi ) and if I reload the page it gives 404
   • http://ec2-52-23-161-9.compute-1.amazonaws.com/wiki/Management_Interfaces.html opens http://ec2-52-23-161-9.compute-1.amazonaws.com/wiki/Management_Interfaces.html without any redirection text
5. 404 pages don't have a template, it's the nginx default page
6. The search box doesn't search when pressing enter, you have to click the button

Thanks for the look! Since my goal here is 'adequate in an emergency' I'm going to set aside many of your concerns as aesthetic rather than functional, but I would like to make search work properly.

I agree that the initial page load is quite slow; as far as I can tell that's the price of static search although it would be nice to have the loading happen asynchronously. I'll see if there's a simple way to arrange that.

Hello @Volans !

I think I've addressed almost all of your bullet points above -- do you mind retesting? I needed to rebuild the VM, so the new site is at:

http://ec2-54-81-201-239.compute-1.amazonaws.com/wiki/Main_Page.html

btw I didn't do anything about the 404 page; do you regard that as necessary for functionality or is it just cosmetic? Anyway hopefully you won't see it as much this time around.

Thanks for the fixes @Andrew, I did another pass, replying to my own comments inline:

In T376400#10694816, @Volans wrote:

1. The first page is very very slow to load, I think it's because it downloads lunr_index.js that is 49.5MB and takes quite some time to download (order of 10~20 seconds)

The lunr_index.js file is now 18MB and much quicker to download and load. Was it having duplicated info or did you apply some other optimization?

2. While the JS is loading the whole page is blank beside the logo and the header menu on the left, giving the false impression that the site is broken/empty

With the added responsiveness this is not perceived anymore.

3. If you are in a nested page, say /wiki/DNS/Netbox.html and search for something, say Cumin, the resulting links in the search results are broken because don't remove the nested path, pointing to say /wiki/DNS/Cumin.html instead of /wiki/Cumin.html and all return 404s

This is not happening anymore, but for the above example the Cumin page is not anymore in the search results. I tried also Netbox and DNS without luck.

4. If a page has aliases it will result in the result page multiple times, pointing to the same page with different urls. For example if you search for ipmi the first 3 results are:
   • http://ec2-52-23-161-9.compute-1.amazonaws.com/wiki/IPMI-2.html: opens http://ec2-52-23-161-9.compute-1.amazonaws.com/wiki/Management_Interfaces with the text (Redirected from IPMI ) and if I reload the page it gives 404
   • http://ec2-52-23-161-9.compute-1.amazonaws.com/wiki/Ipmi.html opens http://ec2-52-23-161-9.compute-1.amazonaws.com/wiki/Management_Interfaces with the text (Redirected from Ipmi ) and if I reload the page it gives 404
   • http://ec2-52-23-161-9.compute-1.amazonaws.com/wiki/Management_Interfaces.html opens http://ec2-52-23-161-9.compute-1.amazonaws.com/wiki/Management_Interfaces.html without any redirection text

This seems resolved.

5. 404 pages don't have a template, it's the nginx default page

As mentioned above this was not fixed. I'll consider it a nice to have, surely not a hard blocker.

6. The search box doesn't search when pressing enter, you have to click the button

This now works.

I noticed than when searching the search results are shown above the current page. Is there a way to "close them" and go back to the current page without refreshing?
Another small thing, trying to exit I got the page totally unresponsive (cannot select text, click, esc, etc...), after a bit got an nginx 400 bad request page.

Regarding item 1 and 3, I think I my check for detecting redirect pages was too broad and excluded other useful pages. I'm rebuilding now with a different rule, https://gitlab.wikimedia.org/repos/sre/wikitech-static-docker/-/commit/0f509ad6c96c608319229044400b6ce1538f1225

Ack, I can confirm the pages I was having trouble with are now found in search (at the cost of a larger index, I think is around 37MB now). Thanks for the fixes.

Hi @RobH @Andrew , we have Meta Monitoring enabled in the Wikitech static Rackspace host. Could you please provide the o11y team with access to the new host so we can enable meta-monitoring for Icinga on it? I created T393625 to track this. Thank you!

So I actually have no login rights (and don't need them) for the new AWS hosted wikitech static deployment. I just pay the AWS bill, Andrew will have to help you with this, sorry!

The site at http://ec2-54-81-201-239.compute-1.amazonaws.com/ seems to embed images from upload.wikimedia.org, for pages like network diagrams those should also be hosted off-cluster.

In T376400#10825452, @taavi wrote:

The site at http://ec2-54-81-201-239.compute-1.amazonaws.com/ seems to embed images from upload.wikimedia.org, for pages like network diagrams those should also be hosted off-cluster.

Can you point me to some specific examples? My half-baked spot checks (e.g. http://ec2-54-81-201-239.compute-1.amazonaws.com/wiki/Eqiad_data_center.html#/media/File:Eqiad_logical.png) seem to be hosted locally (or I am misunderstanding something fundamental).

In T376400#10834856, @Andrew wrote:

Can you point me to some specific examples? My half-baked spot checks (e.g. http://ec2-54-81-201-239.compute-1.amazonaws.com/wiki/Eqiad_data_center.html#/media/File:Eqiad_logical.png) seem to be hosted locally (or I am misunderstanding something fundamental).

Sure, for example the first image on http://ec2-54-81-201-239.compute-1.amazonaws.com/wiki/Network_design.html uses src="http://upload.wikimedia.org/wikipedia/labs/thumb/5/5f/Wikimedia_network_overview.png/960px-Wikimedia_network_overview.png" and src="http://upload.wikimedia.org/wikipedia/labs/thumb/5/5f/Wikimedia_network_overview.png/1280px-Wikimedia_network_overview.png" when you click of it.

yep, I see it now.

I am pasting comments I have made on a Slack thread:

We had Extension:DumpHTML which was once created specially for that purpose but that has long been archived. I think that was to provide HTML dumps on https://dumps.wikimedia.org/

Anyway, labswiki has XML dumps generated on https://dumps.wikimedia.org/labswiki/ and you can render them using maintenance/RenderDump.php. Some care would be needed to retrieve links to images and have them stored as well.

In T376400#10834875, @taavi wrote:

In T376400#10834856, @Andrew wrote:

Can you point me to some specific examples? My half-baked spot checks (e.g. http://ec2-54-81-201-239.compute-1.amazonaws.com/wiki/Eqiad_data_center.html#/media/File:Eqiad_logical.png) seem to be hosted locally (or I am misunderstanding something fundamental).

Sure, for example the first image on http://ec2-54-81-201-239.compute-1.amazonaws.com/wiki/Network_design.html uses src="http://upload.wikimedia.org/wikipedia/labs/thumb/5/5f/Wikimedia_network_overview.png/960px-Wikimedia_network_overview.png" and src="http://upload.wikimedia.org/wikipedia/labs/thumb/5/5f/Wikimedia_network_overview.png/1280px-Wikimedia_network_overview.png" when you click of it.

These images are now locally hosted. There's something a bit busted with the image viewer but I'm not sure I want to go down that rabbit hole.

There are still a /few/ images that don't get scraped properly, but when I include commons in the download scope the image becomes unmanageably large so I'm inclined to declare victory.

Change #1218333 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/dns@master] wikitech-static: associate with an elastic IP stored in AWS

https://gerrit.wikimedia.org/r/1218333

Change #1218344 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] icinga: remove wikitech-static mediawiki version check

https://gerrit.wikimedia.org/r/1218344

Change #1218344 merged by Andrew Bogott:

[operations/puppet@production] icinga: remove wikitech-static mediawiki version check

https://gerrit.wikimedia.org/r/1218344

Change #1223243 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/dns@master] Add temporary record for wikitech-static-dev

https://gerrit.wikimedia.org/r/1223243

Change #1223243 merged by Andrew Bogott:

[operations/dns@master] Add temporary record for wikitech-static-dev

https://gerrit.wikimedia.org/r/1223243

Change #1218333 merged by Andrew Bogott:

[operations/dns@master] wikitech-static: associate with an elastic IP stored in AWS

https://gerrit.wikimedia.org/r/1218333

In T376400#11345384, @Andrew wrote:

Sure, for example the first image on http://ec2-54-81-201-239.compute-1.amazonaws.com/wiki/Network_design.html uses src="http://upload.wikimedia.org/wikipedia/labs/thumb/5/5f/Wikimedia_network_overview.png/960px-Wikimedia_network_overview.png" and src="http://upload.wikimedia.org/wikipedia/labs/thumb/5/5f/Wikimedia_network_overview.png/1280px-Wikimedia_network_overview.png" when you click of it.

These images are now locally hosted. There's something a bit busted with the image viewer but I'm not sure I want to go down that rabbit hole.

This seems to have regressed, unfortunately. https://wikitech-static.wikimedia.org/wiki/Network_design.html still loads all of its images from upload.wm.o, even the ones that have been uploaded locally on Wikitech (instead of on Commons). (This is particularly visible if you add a CSP blocking external media, live I've done for my mirror.)

I'm pretty sure I have images getting rolled into the static site now. Going to wait a few days for @taavi to find another issue and then I'll close this if he doesn't.