Change 200725 had a related patch set uploaded (by Dzahn):
dumps: add classes for zim dumps, add imagemagick

https://gerrit.wikimedia.org/r/200725

For ZIM dumps, we'll additionally need other tools like imagemagick

how to get a list of the other tools besides imagemagick?

@Dzahn, the full list is at https://wikitech.wikimedia.org/wiki/Nova_Resource_Talk:Mwoffliner#Ubuntu_basic_setup. It's quite scary, and I'm not convinced that we should install & execute all those tools directly on bare metal.

In any case, the precondition for efficient ZIM dumps is HTML dumps. For those, we need:

• nodejs
• nodejs-legacy
• nodejs-dev
• npm
• libsqlite3
• libsqlite3-dev

I figure we want to avoid npm being another package manager that would install more software via another channel.

You are right that we don't need it for regular deploys (as dependencies are compiled elsewhere). However, we also use npm start to start services and npm test to run tests. During debugging it can also be useful to temporarily install debug tools like the heapdump module, which we don't normally include in our dependencies.

In T94457#1165509, @GWicke wrote:

It's quite scary, and I'm not convinced that we should install & execute all those tools directly on bare metal.

whether we install it on bare metal or a virtual machine or in a container would not make a difference for the puppet manifest though

Change 200790 had a related patch set uploaded (by Dzahn):
ZIM dumps: install nodejs and libsqlite

https://gerrit.wikimedia.org/r/200790

In T94457#1165579, @Dzahn wrote:

In T94457#1165509, @GWicke wrote:

It's quite scary, and I'm not convinced that we should install & execute all those tools directly on bare metal.

whether we install it on bare metal or a virtual machine or in a container would not make a difference for the puppet manifest though

That depends on whether we lock things down with a dedicated container image or not. If the software is in a read-only image, then we don't need to install it in the host.

Can we please avoid npm, and -dev packages on the host please? There is no reason to increase the attack surface by adding dev tools and another package manager that downloads unsigned, unverified software off the internet on a production machine. As for npm used to run npm start, it bypasses the standard approach of starting services (upstart/systemd/init) and I would rather we avoided it.

Obviously nodejs and libsqlite3 I am fine with :-)

FWIW, all hosts that are running node services have npm installed, and it has been useful on occasion to manually install a debug library like heapdump to debug memory leaks. Also, to repeat myself, npm is never used to install dependencies during normal deploys. Even if tried, that wouldn't work without messing with proxy env vars etc.

Services normally run under a user that does not have write access to its own code, but does have write access to /tmp. I am all in favor of locking this down further (containers, apparmor), as most stateless services really don't need any capabilities or write access to the filesystem.

As for npm used to run npm start, it bypasses the standard approach of starting services (upstart/systemd/init) and I would rather we avoided it.

Not at all. Either way, it's the normal init system calling 'npm start' or 'node somefile.js'.

In T94457#1167094, @GWicke wrote:

has been useful on occasion to manually install a debug library

I think "manually install" is the key part here we want to avoid in production.

I think "manually install" is the key part here we want to avoid in production.

The choices are:

1. Always compile & install debug libraries by requiring them as a dependency. I used to do this for Parsoid, precisely to avoid the need to install things manually in production later. Main problem is that this increases the number of (binary) dependencies, and low-level libraries like a heap debugger often fail for specific node versions. This is a problem especially for third-party users.

2. Don't install those debugging libs by default, and install them manually in the rare case they are needed. This is similar to the way we deal with things like strace etc in production, which afaik also aren't installed on each host by default. The installation of this library is temporary, and will be wiped out with the next install. So far this has worked well in practice, and is thus what we do with Parsoid, RESTBase and others.

In T94457#1165509, @GWicke wrote:

That depends on whether we lock things down with a dedicated container image or not. If the software is in a read-only image, then we don't need to install it in the host.

That still doesn't change the code of the puppet manifest we are applying!?

That still doesn't change the code of the puppet manifest we are applying!?

In many container scenarios there is no puppet *inside* the container. It can configure the container from the outside though. My point is that no software needs to be installed on the host system if that is handled by a container image.

In T94457#1167129, @GWicke wrote:

This is similar to the way we deal with things like strace etc in production, which afaik also aren't installed on each host by default.

I think they are. I just checked on a freshly installed poolcounter host in codfw that hasn't been touched and it does have strace installed.

In T94457#1167146, @GWicke wrote:

In many container scenarios there is no puppet *inside* the container.

In case the assumption really is we don't use puppet to install software anymore that is completely new to me and i think it needs more than just this ticket to discuss that then. My intention was just helping out with the puppet manifest.

Unless you're suggesting to hold-off this work until we have some sort of ideal-in-your-mind setup with containers in some undetermined amount of time, I'd suggest to not plaster containers (or similar) ideas all over random tickets but work with what we currently have instead. This should help moving a bit quicker and avoid wasting everyone's time.

Unless I've completely misunderstood what is this about, installing a dozen packages or so on bare metal should be totally fine.

I think they are. I just checked on a freshly installed poolcounter host in codfw that hasn't been touched and it does have strace installed.

It isn't installed on the restbase hosts, for example. I also don't see it mentioned in puppet.

Unless I've completely misunderstood what is this about, installing a dozen packages or so on bare metal should be totally fine.

Installing might be fine, but I'm not keen on running the mwoffliner scripts with all its shell-outs without proper security auditing and lock-down.

Change 200725 merged by Dzahn:
dumps: add classes for zim dumps, add imagemagick

https://gerrit.wikimedia.org/r/200725

Change 200790 merged by Dzahn:
ZIM dumps: install nodejs and libsqlite

https://gerrit.wikimedia.org/r/200790

As a step forward i created class role::dumps::zim and modules/dumps/manifests/zim.pp which install nodejs, nodejs-legacy, libsqlite3 and imagemagick, to start with. so this is merged in the puppet repo just not applied on a node yet.

This way it could be applied on labs instances via the wikitech UI and tested, amended to, then when like it enough we can just apply it on production. How's that?

https://gerrit.wikimedia.org/r/#/c/204257/ nginx setup for the html dumps producing host, which will temporarily be serving its dumps as well; proxy service added to dumps.wm.org. If we like this, we could enable it on francium and put some dumps over there to get service started.

Change 204617 had a related patch set uploaded (by Dzahn):
site.pp: add node francium.eqiad.wmnet

https://gerrit.wikimedia.org/r/204617

Change 204617 merged by Dzahn:
site.pp: add node francium.eqiad.wmnet

https://gerrit.wikimedia.org/r/204617

Change 204618 had a related patch set uploaded (by Dzahn):
dumps::zim: add role and firewall to francium

https://gerrit.wikimedia.org/r/204618

Change 204618 merged by Dzahn:
dumps::zim: add role and firewall to francium

https://gerrit.wikimedia.org/r/204618

In T94457#1209183, @ArielGlenn wrote:

https://gerrit.wikimedia.org/r/#/c/204257/ nginx setup for the html dumps producing host, which will temporarily be serving its dumps as well; proxy service added to dumps.wm.org. If we like this, we could enable it on francium and put some dumps over there to get service started.

that has been merged. the proxy setup is on dumps1001/ms1001.

[francium:~] $ dpkg -l | grep nodejs
ii  nodejs                              0.10.25~dfsg2-2ubuntu1           amd64        evented I/O for V8 javascript
ii  nodejs-legacy                       0.10.25~dfsg2-2ubuntu1           all          evented I/O for V8 javascript (legacy symlink)
[francium:~] $ dpkg -l | grep nginx
ii  nginx-common                        1.4.6-1ubuntu3.2                 all          small, powerful, scalable web/proxy server - common files
ii  nginx-extras                        1.4.6-1ubuntu3.2                 amd64        nginx web/proxy server (extended version)

nodejs and nginx installed by puppet now

Change 204625 had a related patch set uploaded (by Dzahn):
dumps::zim: libsqlite3 is actually libsqlite3-0

https://gerrit.wikimedia.org/r/204625

Change 204625 merged by Dzahn:
dumps::zim: libsqlite3 is actually libsqlite3-0

https://gerrit.wikimedia.org/r/204625

ii  libsqlite3-0:amd64                  3.8.2-1ubuntu2                   amd64        SQLite 3 shared library

^ now installed.

next todo is fix nginx setup:

Error: /Stage[main]/Dumps::Zim/Nginx::Site[zim]/File[/etc/nginx/sites-available/zim]: Could not evaluate: Could not retrieve information from environment production source(s) puppet:///modules/dumps/nginx.zim.conf
Notice: /Stage[main]/Dumps::Zim/Nginx::Site[zim]/File[/etc/nginx/sites-enabled/zim]: Dependency File[/etc/nginx/sites-available/zim] has failures: true
Warning: /Stage[main]/Dumps::Zim/Nginx::Site[zim]/File[/etc/nginx/sites-enabled/zim]: Skipping because of failed dependencies
Notice: /Stage[main]/Nginx/Service[nginx]: Dependency File[/etc/nginx/sites-available/zim] has failures: true
Warning: /Stage[main]/Nginx/Service[nginx]: Skipping because of failed dependencies

so to summarize:

original request was "nodejs, nodejs-dev, npm, libsqlite3 and libsqlite3-dev .. additionally other tools like imagemagick"

done: nodejs, libsqlite3, imagemagick

not done so far: nodejs-dev, libsqlite3-dev, npm (because of gerrit comments )

also, re: "useful to set up an nginx instance serving a data directory, which is then proxied to via the download servers"

the data directory exists (/srv/www/htmldumps), the proxy config on the download servers exists, what must be done is fix the puppet run on francium, see comment above.

Change 204636 had a related patch set uploaded (by Dzahn):
dumps::zim: fix nginx setup / basic site template

https://gerrit.wikimedia.org/r/204636

Change 204636 merged by Dzahn:
dumps::zim: fix nginx setup / basic site template

https://gerrit.wikimedia.org/r/204636

Change 204640 had a related patch set uploaded (by Dzahn):
dumps::zim: fix template source line

https://gerrit.wikimedia.org/r/204640

Change 204640 merged by Dzahn:
dumps::zim: fix template source line

https://gerrit.wikimedia.org/r/204640

nginx now running:

Info: /Stage[main]/Dumps::Zim/Nginx::Site[zim]/File[/etc/nginx/sites-enabled/zim]: Scheduling refresh of Service[nginx]
Info: Nginx::Site[zim]: Scheduling refresh of Service[nginx]
Notice: /Stage[main]/Nginx/Service[nginx]: Triggered 'refresh' from 3 events
Notice: Finished catalog run in 11.31 seconds

[francium:/srv/www/htmldumps] $ curl localhost
<html>
<head><title>Index of /</title></head>

Change 204649 had a related patch set uploaded (by Dzahn):
dumps::zim: open port 80 for http connections

https://gerrit.wikimedia.org/r/204649

Change 204649 merged by Dzahn:
dumps::zim: open port 80 for http connections

https://gerrit.wikimedia.org/r/204649

firewall opened on 80 on francium.

current error is it refuses connection from dataset1001:

[error] 24810#0: *88980 connect() failed (111: Connection refused) while connecting to upstream, client: ::ffff:198.73.209.4, server: dumps.wikimedia.org, request: "GET /htmldumps HTTP/1.1", upstream: "http://10.64.32.168:80/htmldumps", host: "dumps.wikimedia.org"

Change 204661 had a related patch set uploaded (by Dzahn):
dumps::zim: fix nginx listening port and docroot

https://gerrit.wikimedia.org/r/204661

Change 204661 merged by Dzahn:
dumps::zim: fix nginx listening port and docroot

https://gerrit.wikimedia.org/r/204661

nginx proxy setup working now. see patches above.

added placeholder page:

http://dumps.wikimedia.org/htmldumps/

^ this is now served by francium proxied via dataset1001 as requested

I tweaked the puppet manifest for nginx on francium a little, no big deal.

GWicke: I am ready to put some dumps from the labs instance on francium to be served if you let me know what directory structure you have in mind.

@ArielGlenn, as described in https://phabricator.wikimedia.org/T94093#1214628 the dump-all-wikis script currently expects one working directory and one dump directory for compressed dumps. It decompresses an existing dump (if available in the dump directory) to the working directory, updates the database incrementally and then recompresses it before moving it to the dump directory.

I have a dump running on the external VM since last night. So far it has created 55 dumps, but with the big ones still missing. Running this on francium instead would speed things up.

sqlite3 and pixz have been added by

https://gerrit.wikimedia.org/r/#/c/204684/

Marko and Eric reviewed the code at https://github.com/wikimedia/htmldumper/pull/3. It is also pushed to https://gerrit.wikimedia.org/r/#/admin/projects/operations/dumps/html, in the 'incoming' branch (and now also master). I did not find a way to create a gerrit merge changeset against an empty master branch that would allow review of the entire history so far in gerrit. Gerrit kept rejecting my push / git review attempts.

Change 204964 had a related patch set uploaded (by GWicke):
htmldumper 0.1.0 with dependencies

https://gerrit.wikimedia.org/r/204964

And i guess there's a reason why we should work on our internal tools rather than github.

'work on', yeah ;)

I've made some comments on the gerrit changeset.

are there any updates on making HTML dumps generally available?

Change 204964 abandoned by GWicke:
htmldumper 0.1.0 with dependencies

Reason:
Out of date with https://github.com/wikimedia/htmldumper, so abandoning.

We would still like to set up HTML dumps, but would need a serious commitment from operations to make progress.

https://gerrit.wikimedia.org/r/204964

@ArielGlenn: Hi, all related patches in Gerrit have been merged or abandoned. Do you still plan to work on this task / is there still more to do here? Asking as you are set as task assignee. Thanks.

This should be closed as no longer relevant at this point. What status is that? :-)