Per our conversation during Standup (and @bmansurov's review of this task), I'm moving this onto the kanban board so that we can unblock the conversation on the parent task.

I'm gonna tackle this while waiting for review on T178501: Limit resources used by Chromium in order to make the chromium-render service ready for production from Piotr.

The puppeteer version we're using is 0.11 and the chromium version that comes with it is:

./node_modules/puppeteer/.local-chromium/linux-499413/chrome-linux/chrome --version
Chromium 63.0.3205.0

I've started a new Debian Stretch instance, but I canonot ssh to it. Inspecting the logs, I see the following:

Failed to start Puppet agent.
2017-11-09T23:43:41.979717+00:00 chromium-render-t180037 systemd[1]: puppet.service: Unit entered failed state.
2017-11-09T23:43:41.979883+00:00 chromium-render-t180037 systemd[1]: puppet.service: Failed with result 'timeout'.

@bd808 Would you be able to help us resolve the issue with puppet on the chromium-render-T180037 instance of the reading-web-staging project? Is it a known issue with Debian Stretch?

In T180037#3749597, @bmansurov wrote:

I've started a new Debian Stretch instance, but I canonot ssh to it. Inspecting the logs, I see the following:

Failed to start Puppet agent.
2017-11-09T23:43:41.979717+00:00 chromium-render-t180037 systemd[1]: puppet.service: Unit entered failed state.
2017-11-09T23:43:41.979883+00:00 chromium-render-t180037 systemd[1]: puppet.service: Failed with result 'timeout'.

@bd808 Would you be able to help us resolve the issue with puppet on the chromium-render-T180037 instance of the reading-web-staging project? Is it a known issue with Debian Stretch?

The puppet.service failing to start is actually expected, but if the log stops there that is not expected.

I am unable to login to the instance as well using my Cloud VPS root key. It seems to have died in the "firstboot" process which sets up the instance for basic access via ssh. This happens occasionally due to some race conditions that are not well understood in the process and/or small hiccups in our infrastructure that show up at just the wrong time. The "fix" is to delete the instance and create another one. It is recommended to also make sure that the name of the new instance is unique. There are some other bugs that can crop up with recycled instance names.

I did check to see if this was a larger systemic problem, but I was able to create a new instance using the debian stretch base image with no problems. I think you just had bad luck with this one.

The puppet.service failing to start is actually expected, but if the log stops there that is not expected.

I think the logs went on after that.

It is recommended to also make sure that the name of the new instance is unique.

I suppose you mean, unique within the project?

It was actually my second stretch instance that was having the problem. The one before it had the same issues, and I deleted it to create this one. I've also deleted this instance and created a new one, but I'm still getting the same issue. The instance name is 'chromium-render' this time.

Mentioned in SAL (#wikimedia-cloud) [2017-11-13T17:03:55Z] <bd808> Joined project as admin user to help debug T180037

Mentioned in SAL (#wikimedia-cloud) [2017-11-13T17:11:57Z] <bd808> Removed role::labs::mediawiki_vagrant from project global puppet config in horizon (T180037)

Mentioned in SAL (#wikimedia-cloud) [2017-11-13T17:18:11Z] <bd808> Attempting to create chromium-render-stretch.reading-web-staging.eqiad.wmflabs instance (T180037)

I got a node running Debian Stretch up and running in the project.

$ ssh chromium-render-stretch.reading-web-staging.eqiad.wmflabs
Linux chromium-render-stretch 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u5 (2017-09-19) x86_64
Debian GNU/Linux 9.1 (stretch)
The last Puppet run was at Mon Nov 13 17:20:16 UTC 2017 (2 minutes ago).
Last login: Mon Nov 13 17:22:03 2017 from 10.68.17.232
bd808@chromium-render-stretch:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 9.1 (stretch)
Release:        9.1
Codename:       stretch
bd808@chromium-render-stretch:~$

The only change I made was T180037#3755631 where I removed the role::labs::mediawiki_vagrant Puppet setting from project global Puppet config for the reading-web-staging project via the interface at https://horizon.wikimedia.org/. My reason for doing this was to remove any extra potential Puppet configuration that could be causing your multiple initial provisioning failures. Removing this global role will not remove existing MediaWiki-Vagrant packages that are on other instances, but it will prevent MediaWiki-Vagrant from being automatically installed on this and future instances in your project. It would be a good idea to go through the existing instances and enable role::labs::mediawiki_vagrant directly on the nodes where you are actively using it.

I will open a separate ticket to investigate if there are changes needed in role::labs::mediawiki_vagrant to support Debian Stretch as a host platform.

Thanks! I see the 'reading-web-staging-3' instance already has the mediawiki_vagrant role. Other instance don't need it, I think.

Steps to re-produce

• Updated debian: sudo apt-get update;
• Installed chromium: sudo apt-get install chromium;
• Checked the chromium version: chromium --version, which resulted in Chromium 62.0.3202.89 built on Debian 9.1, running on Debian 9.1;
• Set up the nodesource repo: curl -sL https://deb.nodesource.com/setup_6.x | sudo -E bash -;
• Created a file called /etc/apt/preferences.d$ cat nodesource.pref with the following content:

Package: *
Pin: origin deb.nodesource.com
Pin-Priority: 1002

• Installed nodejs: sudo apt-get install nodejs;
• Checked the nodejs version: nodejs --version, which resulted in v6.11.0;
• Checked the npm version: npm --version, which resulted in 3.10.10;
• Installed dependencies: sudo apt-get install gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget;
• Cloned the repo: cd /srv && sudo git clone https://gerrit.wikimedia.org/r/mediawiki/services/chromium-render;
• Fixed permissions: sudo chown -R bmansurov:users chromium-render/;
• Installed npm dependencies: cd /srv/chromium-render && PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=1 npm install;
• Made sure puppeteer didn't download its own Chromium:

$ ls /srv/chromium-render/node_modules/puppeteer/.local-chromium
ls: cannot access 'node_modules/puppeteer/.local-chromium': No such file or directory

• Edited renderer.js to use the Debian version of Chromium:

bmansurov@chromium-render-stretch:/srv/chromium-render$ git diff
diff --git a/lib/renderer.js b/lib/renderer.js
index dae34f1..3af3bbf 100644
--- a/lib/renderer.js
+++ b/lib/renderer.js
@@ -14,7 +14,7 @@ exports.articleToPdf = function articleToPdf(url, format, puppeteerFlags, pdfOpt
     let browser;
     let page;
 
-    return puppeteer.launch({ args: puppeteerFlags })
+    return puppeteer.launch({executablePath: '/usr/bin/chromium', args: puppeteerFlags })
         .then((browser_) => {
             browser = browser_;
             return browser.newPage();

• Ran the server locally: cd /srv/chromium-render && node server.js
• Using another ssh connection, download a PDF: wget http://localhost:3030/en.wikipedia.org/v1/pdf/Book/letter -O book.pdf
• Here is the resulting book.pdf:

Conclusion

The current puppeteer version "^0.11.0" works with the version "62.0.3202.89" of Chromium that comes with Debian Stretch.

There's not much code, but a review of the approach would be appreciated.

missing chromium in sudo apt-get install;. We would have to lock the chromium to version 62.0.3202.89-1~deb9u1 as it might change in the future. Good work @bmansurov,

As a review I'll try to run this locally on virtualbox using stretch-mini-iso

The guide has couple minor issues like missing chromium when calling apt-get install chromium or missing git dependency, also while calling chown would be nice to pass -R so everything gets user as an owner.

Overall I was able to install chromium service locally on newly created Virtualbox instance using Debian stretch mini-iso, I'm also able to generate PDFs by sending a web request to the service. I'm calling this task as done and moving it to ready-for-signoff

I've added the missing 'chromium'. I don't think I installed git explicitly, it was already there and chown already has -R.

Thanks for the detailed notes, @bmansurov and @pmiazga.

Also, thank you @bd808 for your help with getting the Debian Stretch Labs instance set up :]

An example how to set up the chromium renderer using docker debian stretch image: https://hub.docker.com/r/polishdeveloper/chromium-render/~/dockerfile/ (following @bmansurov guide)

Hi all, thank you for the great research work on this!

We have one blocker for productionizing this work, which is that npm is not available in debian itself, and we can't really rely on downloading a script with curl (which btw just does the work of adding one apt list to the system and to use it to install node and npm, tangentially) and executing it in a shell.

There is some discussion of the options we have in T180524, if you're interested.

@Joe, won't we be using a deploy repo to push npm packages? We have a repo called chromium-render-deploy, that will contain all necessary packages, thus we won't need npm installed in Debian.

@bmansurov heh, I though about the building process, but of course locally you can just use a stretch container with the packages fetched from nodesource.com (this is only important if your dependencies use any sort of shared library, which is sometimes the case).

I'll update you once we have news, but it's not a blocker at all - I got derailed by thinking about how to base your container upon our own docker images :)

OK, thank you.

heh, I thought about the building process, but of course, locally you can just use a stretch container with the packages fetched from nodesource.com (this is only important if your dependencies use any sort of shared library, which is sometimes the case).

@Joe in case the standard service-runner based docker script will be used for building the deploy repo, not having NPM is not an issue - we use NVM to select the appropriate node version and NVM installs NPM appropriate for the node, so we don't need to install NPM from the apt repo. I've created https://github.com/wikimedia/service-runner/pull/181 to make this happen.