With help from SRE people (specially Moritz). I have used spectre-meltdown-checker 0.42 from Stretch and ran it on both instances:

$ diff -U0 jessie stretch
--- jessie	2019-10-30 11:35:43.043507046 +0100
+++ stretch	2019-10-30 11:35:37.411478275 +0100
@@ -4 +4 @@
-Kernel is Linux 4.9.0-0.bpo.11-amd64 #1 SMP Debian 4.9.189-3+deb9u1~deb8u1 (2019-09-30) x86_64
+Kernel is Linux 4.9.0-11-amd64 #1 SMP Debian 4.9.189-3+deb9u1 (2019-09-20) x86_64
@@ -87 +87 @@
-* SSB mitigation currently active for selected processes: NO (no process found using SSB mitigation through prctl)
+* SSB mitigation currently active for selected processes: YES (systemd-journald systemd-logind systemd-timesyncd systemd-udevd)

Both report kernel 4.9.189-3+deb9u1. But Stretch has SSB mitigation enabled on systemd process.

TLDR: The slowdown on Stretch goes out when setting spec_store_bypass_disable=off

Which is reported by spectre-meltdown-checker as:

CVE-2018-3639 aka 'Variant 4, speculative store bypass'
* Mitigated according to the /sys interface:  NO  (Vulnerable)
* Kernel supports disabling speculative store bypass (SSB):  YES  (found in /proc/self/status)
* SSB mitigation is enabled and active:  NO 
> STATUS:  VULNERABLE  (your CPU and kernel both support SSBD but the mitigation is not active)

No idea why jessie is not affected by the slow down.

The kernel boot option mitigations=off sets several flags.

mitigations=off
        Disable all optional CPU mitigations.  This
        improves system performance, but it may also
        expose users to several CPU vulnerabilities.
        Equivalent to: nopti [X86]
                       nospectre_v1 [X86]
                       nospectre_v2 [X86]
                       spectre_v2_user=off [X86]
                       spec_store_bypass_disable=off [X86]
                       l1tf=off [X86]
                       mds=off [X86]

I went setting them each one by one in /etc/default/grub , then update-grub and reboot. Ran test, rinse repeat.

Done on integration-agent-docker-1005 with kernel 4.9.189-3+deb9u1. Result:

Moritz are you aware of any difference between Jessie and Stretch kernels?

On Jessie spec_store_bypass_disable on or off does not change anything.
On Stretch spec_store_bypass_disable=off makes it way faster.

There's no difference, the 4.9.189-3+deb9u1 and 4.9.189-3+deb9u1~deb8u1 are identical feature-wise.

That is what I have guessed looking at the source and /boot/config* files :-\

I just can not find why spec_store_bypass_disable=off barely makes a difference on Jessie while it has a large impact on Stretch.

Could it be related to the gcc version used for the kernel? Reviewing the Debian changelog, I noticed the Jessie one is not compiled with gcc 6 since it is not in Jessie and thus uses gcc 4.9

From /proc/version

(4.19 kernel on Stretch is affected as well)

Obviously the jessie kernel is built with the jessie GCC. but there's no toolchain change which would explain the difference. You can try install the jessie kernel on stretch and vice-versa for further tests, though.

Thanks!

So on a Stretch instance I grabbed the Jessie kernel and installed it:

wget http://security.debian.org/debian-security/pool/updates/main/l/linux-4.9/linux-image-4.9.0-0.bpo.11-amd64_4.9.189-3+deb9u1~deb8u1_amd64.deb
dpkg -i linux-image-4.9.0-0.bpo.11-amd64_4.9.189-3+deb9u1~deb8u1_amd64.deb

Looked at menuentry in /boot/grub/grub.cfg to figure out which one to reboot on. Ended up with 4 so:

sudo grub-reboot '1>4'
cat|sudo reboot

stretch$ uname -a
Linux integration-agent-docker-1005 4.9.0-0.bpo.11-amd64 #1 SMP Debian 4.9.189-3+deb9u1~deb8u1 (2019-09-30) x86_64 GNU/Linux

stretch$ cat /proc/version 
Linux version 4.9.0-0.bpo.11-amd64 (debian-kernel@lists.debian.org) (gcc version 4.9.2 (Debian 4.9.2-10+deb8u2) ) #1 SMP Debian 4.9.189-3+deb9u1~deb8u1 (2019-09-30)

So I have a gcc4.9. compiled kernel on Stretch \o/

sudo spectre-meltdown-checker states everything is fine.

Ran my test again and it still takes 1m8s

Passed spec_store_bypass_disable=off and sudo grub-reboot '1>4 && reboot. Confirmed the kernel uses 4.9 and spectre-meltdown-checker reports an issue.

Ran test again and it is down to 57s.

The Jessie kernel does not change anything compared to the Stretch ones (4.9/4.19). They all show slowness with spec_store_bypass_disable=on which disappear when set to off.

I guess that rules out the kernel / gcc version. The issue must be somewhere up in the stack bah :-\

It is not the kernel, but Docker.

Somehow spec_store_bypass_disable=on and 18.09.7~3-0~debian-stretch do not play well.

Made it public, there is nothing secret.