I have rebuild the job which happened to run on 1002. It failed with the exact same reason. So that does not seem to be due to differences between slaves :-(

Well done @Ladsgroup

In T171562#3469892, @Mholloway wrote:

@hashar , has any configuration around these machines (integration-slave-jessie-1001, integration-slave-jessie-1002) changed in the past 24 hours? I can't think of any change on our end that could be causing this.

@mpopov using twitter to get the size was a smart move :-]

Thank you @thcipriani for the analysis and the patch!

Could it be that /tmp is a tmpfs and thus moving files under / is actually a copy+delete?

The fix is https://gerrit.wikimedia.org/r/#/c/366496/ by @Legoktm

They all have been edited:

$ mwgrep 'wikipedia/en/0/00/Lock_icon_blue.gif'

@Krinkle regarding the bad slave being connected, that is entirely my fault. I created the new slave in Jenkins by copying saucelabs02. And apparently Jenkins connected to that host and kept that host ssh key. That sounds like a bug in Jenkins.

In logstash, the last trace matching idSerialization is 2017-07-20T16:56:19 and had:

/w/api.php?action=wbcheckconstraints&format=json&uselang=en&id=P3212   InvalidArgumentException from line 39 of /srv/mediawiki/php-1.30.0-wmf.9/extensions/Wikidata/vendor/wikibase/data-model/src/Entity/ItemId.php: $idSerialization must match /^Q[1-9]\d{0,9}\z/i

I will visit this task later on this week. Came here to mention I have hit a similar wall with thumbnailBeforeProduceHTML hook which change the output expected by core or other extensions. T69302

@Joe proposed a rewriting of the Puppet Rakefile as part of T166888 Patch is https://gerrit.wikimedia.org/r/#/c/366591/ and implements the logic described in previous comment.

@Krinkle definitely. Will probably want to limit it to the master branch for a while.

OpenStack infrastructure has a python based utility to generate Grafana board based on a YAML DSL. That is similar to their Jenkins Job Builder used to generate jobs.

I have removed faulty puppet classes, ran puppet, restarted nslcd and reapplied the puppet classes

I have marked Cards read-only in Gerrit

Potentially you could reuse the QA list, it is very low traffic nowadays https://lists.wikimedia.org/pipermail/qa/ :]

The detection is based on a regular expression in integration/config zuul/layout.yaml:

So that is partly resolved thanks to @Mholloway . He made patches that made it possible to enable QEMU2 and thus let the emulator use multiple CPU. Thus the run time went from 45 minutes to 11 minutes, largely because retrieving screenshots is way faster.

status update

Note after some magic happened, the job runs in ~ 11 minutes (was 45 minutes) \o/ ( T150623 )

Definitely yes. Looks like I have messed it up. Paladox did submit the same change https://gerrit.wikimedia.org/r/#/c/337225/ which I have abandoned, but I forgot to abandon mine.

Yup because I have added novaadmin as a member of the deployment-prep tenant. But for tools it is still empty:

$ curl 'https://wikitech.wikimedia.org/w/api.php?action=query&list=novainstances&niregion=eqiad&format=json&niproject=deployment-prep' 
{"batchcomplete":"","query":{"novainstances":[]}}

The Icinga alert should probably be more noisy. Left to figure out is whether novaadmin should actually be a member.

I created the repo in Gerrit with the wrong name, thus I have marked the repo hidden and recreated it as wikibase/wikiba.se.git

I have created the repo and imported all references from github (including pull requests refs, though they are not recognized as changes by gerrit):

And in the nova logs, I also see 401 for the tools project for requests from Silver

Code is in api/ApiListNovaInstances.php. Replaying it on silver:

$ mwscript eval.php --wiki=labswiki
> global $wgOpenStackManagerLDAPUsername;
> global $wgOpenStackManagerLDAPUserPassword;
> $user = new OpenStackNovaUser( $wgOpenStackManagerLDAPUsername );

There is already a /wikibase project, so maybe /wikibase/www or /wikibase/wikibase.se It is all up to you :]

Thank you @Samtar I guess I was assuming that lot of bot operators already have global edit interface that is why I suggested reached out to them to automatize the edition. But your idea is way easier and as you said, it is "only" 66 edits :-}

@Bawolff sorry I failed to notice the fix made it to master age ago and effectively made it to REL1_29 when we branched. Thanks :-}

I filled this task in the hope someone could figure out the links to be added in Gdash. Nowadays that can be done via Grafana hence I reopened this task and rephrased the topic.

Andrew has deleted the instance and created a new one. The instance has been created when LDAP/Certs etc were screwed out that certainly explains the issue.

I did remove profile::recommendation_api on deployment-sca01 earlier but was hitting another puppet issue. That got fixed meanwhile.

Announced on the QA list pointing back to this task

So the state as I understand it right now:

https://wikitech.wikimedia.org/wiki/Incident_documentation/20170719-ldap#CI.2Fbeta

https://wikitech.wikimedia.org/wiki/Incident_documentation/20170719-ldap#CI.2Fbeta

https://wikitech.wikimedia.org/wiki/Incident_documentation/20170719-ldap#CI.2Fbeta

Seems the initial puppet run refuses to process for whatever reason. The instance is not even in salt so we can't access it at all. Trying to ssh into it yields Password:

I have added profile::recommendation_api back on deployment-sca01.

On deployment-sca01 I have removed profile::recommendation_api puppet then fails with:

Error: Failed to apply catalog:
  Could not find dependent Service[eventlogging/init]
  for File[/usr/local/lib/eventlogging/filters.py]
  at /etc/puppet/modules/eventlogging/manifests/plugin.pp:49?[0m

deployment-trending01.deployment-prep.eqiad.wmflabs has a similar issue:

(Exec[trendingedits config deploy] => Service::Node::Config::Scap3[trendingedits] => Scap::Target[trending-edits/deploy] => User[deploy-service] => Exec[trendingedits config deploy])

Beta cluster instances have the exact same issue. Filled as T171174

I can confirm that resolved the issue completely. Thank you!

Seems the nova database is on m5-master.eqiad.wmnet db name nova.

The Nodepool launch errors https://grafana.wikimedia.org/dashboard/db/nodepool?panelId=12&fullscreen&orgId=1&from=now-20h&to=now

That is happening again after something got restarted yesterday. Filled as T171158

labnet1001.eqiad.wmnet has a lot of such errors in /var/log/nova/nova-network.log*

I have updated the workaround using the one I originally wrote on T148929. The proposed one did not work for me on CI instances with a self puppet master.

I have removed integration-slave-docker-1000 since puppet is completely broken on it.

I have manually repopulated the cache for operations/puppet.git by triggering https://integration.wikimedia.org/ci/job/operations-puppet-cache-update-jessie/

I have booted a Jessie instance with the latest labs image and it comes with puppet 3.8.5:

apt-cache policy puppet
puppet:
  Installed: 3.8.5-2~bpo8+2
  Candidate: 3.8.5-2~bpo8+2
  Version table:
     4.8.2-5~bpo8+1 0
        100 http://mirrors.wikimedia.org/debian/ jessie-backports/main amd64 Packages
 *** 3.8.5-2~bpo8+2 0
       1001 http://apt.wikimedia.org/wikimedia/ jessie-wikimedia/backports amd64 Packages
        100 /var/lib/dpkg/status
     3.7.2-4+deb8u1 0
        500 http://security.debian.org/ jessie/updates/main amd64 Packages
     3.7.2-4 0
        500 http://httpredir.debian.org/debian/ jessie/main amd64 Packages

From the console log, puppet-agent on boot reports:

SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: virt1000.wikimedia.org]

gdb wait-for-devices complained with:

Your emulator is out of date, please update by launching Android Studio:
 - Start Android Studio
 - Select menu "Tools > Android > SDK Manager"
 - Click "SDK Tools" tab
 - Check "Android SDK Tools" checkbox
 - Click "OK"

I have updated the list. There are 66 entries.