Page MenuHomePhabricator

String vs Binary issues while running the puppet compiler
Closed, ResolvedPublicSecurity

Description

Compiling 644002 on node(s) kafka-main2001.codfw.wmnet,logstash1010.eqiad.wmnet,kafka-jumbo1001.eqiad.wmnet,kafka-main1001.eqiad.wmnet...
Your build URL is https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/26756
Started by user elukey
Running as SYSTEM
Building remotely on compiler1002.puppet-diffs.eqiad.wmflabs (puppet-compiler-node) in workspace /srv/jenkins-workspace/workspace/operations-puppet-catalog-compiler
[operations-puppet-catalog-compiler] $ /bin/bash -xe /tmp/jenkins9379476020989353717.sh
+ NUM_THREADS=2
+ MODE=change
+ CHANGE=644002
+ NODES=kafka-main2001.codfw.wmnet,logstash1010.eqiad.wmnet,kafka-jumbo1001.eqiad.wmnet,kafka-main1001.eqiad.wmnet
+ puppet-compiler
[ 2020-11-30T09:45:56 ] INFO: Working on change 644002
[ 2020-11-30T09:45:56 ] INFO: run manually with: ./utils/pcc 644002 kafka-main2001.codfw.wmnet,logstash1010.eqiad.wmnet,kafka-jumbo1001.eqiad.wmnet,kafka-main1001.eqiad.wmnet
[ 2020-11-30T09:45:57 ] INFO: Refreshing the common repos from upstream if needed
[ 2020-11-30T09:46:01 ] INFO: Creating directories under /srv/jenkins-workspace/puppet-compiler

Note: checking out 'FETCH_HEAD'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

git checkout -b <new-branch-name>

HEAD is now at 33c2e6a1d7 kafka: Migrate hiera() to lookup() and setting datatype in monitoring
From /var/lib/catalog-differ/production
* branch                  production -> FETCH_HEAD
First, rewinding head to replay your work on top of it...
Applying: kafka: Migrate hiera() to lookup() and setting datatype in monitoring
[ 2020-11-30T09:46:13 ] INFO: Starting run in mode change
[ 2020-11-30T09:46:13 ] INFO: Compiling host kafka-main2001.codfw.wmnet (prod)
[ 2020-11-30T09:46:13 ] INFO: Compiling host kafka-main1001.eqiad.wmnet (prod)

[ 2020-11-30T09:46:27 ] INFO: Error in payload
[ 2020-11-30T09:46:27 ] INFO: Compiling host kafka-jumbo1001.eqiad.wmnet (prod)
[ 2020-11-30T09:46:27 ] INFO: Error in payload
[ 2020-11-30T09:46:27 ] INFO: Compiling host logstash1010.eqiad.wmnet (prod)

[ 2020-11-30T09:46:40 ] INFO: Error in payload
[ 2020-11-30T09:46:41 ] INFO: Compiling host logstash1010.eqiad.wmnet (change)

[ 2020-11-30T09:46:55 ] INFO: Calculating diffs for logstash1010.eqiad.wmnet
[ 2020-11-30T09:46:55 ] CRITICAL: Unexpected error running the payload: cannot use a string pattern on a bytes-like object
[ 2020-11-30T09:46:55 ] INFO: [change] Nodes: 1 FAIL
[ 2020-11-30T09:46:55 ] CRITICAL: Unexpected error running the payload: cannot use a string pattern on a bytes-like object
[ 2020-11-30T09:46:55 ] INFO: [change] Nodes: 2 FAIL
[ 2020-11-30T09:46:55 ] CRITICAL: Unexpected error running the payload: cannot use a string pattern on a bytes-like object
[ 2020-11-30T09:46:55 ] INFO: [change] Nodes: 3 FAIL
[ 2020-11-30T09:46:55 ] INFO: [change] Nodes: 3 FAIL 1 NOOP
[ 2020-11-30T09:46:56 ] INFO: Run finished for mode change; see your results at https://puppet-compiler.wmflabs.org/compiler1002/26756/
Build step 'Execute shell' marked build as failure
Finished:
FAIL

I am not sure exactly how to debug this further, but the error seems to come from the on_node_compiled function..

Event Timeline

jbond claimed this task.

This looks like it was an artefact of the python2 -> python3 migration. I have pushed the PS that drops all the hacks added to support both python versions and this is running correctly now.

Let me know if you see any further issues

Krenair set Security to Software security bug.Dec 1 2020, 2:00 AM
Krenair added projects: Security, Security-Team.
Krenair changed the visibility from "Public (No Login Required)" to "Custom Policy".
Krenair changed the subtype of this task from "Task" to "Security Issue".
Krenair added a subscriber: Krenair.

Protecting as security issue due to presence of what appears to be a Jenkins API token in the task description, based on https://wikitech.wikimedia.org/wiki/Help:Puppet-compiler#Catalog_compiler_local_run_(pcc_utility)

reopening too to ensure this gets looked at

Legoktm added a subscriber: Legoktm.

I went to https://integration.wikimedia.org/ci/user/elukey/configure and disabled the "pcc" token.

@elukey you should be able to create a new token like normal.

@Krenair @Legoktm wow sorry I was really sloppy, I didn't really realize, first time that happens :(

Thanks for following up, I'll be more careful next time!

Legoktm changed the visibility from "Custom Policy" to "Public (No Login Required)".