Page Menu
Home
Phabricator
Search
Configure Global Search
Log In
Paste
P8335
instance metadata breakage
Active
Public
Actions
Authored by
Krenair
on Apr 3 2019, 12:06 PM.
Edit Paste
Archive Paste
View Raw File
Subscribe
Mute Notifications
Award Token
Flag For Later
Tags
Cloud-VPS
Referenced Files
F28569903: raw.txt
Apr 3 2019, 12:06 PM
2019-04-03 12:06:48 (UTC+0)
Subscribers
None
krenair@deployment-cumin:~$ sudo cumin '*' 'curl "http://169.254.169.254/latest/meta-data/instance-type" -s'
69 hosts will be targeted:
deployment-acme-chief[03-04].deployment-prep.eqiad.wmflabs,deployment-aqs[01-03].deployment-prep.eqiad.wmflabs,deployment-cache-text05.deployment-prep.eqiad.wmflabs,deployment-cache-upload04.deployment-prep.eqiad.wmflabs,deployment-changeprop.deployment-prep.eqiad.wmflabs,deployment-chromium[01-02].deployment-prep.eqiad.wmflabs,deployment-conf03.deployment-prep.eqiad.wmflabs,deployment-cpjobqueue.deployment-prep.eqiad.wmflabs,deployment-cumin.deployment-prep.eqiad.wmflabs,deployment-db[05-06].deployment-prep.eqiad.wmflabs,deployment-deploy[01-02].deployment-prep.eqiad.wmflabs,deployment-dumps-puppetmaster02.deployment-prep.eqiad.wmflabs,deployment-elastic[05-07].deployment-prep.eqiad.wmflabs,deployment-etcd-01.deployment-prep.eqiad.wmflabs,deployment-eventgate-analytics-1.deployment-prep.eqiad.wmflabs,deployment-eventlog05.deployment-prep.eqiad.wmflabs,deployment-fluorine02.deployment-prep.eqiad.wmflabs,deployment-imagescaler[01-03].deployment-prep.eqiad.wmflabs,deployment-ircd.deployment-prep.eqiad.wmflabs,deployment-jobrunner03.deployment-prep.eqiad.wmflabs,deployment-kafka-jumbo-[1-2].deployment-prep.eqiad.wmflabs,deployment-kafka-main-[1-2].deployment-prep.eqiad.wmflabs,deployment-logstash2.deployment-prep.eqiad.wmflabs,deployment-maps[04-05].deployment-prep.eqiad.wmflabs,deployment-mathoid.deployment-prep.eqiad.wmflabs,deployment-mcs01.deployment-prep.eqiad.wmflabs,deployment-mediawiki-[07,09].deployment-prep.eqiad.wmflabs,deployment-memc[04-07].deployment-prep.eqiad.wmflabs,deployment-ms-be[03-04].deployment-prep.eqiad.wmflabs,deployment-ms-fe02.deployment-prep.eqiad.wmflabs,deployment-mwmaint01.deployment-prep.eqiad.wmflabs,deployment-mx02.deployment-prep.eqiad.wmflabs,deployment-ores01.deployment-prep.eqiad.wmflabs,deployment-parsoid09.deployment-prep.eqiad.wmflabs,deployment-pdfrender02.deployment-prep.eqiad.wmflabs,deployment-poolcounter04.deployment-prep.eqiad.wmflabs,deployment-prometheus02.deployment-prep.eqiad.wmflabs,deployment-puppetdb02.deployment-prep.eqiad.wmflabs,deployment-puppetmaster03.deployment-prep.eqiad.wmflabs,deployment-restbase[01-02].deployment-prep.eqiad.wmflabs,deployment-sca[01-02,04].deployment-prep.eqiad.wmflabs,deployment-sentry01.deployment-prep.eqiad.wmflabs,deployment-sessionstore01.deployment-prep.eqiad.wmflabs,deployment-snapshot01.deployment-prep.eqiad.wmflabs,deployment-urldownloader02.deployment-prep.eqiad.wmflabs,deployment-webperf[11-12].deployment-prep.eqiad.wmflabs,deployment-zookeeper02.deployment-prep.eqiad.wmflabs
Confirm to continue [y/n]? y
===== NODE GROUP =====
(47) deployment-acme-chief[03-04].deployment-prep.eqiad.wmflabs,deployment-aqs03.deployment-prep.eqiad.wmflabs,deployment-chromium02.deployment-prep.eqiad.wmflabs,deployment-conf03.deployment-prep.eqiad.wmflabs,deployment-cpjobqueue.deployment-prep.eqiad.wmflabs,deployment-cumin.deployment-prep.eqiad.wmflabs,deployment-db[05-06].deployment-prep.eqiad.wmflabs,deployment-dumps-puppetmaster02.deployment-prep.eqiad.wmflabs,deployment-elastic[05-07].deployment-prep.eqiad.wmflabs,deployment-etcd-01.deployment-prep.eqiad.wmflabs,deployment-eventgate-analytics-1.deployment-prep.eqiad.wmflabs,deployment-eventlog05.deployment-prep.eqiad.wmflabs,deployment-fluorine02.deployment-prep.eqiad.wmflabs,deployment-imagescaler02.deployment-prep.eqiad.wmflabs,deployment-ircd.deployment-prep.eqiad.wmflabs,deployment-jobrunner03.deployment-prep.eqiad.wmflabs,deployment-kafka-jumbo-[1-2].deployment-prep.eqiad.wmflabs,deployment-kafka-main-[1-2].deployment-prep.eqiad.wmflabs,deployment-mcs01.deployment-prep.eqiad.wmflabs,deployment-mediawiki-07.deployment-prep.eqiad.wmflabs,deployment-memc[04-06].deployment-prep.eqiad.wmflabs,deployment-ms-be[03-04].deployment-prep.eqiad.wmflabs,deployment-mwmaint01.deployment-prep.eqiad.wmflabs,deployment-mx02.deployment-prep.eqiad.wmflabs,deployment-ores01.deployment-prep.eqiad.wmflabs,deployment-parsoid09.deployment-prep.eqiad.wmflabs,deployment-pdfrender02.deployment-prep.eqiad.wmflabs,deployment-prometheus02.deployment-prep.eqiad.wmflabs,deployment-puppetmaster03.deployment-prep.eqiad.wmflabs,deployment-restbase02.deployment-prep.eqiad.wmflabs,deployment-sca[01-02,04].deployment-prep.eqiad.wmflabs,deployment-sentry01.deployment-prep.eqiad.wmflabs,deployment-sessionstore01.deployment-prep.eqiad.wmflabs,deployment-snapshot01.deployment-prep.eqiad.wmflabs,deployment-urldownloader02.deployment-prep.eqiad.wmflabs,deployment-webperf11.deployment-prep.eqiad.wmflabs
----- OUTPUT of 'curl "http://169...nstance-type" -s' -----
500 Internal Server Error
Remote metadata server experienced an internal server error.
===== NODE GROUP =====
(8) deployment-changeprop.deployment-prep.eqiad.wmflabs,deployment-chromium01.deployment-prep.eqiad.wmflabs,deployment-mathoid.deployment-prep.eqiad.wmflabs,deployment-ms-fe02.deployment-prep.eqiad.wmflabs,deployment-poolcounter04.deployment-prep.eqiad.wmflabs,deployment-puppetdb02.deployment-prep.eqiad.wmflabs,deployment-webperf12.deployment-prep.eqiad.wmflabs,deployment-zookeeper02.deployment-prep.eqiad.wmflabs
----- OUTPUT of 'curl "http://169...nstance-type" -s' -----
m1.small
===== NODE GROUP =====
(2) deployment-deploy[01-02].deployment-prep.eqiad.wmflabs
----- OUTPUT of 'curl "http://169...nstance-type" -s' -----
c8.m8.s60
===== NODE GROUP =====
(1) deployment-logstash2.deployment-prep.eqiad.wmflabs
----- OUTPUT of 'curl "http://169...nstance-type" -s' -----
m1.xlarge
===== NODE GROUP =====
(7) deployment-aqs[01-02].deployment-prep.eqiad.wmflabs,deployment-cache-text05.deployment-prep.eqiad.wmflabs,deployment-cache-upload04.deployment-prep.eqiad.wmflabs,deployment-imagescaler[01,03].deployment-prep.eqiad.wmflabs,deployment-memc07.deployment-prep.eqiad.wmflabs
----- OUTPUT of 'curl "http://169...nstance-type" -s' -----
m1.medium
===== NODE GROUP =====
(4) deployment-maps[04-05].deployment-prep.eqiad.wmflabs,deployment-mediawiki-09.deployment-prep.eqiad.wmflabs,deployment-restbase01.deployment-prep.eqiad.wmflabs
----- OUTPUT of 'curl "http://169...nstance-type" -s' -----
m1.large
================
PASS: |█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (69/69) [00:35<00:00, 1.94hosts/s]
FAIL: | | 0% (0/69) [00:35<?, ?hosts/s]
100.0% (69/69) success ratio (>= 100.0% threshold) for command: 'curl "http://169...nstance-type" -s'.
100.0% (69/69) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Event Timeline
Krenair
created this paste.
Apr 3 2019, 12:06 PM
2019-04-03 12:06:48 (UTC+0)
Krenair
mentioned this in
T219983: Instance metadata service returning HTTP 500 for majority of instances checked
.
Apr 3 2019, 12:19 PM
2019-04-03 12:19:08 (UTC+0)
Log In to Comment