Page MenuHomePhabricator

RKemper (Ryan Kemper)
User

Projects

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Wednesday

  • Clear sailing ahead.

User Details

User Since
May 1 2020, 10:28 PM (89 w, 2 d)
Availability
Available
LDAP User
Unknown
MediaWiki User
RKemper (WMF) [ Global Accounts ]

Recent Activity

Fri, Jan 14

RKemper moved T299177: Fix package dependency error on elasticsearch puppet config from Incoming to Needs Reporting on the Discovery-Search (Current work) board.

elastic2051 came up fine after the new re-image

Fri, Jan 14, 11:18 PM · Patch-For-Review, Discovery-Search (Current work)
RKemper moved T298674: Degraded RAID on elastic2051 from Incoming to Needs Reporting on the Discovery-Search (Current work) board.

Re-image is complete

Fri, Jan 14, 11:15 PM · Discovery-Search (Current work), Patch-For-Review, SRE, ops-codfw
RKemper closed T298674: Degraded RAID on elastic2051 as Resolved.
Fri, Jan 14, 11:15 PM · Discovery-Search (Current work), Patch-For-Review, SRE, ops-codfw
RKemper triaged T298674: Degraded RAID on elastic2051 as Medium priority.
Fri, Jan 14, 11:12 PM · Discovery-Search (Current work), Patch-For-Review, SRE, ops-codfw
RKemper added a comment to T299177: Fix package dependency error on elasticsearch puppet config.

This latest patch at least isn't broken on the old/working hosts. We're reimaging elastic2051 to make sure it works on a fresh host.

Fri, Jan 14, 10:27 PM · Patch-For-Review, Discovery-Search (Current work)

Thu, Jan 13

RKemper added a comment to P18732 elastic2051.codfw.wmnet reimage failure.
ryankemper@elastic2051:~$ sudo run-puppet-agent
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for elastic2051.codfw.wmnet
Info: Applying configuration version '(0cdcafe1ab) joal - Fix error in network_internal_flows druid job'
Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install elasticsearch-oss' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
  elasticsearch-oss
INFO:debmonitor:Got 1 updates from dpkg hook version 3
INFO:debmonitor:Successfully sent the dpkg_hook update to the DebMonitor server
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
Need to get 0 B/34.2 MB of archives.
After this operation, 38.4 MB of additional disk space will be used.
(Reading database ... 46070 files and directories currently installed.)
Preparing to unpack .../elasticsearch-oss_6.5.4_all.deb ...
could not find java; set JAVA_HOME or ensure java is in PATH
dpkg: error processing archive /var/cache/apt/archives/elasticsearch-oss_6.5.4_all.deb (--unpack):
 subprocess new pre-installation script returned error exit status 1
Errors were encountered while processing:
 /var/cache/apt/archives/elasticsearch-oss_6.5.4_all.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
Error: /Stage[main]/Elasticsearch::Packages/Package[elasticsearch]/ensure: change from 'purged' to 'present' failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install elasticsearch-oss' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
  elasticsearch-oss
INFO:debmonitor:Got 1 updates from dpkg hook version 3
INFO:debmonitor:Successfully sent the dpkg_hook update to the DebMonitor server
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
Need to get 0 B/34.2 MB of archives.
After this operation, 38.4 MB of additional disk space will be used.
(Reading database ... 46070 files and directories currently installed.)
Preparing to unpack .../elasticsearch-oss_6.5.4_all.deb ...
could not find java; set JAVA_HOME or ensure java is in PATH
dpkg: error processing archive /var/cache/apt/archives/elasticsearch-oss_6.5.4_all.deb (--unpack):
 subprocess new pre-installation script returned error exit status 1
Errors were encountered while processing:
 /var/cache/apt/archives/elasticsearch-oss_6.5.4_all.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
Error: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /usr/share/elasticsearch/lib (file: /etc/puppet/modules/elasticsearch/manifests/packages.pp, line: 29)
Error: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /usr/share/elasticsearch/lib (file: /etc/puppet/modules/elasticsearch/manifests/packages.pp, line: 29)
Wrapped exception:
No such file or directory @ dir_chdir - /usr/share/elasticsearch/lib
Error: /Stage[main]/Elasticsearch::Packages/File[/usr/share/elasticsearch/lib/logstash-gelf.jar]/ensure: change from 'absent' to 'link' failed: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /usr/share/elasticsearch/lib (file: /etc/puppet/modules/elasticsearch/manifests/packages.pp, line: 29)
Error: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /usr/share/elasticsearch/lib (file: /etc/puppet/modules/elasticsearch/manifests/packages.pp, line: 33)
Error: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /usr/share/elasticsearch/lib (file: /etc/puppet/modules/elasticsearch/manifests/packages.pp, line: 33)
Wrapped exception:
No such file or directory @ dir_chdir - /usr/share/elasticsearch/lib
Error: /Stage[main]/Elasticsearch::Packages/File[/usr/share/elasticsearch/lib/json-simple.jar]/ensure: change from 'absent' to 'link' failed: Could not set 'link' on ensure: No such file or directory @ dir_chdir - /usr/share/elasticsearch/lib (file: /etc/puppet/modules/elasticsearch/manifests/packages.pp, line: 33)
Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install wmf-elasticsearch-search-plugins' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
wmf-elasticsearch-search-plugins is already the newest version (6.5.4-7~stretch).
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 wmf-elasticsearch-search-plugins : Depends: elasticsearch-oss (= 6.5.4) but it is not going to be installed or
                                             elasticsearch (= 6.5.4) but it is not installable
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).
Error: /Stage[main]/Profile::Elasticsearch::Cirrus/Package[wmf-elasticsearch-search-plugins]/ensure: change from 'absent' to 'present' failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install wmf-elasticsearch-search-plugins' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
wmf-elasticsearch-search-plugins is already the newest version (6.5.4-7~stretch).
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 wmf-elasticsearch-search-plugins : Depends: elasticsearch-oss (= 6.5.4) but it is not going to be installed or
                                             elasticsearch (= 6.5.4) but it is not installable
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution). (corrective)
Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install openjdk-8-jdk' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 openjdk-8-jdk : Depends: openjdk-8-jre (= 8u312-b07-1~deb9u1) but it is not going to be installed
                 Depends: openjdk-8-jdk-headless (= 8u312-b07-1~deb9u1) but it is not going to be installed
                 Depends: libx11-6 but it is not going to be installed
 wmf-elasticsearch-search-plugins : Depends: elasticsearch-oss (= 6.5.4) but it is not going to be installed or
                                             elasticsearch (= 6.5.4) but it is not installable
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).
Error: /Stage[main]/Java/Java::Package[openjdk-jdk-8]/Package[openjdk-8-jdk]/ensure: change from 'purged' to 'present' failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install openjdk-8-jdk' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 openjdk-8-jdk : Depends: openjdk-8-jre (= 8u312-b07-1~deb9u1) but it is not going to be installed
                 Depends: openjdk-8-jdk-headless (= 8u312-b07-1~deb9u1) but it is not going to be installed
                 Depends: libx11-6 but it is not going to be installed
 wmf-elasticsearch-search-plugins : Depends: elasticsearch-oss (= 6.5.4) but it is not going to be installed or
                                             elasticsearch (= 6.5.4) but it is not installable
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).
Notice: /Stage[main]/Logstash/Package[logstash]: Dependency Package[openjdk-8-jdk] has failures: true
Warning: /Stage[main]/Logstash/Package[logstash]: Skipping because of failed dependencies
Warning: /Stage[main]/Logstash/File[/usr/local/bin/logstash-config-test]: Skipping because of failed dependencies
Warning: /Stage[main]/Logstash/File[/etc/default/logstash]: Skipping because of failed dependencies
Warning: /Stage[main]/Logstash/File[/etc/logstash/jvm.options]: Skipping because of failed dependencies
Warning: /Stage[main]/Logstash/File[/etc/logstash/log4j2.properties]: Skipping because of failed dependencies
Warning: /Stage[main]/Logstash/File[/etc/logstash/logstash.yml]: Skipping because of failed dependencies
Warning: /Stage[main]/Logstash/File[/etc/logstash/conf.d/README]: Skipping because of failed dependencies
Warning: /Stage[main]/Logstash/File[/etc/init/logstash.conf]: Skipping because of failed dependencies
Warning: /Stage[main]/Logstash/File[/etc/init/logstash-web.conf]: Skipping because of failed dependencies
Warning: /Stage[main]/Java/Alternatives::Java[8]/Exec[update_java_alternatives_8]: Skipping because of failed dependencies
Warning: /Stage[main]/Java/Java::Cacert[wmf:puppetca.pem]/Exec[java__cacert_wmf:puppetca.pem]: Skipping because of failed dependencies
Warning: /Stage[main]/Java/Java::Cacert[wmf:Wikimedia_Internal_Root_CA]/Exec[java__cacert_wmf:Wikimedia_Internal_Root_CA]: Skipping because of failed dependencies
Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install elasticsearch-curator' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 wmf-elasticsearch-search-plugins : Depends: elasticsearch-oss (= 6.5.4) but it is not going to be installed or
                                             elasticsearch (= 6.5.4) but it is not installable
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).
Error: /Stage[main]/Elasticsearch::Curator/Package[elasticsearch-curator]/ensure: change from 'purged' to 'present' failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install elasticsearch-curator' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 wmf-elasticsearch-search-plugins : Depends: elasticsearch-oss (= 6.5.4) but it is not going to be installed or
                                             elasticsearch (= 6.5.4) but it is not installable
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).
Notice: /Stage[main]/Elasticsearch/File[/etc/default/elasticsearch]: Dependency Package[elasticsearch] has failures: true
Warning: /Stage[main]/Elasticsearch/File[/etc/default/elasticsearch]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch/File[/srv/elasticsearch]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch/File[/var/log/elasticsearch]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch/File[/var/log/elasticsearch/elasticsearch.log]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch/File[/var/log/elasticsearch/elasticsearch_index_indexing_slowlog.log]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch/File[/var/log/elasticsearch/elasticsearch_index_search_slowlog.log]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch/Service[elasticsearch]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-codfw]/File[/etc/elasticsearch/production-search-codfw/elasticsearch.yml]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-codfw]/File[/etc/elasticsearch/production-search-codfw/log4j2.properties]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-codfw]/File[/etc/elasticsearch/production-search-codfw/jvm.options]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-codfw]/File[/etc/elasticsearch/production-search-codfw/scripts]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-codfw]/Exec[elasticsearch-create-keystore-production-search-codfw]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-codfw]/File[/etc/elasticsearch/production-search-codfw/elasticsearch.keystore]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-codfw]/File[/srv/elasticsearch/production-search-codfw]: Skipping because of failed dependencies
Notice: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-codfw]/Service[elasticsearch_6@production-search-codfw]: Dependency Package[wmf-elasticsearch-search-plugins] has failures: true
Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-codfw]/Service[elasticsearch_6@production-search-codfw]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-omega-codfw]/File[/etc/elasticsearch/production-search-omega-codfw/elasticsearch.yml]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-omega-codfw]/File[/etc/elasticsearch/production-search-omega-codfw/log4j2.properties]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-omega-codfw]/File[/etc/elasticsearch/production-search-omega-codfw/jvm.options]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-omega-codfw]/File[/etc/elasticsearch/production-search-omega-codfw/scripts]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-omega-codfw]/Exec[elasticsearch-create-keystore-production-search-omega-codfw]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-omega-codfw]/File[/etc/elasticsearch/production-search-omega-codfw/elasticsearch.keystore]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-omega-codfw]/File[/srv/elasticsearch/production-search-omega-codfw]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch/Elasticsearch::Instance[production-search-omega-codfw]/Service[elasticsearch_6@production-search-omega-codfw]: Skipping because of failed dependencies
Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install prometheus-elasticsearch-exporter' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 prometheus-elasticsearch-exporter : Depends: daemon but it is not going to be installed
 wmf-elasticsearch-search-plugins : Depends: elasticsearch-oss (= 6.5.4) but it is not going to be installed or
                                             elasticsearch (= 6.5.4) but it is not installable
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).
Error: /Stage[main]/Prometheus::Elasticsearch_exporter::Common/Package[prometheus-elasticsearch-exporter]/ensure: change from 'purged' to 'present' failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install prometheus-elasticsearch-exporter' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 prometheus-elasticsearch-exporter : Depends: daemon but it is not going to be installed
 wmf-elasticsearch-search-plugins : Depends: elasticsearch-oss (= 6.5.4) but it is not going to be installed or
                                             elasticsearch (= 6.5.4) but it is not installable
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).
Notice: /Stage[main]/Prometheus::Elasticsearch_exporter::Common/Service[prometheus-elasticsearch-exporter]: Dependency Package[prometheus-elasticsearch-exporter] has failures: true
Warning: /Stage[main]/Prometheus::Elasticsearch_exporter::Common/Service[prometheus-elasticsearch-exporter]: Skipping because of failed dependencies
Error: Could not set 'present' on ensure: No such file or directory - A directory component in /etc/logstash/conf.d/10-input-gelf-gelf_relay.conf20220113-7303-oqd0c5.lock does not exist or is a dangling symbolic link (file: /etc/puppet/modules/logstash/manifests/conf.pp, line: 33)
Error: Could not set 'present' on ensure: No such file or directory - A directory component in /etc/logstash/conf.d/10-input-gelf-gelf_relay.conf20220113-7303-oqd0c5.lock does not exist or is a dangling symbolic link (file: /etc/puppet/modules/logstash/manifests/conf.pp, line: 33)
Wrapped exception:
No such file or directory - A directory component in /etc/logstash/conf.d/10-input-gelf-gelf_relay.conf20220113-7303-oqd0c5.lock does not exist or is a dangling symbolic link
Error: /Stage[main]/Profile::Logstash::Gelf_relay/Logstash::Input::Gelf[gelf_relay]/Logstash::Conf[input-gelf-gelf_relay]/File[/etc/logstash/conf.d/10-input-gelf-gelf_relay.conf]/ensure: change from 'absent' to 'present' failed: Could not set 'present' on ensure: No such file or directory - A directory component in /etc/logstash/conf.d/10-input-gelf-gelf_relay.conf20220113-7303-oqd0c5.lock does not exist or is a dangling symbolic link (file: /etc/puppet/modules/logstash/manifests/conf.pp, line: 33)
Error: Could not set 'present' on ensure: No such file or directory - A directory component in /etc/logstash/conf.d/90-output-udp-gelf_relay.conf20220113-7303-lekz32.lock does not exist or is a dangling symbolic link (file: /etc/puppet/modules/logstash/manifests/conf.pp, line: 33)
Error: Could not set 'present' on ensure: No such file or directory - A directory component in /etc/logstash/conf.d/90-output-udp-gelf_relay.conf20220113-7303-lekz32.lock does not exist or is a dangling symbolic link (file: /etc/puppet/modules/logstash/manifests/conf.pp, line: 33)
Wrapped exception:
No such file or directory - A directory component in /etc/logstash/conf.d/90-output-udp-gelf_relay.conf20220113-7303-lekz32.lock does not exist or is a dangling symbolic link
Error: /Stage[main]/Profile::Logstash::Gelf_relay/Logstash::Output::Udp[gelf_relay]/Logstash::Conf[output-udp-gelf_relay]/File[/etc/logstash/conf.d/90-output-udp-gelf_relay.conf]/ensure: change from 'absent' to 'present' failed: Could not set 'present' on ensure: No such file or directory - A directory component in /etc/logstash/conf.d/90-output-udp-gelf_relay.conf20220113-7303-lekz32.lock does not exist or is a dangling symbolic link (file: /etc/puppet/modules/logstash/manifests/conf.pp, line: 33)
Notice: /Stage[main]/Logstash/Service[logstash]: Dependency File[/etc/logstash/conf.d/10-input-gelf-gelf_relay.conf] has failures: true
Notice: /Stage[main]/Logstash/Service[logstash]: Dependency File[/etc/logstash/conf.d/90-output-udp-gelf_relay.conf] has failures: true
Warning: /Stage[main]/Logstash/Service[logstash]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch::Log::Hot_threads/Systemd::Timer::Job[elasticsearch-hot-threads-log]/Systemd::Unit[elasticsearch-hot-threads-log.service]/File[/lib/systemd/system/elasticsearch-hot-threads-log.service]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch::Log::Hot_threads/Systemd::Timer::Job[elasticsearch-hot-threads-log]/Systemd::Unit[elasticsearch-hot-threads-log.service]/Exec[systemd daemon-reload for elasticsearch-hot-threads-log.service]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Elasticsearch::Cirrus/Profile::Prometheus::Elasticsearch_exporter[elastic2051:9200]/Prometheus::Elasticsearch_exporter[localhost:9200]/Systemd::Service[prometheus-elasticsearch-exporter-9200]/Systemd::Unit[prometheus-elasticsearch-exporter-9200]/File[/lib/systemd/system/prometheus-elasticsearch-exporter-9200.service]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Elasticsearch::Cirrus/Profile::Prometheus::Elasticsearch_exporter[elastic2051:9200]/Prometheus::Elasticsearch_exporter[localhost:9200]/Systemd::Service[prometheus-elasticsearch-exporter-9200]/Systemd::Unit[prometheus-elasticsearch-exporter-9200]/Exec[systemd daemon-reload for prometheus-elasticsearch-exporter-9200.service]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Elasticsearch::Cirrus/Profile::Prometheus::Elasticsearch_exporter[elastic2051:9200]/Prometheus::Elasticsearch_exporter[localhost:9200]/Systemd::Service[prometheus-elasticsearch-exporter-9200]/Service[prometheus-elasticsearch-exporter-9200]: Skipping because of failed dependencies
Notice: /Stage[main]/Profile::Elasticsearch::Cirrus/Profile::Prometheus::Wmf_elasticsearch_exporter[elastic2051:9200]/Prometheus::Wmf_elasticsearch_exporter[elastic2051:9200]/Systemd::Service[prometheus-wmf-elasticsearch-exporter-9200]/Service[prometheus-wmf-elasticsearch-exporter-9200]/ensure: ensure changed 'stopped' to 'running' (corrective)
Info: /Stage[main]/Profile::Elasticsearch::Cirrus/Profile::Prometheus::Wmf_elasticsearch_exporter[elastic2051:9200]/Prometheus::Wmf_elasticsearch_exporter[elastic2051:9200]/Systemd::Service[prometheus-wmf-elasticsearch-exporter-9200]/Service[prometheus-wmf-elasticsearch-exporter-9200]: Unscheduling refresh on Service[prometheus-wmf-elasticsearch-exporter-9200]
Warning: /Stage[main]/Profile::Elasticsearch::Cirrus/Profile::Prometheus::Elasticsearch_exporter[elastic2051:9400]/Prometheus::Elasticsearch_exporter[localhost:9400]/Systemd::Service[prometheus-elasticsearch-exporter-9400]/Systemd::Unit[prometheus-elasticsearch-exporter-9400]/File[/lib/systemd/system/prometheus-elasticsearch-exporter-9400.service]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Elasticsearch::Cirrus/Profile::Prometheus::Elasticsearch_exporter[elastic2051:9400]/Prometheus::Elasticsearch_exporter[localhost:9400]/Systemd::Service[prometheus-elasticsearch-exporter-9400]/Systemd::Unit[prometheus-elasticsearch-exporter-9400]/Exec[systemd daemon-reload for prometheus-elasticsearch-exporter-9400.service]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Elasticsearch::Cirrus/Profile::Prometheus::Elasticsearch_exporter[elastic2051:9400]/Prometheus::Elasticsearch_exporter[localhost:9400]/Systemd::Service[prometheus-elasticsearch-exporter-9400]/Service[prometheus-elasticsearch-exporter-9400]: Skipping because of failed dependencies
Notice: /Stage[main]/Profile::Elasticsearch::Cirrus/Profile::Prometheus::Wmf_elasticsearch_exporter[elastic2051:9400]/Prometheus::Wmf_elasticsearch_exporter[elastic2051:9400]/Systemd::Service[prometheus-wmf-elasticsearch-exporter-9400]/Service[prometheus-wmf-elasticsearch-exporter-9400]/ensure: ensure changed 'stopped' to 'running' (corrective)
Info: /Stage[main]/Profile::Elasticsearch::Cirrus/Profile::Prometheus::Wmf_elasticsearch_exporter[elastic2051:9400]/Prometheus::Wmf_elasticsearch_exporter[elastic2051:9400]/Systemd::Service[prometheus-wmf-elasticsearch-exporter-9400]/Service[prometheus-wmf-elasticsearch-exporter-9400]: Unscheduling refresh on Service[prometheus-wmf-elasticsearch-exporter-9400]
Warning: /Stage[main]/Elasticsearch::Log::Hot_threads/Systemd::Timer::Job[elasticsearch-hot-threads-log]/Systemd::Timer[elasticsearch-hot-threads-log]/Systemd::Service[elasticsearch-hot-threads-log]/Systemd::Unit[elasticsearch-hot-threads-log.timer]/File[/lib/systemd/system/elasticsearch-hot-threads-log.timer]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch::Log::Hot_threads/Systemd::Timer::Job[elasticsearch-hot-threads-log]/Systemd::Timer[elasticsearch-hot-threads-log]/Systemd::Service[elasticsearch-hot-threads-log]/Systemd::Unit[elasticsearch-hot-threads-log.timer]/Exec[systemd daemon-reload for elasticsearch-hot-threads-log.timer]: Skipping because of failed dependencies
Warning: /Stage[main]/Elasticsearch::Log::Hot_threads/Systemd::Timer::Job[elasticsearch-hot-threads-log]/Systemd::Timer[elasticsearch-hot-threads-log]/Systemd::Service[elasticsearch-hot-threads-log]/Service[elasticsearch-hot-threads-log.timer]: Skipping because of failed dependencies
Info: Class[Profile::Elasticsearch::Cirrus]: Unscheduling all events on Class[Profile::Elasticsearch::Cirrus]
Notice: Applied catalog in 22.24 seconds
Thu, Jan 13, 8:43 PM
RKemper added a comment to P18732 elastic2051.codfw.wmnet reimage failure.
ryankemper@elastic2054:~$ apt-cache rdepends --installed java-common
java-common
Reverse Depends:
  openjdk-8-jre-headless
  openjdk-8-jre-headless
Thu, Jan 13, 7:36 PM
RKemper added a comment to P18732 elastic2051.codfw.wmnet reimage failure.

elastic2051 doesn't have java installed:
java-common/oldoldstable 0.58+deb9u1 all

Thu, Jan 13, 7:30 PM
RKemper added a comment to P18732 elastic2051.codfw.wmnet reimage failure.

Kicked off another puppet agent run on the off chance it was a first-puppet-run-failure type thing. It looks pretty borked. Probably going to try the apt --fix-broken-install command, but going to read through the output below first and see if anything obvious jumps out at me.

Thu, Jan 13, 7:26 PM

Wed, Jan 12

RKemper renamed T299098: hw troubleshooting: IPMI Power Supply Failure (PS2) for wdqs2003.codfw.wmnet from hw troubleshooting: IPMI Power Supply Failure for wdqs2003.codfw.wmnet to hw troubleshooting: IPMI Power Supply Failure (PS2) for wdqs2003.codfw.wmnet.
Wed, Jan 12, 9:17 PM · SRE, Discovery-Search (Current work), ops-codfw, DC-Ops
RKemper updated the task description for T299098: hw troubleshooting: IPMI Power Supply Failure (PS2) for wdqs2003.codfw.wmnet.
Wed, Jan 12, 9:17 PM · SRE, Discovery-Search (Current work), ops-codfw, DC-Ops
RKemper updated the task description for T299098: hw troubleshooting: IPMI Power Supply Failure (PS2) for wdqs2003.codfw.wmnet.
Wed, Jan 12, 9:17 PM · SRE, Discovery-Search (Current work), ops-codfw, DC-Ops
RKemper updated the task description for T299098: hw troubleshooting: IPMI Power Supply Failure (PS2) for wdqs2003.codfw.wmnet.
Wed, Jan 12, 9:16 PM · SRE, Discovery-Search (Current work), ops-codfw, DC-Ops
RKemper moved T299098: hw troubleshooting: IPMI Power Supply Failure (PS2) for wdqs2003.codfw.wmnet from Incoming to Blocked (from outside the team) on the Discovery-Search (Current work) board.
Wed, Jan 12, 9:15 PM · SRE, Discovery-Search (Current work), ops-codfw, DC-Ops
RKemper updated the task description for T299098: hw troubleshooting: IPMI Power Supply Failure (PS2) for wdqs2003.codfw.wmnet.
Wed, Jan 12, 9:15 PM · SRE, Discovery-Search (Current work), ops-codfw, DC-Ops
RKemper created T299098: hw troubleshooting: IPMI Power Supply Failure (PS2) for wdqs2003.codfw.wmnet.
Wed, Jan 12, 9:15 PM · SRE, Discovery-Search (Current work), ops-codfw, DC-Ops
RKemper added a comment to T298853: Degraded RAID on elastic2035.

@wiki_willy Yes, please go ahead and ignore/resolve

Wed, Jan 12, 6:35 PM · Discovery-Search (Current work), SRE, ops-codfw

Tue, Jan 11

RKemper added a comment to T297907: SRE Onboarding - Brian King, Search Platform team.

Added Brian to https://phabricator.wikimedia.org/project/profile/29/ so he can see (in this case) dc-ops stuff like https://phabricator.wikimedia.org/T297645

Tue, Jan 11, 10:38 PM · Discovery-Search (Current work)
RKemper updated the task description for T297907: SRE Onboarding - Brian King, Search Platform team.
Tue, Jan 11, 10:38 PM · Discovery-Search (Current work)
RKemper added a member for acl*sre-team: bking.
Tue, Jan 11, 10:37 PM
RKemper created P18616 ssh config snippet.
Tue, Jan 11, 10:08 PM

Mon, Jan 10

RKemper updated the task description for T297986: [Tracking task] Pair with brian king on various operational tasks.
Mon, Jan 10, 4:47 PM · Discovery-Search (Current work)

Wed, Dec 22

RKemper created P18254 Ban / Unban Elasticsearch Node (crappy notes).
Wed, Dec 22, 6:05 PM · Discovery-Search

Mon, Dec 20

RKemper updated the task description for T297986: [Tracking task] Pair with brian king on various operational tasks.
Mon, Dec 20, 8:02 PM · Discovery-Search (Current work)
RKemper created P18246 WDQS Abridged Deploy Notes.
Mon, Dec 20, 6:12 PM · Discovery-Search
RKemper added a comment to T297735: elastic1043.eqiad.wmnet stuck in power off state.

We're not going to open a dc-ops ticket. This host will be replaced when we add the new eqiad elastic hosts in early January.

Mon, Dec 20, 4:53 PM · Discovery-Search (Current work)

Dec 17 2021

RKemper updated the task description for T297986: [Tracking task] Pair with brian king on various operational tasks.
Dec 17 2021, 11:37 PM · Discovery-Search (Current work)
RKemper updated the task description for T297907: SRE Onboarding - Brian King, Search Platform team.
Dec 17 2021, 11:35 PM · Discovery-Search (Current work)
RKemper closed T297910: Requesting shell access for Brian King as Resolved.
Dec 17 2021, 11:12 PM · SRE, LDAP-Access-Requests, SRE-Access-Requests
RKemper triaged T297986: [Tracking task] Pair with brian king on various operational tasks as Medium priority.
Dec 17 2021, 10:58 PM · Discovery-Search (Current work)
RKemper created T297986: [Tracking task] Pair with brian king on various operational tasks.
Dec 17 2021, 10:56 PM · Discovery-Search (Current work)
RKemper added a comment to T297910: Requesting shell access for Brian King.

Added bking to LDAP users and merged the puppet patches. This should be all done AFAICT.

Dec 17 2021, 8:04 PM · SRE, LDAP-Access-Requests, SRE-Access-Requests
RKemper updated the task description for T297907: SRE Onboarding - Brian King, Search Platform team.
Dec 17 2021, 7:29 PM · Discovery-Search (Current work)
RKemper added a comment to T297910: Requesting shell access for Brian King.

Brian and I will pair tomorrow on making the various puppet patches for the access request.

Dec 17 2021, 5:52 AM · SRE, LDAP-Access-Requests, SRE-Access-Requests

Dec 16 2021

RKemper updated the task description for T297910: Requesting shell access for Brian King.
Dec 16 2021, 9:09 PM · SRE, LDAP-Access-Requests, SRE-Access-Requests
RKemper updated the task description for T297910: Requesting shell access for Brian King.
Dec 16 2021, 9:03 PM · SRE, LDAP-Access-Requests, SRE-Access-Requests
RKemper updated the task description for T297908: Requesting access to analytics-privatedata-users for ryankemper.
Dec 16 2021, 8:53 PM · SRE, SRE-Access-Requests
RKemper created T297908: Requesting access to analytics-privatedata-users for ryankemper.
Dec 16 2021, 8:52 PM · SRE, SRE-Access-Requests
RKemper claimed T297907: SRE Onboarding - Brian King, Search Platform team.
Dec 16 2021, 8:01 PM · Discovery-Search (Current work)
RKemper created T297907: SRE Onboarding - Brian King, Search Platform team.
Dec 16 2021, 8:00 PM · Discovery-Search (Current work)

Dec 14 2021

RKemper updated the task description for T297735: elastic1043.eqiad.wmnet stuck in power off state.
Dec 14 2021, 6:11 PM · Discovery-Search (Current work)
RKemper created T297735: elastic1043.eqiad.wmnet stuck in power off state.
Dec 14 2021, 6:09 PM · Discovery-Search (Current work)

Dec 12 2021

RKemper added a parent task for T264181: Migrate WDQS to profile::java: T297555: [epic] Brian's onboarding to the Search Platform team.
Dec 12 2021, 2:22 AM · wdwb-tech, Wikidata-Query-Service, Wikidata, SRE
RKemper added a parent task for T178328: Create maintenance script to clean up stale indexes: T297555: [epic] Brian's onboarding to the Search Platform team.
Dec 12 2021, 2:21 AM · Discovery-Search, CirrusSearch
RKemper added a subtask for T297555: [epic] Brian's onboarding to the Search Platform team: T178328: Create maintenance script to clean up stale indexes.
Dec 12 2021, 2:21 AM · Discovery-Search (Current work), Epic
RKemper added a parent task for T193473: Add HTTPS support to wdqs-internal service: T297555: [epic] Brian's onboarding to the Search Platform team.
Dec 12 2021, 2:21 AM · wdwb-tech, SRE, Discovery, Wikidata, Wikidata-Query-Service
RKemper added a subtask for T297555: [epic] Brian's onboarding to the Search Platform team: T193473: Add HTTPS support to wdqs-internal service.
Dec 12 2021, 2:21 AM · Discovery-Search (Current work), Epic
RKemper added a subtask for T297555: [epic] Brian's onboarding to the Search Platform team: T270614: Automatically depool wdqs servers that are "lagged".
Dec 12 2021, 2:20 AM · Discovery-Search (Current work), Epic
RKemper added a parent task for T270614: Automatically depool wdqs servers that are "lagged": T297555: [epic] Brian's onboarding to the Search Platform team.
Dec 12 2021, 2:20 AM · Wikidata, Wikidata-Query-Service
RKemper added a subtask for T297555: [epic] Brian's onboarding to the Search Platform team: T264181: Migrate WDQS to profile::java.
Dec 12 2021, 2:20 AM · Discovery-Search (Current work), Epic
RKemper created T297555: [epic] Brian's onboarding to the Search Platform team.
Dec 12 2021, 2:19 AM · Discovery-Search (Current work), Epic
RKemper added a comment to T294152: Q2:(Need By: 2021-12-17) rack/setup/install elastic108[4-8].

@RKemper rack A6 is not 10g rack b4 has no space. Are there any other requirements before i rack in same rows 10g racks?

Dec 12 2021, 1:39 AM · SRE, Discovery-Search, Elasticsearch, ops-eqiad, DC-Ops

Dec 7 2021

RKemper updated the task description for T193473: Add HTTPS support to wdqs-internal service.
Dec 7 2021, 7:42 PM · wdwb-tech, SRE, Discovery, Wikidata, Wikidata-Query-Service

Nov 30 2021

RKemper moved T294961: Resolve kernel hang on wcqs* instances from Needs review to Needs Reporting on the Discovery-Search (Current work) board.
Nov 30 2021, 10:50 PM · wdwb-tech, SRE, Discovery-Search (Current work), Wikidata-Query-Service, Wikidata

Nov 25 2021

RKemper added a comment to T294154: Q2:(Need By: TBD) rack/setup/install elastic20[61-72].

@RKemper @Gehel all the servers are ready to put in service but not elastic2069 for some reason i can not login to it so I will have to check that next week.

Nov 25 2021, 4:17 AM · SRE, Discovery-Search, Elasticsearch, ops-codfw, DC-Ops

Nov 22 2021

RKemper moved T280008: Set up WCQS monitoring from Needs review to To Be Deployed on the Discovery-Search (Current work) board.
Nov 22 2021, 7:55 PM · Discovery-Search (Current work), Wikidata-Query-Service, Wikidata

Nov 18 2021

RKemper added a comment to T295705: Cleanup missing Commons index on Elasticsearch eqiad.

Working on the plugin build. Getting an error at basically the final step:

ryankemper@apt1001:~$ sudo -E reprepro -C component/elastic65 include stretch-wikimedia /home/ryankemper/Elastic_Plugins/wmf-elasticsearch-search-plugins_6.5.4-7_amd64.changes && sudo rm -rfv ~/Elastic_Plugins
File "pool/component/elastic65/w/wmf-elasticsearch-search-plugins/wmf-elasticsearch-search-plugins_6.5.4-6~stretch_all.deb" is already registered with different checksums!
md5 expected: d24b364d64e6898dd0d6c131de039710, got: 9259317418b1bd1ce34a43b806b8e879
sha1 expected: f78b1bd75539aca14e79a9b67b594bba5ed66c2b, got: 56418574cf370dcfc9ed31f9d1782659e697e651
sha256 expected: 05854b89803030182da3e4b3dffc60f668efb66222a5be2eaec1952b668a84bd, got: 237854d649b0912fbf38dc42805b59380657622754bfefc1f0760b2c2660eead
size expected: 31251912, got: 35086032
There have been errors!

I haven't actually merged https://gerrit.wikimedia.org/r/c/operations/software/elasticsearch/plugins/+/738979 so maybe that's why it's still "seeing" 6.5.4-6 instead of 6.5.4-7


Here's the contents of the changes file:

ryankemper@apt1001:~$ cat /home/ryankemper/Elastic_Plugins/wmf-elasticsearch-search-plugins_6.5.4-7_amd64.changes
Format: 1.8
Date: Mon, 15 Nov 2021 10:02:14 -0800
Source: wmf-elasticsearch-search-plugins
Binary: wmf-elasticsearch-search-plugins
Architecture: source all
Version: 6.5.4-7
Distribution: stretch
Urgency: medium
Maintainer: David Causse <dcausse@wikimedia.org>
Changed-By: Erik <ebernhardson@ebernhardson-ThinkPad-X1-Carbon-7th>
Description:
 wmf-elasticsearch-search-plugins - Elasticsearch plugins for search
Changes:
 wmf-elasticsearch-search-plugins (6.5.4-7) stretch; urgency=medium
 .
   * Add repository-swift plugin to support T295705
Checksums-Sha1:
 79474c5b44236358bacd58f26490c7dbc32978b5 688 wmf-elasticsearch-search-plugins_6.5.4-7.dsc
 e7ec820c548822754da4daf70505f312e2501935 40718258 wmf-elasticsearch-search-plugins_6.5.4-7.tar.gz
 56418574cf370dcfc9ed31f9d1782659e697e651 35086032 wmf-elasticsearch-search-plugins_6.5.4-6~stretch_all.deb
 0c030af7184c5fcdda276e819b52b9316b61dde7 6206 wmf-elasticsearch-search-plugins_6.5.4-7_amd64.buildinfo
Checksums-Sha256:
 1d7055d4b29b55b9048b122d7b59f856da4b97fc1b0aa80f83e90a6a2cc88a87 688 wmf-elasticsearch-search-plugins_6.5.4-7.dsc
 66f58ebc5a786f4961f83bef8b860614b9cba3041a790456096bf12154b030eb 40718258 wmf-elasticsearch-search-plugins_6.5.4-7.tar.gz
 237854d649b0912fbf38dc42805b59380657622754bfefc1f0760b2c2660eead 35086032 wmf-elasticsearch-search-plugins_6.5.4-6~stretch_all.deb
 98775ab3ca0a2f1f8302630cef6f2cb0e5eda1ba26f242bd2538cab5af23cd0e 6206 wmf-elasticsearch-search-plugins_6.5.4-7_amd64.buildinfo
Files:
 5a642fa7ea58533e033fb23fca2b5ecf 688 database optional wmf-elasticsearch-search-plugins_6.5.4-7.dsc
 1c709315d373f5ec3b1dee13dfbc72fe 40718258 database optional wmf-elasticsearch-search-plugins_6.5.4-7.tar.gz
 9259317418b1bd1ce34a43b806b8e879 35086032 database optional wmf-elasticsearch-search-plugins_6.5.4-6~stretch_all.deb
 fb08db3ad6979fc478afa8c8666ee3cf 6206 database optional wmf-elasticsearch-search-plugins_6.5.4-7_amd64.buildinfo
Nov 18 2021, 1:27 AM · Patch-For-Review, Discovery-Search (Current work)
RKemper added a comment to T295705: Cleanup missing Commons index on Elasticsearch eqiad.

Working on the plugin build. Getting an error at basically the final step:

Nov 18 2021, 12:39 AM · Patch-For-Review, Discovery-Search (Current work)

Nov 17 2021

RKemper moved T276198: /var/run/elasticsearch deleted by elasticsearch from To Be Deployed to Waiting on the Discovery-Search (Current work) board.

We can probably move this to Needs Reporting but I'll check with @dcausse and others in our Search team Wednesday meeting. For now I'll stick this in Waiting while we make sure that the /var/run/elasticsearch gets created when intended (we might even want to simulate nuking /var/run/elasticsearch after stopping the service on a single host)

Nov 17 2021, 2:04 AM · Discovery-Search (Current work), SRE
RKemper added a comment to T290902: cirrussearch: iron out and document procedure for puppet changes triggering restart.

Just deployed https://gerrit.wikimedia.org/r/c/operations/puppet/+/721644 / https://gerrit.wikimedia.org/r/c/operations/puppet/+/739375 so was able to poke around a bit regarding puppet / systemd restart behavior wrt elasticsearch. I can see changing the unit file leads to a systemd daemon-reload:

Nov 17 2021, 12:35 AM · Discovery-Search (Current work), Sustainability (Incident Followup)

Nov 10 2021

RKemper added a comment to T295480: Searching for files on Commons returns error.

Incident report will be coming up, but the short answer is the commonswiki_file index on eqiad cluster went missing. Traffic has been moved to a cluster that still has the index, working up processes now to copy the good index between clusters and restore the eqiad cluster.

Nov 10 2021, 9:22 PM · Commons, Discovery-Search

Nov 8 2021

RKemper updated the task description for T280221: Refactor elasticsearch rolling $operation cookbooks into a single generic cookbook.
Nov 8 2021, 4:47 PM · Patch-For-Review, Discovery-Search (Current work)
RKemper added a comment to T289135: Upgrade Cirrus Elasticsearch clusters to Debian Bullseye.

Note from Search triage meeting: This ticket is for the debian upgrade specifically, so we can upgrade the OS itself (re-imaging the fleet) before we upgrade Elasticsearch itself, which will keep things relatively straightforward for us (otherwise we'd have to do a much more complicated "online upgrade" procedure)

Nov 8 2021, 4:38 PM · Discovery-Search (Current work), SRE
RKemper added a comment to T294961: Resolve kernel hang on wcqs* instances.

Let's make sure the kernel version is pinned somewhere in our puppet code! Then we can wait to see if the problem is reproduced or not.

Nov 8 2021, 4:33 PM · wdwb-tech, SRE, Discovery-Search (Current work), Wikidata-Query-Service, Wikidata
RKemper merged T294865: wcqs1002 and wcqs2001 unresponsive into T294961: Resolve kernel hang on wcqs* instances.
Nov 8 2021, 4:30 PM · wdwb-tech, SRE, Discovery-Search (Current work), Wikidata-Query-Service, Wikidata
RKemper merged task T294865: wcqs1002 and wcqs2001 unresponsive into T294961: Resolve kernel hang on wcqs* instances.
Nov 8 2021, 4:29 PM · Discovery-Search (Current work), Patch-For-Review, wdwb-tech, Wikidata, Wikidata-Query-Service, SRE

Nov 3 2021

RKemper created P17676 T294961 dns changes log.
Nov 3 2021, 11:25 PM
RKemper created P17674 T294961 puppet changes log.
Nov 3 2021, 11:24 PM
RKemper added a comment to T294961: Resolve kernel hang on wcqs* instances.
  1. Puppet changes
Nov 3 2021, 9:54 PM · wdwb-tech, SRE, Discovery-Search (Current work), Wikidata-Query-Service, Wikidata
RKemper added a comment to T294961: Resolve kernel hang on wcqs* instances.
  1. DNS changes from https://gerrit.wikimedia.org/r/c/operations/dns/+/736585
Nov 3 2021, 9:47 PM · wdwb-tech, SRE, Discovery-Search (Current work), Wikidata-Query-Service, Wikidata
RKemper added a subtask for T294865: wcqs1002 and wcqs2001 unresponsive: T294961: Resolve kernel hang on wcqs* instances.
Nov 3 2021, 9:06 PM · Discovery-Search (Current work), Patch-For-Review, wdwb-tech, Wikidata, Wikidata-Query-Service, SRE
RKemper added a parent task for T294961: Resolve kernel hang on wcqs* instances: T294865: wcqs1002 and wcqs2001 unresponsive.
Nov 3 2021, 9:06 PM · wdwb-tech, SRE, Discovery-Search (Current work), Wikidata-Query-Service, Wikidata
RKemper added a subtask for T260568: [EPIC] Productionize WCQS: T294865: wcqs1002 and wcqs2001 unresponsive.
Nov 3 2021, 9:05 PM · Discovery-Search (Current work), Wikidata-Query-Service, Wikidata, Epic
RKemper added a parent task for T294865: wcqs1002 and wcqs2001 unresponsive: T260568: [EPIC] Productionize WCQS.
Nov 3 2021, 9:05 PM · Discovery-Search (Current work), Patch-For-Review, wdwb-tech, Wikidata, Wikidata-Query-Service, SRE

Nov 2 2021

RKemper updated the task description for T294805: Service implementation for elastic10[68-83].eqiad.wmnet.
Nov 2 2021, 6:50 AM · Patch-For-Review, CirrusSearch, Discovery-Search (Current work)
RKemper updated the task description for T294805: Service implementation for elastic10[68-83].eqiad.wmnet.
Nov 2 2021, 6:44 AM · Patch-For-Review, CirrusSearch, Discovery-Search (Current work)
RKemper updated the task description for T294805: Service implementation for elastic10[68-83].eqiad.wmnet.
Nov 2 2021, 6:42 AM · Patch-For-Review, CirrusSearch, Discovery-Search (Current work)
RKemper added a comment to T294805: Service implementation for elastic10[68-83].eqiad.wmnet.
  1. Step 3
Nov 2 2021, 6:41 AM · Patch-For-Review, CirrusSearch, Discovery-Search (Current work)
RKemper updated the task description for T294805: Service implementation for elastic10[68-83].eqiad.wmnet.
Nov 2 2021, 6:40 AM · Patch-For-Review, CirrusSearch, Discovery-Search (Current work)
RKemper added a comment to T294805: Service implementation for elastic10[68-83].eqiad.wmnet.
  1. [Hosts that will be decom'd] elastic1032 Active — Server HP ProLiant DL360 Gen9 Equinix Ashburn A3 2620:0:861:101:10:64:0:233/64 elastic1033 Active — Server HP ProLiant DL360 Gen9 Equinix Ashburn A3 2620:0:861:101:10:64:0:234/64 elastic1034 Active — Server HP ProLiant DL360 Gen9 Equinix Ashburn A3 2620:0:861:101:10:64:0:235/64 elastic1035 Active — Server HP ProLiant DL360 Gen9 Equinix Ashburn A3 2620:0:861:101:10:64:0:236/64 elastic1036 Active — Server HP ProLiant DL360 Gen9 Equinix Ashburn B3 2620:0:861:102:10:64:16:45/64 elastic1037 Active — Server HP ProLiant DL360 Gen9 Equinix Ashburn B3 2620:0:861:102:10:64:16:46/64 elastic1038 Active — Server HP ProLiant DL360 Gen9 Equinix Ashburn B3 2620:0:861:102:10:64:16:47/64 elastic1039 Failed — Server HP ProLiant DL360 Gen9 Equinix Ashburn B3 2620:0:861:102:10:64:16:48/64 elastic1040 Active — Server HP ProLiant DL360 Gen9 Equinix Ashburn C5 2620:0:861:103:10:64:32:108/64 elastic1041 Active — Server HP ProLiant DL360 Gen9 Equinix Ashburn C5 2620:0:861:103:10:64:32:109/64 elastic1042 Active — Server HP ProLiant DL360 Gen9 Equinix Ashburn C5 2620:0:861:103:10:64:32:110/64 elastic1043 Active — Server HP ProLiant DL360 Gen9 Equinix Ashburn C5 2620:0:861:103:10:64:32:111/64 elastic1044 Active — Server HP ProLiant DL360 Gen9 Equinix Ashburn A6 2620:0:861:101:10:64:0:85/64 elastic1045 Active — Server HP ProLiant DL360 Gen9 Equinix Ashburn A6 2620:0:861:101:10:64:0:86/64 elastic1046 Active — Server HP ProLiant DL360 Gen9 Equinix Ashburn B6 2620:0:861:102:10:64:16:70/64 elastic1047 Active — Server HP ProLiant DL360 Gen9 Equinix Ashburn B6 2620:0:861:102:10:64:16:71/64
Nov 2 2021, 6:31 AM · Patch-For-Review, CirrusSearch, Discovery-Search (Current work)
RKemper triaged T294805: Service implementation for elastic10[68-83].eqiad.wmnet as Medium priority.
Nov 2 2021, 6:18 AM · Patch-For-Review, CirrusSearch, Discovery-Search (Current work)
RKemper added a subtask for T294806: [Epic] Search platform - Hardware requests for 2021-2022: Unknown Object (Task).
Nov 2 2021, 6:03 AM · Discovery-Search (Current work)
RKemper added a subtask for T294806: [Epic] Search platform - Hardware requests for 2021-2022: Unknown Object (Task).
Nov 2 2021, 6:02 AM · Discovery-Search (Current work)
RKemper added a subtask for T294806: [Epic] Search platform - Hardware requests for 2021-2022: Unknown Object (Task).
Nov 2 2021, 6:02 AM · Discovery-Search (Current work)
RKemper added a subtask for T294806: [Epic] Search platform - Hardware requests for 2021-2022: Unknown Object (Task).
Nov 2 2021, 5:58 AM · Discovery-Search (Current work)
RKemper created T294806: [Epic] Search platform - Hardware requests for 2021-2022.
Nov 2 2021, 5:57 AM · Discovery-Search (Current work)
RKemper updated the task description for T294805: Service implementation for elastic10[68-83].eqiad.wmnet.
Nov 2 2021, 5:54 AM · Patch-For-Review, CirrusSearch, Discovery-Search (Current work)
RKemper created T294805: Service implementation for elastic10[68-83].eqiad.wmnet.
Nov 2 2021, 5:54 AM · Patch-For-Review, CirrusSearch, Discovery-Search (Current work)

Oct 27 2021

RKemper renamed T294379: Requesting access to production for ejoseph from (WIP) Requesting access to production for ejoseph to Requesting access to production for ejoseph.
Oct 27 2021, 4:34 PM · SRE, SRE-Access-Requests
RKemper updated the task description for T294379: Requesting access to production for ejoseph.
Oct 27 2021, 4:27 PM · SRE, SRE-Access-Requests
RKemper updated the task description for T294379: Requesting access to production for ejoseph.
Oct 27 2021, 6:49 AM · SRE, SRE-Access-Requests

Oct 26 2021

RKemper added a comment to T294379: Requesting access to production for ejoseph.

I can upload some initial patches for the membership changes in a couple hours

Oct 26 2021, 6:41 PM · SRE, SRE-Access-Requests
RKemper updated the task description for T294379: Requesting access to production for ejoseph.
Oct 26 2021, 6:30 PM · SRE, SRE-Access-Requests
RKemper created T294379: Requesting access to production for ejoseph.
Oct 26 2021, 6:29 PM · SRE, SRE-Access-Requests

Oct 25 2021

RKemper committed rLPRI17acb447e9b0: wcqs: add dummy oauth_access_token_secret (authored by RKemper).
wcqs: add dummy oauth_access_token_secret
Oct 25 2021, 10:35 PM

Oct 20 2021

RKemper awarded T255509: Update gerrit submit type for discovery repositories in gerrit a Love token.
Oct 20 2021, 7:16 PM · Discovery-Search (Current work), Release-Engineering-Team (Radar), Gerrit, Discovery

Oct 19 2021

RKemper updated the task description for T288231: Deploy the wdqs streaming updater to production.
Oct 19 2021, 4:32 AM · Patch-For-Review, Discovery-Search (Current work), Wikidata, Wikidata-Query-Service

Oct 18 2021

RKemper updated the task description for T293638: Create and update the process for separate WCQS mutation topic.
Oct 18 2021, 3:53 PM · Discovery-Search (Current work), Wikidata-Query-Service, Wikidata

Oct 15 2021

RKemper updated the task description for T288231: Deploy the wdqs streaming updater to production.
Oct 15 2021, 11:28 PM · Patch-For-Review, Discovery-Search (Current work), Wikidata, Wikidata-Query-Service
RKemper updated the task description for T288231: Deploy the wdqs streaming updater to production.
Oct 15 2021, 9:48 PM · Patch-For-Review, Discovery-Search (Current work), Wikidata, Wikidata-Query-Service