Page MenuHomePhabricator

Update `sql` command to use new wiki replica servers
Closed, ResolvedPublic

Description

Toolforge provides a convenience script in /usr/bin/sql that makes connecting to a Wiki Replica server easier. The current script connects to the *.labsdb hosts which are being decommissioned. The script should be changed to connect to the *.{analytics,web}.db.svc.eqiad.wmflabs hosts.

This change may need to be done in stages to avoid breaking usage that relies on access to user created tables on the s1 or s3 hosts.

Details

Related Gerrit Patches:
operations/puppet : productiontoolforge: Remove /usr/bin/sql
labs/toollabs : mastersql: Update to allow connecting to new cluster

Event Timeline

bd808 created this task.Sep 26 2017, 12:12 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 26 2017, 12:12 AM

Change 380684 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[labs/toollabs@master] sql: Update to allow connecting to new cluster

https://gerrit.wikimedia.org/r/380684

Change 380685 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[operations/puppet@production] toolforge: Remove /usr/bin/sql

https://gerrit.wikimedia.org/r/380685

Change 380684 merged by jenkins-bot:
[labs/toollabs@master] sql: Update to allow connecting to new cluster

https://gerrit.wikimedia.org/r/380684

Restricted Application added a project: User-bd808. · View Herald TranscriptSep 27 2017, 4:47 PM

Change 380685 merged by Madhuvishy:
[operations/puppet@production] toolforge: Remove /usr/bin/sql

https://gerrit.wikimedia.org/r/380685

bd808 added a comment.Sep 27 2017, 8:28 PM
$ ssh tools-clushmaster-01.tools.eqiad.wmflabs
$ clush -w @all -b '/usr/bin/dpkg -s misctools &>/dev/null && sudo apt-get update -qq && sudo /usr/bin/env DEBIAN_FRONTEND=noninteractive apt-get install -q -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" misctools'
tools-docker-builder-05.tools.eqiad.wmflabs: ssh: connect to host tools-docker-builder-05.tools.eqiad.wmflabs port 22: Connection timed out
tools-exec-1411.tools.eqiad.wmflabs: E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable)
tools-exec-1411.tools.eqiad.wmflabs: E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?
tools-exec-1420.tools.eqiad.wmflabs: E: Could not open file /var/lib/apt/lists/nova.clouds.archive.ubuntu.com_ubuntu_dists_trusty_multiverse_i18n_Translation-en - open (2: No such file or directory)
tools-exec-1420.tools.eqiad.wmflabs: E: Could not open file /var/lib/apt/lists/nova.clouds.archive.ubuntu.com_ubuntu_dists_trusty_main_i18n_Translation-en - open (2: No such file or directory)
tools-exec-1420.tools.eqiad.wmflabs: E: Could not open file /var/lib/apt/lists/nova.clouds.archive.ubuntu.com_ubuntu_dists_trusty_multiverse_binary-amd64_Packages - open (2: No such file or directory)
tools-exec-1420.tools.eqiad.wmflabs: E: Could not open file /var/lib/apt/lists/nova.clouds.archive.ubuntu.com_ubuntu_dists_trusty_universe_binary-amd64_Packages - open (2: No such file or directory)
tools-puppetmaster-01.tools.eqiad.wmflabs: Permission denied (publickey,hostbased).
---------------
{... snip ...}
clush: tools-checker-[01-02].tools.eqiad.wmflabs,tools-clushmaster-01.tools.eqiad.wmflabs,tools-docker-registry-[01-02].tools.eqiad.wmflabs,tools-elastic-[01-03].tools.eqiad.wmflabs,tools-flannel-etcd-[01-03].tools.eqiad.wmflabs,tools-grid-master.tools.eqiad.wmflabs,tools-grid-shadow.tools.eqiad.wmflabs,tools-k8s-etcd-[01-03].tools.eqiad.wmflabs,tools-k8s-master-01.tools.eqiad.wmflabs,tools-logs-02.tools.eqiad.wmflabs,tools-mail.tools.eqiad.wmflabs,tools-package-builder-01.tools.eqiad.wmflabs,tools-paws-master-01.tools.eqiad.wmflabs,tools-paws-worker-[1001-1003,1005-1007,1010,1013,1016-1017].tools.eqiad.wmflabs,tools-prometheus-[01-02].tools.eqiad.wmflabs,tools-proxy-[01-02].tools.eqiad.wmflabs,tools-redis-[1001-1002].tools.eqiad.wmflabs,tools-services-[01-02].tools.eqiad.wmflabs,tools-static-[10-11].tools.eqiad.wmflabs,tools-worker-[1001-1023,1025-1027].tools.eqiad.wmflabs (67): exited with exit code 1
clush: tools-exec-[1411,1420].tools.eqiad.wmflabs (2): exited with exit code 100
clush: tools-docker-builder-05.tools.eqiad.wmflabs,tools-puppetmaster-01.tools.eqiad.wmflabs (2): exited with exit code 255
bd808 added a comment.Sep 27 2017, 9:56 PM

A second clush run showed clean for the prior failures on tools-exec-[1411,1420].tools.eqiad.wmflabs. The tools-docker and tools-puppetmaster failures are unrelated and unimportant.

bd808 closed this task as Resolved.Oct 1 2017, 3:35 AM