Toolforge provides a convenience script in /usr/bin/sql that makes connecting to a Wiki Replica server easier. The current script connects to the *.labsdb hosts which are being decommissioned. The script should be changed to connect to the *.{analytics,web}.db.svc.eqiad.wmflabs hosts.
This change may need to be done in stages to avoid breaking usage that relies on access to user created tables on the s1 or s3 hosts.
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| toolforge: Remove /usr/bin/sql | operations/puppet | production | +0 -128 | |
| sql: Update to allow connecting to new cluster | labs/toollabs | master | +78 -46 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | jcrespo | T140788 Labs databases rearchitecture (tracking) | |||
| Resolved | bd808 | T166402 Program 7 Outcome 3: data services | |||
| Resolved | bd808 | T142807 Migrate all users to new Wiki Replica cluster and decommission old hardware | |||
| Resolved | bd808 | T176688 Update `sql` command to use new wiki replica servers |
Change 380684 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[labs/toollabs@master] sql: Update to allow connecting to new cluster
Change 380685 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[operations/puppet@production] toolforge: Remove /usr/bin/sql
Change 380684 merged by jenkins-bot:
[labs/toollabs@master] sql: Update to allow connecting to new cluster
Change 380685 merged by Madhuvishy:
[operations/puppet@production] toolforge: Remove /usr/bin/sql
$ ssh tools-clushmaster-01.tools.eqiad.wmflabs $ clush -w @all -b '/usr/bin/dpkg -s misctools &>/dev/null && sudo apt-get update -qq && sudo /usr/bin/env DEBIAN_FRONTEND=noninteractive apt-get install -q -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" misctools' tools-docker-builder-05.tools.eqiad.wmflabs: ssh: connect to host tools-docker-builder-05.tools.eqiad.wmflabs port 22: Connection timed out tools-exec-1411.tools.eqiad.wmflabs: E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable) tools-exec-1411.tools.eqiad.wmflabs: E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it? tools-exec-1420.tools.eqiad.wmflabs: E: Could not open file /var/lib/apt/lists/nova.clouds.archive.ubuntu.com_ubuntu_dists_trusty_multiverse_i18n_Translation-en - open (2: No such file or directory) tools-exec-1420.tools.eqiad.wmflabs: E: Could not open file /var/lib/apt/lists/nova.clouds.archive.ubuntu.com_ubuntu_dists_trusty_main_i18n_Translation-en - open (2: No such file or directory) tools-exec-1420.tools.eqiad.wmflabs: E: Could not open file /var/lib/apt/lists/nova.clouds.archive.ubuntu.com_ubuntu_dists_trusty_multiverse_binary-amd64_Packages - open (2: No such file or directory) tools-exec-1420.tools.eqiad.wmflabs: E: Could not open file /var/lib/apt/lists/nova.clouds.archive.ubuntu.com_ubuntu_dists_trusty_universe_binary-amd64_Packages - open (2: No such file or directory) tools-puppetmaster-01.tools.eqiad.wmflabs: Permission denied (publickey,hostbased). --------------- {... snip ...} clush: tools-checker-[01-02].tools.eqiad.wmflabs,tools-clushmaster-01.tools.eqiad.wmflabs,tools-docker-registry-[01-02].tools.eqiad.wmflabs,tools-elastic-[01-03].tools.eqiad.wmflabs,tools-flannel-etcd-[01-03].tools.eqiad.wmflabs,tools-grid-master.tools.eqiad.wmflabs,tools-grid-shadow.tools.eqiad.wmflabs,tools-k8s-etcd-[01-03].tools.eqiad.wmflabs,tools-k8s-master-01.tools.eqiad.wmflabs,tools-logs-02.tools.eqiad.wmflabs,tools-mail.tools.eqiad.wmflabs,tools-package-builder-01.tools.eqiad.wmflabs,tools-paws-master-01.tools.eqiad.wmflabs,tools-paws-worker-[1001-1003,1005-1007,1010,1013,1016-1017].tools.eqiad.wmflabs,tools-prometheus-[01-02].tools.eqiad.wmflabs,tools-proxy-[01-02].tools.eqiad.wmflabs,tools-redis-[1001-1002].tools.eqiad.wmflabs,tools-services-[01-02].tools.eqiad.wmflabs,tools-static-[10-11].tools.eqiad.wmflabs,tools-worker-[1001-1023,1025-1027].tools.eqiad.wmflabs (67): exited with exit code 1 clush: tools-exec-[1411,1420].tools.eqiad.wmflabs (2): exited with exit code 100 clush: tools-docker-builder-05.tools.eqiad.wmflabs,tools-puppetmaster-01.tools.eqiad.wmflabs (2): exited with exit code 255
A second clush run showed clean for the prior failures on tools-exec-[1411,1420].tools.eqiad.wmflabs. The tools-docker and tools-puppetmaster failures are unrelated and unimportant.