@tstarling Hey, timing wise this seems to match up with https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/683465/, could yiu take a look?

tools-db-1.tools.eqiad1.wikimedia.cloud now is now replicating. The only problem so far is that users and grants didn't seem to get copied over for some reason.

Closing this years-old ticket. The current plan is to upgrade to MariaDB 10.4.

Closing this years-old ticket. The current plan is to upgrade to MariaDB 10.4.

The patch above works fine for new VMs:

toolsbeta-test-k8s-worker-7    Ready    <none>                 32s    v1.21.8   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=toolsbeta-test-k8s-worker-7,kubernetes.io/os=linux,kubernetes.wmcloud.org/nfs-mounted=true,toolforge.org/nfs-mounted=true

I'm going to write a script to backfill the labels to existing ones.

Procedural note: Access to the labs/tools/graphql repo should only be granted to maintainers of the graphql Toolforge tool. The process for taking over an abandoned tool is described at https://wikitech.wikimedia.org/wiki/Help:Toolforge/Abandoned_tool_policy.

+1

+1

In T311412#8029637, @dcaro wrote:

This might be the reason why this alert triggered:

Hi! As far as I can see, tools-db is not currently in read only mode. Your code file is quite large, do you have the exact statement that is failing?

Support. I don't consider the lack of existing reviews to be a problem, @Dreamy_Jazz's previous contributions and on-wiki functionary roles clearly demonstrate that we can trust them to make good decisions about patches in areas they're familiar with and ask for help if they're unsure.

Done!

https://gerrit.wikimedia.org/r/c/operations/puppet/+/655762

In T310974#8022000, @Kudpung wrote:

@JJMC89 could we have an answer please?

Is this still needed?

Done, since this is not a WMF deployed extension and this task had an endorsement for someone who already has rights to that extension. Sorry for the delay!

Done. Sorry for the delay!

This is covered by the generic Prometheus target down alert.

The following SGE jobs are still running on the exec grid:

tools.alchimista 8787914 tlgm
tools.ammarbot 9058295 cine_review
tools.avicbot 400742 uaaby5min
tools.avicbot 6444029 avicbotirc
tools.cewbot 3280296 cron-tools.cewbot-IRC
tools.earwigbot 9782320 earwigbot
tools.germancon-mobile 400540 germancon-mobile-parser
tools.hat-collector 3965351 go
tools.listeria 3216144 bot
tools.nokib-bot 1593642 daily
tools.patest 6882116 script_wui
tools.phetools 2742445 ws_ocr_daemon
tools.phetools 2742447 verify_match
tools.phetools 2742450 extract_text_layer
tools.sdzerobot 6303775 stream
tools.signature-checker 2297341 cron-20170515.signature_check.wikinews
tools.signature-checker 2297344 cron-20170515.signature_check.wikisource
tools.signature-checker 2297345 cron-20170515.signature_check.wikiversity
tools.signature-checker 4615899 cron-20170515.signature_check.zh
tools.signature-checker 5086089 cron-20170515.signature_check.simple
tools.signature-checker 5969401 cron-20170515.signature_check.wiktionary
tools.signature-checker 6091396 cron-20170515.signature_check.zh-classical
tools.signature-manquante-bot 4769872 signature-manquante
tools.telegram-wikilinksbot 5699376 wikilinksbot
tools.toc 1826241 cron-20170915.topic_list.ja
tools.toc 3609056 cron-20170915.topic_list.wikinews
tools.toc 3609088 cron-20170915.topic_list.wikisource
tools.toc 3609172 cron-20170915.topic_list.wikiversity
tools.toc 5086111 cron-20170915.topic_list.wiktionary
tools.toc 5287268 cron-20170915.topic_list.moegirl
tools.urbanecmbot 2811129 patrolAfterPatrol
tools.urbanecmbot 6443111 patrolSandbox
tools.wikidata-todo 2840123 wc_all
tools.wikidata-todo 2840124 wc_rc
tools.wikitanvirbot 1366931 doubredi
tools.wikitanvirbot 5512585 sand-wbbn
tools.wikitanvirbot 5512746 corona
tools.wikitanvirbot 5816767 sand-wtbn

I'm going to kill those sometime tomorrow (so 2022-06-23).

Hi @SLyngshede-WMF, please also add myself to the ciadmin ldap group as requested in the task description. Thanks!

It does a bit, but that can be worked around if needed. Previously I'd been working with the assumption that we do want to keep basic health metrics for all projects. If that's not something we want, that's fine, I just want that explicitely documented somewhere.

Decided to be bold and created the second address:

taavi@cloudcontrol1004 ~ $ os port create --network 7425e328-560c-4f00-8e99-706f3fb90bb4 --description "reserved address for monitoring" "metricsinfra reserved address #2"

This means we now have the following:

taavi@cloudcontrol1004 ~ $ os port list | grep metricsinfra
| 02099d16-afab-4f27-83fc-8ffe0663e421 | metricsinfra reserved address #2                   | fa:16:3e:f8:4f:36 | ip_address='172.16.6.65', subnet_id='a69bdfad-d7d2-4cfa-8231-3d6d3e0074c9'   | DOWN   |
| 2e67a486-e840-4800-b974-d9220f5e107a | metricsinfra reserved address #1                   | fa:16:3e:c1:4d:69 | ip_address='172.16.0.229', subnet_id='a69bdfad-d7d2-4cfa-8231-3d6d3e0074c9'  | ACTIVE |

@Andrew Hello! I think T310799: Upgrade metricsinfra prometheus to bullseye gives me the reason I needed to finally get this done. In short:

• I want to create a new pair of instances, metricsinfra-prometheus-2 and metricsinfra-prometheus-3, with the g3.cores4.ram8.disk20 flavor. Those new instances should have access to the node-exporter port on all VMs on all projects.
• In addition to Neutron security groups, the new instances also need to be added to the profile::wmcs::instance::metricsinfra_prometheus_nodes hiera key.
• The existing instance was given a reserved address in T250206#6056467. We can re-use that for one of the instances and create a new similarly reserved port for the other one. That gives us stable IPs that don't change the next time we're upgrading these VMs.
• I believe we can update wmfkeystonehooks to create the needed security group rules by default and write some scripts to backfill it to existing projects.

Hi and sorry for the delay!

I wanted to use this to ensure we only have one active cron runner:

# if this is not the active cron runner, block tool crons for easier migration between nodes,
# but allow root owned crons (most imporantly puppet runs) to still run as intended
# for more details, see crontab(1)
file { '/etc/cron.allow':
    ensure  => $is_active.bool2str('absent', 'file'),
    content => "root\n",
}

That does not work. /etc/cron.allow controls who can use the crontab command, but the cron daemon itself ignores it.