I do lots of things. See https://www.mediawiki.org/wiki/User:Majavah.
Fediverse: https://mastodon.technology/@taavi
User Details
User Details
- User Since
- Feb 24 2019, 3:58 PM (169 w, 3 d)
- Availability
- Available
- IRC Nick
- taavi
- LDAP User
- Majavah
- MediaWiki User
- Majavah [ Global Accounts ]
Today
Today
Majavah edited projects for T303359: Remove items from Meta-Wiki page [[Special:Contact/requestlicense]], added: Wikimedia-Site-requests, WikimediaMessages; removed MediaWiki-extensions-ContactPage.
Majavah added a comment to T304328: Move Termbox SSR for Beta Wikidata into deployment-prep project.
This new instance is failing to run Puppet:
taavi@deployment-termbox-ssr:~$ sudo run-puppet-agent Warning: Unable to fetch my node definition, but the agent run will continue: Warning: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain): [self signed certificate in certificate chain for /CN=Puppet CA: deployment-puppetmaster03.deployment-prep.eqiad.wmflabs] Info: Retrieving pluginfacts Error: /File[/var/lib/puppet/facts.d]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain): [self signed certificate in certificate chain for /CN=Puppet CA: deployment-puppetmaster03.deployment-prep.eqiad.wmflabs] Error: /File[/var/lib/puppet/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet:///pluginfacts: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain): [self signed certificate in certificate chain for /CN=Puppet CA: deployment-puppetmaster03.deployment-prep.eqiad.wmflabs] Info: Retrieving plugin Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain): [self signed certificate in certificate chain for /CN=Puppet CA: deployment-puppetmaster03.deployment-prep.eqiad.wmflabs] Error: /File[/var/lib/puppet/lib]: Could not evaluate: Could not retrieve file metadata for puppet:///plugins: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain): [self signed certificate in certificate chain for /CN=Puppet CA: deployment-puppetmaster03.deployment-prep.eqiad.wmflabs] Error: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain): [self signed certificate in certificate chain for /CN=Puppet CA: deployment-puppetmaster03.deployment-prep.eqiad.wmflabs] Warning: Not using cache on failed catalog Error: Could not retrieve catalog; skipping run Error: Could not send report: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain): [self signed certificate in certificate chain for /CN=Puppet CA: deployment-puppetmaster03.deployment-prep.eqiad.wmflabs]
Please fix. All deployment-prep instances must be fully configured via Puppet and not by hand / separate Ansible cookbooks.
Majavah changed E614: ArchCom RFC Meeting: Reading List service (2017-06-14, #wikimedia-office) to repeat until Tue, May 24, 9:00 PM.
Yesterday
Yesterday
Majavah added a comment to T308989: Refill tool stuck "waiting for an available worker".
If the worker is using a custom k8s deployment, consider configuring liveliness/readiness probes to make kubernetes restart the container when it gets stuck.
Mon, May 23
Mon, May 23
Majavah closed T308955: Maintain kubeusers duplicates context and certificate entries in kubeconfig if the context is not named toolforge as Resolved.
Fixed so maintain-kubeusers won't generate any new broken configs.
user: fix renewals on paws / toolsbeta
Majavah added a comment to T308995: wikibugs has stopped showing phab/gerrit comments on IRC as of 2022-05-22Z17:00.
This looks very similar to T304180 and T291129.
2022-05-23 13:13:42,815 - irc3.wikibugs - DEBUG - Register plugin 'irc3.plugins.ctcp.CTCP' 2022-05-23 13:13:42,826 - irc3.wikibugs - DEBUG - Register plugin 'irc3.plugins.autojoins.AutoJoins' 2022-05-23 13:13:42,859 - irc3.wikibugs - DEBUG - Register plugin 'irc3.plugins.sasl.Sasl' 2022-05-23 13:13:42,924 - irc3.wikibugs - DEBUG - Starting wikibugs... 2022-05-23 13:13:43,207 - irc3.wikibugs - DEBUG - Connected 2022-05-23 13:13:43,208 - irc3.wikibugs - DEBUG - CONNECT ping-pong () 2022-05-23 13:14:24,806 - irc3.wikibugs - CRITICAL - connection lost (139787222388544): None 2022-05-23 13:14:24,809 - irc3.wikibugs - CRITICAL - closing old transport (139787222388544) 2022-05-23 13:14:26,812 - irc3.wikibugs - DEBUG - Starting wikibugs... 2022-05-23 13:14:27,360 - irc3.wikibugs - DEBUG - Connected 2022-05-23 13:14:27,361 - irc3.wikibugs - DEBUG - CONNECT ping-pong () 2022-05-23 13:15:10,604 - irc3.wikibugs - CRITICAL - connection lost (139787222129536): None 2022-05-23 13:15:10,606 - irc3.wikibugs - CRITICAL - closing old transport (139787222129536) 2022-05-23 13:15:12,608 - irc3.wikibugs - DEBUG - Starting wikibugs... 2022-05-23 13:15:12,884 - irc3.wikibugs - DEBUG - Connected 2022-05-23 13:15:12,885 - irc3.wikibugs - DEBUG - CONNECT ping-pong () 2022-05-23 13:15:56,808 - irc3.wikibugs - CRITICAL - connection lost (139787222129824): None 2022-05-23 13:15:56,810 - irc3.wikibugs - CRITICAL - closing old transport (139787222129824) 2022-05-23 13:15:58,813 - irc3.wikibugs - DEBUG - Starting wikibugs... 2022-05-23 13:15:59,034 - irc3.wikibugs - DEBUG - Connected 2022-05-23 13:15:59,035 - irc3.wikibugs - DEBUG - CONNECT ping-pong () 2022-05-23 13:16:40,593 - irc3.wikibugs - CRITICAL - connection lost (139787222129920): None 2022-05-23 13:16:40,594 - irc3.wikibugs - CRITICAL - closing old transport (139787222129920) 2022-05-23 13:16:42,597 - irc3.wikibugs - DEBUG - Starting wikibugs... 2022-05-23 13:16:42,869 - irc3.wikibugs - DEBUG - Connected 2022-05-23 13:16:42,870 - irc3.wikibugs - DEBUG - CONNECT ping-pong ()
Sun, May 22
Sun, May 22
user: handle empty kubeconfig files
Sat, May 21
Sat, May 21
Majavah added a comment to T308895: GlobalRename not renaming some accounts.
Suprisingly this doesn't seem to have caused any unattached local accounts, since the renameuser_status protections in LocalRenameJob prevented the actual renames from happening on the wiki where the rename was requested from.
Majavah placed T308914: Display "Page Not Found" for incorrect request up for grabs.
Majavah added a comment to T308895: GlobalRename not renaming some accounts.
@Zabe I wonder if this can be caused by https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/774972? That could explain the Username@somewiki suffix that we're seeing in the rename logs?
Wed, May 18
Wed, May 18
Majavah added a comment to T308381: toolforge: Scrape Kubernetes controller-manager and apiserver metrics into Prometheus.
Useful reading:
Majavah added a comment to T308381: toolforge: Scrape Kubernetes controller-manager and apiserver metrics into Prometheus.
Majavah added a comment to T308682: Cleanup two LDAP users with invalid `cn` attributes.
I'm happy to fix it myself if it helps, but thought it might be best to simply create a ticket and tag it with LDAP and SRE to begin with.
Majavah added a comment to T308682: Cleanup two LDAP users with invalid `cn` attributes.
Those are not invalid values, those are just people whose usernames contain non-ASCII characters. Our existing stack fully supports them, and I'd argue that any software that does not like non-ascii values in usernames is bugged and should be fixed.
Majavah added a comment to T305847: Migrate SRE paging alerts off Icinga and to Alertmanager.
How was the above list generated? It's missing WMCS paging alerts, AIUI everything sent to the wmcs-team contact group is paging even if it has page => false set.
Majavah added a comment to P27926 Error on first puppet run (went away on second run).
Anyhow our work in T308601: Puppet fails on new cloud-vps VMs (with new base images) due to wanting /usr/local/lib/nagios/plugins should fix this issue entirely so probably not worth it to figure out what caused this
Majavah added a comment to P27926 Error on first puppet run (went away on second run).
Was this on a cloud vps vm or a production host?
Add dummy authdns keys to fix PCC
Toolforge or Cloud VPS doesn't currently offer a managed Prometheus-style monitoring or alerting service. There have been talks of building one for a while, but so far other projects have consumed our very limited engineering resources.
Tue, May 17
Tue, May 17
Dzahn awarded T308486: openstack-browser stopped showing puppet classes in use a Like token.
Majavah closed T301993: [toolsdb] Enable gtid to help replication recovery, a subtask of T301949: ToolsDB upgrade => Bullseye, MariaDB 10.4, as Resolved.
Looks good:
MariaDB [(none)]> show slave status\G
[...]
Using_Gtid: No
Gtid_IO_Pos:
[...]Majavah closed T308486: openstack-browser stopped showing puppet classes in use, a subtask of T274666: Add keystone auth middleware to the puppet enc api, as Resolved.
Majavah committed R2073:81343c373338: puppetclasses: Fix keystone authentication (authored by Majavah).
puppetclasses: Fix keystone authentication
Mon, May 16
Mon, May 16
Majavah edited projects for T308486: openstack-browser stopped showing puppet classes in use, added: Tool-openstack-browser; removed Toolforge.
Majavah closed T306494: Some of parameters in MediaWiki:Blockedtext are not parsed on Special:CreateAccount as Resolved.
The fix will be included in this week's train.
Majavah added a comment to T301993: [toolsdb] Enable gtid to help replication recovery.
This is clouddb1002 (secondary) after setting gtid_domain_id on both servers:
MariaDB [(none)]> SELECT @@GLOBAL.gtid_slave_pos; +------------------------------------------------------+ | @@GLOBAL.gtid_slave_pos | +------------------------------------------------------+ | 0-2886731673-33519859088,2886731673-2886731673-18688 | +------------------------------------------------------+ 1 row in set (0.00 sec)
Is it intentional that there are two entries and the first one starts with 0-?
Majavah updated the task description for T308013: Assign SPDX headers to puppet.git.
Sun, May 15
Sun, May 15
Majavah set Due Date to Thu, Jun 2, 9:00 PM on T308402: toolforge: Refresh certs that are not controlled by kubeadm (mid 2022 edition).
Majavah triaged T308402: toolforge: Refresh certs that are not controlled by kubeadm (mid 2022 edition) as High priority.
Sat, May 14
Sat, May 14
Majavah added a comment to T308388: User unable to merge accounts.
This seems to be a display bug with Special:CentralAuth: we have some 'corrupted' localuser rows for this specific account that look otherwise good but have lu_attached_method set as NULL. SpecialCentralAuth uses the centralauth-admin-unattached message as a placeholder for missing data.
Majavah added a comment to T292945: Automate kubeadm config change deployment.
A simple automation for step 3 would be to simply to replace the 3 control nodes with new ones as the cluster join process creates the manifest files from what's stored in the config maps.
Looks like the Kubernetes cronjob scheduler may be getting overloaded at midnight given how many tools are running jobs at that point. I've increased the scheduling tolerance of your jobs to make Kubernetes start your jobs even if that means it'll be a little off the scheduled date. However, a better solution would be to, if possible, run hourly/daily jobs at a random time (say, 18:37 daily) instead of exactly midnight or top of the hour.
Fri, May 13
Fri, May 13
Majavah added a comment to T308283: Beta Cluster Tech Decision Forum.
Majavah added a comment to T308283: Beta Cluster Tech Decision Forum.
Am I reading this correctly that the proposed Beta replacement would be directly using and updating production data and also share the production bottlenecks (such as databases)?
Majavah renamed T307943: Update Kubernetes clusters to v1.23 from Update Kubernets clusters to v1.23 to Update Kubernetes clusters to v1.23.
Majavah closed T305317: Request membership in extension-HidePrefix group for Zoranzoki21 as Resolved.
Done.
Legoktm awarded T308203: Create a Toolschecker check for K8s cronjobs a The World Burns token.
Thu, May 12
Thu, May 12
Majavah updated the task description for T308203: Create a Toolschecker check for K8s cronjobs.
Looks like this was caused by an unrelated Puppet change. Fixed by updating the hiera data in Horizon.
Majavah closed T308189: Toolforge jobs stopped getting scheduled around the same time as the Toolforge k8s cluster upgrade as Resolved.
Everything seems to be working again properly. I've filed some actionables and marked those as subtasks, so closing this task.
Seems to have worked fine. Tentatively closing.
Majavah added a comment to T308205: Re-enable CronJobControllerV2.
Majavah triaged T308204: toolforge-jobs should set startingDeadlineSeconds by default as High priority.
Majavah raised the priority of T308203: Create a Toolschecker check for K8s cronjobs from High to Needs Triage.
Wed, May 11
Wed, May 11
Majavah updated the task description for T295190: Upgrade all third-party Toolforge Kubernetes components to versions supporting Kubernetes 1.22.
Majavah added a parent task for T308172: Upgrade PAWS to Kubernetes 1.21: T286856: Upgrade Toolforge Kubernetes to latest 1.22.
Majavah added a parent task for T282942: Upgrade Toolforge Kubernetes to latest 1.21: T308172: Upgrade PAWS to Kubernetes 1.21.
Majavah closed T282942: Upgrade Toolforge Kubernetes to latest 1.21, a subtask of T286856: Upgrade Toolforge Kubernetes to latest 1.22, as Resolved.
Majavah added a comment to T308102: Delete Cloud VPS projects ores and ores-staging.
Hey! I still see a VM in the ores project, is it ok to delete that too?
taavi@cloudcontrol1004 ~ $ os server list --project ores +--------------------------------------+-------------+--------+----------------------------------------+--------------------------------------------+-----------------------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+-------------+--------+----------------------------------------+--------------------------------------------+-----------------------+ | c0252f8f-6953-4d08-9ffc-57f8dcf7ba18 | calbon-test | ACTIVE | lan-flat-cloudinstances2b=172.16.0.200 | debian-10.0-buster (deprecated 2020-10-16) | g2.cores1.ram2.disk20 | +--------------------------------------+-------------+--------+----------------------------------------+--------------------------------------------+-----------------------+
Majavah renamed T308102: Delete Cloud VPS projects ores and ores-staging from Delete Horizon projects ores and ores-staging to Delete Cloud VPS projects ores and ores-staging.
Tue, May 10
Tue, May 10
Majavah added a comment to T290494: Revisit Toolforge automated package updates and version pinnings.
Majavah added a comment to T290494: Revisit Toolforge automated package updates and version pinnings.
Oh, the kernel pinnings don't work at all. This is an older host that does not
taavi@tools-k8s-worker-42:~ $ uname -a Linux tools-k8s-worker-42 4.19.0-14-amd64 #1 SMP Debian 4.19.171-2 (2021-01-30) x86_64 GNU/Linux taavi@tools-k8s-control-2:~ $ kubectl sudo get node tools-k8s-worker-42 -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME tools-k8s-worker-42 Ready,SchedulingDisabled <none> 2y72d v1.20.11 172.16.1.74 <none> Debian GNU/Linux 10 (buster) 4.19.0-14-amd64 docker://20.10.8
Majavah added a comment to T290494: Revisit Toolforge automated package updates and version pinnings.
Just enabled unattended-upgrades. That still leaves the apt pinnings. Also note that the kernel pinning does not work for new hosts (all bullseye hosts and some buster ones) use the cloud kernel variants, which our pinnings don't seem to apply to.
Mon, May 9
Mon, May 9
Majavah added a comment to T127607: Fix canonical namespaces for rowiki.
taavi@mwmaint1002 ~ $ mwscript namespaceDupes.php --wiki rowiki --fix 0 pages to fix, 0 were resolvable.
Fri, May 6
Fri, May 6
Majavah added a project to T307800: mysql collation utf8mb4_0900_ai_ci does not work correctly: Wikimedia-Rdbms.
Done. Sorry for the delay!
Majavah removed a project from T307768: Outreachdashboard application Internal Sever error: Wikimedia-production-error.
Majavah added a comment to T307648: Audit database usage of GlobalBlocking extension.
Thu, May 5
Thu, May 5
Looks like I managed to sneak in a bug in the last webservice update which broke starting all Java web services. Fixed now, sorry about that!
Majavah edited projects for T307693: Bad Request error when starting a Kubernetes service, added: Toolforge; removed Tools.