$ sudo ls -lh /home/rluts/2fa-reset-request.txt -rw------- 1 rluts wikidev 42 Apr 22 11:57 /home/rluts/2fa-reset-request.txt $ sudo cat /home/rluts/2fa-reset-request.txt https://phabricator.wikimedia.org/T363092
$ mwscript extensions/OATHAuth/maintenance/disableOATHAuthForUser.php --wiki=labswiki 'RLuts' Two-factor authentication disabled for RLuts.
The root problem here seems to be:
[step-build] 2024-04-19T08:31:50.877313171Z Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::19). - connect (101: Network is unreachable)
For currently unexplained reasons, the DNS resolver used by the builder process resolved archive.ubuntu.com to an IPv6 address. The network in Cloud VPS & Toolforge is IPv4 only, so this in turn led to nothing working.
There aren't any open tasks in the search, so somehow this came to fruition.
Build service probably makes this investigation and implementation obsolete in that we are actively directing folks away from the shared containers and towards building their own.
GitLab EE has mechanisms for jobs to authenticate to a Hashicorp Vault service and fetch secrets. It is not completely clear from reading https://docs.gitlab.com/ee/ci/secrets/ if there is really any support for this in CE or not.
One concrete use case for this feature would be installing toolforge-*-cli packages in a buildservice managed container. This is an idea that @Anomie and I have discussed as a potential partial solution for T356377: [toolforge] simplify calling the different toolforge apis from within the containers / T321919: Figure out and document how to call the Kubernetes API as your tool user from inside a pod.
Play around with it via curl --no-buffer -H 'Accept: text/event-stream' 'https://gitlab-webhooks.toolforge.org/sse/'
I was able to create R3385 bd808-testing-T362909 via the Phabricator web UI. Is it only the Conduit API that is failing? Are the failures intermittent?
This is basically a missing integration with the system that @Andrew built for T170355: Figure out process for deleting an unused tool (wikitech docs). There is also nothing currently deleting or otherwise marking GitLab projects and their associated Diffusion mirrors as abandoned or archived when the tool that created them is deleted.
The tool in question is https://toolsadmin.wikimedia.org/tools/id/api-docs. Things are definitely goofy here with there being no toolinfo record. There are several related log events in logstash, and they seem to start with this error that seems to echo the name validation failure.
Traceback (most recent call last): File "/srv/app/striker/tools/views/tool.py", line 83, in create tool = utils.create_tool(form.cleaned_data["name"], req.user) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/srv/app/striker/tools/utils.py", line 86, in create_tool sudoers.save() File "/opt/lib/poetry/striker-2uZo5AhP-py3.11/lib/python3.11/site-packages/django/db/models/base.py", line 743, in save self.save_base(using=using, force_insert=force_insert, File "/opt/lib/poetry/striker-2uZo5AhP-py3.11/lib/python3.11/site-packages/django/db/models/base.py", line 780, in save_base updated = self._save_table( ^^^^^^^^^^^^^^^^^ File "/opt/lib/poetry/striker-2uZo5AhP-py3.11/lib/python3.11/site-packages/ldapdb/models/base.py", line 120, in _save_table connection.add_s(new_dn, new_values) File "/opt/lib/poetry/striker-2uZo5AhP-py3.11/lib/python3.11/site-packages/ldapdb/backends/ldap/base.py", line 317, in add_s return cursor.connection.add_s(dn, modlist) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/lib/poetry/striker-2uZo5AhP-py3.11/lib/python3.11/site-packages/ldap/ldapobject.py", line 236, in add_s return self.add_ext_s(dn,modlist,None,None) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/lib/poetry/striker-2uZo5AhP-py3.11/lib/python3.11/site-packages/ldap/ldapobject.py", line 1012, in add_ext_s return self._apply_method_s(SimpleLDAPObject.add_ext_s,*args,**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/lib/poetry/striker-2uZo5AhP-py3.11/lib/python3.11/site-packages/ldap/ldapobject.py", line 972, in _apply_method_s return func(self,*args,**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/lib/poetry/striker-2uZo5AhP-py3.11/lib/python3.11/site-packages/ldap/ldapobject.py", line 222, in add_ext_s resp_type, resp_data, resp_msgid, resp_ctrls = self.result3(msgid,all=1,timeout=self.timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/lib/poetry/striker-2uZo5AhP-py3.11/lib/python3.11/site-packages/ldap/ldapobject.py", line 543, in result3 resp_type, resp_data, resp_msgid, decoded_resp_ctrls, retoid, retval = self.result4( ^^^^^^^^^^^^^ File "/opt/lib/poetry/striker-2uZo5AhP-py3.11/lib/python3.11/site-packages/ldap/ldapobject.py", line 553, in result4 ldap_result = self._ldap_call(self._l.result4,msgid,all,timeout,add_ctrls,add_intermediates,add_extop) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/lib/poetry/striker-2uZo5AhP-py3.11/lib/python3.11/site-packages/ldap/ldapobject.py", line 128, in _ldap_call result = func(*args,**kwargs) ^^^^^^^^^^^^^^^^^^^^ ldap.ALREADY_EXISTS: {'msgtype': 105, 'msgid': 17, 'result': 68, 'desc': 'Already exists', 'ctrls': []}
Current lag is very likely caused by T352010: Gradually drop old pagelinks columns as shown on https://wikitech.wikimedia.org/wiki/Map_of_database_maintenance which is linked to by https://replag.toolforge.org/ prominently.
As explained briefly in the irc snippet, I have a use case where Tool-bridgebot uses a 3rd party golang component (matterbridge) which expects to read configuration from a static file. I want to move this tool from a legacy hand built deployment to a modern stack of a toolforge jobs managed deployment using a toolforge build generated image. To do so I need to replace the current chmod 0600 config file in the tool's $HOME with configuration that uses toolforge envvars for transmitting secrets into the running container.
https://wikitech.wikimedia.org/wiki/Help:Toolforge/Jobs_framework does this work now.
I was doing some other cleanup of pending GitLab accounts and I think I rejected the https://gitlab.wikimedia.org/safan41 account. Logging in to GitLab again will restore the account to pending status.
I was doing some other cleanup of pending GitLab accounts and I think I may have rejected the https://gitlab.wikimedia.org/sethabathaba account. Attempting to login again will restore the account to pending status.
Rendering of T1152#9715566 via https://web.libera.chat/. Note that this client does not render the strikethrough style, but it also does not show any strange formatting characters:
another closed task comment
Rendering of T1152#9715558 in my Textual client with some additional styling triggered by my irc nick being part of the the message:
test comment on closed task
I'm going to switch the formatting from grey foreground to strikethrough text style. The strikethrough will only be visible to some IRC clients, but is expected to be more intuitive as it matches the Phabricator styling for closed tasks. HexChat, IRCCloud, The Lounge, & Textual are explicitly listed at https://defs.ircdocs.horse/info/formatting as supporting the strikethrough style.
I really need to work on this again. The bot breaks a lot and @LucasWerkmeister is amazing for noticing and restarting it. This really just needs me to take a little time to understand the golang build chain used in the upstream project so I can easily iterate on a fix like the one @Platonides proposed years ago in T305487#7915998.
The open and assigned task nag script has complained that I haven't closed this task, so whatever it is closed now. Maybe someday I'll write up the reasons I added this feature to blubber. :shrug:
Getting comment, pipeline, and job events out of GitLab would be a nice addition to the data that is currently collected by gitlab-webhooks using the system hooks integration. Unfortunately these are not as easy for us to manage centrally as they must be configured at the project level in GitLab CE. Group level integration for these events is only possible with a paid subscription.
Quick mockup using jq of data likely to be used in building messages:
$ grep '"object_kind": "merge_request"' events.log | grep "bd808" | jq '{"action": .object_attributes.action, "user": .user.username, "message": .object_attributes.title, "repo": .object_attributes.target.path_with_namespace, "branch": .object_attributes.target_branch, "url": .object_attributes.url, "owner": .object_attributes.author_id, "state": .object_attributes.state }' { "action": "open", "user": "bd808", "message": "Relicense under GPL-3.0-or-later", "repo": "toolforge-repos/wikibugs2", "branch": "main", "url": "https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_request s/22", "owner": 14, "state": "opened" } { "action": "approved", "user": "bd808", "message": "Relicense under GPL-3.0-or-later", "repo": "toolforge-repos/wikibugs2", "branch": "main", "url": "https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/22", "owner": 14, "state": "opened" } { "action": "merge", "user": "bd808", "message": "Relicense under GPL-3.0-or-later", "repo": "toolforge-repos/wikibugs2", "branch": "main", "url": "https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/22", "owner": 14, "state": "merged" }
The 2014-04-08 scap triggered outage that "earned" me the shirt (and my Chaos Monkey badge):
In 2016 I started making a sticker version because 1) easier to carry places, 2) no worries about not having the correct size, 3) cheaper.
So bd808 made one: http://oi61.tinypic.com/2ryj5kz.jpg
tinypic.com has been lost to the sands of time. That photo was:
The photo of the just arrived shirts that I posted to twitter on 2014-09-15:
test
tools.wikibugs-testing@tools-bastion-12:~$ toolforge build start --ref work/bd808/expect-phorge-api-errors https://gitlab.wikimedia.org/toolforge-repos/wikibugs2 Waiting for the logs... if the build just started this might take a minute [place-tools] 2024-04-11T17:49:59.348045941Z 2024/04/11 17:49:59 Copied /ko-app/entrypoint to /tekton/bin/entrypoint [place-tools] failed to create fsnotify watcher: too many open files [step-init] 2024-04-11T17:49:59.913177243Z 2024/04/11 17:49:59 Setup /step directories [step-init] failed to create fsnotify watcher: too many open files [place-scripts] 2024-04-11T17:50:01.125871795Z 2024/04/11 17:50:01 Decoded script /tekton/scripts/script-0-vmtp5 [place-scripts] 2024-04-11T17:50:01.150840971Z 2024/04/11 17:50:01 Decoded script /tekton/scripts/script-2-wldbl [place-scripts] 2024-04-11T17:50:01.187348966Z 2024/04/11 17:50:01 Decoded script /tekton/scripts/script-3-l5vh7 [place-scripts] 2024-04-11T17:50:01.214228628Z 2024/04/11 17:50:01 Decoded script /tekton/scripts/script-9-s6s4t [place-scripts] 2024-04-11T17:50:01.237079271Z 2024/04/11 17:50:01 Decoded script /tekton/scripts/script-10-6th52 [place-scripts] 2024-04-11T17:50:01.270148540Z 2024/04/11 17:50:01 Decoded script /tekton/scripts/script-11-zcrvx [place-scripts] 2024-04-11T17:50:01.304439329Z 2024/04/11 17:50:01 Decoded script /tekton/scripts/script-12-4gvkt [place-scripts] failed to create fsnotify watcher: too many open files [step-clone] failed to create fsnotify watcher: too many open files [step-copy-builder-to-tmp] 2024-04-11T17:50:02.407982038Z 2024/04/11 17:50:02 warning: unsuccessful cred copy: ".docker" from "/tekton/creds" to "/tekton/home": unable to open destination: open /tekton/home/.docker/config.json: permission denied [step-copy-builder-to-tmp] failed to create fsnotify watcher: too many open files [step-prepare] failed to create fsnotify watcher: too many open files [step-inject-buildpacks] failed to create fsnotify watcher: too many open files [step-copy-stack-toml] 2024-04-11T17:50:03.342859273Z 2024/04/11 17:50:03 warning: unsuccessful cred copy: ".docker" from "/tekton/creds" to "/tekton/home": unable to open destination: open /tekton/home/.docker/config.json: permission denied [step-copy-stack-toml] failed to create fsnotify watcher: too many open files [step-detect] failed to create fsnotify watcher: too many open files [step-analyze] 2024-04-11T17:50:04.254967516Z 2024/04/11 17:50:04 warning: unsuccessful cred copy: ".docker" from "/tekton/creds" to "/tekton/home": unable to open destination: open /tekton/home/.docker/config.json: permission denied [step-analyze] failed to create fsnotify watcher: too many open files [step-restore] 2024-04-11T17:50:04.688674016Z 2024/04/11 17:50:04 warning: unsuccessful cred copy: ".docker" from "/tekton/creds" to "/tekton/home": unable to open destination: open /tekton/home/.docker/config.json: permission denied [step-restore] failed to create fsnotify watcher: too many open files [step-build] failed to create fsnotify watcher: too many open files [step-fix-nested-procfile-launcher] failed to create fsnotify watcher: too many open files [step-fix-permissions] failed to create fsnotify watcher: too many open files [step-export] 2024-04-11T17:50:06.064284001Z 2024/04/11 17:50:06 warning: unsuccessful cred copy: ".docker" from "/tekton/creds" to "/tekton/home": unable to open destination: open /tekton/home/.docker/config.json: permission denied [step-export] failed to create fsnotify watcher: too many open files [step-results] failed to create fsnotify watcher: too many open files
3rd retry (4th call overall) worked.
I thought that was part of {T316419}, but apparently I was mistaken. +1 from me.
I have a hacky script for this kind of stuff at cloudweb1003:/home/bd808/projects/striker/debug-striker.sh. It was inspired by the hacky script I had to make to manage Toolhub.
#!/usr/bin/env bash # Run the Striker container locally. # Typical usage: # ./debug-striker.sh -it --entrypoint=/bin/bash set -euxo pipefail
You seem to be reporting that your computer does not have a font installed that supports the ≯ unicode glyph (code point 226F).
I'm not quite sure how this problem can be solved in a general sense. We have historically avoided shipping web fonts and instead expect users to have local fonts installed to support unicode content.
I talked with @dancy on irc and have decided to try this variant:
test
test
We've gotten by for ~10 years now without irc notifs for pastes, etc and it looks like basically no one has missed that functionality. Declining without prejudice on that basis. If someone can present a reasonable use case and output format for notifs for things other than tasks then we should look into that.
With Gerrit events and the likely implementation of GitLab events both being unable to time travel it doesn't seem important to add this support to the Phorge client at this time.
The bot has been running for a couple of weeks with only the default flood protections from irc3.IrcBot + ZNC mentioned in T359753#9618035. There have not been any issues in this time which I think reasonably shows that this is working. If we need to tune ZNC or irc3.IrcBot in the future as more message sources and sinks are added we can do so as desired.
test