Page MenuHomePhabricator

bd808 (Bryan Davis)
Principal Software Engineer, Technical EngagementAdministrator

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Wednesday

  • Clear sailing ahead.

User Details

User Since
Oct 3 2014, 2:36 PM (403 w, 2 d)
Roles
Administrator
Availability
Available
IRC Nick
bd808
LDAP User
BryanDavis
MediaWiki User
BDavis (WMF) [ Global Accounts ]

I'm BDavis (WMF) on wiki, bd808 on irc, and BryanDavis on Gerrit and Wikitech.

I've got a thing for 🦄s. Don't judge.

I work for or provide services to the Wikimedia Foundation, but this is my only Phabricator account. Edits, statements, or other contributions made from this account are my own, and may not reflect the views of the Foundation.

Recent Activity

Fri, Jun 24

bd808 moved T311297: activity.toolforge.org loads third-party content for d3 from Incoming to Privacy / Other cleanup on the Toolforge-standards-committee board.
Fri, Jun 24, 3:45 PM · Toolforge-standards-committee, Privacy, Tools
bd808 added a project to T311297: activity.toolforge.org loads third-party content for d3: Toolforge-standards-committee.
Fri, Jun 24, 3:44 PM · Toolforge-standards-committee, Privacy, Tools
bd808 added a parent task for T311297: activity.toolforge.org loads third-party content for d3: T103934: Convert all Toolforge tools to use cdnjs for static libraries and fonts.
Fri, Jun 24, 3:44 PM · Toolforge-standards-committee, Privacy, Tools
bd808 added a subtask for T103934: Convert all Toolforge tools to use cdnjs for static libraries and fonts: T311297: activity.toolforge.org loads third-party content for d3.
Fri, Jun 24, 3:44 PM · Toolforge-standards-committee, Tools, Epic
bd808 added a comment to T311302: Request increased quota for devtools Cloud VPS project.

+1

Fri, Jun 24, 12:41 PM · cloud-services-team (Kanban), Release-Engineering-Team, Cloud-VPS (Quota-requests)
bd808 moved T311302: Request increased quota for devtools Cloud VPS project from Inbox to Clinic Duty on the cloud-services-team (Kanban) board.
Fri, Jun 24, 12:40 PM · cloud-services-team (Kanban), Release-Engineering-Team, Cloud-VPS (Quota-requests)
bd808 added a project to T311302: Request increased quota for devtools Cloud VPS project: cloud-services-team (Kanban).
Fri, Jun 24, 12:40 PM · cloud-services-team (Kanban), Release-Engineering-Team, Cloud-VPS (Quota-requests)

Wed, Jun 22

bd808 changed the subtype of T310724: UnicodeDecodeError on poetry run mkdocs --verbose build from "Task" to "Bug Report".
Wed, Jun 22, 1:39 PM · Patch-For-Review, Wikimedia-Developer-Portal
bd808 moved T310724: UnicodeDecodeError on poetry run mkdocs --verbose build from Inbox to Implementation on the Wikimedia-Developer-Portal board.
Wed, Jun 22, 1:38 PM · Patch-For-Review, Wikimedia-Developer-Portal
bd808 moved T310727: Add OpenGraph meta tags from Inbox to Content & design on the Wikimedia-Developer-Portal board.
Wed, Jun 22, 1:38 PM · Wikimedia-Developer-Portal
bd808 moved T311134: Setup monitoring dashboard for developer.wikimedia.org from Inbox to Implementation on the Wikimedia-Developer-Portal board.
Wed, Jun 22, 1:38 PM · User-bd808, Wikimedia-Developer-Portal
bd808 created T311134: Setup monitoring dashboard for developer.wikimedia.org.
Wed, Jun 22, 1:34 PM · User-bd808, Wikimedia-Developer-Portal
bd808 added a comment to T297140: New Service Request: developer-portal.

This has been deployed for some time so I moved it to the Done column, but I see 2 remaining unchecked items in the Checklist section of the task

  • Monitoring dashboard
  • Integration and Acceptance tests

@bd808, any news on those?

Wed, Jun 22, 1:30 PM · Goal, serviceops, Wikimedia-Developer-Portal, Service-deployment-requests
bd808 added a comment to T311087: Fix locally importing Bitergia database dumps.

@Aklapper do you think this is something that it would be technically possible to automate? I'm wondering specifically if we could create an dedicated account on the bitergia side to be used for a scripted export of the data and also create a Cloud VPS project to host the local dump and any data cleaning/viewing scripts that might be useful.

Wed, Jun 22, 4:24 AM · Developer-Advocacy (Apr-Jun 2022), wikimedia.biterg.io
bd808 added a comment to T310727: Add OpenGraph meta tags.

Looking at the Twitter docs, it seems like svg files aren't supported. How about something like this instead of one of the illustrations:

test2.png (800×1 px, 30 KB)

Wed, Jun 22, 4:16 AM · Wikimedia-Developer-Portal

Tue, Jun 21

bd808 closed T310493: Renamed tool diffusion repository failed to be updated in toolsadmin.wikimedia.org as Declined.

I am going to close this as declined as any work on Striker for git repos should really be put into T296893: Replace Diffusion integration with Gitlab integration in Striker (toolsadmin) at this point. I also don't think we will ever put in the effort to find a way to track arbitrary changes made from external workflows. The change history at https://phabricator.wikimedia.org/source/crystal/manage/history/ shows manual changes to just about everything that Striker originally setup. This is fine by policy, but not something that is considered "supported" as a workflow.

Tue, Jun 21, 3:47 PM · Striker, Diffusion
bd808 moved T284299: Install setuptools >= 49.4.0 on Toolforge from Backlog to Job grid on the Toolforge (Software install/update) board.
Tue, Jun 21, 3:14 PM · Toolforge (Software install/update), cloud-services-team (Kanban), Pywikibot
bd808 moved T306144: Install wget2 on Toolforge hosts from Backlog to Job grid on the Toolforge (Software install/update) board.
Tue, Jun 21, 3:12 PM · Toolforge (Software install/update)
bd808 closed T308988: Add make and gdal package to tf-bullseye-std toolforge-jobs image, a subtask of T308886: Hacking on bldrwnsch.toolforge.org, as Declined.
Tue, Jun 21, 3:10 PM · Wikimedia-Hackathon-2022
bd808 closed T308988: Add make and gdal package to tf-bullseye-std toolforge-jobs image as Declined.

Gdal is a specialized application which is not currently a good fit for any Toolforge Kubernetes container. We are actively working on solutions for T194332: [Epic] Make Toolforge a proper platform as a service with push-to-deploy and build packs which will eventually provide a solution for tools which need to install custom applications and libraries in their containers.

Tue, Jun 21, 3:10 PM · Toolforge (Software install/update), Wikimedia-Hackathon-2022
bd808 added a comment to T310493: Renamed tool diffusion repository failed to be updated in toolsadmin.wikimedia.org.

How would/should/could Striker know that you renamed the repository with a direct management workflow?

Tue, Jun 21, 3:08 PM · Striker, Diffusion
bd808 added a comment to T306458: Separate search content by language.

T307141: Multi-lingual search is not consistently multi-lingual is related and might imply that some locales would not have any search at all once this task is resolved.

Tue, Jun 21, 1:49 AM · Wikimedia-Developer-Portal
bd808 added a comment to T310534: developer.wikimedia.org accessibility: language selector menu improperly structured for screen readers.

When someone figures out what HTML changes are needed for this we should make sure to submit a patch to the upstream at https://github.com/squidfunk/mkdocs-material. We can also patch locally via theme overrides, but it would be ideal for this to be upstreamed.

Tue, Jun 21, 1:41 AM · Accessibility, Wikimedia-Developer-Portal
bd808 added a comment to T310795: Abolish labtestwikitech.

@bd808 I'm interested in your thoughts about this -- is that wiki good for anything that I'm forgetting? I would say 'striker 2fa' except in theory we have better test/dev options for striker these days.

Tue, Jun 21, 1:09 AM · cloud-services-team (Kanban), Cloud-VPS

Fri, Jun 3

bd808 added a comment to P29377 Tools Broken on Buster.

wikinity is an old uwsgi-python service that needs it's venv to be rebuilt. Ideally it would move to k8s too.

Fri, Jun 3, 10:13 PM
bd808 added a comment to P29377 Tools Broken on Buster.

weeklypedia is an old uwsgi-python service that needs it's venv to be rebuilt. Ideally it would move to k8s too.

Fri, Jun 3, 10:13 PM
bd808 added a comment to P29377 Tools Broken on Buster.

stereoskopie is tomcat (see P29377#124187).

Fri, Jun 3, 10:10 PM
bd808 added a comment to P29377 Tools Broken on Buster.

saami looks like it supposed to be a deno tool. Currently crashing because the startup script given in it's $HOME/service.manifest does not exist. I shut it down.

Fri, Jun 3, 10:07 PM
bd808 added a comment to P29377 Tools Broken on Buster.

ramp has a bad $HOME/.lighttpd.conf file that was last modified in 2014. Probably broken since migration to the stretch grid. I stopped the webservice.

Fri, Jun 3, 10:04 PM
bd808 added a comment to P29377 Tools Broken on Buster.

pmidtool is an old uwsgi-plain service that needs to have it's venv rebuilt for a modern python version (and ideally moved to k8s if possible).

Fri, Jun 3, 10:01 PM
bd808 added a comment to P29377 Tools Broken on Buster.

movestats is an old uwsgi-plain service that needs to have it's venv rebuilt for a modern python version (and ideally moved to k8s if possible). This tool doesn't really look like it has worked since the toolforge.org domain migration based on the routes in it's flask source file.

Fri, Jun 3, 10:00 PM
bd808 added a comment to P29377 Tools Broken on Buster.

lalm is tomcat (see P29377#124187).

Fri, Jun 3, 9:57 PM
bd808 added a comment to P29377 Tools Broken on Buster.

ensocksfinder is compiled rust. it is crashing on startup as it tries and fails to allocate ~4GiB of memory.

Listening on localhost:39677...
Loading index...
memory allocation of 4294967296 bytes failed
/data/project/ensocksfinder/socksfinder-webservice.sh: line 2: 29806 Aborted                 (core dumped) /data/project/ensocksfinder/socksfinder/target/release/socksfinder serve --hostname localhost --port $PORT /data/project/ensocksfinder/socksfinder/target/release/enwiki-latest.idx
Traceback (most recent call last):
  File "/usr/bin/webservice-runner", line 47, in <module>
    webservice.run(port)
  File "/usr/lib/python3/dist-packages/toolsws/wstypes/generic.py", line 24, in run
    cwd=self.tool.home,
  File "/usr/lib/python3.5/subprocess.py", line 271, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/data/project/ensocksfinder/socksfinder-webservice.sh']' returned non-zero exit status 134
Fri, Jun 3, 9:56 PM
bd808 added a comment to P29377 Tools Broken on Buster.

idsgen is another tomcat service. (see P29377#124174)

Fri, Jun 3, 9:51 PM
bd808 added a comment to P29377 Tools Broken on Buster.

himo was crashing for $HOME/.lighttpd.conf errors. Fixed by moving it out of the way since the content was not really useful anyway.

Fri, Jun 3, 9:42 PM
bd808 added a comment to P29377 Tools Broken on Buster.

fountain-test (and fountain) are csharp/dotnet using custom compiled interpreters which need to be rebuilt for Buster. Crashing with "No usable version of the libssl was found" error message and a core dump. I moved fountain back to the Stretch grid earlier today after community folks poking on irc and via staff friends on slack.

Fri, Jun 3, 9:30 PM
bd808 added a comment to P29377 Tools Broken on Buster.

facebook-messenger-chatbot is ruby + rackup. Bundle probably needs to be rebuilt for new version.

Fri, Jun 3, 9:25 PM
bd808 added a comment to P29377 Tools Broken on Buster.

crosswatch is a custom python tool attempting to run from a python3.4 venv built 2015

Fri, Jun 3, 9:21 PM
bd808 added a comment to P29377 Tools Broken on Buster.

clickstream-api is an ancient python2 uwsgi app that looks to have been broken for a very long time (maybe since migration to the stretch grid?). I shut it down.

Fri, Jun 3, 9:03 PM
bd808 added a comment to P29377 Tools Broken on Buster.

asurabot has a similar $HOME/.lighttpd.conf problem to cobain. Doesn't look like the tool has been doing bot things since early 2019. No crontab or k8s jobs. I shut it down.

Fri, Jun 3, 8:56 PM
bd808 added a comment to P29377 Tools Broken on Buster.

isbn & isbn-tmptest are both tomcat tools and crashing because /usr/bin/deprecated-tomcat-starter is broken on Buster. I shut isbn-tmptest down

Fri, Jun 3, 8:48 PM
bd808 added a comment to P29377 Tools Broken on Buster.

I shut cobain down. It has a $HOME/.lighttpd.conf that needs to be updated. The index page for the site has not been updated for the toolforge.org domain migration either so even when it is running without deep links there's not much there to see.

Fri, Jun 3, 8:46 PM
bd808 edited Description on Security.
Fri, Jun 3, 8:22 PM
bd808 added a parent task for T307531: Formally EOL REL1_36: T309864: Tidy up references to REL1_36 now it is EOL.
Fri, Jun 3, 4:52 PM · LibUp, Patch-For-Review, Documentation, MW-1.36-release, MediaWiki-General, translatewiki.net, Continuous-Integration-Config
bd808 added a subtask for T309864: Tidy up references to REL1_36 now it is EOL: T307531: Formally EOL REL1_36.
Fri, Jun 3, 4:52 PM · Continuous-Integration-Config, WMF-General-or-Unknown
bd808 added a comment to T309821: Buster webservice grid went BOOM!.

Things are a little better after deploying the hacked version tools-webservice. Many more webservices are running, but certainly not all. Some nodes are still being overloaded too with jobs failing due to "fork failed: Cannot allocate memory" and similar errors.

Fri, Jun 3, 4:07 AM · Patch-For-Review, cloud-services-team (Kanban), Toolforge
bd808 added a comment to T309821: Buster webservice grid went BOOM!.
tools.bd808-test2@tools-sgewebgen-10-1:~$ /usr/bin/webservice-runner --register-proxy --type lighttpd
Traceback (most recent call last):
  File "/usr/bin/webservice-runner", line 46, in <module>
    proxy.register(port)
  File "/usr/lib/python3/dist-packages/toolsws/proxy.py", line 114, in register
    register_dynamicproxy(port)
  File "/usr/lib/python3/dist-packages/toolsws/proxy.py", line 58, in register_dynamicproxy
    raise ProxyException("Port registration failed!")
toolsws.proxy.ProxyException: Port registration failed!

But on tools-proxy-06:

2022-06-03 00:28:40,686 Received request from bd808-test2 for .* to http://172.16.7.61:38033

Is the ack not getting back to the caller?

Fri, Jun 3, 12:30 AM · Patch-For-Review, cloud-services-team (Kanban), Toolforge
bd808 added a comment to T309821: Buster webservice grid went BOOM!.

Example log events on the front proxy:

$ grep cobain /var/log/proxylistener |tail -10
2022-06-03 00:08:26,308 Received request from cobain for .* to http://172.16.1.75:60743
2022-06-03 00:08:26,865 Cleaning up request from cobain for .*
2022-06-03 00:08:27,081 Received request from cobain for .* to http://172.16.1.75:33763
2022-06-03 00:08:27,449 Cleaning up request from cobain for .*
2022-06-03 00:08:27,752 Received request from cobain for .* to http://172.16.1.69:56789
2022-06-03 00:08:28,260 Cleaning up request from cobain for .*
2022-06-03 00:08:28,865 Received request from cobain for .* to http://172.16.1.69:34515
2022-06-03 00:08:29,349 Cleaning up request from cobain for .*
2022-06-03 00:08:29,637 Received request from cobain for .* to http://172.16.1.72:58491
2022-06-03 00:08:29,986 Cleaning up request from cobain for .*
Fri, Jun 3, 12:26 AM · Patch-For-Review, cloud-services-team (Kanban), Toolforge
bd808 added a comment to T309821: Buster webservice grid went BOOM!.

Emails to root@tools.wmflabs.org show that most (all?) of the failed jobs are variations of execvlp(/bin/bash, "-bash" "-c" "/usr/bin/webservice-runner --register-proxy --type lighttpd ") failing which then fires the eplilog to cleanup the proxy registration and that fails as well. Watching on the front proxy with tail -f /var/log/proxylistener I think I see both the registration and the removal happening, but apparently the python processes back on the exec nodes are not seeing the expected acks for those events?

Fri, Jun 3, 12:24 AM · Patch-For-Review, cloud-services-team (Kanban), Toolforge

Thu, Jun 2

bd808 triaged T309821: Buster webservice grid went BOOM! as High priority.
Thu, Jun 2, 11:34 PM · Patch-For-Review, cloud-services-team (Kanban), Toolforge
bd808 added a subtask for T277653: Toolforge: add Debian Buster to the grid and eliminate Debian Stretch: T309821: Buster webservice grid went BOOM!.
Thu, Jun 2, 11:33 PM · Patch-For-Review, Toolforge, cloud-services-team (Kanban)
bd808 added a parent task for T309821: Buster webservice grid went BOOM!: T277653: Toolforge: add Debian Buster to the grid and eliminate Debian Stretch.
Thu, Jun 2, 11:33 PM · Patch-For-Review, cloud-services-team (Kanban), Toolforge
bd808 updated subscribers of T309821: Buster webservice grid went BOOM!.
[22:24] bd808 figures out that tools-sgeweblight-10-1.tools.eqiad1.wikimedia.cloud is sick somehow
[22:26]  <    bd808> !log tools Rebooting tools-sgeweblight-10-1.tools.eqiad1.wikimedia.cloud. Node is full of jobs that are not tracked by grid master and failing to spawn new jobs sent by the scheduler

Not long after this I did the things from https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin#Clearing_error_state. When the queues flipped right back to E state I cried for help in -cloud-admin and got @nskaggs involved. Then I started this task.

Thu, Jun 2, 11:17 PM · Patch-For-Review, cloud-services-team (Kanban), Toolforge
bd808 updated subscribers of T309821: Buster webservice grid went BOOM!.

Initial investigation started because of a report by @-jem- that their jembot webservice was misbehaving.

[20:53]  <    bd808> lots of confusing state in this tool right now jem. $HOME/service.manifest is in a state that would indicate that the tool is down. But it is running. qstat shows the running version to be in 'dr' deleted state (but it is obviously running). The job that is in dr state was started on 2022-04-09 and is running on a Buster node at least.
[20:55]  <    bd808> !log tools.jembot Force deleted stuck webservice job
[20:55]  < stashbot> Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.jembot/SAL
[20:56]  <    bd808> ummm.... and yet it is still running?
[21:00]  <    bd808> very confused right now about where the webservice is actually running. the grid has lost track, but the front proxy apparently has not.
[21:13]  <    bd808> this is so weird. usually if the front proxy gets confused like this it is possible to fix by starting and then stopping a grid webservice. The start should register a new backend ip/port and then stopping removes it again. I've done that twice now and the front proxy is still pretty obviously pointed at the "lost" job
[21:13] bd808 will keep poking
[21:14] jem supports mentally
[21:33]  <    bd808> !log tools.jembot Found orphan grid job by fetching host and port from front proxy redis. Killed related processes on tools-sgeweblight-10-4
[21:41]  <    bd808> !log tools Updated hiera to use fqdn of 'tools-proxy-06.tools.eqiad1.wikimedia.cloud' for active_redis key
[21:41]  < stashbot> Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[21:48]  <    bd808> puppet is taking approximately forever to run on tools-sgebastion-11 :/
[21:55]  <    bd808> !log tools Updated hiera to use fqdn of 'tools-proxy-06.tools.eqiad1.wikimedia.cloud' for profile::toolforge::active_proxy_host key
[21:55]  < stashbot> Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[21:56]  <    bd808> !log tools Removed legacy "active_proxy_host" hiera setting
[21:56]  < stashbot> Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[22:06] jem is back from dinner
[22:09]  <    bd808> jem: I haven't given up, but things are still mysterious. I found and killed the rogue webservice. I have not yet been able to get a new webservice to start however. The jobs submits but gets stuck in queue wait state.

The "active_redis key" mentioned was a typo and should have been "active_proxy_host" (the one I deleted later).

Thu, Jun 2, 11:14 PM · Patch-For-Review, cloud-services-team (Kanban), Toolforge
bd808 created T309821: Buster webservice grid went BOOM!.
Thu, Jun 2, 11:09 PM · Patch-For-Review, cloud-services-team (Kanban), Toolforge
bd808 created P29364 (An Untitled Masterwork).
Thu, Jun 2, 11:08 PM

Wed, Jun 1

bd808 added a member for Trusted-Contributors: KBach.
Wed, Jun 1, 4:25 PM
bd808 closed T283679: Replace Content::getNativeData() calls with TextContent::getText() in Math as Resolved.
Wed, Jun 1, 3:59 PM · MW-1.39-notes (1.39.0-wmf.15; 2022-06-06), Math
bd808 closed T283679: Replace Content::getNativeData() calls with TextContent::getText() in Math, a subtask of T155582: Deprecate Content::getNativeData(), defined TextContent::getText() to replace it. , as Resolved.
Wed, Jun 1, 3:59 PM · MW-1.37-notes (1.37.0-wmf.18; 2021-08-09), Platform Engineering (Icebox), MW-1.34-notes (1.34.0-wmf.3; 2019-04-30), MW-1.33-notes (1.33.0-wmf.24; 2019-04-02), Technical-Debt, User-Daniel, MediaWiki-ContentHandler
bd808 closed T309663: Consider moving "About" from "Get Help" to footer as Resolved.
Wed, Jun 1, 3:43 PM · Wikimedia-Developer-Portal
bd808 assigned T309676: Adoption request for signpost to Jarry1250.

Found the notification record in the Striker database showing that @Jarry1250 added @EpicPupper as a co-maintainer on 2022-05-14.

Wed, Jun 1, 3:28 PM · Toolforge-standards-committee
bd808 closed T309676: Adoption request for signpost as Resolved.
In T309676#7973344, @Majavah wrote:

As far as I can tell, EpicPupper is already a maintainer of that tool?

Wed, Jun 1, 3:23 PM · Toolforge-standards-committee
bd808 added a comment to T309308: Add tfj as a shortcut for toolforge-jobs command.

On https://github.com/wikimedia/cloud-toolforge-jobs-framework-cli it only says "We currently deploy this code into Toolforge using a debian package that is built from this very source tree." Do you know how it is actually built and deployed? I wonder if I will be able to learn it, replicate it on my local VM, then submit a patch.

Wed, Jun 1, 12:05 AM · Patch-For-Review, Toolforge Jobs framework

Tue, May 31

bd808 awarded Blog Post: GitLab-a-thon! a Meh! token.
Tue, May 31, 9:45 PM · GitLab
bd808 edited projects for T309659: Two volumes not deleting/creating on deployment-prep, added: cloud-services-team (Kanban), Beta-Cluster-Infrastructure; removed cloud-services-team.

(wrong tag, apologies cloud-services-team)

Tue, May 31, 8:47 PM · Cloud-VPS, Beta-Cluster-Infrastructure, cloud-services-team (Kanban)
bd808 added a comment to T309308: Add tfj as a shortcut for toolforge-jobs command.

Neither of those would need a Gerrit patch, IIUC. Both require a server admin to make the change on all relevant machines.

Tue, May 31, 8:39 PM · Patch-For-Review, Toolforge Jobs framework
bd808 edited Description on Cloud-Services.
Tue, May 31, 8:25 PM
bd808 added a hashtag to Toolforge Build Service: #toolforge-build.
Tue, May 31, 8:24 PM
bd808 edited Description on Toolforge Jobs framework.
Tue, May 31, 8:19 PM
bd808 edited projects for T249787: Create Docker image for Toolforge that is purpose built to run pywikibot scripts, added: Toolforge Jobs framework; removed Patch-For-Review, Toolforge.
Tue, May 31, 8:14 PM · Toolforge Jobs framework, Pywikibot, cloud-services-team (Kanban)
bd808 renamed T292981: Special consideration needed for toolforge-jobs when performing kubernetes cluster upgrades? from toolforge-jobs and kubernetes cluster upgrades to Special consideration needed for toolforge-jobs when performing kubernetes cluster upgrades?.
Tue, May 31, 8:12 PM · Toolforge Jobs framework, cloud-services-team (Kanban)
bd808 edited projects for T292981: Special consideration needed for toolforge-jobs when performing kubernetes cluster upgrades?, added: Toolforge Jobs framework; removed Toolforge.
Tue, May 31, 8:12 PM · Toolforge Jobs framework, cloud-services-team (Kanban)
bd808 edited projects for T299121: Job getting killed on k8s, added: Toolforge Jobs framework; removed Toolforge.
Tue, May 31, 8:10 PM · Toolforge Jobs framework, Kubernetes
bd808 renamed Toolforge Jobs framework from Toolforge Jobs Service to Toolforge Jobs framework.
Tue, May 31, 8:08 PM
bd808 edited projects for T285944: Toolforge: beta phase for the new jobs framework, added: Toolforge Jobs framework; removed Toolforge.
Tue, May 31, 8:03 PM · Toolforge Jobs framework, cloud-services-team (Kanban)
bd808 edited projects for T301901: Allow specifying the path for log files for jobs executed on the new toolforge Jobs framework, added: Toolforge Jobs framework; removed Toolforge.
Tue, May 31, 8:02 PM · Toolforge Jobs framework, Patch-For-Review, cloud-services-team (Kanban)
bd808 edited projects for T304421: Allow customizing the out/err files with toolforge-jobs, added: Toolforge Jobs framework; removed Toolforge.
Tue, May 31, 8:02 PM · Toolforge Jobs framework, cloud-services-team (Kanban)
bd808 edited projects for T302211: toolforge-jobs: merge stdout/stderr output, added: Toolforge Jobs framework; removed Toolforge.
Tue, May 31, 8:02 PM · Toolforge Jobs framework
bd808 edited projects for T305780: toolforge-jobs – wikihistory needs a container with both php7 and mono, added: Toolforge Jobs framework; removed Toolforge.
Tue, May 31, 8:02 PM · Toolforge Jobs framework
bd808 edited projects for T306309: toolforge-jobs job emails should have timestamps in events, added: Toolforge Jobs framework; removed Toolforge.
Tue, May 31, 8:02 PM · Toolforge Jobs framework
bd808 edited projects for T304893: Stderr is doubled with toolforge-jobs, added: Toolforge Jobs framework; removed Toolforge.
Tue, May 31, 8:02 PM · Toolforge Jobs framework
bd808 edited projects for T304900: toolforge-jobs should properly process 'out of quota' errors, added: Toolforge Jobs framework; removed Toolforge.
Tue, May 31, 8:02 PM · Toolforge Jobs framework
bd808 edited projects for T306311: toolforge-jobs should have a method to easily restart a specific job, added: Toolforge Jobs framework; removed Toolforge.
Tue, May 31, 8:02 PM · Toolforge Jobs framework
bd808 edited projects for T306312: toolforge-jobs should have a method to load a single task (or a subset of tasks) from a yaml file, added: Toolforge Jobs framework; removed Toolforge.
Tue, May 31, 8:02 PM · Toolforge Jobs framework
bd808 edited projects for T306310: toolforge-jobs job emails should have information on why events happened, added: Toolforge Jobs framework; removed Toolforge.
Tue, May 31, 8:02 PM · Toolforge Jobs framework
bd808 edited projects for T306391: Allow Toolforge scheduled jobs to have a maximum runtime, added: Toolforge Jobs framework; removed Toolforge.
Tue, May 31, 8:02 PM · Toolforge Jobs framework, Kubernetes
bd808 edited projects for T307847: toolforge-jobs should be able to run git with ssh, added: Toolforge Jobs framework; removed Toolforge.
Tue, May 31, 8:02 PM · Toolforge Jobs framework, cloud-services-team (Kanban)
bd808 edited projects for T309308: Add tfj as a shortcut for toolforge-jobs command, added: Toolforge Jobs framework; removed Toolforge.
Tue, May 31, 7:59 PM · Patch-For-Review, Toolforge Jobs framework
bd808 created Toolforge Jobs framework.
Tue, May 31, 7:59 PM
bd808 added a comment to T309644: Page titles missing from search.

I wonder if this is in any way affected by the fact that there is more than one page with "Get started" as it's title? The first hit for me is https://developer-portal.wmcloud.org/build-tools/get-started/

Tue, May 31, 7:48 PM · Wikimedia-Developer-Portal
bd808 added a comment to T287500: Accessibility evaluation.

Ideas for screenreaders to try

https://techblog.wikimedia.org/2020/07/02/an-orca-screen-reader-tutorial/

Tue, May 31, 5:58 PM · Accessibility, Wikimedia-Developer-Portal
bd808 updated the task description for T297140: New Service Request: developer-portal.
Tue, May 31, 5:50 PM · Goal, serviceops, Wikimedia-Developer-Portal, Service-deployment-requests
bd808 updated the task description for T280881: New Service Request Toolhub.
Tue, May 31, 5:45 PM · Patch-For-Review, serviceops, Toolhub, Service-deployment-requests, Services, SRE
bd808 removed a project from T297140: New Service Request: developer-portal: Patch-For-Review.
Tue, May 31, 5:43 PM · Goal, serviceops, Wikimedia-Developer-Portal, Service-deployment-requests
bd808 updated the task description for T305487: Bridgebot freaks out and sends double messages from IRC to Telegram.
Tue, May 31, 2:58 PM · User-bd808, Upstream, Tool-bridgebot

Mon, May 30

bd808 updated subscribers of T301901: Allow specifying the path for log files for jobs executed on the new toolforge Jobs framework.

All of the above could translate to a change as simple as ${job} 1>>${logdir}${jobname}.out 2>>${logdir}${jobname}.err where the default value of logdir is blank. (Obviously, we will remind users that their --log-dir value must include the trailing / which is why we are not hard-coding it in this changed version.)

Any more nuanced solutions (like allowing a different path for the err versus out file) could be discussed separately, preferably after @aborrero is back.

Mon, May 30, 4:48 PM · Toolforge Jobs framework, Patch-For-Review, cloud-services-team (Kanban)
bd808 added a comment to T306550: Move dumps.wikimedia.org HTTP service behind CDN edge.

Once this is done, could those severs live with private IPs to not "waste" public ones?
I see a mention of rsync on CR793525 for example, is that a blocker? What are the other flows on those boxes?

Mon, May 30, 4:00 PM · Patch-For-Review, cloud-services-team (Kanban), Data-Services
bd808 claimed T303444: Ensure Toolhub is compatible with Elasticsearch 7.10.

We are planning on migrating to Elasticsearch 7.x around the end of June (exact date to be defined). @bd808 / @Raymond_Ndibe: is that timeline reasonable on your side?

Mon, May 30, 2:55 PM · User-bd808, Patch-For-Review, User-Raymond_Ndibe, Discovery-Search (Current work), Toolhub

Sun, May 29

bd808 added a comment to T307122: Change @wikimediatech Twitter account authentication from verification email to 2FA code shown in 1Password.

@bd808 Do you still receive email spam? I logged in two or three times over the last two days since I switched to 2FA.

Sun, May 29, 8:47 PM · Developer-Advocacy (Apr-Jun 2022)

May 27 2022

bd808 closed T309419: Grant Zabe admin access to deployment-prep as Resolved.
May 27 2022, 9:45 PM · User-bd808, Beta-Cluster-Infrastructure