Page MenuHomePhabricator

valhallasw (Merlijn van Deen)
Pywikibot / Tool Labs

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Tuesday

  • Clear sailing ahead.

User Details

User Since
Oct 3 2014, 1:14 PM (233 w, 1 d)
Availability
Available
IRC Nick
valhallasw (typically around in European evenings)
LDAP User
Merlijn van Deen
MediaWiki User
Valhallasw [ Global Accounts ]

Recent Activity

Thu, Mar 21

GitHub <noreply@github.com> committed rTGPUd74b523e7f6d: Merge pull request #20 from valhallasw/stretch-upgrade (authored by valhallasw).
Merge pull request #20 from valhallasw/stretch-upgrade
Thu, Mar 21, 12:47 AM
valhallasw committed rTGPU4bffb42e2dd9: Update crontab (authored by valhallasw).
Update crontab
Thu, Mar 21, 12:47 AM
valhallasw committed rTGPU8c87cfa61f2c: Install flake8 via requirements.txt (authored by valhallasw).
Install flake8 via requirements.txt
Thu, Mar 21, 12:47 AM
valhallasw committed rTGPUa6d7d96eed26: Mailbox generates bytes, not str (authored by valhallasw).
Mailbox generates bytes, not str
Thu, Mar 21, 12:47 AM
valhallasw committed rTGPU08958244aa15: Add Cirrus CI (authored by valhallasw).
Add Cirrus CI
Thu, Mar 21, 12:47 AM
valhallasw committed rTGPU5d29344cc4e3: Re-add simplified online test (authored by valhallasw).
Re-add simplified online test
Thu, Mar 21, 12:47 AM
valhallasw committed rTGPU34eb311d300a: Add pop3bot tests (authored by valhallasw).
Add pop3bot tests
Thu, Mar 21, 12:47 AM
valhallasw committed rTGPUb988978662d8: Rename test.py to something more sensible (authored by valhallasw).
Rename test.py to something more sensible
Thu, Mar 21, 12:47 AM
valhallasw committed rTGPU93a2803ca0e5: Move test to tests/ (authored by valhallasw).
Move test to tests/
Thu, Mar 21, 12:47 AM
valhallasw committed rTGPU0a0f41721a94: Remove some old test files (authored by valhallasw).
Remove some old test files
Thu, Mar 21, 12:47 AM
valhallasw committed rTGPUb376b5d5abd5: Ignore Pycharm files (authored by valhallasw).
Ignore Pycharm files
Thu, Mar 21, 12:47 AM
valhallasw committed rTGPUd3050fe6ad61: Cleanup, pep8, etc (authored by valhallasw).
Cleanup, pep8, etc
Thu, Mar 21, 12:47 AM
valhallasw committed rTGPU3002ce4d69ed: Move get_changeset, filter_reviewers and get_reviewers_for_changeset to more… (authored by valhallasw).
Move get_changeset, filter_reviewers and get_reviewers_for_changeset to more…
Thu, Mar 21, 12:47 AM
valhallasw committed rTGPU406e2edc78b7: 2to3 conversion (authored by valhallasw).
2to3 conversion
Thu, Mar 21, 12:47 AM
valhallasw committed rTGPU4e93e216345a: Add SGE scripts (authored by valhallasw).
Add SGE scripts
Thu, Mar 21, 12:47 AM
valhallasw committed rTGPUb5466ac896d9: Add crontab as deployed on Trusty (authored by valhallasw).
Add crontab as deployed on Trusty
Thu, Mar 21, 12:47 AM

Tue, Mar 19

valhallasw closed T97790: zoomviewer seems to be down as Resolved.

Seems to work, so closing (and unassigning myself).

Tue, Mar 19, 8:32 PM · Cloud-Services, Toolforge
valhallasw placed T143217: archivebot.py exception: ValueError: unsupported format character '|' (0x7c) at index 43 up for grabs.

Apparently I was still assigned to this; unassigning. (or we could close this -- maybe crashing isn't so bad, as this did lead to the issue being noticed)

Tue, Mar 19, 8:32 PM · Pywikibot-tests, Pywikibot, Pywikibot-i18n
valhallasw placed T166951: Tag detection is broken up for grabs.

Unassigning; if anyone wants to pick this up they are more than welcome, but it's not on my radar to fix.

Tue, Mar 19, 8:29 PM · Wikibugs
valhallasw closed T216354: Upgrade valhallasw's tools to Stretch as Resolved.
Tue, Mar 19, 8:26 PM · Tools
valhallasw updated the task description for T216354: Upgrade valhallasw's tools to Stretch.
Tue, Mar 19, 8:26 PM · Tools
valhallasw added a comment to T218612: wb2-phab having issues.

Looking at the logs, my hypothesis is the following:

Tue, Mar 19, 6:53 AM · Patch-For-Review, Wikibugs

Sun, Mar 17

GitHub <noreply@github.com> committed rTGPUfe8d9c3d99f7: Merge 521b044a13ec5190429092ce0dcfb9db59fb44a3 into… (authored by valhallasw).
Merge 521b044a13ec5190429092ce0dcfb9db59fb44a3 into…
Sun, Mar 17, 4:25 PM
GitHub <noreply@github.com> committed rTGPU2e99df2344cf: Merge 87e0a75b41a6170418ac04d182e2e123915d0141 into… (authored by valhallasw).
Merge 87e0a75b41a6170418ac04d182e2e123915d0141 into…
Sun, Mar 17, 4:20 PM
valhallasw committed rTGPU87e0a75b41a6: Add Cirrus CI (authored by valhallasw).
Add Cirrus CI
Sun, Mar 17, 4:20 PM
valhallasw committed rTGPU521b044a13ec: Add Cirrus CI (authored by valhallasw).
Add Cirrus CI
Sun, Mar 17, 4:20 PM
GitHub <noreply@github.com> committed rTGPU37a40f73085d: Merge 4fb97aee688e96ecc5e1240b12b13153ea5344a8 into… (authored by valhallasw).
Merge 4fb97aee688e96ecc5e1240b12b13153ea5344a8 into…
Sun, Mar 17, 4:04 PM
GitHub <noreply@github.com> committed rTGPU7c34c3462395: Create .cirrus.yml (authored by valhallasw).
Create .cirrus.yml
Sun, Mar 17, 4:04 PM
valhallasw committed rTGPU4fb97aee688e: Add Cirrus CI (authored by valhallasw).
Add Cirrus CI
Sun, Mar 17, 4:04 PM
valhallasw committed rTGPUbfda6f41f1fe: Re-add simplified online test (authored by valhallasw).
Re-add simplified online test
Sun, Mar 17, 4:04 PM
valhallasw committed rTGPUe57f6edd1281: Make test_create_gerrit_json manual -- it's not really a test, after all (authored by valhallasw).
Make test_create_gerrit_json manual -- it's not really a test, after all
Sun, Mar 17, 4:04 PM
valhallasw committed rTGPU72b75fbbf7e9: Add pop3bot tests (authored by valhallasw).
Add pop3bot tests
Sun, Mar 17, 4:04 PM
valhallasw committed rTGPU392987d5b723: Rename test.py to something more sensible (authored by valhallasw).
Rename test.py to something more sensible
Sun, Mar 17, 4:04 PM
valhallasw committed rTGPUf8fa7d9bb934: Move test to tests/ (authored by valhallasw).
Move test to tests/
Sun, Mar 17, 4:04 PM
valhallasw committed rTGPUc6eec91a8701: Remove some old test files (authored by valhallasw).
Remove some old test files
Sun, Mar 17, 4:04 PM
valhallasw added a comment to T218494: `become`, `crontab` et al missing from Trusty hosts.
  • become mytoolname: sudo -niu tools.mytoolname
    • crontab -l: ssh cat /etc/toollabs-cronhost` crontab -l`
Sun, Mar 17, 10:26 AM · Patch-For-Review, cloud-services-team (Kanban), Toolforge
valhallasw created T218504: Install `qstat-full` in `/usr/local/bin` instead of `/usr/local/sbin/qstat-full`.
Sun, Mar 17, 10:25 AM · Patch-For-Review, Toolforge

Sat, Mar 16

valhallasw created T218494: `become`, `crontab` et al missing from Trusty hosts.
Sat, Mar 16, 9:06 PM · Patch-For-Review, cloud-services-team (Kanban), Toolforge
valhallasw added a comment to T218486: Grid jobs stuck on host.

@MarioFinale: as a workaround, I would suggest ignoring these jobs, and resubmitting them under a different name. Once the NFS issues are resolved (or the host is rebooted), they should disappear.

Sat, Mar 16, 6:03 PM · cloud-services-team (Kanban), Toolforge
valhallasw added a comment to T218486: Grid jobs stuck on host.

Attempted a force-umount using sudo umount -fr /mnt/nfs/labstore-secondary-tools-project, but this only resulted in umount.nfs4: /mnt/nfs/labstore-secondary-tools-project: device is busy.

Sat, Mar 16, 6:01 PM · cloud-services-team (Kanban), Toolforge
valhallasw edited projects for T218486: Grid jobs stuck on host, added: Toolforge; removed Cloud-VPS.

I've done some initial investigation, but I'm unable to find the root cause. Some observations:

Sat, Mar 16, 5:56 PM · cloud-services-team (Kanban), Toolforge
valhallasw added a comment to T218474: @ReleaseTaggerBot not tagging since March 6, 2019.

Thanks for taking a look at this. I migrated the bot to Stretch and Python 3 around that time, so that is likely the origin of the issue. I'll take a look in a bit more detail.

Sat, Mar 16, 1:04 PM · ReleaseTaggerBot

Fri, Mar 15

valhallasw committed rTGPUfd43678d2865: Ignore Pycharm files (authored by valhallasw).
Ignore Pycharm files
Fri, Mar 15, 12:56 AM
valhallasw committed rTGPU73d999451ac2: Cleanup, pep8, etc (authored by valhallasw).
Cleanup, pep8, etc
Fri, Mar 15, 12:56 AM
valhallasw committed rTGPU7aabe6c06464: Move get_changeset, filter_reviewers and get_reviewers_for_changeset to more… (authored by valhallasw).
Move get_changeset, filter_reviewers and get_reviewers_for_changeset to more…
Fri, Mar 15, 12:56 AM
valhallasw committed rTGPU604754bcd514: 2to3 conversion (authored by valhallasw).
2to3 conversion
Fri, Mar 15, 12:56 AM

Thu, Mar 14

valhallasw added a comment to T216354: Upgrade valhallasw's tools to Stretch.

Notes for gerrit-reviewer-bot.

Thu, Mar 14, 7:26 PM · Tools

Wed, Mar 13

valhallasw added a comment to T216354: Upgrade valhallasw's tools to Stretch.

tsreports was surprisingly easy, but was not converted to Python 3 - so it will disappear when Python 2's deprecation (2020... maybe 2022 for Stretch) comes along. I added a sitenotice with steps to convert queries to Quarry instead.

Wed, Mar 13, 8:33 PM · Tools
valhallasw updated the task description for T216354: Upgrade valhallasw's tools to Stretch.
Wed, Mar 13, 8:32 PM · Tools

Thu, Mar 7

valhallasw updated the task description for T216354: Upgrade valhallasw's tools to Stretch.
Thu, Mar 7, 8:13 PM · Tools
valhallasw added a comment to T216354: Upgrade valhallasw's tools to Stretch.

gerrit-patch-uploader done -- it's insanely fast now!

Thu, Mar 7, 8:13 PM · Tools

Wed, Mar 6

valhallasw updated the task description for T216354: Upgrade valhallasw's tools to Stretch.
Wed, Mar 6, 8:20 PM · Tools
valhallasw updated the task description for T216354: Upgrade valhallasw's tools to Stretch.
Wed, Mar 6, 7:56 PM · Tools
valhallasw closed T216650: Convert Wikibugs and ReleaseTaggerBot to stretch as Resolved.

RTB is now also converted.

Wed, Mar 6, 7:56 PM · ReleaseTaggerBot, Wikibugs
valhallasw added a comment to T216650: Convert Wikibugs and ReleaseTaggerBot to stretch.

Wikibugs is now fully Trusty-free! Next up: RTB. Hopefully as painless...

Wed, Mar 6, 7:38 PM · ReleaseTaggerBot, Wikibugs
valhallasw added a comment to T216650: Convert Wikibugs and ReleaseTaggerBot to stretch.

👍 after upgrading BS4 it seems to run smoothly again

Wed, Mar 6, 7:33 PM · ReleaseTaggerBot, Wikibugs
valhallasw added a comment to T216650: Convert Wikibugs and ReleaseTaggerBot to stretch.

🍪

Wed, Mar 6, 7:30 PM · ReleaseTaggerBot, Wikibugs
valhallasw added a comment to T216650: Convert Wikibugs and ReleaseTaggerBot to stretch.

Less luck there:

  File "/data/project/wikibugs/wikibugs2/wikibugs.py", line 8, in <module>
    from bs4 import BeautifulSoup
  File "/mnt/nfs/labstore-secondary-tools-project/wikibugs/py35-stretch/lib/python3.5/site-packages/bs4/__init__.py", line 30, in <module>
    from .builder import builder_registry, ParserRejectedMarkup
  File "/mnt/nfs/labstore-secondary-tools-project/wikibugs/py35-stretch/lib/python3.5/site-packages/bs4/builder/__init__.py", line 308, in <module>
    from . import _htmlparser
  File "/mnt/nfs/labstore-secondary-tools-project/wikibugs/py35-stretch/lib/python3.5/site-packages/bs4/builder/_htmlparser.py", line 7, in <module>
    from html.parser import (
ImportError: cannot import name 'HTMLParseError'
Wed, Mar 6, 7:27 PM · ReleaseTaggerBot, Wikibugs
valhallasw added a comment to T216650: Convert Wikibugs and ReleaseTaggerBot to stretch.

✔ wb2-irc works, now testing wb2-phab.

Wed, Mar 6, 7:26 PM · ReleaseTaggerBot, Wikibugs
valhallasw added a comment to T216650: Convert Wikibugs and ReleaseTaggerBot to stretch.

😊 emoji test!

Wed, Mar 6, 7:24 PM · ReleaseTaggerBot, Wikibugs
valhallasw added a comment to T216650: Convert Wikibugs and ReleaseTaggerBot to stretch.

And again.

Wed, Mar 6, 7:23 PM · ReleaseTaggerBot, Wikibugs
valhallasw added a comment to T216650: Convert Wikibugs and ReleaseTaggerBot to stretch.

Testing IRC bot

Wed, Mar 6, 7:22 PM · ReleaseTaggerBot, Wikibugs
valhallasw added a comment to T216650: Convert Wikibugs and ReleaseTaggerBot to stretch.

Taxonomy is re-added as crontab. Now looking at the rest of the processes.

Wed, Mar 6, 7:14 PM · ReleaseTaggerBot, Wikibugs

Mon, Mar 4

valhallasw added a comment to T216354: Upgrade valhallasw's tools to Stretch.

Contact converted to k8s uwsgi webservice. Notes:

Mon, Mar 4, 9:21 PM · Tools
valhallasw updated the task description for T216354: Upgrade valhallasw's tools to Stretch.
Mon, Mar 4, 9:13 PM · Tools

Sun, Mar 3

valhallasw updated the task description for T217513: newitem.py: AttributeError: 'function' object has no attribute 'startswith'.
Sun, Mar 3, 3:07 PM · Patch-For-Review, User-Zoranzoki21, good first bug, Pywikibot
valhallasw added a comment to T216355: Upgrade nlwikibots to Stretch.

Took a look at tvpmelder -- this seems more of an headache:

Sun, Mar 3, 12:14 PM · Tool-nlwikibots

Sat, Mar 2

valhallasw added a comment to T216355: Upgrade nlwikibots to Stretch.

I have changed the overall structure of the project directory a bit. There are now two shared virtualenvs:

Sat, Mar 2, 9:34 PM · Tool-nlwikibots
valhallasw updated subscribers of T217280: LDAP server running out of memory frequently and disrupting Cloud VPS clients.

@MarcoAurelio mentioned on IRC that this also happens on login, and I could reproduce it as well:

Last login: Sat Mar  2 12:10:40 2019 from [...]
groups: cannot find name for group ID 50062
valhallasw@tools-sgebastion-07:~$ id
Sat, Mar 2, 12:18 PM · cloud-services-team (Kanban), Patch-For-Review, Operations, Cloud-VPS, LDAP, Toolforge
valhallasw lowered the priority of T217472: Heavy load on all machines of toolforge -- jobs do not start from Unbreak Now! to Needs Triage.

/mnt/nfs/labstore1003-scratch also seems accessible again, so I think the problem should resolve itself now, but I think this still needs some investigation of why this happened.

Sat, Mar 2, 12:00 PM · Data-Services, cloud-services-team (Kanban), Toolforge
valhallasw added a comment to T217472: Heavy load on all machines of toolforge -- jobs do not start.

labstore1006 seems to have recovered:

Sat, Mar 2, 11:59 AM · Data-Services, cloud-services-team (Kanban), Toolforge
valhallasw added a comment to T217472: Heavy load on all machines of toolforge -- jobs do not start.

I'm not sure whether the Puppet are related -- the puppetmaster does not (always?) start correctly when it needs to respond to requests:

Sat, Mar 2, 11:50 AM · Data-Services, cloud-services-team (Kanban), Toolforge
valhallasw added a comment to T217472: Heavy load on all machines of toolforge -- jobs do not start.

sudo mount -o remount /mnt/nfs/labstore1003-scratch does not help to bring the scratch mount back online. Note, the mount parameters changed from

Sat, Mar 2, 11:42 AM · Data-Services, cloud-services-team (Kanban), Toolforge
valhallasw added a comment to T217472: Heavy load on all machines of toolforge -- jobs do not start.

The two problematic mounts are set to soft,timeo=300, which means any activity should time out after 30 seconds. But this is not consistent with what I see: ls does not time out, and there are many prometheus processes older than 30 seconds.

Sat, Mar 2, 11:29 AM · Data-Services, cloud-services-team (Kanban), Toolforge
valhallasw added a comment to T217472: Heavy load on all machines of toolforge -- jobs do not start.

Something strange is going on here. Taking tools-exec-1401 as an example:

Sat, Mar 2, 11:18 AM · Data-Services, cloud-services-team (Kanban), Toolforge

Feb 20 2019

valhallasw added a comment to T216650: Convert Wikibugs and ReleaseTaggerBot to stretch.
virtualenv -p python3 py35-stretch
~/py35-stretch/bin/pip install -r requirements.txt
# Successfully installed PyYAML-3.13 asyncio-redis-0.12.3 atomicwrites-1.3.0 attrs-18.2.0 beautifulsoup4-4.3.2 certifi-2018.11.29 chardet-3.0.4 click-6.7 docopt-0.6.2 fab-1.4.1 idna-2.8 irc3-0.5.1 more-itertools-6.0.0 pathlib2-2.3.3 pluggy-0.8.1 py-1.7.0 pytest-4.3.0 redis-2.10.3 requests-2.21.0 six-1.12.0 urllib3-1.24.1 venusian-1.2.0
Feb 20 2019, 8:12 PM · ReleaseTaggerBot, Wikibugs
valhallasw created T216650: Convert Wikibugs and ReleaseTaggerBot to stretch.
Feb 20 2019, 7:35 PM · ReleaseTaggerBot, Wikibugs

Feb 18 2019

Mahveotm awarded T60878: Report sensible error when family file cannot be loaded a Like token.
Feb 18 2019, 2:02 AM · good first bug, Pywikibot, Pywikibot-General

Feb 17 2019

valhallasw created T216355: Upgrade nlwikibots to Stretch.
Feb 17 2019, 1:48 PM · Tool-nlwikibots
valhallasw added a comment to T216354: Upgrade valhallasw's tools to Stretch.

Web tools

  • contact
    • Runs as a uWSGI Python tool, 2.7.6
    • should probably converted to an uWSGI k8s tool, possibly on Python3?
Feb 17 2019, 1:46 PM · Tools
valhallasw created T216354: Upgrade valhallasw's tools to Stretch.
Feb 17 2019, 1:07 PM · Tools

Feb 15 2019

Dalba awarded T104374: support python3 uwsgi apps a Manufacturing Defect? token.
Feb 15 2019, 7:41 AM · Cloud-Services, Toolforge

Jan 28 2019

valhallasw added a comment to T214809: wikibugs test warnings.

Reading through the bs4 changelog: https://bazaar.launchpad.net/~leonardr/beautifulsoup/bs4/view/head:/CHANGELOG I think this should be resolved in 4.4+. The warning seems gone with 4.7.1, so let's go to that version :-)

Jan 28 2019, 8:14 PM · Patch-For-Review, Wikibugs

Jan 27 2019

valhallasw requested review of D1141: Add .arcconfig to support arc diff.
Jan 27 2019, 2:39 PM
valhallasw added a reviewer for D1140: Add link to Toolforge admin console: bd808.
Jan 27 2019, 2:37 PM
valhallasw requested review of D1140: Add link to Toolforge admin console.
Jan 27 2019, 2:36 PM

Jan 22 2019

valhallasw added a comment to T214339: qsub job running for over 4 months on tool labs exec.

Sorry to hear that your bot was offline for 4 months. In general, a job that runs for so long is not an issue -- for example, Wikibugs regularly runs for months at a time without job resubmission.

Jan 22 2019, 8:29 PM · Toolforge

Jan 20 2019

valhallasw added a comment to T213965: Request to install exiv on Toolforge.

I fiddled around a bit with docker to try and find the minimal dependencies needed to pip install py3exiv2:

Jan 20 2019, 9:17 PM · cloud-services-team (Kanban), Toolforge
valhallasw added a comment to T122091: fixing_redirects.py / replace_links fails on links such as [[{{2001}}]].

Looking at the stack trace, the user is running a very old version of pywikibot; the use of textlib.replace_links was removed in november 2015 (8a7c42f5).

Jan 20 2019, 4:06 PM · Patch-For-Review, Pywikibot-textlib.py, Pywikibot
valhallasw changed the status of T212812: An extra blank line added below when editing a paragraph from Stalled to Open.

Hi @Inaki-LL , thank you for reporting this. I'm moving this task to the VisualEditor project; I think the maintainers of that project should be able to determine whether this is expected and if not, what can be done about it.

Jan 20 2019, 3:58 PM · VisualEditor

Jan 16 2019

valhallasw added a parent task for T213965: Request to install exiv on Toolforge: T55704: Packages to be added to toollabs puppet.
Jan 16 2019, 8:39 PM · cloud-services-team (Kanban), Toolforge
valhallasw added a subtask for T55704: Packages to be added to toollabs puppet: T213965: Request to install exiv on Toolforge.
Jan 16 2019, 8:39 PM · Cloud-Services, Tracking, Toolforge
valhallasw added a comment to T213965: Request to install exiv on Toolforge.

Unfortunately, PPA's cannot be used due to security concerns, but I think the following should work:

Jan 16 2019, 8:38 PM · cloud-services-team (Kanban), Toolforge

Dec 17 2018

valhallasw closed T211912: add wikibugs to #wikimedia-serviceops as Resolved.
Dec 17 2018, 8:27 PM · serviceops, Wikibugs
valhallasw closed T211912: add wikibugs to #wikimedia-serviceops, a subtask of T211902: create IRC channel for the Service Operations SRE subteam, as Resolved.
Dec 17 2018, 8:26 PM · serviceops, User-jijiki, Operations
valhallasw closed T211912: add wikibugs to #wikimedia-serviceops as Resolved.
Dec 17 2018, 8:25 PM · serviceops, Wikibugs
valhallasw closed T211912: add wikibugs to #wikimedia-serviceops, a subtask of T211902: create IRC channel for the Service Operations SRE subteam, as Resolved.
Dec 17 2018, 8:25 PM · serviceops, User-jijiki, Operations
valhallasw added a project to T211912: add wikibugs to #wikimedia-serviceops: serviceops.

Testing wikibugs.

Dec 17 2018, 8:25 PM · serviceops, Wikibugs
valhallasw added a comment to T211916: Whitelist Wikibugs in #wmcz-tracker.

Testing wikibugs.

Dec 17 2018, 8:24 PM · WMCZ-Tracker, Wikibugs

Dec 14 2018

valhallasw added a comment to T208118: Import arXiv ID (P818) and "full work available at" (P953) from unpaywall dataset.

There are a number of 'nan' values imported. This seems to be due to the somewhat hacky filtering for unpaywall entries with arXiv identifiers. For example, there may be a link to http://cds.cern.ch/record/681502/files/arXiv:hep-ph_0411095.pdf, but not to the corresponding arXiv page. In those cases, unpaywall_doi_to_arxiv_zonder_initial_sorted would not contain an arXiv identifier, and pandas would substitute it with 'nan' (which is also an odd choice...)

Dec 14 2018, 2:58 PM · WikiCite, Wikistorm