Page MenuHomePhabricator

Create a "my first Pywikibot bot" tutorial for Toolforge
Open, LowPublic


Wanted: A detailed but not technical jargon heavy guide to setting up a new Pywikibot tool.

[10:25]  <theglobetrotter>	How do I set up a bot on toolforge that edits the wiki at regular intervals?

This question was asked in the #wikimedia-cloud irc channel, unfortunately at a time when no one was providing answers. It is a good question though. There are many possible ways to answer, but a reasonably common solution would be to first create a pywikibot script to perform the edits and then to setup a cron job to run that script at some defined interval.

Tutorial should cover:

  • Standard tutorial header material (prerequisites, overview of steps)
  • Setting up Owner-only OAuth for pywikibot (maybe something to document on mw.o if not already there)
  • Using shared pywikibot code on Toolforge
  • Example of running a pywikibot provided script
  • Example of running a trivial custom script
  • Example of Grid Engine cron job
  • Example of Kubernetes CronJob object (might be a good excuse to work on some helper script for this)
  • Standard tutorial footer material
    • Choosing a license
    • Publishing source code
    • Adding a Tool:* documentation page
    • Where to look next for help with more advanced use cases

Should include:

  • Screenshots!
  • Cut-n-paste instructions for everything

Event Timeline

bd808 created this task.May 5 2016, 3:14 PM
Restricted Application added subscribers: Zppix, Aklapper. · View Herald TranscriptMay 5 2016, 3:14 PM

Steal some from Also I want us to support crons in PAWS, at which point it'll become much more useful...

and interactive bots can already be run there and are being run there

jayvdb added a subscriber: jayvdb.May 5 2016, 3:20 PM

As an aside, I think we should be trying to enable the "my first Pywikibot bot" experience to be within PAWS. For that, there is , and WMID has created an Indonesian language booklet with pretty pictures, etc. We can merge the print quality version into the MediaWiki page.

When PAWS has cron functionality , it could even be the "my super-dooper critical Pywikibot bot" experience.

Danny_B moved this task from Backlog to Cloud Services on the Documentation board.May 6 2016, 10:23 PM
valhallasw lowered the priority of this task from Low to Lowest.May 27 2016, 12:47 PM
valhallasw moved this task from Triage to Backlog on the Toolforge board.
valhallasw raised the priority of this task from Lowest to Low.May 27 2016, 12:49 PM
Vaishnav980123 added a subscriber: Vaishnav980123.

I would like to work on this bug.

Vaishnav980123 removed Vaishnav980123 as the assignee of this task.Oct 2 2016, 1:32 PM
bd808 added a comment.Oct 3 2016, 1:32 AM

I would like to work on this bug.

@Vaishnav980123 awesome! Do you have experience with Tool Labs and Pywikibot already? I think it's totally ok if you don't, but you will need to seek out more help from others if not.

I think the main focus of the "My first X" tutorial series should be to help show people who are not familiar with Tool Labs detailed steps needed to make something useful. I think reading some tutorials from the Heroku getting started series might be good inspiration for the kind of thing I'm hoping we will end up with. The awesome part of doing things like this on wikitech is that we can get help from lots of people in making the directions more clear and informative. Don't worry about getting everything perfect on your first try. Just be bold, start a page on wikitech, and start working. It would probably be a good idea to make the page with the {{draft}} template at first so that people are warned if they find while looking for help.

@Xephyr826: Thanks! Please tell us if you have any specific questions or get stuck so we can provide input!

Thanks! I'll come back to this next week when I have more time.

Xephyr826 removed Xephyr826 as the assignee of this task.Mar 8 2017, 12:03 PM
This comment was removed by Xephyr826.
Xephyr826 added a comment.EditedApr 26 2017, 5:11 AM

@bd808 or anyone, Two questions:

@bd808 or anyone, Two questions:

  • Should this tutorial use PAWS to introduce Pywikibot as suggested above?

Feel free to, though PAWS is quite different from Tool Labs in many aspects (eg. Grid), but users should be able to familiarize themselves with Pywikibot with PAWS.

Definitely not; that does not show how the tutorial is related to tool labs. Personally I'd recommend somewhere inside as it is where a few other "my first..." tutorials for tool labs live)

There is also a very old (and perhaps outdated) tutorial at I'd recommend rewriting that to wikitech and redirect this page to the new tutorial.

@Xephyr826: Thanks for working on this!
I see this got moved to , great!
Is there more work needed on this task? Or is this task "resolved"?

bd808 added a comment.Oct 10 2017, 2:41 AM

Is there more work needed on this task? Or is this task "resolved"?

I think there is quite a bit more work that can be done on this tutorial, but it is nice to have something for people to iterate on.

I'm back for another crack at this one :-)

Xephyr826 renamed this task from Create a "my first Pywikibot bot" tutorial for Tool Labs to Create a "my first Pywikibot bot" tutorial for Toolforge.Mar 24 2018, 8:41 PM
Xephyr826 removed a project: Cloud-Services.
Xephyr826 edited projects, added Cloud-Services; removed Toolforge.Mar 24 2018, 8:43 PM
Xephyr826 edited projects, added Toolforge; removed Cloud-Services.

First a little research on what documentation is out there now. Seems there are two main pages in use related to Toolforge/Pywikibot:
^ I suggest using the shared pywikibot as dosumented at

User:Russell Blau/Using pywikibot on Labs Might be outdated.

It looks like written in the pmtpa times, but it's not too bad of a current guide for a shared install. From a quick glance only minor details like the bash prompt and the pywikibot branch are different.

mw:Manual:Pywikibot/Installation/Toolforge Detailed instructions but some parts marked outdated.

Documents the method of having a tool running its own copy of pywikibot.
Section Setup the webspace is hopelessly outdated (pmtpa instructions, when we still had a global apache).
Section Setup the job submission describes the conversion from toolserver command to toolforge command and is not relevant. Syntax like -m 512M is preferred over -l h_vmem=512M as the former also specifies virtual_free in addition to h_vmem. And note that 512M is now the default after T120517.
Section Automatic updating git on Toolforge is really unnecessary. If someone wants to auto-update, they should use the shared pywikibot, which is updated twice a day AFAIK.

FWIW, because of issues like, I do not personally use that export PYTHONPATH thing. It should be unnecessary anyways. If a custom environment must be specified, eg. for virtualenvs or C dependencies, I usually use workarounds, eg.

  • using /absolute/path/to/virtualenv/bin/python as the python interpretor
  • using .local as the virtualenv
  • using wrapper scripts like


  • compat isn't maintained anymore. I'd say don't even mention it... causes more confusion than good.
  • Section setup web-space & Setup job submission suffers from the same outdatedness as mw:Manual:Pywikibot/Installation/Toolforge.

Expanded on the instructions to request access to Toolforge:

Pulled in simplified instructions from a few places so all the instructions are in one section.

Restricted Application added a subscriber: pywikibot-bugs-list. · View Herald TranscriptJun 19 2018, 8:28 AM
srodlund claimed this task.Aug 28 2018, 8:21 PM
binbot added a subscriber: binbot.Aug 29 2018, 7:14 AM
Wenyi added a subscriber: Wenyi.Sep 18 2018, 2:28 PM
zhuyifei1999 added a comment.EditedFeb 26 2019, 10:26 PM

both pretty good

both interior to the other in some aspects.

  • Should use /shared/pywikibot/core (if grid-only) or /data/project/shared/pywikibot/core (works also in k8s, but I very few run bots in k8s...)
  • Modifying PYTHONPATH is evil. There is no guarantee that bash_profile will be loaded at all. (T134495#4078967, T164277: jsub/jstart inconsistency: non-continuous jobs spawns a login bash shell that loads .bash_profile, but continuous jobs doesn't load either .bash_profile or .bashrc)
    • Highly recommend either using venv and/or the in the shared nightly clone.
    • Honestly, some people tell you to modify env vars with .bashrc, some .bash_profile, some .profile, and some .bash_aliases. The outcome? A terrible mess. (My opinion: .bash_aliases should go into .bashrc and .bashrc should be loaded by .profile after testing if the shell is bash; .bash_profile should go into .profile. Keep login shell configuration in .profile and non-login shell configuration in .bashrc)
  • Doesn't explain T60784#3925224 (Granted, people are very unlikely to hit this bug if they use pywikibot's own scripts, but if some more advanced people start to code their own pywikibot scripts and start adding some print statements, oh, gotta expect some fun...)
  • is evil. (my opinion doesn't help does it? admittedly, the script does help newcomers a lot, so, ignore this complaint)
  • Should seriously stop talking about compat. It ain't supported.
  • Setup web-space is like 4 years outdated. I'd recommend just remove the section.
  • qcronsub?! Is this toolserver?
  • Why h_vmem when we have -mem?
  • 256MB was default limit... until 3 years ago
  • Should start talking about python3 rather than this aged python2

tl;dr: badly outdated.

  • Nice docs on how to access toolforge
  • Local clone is the only way? This should not be used unless you're developing pywikibot code itself.
  • The instruction for webpage... while it works, it has nothing to do with pywikibot.
  • Doesn't even talk about how to submit a job to wither grid or k8s.
  • And yeah... empty sections

tl;dr: nice start, but bad finish, lacking in so much stuffs that should be said.

Looking forward to it :)

Dvorapa added a comment.EditedFeb 26 2019, 11:26 PM

@zhuyifei1999 Agree, python2, compat, 2.0, all these things should be eliminated by force from Pywikibot manuals (and brutal force to still occurring pywikipedia and trunk or rewrite! Are we still at the beginning of the century?)

bd808 updated the task description. (Show Details)Mar 7 2020, 10:35 PM

("screenshots"? of command line interface? wtf?)

srodlund removed srodlund as the assignee of this task.Jun 25 2020, 8:56 PM
srodlund added a subscriber: srodlund.
srodlund claimed this task.EditedJun 29 2020, 5:16 PM

Retaking this task.

Will be looking at the current tutorial and for updates and improvements.

See staging doc here:

MJL added a subscriber: MJL.Sep 15 2020, 5:16 AM