Page MenuHomePhabricator

Make a pywikibot entry point for scripts
Open, LowPublic

Description

Currently most scripts in pywikibot are just in the scripts/ folder. It may make sense to create a console script named pwb and then configure things to run using something simple like pwb <scriptname> irrespective of the folder location and so on.

This way, other users can simply make a pywikibot-<script_name> package on pypi and it can be linked to pwb as a plugin with entry points. Hence, it gives pywikibot the ability to make "plugins" which are run using a unified interface.

Check out how pytest does it to get a better idea: https://pytest.org/latest/writing_plugins.html

Event Timeline

AbdealiJK created this task.Jul 1 2016, 5:25 AM
Restricted Application added subscribers: Zppix, Aklapper. · View Herald TranscriptJul 1 2016, 5:25 AM
Restricted Application added a subscriber: pywikibot-bugs-list. · View Herald TranscriptJul 1 2016, 5:25 AM
AbdealiJK updated the task description. (Show Details)Jul 1 2016, 5:31 AM
jayvdb added a comment.Jul 1 2016, 4:49 PM

This concept needs to be fleshed out some more. If it is only the scripts in pywikibot-core repo, T139144: Making a pypi package pywikibot-scripts for officially supported scripts would be the correct approach, and the current pwb.py script is probably not very useful for that.

If it is goes to support other scripts, that are separate packages, how will it find them?

One approach is to create a PEP420 namespace, but that doesnt work in Python 2. For T104130: Family packages, https://gerrit.wikimedia.org/r/#/c/221637/ has a Python 2 mechanism that replicates PEP420 behaviour. Maybe I should extract that out into a separate package, and then it can be used for scripts also.

Another approach is to register the scripts as plugins using setuptools (like flake8 v2).

Other ideas?

The library does need a mechanism to create a user-config.py, and that could be a console script in the pywikibot package, and could simply be the existing generate_user_config.py.

Related patches are:

Xqt added a subscriber: Xqt.Jul 14 2017, 11:45 AM

Isn't this the same as in T139141 ?

Xqt triaged this task as Low priority.Sep 27 2018, 2:11 PM
This comment was removed by Dvorapa.
Restricted Application added a subscriber: RhinosF1. · View Herald TranscriptDec 21 2019, 7:25 AM
Dvorapa added a comment.EditedDec 21 2019, 7:35 AM

Isn't this the same as in T139141 ?

No, T139141 is about T107629 (merged those two)

Xqt claimed this task.May 26 2020, 7:03 AM
Xqt added a comment.May 27 2020, 12:10 PM

Add pwb.py wrapper to pywkibot package
Explanations for https://gerrit.wikimedia.org/r/#/c/pywikibot/core/+/560057/
see also for example

The main solution is a console scripts entry point inside setup.py:

entry_points={'console_scripts': [
    'pwb=pywikibot.scripts.pwb:run',
]},

This creates an entry_point text file inside pywikibot egg info when creating the package and has this content:

[console_scripts]
pwb = pywikibot.scripts.pwb:run

When installing the pywikibot package Python creates two short files inside his own Scripts folder:

  • pwb.exe (in Windows), a small application file which just calls the bootstrap Python script pwb_script.py
  • pwb_script.py a python script which calls the entry point script given above

Entry point script
The script entry Point consist of the script path (pywikibot.scripts), the script name (pwb) and the entry point function (run).

The entry point script is located in inside scripts folder inside pywikibot package (the scripts name is inspired from Scripts folder used by Python). The entry point function may be different from default entry point if the script is from __main__. The main entry point function is main() but the package entry point is run(). This ensures that we are able to differentiate between package and directory mode installations which may be slightly different in ist behaviou. Maybe this will be rreplaced by the Path information (scripts starts inside the 'side-package' folder which can be detected too).

The Location inside the pywikibot folder is mandatory to ensure to collect the scripts to the package

creating the package

A simple batch does it:

@echo ### copy script entry points to pywikibot\scripts
copy pwb.py .\pywikibot\scripts\*
copy generate*.py .\pywikibot\scripts\*

@echo
@echo ### create a new package
py -3 setup.py sdist

@echo
@echo ### delete script entry points
del pywikibot\scripts\pwb.py
del pywikibot\scripts\generate*

It is not necessary to change the git repository.

Installing the local package
Having al local package created by the commands above it can be installed as a side package without the pypi index is needed:

pip uninstall pywikibot
pip install --no-index --find-links=dist pywikibot

Running the pwb wrapper
create a minimal user-config.py:

mylang = 'en'  # the default site code you are working on
family = 'wikipedia'  # the default family
usernames['wikipedia']['en'] = 'Test'  # the bot account name
user_script_paths = ['c:\\pwb.git.core.scripts']

The user_script_path is important. You can add any path to it including side package paths (but there are some TODOs, e.g. find the absolute side package path. probably a different side package path is appropriate)
run the script
pwb.exe <global option> <script name> <global and local options>
for example:
pwb.exe touche -page:user:xqt <-- yes it works with touche
and have the full benefit of the pwb script wrapper:

  • find any script in any folder given by user_scripts_path
  • add .py ending automatically
  • similar search for script names, ignore spelling mistakes
  • enable global options even if the script does not support it
  • and in future: enable scripts installed as side package
Dvorapa added a comment.EditedMay 27 2020, 12:53 PM

I tested the patch:

$ git review -d I337beab3732d6a4e9567f56af885a27daa004a0d
Downloading refs/changes/57/560057/11 from gerrit
Switched to branch "review/xqt/T107629"
$ python setup.py sdist
running sdist
running egg_info
creating pywikibot.egg-info
writing pywikibot.egg-info/PKG-INFO
writing dependency_links to pywikibot.egg-info/dependency_links.txt
writing entry points to pywikibot.egg-info/entry_points.txt
writing requirements to pywikibot.egg-info/requires.txt
writing top-level names to pywikibot.egg-info/top_level.txt
writing manifest file 'pywikibot.egg-info/SOURCES.txt'
reading manifest file 'pywikibot.egg-info/SOURCES.txt'
writing manifest file 'pywikibot.egg-info/SOURCES.txt'
running check
creating pywikibot-3.0.20200527
creating pywikibot-3.0.20200527/pywikibot
creating pywikibot-3.0.20200527/pywikibot.egg-info
creating pywikibot-3.0.20200527/pywikibot/comms
creating pywikibot-3.0.20200527/pywikibot/data
creating pywikibot-3.0.20200527/pywikibot/families
creating pywikibot-3.0.20200527/pywikibot/page
creating pywikibot-3.0.20200527/pywikibot/scripts
creating pywikibot-3.0.20200527/pywikibot/site
creating pywikibot-3.0.20200527/pywikibot/specialbots
creating pywikibot-3.0.20200527/pywikibot/tools
creating pywikibot-3.0.20200527/pywikibot/userinterfaces
copying files to pywikibot-3.0.20200527...
copying README.rst -> pywikibot-3.0.20200527
copying setup.py -> pywikibot-3.0.20200527
copying pywikibot/__init__.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/_wbtypes.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/bot.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/bot_choice.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/config2.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/cosmetic_changes.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/daemonize.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/date.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/diff.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/echo.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/editor.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/exceptions.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/family.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/fixes.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/flow.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/i18n.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/interwiki_graph.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/logentries.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/logging.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/login.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/pagegenerators.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/plural.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/proofreadpage.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/site_detect.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/textlib.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/throttle.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/titletranslate.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/version.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/xmlreader.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot.egg-info/PKG-INFO -> pywikibot-3.0.20200527/pywikibot.egg-info
copying pywikibot.egg-info/SOURCES.txt -> pywikibot-3.0.20200527/pywikibot.egg-info
copying pywikibot.egg-info/dependency_links.txt -> pywikibot-3.0.20200527/pywikibot.egg-info
copying pywikibot.egg-info/entry_points.txt -> pywikibot-3.0.20200527/pywikibot.egg-info
copying pywikibot.egg-info/requires.txt -> pywikibot-3.0.20200527/pywikibot.egg-info
copying pywikibot.egg-info/top_level.txt -> pywikibot-3.0.20200527/pywikibot.egg-info
copying pywikibot/comms/__init__.py -> pywikibot-3.0.20200527/pywikibot/comms
copying pywikibot/comms/eventstreams.py -> pywikibot-3.0.20200527/pywikibot/comms
copying pywikibot/comms/http.py -> pywikibot-3.0.20200527/pywikibot/comms
copying pywikibot/comms/threadedhttp.py -> pywikibot-3.0.20200527/pywikibot/comms
copying pywikibot/data/__init__.py -> pywikibot-3.0.20200527/pywikibot/data
copying pywikibot/data/api.py -> pywikibot-3.0.20200527/pywikibot/data
copying pywikibot/data/mysql.py -> pywikibot-3.0.20200527/pywikibot/data
copying pywikibot/data/sparql.py -> pywikibot-3.0.20200527/pywikibot/data
copying pywikibot/data/wikistats.py -> pywikibot-3.0.20200527/pywikibot/data
copying pywikibot/families/__init__.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/commons_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/foundation_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/i18n_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/incubator_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/lyricwiki_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/mediawiki_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/meta_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/omegawiki_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/osm_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/outreach_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/species_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/vikidia_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/wikibooks_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/wikidata_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/wikimania_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/wikimediachapter_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/wikinews_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/wikipedia_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/wikiquote_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/wikisource_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/wikitech_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/wikiversity_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/wikivoyage_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/wiktionary_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/wowwiki_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/page/__init__.py -> pywikibot-3.0.20200527/pywikibot/page
copying pywikibot/scripts/__init__.py -> pywikibot-3.0.20200527/pywikibot/scripts
copying pywikibot/site/__init__.py -> pywikibot-3.0.20200527/pywikibot/site
copying pywikibot/specialbots/__init__.py -> pywikibot-3.0.20200527/pywikibot/specialbots
copying pywikibot/specialbots/_unlink.py -> pywikibot-3.0.20200527/pywikibot/specialbots
copying pywikibot/specialbots/_upload.py -> pywikibot-3.0.20200527/pywikibot/specialbots
copying pywikibot/tools/__init__.py -> pywikibot-3.0.20200527/pywikibot/tools
copying pywikibot/tools/_logging.py -> pywikibot-3.0.20200527/pywikibot/tools
copying pywikibot/tools/chars.py -> pywikibot-3.0.20200527/pywikibot/tools
copying pywikibot/tools/djvu.py -> pywikibot-3.0.20200527/pywikibot/tools
copying pywikibot/tools/formatter.py -> pywikibot-3.0.20200527/pywikibot/tools
copying pywikibot/tools/ip.py -> pywikibot-3.0.20200527/pywikibot/tools
copying pywikibot/userinterfaces/__init__.py -> pywikibot-3.0.20200527/pywikibot/userinterfaces
copying pywikibot/userinterfaces/gui.py -> pywikibot-3.0.20200527/pywikibot/userinterfaces
copying pywikibot/userinterfaces/terminal_interface.py -> pywikibot-3.0.20200527/pywikibot/userinterfaces
copying pywikibot/userinterfaces/terminal_interface_base.py -> pywikibot-3.0.20200527/pywikibot/userinterfaces
copying pywikibot/userinterfaces/terminal_interface_unix.py -> pywikibot-3.0.20200527/pywikibot/userinterfaces
copying pywikibot/userinterfaces/terminal_interface_win32.py -> pywikibot-3.0.20200527/pywikibot/userinterfaces
copying pywikibot/userinterfaces/transliteration.py -> pywikibot-3.0.20200527/pywikibot/userinterfaces
copying pywikibot/userinterfaces/win32_unicode.py -> pywikibot-3.0.20200527/pywikibot/userinterfaces
Writing pywikibot-3.0.20200527/setup.cfg
creating dist
Creating tar archive
removing 'pywikibot-3.0.20200527' (and everything under it)
$ cd dist/
$ sudo pip install pywikibot-3.0.20200527.tar.gz 
[sudo] heslo pro user: 
Processing ./pywikibot-3.0.20200527.tar.gz
Requirement already satisfied: requests>=2.20.1 in /usr/lib/python3.8/site-packages (from pywikibot==3.0.20200527) (2.23.0)
Requirement already satisfied: chardet>=3.0.2 in /usr/lib/python3.8/site-packages (from requests>=2.20.1->pywikibot==3.0.20200527) (3.0.4)
Requirement already satisfied: idna>=2.5 in /usr/lib/python3.8/site-packages (from requests>=2.20.1->pywikibot==3.0.20200527) (2.9)
Requirement already satisfied: urllib3>=1.21.1 in /usr/lib/python3.8/site-packages (from requests>=2.20.1->pywikibot==3.0.20200527) (1.25.9)
Installing collected packages: pywikibot
    Running setup.py install for pywikibot ... done
Successfully installed pywikibot-3.0.20200527

This creates the files pwb, gff, guf... (Python executables; like pwb.py, but without .py) in /usr/bin folder (whereas Pywikibot is installed into /usr/lib/python3.8/site-packages/pywikibot) with the following content:

/usr/bin/pwb
#!/usr/bin/python
# EASY-INSTALL-ENTRY-SCRIPT: 'pywikibot==3.0.20200527','console_scripts','pwb'
__requires__ = 'pywikibot==3.0.20200527'
import re
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(
        load_entry_point('pywikibot==3.0.20200527', 'console_scripts', 'pwb')()
    )

So far so good.

But if I run any of these, there is the following traceback:

$ guf
Traceback (most recent call last):
  File "/usr/bin/guf", line 11, in <module>
    load_entry_point('pywikibot==3.0.20200527', 'console_scripts', 'guf')()
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 490, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2853, in load_entry_point
    return ep.load()
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2453, in load
    return self.resolve()
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2459, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/home/user/.local/lib/python3.8/site-packages/pywikibot/__init__.py", line 25, in <module>
    from pywikibot.bot import (
  File "/home/user/.local/lib/python3.8/site-packages/pywikibot/bot.py", line 114, in <module>
    from pywikibot import config2 as config
  File "/home/user/.local/lib/python3.8/site-packages/pywikibot/config2.py", line 377, in <module>
    base_dir = get_base_dir()
  File "/home/user/.local/lib/python3.8/site-packages/pywikibot/config2.py", line 371, in get_base_dir
    raise RuntimeError(exc_text)
RuntimeError: No user-config.py found in directory '/mnt/B4D0D63BD0D60410/Tvorba/pywikibota/dist'.
  Please check that user-config.py is stored in the correct location.
  Directory where user-config.py is searched is determined as follows:

    Return the directory in which user-specific information is stored.

    This is determined in the following order:
     1.  If the script was called with a -dir: argument, use the directory
         provided in this argument.
     2.  If the user has a PYWIKIBOT_DIR environment variable, use the value
         of it.
     3.  If user-config is present in current directory, use the current
         directory.
     4.  If user-config is present in pwb.py directory, use that directory
     5.  Use (and if necessary create) a 'pywikibot' folder under
         'Application Data' or 'AppData\Roaming' (Windows) or
         '.pywikibot' directory (Unix and similar) under the user's home
         directory.

    Set PYWIKIBOT_NO_USER_CONFIG=1 to disable loading user-config.py

    @param test_directory: Assume that a user config file exists in this
        directory. Used to test whether placing a user config file in this
        directory will cause it to be selected as the base directory.
    @type test_directory: str or None
    @rtype: str
Xqt added a comment.May 27 2020, 1:18 PM

Oh, that looks great. I had the same. At this point the user-config.py is required. I did it by hand but guf should be do the job later.

guf and gff should always work without user-config.py. Anyway, a simple workaround of an empty user-config.py file works usually too.

Change 560057 had a related patch set uploaded (by Xqt; owner: Xqt):
[pywikibot/core@master] [dist] Add pwb.py wrapper and generate*.py to pywkibot package

https://gerrit.wikimedia.org/r/560057