Make a pywikibot entry point for scripts
Open, LowPublic
Actions

Assigned To

Authored By

	AbdealiJK
	Jul 1 2016, 5:25 AM

Description

Currently most scripts in pywikibot are just in the scripts/ folder. It may make sense to create a console script named pwb and then configure things to run using something simple like pwb <scriptname> irrespective of the folder location and so on.

This way, other users can simply make a pywikibot-<script_name> package on pypi and it can be linked to pwb as a plugin with entry points. Hence, it gives pywikibot the ability to make "plugins" which are run using a unified interface.

Check out how pytest does it to get a better idea: https://pytest.org/latest/writing_plugins.html

Details

	Subject	Repo	Branch	Lines +/-
	[IMPR] Provide an entry point to connect foreign scripts with pwb wapper	pywikibot/core	master	+312 -13
	[IMP] Make a pywikibot entry point for scripts	pywikibot/core	master	+107 -26

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
Declined		None	T164095 pypi package does not contain scripts folder
Resolved		Xqt	T107629 generate_user_files and generate_family_file helper scripts aren't installed via pip
Resolved	Feature	Xqt	T270480 add version.py script to pywikibot pypi package
Open		Xqt	T139144 Making a pypi package pywikibot-scripts for officially supported scripts
Declined		None	T115784 pip install . in pywikibot should install scripts/ in /usr/local/bin
Open		Xqt	T139143 Make a pywikibot entry point for scripts
Resolved	Feature	Xqt	T270474 Allow using pywikibot without user-config.py and PYWIKIBOT_NO_USER_CONFIG=1

Event Timeline

AbdealiJK created this task.Jul 1 2016, 5:25 AM

Restricted Application added subscribers: Zppix, Aklapper. · View Herald TranscriptJul 1 2016, 5:25 AM

AbdealiJK added projects: Pywikibot-catfiles, Pywikibot.Jul 1 2016, 5:25 AM

AbdealiJK added subscribers: jayvdb, DrTrigon.

Restricted Application added a subscriber: pywikibot-bugs-list. · View Herald TranscriptJul 1 2016, 5:25 AM

AbdealiJK updated the task description. (Show Details)Jul 1 2016, 5:31 AM

This concept needs to be fleshed out some more. If it is only the scripts in pywikibot-core repo, T139144: Making a pypi package pywikibot-scripts for officially supported scripts would be the correct approach, and the current pwb.py script is probably not very useful for that.

If it is goes to support other scripts, that are separate packages, how will it find them?

One approach is to create a PEP420 namespace, but that doesnt work in Python 2. For T104130: Family packages, https://gerrit.wikimedia.org/r/#/c/221637/ has a Python 2 mechanism that replicates PEP420 behaviour. Maybe I should extract that out into a separate package, and then it can be used for scripts also.

Another approach is to register the scripts as plugins using setuptools (like flake8 v2).

Other ideas?

The library does need a mechanism to create a user-config.py, and that could be a console script in the pywikibot package, and could simply be the existing generate_user_config.py.

Related patches are:

jayvdb mentioned this in T139144: Making a pypi package pywikibot-scripts for officially supported scripts.Jul 1 2016, 5:09 PM

Isn't this the same as in T139141 ?

Xqt triaged this task as Low priority.Sep 27 2018, 2:11 PM

Xqt added a parent task: T139144: Making a pypi package pywikibot-scripts for officially supported scripts.Dec 21 2019, 4:47 AM

Dvorapa subscribed.Dec 21 2019, 7:25 AM

This comment was removed by Dvorapa.

Restricted Application added a subscriber: RhinosF1. · View Herald TranscriptDec 21 2019, 7:25 AM

Dvorapa removed a parent task: T139144: Making a pypi package pywikibot-scripts for officially supported scripts.Dec 21 2019, 7:25 AM

Dvorapa added a subtask: T139144: Making a pypi package pywikibot-scripts for officially supported scripts.

Dvorapa removed a subtask: T139144: Making a pypi package pywikibot-scripts for officially supported scripts.Dec 21 2019, 7:28 AM

Dvorapa added a parent task: T139144: Making a pypi package pywikibot-scripts for officially supported scripts.

In T139143#3438931, @Xqt wrote:

Isn't this the same as in T139141 ?

No, T139141 is about T107629 (merged those two)

Xqt added a subtask: T244684: Allow pywikibot import without presence of user-config.py.Feb 10 2020, 5:26 AM

Xqt removed a project: Pywikibot-catfiles.May 25 2020, 6:43 AM

Xqt added a parent task: T107629: generate_user_files and generate_family_file helper scripts aren't installed via pip.May 25 2020, 9:32 AM

Xqt claimed this task.May 26 2020, 7:03 AM

Add pwb.py wrapper to pywkibot package
Explanations for https://gerrit.wikimedia.org/r/#/c/pywikibot/core/+/560057/
see also for example

The main solution is a console scripts entry point inside setup.py:

entry_points={'console_scripts': [
    'pwb=pywikibot.scripts.pwb:run',
]},

This creates an entry_point text file inside pywikibot egg info when creating the package and has this content:

[console_scripts]
pwb = pywikibot.scripts.pwb:run

When installing the pywikibot package Python creates two short files inside his own Scripts folder:

pwb.exe (in Windows), a small application file which just calls the bootstrap Python script pwb_script.py
pwb_script.py a python script which calls the entry point script given above

Entry point script
The script entry Point consist of the script path (pywikibot.scripts), the script name (pwb) and the entry point function (run).

The entry point script is located in inside scripts folder inside pywikibot package (the scripts name is inspired from Scripts folder used by Python). The entry point function may be different from default entry point if the script is from __main__. The main entry point function is main() but the package entry point is run(). This ensures that we are able to differentiate between package and directory mode installations which may be slightly different in ist behaviou. Maybe this will be rreplaced by the Path information (scripts starts inside the 'side-package' folder which can be detected too).

The Location inside the pywikibot folder is mandatory to ensure to collect the scripts to the package

creating the package

A simple batch does it:

@echo ### copy script entry points to pywikibot\scripts
copy pwb.py .\pywikibot\scripts\*
copy generate*.py .\pywikibot\scripts\*

@echo
@echo ### create a new package
py -3 setup.py sdist

@echo
@echo ### delete script entry points
del pywikibot\scripts\pwb.py
del pywikibot\scripts\generate*

It is not necessary to change the git repository.

Installing the local package
Having al local package created by the commands above it can be installed as a side package without the pypi index is needed:

pip uninstall pywikibot
pip install --no-index --find-links=dist pywikibot

Running the pwb wrapper
create a minimal user-config.py:

mylang = 'en'  # the default site code you are working on
family = 'wikipedia'  # the default family
usernames['wikipedia']['en'] = 'Test'  # the bot account name
user_script_paths = ['c:\\pwb.git.core.scripts']

The user_script_path is important. You can add any path to it including side package paths (but there are some TODOs, e.g. find the absolute side package path. probably a different side package path is appropriate)
run the script
pwb.exe <global option> <script name> <global and local options>
for example:
pwb.exe touche -page:user:xqt <-- yes it works with touche
and have the full benefit of the pwb script wrapper:

find any script in any folder given by user_scripts_path
add .py ending automatically
similar search for script names, ignore spelling mistakes
enable global options even if the script does not support it
and in future: enable scripts installed as side package

I tested the patch:

$ git review -d I337beab3732d6a4e9567f56af885a27daa004a0d
Downloading refs/changes/57/560057/11 from gerrit
Switched to branch "review/xqt/T107629"
$ python setup.py sdist
running sdist
running egg_info
creating pywikibot.egg-info
writing pywikibot.egg-info/PKG-INFO
writing dependency_links to pywikibot.egg-info/dependency_links.txt
writing entry points to pywikibot.egg-info/entry_points.txt
writing requirements to pywikibot.egg-info/requires.txt
writing top-level names to pywikibot.egg-info/top_level.txt
writing manifest file 'pywikibot.egg-info/SOURCES.txt'
reading manifest file 'pywikibot.egg-info/SOURCES.txt'
writing manifest file 'pywikibot.egg-info/SOURCES.txt'
running check
creating pywikibot-3.0.20200527
creating pywikibot-3.0.20200527/pywikibot
creating pywikibot-3.0.20200527/pywikibot.egg-info
creating pywikibot-3.0.20200527/pywikibot/comms
creating pywikibot-3.0.20200527/pywikibot/data
creating pywikibot-3.0.20200527/pywikibot/families
creating pywikibot-3.0.20200527/pywikibot/page
creating pywikibot-3.0.20200527/pywikibot/scripts
creating pywikibot-3.0.20200527/pywikibot/site
creating pywikibot-3.0.20200527/pywikibot/specialbots
creating pywikibot-3.0.20200527/pywikibot/tools
creating pywikibot-3.0.20200527/pywikibot/userinterfaces
copying files to pywikibot-3.0.20200527...
copying README.rst -> pywikibot-3.0.20200527
copying setup.py -> pywikibot-3.0.20200527
copying pywikibot/__init__.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/_wbtypes.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/bot.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/bot_choice.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/config2.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/cosmetic_changes.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/daemonize.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/date.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/diff.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/echo.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/editor.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/exceptions.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/family.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/fixes.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/flow.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/i18n.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/interwiki_graph.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/logentries.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/logging.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/login.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/pagegenerators.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/plural.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/proofreadpage.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/site_detect.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/textlib.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/throttle.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/titletranslate.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/version.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot/xmlreader.py -> pywikibot-3.0.20200527/pywikibot
copying pywikibot.egg-info/PKG-INFO -> pywikibot-3.0.20200527/pywikibot.egg-info
copying pywikibot.egg-info/SOURCES.txt -> pywikibot-3.0.20200527/pywikibot.egg-info
copying pywikibot.egg-info/dependency_links.txt -> pywikibot-3.0.20200527/pywikibot.egg-info
copying pywikibot.egg-info/entry_points.txt -> pywikibot-3.0.20200527/pywikibot.egg-info
copying pywikibot.egg-info/requires.txt -> pywikibot-3.0.20200527/pywikibot.egg-info
copying pywikibot.egg-info/top_level.txt -> pywikibot-3.0.20200527/pywikibot.egg-info
copying pywikibot/comms/__init__.py -> pywikibot-3.0.20200527/pywikibot/comms
copying pywikibot/comms/eventstreams.py -> pywikibot-3.0.20200527/pywikibot/comms
copying pywikibot/comms/http.py -> pywikibot-3.0.20200527/pywikibot/comms
copying pywikibot/comms/threadedhttp.py -> pywikibot-3.0.20200527/pywikibot/comms
copying pywikibot/data/__init__.py -> pywikibot-3.0.20200527/pywikibot/data
copying pywikibot/data/api.py -> pywikibot-3.0.20200527/pywikibot/data
copying pywikibot/data/mysql.py -> pywikibot-3.0.20200527/pywikibot/data
copying pywikibot/data/sparql.py -> pywikibot-3.0.20200527/pywikibot/data
copying pywikibot/data/wikistats.py -> pywikibot-3.0.20200527/pywikibot/data
copying pywikibot/families/__init__.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/commons_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/foundation_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/i18n_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/incubator_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/lyricwiki_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/mediawiki_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/meta_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/omegawiki_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/osm_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/outreach_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/species_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/vikidia_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/wikibooks_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/wikidata_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/wikimania_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/wikimediachapter_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/wikinews_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/wikipedia_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/wikiquote_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/wikisource_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/wikitech_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/wikiversity_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/wikivoyage_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/wiktionary_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/families/wowwiki_family.py -> pywikibot-3.0.20200527/pywikibot/families
copying pywikibot/page/__init__.py -> pywikibot-3.0.20200527/pywikibot/page
copying pywikibot/scripts/__init__.py -> pywikibot-3.0.20200527/pywikibot/scripts
copying pywikibot/site/__init__.py -> pywikibot-3.0.20200527/pywikibot/site
copying pywikibot/specialbots/__init__.py -> pywikibot-3.0.20200527/pywikibot/specialbots
copying pywikibot/specialbots/_unlink.py -> pywikibot-3.0.20200527/pywikibot/specialbots
copying pywikibot/specialbots/_upload.py -> pywikibot-3.0.20200527/pywikibot/specialbots
copying pywikibot/tools/__init__.py -> pywikibot-3.0.20200527/pywikibot/tools
copying pywikibot/tools/_logging.py -> pywikibot-3.0.20200527/pywikibot/tools
copying pywikibot/tools/chars.py -> pywikibot-3.0.20200527/pywikibot/tools
copying pywikibot/tools/djvu.py -> pywikibot-3.0.20200527/pywikibot/tools
copying pywikibot/tools/formatter.py -> pywikibot-3.0.20200527/pywikibot/tools
copying pywikibot/tools/ip.py -> pywikibot-3.0.20200527/pywikibot/tools
copying pywikibot/userinterfaces/__init__.py -> pywikibot-3.0.20200527/pywikibot/userinterfaces
copying pywikibot/userinterfaces/gui.py -> pywikibot-3.0.20200527/pywikibot/userinterfaces
copying pywikibot/userinterfaces/terminal_interface.py -> pywikibot-3.0.20200527/pywikibot/userinterfaces
copying pywikibot/userinterfaces/terminal_interface_base.py -> pywikibot-3.0.20200527/pywikibot/userinterfaces
copying pywikibot/userinterfaces/terminal_interface_unix.py -> pywikibot-3.0.20200527/pywikibot/userinterfaces
copying pywikibot/userinterfaces/terminal_interface_win32.py -> pywikibot-3.0.20200527/pywikibot/userinterfaces
copying pywikibot/userinterfaces/transliteration.py -> pywikibot-3.0.20200527/pywikibot/userinterfaces
copying pywikibot/userinterfaces/win32_unicode.py -> pywikibot-3.0.20200527/pywikibot/userinterfaces
Writing pywikibot-3.0.20200527/setup.cfg
creating dist
Creating tar archive
removing 'pywikibot-3.0.20200527' (and everything under it)
$ cd dist/
$ sudo pip install pywikibot-3.0.20200527.tar.gz 
[sudo] heslo pro user: 
Processing ./pywikibot-3.0.20200527.tar.gz
Requirement already satisfied: requests>=2.20.1 in /usr/lib/python3.8/site-packages (from pywikibot==3.0.20200527) (2.23.0)
Requirement already satisfied: chardet>=3.0.2 in /usr/lib/python3.8/site-packages (from requests>=2.20.1->pywikibot==3.0.20200527) (3.0.4)
Requirement already satisfied: idna>=2.5 in /usr/lib/python3.8/site-packages (from requests>=2.20.1->pywikibot==3.0.20200527) (2.9)
Requirement already satisfied: urllib3>=1.21.1 in /usr/lib/python3.8/site-packages (from requests>=2.20.1->pywikibot==3.0.20200527) (1.25.9)
Installing collected packages: pywikibot
    Running setup.py install for pywikibot ... done
Successfully installed pywikibot-3.0.20200527

This creates the files pwb, gff, guf... (Python executables; like pwb.py, but without .py) in /usr/bin folder (whereas Pywikibot is installed into /usr/lib/python3.8/site-packages/pywikibot) with the following content:

/usr/bin/pwb

#!/usr/bin/python
# EASY-INSTALL-ENTRY-SCRIPT: 'pywikibot==3.0.20200527','console_scripts','pwb'
__requires__ = 'pywikibot==3.0.20200527'
import re
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(
        load_entry_point('pywikibot==3.0.20200527', 'console_scripts', 'pwb')()
    )

So far so good.

But if I run any of these, there is the following traceback:

$ guf
Traceback (most recent call last):
  File "/usr/bin/guf", line 11, in <module>
    load_entry_point('pywikibot==3.0.20200527', 'console_scripts', 'guf')()
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 490, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2853, in load_entry_point
    return ep.load()
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2453, in load
    return self.resolve()
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2459, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/home/user/.local/lib/python3.8/site-packages/pywikibot/__init__.py", line 25, in <module>
    from pywikibot.bot import (
  File "/home/user/.local/lib/python3.8/site-packages/pywikibot/bot.py", line 114, in <module>
    from pywikibot import config2 as config
  File "/home/user/.local/lib/python3.8/site-packages/pywikibot/config2.py", line 377, in <module>
    base_dir = get_base_dir()
  File "/home/user/.local/lib/python3.8/site-packages/pywikibot/config2.py", line 371, in get_base_dir
    raise RuntimeError(exc_text)
RuntimeError: No user-config.py found in directory '/mnt/B4D0D63BD0D60410/Tvorba/pywikibota/dist'.
  Please check that user-config.py is stored in the correct location.
  Directory where user-config.py is searched is determined as follows:

    Return the directory in which user-specific information is stored.

    This is determined in the following order:
     1.  If the script was called with a -dir: argument, use the directory
         provided in this argument.
     2.  If the user has a PYWIKIBOT_DIR environment variable, use the value
         of it.
     3.  If user-config is present in current directory, use the current
         directory.
     4.  If user-config is present in pwb.py directory, use that directory
     5.  Use (and if necessary create) a 'pywikibot' folder under
         'Application Data' or 'AppData\Roaming' (Windows) or
         '.pywikibot' directory (Unix and similar) under the user's home
         directory.

    Set PYWIKIBOT_NO_USER_CONFIG=1 to disable loading user-config.py

    @param test_directory: Assume that a user config file exists in this
        directory. Used to test whether placing a user config file in this
        directory will cause it to be selected as the base directory.
    @type test_directory: str or None
    @rtype: str

Oh, that looks great. I had the same. At this point the user-config.py is required. I did it by hand but guf should be do the job later.

guf and gff should always work without user-config.py. Anyway, a simple workaround of an empty user-config.py file works usually too.

Change 560057 had a related patch set uploaded (by Xqt; owner: Xqt):
[pywikibot/core@master] [dist] Add pwb.py wrapper and generate*.py to pywkibot package

https://gerrit.wikimedia.org/r/560057

gerritbot added a project: Patch-For-Review.May 28 2020, 10:32 AM

Xqt added a subtask: T270474: Allow using pywikibot without user-config.py and PYWIKIBOT_NO_USER_CONFIG=1.Dec 18 2020, 8:19 AM

Xqt removed a subtask: T244684: Allow pywikibot import without presence of user-config.py.Dec 18 2020, 11:43 AM

Xqt added a parent task: T270480: add version.py script to pywikibot pypi package.Dec 18 2020, 11:45 AM

Xqt closed subtask T270474: Allow using pywikibot without user-config.py and PYWIKIBOT_NO_USER_CONFIG=1 as Resolved.Dec 18 2020, 8:20 PM

Change 738576 had a related patch set uploaded (by Xqt; author: Xqt):

[pywikibot/core@master] [IMP] Make a pywikibot entry point for scripts

https://gerrit.wikimedia.org/r/738576

Note: Python 3.6 is the minimal requirement to add entry points of scripts packages to be runned with Pywikibot's pwb wrapper script