remove superfluous sleep between archiving each page
Closed, ResolvedPublic
Actions

Assigned To

None

Authored By

	whym
	Jun 27 2015, 3:02 AM

Description

The sleep in

try:
    archiver = PageArchiver(pg, a, salt, force)
    archiver.run()
    time.sleep(10)
except Exception:
    pywikibot.error(u'Error occurred while processing page %s' % pg)
    pywikibot.exception(tb=True)

appears to be of little use because

There is no need to wait for user action, because the script is normally run periodically, not interactively.
There is no need to limit API requests, because requests are sequentially emitted in this script.

This was suggested in https://gerrit.wikimedia.org/r/#/c/215871/.

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		None	T104046 remove superfluous sleep between archiving each page
		Resolved		murfel	T120894 Change name of checkimages.py argument -time to be -sleep

Event Timeline

whym created this task.Jun 27 2015, 3:02 AM

whym raised the priority of this task from to Needs Triage.

whym updated the task description. (Show Details)

whym added a project: Pywikibot-archivebot.py.

whym added subscribers: whym, jayvdb.

Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald TranscriptJun 27 2015, 3:02 AM

whym updated the task description. (Show Details)Jun 27 2015, 3:07 AM

whym set Security to None.

whym updated the task description. (Show Details)

whym added subscribers: Ladsgroup, Ricordisamoa.Jun 27 2015, 3:14 AM

Quickly looking at the other scripts, I see

quite a few sleeps are because of a server-error-retry cycle. some of these are also dubious in core, as the http and api layer have already retried the request; but e.g. the flickr api code does sleep-retry, which is appropriate because it doesnt use the api layer.
weblinkchecker.py does some very small sleeps while spinning down the threads
checkimages.py has a sleep argument -time: to slow down the bot
replace.py has a sleep argument -sleep: to slow down the bot between regex on each page? Quite odd : 9d415671b3
clean_sandbox.py and welcome.py has a sleep between bot runs - i.e. a delay before restarting the process again, which is effectively waiting for humans to do stuff

jayvdb added a subscriber: valhallasw.Jun 27 2015, 8:26 AM

jayvdb renamed this task from remove superfluous sleep made during moving from one page to another to remove superfluous sleep between archiving each page.Dec 2 2015, 11:53 AM

Restricted Application added a subscriber: StudiesWorld. · View Herald TranscriptDec 2 2015, 11:53 AM

I will mentor this task.

https://codein.withgoogle.com/dashboard/tasks/6239503404498944/

Aklapper moved this task from Proposed to Imported into GCI site on the Google-Code-In-2015 board.Dec 3 2015, 6:01 PM

For now, sleep time can be specified through a CLI option. If this option is specified, is it better to ignore it or to output a deprecation notice? Or is it better to leave the CLI option as is but change its default value from 10 to 0?

The archivebot doesnt currently have a sleep command line option, and I dont believe this script needs a custom command line option for sleeping. The sleep related code in archivebot can be deleted.
There is a global option -put_throttle which the user can can slow down the bot if they want to.

I added the -sleep option to archivebot in https://gerrit.wikimedia.org/r/#/c/222727/ in July - whether or not it should go is up to discussion. (Sorry for not linking the change here.)

jayvdb, I think there is.

for v in if_arg_value(arg, '-sleep'):
    sleep = int(v)

murfel closed subtask T120894: Change name of checkimages.py argument -time to be -sleep as Resolved.Dec 9 2015, 1:39 PM

https://gerrit.wikimedia.org/r/#/c/272309/

remove superfluous sleep between archiving each pageClosed, ResolvedPublicActions

Description

Related ObjectsSearch...

Event Timeline

remove superfluous sleep between archiving each page
Closed, ResolvedPublic
Actions

Related Objects
Search...