Page MenuHomePhabricator

In category_graph.py, Gracefully Handle Invalid File Path Passed in as Argument
Closed, ResolvedPublic

Description

Currently, invalid file paths passed in via the -to argument result in an unhandled FileNotFoundError exception. This exception only occurs after all of the subcategories have been added to the graph object, which could take a considerable amount of time on large categories.

Full Traceback:

Traceback (most recent call last):
  File "C:\git\core\pwb.py", line 39, in <module>
    sys.exit(main())
             ^^^^^^
  File "C:\git\core\pwb.py", line 35, in main
    runpy.run_path(str(path), run_name='__main__')
  File "<frozen runpy>", line 291, in run_path
  File "<frozen runpy>", line 98, in _run_module_code
  File "<frozen runpy>", line 88, in _run_code
  File "C:\git\core\pywikibot\scripts\wrapper.py", line 513, in <module>
    main()
  File "C:\git\core\pywikibot\scripts\wrapper.py", line 497, in main
    if not execute():
           ^^^^^^^^^
  File "C:\git\core\pywikibot\scripts\wrapper.py", line 484, in execute
    run_python_file(filename, script_args, module)
  File "C:\git\core\pywikibot\scripts\wrapper.py", line 147, in run_python_file
    exec(compile(source, filename, 'exec', dont_inherit=True),
  File "C:\git\core\scripts\category_graph.py", line 209, in <module>
    bot.run()
  File "C:\git\core\scripts\category_graph.py", line 179, in run
    self.dot.write(self.to + '.gv', encoding='utf-8')
  File "C:\python\Lib\site-packages\pydot.py", line 1825, in write
    with io.open(path, mode='wt', encoding=encoding) as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'nonexistent_directory/equations.gv'
Dropped throttle(s).
Closing network session.
Traceback (most recent call last):
  File "C:\git\core\pwb.py", line 39, in <module>
    sys.exit(main())
             ^^^^^^
  File "C:\git\core\pwb.py", line 35, in main
    runpy.run_path(str(path), run_name='__main__')
  File "<frozen runpy>", line 291, in run_path
  File "<frozen runpy>", line 98, in _run_module_code
  File "<frozen runpy>", line 88, in _run_code
  File "C:\git\core\pywikibot\scripts\wrapper.py", line 513, in <module>
    main()
  File "C:\git\core\pywikibot\scripts\wrapper.py", line 497, in main
    if not execute():
           ^^^^^^^^^
  File "C:\git\core\pywikibot\scripts\wrapper.py", line 484, in execute
    run_python_file(filename, script_args, module)
  File "C:\git\core\pywikibot\scripts\wrapper.py", line 147, in run_python_file
    exec(compile(source, filename, 'exec', dont_inherit=True),
  File "C:\git\core\scripts\category_graph.py", line 209, in <module>
    bot.run()
  File "C:\git\core\scripts\category_graph.py", line 179, in run
    self.dot.write(self.to + '.gv', encoding='utf-8')
  File "C:\python\Lib\site-packages\pydot.py", line 1825, in write
    with io.open(path, mode='wt', encoding=encoding) as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'nonexistent_directory/equations.gv'

CRITICAL: Exiting due to uncaught exception FileNotFoundError: [Errno 2] No such file or directory: 'nonexistent_directory/equations.gv'

This input should be either checked at the beginning of program execution and rejected gracefully if invalid, or the program should create the directories necessary for the file path if they do not exist.

Event Timeline

Change 960227 had a related patch set uploaded (by Enag2000; author: Enag2000):

[pywikibot/core@master] Validate file path input for category_graph.py.

https://gerrit.wikimedia.org/r/960227

Change 961275 had a related patch set uploaded (by Enag2000; author: Enag2000):

[pywikibot/core@master] Validate file path input for category_graph.py.

https://gerrit.wikimedia.org/r/961275

I again accidentally submitted for review twice. My mistake.

Change 960227 abandoned by Xqt:

[pywikibot/core@master] Validate file path input for category_graph.py.

Reason:

superseeded by https://gerrit.wikimedia.org/r/c/pywikibot/core/+/961275

https://gerrit.wikimedia.org/r/960227

Code check passed. Still awaiting completion of code review.

@Enag2000: Could you please provide a command line wich produces this exception above for CR

@Enag2000: Could you please provide a command line wich produces this exception above for CR

Yes. Running the following on English Wikipedia will produce the exception seen above. This assumes that the directory "InvalidDirectory" does not exist.

pwb.py -v category_graph -from Equations -to InvalidDirectory\Equations

@Enag2000: Could you please provide a command line wich produces this exception above for CR

Yes. Running the following on English Wikipedia will produce the exception seen above. This assumes that the directory "InvalidDirectory" does not exist.

pwb.py -v category_graph -from Equations -to InvalidDirectory\Equations

But your patch does not solve that:

D:\pwb\GIT\core>pwb.py  -simulate category_graph -from Equations -to InvalidDirectory\Equations
Scanning 'Equations'
Saving results
InvalidDirectory\Equations.gv
Traceback (most recent call last):
  File "D:\pwb\GIT\core\pwb.py", line 39, in <module>
    sys.exit(main())
             ^^^^^^
  File "D:\pwb\GIT\core\pwb.py", line 35, in main
    runpy.run_path(str(path), run_name='__main__')
  File "<frozen runpy>", line 291, in run_path
  File "<frozen runpy>", line 98, in _run_module_code
  File "<frozen runpy>", line 88, in _run_code
  File "D:\pwb\GIT\core\pywikibot\scripts\wrapper.py", line 515, in <module>
    main()
  File "D:\pwb\GIT\core\pywikibot\scripts\wrapper.py", line 499, in main
    if not execute():
           ^^^^^^^^^
  File "D:\pwb\GIT\core\pywikibot\scripts\wrapper.py", line 486, in execute
    run_python_file(filename, script_args, module)
  File "D:\pwb\GIT\core\pywikibot\scripts\wrapper.py", line 147, in run_python_file
    exec(compile(source, filename, 'exec', dont_inherit=True),
  File "D:\pwb\GIT\core\scripts\category_graph.py", line 230, in <module>
    bot.run()
  File "D:\pwb\GIT\core\scripts\category_graph.py", line 184, in run
    self.dot.write(self.to + '.gv', encoding='utf-8')
  File "C:\Python311\Lib\site-packages\pydot.py", line 1825, in write
    with io.open(path, mode='wt', encoding=encoding) as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'InvalidDirectory\\Equations.gv'
CRITICAL: Exiting due to uncaught exception FileNotFoundError: [Errno 2] No such file or directory: 'InvalidDirectory\\Equations.gv'

@Enag2000: Could you please provide a command line wich produces this exception above for CR

Yes. Running the following on English Wikipedia will produce the exception seen above. This assumes that the directory "InvalidDirectory" does not exist.

pwb.py -v category_graph -from Equations -to InvalidDirectory\Equations

But your patch does not solve that:

D:\pwb\GIT\core>pwb.py  -simulate category_graph -from Equations -to InvalidDirectory\Equations
Scanning 'Equations'
Saving results
InvalidDirectory\Equations.gv
Traceback (most recent call last):
  File "D:\pwb\GIT\core\pwb.py", line 39, in <module>
    sys.exit(main())
             ^^^^^^
  File "D:\pwb\GIT\core\pwb.py", line 35, in main
    runpy.run_path(str(path), run_name='__main__')
  File "<frozen runpy>", line 291, in run_path
  File "<frozen runpy>", line 98, in _run_module_code
  File "<frozen runpy>", line 88, in _run_code
  File "D:\pwb\GIT\core\pywikibot\scripts\wrapper.py", line 515, in <module>
    main()
  File "D:\pwb\GIT\core\pywikibot\scripts\wrapper.py", line 499, in main
    if not execute():
           ^^^^^^^^^
  File "D:\pwb\GIT\core\pywikibot\scripts\wrapper.py", line 486, in execute
    run_python_file(filename, script_args, module)
  File "D:\pwb\GIT\core\pywikibot\scripts\wrapper.py", line 147, in run_python_file
    exec(compile(source, filename, 'exec', dont_inherit=True),
  File "D:\pwb\GIT\core\scripts\category_graph.py", line 230, in <module>
    bot.run()
  File "D:\pwb\GIT\core\scripts\category_graph.py", line 184, in run
    self.dot.write(self.to + '.gv', encoding='utf-8')
  File "C:\Python311\Lib\site-packages\pydot.py", line 1825, in write
    with io.open(path, mode='wt', encoding=encoding) as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'InvalidDirectory\\Equations.gv'
CRITICAL: Exiting due to uncaught exception FileNotFoundError: [Errno 2] No such file or directory: 'InvalidDirectory\\Equations.gv'

Interesting. I just realized that I provided a path with '\' instead of '/' in my comment. The original exception in the issue body uses '/' and the solution was coded for ONLY '/', which is why the exception occurs with the command I sent. Running pwb.py -v category_graph -from Equations -to InvalidDirectory/Equations will not produce the exception. Given that file paths with either slash direction are valid, I can revise the solution to account for either scenario? I missed this when originally looking at the issue. Good catch and thanks for taking the time to review.

Change 961275 merged by jenkins-bot:

[pywikibot/core@master] Validate file path input for category_graph.py.

https://gerrit.wikimedia.org/r/961275