Page MenuHomePhabricator
Create Task
Maniphest T359787

ImportError: cannot import name 'where' from 'certifi' (unknown location)
Closed, ResolvedPublic
Actions

Description

Seen while running scap backport today:

07:54:55 Started sync-masters
07:54:56 ['/usr/bin/scap', 'pull-master', 'deploy2002.codfw.wmnet'] (ran as mwdeploy@deploy1002.eqiad.wmnet) returned [1]: Traceback (most recent call last):
  File "/usr/bin/scap", line 32, in <module>
    from scap import cli
  File "/var/lib/scap/scap/lib/python3.7/site-packages/scap/__init__.py", line 27, in <module>
    import scap.plugins
  File "/var/lib/scap/scap/lib/python3.7/site-packages/scap/plugins/__init__.py", line 41, in <module>
    from ..cli import Application
  File "/var/lib/scap/scap/lib/python3.7/site-packages/scap/cli.py", line 36, in <module>
    import scap.config as config
  File "/var/lib/scap/scap/lib/python3.7/site-packages/scap/config.py", line 28, in <module>
    import scap.utils as utils
  File "/var/lib/scap/scap/lib/python3.7/site-packages/scap/utils.py", line 26, in <module>
    import requests
  File "/var/lib/scap/scap/lib/python3.7/site-packages/requests/__init__.py", line 147, in <module>
    from . import packages, utils
  File "/var/lib/scap/scap/lib/python3.7/site-packages/requests/utils.py", line 24, in <module>
    from . import certs
  File "/var/lib/scap/scap/lib/python3.7/site-packages/requests/certs.py", line 14, in <module>
    from certifi import where
ImportError: cannot import name 'where' from 'certifi' (unknown location)

07:54:56 sync-masters: 100% (in-flight: 0; ok: 0; fail: 1; left: 0)
07:54:56 1 masters had sync errors
07:54:56 Finished sync-masters (duration: 00m 01s)
07:54:56 Started sync-testservers-k8s
07:54:57 Started Running helmfile -e codfw --selector name=main apply in /srv/deployment-charts/helmfile.d/services/mw-misc
07:54:57 Started Running helmfile -e codfw --selector name=pinkunicorn apply in /srv/deployment-charts/helmfile.d/services/mw-debug
07:54:57 Started Running helmfile -e eqiad --selector name=main apply in /srv/deployment-charts/helmfile.d/services/mw-misc
07:54:57 Started Running helmfile -e eqiad --selector name=pinkunicorn apply in /srv/deployment-charts/helmfile.d/services/mw-debug
07:55:22 Finished Running helmfile -e eqiad --selector name=main apply in /srv/deployment-charts/helmfile.d/services/mw-misc (duration: 00m 24s)
07:55:24 Finished Running helmfile -e codfw --selector name=main apply in /srv/deployment-charts/helmfile.d/services/mw-misc (duration: 00m 27s)
07:55:25 Finished Running helmfile -e codfw --selector name=pinkunicorn apply in /srv/deployment-charts/helmfile.d/services/mw-debug (duration: 00m 27s)
07:55:34 Finished Running helmfile -e eqiad --selector name=pinkunicorn apply in /srv/deployment-charts/helmfile.d/services/mw-debug (duration: 00m 37s)
07:55:34 Finished sync-testservers-k8s (duration: 00m 37s)
07:55:34 Started sync-testservers
07:55:47 sync-testservers: 100% (in-flight: 0; ok: 4; fail: 0; left: 0)
07:55:47 Per-host sync duration: average 10.9s, median 10.7s
07:55:47 rsync transfer: average 462,950 bytes/host, total 1,851,800 bytes
07:55:47 Finished sync-testservers (duration: 00m 12s)
07:55:47 Started check-testservers
07:55:47 Executing check 'check_testservers_baremetal'
07:55:47 Executing check 'check_testservers_k8s'
07:56:30 Finished check-testservers (duration: 00m 43s)
07:56:30 kharlan and jdlrobson: Backport for [[gerrit:1009790|Exclude non-functional pages from night mode (T359183)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
Changes synced to the testservers. (see https://wikitech.wikimedia.org/wiki/Mwdebug)
Please do any necessary checks before continuing.
Continue with sync? (y/n): n
08:04:46 Sync cancelled.

Related Objects

Event Timeline

kostajh created this task.Mar 11 2024, 7:55 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 11 2024, 7:55 AM
kostajh added a comment.Mar 11 2024, 8:02 AM

It seems that scap pull-master doesn't work as a result, per https://gitlab.wikimedia.org/repos/releng/scap/-/blob/master/scap/main.py#L347 and https://gitlab.wikimedia.org/repos/releng/scap/-/blob/master/scap/main.py#L983

Mabualruz mentioned this in T359183: Exclude non-functional pages from night mode.Mar 11 2024, 8:06 AM
kostajh triaged this task as Unbreak Now! priority.Mar 11 2024, 8:06 AM
kostajh added a project: Release-Engineering-Team.
kostajh added a subscriber: dancy.

I've marked as UBN as it seems like this might cause problems in deployment, and I canceled the UTC morning backport as a result. Please lower the priority if I've misunderstood.

kostajh updated the task description. (Show Details)Mar 11 2024, 8:09 AM
hashar added subscribers: jnuche, hashar.Mar 11 2024, 8:41 AM

The list of requirements from the scap git repository:

requirements.txt
certifi==2022.9.24
requests==2.31.0

Which have not changed since 2022-09-26 and 2023-09-11 respectively.

The list of requirements is supposedly exhaustive and CI should fail if any is missing (I did work on that front a couple years ago via T316755 / 50673b4de284f152e71e12ff117d5de4852c7150 ). However when I look at the python environment on each of the deployment servers (/var/lib/scap/scap/bin/pip freeze), deploy1002 does not have certifi listed which is I guess why it is failing?

There is a /var/lib/scap/scap/lib/python3.7/site-packages/certifi/__pycache__:

drwxr-xr-x 2 scap scap 4096 Nov 10 14:13 ./
drwxr-xr-x 3 scap scap 4096 Mar  8 10:29 ../
-rw-r--r-- 1 root root 1896 Nov 10 14:13 core.cpython-37.pyc
-rw-r--r-- 1 root root  266 Nov 10 14:13 __init__.cpython-37.pyc
-rw-r--r-- 1 scap scap  389 Sep 18 16:13 __main__.cpython-37.pyc

Whereas deploy2002 has files from March 8th. My guess is scap needs to be reployed/reinstalled on deploy1002?

jnuche lowered the priority of this task from Unbreak Now! to Low.Mar 11 2024, 8:59 AM

I've tried reinstalling scap and that has re-added the missing dependency files:

[jnuche@deploy1002 ~]$ scap version
Traceback (most recent call last):
  File "/usr/bin/scap", line 32, in <module>
    from scap import cli
  File "/var/lib/scap/scap/lib/python3.7/site-packages/scap/__init__.py", line 27, in <module>
    import scap.plugins
  File "/var/lib/scap/scap/lib/python3.7/site-packages/scap/plugins/__init__.py", line 41, in <module>
    from ..cli import Application
  File "/var/lib/scap/scap/lib/python3.7/site-packages/scap/cli.py", line 36, in <module>
    import scap.config as config
  File "/var/lib/scap/scap/lib/python3.7/site-packages/scap/config.py", line 28, in <module>
    import scap.utils as utils
  File "/var/lib/scap/scap/lib/python3.7/site-packages/scap/utils.py", line 26, in <module>
    import requests
  File "/var/lib/scap/scap/lib/python3.7/site-packages/requests/__init__.py", line 147, in <module>
    from . import packages, utils
  File "/var/lib/scap/scap/lib/python3.7/site-packages/requests/utils.py", line 24, in <module>
    from . import certs
  File "/var/lib/scap/scap/lib/python3.7/site-packages/requests/certs.py", line 14, in <module>
    from certifi import where
ImportError: cannot import name 'where' from 'certifi' (unknown location)
[jnuche@deploy1002 ~]$ scap version
4.70.1-1

Backports should be working again.

I don't know what could have caused that dependency to disappear, or somehow be removed.

kostajh closed this task as Resolved.Mar 11 2024, 9:37 AM
kostajh claimed this task.

This time there was no issue:

09:36:25 Started sync-masters
09:36:35 sync-masters: 100% (in-flight: 0; ok: 1; fail: 0; left: 0)
09:36:35 Finished sync-masters (duration: 00m 09s)

thanks @jnuche and @hashar!

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL