Page MenuHomePhabricator

APT sources are broken on Quarry hosts: "GPG error, the following signatures were invalid"
Closed, ResolvedPublic

Description

Since 2020-10-20, apt lists are broken on quarry hosts (web and db at least).
As it refers to obscures parts of wmcs infra I prefer letting Cloud team looking into :)

First occurrence found in logs is the following (quarry-db-01):

Oct 20 16:00:59 quarry-web-01 puppet-agent-cronjob: W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://apt.wikimedia.org/wikimedia jessie-wikimedia InRelease: The following signatures were invalid: DB3DC2BD4CD504EF2D908FC509DBD9F93F6CD44A
Oct 20 16:00:59 quarry-web-01 puppet-agent-cronjob: W: Failed to fetch http://apt.wikimedia.org/wikimedia/dists/jessie-wikimedia/InRelease  The following signatures were invalid: DB3DC2BD4CD504EF2D908FC509DBD9F93F6CD44A
Oct 20 16:00:59 quarry-web-01 puppet-agent-cronjob: W: Some index files failed to download. They have been ignored, or old ones used instead.

And puppet can't do its job:

Oct 23 20:34:10 quarry-web-01 puppet-agent[13893]: Applying configuration version '(4241fd4a30) Dzahn - Update update_parsoid.sh script for use on testreduce1001'
Oct 23 20:34:13 quarry-web-01 puppet-agent[13893]: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install python3-novaclient' returned 100: Reading package lists...
Oct 23 20:34:13 quarry-web-01 puppet-agent[13893]: Building dependency tree...
Oct 23 20:34:13 quarry-web-01 puppet-agent[13893]: Reading state information...
Oct 23 20:34:13 quarry-web-01 puppet-agent[13893]: Some packages could not be installed. This may mean that you have
Oct 23 20:34:13 quarry-web-01 puppet-agent[13893]: requested an impossible situation or if you are using the unstable
Oct 23 20:34:13 quarry-web-01 puppet-agent[13893]: distribution that some required packages have not yet been created
Oct 23 20:34:13 quarry-web-01 puppet-agent[13893]: or been moved out of Incoming.
Oct 23 20:34:13 quarry-web-01 puppet-agent[13893]: The following information may help to resolve the situation:
Oct 23 20:34:13 quarry-web-01 puppet-agent[13893]: 
Oct 23 20:34:13 quarry-web-01 puppet-agent[13893]: The following packages have unmet dependencies:
Oct 23 20:34:13 quarry-web-01 puppet-agent[13893]:  python3-novaclient : Depends: python3-oslo.serialization (>= 1.10.0) but it is not going to be installed
Oct 23 20:34:13 quarry-web-01 puppet-agent[13893]: E: Unable to correct problems, you have held broken packages.
Oct 23 20:34:13 quarry-web-01 puppet-agent[13893]: (/Stage[main]/Openstack::Clientpackages::Vms::Common/Package[python3-novaclient]/ensure) change from 'purged' to 'present' failed: Execution of '/usr/bin/ap
Oct 23 20:34:13 quarry-web-01 puppet-agent[13893]: (/Stage[main]/Openstack::Clientpackages::Vms::Common/Package[python3-novaclient]/ensure) Building dependency tree...
Oct 23 20:34:13 quarry-web-01 puppet-agent[13893]: (/Stage[main]/Openstack::Clientpackages::Vms::Common/Package[python3-novaclient]/ensure) Reading state information...
Oct 23 20:34:13 quarry-web-01 puppet-agent[13893]: (/Stage[main]/Openstack::Clientpackages::Vms::Common/Package[python3-novaclient]/ensure) Some packages could not be installed. This may mean that you have
Oct 23 20:34:13 quarry-web-01 puppet-agent[13893]: (/Stage[main]/Openstack::Clientpackages::Vms::Common/Package[python3-novaclient]/ensure) requested an impossible situation or if you are using the unstable
Oct 23 20:34:13 quarry-web-01 puppet-agent[13893]: (/Stage[main]/Openstack::Clientpackages::Vms::Common/Package[python3-novaclient]/ensure) distribution that some required packages have not yet been created
Oct 23 20:34:13 quarry-web-01 puppet-agent[13893]: (/Stage[main]/Openstack::Clientpackages::Vms::Common/Package[python3-novaclient]/ensure) or been moved out of Incoming.
Oct 23 20:34:13 quarry-web-01 puppet-agent[13893]: (/Stage[main]/Openstack::Clientpackages::Vms::Common/Package[python3-novaclient]/ensure) The following information may help to resolve the situation:
Oct 23 20:34:13 quarry-web-01 puppet-agent[13893]: (/Stage[main]/Openstack::Clientpackages::Vms::Common/Package[python3-novaclient]/ensure)
Oct 23 20:34:13 quarry-web-01 puppet-agent[13893]: (/Stage[main]/Openstack::Clientpackages::Vms::Common/Package[python3-novaclient]/ensure) The following packages have unmet dependencies:
Oct 23 20:34:13 quarry-web-01 puppet-agent[13893]: (/Stage[main]/Openstack::Clientpackages::Vms::Common/Package[python3-novaclient]/ensure)  python3-novaclient : Depends: python3-oslo.serialization (>= 1.10.
Oct 23 20:34:13 quarry-web-01 puppet-agent[13893]: (/Stage[main]/Openstack::Clientpackages::Vms::Common/Package[python3-novaclient]/ensure) E: Unable to correct problems, you have held broken packages.
Oct 23 20:34:13 quarry-web-01 puppet-agent[13893]: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install python3-glanceclient' returned 100: Reading package lists...
Oct 23 20:34:13 quarry-web-01 puppet-agent[13893]: Building dependency tree...
Oct 23 20:34:13 quarry-web-01 puppet-agent[13893]: Reading state information...
Oct 23 20:34:13 quarry-web-01 puppet-agent[13893]: Some packages could not be installed. This may mean that you have
Oct 23 20:34:13 quarry-web-01 puppet-agent[13893]: requested an impossible situation or if you are using the unstable
Oct 23 20:34:13 quarry-web-01 puppet-agent[13893]: distribution that some required packages have not yet been created
Oct 23 20:34:13 quarry-web-01 puppet-agent[13893]: or been moved out of Incoming.
framawiki@quarry-web-01:/var/log$ sudo apt-get update
Hit:1 http://mirrors.wikimedia.org/debian stretch-backports InRelease
Ign:2 http://deb.debian.org/debian stretch InRelease                                                                                            
Hit:3 http://deb.debian.org/debian stretch-updates InRelease                                                                                    
Hit:4 http://security.debian.org stretch/updates InRelease                                                                                      
Hit:5 http://apt.wikimedia.org/wikimedia stretch-wikimedia InRelease                                                                            
Get:6 http://apt.wikimedia.org/wikimedia jessie-wikimedia InRelease [54.3 kB]                                                                   
Hit:7 http://deb.debian.org/debian stretch Release                                                                                              
Err:6 http://apt.wikimedia.org/wikimedia jessie-wikimedia InRelease
  The following signatures were invalid: DB3DC2BD4CD504EF2D908FC509DBD9F93F6CD44A
Fetched 54.3 kB in 0s (107 kB/s)
Reading package lists... Done
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://apt.wikimedia.org/wikimedia jessie-wikimedia InRelease: The following signatures were invalid: DB3DC2BD4CD504EF2D908FC509DBD9F93F6CD44A
W: Failed to fetch http://apt.wikimedia.org/wikimedia/dists/jessie-wikimedia/InRelease  The following signatures were invalid: DB3DC2BD4CD504EF2D908FC509DBD9F93F6CD44A
W: Some index files failed to download. They have been ignored, or old ones used instead.

framawiki@quarry-web-01:/var/log$ sudo apt-get upgrade
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
  python-osc-lib python-rfc3986
Use 'sudo apt autoremove' to remove them.
The following packages have been kept back:
  debdeploy-client git git-man libpam-systemd libsystemd0 linux-image-amd64 systemd
The following packages will be DOWNGRADED:
  ifupdown libapparmor1 libsodium18 libsqlite3-0 python-cffi-backend python-cinderclient python-dateutil python-debtcollector python-designateclient python-funcsigs python-functools32 python-glanceclient
  python-idna python-ipaddress python-iso8601 python-jsonschema python-keyring python-keystoneclient python-monotonic python-msgpack python-netaddr python-neutronclient python-novaclient
  python-openstackclient python-openstacksdk python-os-client-config python-oslo.config python-oslo.i18n python-oslo.serialization python-pbr python-pkg-resources python-positional python-pyasn1
  python-redis python-requests python-requestsexceptions python-setuptools python-six python-urllib3 python-warlock python3-pkg-resources python3-requests python3-six python3-urllib3 ruby-deep-merge
0 upgraded, 0 newly installed, 45 downgraded, 0 to remove and 13 not upgraded.
Need to get 3,862 kB of archives.
After this operation, 897 kB disk space will be freed.
Do you want to continue? [Y/n] 
Err:1 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 libsqlite3-0 amd64 3.16.2-3~bpo8+1
  404  Not Found [IP: 208.80.154.30 80]
Err:2 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 libsodium18 amd64 1.0.11-1~bpo8+1
  404  Not Found [IP: 208.80.154.30 80]
Err:3 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 ifupdown amd64 0.8.13~bpo8+1
  404  Not Found [IP: 208.80.154.30 80]
Err:4 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 libapparmor1 amd64 2.10.95-4~bpo8+2
  404  Not Found [IP: 208.80.154.30 80]
Err:5 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 python-cffi-backend amd64 1.9.1-2~bpo8+1
  404  Not Found [IP: 208.80.154.30 80]
Err:6 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 python-setuptools all 33.1.1-1~bpo8+1
  404  Not Found [IP: 208.80.154.30 80]
Err:7 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 python-pkg-resources all 33.1.1-1~bpo8+1
  404  Not Found [IP: 208.80.154.30 80]
Err:8 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 python-six all 1.10.0-3~bpo8+1
  404  Not Found [IP: 208.80.154.30 80]
Err:9 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 python-pbr all 1.8.0-4.1~bpo8+1
  404  Not Found [IP: 208.80.154.30 80]
Err:10 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 python-funcsigs all 1.0.2-3~bpo8+1
  404  Not Found [IP: 208.80.154.30 80]
Err:11 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 python-debtcollector all 1.3.0-2~bpo8+1
  404  Not Found [IP: 208.80.154.30 80]
Err:12 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 python-iso8601 all 0.1.11-1~bpo8+1
  404  Not Found [IP: 208.80.154.30 80]
Err:13 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 python-keyring all 8.5.1-1~bpo8+1
  404  Not Found [IP: 208.80.154.30 80]
Err:14 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 python-netaddr all 0.7.18-1~bpo8+1
  404  Not Found [IP: 208.80.154.30 80]
Err:15 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 python-openstackclient all 2.3.0-2~bpo8+1
  404  Not Found [IP: 208.80.154.30 80]
Err:16 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 python-keystoneclient all 1:2.3.1-2~bpo8+1
  404  Not Found [IP: 208.80.154.30 80]
Err:17 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 python-oslo.config all 1:3.9.0-3~bpo8+1
  404  Not Found [IP: 208.80.154.30 80]
Err:18 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 python-oslo.i18n all 3.5.0-2~bpo8+1
  404  Not Found [IP: 208.80.154.30 80]
Err:19 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 python-msgpack amd64 0.4.6-1~bpo8+1
  404  Not Found [IP: 208.80.154.30 80]
Err:20 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 python-oslo.serialization all 2.4.0-2~bpo8+1
  404  Not Found [IP: 208.80.154.30 80]
Err:21 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 python-positional all 1.0.1-2~bpo8+1
  404  Not Found [IP: 208.80.154.30 80]
Err:22 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 python-requests all 2.11.1-1~bpo8+1
  404  Not Found [IP: 208.80.154.30 80]
Err:23 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 python-urllib3 all 1.16-1~bpo8+1
  404  Not Found [IP: 208.80.154.30 80]
Err:24 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 python-functools32 all 3.2.3.2-2~bpo8+1
  404  Not Found [IP: 208.80.154.30 80]
Err:25 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 python-jsonschema all 2.5.1-4~bpo8+1
  404  Not Found [IP: 208.80.154.30 80]
Err:26 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 python-warlock all 1.1.0-1~bpo8+1
  404  Not Found [IP: 208.80.154.30 80]
Err:27 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 python-glanceclient all 1:2.0.0-2~bpo8+1
  404  Not Found [IP: 208.80.154.30 80]
Err:28 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 python-requestsexceptions all 1.1.1-1~bpo8+1
  404  Not Found [IP: 208.80.154.30 80]
Err:29 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 python-os-client-config all 1.16.0-1~bpo8+1
  404  Not Found [IP: 208.80.154.30 80]
Err:30 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 python-neutronclient all 1:4.1.1-2~bpo8+1
  404  Not Found [IP: 208.80.154.30 80]
Err:31 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 python-novaclient all 2:3.3.1-3~bpo8+1
  404  Not Found [IP: 208.80.154.30 80]
Err:32 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 python-openstacksdk all 0.8.1-2~bpo8+1
  404  Not Found [IP: 208.80.154.30 80]
Err:33 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 python-cinderclient all 1:1.6.0-2~bpo8+1
  404  Not Found [IP: 208.80.154.30 80]
Err:34 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 python-dateutil all 2.4.2-1~bpo8+1
  404  Not Found [IP: 208.80.154.30 80]
Err:35 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 python-designateclient all 2.1.0-2~bpo8+1
  404  Not Found [IP: 208.80.154.30 80]
Err:36 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 python-idna all 2.0-3~bpo8+1
  404  Not Found [IP: 208.80.154.30 80]
Err:37 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 python-ipaddress all 1.0.16-1~bpo8+1
  404  Not Found [IP: 208.80.154.30 80]
Err:38 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 python-monotonic all 0.6-2~bpo8+1
  404  Not Found [IP: 208.80.154.30 80]
Err:39 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 python-pyasn1 all 0.1.9-1~bpo8+1
  404  Not Found [IP: 208.80.154.30 80]
Err:40 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 python-redis all 2.10.5-2~bpo8+1
  404  Not Found [IP: 208.80.154.30 80]
Err:41 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 python3-pkg-resources all 33.1.1-1~bpo8+1
  404  Not Found [IP: 208.80.154.30 80]
Err:42 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 python3-six all 1.10.0-3~bpo8+1
  404  Not Found [IP: 208.80.154.30 80]
Err:43 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 python3-requests all 2.11.1-1~bpo8+1
  404  Not Found [IP: 208.80.154.30 80]
Err:44 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 python3-urllib3 all 1.16-1~bpo8+1
  404  Not Found [IP: 208.80.154.30 80]
Err:45 http://apt.wikimedia.org/wikimedia jessie-wikimedia/openstack-mitaka-jessie amd64 ruby-deep-merge all 1.1.1-1~bpo8+1
  404  Not Found [IP: 208.80.154.30 80]
E: Failed to fetch http://apt.wikimedia.org/wikimedia/pool/openstack-mitaka-jessie/s/sqlite3/libsqlite3-0_3.16.2-3~bpo8+1_amd64.deb  404  Not Found [IP: 208.80.154.30 80]
E: Failed to fetch http://apt.wikimedia.org/wikimedia/pool/openstack-mitaka-jessie/libs/libsodium/libsodium18_1.0.11-1~bpo8+1_amd64.deb  404  Not Found [IP: 208.80.154.30 80]
E: Failed to fetch http://apt.wikimedia.org/wikimedia/pool/openstack-mitaka-jessie/i/ifupdown/ifupdown_0.8.13~bpo8+1_amd64.deb  404  Not Found [IP: 208.80.154.30 80]
E: Failed to fetch http://apt.wikimedia.org/wikimedia/pool/openstack-mitaka-jessie/a/apparmor/libapparmor1_2.10.95-4~bpo8+2_amd64.deb  404  Not Found [IP: 208.80.154.30 80]
E: Failed to fetch http://apt.wikimedia.org/wikimedia/pool/openstack-mitaka-jessie/p/python-cffi/python-cffi-backend_1.9.1-2~bpo8+1_amd64.deb  404  Not Found [IP: 208.80.154.30 80]
E: Failed to fetch http://apt.wikimedia.org/wikimedia/pool/openstack-mitaka-jessie/p/python-setuptools/python-setuptools_33.1.1-1~bpo8+1_all.deb  404  Not Found [IP: 208.80.154.30 80]
E: Failed to fetch http://apt.wikimedia.org/wikimedia/pool/openstack-mitaka-jessie/p/python-setuptools/python-pkg-resources_33.1.1-1~bpo8+1_all.deb  404  Not Found [IP: 208.80.154.30 80]
E: Failed to fetch http://apt.wikimedia.org/wikimedia/pool/openstack-mitaka-jessie/s/six/python-six_1.10.0-3~bpo8+1_all.deb  404  Not Found [IP: 208.80.154.30 80]
E: Failed to fetch http://apt.wikimedia.org/wikimedia/pool/openstack-mitaka-jessie/p/python-pbr/python-pbr_1.8.0-4.1~bpo8+1_all.deb  404  Not Found [IP: 208.80.154.30 80]
E: Failed to fetch http://apt.wikimedia.org/wikimedia/pool/openstack-mitaka-jessie/p/python-funcsigs/python-funcsigs_1.0.2-3~bpo8+1_all.deb  404  Not Found [IP: 208.80.154.30 80]
E: Failed to fetch http://apt.wikimedia.org/wikimedia/pool/openstack-mitaka-jessie/p/python-debtcollector/python-debtcollector_1.3.0-2~bpo8+1_all.deb  404  Not Found [IP: 208.80.154.30 80]
E: Failed to fetch http://apt.wikimedia.org/wikimedia/pool/openstack-mitaka-jessie/p/python-iso8601/python-iso8601_0.1.11-1~bpo8+1_all.deb  404  Not Found [IP: 208.80.154.30 80]
E: Failed to fetch http://apt.wikimedia.org/wikimedia/pool/openstack-mitaka-jessie/p/python-keyring/python-keyring_8.5.1-1~bpo8+1_all.deb  404  Not Found [IP: 208.80.154.30 80]
E: Failed to fetch http://apt.wikimedia.org/wikimedia/pool/openstack-mitaka-jessie/p/python-netaddr/python-netaddr_0.7.18-1~bpo8+1_all.deb  404  Not Found [IP: 208.80.154.30 80]
E: Failed to fetch http://apt.wikimedia.org/wikimedia/pool/openstack-mitaka-jessie/p/python-openstackclient/python-openstackclient_2.3.0-2~bpo8+1_all.deb  404  Not Found [IP: 208.80.154.30 80]
E: Failed to fetch http://apt.wikimedia.org/wikimedia/pool/openstack-mitaka-jessie/p/python-keystoneclient/python-keystoneclient_2.3.1-2~bpo8+1_all.deb  404  Not Found [IP: 208.80.154.30 80]
E: Failed to fetch http://apt.wikimedia.org/wikimedia/pool/openstack-mitaka-jessie/p/python-oslo.config/python-oslo.config_3.9.0-3~bpo8+1_all.deb  404  Not Found [IP: 208.80.154.30 80]
E: Failed to fetch http://apt.wikimedia.org/wikimedia/pool/openstack-mitaka-jessie/p/python-oslo.i18n/python-oslo.i18n_3.5.0-2~bpo8+1_all.deb  404  Not Found [IP: 208.80.154.30 80]
E: Failed to fetch http://apt.wikimedia.org/wikimedia/pool/openstack-mitaka-jessie/m/msgpack-python/python-msgpack_0.4.6-1~bpo8+1_amd64.deb  404  Not Found [IP: 208.80.154.30 80]
E: Failed to fetch http://apt.wikimedia.org/wikimedia/pool/openstack-mitaka-jessie/p/python-oslo.serialization/python-oslo.serialization_2.4.0-2~bpo8+1_all.deb  404  Not Found [IP: 208.80.154.30 80]
E: Failed to fetch http://apt.wikimedia.org/wikimedia/pool/openstack-mitaka-jessie/p/python-positional/python-positional_1.0.1-2~bpo8+1_all.deb  404  Not Found [IP: 208.80.154.30 80]
E: Failed to fetch http://apt.wikimedia.org/wikimedia/pool/openstack-mitaka-jessie/r/requests/python-requests_2.11.1-1~bpo8+1_all.deb  404  Not Found [IP: 208.80.154.30 80]
E: Failed to fetch http://apt.wikimedia.org/wikimedia/pool/openstack-mitaka-jessie/p/python-urllib3/python-urllib3_1.16-1~bpo8+1_all.deb  404  Not Found [IP: 208.80.154.30 80]
E: Failed to fetch http://apt.wikimedia.org/wikimedia/pool/openstack-mitaka-jessie/p/python-functools32/python-functools32_3.2.3.2-2~bpo8+1_all.deb  404  Not Found [IP: 208.80.154.30 80]
E: Failed to fetch http://apt.wikimedia.org/wikimedia/pool/openstack-mitaka-jessie/p/python-jsonschema/python-jsonschema_2.5.1-4~bpo8+1_all.deb  404  Not Found [IP: 208.80.154.30 80]
E: Failed to fetch http://apt.wikimedia.org/wikimedia/pool/openstack-mitaka-jessie/p/python-warlock/python-warlock_1.1.0-1~bpo8+1_all.deb  404  Not Found [IP: 208.80.154.30 80]
E: Failed to fetch http://apt.wikimedia.org/wikimedia/pool/openstack-mitaka-jessie/p/python-glanceclient/python-glanceclient_2.0.0-2~bpo8+1_all.deb  404  Not Found [IP: 208.80.154.30 80]
E: Failed to fetch http://apt.wikimedia.org/wikimedia/pool/openstack-mitaka-jessie/p/python-requestsexceptions/python-requestsexceptions_1.1.1-1~bpo8+1_all.deb  404  Not Found [IP: 208.80.154.30 80]
E: Failed to fetch http://apt.wikimedia.org/wikimedia/pool/openstack-mitaka-jessie/p/python-os-client-config/python-os-client-config_1.16.0-1~bpo8+1_all.deb  404  Not Found [IP: 208.80.154.30 80]
E: Failed to fetch http://apt.wikimedia.org/wikimedia/pool/openstack-mitaka-jessie/p/python-neutronclient/python-neutronclient_4.1.1-2~bpo8+1_all.deb  404  Not Found [IP: 208.80.154.30 80]
E: Failed to fetch http://apt.wikimedia.org/wikimedia/pool/openstack-mitaka-jessie/p/python-novaclient/python-novaclient_3.3.1-3~bpo8+1_all.deb  404  Not Found [IP: 208.80.154.30 80]
E: Failed to fetch http://apt.wikimedia.org/wikimedia/pool/openstack-mitaka-jessie/p/python-openstacksdk/python-openstacksdk_0.8.1-2~bpo8+1_all.deb  404  Not Found [IP: 208.80.154.30 80]
E: Failed to fetch http://apt.wikimedia.org/wikimedia/pool/openstack-mitaka-jessie/p/python-cinderclient/python-cinderclient_1.6.0-2~bpo8+1_all.deb  404  Not Found [IP: 208.80.154.30 80]
E: Failed to fetch http://apt.wikimedia.org/wikimedia/pool/openstack-mitaka-jessie/p/python-dateutil/python-dateutil_2.4.2-1~bpo8+1_all.deb  404  Not Found [IP: 208.80.154.30 80]
E: Failed to fetch http://apt.wikimedia.org/wikimedia/pool/openstack-mitaka-jessie/p/python-designateclient/python-designateclient_2.1.0-2~bpo8+1_all.deb  404  Not Found [IP: 208.80.154.30 80]
E: Failed to fetch http://apt.wikimedia.org/wikimedia/pool/openstack-mitaka-jessie/p/python-idna/python-idna_2.0-3~bpo8+1_all.deb  404  Not Found [IP: 208.80.154.30 80]
E: Failed to fetch http://apt.wikimedia.org/wikimedia/pool/openstack-mitaka-jessie/p/python-ipaddress/python-ipaddress_1.0.16-1~bpo8+1_all.deb  404  Not Found [IP: 208.80.154.30 80]
E: Failed to fetch http://apt.wikimedia.org/wikimedia/pool/openstack-mitaka-jessie/p/python-monotonic/python-monotonic_0.6-2~bpo8+1_all.deb  404  Not Found [IP: 208.80.154.30 80]
E: Failed to fetch http://apt.wikimedia.org/wikimedia/pool/openstack-mitaka-jessie/p/pyasn1/python-pyasn1_0.1.9-1~bpo8+1_all.deb  404  Not Found [IP: 208.80.154.30 80]
E: Failed to fetch http://apt.wikimedia.org/wikimedia/pool/openstack-mitaka-jessie/p/python-redis/python-redis_2.10.5-2~bpo8+1_all.deb  404  Not Found [IP: 208.80.154.30 80]
E: Failed to fetch http://apt.wikimedia.org/wikimedia/pool/openstack-mitaka-jessie/p/python-setuptools/python3-pkg-resources_33.1.1-1~bpo8+1_all.deb  404  Not Found [IP: 208.80.154.30 80]
E: Failed to fetch http://apt.wikimedia.org/wikimedia/pool/openstack-mitaka-jessie/s/six/python3-six_1.10.0-3~bpo8+1_all.deb  404  Not Found [IP: 208.80.154.30 80]
E: Failed to fetch http://apt.wikimedia.org/wikimedia/pool/openstack-mitaka-jessie/r/requests/python3-requests_2.11.1-1~bpo8+1_all.deb  404  Not Found [IP: 208.80.154.30 80]
E: Failed to fetch http://apt.wikimedia.org/wikimedia/pool/openstack-mitaka-jessie/p/python-urllib3/python3-urllib3_1.16-1~bpo8+1_all.deb  404  Not Found [IP: 208.80.154.30 80]
E: Failed to fetch http://apt.wikimedia.org/wikimedia/pool/openstack-mitaka-jessie/r/ruby-deep-merge/ruby-deep-merge_1.1.1-1~bpo8+1_all.deb  404  Not Found [IP: 208.80.154.30 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
framawiki@quarry-web-01:/var/log$