- Queries
- All Stories
- Search
- Advanced Search
- Transactions
- Transaction Logs
Advanced Search
Today
Yesterday
No, not really.. https://sql-optimizer.toolforge.org gives a good idea why it's so slow (it's doing a filesort on the page table as well as going through 34M+ rows of the revision table), but I don't have an idea why the execution speeds are so different. Anyhow, given that what you're running is a scheduled job, can you try running it on the analytics replicas instead of the web ones?
And it's down again. I ran sudo puppet node deactivate parse1002.eqiad.wmnet again to have it removed from the scap mediawiki image pulling list.
Sun, May 19
Sat, May 18
Fri, May 17
We need to have some quota in place to prevent a misbehaving tool from taking kube-apiserver down by creating hundreds or thousands of unfillable ReplicaSets (a similar thing has happened in the past, see T301081), but I have no objections to raising the deployment quota to match the pod one for example.
Thu, May 16
Created new aggregates for testing in codfw1dev:
taavi@cloudcontrol2004-dev ~ $ os aggregate create network-ovs taavi@cloudcontrol2004-dev ~ $ os aggregate create network-linuxbridge taavi@cloudcontrol2004-dev ~ $ os aggregate set --property network-agent=ovs network-ovs taavi@cloudcontrol2004-dev ~ $ os aggregate set --property network-agent=linuxbridge network-linuxbridge taavi@cloudcontrol2004-dev ~ $ os aggregate add host network-ovs cloudvirt2001-dev taavi@cloudcontrol2004-dev ~ $ os aggregate add host network-ovs cloudvirt2002-dev taavi@cloudcontrol2004-dev ~ $ os aggregate add host network-linuxbridge cloudvirt2003-dev taavi@cloudcontrol2004-dev ~ $ os aggregate add host network-linuxbridge cloudvirt2004-dev taavi@cloudcontrol2004-dev ~ $ os aggregate add host network-linuxbridge cloudvirt2005-dev taavi@cloudcontrol2004-dev ~ $ os aggregate add host network-linuxbridge cloudvirt2006-dev
And documented them at https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Host_aggregates.
Seems like Nova (or Placement) do not support a flavor targeting the lack of an aggregate property value on a host. So we will temporarily need to create two aggregates, and once the migration is done we can to remove the OVS/Linuxbridge filters from the flavors and retire those aggregates.
Done, thank you!
@Andrew, this seems related to the recent Horizon Django upgrade:
[Thu May 16 07:06:34.034090 2024] [wsgi:error] [pid 8:tid 139706610005760] [remote 208.80.154.150:59838] Internal Server Error: /project/member/add/ [Thu May 16 07:06:34.034114 2024] [wsgi:error] [pid 8:tid 139706610005760] [remote 208.80.154.150:59838] Traceback (most recent call last): [Thu May 16 07:06:34.034117 2024] [wsgi:error] [pid 8:tid 139706610005760] [remote 208.80.154.150:59838] File "/opt/lib/python/site-packages/django/core/handlers/exception.py", line 55, in inner [Thu May 16 07:06:34.034118 2024] [wsgi:error] [pid 8:tid 139706610005760] [remote 208.80.154.150:59838] response = get_response(request) [Thu May 16 07:06:34.034120 2024] [wsgi:error] [pid 8:tid 139706610005760] [remote 208.80.154.150:59838] File "/opt/lib/python/site-packages/django/core/handlers/base.py", line 197, in _get_response [Thu May 16 07:06:34.034122 2024] [wsgi:error] [pid 8:tid 139706610005760] [remote 208.80.154.150:59838] response = wrapped_callback(request, *callback_args, **callback_kwargs) [Thu May 16 07:06:34.034123 2024] [wsgi:error] [pid 8:tid 139706610005760] [remote 208.80.154.150:59838] File "/opt/lib/python/site-packages/horizon/decorators.py", line 51, in dec [Thu May 16 07:06:34.034124 2024] [wsgi:error] [pid 8:tid 139706610005760] [remote 208.80.154.150:59838] return view_func(request, *args, **kwargs) [Thu May 16 07:06:34.034125 2024] [wsgi:error] [pid 8:tid 139706610005760] [remote 208.80.154.150:59838] File "/opt/lib/python/site-packages/horizon/decorators.py", line 35, in dec [Thu May 16 07:06:34.034144 2024] [wsgi:error] [pid 8:tid 139706610005760] [remote 208.80.154.150:59838] return view_func(request, *args, **kwargs) [Thu May 16 07:06:34.034145 2024] [wsgi:error] [pid 8:tid 139706610005760] [remote 208.80.154.150:59838] File "/opt/lib/python/site-packages/horizon/decorators.py", line 35, in dec [Thu May 16 07:06:34.034146 2024] [wsgi:error] [pid 8:tid 139706610005760] [remote 208.80.154.150:59838] return view_func(request, *args, **kwargs) [Thu May 16 07:06:34.034147 2024] [wsgi:error] [pid 8:tid 139706610005760] [remote 208.80.154.150:59838] File "/opt/lib/python/site-packages/horizon/decorators.py", line 111, in dec [Thu May 16 07:06:34.034149 2024] [wsgi:error] [pid 8:tid 139706610005760] [remote 208.80.154.150:59838] return view_func(request, *args, **kwargs) [Thu May 16 07:06:34.034150 2024] [wsgi:error] [pid 8:tid 139706610005760] [remote 208.80.154.150:59838] File "/opt/lib/python/site-packages/django/views/generic/base.py", line 104, in view [Thu May 16 07:06:34.034151 2024] [wsgi:error] [pid 8:tid 139706610005760] [remote 208.80.154.150:59838] return self.dispatch(request, *args, **kwargs) [Thu May 16 07:06:34.034152 2024] [wsgi:error] [pid 8:tid 139706610005760] [remote 208.80.154.150:59838] File "/opt/lib/python/site-packages/django/views/generic/base.py", line 143, in dispatch [Thu May 16 07:06:34.034154 2024] [wsgi:error] [pid 8:tid 139706610005760] [remote 208.80.154.150:59838] return handler(request, *args, **kwargs) [Thu May 16 07:06:34.034155 2024] [wsgi:error] [pid 8:tid 139706610005760] [remote 208.80.154.150:59838] File "/opt/lib/python/site-packages/django/views/generic/edit.py", line 152, in post [Thu May 16 07:06:34.034156 2024] [wsgi:error] [pid 8:tid 139706610005760] [remote 208.80.154.150:59838] if form.is_valid(): [Thu May 16 07:06:34.034158 2024] [wsgi:error] [pid 8:tid 139706610005760] [remote 208.80.154.150:59838] File "/opt/lib/python/site-packages/django/forms/forms.py", line 201, in is_valid [Thu May 16 07:06:34.034159 2024] [wsgi:error] [pid 8:tid 139706610005760] [remote 208.80.154.150:59838] return self.is_bound and not self.errors [Thu May 16 07:06:34.034160 2024] [wsgi:error] [pid 8:tid 139706610005760] [remote 208.80.154.150:59838] File "/opt/lib/python/site-packages/django/forms/forms.py", line 196, in errors [Thu May 16 07:06:34.034161 2024] [wsgi:error] [pid 8:tid 139706610005760] [remote 208.80.154.150:59838] self.full_clean() [Thu May 16 07:06:34.034163 2024] [wsgi:error] [pid 8:tid 139706610005760] [remote 208.80.154.150:59838] File "/opt/lib/python/site-packages/django/forms/forms.py", line 434, in full_clean [Thu May 16 07:06:34.034166 2024] [wsgi:error] [pid 8:tid 139706610005760] [remote 208.80.154.150:59838] self._clean_form() [Thu May 16 07:06:34.034168 2024] [wsgi:error] [pid 8:tid 139706610005760] [remote 208.80.154.150:59838] File "/opt/lib/python/site-packages/django/forms/forms.py", line 455, in _clean_form [Thu May 16 07:06:34.034169 2024] [wsgi:error] [pid 8:tid 139706610005760] [remote 208.80.154.150:59838] cleaned_data = self.clean() [Thu May 16 07:06:34.034170 2024] [wsgi:error] [pid 8:tid 139706610005760] [remote 208.80.154.150:59838] File "/opt/lib/python/site-packages/wikimediamemberdashboard/views.py", line 180, in clean [Thu May 16 07:06:34.034172 2024] [wsgi:error] [pid 8:tid 139706610005760] [remote 208.80.154.150:59838] if cleaned_data['username'] and cleaned_data['shellname']: [Thu May 16 07:06:34.034173 2024] [wsgi:error] [pid 8:tid 139706610005760] [remote 208.80.154.150:59838] KeyError: 'shellname'
Wed, May 15
You should be able to create that proxy via Horizon now, please re-open if that doesn't work for some reason.
The scheduler is working correctly. The first item in "* */6 * * *" is the minute - you have specified a star which means that it's going to run the job every minute when the rest of the conditions (namely the current hour being divisible by six) matches. To make it run once every six hours, change the first star to be some number between 0 and 59.
Tue, May 14
This operation is scheduled for 2024-05-21 starting at around 14:00 UTC: https://lists.wikimedia.org/hyperkitty/list/cloud@lists.wikimedia.org/thread/AYR45M4X2RFUVRDXBDEVT4P7LJRYHV6J/
In T364459#9793907, @dcaro wrote:I have some questions:
- What tests will be run on each checkpoint? (linuxbridge active-ovs inactive, ovs active-linuxbridge inactive, ovs active-ovs inactive)
I need to check what exactly the network tests cookbook does, but possibly that. The main thing to look for is traffic getting in and out of the Neutron-managed VLAN and not getting duplicated.
Mon, May 13
This is surprisingly complicated because many Gruntfiles are using Object.assign() with the directories from extension.json.. https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/54 does its best but there's still a bunch of extensions that just won't parse for now.. I'm planning to simply disable the banana-check autofixer for those cases.
How does Horizon expose these /api/swift URLs? I would assume they'd only be embedded in pages that do have correct logged-out handling and this only happens if you try somehow use the URLs standalone?
May I ask why are we are talking about tools interacting with Horizon? I would hope everything happens either via the APIs directly or via Striker instead of introducing a second admin interface for Toolforge :-)
No, that is for the GitLab interface and this is for the Wikibugs IRC bot.
I created T364733: Add a nicer interface in Spicerack/wmcs-cookbooks to downtime Cloud VPS instances to improve the API somewhat, but silencing things works now so closing this one.
@aborrero can you think of any reason not to delete that zone?
Closing as I don't see any actionable end goal here. Individual projects are already tracked in their own tasks.
Closing this task since I don't see a clear end goal here. Current ongoing and planned work is already tracked in various tasks.
Closing this in favour of the slightly different approach in T358868: Use BGP to announce VM ranges from cloudnet to cloudgw that's likely going to be a part of the instance VLAN->VXLAN migration project.
In T187929#9748100, @cmooney wrote:The aggregate that is used for the cloud-private allocations should come from IPv6 space not announced to the internet/DFZ, or space that is announced but is filtered inbound on our CRs (similar to the WMF production private IPv6 aggregates).
The throttler logic in https://gitlab.wikimedia.org/repos/ci-tools/libup/-/blob/3602f02bf8e05c85a1db5bdab660e9cbc23bfba9/libup/tasks.py#L162 should be taking care if this already..
{{ping}} - this is blocking merges to a Wikimedia-deployed extension.
[Mon May 13 07:48:46.507418 2024] [wsgi:error] [pid 10:tid 139706677188352] [remote 208.80.154.150:34242] Internal Server Error: /api/swift/containers/dcarotest2/object/test.yaml [Mon May 13 07:48:46.507439 2024] [wsgi:error] [pid 10:tid 139706677188352] [remote 208.80.154.150:34242] Traceback (most recent call last): [Mon May 13 07:48:46.507441 2024] [wsgi:error] [pid 10:tid 139706677188352] [remote 208.80.154.150:34242] File "/opt/lib/python/site-packages/django/core/handlers/exception.py", line 55, in inner [Mon May 13 07:48:46.507443 2024] [wsgi:error] [pid 10:tid 139706677188352] [remote 208.80.154.150:34242] response = get_response(request) [Mon May 13 07:48:46.507445 2024] [wsgi:error] [pid 10:tid 139706677188352] [remote 208.80.154.150:34242] File "/opt/lib/python/site-packages/django/core/handlers/base.py", line 197, in _get_response [Mon May 13 07:48:46.507446 2024] [wsgi:error] [pid 10:tid 139706677188352] [remote 208.80.154.150:34242] response = wrapped_callback(request, *callback_args, **callback_kwargs) [Mon May 13 07:48:46.507448 2024] [wsgi:error] [pid 10:tid 139706677188352] [remote 208.80.154.150:34242] File "/opt/lib/python/site-packages/django/views/generic/base.py", line 104, in view [Mon May 13 07:48:46.507449 2024] [wsgi:error] [pid 10:tid 139706677188352] [remote 208.80.154.150:34242] return self.dispatch(request, *args, **kwargs) [Mon May 13 07:48:46.507451 2024] [wsgi:error] [pid 10:tid 139706677188352] [remote 208.80.154.150:34242] File "/opt/lib/python/site-packages/django/views/generic/base.py", line 143, in dispatch [Mon May 13 07:48:46.507452 2024] [wsgi:error] [pid 10:tid 139706677188352] [remote 208.80.154.150:34242] return handler(request, *args, **kwargs) [Mon May 13 07:48:46.507453 2024] [wsgi:error] [pid 10:tid 139706677188352] [remote 208.80.154.150:34242] File "/opt/lib/python/site-packages/openstack_dashboard/api/rest/swift.py", line 239, in get [Mon May 13 07:48:46.507454 2024] [wsgi:error] [pid 10:tid 139706677188352] [remote 208.80.154.150:34242] obj = api.swift.swift_get_object( [Mon May 13 07:48:46.507455 2024] [wsgi:error] [pid 10:tid 139706677188352] [remote 208.80.154.150:34242] File "/opt/lib/python/site-packages/openstack_dashboard/api/swift.py", line 45, in wrapper [Mon May 13 07:48:46.507457 2024] [wsgi:error] [pid 10:tid 139706677188352] [remote 208.80.154.150:34242] return function(*args, **kwargs) [Mon May 13 07:48:46.507458 2024] [wsgi:error] [pid 10:tid 139706677188352] [remote 208.80.154.150:34242] File "/opt/lib/python/site-packages/openstack_dashboard/api/swift.py", line 404, in swift_get_object [Mon May 13 07:48:46.507459 2024] [wsgi:error] [pid 10:tid 139706677188352] [remote 208.80.154.150:34242] headers, data = swift_api(request).get_object( [Mon May 13 07:48:46.507460 2024] [wsgi:error] [pid 10:tid 139706677188352] [remote 208.80.154.150:34242] File "/opt/lib/python/site-packages/openstack_dashboard/api/swift.py", line 132, in swift_api [Mon May 13 07:48:46.507461 2024] [wsgi:error] [pid 10:tid 139706677188352] [remote 208.80.154.150:34242] endpoint = base.url_for(request, 'object-store') [Mon May 13 07:48:46.507463 2024] [wsgi:error] [pid 10:tid 139706677188352] [remote 208.80.154.150:34242] File "/opt/lib/python/site-packages/openstack_dashboard/api/base.py", line 335, in url_for [Mon May 13 07:48:46.507464 2024] [wsgi:error] [pid 10:tid 139706677188352] [remote 208.80.154.150:34242] catalog = request.user.service_catalog [Mon May 13 07:48:46.507465 2024] [wsgi:error] [pid 10:tid 139706677188352] [remote 208.80.154.150:34242] File "/opt/lib/python/site-packages/django/utils/functional.py", line 268, in inner [Mon May 13 07:48:46.507466 2024] [wsgi:error] [pid 10:tid 139706677188352] [remote 208.80.154.150:34242] return func(_wrapped, *args) [Mon May 13 07:48:46.507467 2024] [wsgi:error] [pid 10:tid 139706677188352] [remote 208.80.154.150:34242] AttributeError: 'AnonymousUser' object has no attribute 'service_catalog'