Unable to delete "tiles.wmflabs.org" proxy entry via horizon
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	TheDJ
	Mar 10 2019, 8:45 PM

Description

I was trying to move tiles.wmflabs.org from http://10.68.16.103:80 to http://172.16.5.154:80 of the maps project.

Since the proxy panel only allows create/delete (not modify), I attempted to delete the existing rule for tiles.wmflabs.org but I was met with a Error: Unable to delete proxy: tiles.wmflabs.org. Of note, the confirmation dialog didn't list the name of this entry either: You have selected: . Please confirm your selection. This action cannot be undone.

I experienced a similar issue with trying to adapt security groups on the maps project. @bd808 at that time was able to make the change.
I wonder if some of these entries are so old, that something has gone wrong with permission tracking somewhere along the way, causing my account to not have the right permission to execute the action or something.. not sure.

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		Andrew	T204506 cloudvps: maps project trusty deprecation
		Resolved		bd808	T217992 Unable to delete "tiles.wmflabs.org" proxy entry via horizon

Event Timeline

TheDJ created this task.Mar 10 2019, 8:45 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 10 2019, 8:46 PM

TheDJ added a project: Cloud-VPS.Mar 10 2019, 8:46 PM

TheDJ mentioned this in T204506: cloudvps: maps project trusty deprecation.

TheDJ added a parent task: T204506: cloudvps: maps project trusty deprecation.

TheDJ added a project: cloud-services-team.Mar 10 2019, 9:03 PM

Aklapper edited projects, added Horizon; removed Cloud-VPS.Mar 10 2019, 9:44 PM

Desired backend is maps-tiles1.maps.eqiad.wmflabs (172.16.5.154). I was going to try and fix the proxy records, but I found that there is nothing listening on port 80 of the target host.

I tried to make a d.titles.wmflabs.org proxy just to test things and found that the horizon UI blocks creation of proxies using this type of subdomain notation as well. This likely means that the current proxies were made with older tooling. This naming scheme is also problematic because the proxy server's TLS certificate is only good for *.wmflabs.org and not *.*.wmflabs.org. The {a,b,c}.tiles.wmflabs.org proxies are all currently pointed at the same backing server, so I imagine this naming scheme is to increase parallel content fetching for HTTP/1.1 clients?

bd808 renamed this task from Unable to delete proxy entry via horizon to Unable to delete "tiles.wmflabs.org" proxy entry via horizon.Mar 11 2019, 1:17 AM

I had disabled apache on that server temporarily, because I had already set it up to be the new master renderer, but i didn't want to have two masters at the same time.

This naming scheme is also problematic because the proxy server's TLS certificate is only good for *.wmflabs.org and not *.*.wmflabs.org

Yes this is long known, but the old scheme was (and is) widely used in http contexts and was carried over from the toolserver days.

$ sudo /usr/local/sbin/wmcs-webproxy --project=maps list
domain                                           backend
================================================ ========================
0.wma.wmflabs.org.                               http://10.68.16.70:80
1.wma.wmflabs.org.                               http://10.68.16.70:80
2.wma.wmflabs.org.                               http://10.68.16.70:80
3.wma.wmflabs.org.                               http://10.68.16.70:80
4.wma.wmflabs.org.                               http://10.68.16.70:80
5.wma.wmflabs.org.                               http://10.68.16.70:80
6.wma.wmflabs.org.                               http://10.68.16.70:80
7.wma.wmflabs.org.                               http://10.68.16.70:80
a.tiles.wmflabs.org.                             http://10.68.16.103:80
b.tiles.wmflabs.org.                             http://10.68.16.103:80
c.tiles.wmflabs.org.                             http://10.68.16.103:80
label.wma.wmflabs.org.                           http://10.68.16.70:80
maps.wmflabs.org                                 http://172.16.5.154:80
tiles.wmflabs.org.                               http://10.68.16.103:80
warper.wmflabs.org                               http://172.16.0.158:80
wma.wmflabs.org.                                 http://10.68.16.70:80
$ sudo /usr/local/sbin/wmcs-webproxy --project=maps delete tiles
Traceback (most recent call last):
  File "/usr/local/sbin/wmcs-webproxy", line 152, in <module>
    main()
  File "/usr/local/sbin/wmcs-webproxy", line 148, in main
    args.func(args)
  File "/usr/local/sbin/wmcs-webproxy", line 102, in delete_proxy
    resp.status_code, resp.text))
Exception: HTTP 400 response from dynamicproxy: No such domain

Things are apparently in a "fun" state. I'm going to poke around in the backing database to see if I can figure out what's going wrong here.

$ sqlite3 /etc/dynamicproxy-api/data.db
sqlite> .mode list
sqlite> select * from project where name = 'maps';
18|maps
sqlite> select * from route where project_id = 18;
41|wma.wmflabs.org.|18
42|0.wma.wmflabs.org.|18
43|1.wma.wmflabs.org.|18
44|2.wma.wmflabs.org.|18
45|3.wma.wmflabs.org.|18
46|4.wma.wmflabs.org.|18
47|5.wma.wmflabs.org.|18
48|6.wma.wmflabs.org.|18
49|7.wma.wmflabs.org.|18
69|label.wma.wmflabs.org.|18
157|a.tiles.wmflabs.org.|18
180|tiles.wmflabs.org.|18
181|b.tiles.wmflabs.org.|18
182|c.tiles.wmflabs.org.|18
2373|warper.wmflabs.org|18
2382|maps.wmflabs.org|18

The records that end with . are really old and not supported by the current cli tool or the Horizon interface. Both of those interfaces strip the trailing dot from the name. I hacked up a copy of wmcs-webproxy to leave the trailing dot in place while deleting. This let me delete all of the old proxies.

After doing that I had to delete the tiles.wmflabs.org domain from Designate (OpenStack's DNS system) to re-create the tiles.wmflabs.org proxy. I'm pretty sure we have an open bug about this problem (not being able to add proxies to existing subdomains), but I'm not finding it right now.

$ sudo /usr/local/sbin/wmcs-webproxy --project=maps list
domain                                           backend
================================================ ========================
0.wma.wmflabs.org.                               http://10.68.16.70:80
1.wma.wmflabs.org.                               http://10.68.16.70:80
2.wma.wmflabs.org.                               http://10.68.16.70:80
3.wma.wmflabs.org.                               http://10.68.16.70:80
4.wma.wmflabs.org.                               http://10.68.16.70:80
5.wma.wmflabs.org.                               http://10.68.16.70:80
6.wma.wmflabs.org.                               http://10.68.16.70:80
7.wma.wmflabs.org.                               http://10.68.16.70:80
a.tiles.wmflabs.org                              http://172.16.5.154:80
b.tiles.wmflabs.org                              http://172.16.5.154:80
c.tiles.wmflabs.org                              http://172.16.5.154:80
label.wma.wmflabs.org.                           http://10.68.16.70:80
maps.wmflabs.org                                 http://172.16.5.154:80
tiles.wmflabs.org                                http://172.16.5.154:80
warper.wmflabs.org                               http://172.16.0.158:80
wma.wmflabs.org.                                 http://10.68.16.70:80

Mentioned in SAL (#wikimedia-cloud) [2019-03-12T00:22:15Z] <bd808> Switched *.tiles.wmflabs.org proxies to point to http://172.16.5.154:80 (T217992)

bd808 mentioned this in T218064: dynamicproxy data contains domains ending in '.' which can not be deleted via Horizon or cli tools.Mar 12 2019, 12:32 AM

bd808 closed this task as Resolved.Mar 12 2019, 5:42 AM

bd808 claimed this task.

Hello,
JOSM maintainer here.
Since a few days we're unable to access two imagery layers hosted on tiles.wmflabs.org: 'https://tiles.wmflabs.org/osm-no-labels/{zoom}/{x}/{y}.png' and 'https://tiles.wmflabs.org/bw-mapnik/{zoom}/{x}/{y}.png'.
Is it due to this change? Is the service still provided?

that might be a question best asked on the parent ticket... this one is just about deleting some config so it can be replaced with updated internal IPs.
Edit: quoted your message there

Unable to delete "tiles.wmflabs.org" proxy entry via horizonClosed, ResolvedPublicActions

Description

Related ObjectsSearch...

Event Timeline

Unable to delete "tiles.wmflabs.org" proxy entry via horizon
Closed, ResolvedPublic
Actions

Related Objects
Search...