Page MenuHomePhabricator

dschwen (dschwen)
User

Projects

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Thursday

  • Clear sailing ahead.

User Details

User Since
Dec 8 2014, 9:03 PM (484 w, 12 h)
Availability
Available
LDAP User
dschwen
MediaWiki User
Dschwen [ Global Accounts ]

Recent Activity

Jan 22 2024

dschwen added a comment to T319953: Migrate panoviewer from Toolforge GridEngine to Toolforge Kubernetes.

Tim, I just added you as a maintainer!

Jan 22 2024, 3:17 PM · Grid-Engine-to-K8s-Migration

Dec 31 2023

dschwen added a comment to T351523: Archive Gerrit repository "labs/maps" (20121106).

My maps related stuff is on GitHub. I never knew about this repo.

Dec 31 2023, 3:00 PM · Diffusion-Repository-Administrators, Wikimedia-GitHub, Projects-Cleanup

Oct 31 2023

dschwen added a comment to T343796: ZoomViewer produces a 503 error.

This is weird:

Oct 31 2023, 2:47 PM · Tools, Commons

Sep 27 2023

dschwen added a comment to T138933: Explore moving the Panoviewer gadget/Toolforge tool into production.

@tstarling if you were to integrate VIPS based large image tiling into the framework you could in one fell swoop support both tiled high res panoramics and the zoomviewer!

Sep 27 2023, 2:13 PM · Thumbor, MediaViewer, Tools, Platform Team Legacy (Watching / External), Services (watching), Multimedia

Aug 28 2023

dschwen added a comment to T343796: ZoomViewer produces a 503 error.

I bet it's not a missing file, but rather a broken tif. Those can occur when VIPS crashes/is killed

Aug 28 2023, 3:28 AM · Tools, Commons

Aug 27 2023

dschwen added a comment to T343796: ZoomViewer produces a 503 error.

Ah, ok, that's good to know. I did pull the latest iipserver repo version and rebuilt the server on Wednesday, but if it is indeed due to missing files than this is likely related to the dying VIPS process which gets killed by gridengine(?) for running out of memory. I suppose I could ask for a bigger allocation. What do you think @bd808 ?

Aug 27 2023, 10:09 PM · Tools, Commons
dschwen added a comment to T343796: ZoomViewer produces a 503 error.

The webservice was running, but the fastcgi service was still broken. Stopping and starting the service got it going again. No idea why it fails. Maybe it is getting killed (running out of memory?)

Aug 27 2023, 7:57 PM · Tools, Commons
dschwen added a comment to T343796: ZoomViewer produces a 503 error.

fastcgi is still broken

Aug 27 2023, 7:46 PM · Tools, Commons

Aug 24 2023

dschwen added a comment to T343796: ZoomViewer produces a 503 error.

Yeah, the VIPS process fails for that image:

Aug 24 2023, 10:41 PM · Tools, Commons
dschwen added a comment to T343796: ZoomViewer produces a 503 error.

Added you as a co-maintainer and added a license (that was an oversight). There is minimal doco in the README.md. I shall expand on that. But TBH I'm just winging it, as I'm neither an expert in lighthttpd nor gridengine.

Aug 24 2023, 10:32 PM · Tools, Commons

Aug 23 2023

dschwen added a comment to T343796: ZoomViewer produces a 503 error.

Nope, I see new multi res tiffs popping up in the cache dir right now

Aug 23 2023, 11:33 PM · Tools, Commons
dschwen added a comment to T343796: ZoomViewer produces a 503 error.

Viewer works now

Aug 23 2023, 11:28 PM · Tools, Commons
dschwen added a comment to T343796: ZoomViewer produces a 503 error.

Hm, I'm very much open to having a co-admin on this project. My day job and family do not leave.e as much time as I'd like. I'll take a quick look now (tricky behind the work firewall).

Aug 23 2023, 10:35 PM · Tools, Commons

Feb 6 2023

dschwen added a comment to T323159: Shut down osmdb.eqiad.wmnet (clouddb100[3-4])?.

P.S.: the import of the wp_coords_redX tables is done.

Feb 6 2023, 2:53 AM · cloud-services-team (FY2022/2023-Q3), User-Sandra_Fauconnier_WMSE, Cloud-VPS (Debian Stretch Deprecation)
dschwen added a comment to T323159: Shut down osmdb.eqiad.wmnet (clouddb100[3-4])?.

@Kolossos how old are these POI tables? I notice they use a curious design with a column for each language and most columns being empty for most POIs. I have a pretty robust extraction and reduction level rebuilding going on for the WMA. I think it would be a good idea to just use that dataset instead.

Feb 6 2023, 2:52 AM · cloud-services-team (FY2022/2023-Q3), User-Sandra_Fauconnier_WMSE, Cloud-VPS (Debian Stretch Deprecation)
dschwen added a comment to T323159: Shut down osmdb.eqiad.wmnet (clouddb100[3-4])?.

Thanks. I'm importing the wp_coords_redX tables right now. For the record, the WikiMiniAtlas has already been connecting to this database. I hope that still works after these changes. I'll check this later and will change it to use the new account.

Feb 6 2023, 2:49 AM · cloud-services-team (FY2022/2023-Q3), User-Sandra_Fauconnier_WMSE, Cloud-VPS (Debian Stretch Deprecation)

Jan 15 2023

dschwen added a comment to T323159: Shut down osmdb.eqiad.wmnet (clouddb100[3-4])?.

Yeah, I was waiting for help with the account creation. I'm out skiing this weekend but I'll check where we are with that this week. I can copy the tables. Ideally you just dump them and I import them from the dump.

Jan 15 2023, 7:19 PM · cloud-services-team (FY2022/2023-Q3), User-Sandra_Fauconnier_WMSE, Cloud-VPS (Debian Stretch Deprecation)

Dec 12 2022

dschwen added a comment to T315466: FastCCI-gadget on Wikimedia-Commons returns no images and hangs with "Connecting..." message / FastCCI server status - the backend server deployed on Wikimedia Cloud VPS returns a "504 Gateway Time-out" message.

Yeah, sorry, I didn't see the phabricator task, but somebody emailed me a few months ago. I rerwrote parts of the database generation and the server to allow for larger db sizes (commons is constantky growing). I hope it has been more resilient in the past few months.

Dec 12 2022, 1:52 AM · VPS-Projects, Commons

Dec 8 2022

dschwen added a comment to T323159: Shut down osmdb.eqiad.wmnet (clouddb100[3-4])?.

This one looks like it dumps all the names of roads/streets in a given region (here it is east Germany, around the city of Dresden)

select distinct name from planet_osm_line where way && ST_Transform(ST_SetSRID(ST_MakeBox2D(ST_Point(13.54,50.95),ST_Point(13.94,51.15)),4326),900913)  AND "highway" is not NULL AND name is not NULL order by name LIMIT 10000;
Dec 8 2022, 1:55 PM · cloud-services-team (FY2022/2023-Q3), User-Sandra_Fauconnier_WMSE, Cloud-VPS (Debian Stretch Deprecation)

Dec 3 2022

dschwen added a comment to T323159: Shut down osmdb.eqiad.wmnet (clouddb100[3-4])?.

possible to switch to kartographers data like

Dec 3 2022, 3:28 PM · cloud-services-team (FY2022/2023-Q3), User-Sandra_Fauconnier_WMSE, Cloud-VPS (Debian Stretch Deprecation)
dschwen added a comment to T323159: Shut down osmdb.eqiad.wmnet (clouddb100[3-4])?.

@Kolossos what's the status of the WIWOSM project? I'm using it from WMA. Is it maintained? It would be another candidate to use the new OSM DB.

Dec 3 2022, 1:22 AM · cloud-services-team (FY2022/2023-Q3), User-Sandra_Fauconnier_WMSE, Cloud-VPS (Debian Stretch Deprecation)

Dec 2 2022

dschwen added a comment to T323159: Shut down osmdb.eqiad.wmnet (clouddb100[3-4])?.

Maybe we could make the names more explicit, like osm_admin, osm_ro_wma, etc.? I'll draft the actual SQL commands next week.

Dec 2 2022, 6:35 PM · cloud-services-team (FY2022/2023-Q3), User-Sandra_Fauconnier_WMSE, Cloud-VPS (Debian Stretch Deprecation)
dschwen added a comment to T323159: Shut down osmdb.eqiad.wmnet (clouddb100[3-4])?.

@fnegri my suggestion would be this:

Dec 2 2022, 4:55 PM · cloud-services-team (FY2022/2023-Q3), User-Sandra_Fauconnier_WMSE, Cloud-VPS (Debian Stretch Deprecation)

Nov 28 2022

dschwen added a comment to T323159: Shut down osmdb.eqiad.wmnet (clouddb100[3-4])?.

is it your understanding that the new DB (that you built) contains the same data as this database?

Nov 28 2022, 7:25 PM · cloud-services-team (FY2022/2023-Q3), User-Sandra_Fauconnier_WMSE, Cloud-VPS (Debian Stretch Deprecation)

Nov 10 2022

dschwen added a comment to T322755: shut down cloud-vps 'maps' project.

Yeah, no worries. I suppose I should create some documentation of the bits I'm maintaining. Where would you look for this type of project level docs.?

Nov 10 2022, 5:55 AM · User-Sandra_Fauconnier_WMSE, cloud-services-team (Kanban), Cloud-VPS (Debian Stretch Deprecation)
dschwen added a comment to T322755: shut down cloud-vps 'maps' project.

So upon closer inspection it seems to me that at most the maps-tiles1 instance can be deleted (unless noone else speaks up about it).

Nov 10 2022, 3:47 AM · User-Sandra_Fauconnier_WMSE, cloud-services-team (Kanban), Cloud-VPS (Debian Stretch Deprecation)
dschwen added a comment to T306073: Cloud VPS "fastcci" project Stretch deprecation.

Ok, I deleted that instance and I'm setting up a new one. Why are the other Bullseye instances also marked as "deprecated"?

Nov 10 2022, 2:15 AM · Cloud-VPS (Debian Stretch Deprecation)
dschwen added a comment to T306073: Cloud VPS "fastcci" project Stretch deprecation.

Uh oh. I'll double check if that instance is still needed.

Nov 10 2022, 1:58 AM · Cloud-VPS (Debian Stretch Deprecation)

Nov 9 2022

dschwen added a comment to T322755: shut down cloud-vps 'maps' project.

Wait what?!

Nov 9 2022, 5:24 PM · User-Sandra_Fauconnier_WMSE, cloud-services-team (Kanban), Cloud-VPS (Debian Stretch Deprecation)

Oct 6 2022

dschwen added a comment to T300160: Request increased quota for maps Cloud VPS project.

Hello @rook and @Andrew , I'm in good shape. I have an up to date OSM mirror (see https://wma.wmcloud.org/osmdb.txt) and my tile rendering is working great (https://wma.wmcloud.org/status.php). I could free 80GB by removing my older tile cache volume (that ended up being too small). Either way I should unmount it and detach it from the machine. Cheers and many thanks for your support!

Oct 6 2022, 1:49 PM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)

Jun 30 2022

dschwen added a comment to T285668: Labs OSMdb is outdated/outofsync.

Yeah, I set up a separate OSMDB replica. By the way I could use some help with setting up access to it in a sensible way. It should be discussed if accounts are handed out to individuals, or if there'll be a free for all type of access. I'd prefer the former to have a bit more control over the resource.

Jun 30 2022, 1:11 AM · Patch-For-Review, Data-Services, cloud-services-team (Kanban), Maps

Mar 9 2022

dschwen added a comment to T300160: Request increased quota for maps Cloud VPS project.

Fine so far. I'm at my second import with optimized database settings and the right data layout (using hstore). It'll be. A few more days. You can monitor the progress here https://wma.wmcloud.org/osmdb.txt

Mar 9 2022, 3:31 PM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)

Mar 1 2022

dschwen added a comment to T300160: Request increased quota for maps Cloud VPS project.

No problem, @Andrew! Enjoy your vacation! This is a slow burn project anyways. I'm currently reimporting the planet all over again, this time with the proper hstore usage. Of course I had to ignore the advice online about doing test imports with small subsets (like a US state or city)... :-/

Mar 1 2022, 12:20 AM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)

Feb 28 2022

dschwen added a comment to T300160: Request increased quota for maps Cloud VPS project.

@Andrew The Trove instance is stuck on "Building" and I cannot seem to log into it. I need to set a bunch of postgres.conf options and I need to make sure that the postgis and hstore extensions are installed (is Trove managing the upgrades of those as well, otherwise the Trove advantage of having the server managed seems not applicable for this scenario).

Feb 28 2022, 10:56 PM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)
dschwen added a comment to T300160: Request increased quota for maps Cloud VPS project.

Uaaahhrggh, I forgot to use hstore during import. Now I have an up to the minute updated DB that does not support my queries! Ok, I suppose this was a learning experience. I should try again on trove.

Feb 28 2022, 3:27 AM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)

Feb 24 2022

dschwen added a comment to T285668: Labs OSMdb is outdated/outofsync.

Update: The initial import (with a planet data file from 1/31/22) has finished and I'm currently running a replication process to catch up.

Feb 24 2022, 2:38 AM · Patch-For-Review, Data-Services, cloud-services-team (Kanban), Maps

Feb 22 2022

dschwen added a comment to T300160: Request increased quota for maps Cloud VPS project.

Yes! I resized the FS to 4TB and started a replication update. https://wma.wmcloud.org/osmdb.txt

Feb 22 2022, 11:46 PM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)
dschwen added a comment to T300694: Move cloud-vps Maps nfs share to vm-hosted NFS.

actually, maybe no data needs to be synced. I'll just remount the device.

Feb 22 2022, 11:14 PM · cloud-services-team (Kanban)
dschwen added a comment to T300694: Move cloud-vps Maps nfs share to vm-hosted NFS.

Do I need to reboot my running instance to get this change? I thin the maps-wma2 instance is still connected to the old NFS

Feb 22 2022, 11:12 PM · cloud-services-team (Kanban)
dschwen added a comment to T300160: Request increased quota for maps Cloud VPS project.

Hm, it seems I was simply confused by the display of remaining quota. In the project overview I see there is no volume quota left.

Feb 22 2022, 8:59 PM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)

Feb 21 2022

dschwen added a comment to T300160: Request increased quota for maps Cloud VPS project.

Thanks, any idea why I cannot resize the volume to 4TB?

Feb 21 2022, 10:54 PM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)
dschwen added a comment to T300160: Request increased quota for maps Cloud VPS project.

Still detaching. I'm stuck.

Feb 21 2022, 7:54 PM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)

Feb 19 2022

dschwen added a comment to T300160: Request increased quota for maps Cloud VPS project.

Thanks. The volume my current DB import is on is has now been in the "detaching" state for a while as well. Similar issue? I tried to detach it to resize it. But while it is in that state I cannot do anything but "update metadata".

Feb 19 2022, 5:26 AM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)
dschwen added a comment to T300160: Request increased quota for maps Cloud VPS project.

The maps-postgres db instance is still shown as building. Isn't that a bit curious. Also, I don't see a way to resize a volume. Is that possible via Horizon?

Feb 19 2022, 2:28 AM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)

Feb 17 2022

dschwen updated the task description for T300160: Request increased quota for maps Cloud VPS project.
Feb 17 2022, 7:13 PM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)
dschwen added a comment to T300160: Request increased quota for maps Cloud VPS project.

Yeah, for sure. II also wonder if it would be possible to just transplant config and data director from the VM PG to the Trove container?
I'm also a little suspicious about how tightly the data just happened to fit. I don't see any errors, but it would be quite a coincidence if the import worked with just a few hundred MB of free space left :-/

Feb 17 2022, 6:09 PM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)
dschwen added a comment to T300160: Request increased quota for maps Cloud VPS project.
dschwen@maps-osmdb:~/osmdb$ time osm2pgsql -c -s planet-latest2.osm.pbf -d gis
2022-01-31 20:47:21  osm2pgsql version 1.6.0 (1.6.0)
2022-01-31 20:47:21  Database version: 13.5 (Debian 13.5-0+deb11u1)
2022-01-31 20:47:21  PostGIS version: 3.1
2022-01-31 20:47:21  Setting up table 'planet_osm_point'
2022-01-31 20:47:22  Setting up table 'planet_osm_line'
2022-01-31 20:47:22  Setting up table 'planet_osm_polygon'
2022-01-31 20:47:22  Setting up table 'planet_osm_roads'
2022-02-11 17:52:17  Reading input files done in 939895s (261h 4m 55s).
2022-02-11 17:52:17    Processed 7476676170 nodes in 31789s (8h 49m 49s) - 235k/s
2022-02-11 17:52:17    Processed 833672717 ways in 656592s (182h 23m 12s) - 1k/s
2022-02-11 17:52:17    Processed 9622337 relations in 251514s (69h 51m 54s) - 38/s
2022-02-11 17:52:18  Clustering table 'planet_osm_roads' by geometry...
2022-02-11 17:52:18  Clustering table 'planet_osm_line' by geometry...
2022-02-11 17:52:18  Clustering table 'planet_osm_point' by geometry...
2022-02-11 17:52:18  Clustering table 'planet_osm_polygon' by geometry...
2022-02-11 19:49:23  Creating geometry index on table 'planet_osm_roads'...
2022-02-11 20:05:13  Creating geometry index on table 'planet_osm_point'...
2022-02-11 20:06:37  Creating osm_id index on table 'planet_osm_roads'...
2022-02-11 20:11:56  Analyzing table 'planet_osm_roads'...
2022-02-11 20:13:46  Done postprocessing on table 'planet_osm_nodes' in 0s
2022-02-11 20:13:46  Building index on table 'planet_osm_ways'
2022-02-11 22:57:25  Creating osm_id index on table 'planet_osm_point'...
2022-02-12 00:04:34  Analyzing table 'planet_osm_point'...
2022-02-12 00:05:07  Building index on table 'planet_osm_rels'
2022-02-13 01:50:51  Creating geometry index on table 'planet_osm_line'...
2022-02-13 07:55:49  Creating osm_id index on table 'planet_osm_line'...
2022-02-13 09:16:08  Analyzing table 'planet_osm_line'...
2022-02-13 17:54:52  Creating geometry index on table 'planet_osm_polygon'...
2022-02-14 13:22:19  Creating osm_id index on table 'planet_osm_polygon'...
2022-02-14 15:23:16  Analyzing table 'planet_osm_polygon'...
2022-02-17 01:09:21  Done postprocessing on table 'planet_osm_ways' in 449734s (124h 55m 34s)
2022-02-17 01:09:21  Done postprocessing on table 'planet_osm_rels' in 10962s (3h 2m 42s)
2022-02-17 01:09:21  All postprocessing on table 'planet_osm_point' done in 22368s (6h 12m 48s).
2022-02-17 01:09:21  All postprocessing on table 'planet_osm_line' done in 141865s (39h 24m 25s).
2022-02-17 01:09:21  All postprocessing on table 'planet_osm_polygon' done in 250293s (69h 31m 33s).
2022-02-17 01:09:21  All postprocessing on table 'planet_osm_roads' done in 8487s (2h 21m 27s).
2022-02-17 01:09:21  osm2pgsql took 1398120s (388h 22m 0s) overall.
Feb 17 2022, 5:38 AM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)

Feb 12 2022

dschwen added a comment to T300160: Request increased quota for maps Cloud VPS project.

Just to keep you updated. The import took about 11days and finished yesterday. It is currently building indices and it shows 2.2TB out of 2.9TB used on the DB volume. I'll let this finish to get a better idea as to how big the final (at least with the current state of OSM) DB will be. It's possible that 3TB is a bit tight.

Feb 12 2022, 7:39 PM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)

Feb 9 2022

dschwen added a comment to T300160: Request increased quota for maps Cloud VPS project.

Ok, thanks. The import on the VM did speed up considerably and is in its final stages. I'll let it run and check if it went well, to verify my procedure. If it went fine I'll replicate the steps on the Trove DB.

Feb 9 2022, 3:29 AM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)

Feb 8 2022

dschwen added a comment to T300694: Move cloud-vps Maps nfs share to vm-hosted NFS.

Form my end you can just cut over. Would it be possible to keep the old die around for a few days in case something hit missed?

Feb 8 2022, 1:25 AM · cloud-services-team (Kanban)

Feb 4 2022

dschwen added a comment to T300160: Request increased quota for maps Cloud VPS project.

weird, it shows mysql as the datastore ...

Feb 4 2022, 12:13 AM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)

Feb 3 2022

dschwen added a comment to T300160: Request increased quota for maps Cloud VPS project.

Yeah, I think 3TB would be a good start. I have trouble finding hard and fast numbers for database size. My current import is at 1.3TB with nodes imported, but only a fraction of all ways, and no relations imported.

Feb 3 2022, 9:50 PM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)
dschwen added a comment to T300160: Request increased quota for maps Cloud VPS project.

Hm, ok. I'm in the middle of importing data, so I'd like to see what the performance of the Trove DB is compared to the server on the VM. Would it be possible to get enough quota to start an import? Also how hard/easy is it to resize volumes or Trove DBs in case the DB size grows beyond the current allocation future?

Feb 3 2022, 5:34 PM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)

Feb 2 2022

dschwen added a comment to T300694: Move cloud-vps Maps nfs share to vm-hosted NFS.

Option 3 would be doing a second, final rsync without stopping services. That should be fine for my stuff at least.

Feb 2 2022, 5:25 PM · cloud-services-team (Kanban)
dschwen added a comment to T300694: Move cloud-vps Maps nfs share to vm-hosted NFS.

Andrew, everything i have on NFS is not transient and must be copied. My transient stuff is now on a cinder volume.

Feb 2 2022, 1:52 PM · cloud-services-team (Kanban)

Feb 1 2022

dschwen added a comment to T300160: Request increased quota for maps Cloud VPS project.

Uhm, yes, it is a Postgres / PostGIS database. I thought the whole point here was not to have a DB managed by admins, but to put it in "userspace" (per T285668).

Feb 1 2022, 11:59 PM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)
dschwen added a comment to T285668: Labs OSMdb is outdated/outofsync.

You can peek at the import progress here: https://wma.wmflabs.org/osmdb.txt (updated once a minute)

Feb 1 2022, 3:05 PM · Patch-For-Review, Data-Services, cloud-services-team (Kanban), Maps
dschwen added a comment to T285668: Labs OSMdb is outdated/outofsync.

I see that @dschwen has started a server in maps project, with the potential idea of doing this ?

Feb 1 2022, 3:00 PM · Patch-For-Review, Data-Services, cloud-services-team (Kanban), Maps
dschwen added a comment to T187601: Examine replacing tiles.wmflabs.org with production tile server.

I'm currently importing a new planet file into a database within the maps project. While all nodes completed import last night the ways import goes a lot slower and I expect it to last a few days (up to two weeks).

Feb 1 2022, 2:16 PM · Maps, VPS-Projects

Jan 31 2022

dschwen added a comment to T300160: Request increased quota for maps Cloud VPS project.

I have an up to date planet-latest.osm.pbf that I'm incrementally keeping current with osmupdate. I have the database set up (just have to move it to the cinder volume), and I think I got the replication using osmosis and osm2pgsql figured out. I'm itching to get an import going.

Jan 31 2022, 3:29 AM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)

Jan 26 2022

dschwen added a comment to T300160: Request increased quota for maps Cloud VPS project.

I'm documenting the setup here: https://github.com/Commonists/maps-osmdb

Jan 26 2022, 8:03 PM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)
dschwen created T300160: Request increased quota for maps Cloud VPS project.
Jan 26 2022, 4:16 PM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)
dschwen added a comment to T285668: Labs OSMdb is outdated/outofsync.

How far out of sync is that DB anyways. I just queried

Jan 26 2022, 5:19 AM · Patch-For-Review, Data-Services, cloud-services-team (Kanban), Maps

Jan 25 2022

dschwen added a comment to T285668: Labs OSMdb is outdated/outofsync.

@Multichill the underlying issue here is the OSM db replication not working. Tiles are just a symptom.

Jan 25 2022, 6:40 PM · Patch-For-Review, Data-Services, cloud-services-team (Kanban), Maps

Jan 22 2022

dschwen added a comment to T299585: Request increased quota for maps Cloud VPS project.

I have deleted the old instance. The quota can be rolled back. Thanks!

Jan 22 2022, 10:36 PM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)
dschwen added a comment to T187601: Examine replacing tiles.wmflabs.org with production tile server.

By the way, my rant above is outdated. I rewrote the data extraction for the Wikiminiatlas a year ago, and I just rebuilt the wma VM with Debian 11. Everything should be fully functional.

Jan 22 2022, 7:34 PM · Maps, VPS-Projects
dschwen added a comment to T187601: Examine replacing tiles.wmflabs.org with production tile server.

tiles.wmflabs.org has never generated tiles for the Wikiminiatlas!

Jan 22 2022, 7:32 PM · Maps, VPS-Projects
dschwen added a comment to T299775: maps project wildcard domains.

Thanks that's fine. I'll start migrating everything to wmcloud.org and let you know when the labs domains can be deleted.

Jan 22 2022, 7:30 PM · Cloud-VPS

Jan 21 2022

dschwen added a comment to T299775: maps project wildcard domains.

Ok, I just saw that there already are

Jan 21 2022, 4:03 PM · Cloud-VPS
dschwen created T299775: maps project wildcard domains.
Jan 21 2022, 4:01 PM · Cloud-VPS
dschwen added a comment to T299585: Request increased quota for maps Cloud VPS project.

Would it also be possible to add a wildcard *.wma.wmflabs.org. record (and *.wma.wmcloud.org.) - all pointing to the new VM? Then I could experiment with rotating tile base URLs to permit faster bursts of requests when a map is opened.

Jan 21 2022, 3:37 PM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)
dschwen added a comment to T299585: Request increased quota for maps Cloud VPS project.

@Majavah I think I'm ready to have the proxy mappings changed, so that wma.wmflabs.org points to the new server. If you could also add wma.wmcloud.org as a mapping that'd be great, then I can also start changing the gadget to point to the now blessed domain.

Jan 21 2022, 3:34 PM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)

Jan 20 2022

dschwen added a comment to T299585: Request increased quota for maps Cloud VPS project.

Will do. The new instance is partially running and I'm still finishing setup of some components. Then I have to thoroughly test and point the JavaScript on the projects to the new url.

Jan 20 2022, 2:44 PM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)

Jan 19 2022

dschwen added a comment to T299585: Request increased quota for maps Cloud VPS project.

The quota increase should be temporary and could be reversed once the newly rebuilt maps-wma VM is operational and the old VM is deleted.

Jan 19 2022, 9:08 PM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)
dschwen created T299585: Request increased quota for maps Cloud VPS project.
Jan 19 2022, 9:03 PM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)

Jun 15 2021

dschwen added a comment to T285018: zoomviewer taking up a lot of NFS space -- please clean up.

I've already set up a daily cronjob that prunes out all cache file older than 90 days. That's based on mtime. I suppose atime would be the better measure, but /mnt/nfs/labstore-secondary-tools-project is mounted with noatime. Do you want me to reduce the expiration age to 30 days? Alternatively I could come up with a dabade approach to implement atime myself and purge images that are not being accessed for a while.

Jun 15 2021, 10:07 PM · Tools

Apr 8 2021

dschwen added a comment to T263141: ZoomViewer Error - No response from server iipsrv.fcgi.

Thanks AFBorchert, this is indeed fixed. A path change in the lighttpd config was all that was required.

Apr 8 2021, 4:51 PM · Tools, Commons

Jan 21 2021

dschwen closed T272556: Unexpected rows in external links table as Invalid.
Jan 21 2021, 2:34 AM
dschwen added a comment to T272556: Unexpected rows in external links table.

Ugh... the OSM location map template.... nevermind then :-(

Jan 21 2021, 2:34 AM
dschwen created T272556: Unexpected rows in external links table.
Jan 21 2021, 2:32 AM

Nov 6 2020

dschwen added a comment to T248188: Zoomviewer has ~450,000 files in NFS home directory .

Thanks! Side question do I need to change commonswiki.web.db.svc.eqiad to commonswiki.analytics.db.svc.eqiad.wmflabs (I'm seeing some DB errors in the logs)

Nov 6 2020, 12:40 AM · Tools
dschwen added a comment to T248188: Zoomviewer has ~450,000 files in NFS home directory .

But - also weird - this:

Nov 6 2020, 12:28 AM · Tools
dschwen added a comment to T248188: Zoomviewer has ~450,000 files in NFS home directory .
tools.zoomviewer@tools-sgebastion-07:~$ crontab -l
....
23   2  *   *   *      jsub ${HOME}/prune.sh >/dev/null 2>&1
Nov 6 2020, 12:27 AM · Tools
dschwen added a comment to T248188: Zoomviewer has ~450,000 files in NFS home directory .

Yeah, I guess we could go with that. What's a reasonable time? Also the retention would have to be based on atime rather than ctime/mtime. Do we have that available on that FS?

Nov 6 2020, 12:23 AM · Tools

Nov 5 2020

dschwen added a comment to T248188: Zoomviewer has ~450,000 files in NFS home directory .

Yes, this is expected. The tool generates multiresolution pyramids for very large (gigapixel) images on commons. Generating the MRPs takes a while (job queue) so if we delete them the user experience will worsen.

Nov 5 2020, 11:51 PM · Tools

Aug 27 2020

dschwen added a comment to T259385: Work with community to fix faulty gadgets triggering JS errors.

I don't know why MediaWiki:Wikiminiatlas.js is on that list. The JS Error column shows "Unknown" and I cannot replicate any error on the linked page.

Aug 27 2020, 4:52 PM · Local-Wiki-Template-And-Gadget-Issues

Aug 26 2020

dschwen added a comment to T261354: Zoomviewer uses persitent db connections.

I've made the change

Aug 26 2020, 11:02 PM · Data-Services, Tools

Aug 6 2020

dschwen added a comment to T223051: EXIF location data of image file not imported to MediaWiki File information (Upload with UploadWizard).

Ugh, right, but it only downloads promising candidates, i.e. files which have some GPS stuff in the metadata in the database https://github.com/Commonists/gpsexifbot/blob/master/gps_exif_bot2.py#L59-L63

Aug 6 2020, 10:02 PM · Multimedia, Commons, MediaWiki-File-management, UploadWizard
dschwen added a comment to T223051: EXIF location data of image file not imported to MediaWiki File information (Upload with UploadWizard).

No, it should actually download the file and extract metadata with pyexiv2: https://github.com/Commonists/gpsexifbot/blob/master/gps_exif_bot2.py#L174-L191

Aug 6 2020, 9:56 PM · Multimedia, Commons, MediaWiki-File-management, UploadWizard

May 21 2020

dschwen added a comment to T248188: Zoomviewer has ~450,000 files in NFS home directory .

2 months later and no response on this ticket. :(

May 21 2020, 1:04 AM · Tools

Feb 13 2020

dschwen added a comment to T208404: Check whether fastcci project requires NFS or not.

I'm using /data/project and the fact that I have shared home directories (and I'm accessing replica dbs). That's it (no dumps, no scratch).

Feb 13 2020, 8:02 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS
dschwen added a comment to T208404: Check whether fastcci project requires NFS or not.

Will do once I get to a computer later today.

Feb 13 2020, 4:06 PM · Patch-For-Review, cloud-services-team (Kanban), Cloud-VPS

Dec 25 2019

dschwen added a comment to T241427: WikiMiniAtlas (wma) is down at wma.wmflabs.org.

Ah, stop, it seems to have booted through and is back online.

Dec 25 2019, 12:11 AM · VPS-Projects, Maps
dschwen added a comment to T241427: WikiMiniAtlas (wma) is down at wma.wmflabs.org.

Hm, I soft rebooted and I'm getting that dreaded A stop job is running for ... message :-(
I cannot ssh in.

Dec 25 2019, 12:10 AM · VPS-Projects, Maps
dschwen added a comment to T241427: WikiMiniAtlas (wma) is down at wma.wmflabs.org.

Ugh, Merry Christmas :-)

Dec 25 2019, 12:03 AM · VPS-Projects, Maps

Dec 19 2019

dschwen added a comment to T138933: Explore moving the Panoviewer gadget/Toolforge tool into production.

Just as a quick comment, I think multires tiles are absolutely necessary for the stitched panos that we have (>100 Megapixel). I would suggest not even thinking about a solution that does not permit this.

Dec 19 2019, 4:01 PM · Thumbor, MediaViewer, Tools, Platform Team Legacy (Watching / External), Services (watching), Multimedia

Nov 29 2019

dschwen added a comment to T105789: A new panoramic viewer for commons.

Yes, sorry, that's the one. I could have sworn with over 40 comments the link would be on one of them :-)

Nov 29 2019, 5:30 PM · Tools, Community-Wishlist-Survey-2016, Multimedia, Commons, Wikimania-Hackathon-2015
dschwen added a comment to T105789: A new panoramic viewer for commons.

The panorama viewer described in this ticket exists on tools cloud. I guess we can close this ticket and open new tickets for bugs and improvements.

Nov 29 2019, 2:47 PM · Tools, Community-Wishlist-Survey-2016, Multimedia, Commons, Wikimania-Hackathon-2015

Apr 1 2019

dschwen added a comment to T204506: cloudvps: maps project trusty deprecation.

@Andrew , Yes, I am planning to update my code to use non-dot subdomains. But I'm waiting until the dash issue is resolved.

Apr 1 2019, 3:44 PM · cloud-services-team (Kanban), User-TheDJ, Cloud-VPS (Ubuntu Trusty Deprecation), Maps
dschwen added a comment to T204506: cloudvps: maps project trusty deprecation.

And the dash? (anyhow, just delete all wma proxies containing a dot at your convenience, please)

Apr 1 2019, 3:23 PM · cloud-services-team (Kanban), User-TheDJ, Cloud-VPS (Ubuntu Trusty Deprecation), Maps