Maniphest T190193

Put https proxy into process-osm-data
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Pnorman
	Mar 20 2018, 5:40 PM

Details

	Subject	Repo	Branch	Lines +/-
	Move process-osm-data example URLs to https	operations/puppet	production	+3 -3
	maps: add Java proxy to cleartables_sync cron	operations/puppet	production	+9 -7

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
Open		None	T87268 Copyright license and attribution issues (tracking)
Invalid		None	T149179 Attribution and copyright tasks for the iOS app (tracking)
Open	Feature	None	T157763 Replace Apple Maps with a freely licensed alternative
Invalid		None	T155601 Stabilizing Interactive Products
Open		None	T113008 Epic: Borders aren't always marked as disputed
Resolved		Pnorman	T153282 [epic] Migrate to a new vector tile structure
Declined		None	T156682 Deploying new vector tiles to production
Declined		None	T194107 Convert all production machines to new schema
Declined		None	T194108 Switch traffic to new schema machines
Declined		None	T194109 Seed vector tile store with new schema tiles
Declined		None	T194110 QA new schema in production
Declined		None	T194111 Load data for new schema in production
Declined		None	T194112 Reimage two maps servers per DC as new schema machines
Declined		None	T194113 Pull two maps servers from production serving
Resolved		Gehel	T194106 Deploy new vector tiles on maps-test2004.codfw.wmnet
Resolved		Pnorman	T190193 Put https proxy into process-osm-data
Resolved		Pnorman	T194850 Move external data sources to HTTPS

Event Timeline

Pnorman created this task.Mar 20 2018, 5:40 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 20 2018, 5:40 PM

Pnorman claimed this task.Mar 27 2018, 5:57 PM

Osmosis is java, so is of course different.

My reading of Java is that I can do this with

JAVACMD_OPTIONS="-Dhttp.proxyHost=webproxy.codfw.wmnet -Dhttp.proxyPort=8080 -Dhttps.proxyHost=webproxy.codfw.wmnet -Dhttps.proxyPort=8080"

@Gehel, you've worked with Java a lot, does that sound right?

@Gehel is it better to put the environment variables into the script, or to whatever is calling the script?

If it's the script, is there a string that puppet automagically turns into the right proxy for the datacentre?

Reassigning. I'll still do the changes to the script once Europe is awake and I know the best way.

Using env variables external to the script seems best. At least, we don't want the values hardcoded in the script.

In T190193#4101515, @Pnorman wrote:

Osmosis is java, so is of course different.

My reading of Java is that I can do this with

JAVACMD_OPTIONS="-Dhttp.proxyHost=webproxy.codfw.wmnet -Dhttp.proxyPort=8080 -Dhttps.proxyHost=webproxy.codfw.wmnet -Dhttps.proxyPort=8080"

@Gehel, you've worked with Java a lot, does that sound right?

Yep, that looks good.

The cron job will need to set JAVACMD_OPTIONS for database-update and planet-update, and http_proxy + https_proxy for static-update.

The cron job is set, but I'm leaving this open to document the loading process proxy usage

Change 424247 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] maps: add Java proxy to cleartables_sync cron

https://gerrit.wikimedia.org/r/424247

gerritbot added a project: Patch-For-Review.Apr 5 2018, 9:07 AM

Change 424247 merged by Gehel:
[operations/puppet@production] maps: add Java proxy to cleartables_sync cron

https://gerrit.wikimedia.org/r/424247

Made https://wikitech.wikimedia.org/wiki/Maps/Loading to document how to load on our servers and added https_proxy to it

Pnorman added a parent task: T194106: Deploy new vector tiles on maps-test2004.codfw.wmnet.May 16 2018, 8:08 PM

The documentation now covers enough to have all the commands which need to download data that are run on import.

Some related work is T194850: Move external data sources to HTTPS where I need to change some HTTP URLs to HTTPS

Change 433776 had a related patch set uploaded (by Pnorman; owner: Pnorman):
[operations/puppet@production] Move process-osm-data example URLs to https

https://gerrit.wikimedia.org/r/433776

Pnorman closed subtask T194850: Move external data sources to HTTPS as Resolved.May 18 2018, 6:49 PM

Change 433776 merged by Gehel:
[operations/puppet@production] Move process-osm-data example URLs to https

https://gerrit.wikimedia.org/r/433776

Pnorman moved this task from In progress to Stalled/Waiting on the Maps-Sprint board.May 24 2018, 9:35 AM

Pnorman closed this task as Resolved.May 25 2018, 10:28 PM

• Vvjjkkii reopened subtask T194850: Move external data sources to HTTPS as Open.Jul 1 2018, 1:09 AM

CommunityTechBot closed subtask T194850: Move external data sources to HTTPS as Resolved.Jul 2 2018, 4:06 PM