Page MenuHomePhabricator

Put https proxy into process-osm-data
Closed, ResolvedPublic

Event Timeline

Pnorman created this task.Mar 20 2018, 5:40 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 20 2018, 5:40 PM
Pnorman claimed this task.Mar 27 2018, 5:57 PM

Osmosis is java, so is of course different.

My reading of Java is that I can do this with

JAVACMD_OPTIONS="-Dhttp.proxyHost=webproxy.codfw.wmnet -Dhttp.proxyPort=8080 -Dhttps.proxyHost=webproxy.codfw.wmnet -Dhttps.proxyPort=8080"

@Gehel, you've worked with Java a lot, does that sound right?

@Gehel is it better to put the environment variables into the script, or to whatever is calling the script?

If it's the script, is there a string that puppet automagically turns into the right proxy for the datacentre?

Pnorman reassigned this task from Pnorman to Gehel.Apr 5 2018, 12:20 AM

Reassigning. I'll still do the changes to the script once Europe is awake and I know the best way.

Gehel reassigned this task from Gehel to Pnorman.Apr 5 2018, 8:47 AM

Using env variables external to the script seems best. At least, we don't want the values hardcoded in the script.

Osmosis is java, so is of course different.
My reading of Java is that I can do this with
JAVACMD_OPTIONS="-Dhttp.proxyHost=webproxy.codfw.wmnet -Dhttp.proxyPort=8080 -Dhttps.proxyHost=webproxy.codfw.wmnet -Dhttps.proxyPort=8080"
@Gehel, you've worked with Java a lot, does that sound right?

Yep, that looks good.

The cron job will need to set JAVACMD_OPTIONS for database-update and planet-update, and http_proxy + https_proxy for static-update.

Pnorman moved this task from Backlog to In progress on the Maps-Sprint board.Apr 5 2018, 8:58 AM

The cron job is set, but I'm leaving this open to document the loading process proxy usage

Change 424247 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] maps: add Java proxy to cleartables_sync cron

https://gerrit.wikimedia.org/r/424247

Change 424247 merged by Gehel:
[operations/puppet@production] maps: add Java proxy to cleartables_sync cron

https://gerrit.wikimedia.org/r/424247

Made https://wikitech.wikimedia.org/wiki/Maps/Loading to document how to load on our servers and added https_proxy to it

The documentation now covers enough to have all the commands which need to download data that are run on import.

Some related work is T194850: Move external data sources to HTTPS where I need to change some HTTP URLs to HTTPS

Change 433776 had a related patch set uploaded (by Pnorman; owner: Pnorman):
[operations/puppet@production] Move process-osm-data example URLs to https

https://gerrit.wikimedia.org/r/433776

Change 433776 merged by Gehel:
[operations/puppet@production] Move process-osm-data example URLs to https

https://gerrit.wikimedia.org/r/433776

Pnorman closed this task as Resolved.May 25 2018, 10:28 PM