In T138464#3117955, @Basvb wrote:

@tom29739: That would be wonderful, would it be a good idea to have a small chat on IRC this evening or tomorrow to discuss how we see the project and some next steps?

I can help with this project. I have experience in Python for software development, and I maintain a few tools on Tool Labs. I've also used Wikimedia's OAuth in one of my tools.

The easiest and safest solution would probably be using something like Bucardo (live migration tutorial).

It only happens sometimes, for example this message a few hours before the task in the description links to the correct task:
<wikibugs> Labs, Tool-Labs, Tool-Labs-tools-Database-Queries: nlwiki Labs replica with problems - https://phabricator.wikimedia.org/T138927#3043967 (Sjoerddebruin) Resolved>Open It shouldn't be that this page and two **privacy violations** keep appearing on pages as https://tools.wmflabs.org/wikidata-to...

PyICU requires the underlying C++ library to work, and this is in the libicu52 apt package on Debian Jessie. When this is installed, PyICU can then be installed in a virtualenv. Alternatively, the python-pyicu apt package can be installed, which installs PyICU and the underlying C++ library systemwide. One of the above packages would need to be installed in the python 2 Docker image to be used in Kubernetes containers.

The current version that is on Tool Labs, v0.10.25, was made end of life over 3 months ago. This means that it's a security risk.

In T154551#2917678, @Matthewrbowker wrote:

The rebirth code is written using the Symfony framework: http://symfony.com/. I opted to use the framework because it is very similar to Ruby on Rails, with regard to the Model-View-Controller scheme. Also, it includes caching and profiling out of the box.

@scfc: I ran the tests and reported back in T137146#2388908. I found that while xdebug slowed down the requests by a small amount, it was NFS that slowed down the requests the most.

In T143939#2820868, @scfc wrote:

In T143939#2584266, @valhallasw wrote:

To recap some of the arguments from Cloud-Services:

• ProxyCommand is complex for windows users; a VPN would allow a much more user-friendly connection (directly connecting to e.g. tools-exec-1201.eqiad.wmflabs from putty)

[…]

Is the configuration really that complex? https://wikitech.wikimedia.org/wiki/Help:Access_to_instances_with_PuTTY_and_WinSCP#How_to_set_up_PuTTY_for_proxying_through_bastion.wmflabs.org_to_your_instance looks tedious to repeat for every instance, but fairly simple.

The interim solution of using a compiled wheel of pyenchant no longer works because the wheel at http://tools-docker-builder-01.eqiad.wmflabs/pyenchant-1.6.7-py2.py3.cp27.cp32.cp33.cp34.cp35.pp27.pp33-none-any.whl has been taken down. Having the enchant C library (libenchant1c2a) installed in the image would remove the need for the wheel to be compiled and hosted.

Being able to use custom Docker images would be extremely helpful, because it removes the need to have to compile source or find binaries and/or ask for packages to be installed.

Why not just redirect the .wiki domains to the .org domains?

Graylog (https://github.com/graylog2/graylog2-server, https://www.graylog.org/) seems to cover most of these points:

1. Take load off NFS - logs are stored on an Elasticsearch cluster
2. Make it far faster to see the actual logs from processes - doesn't depend on NFS, so should be fast
3. Be able to search through logs easier - searching is easy: http://docs.graylog.org/en/2.0/pages/queries.html, "The search syntax is very close to the Lucene syntax. By default all message fields are included in the search if you don’t specify a message field to search in."
4. Automatically drop older logs - index rotation can be configured based on message count, index size, or index time: http://docs.graylog.org/en/2.0/pages/index_model.html#eviction-of-indices-and-messages
5. Provide a Filesystem based interface for log ingress - Graylog supports this: http://docs.graylog.org/en/2.0/pages/sending_data.html#reading-from-files, "we provide the Collector Sidecar which acts as a supervisor process for other programs, such as nxlog and Filebeats, which have specifically been built to collect log messages from local files and ship them to remote systems like Graylog."
6. Provide more standard and modern interfaces (gelf? etc) for log ingress - Graylog supports GELF and syslog: http://docs.graylog.org/en/2.0/pages/sending_data.html
7. Provide a filesystem based interface for log reading - I don't think this is supported, but you can export search results to CSV: http://docs.graylog.org/en/2.0/pages/queries.html#export-results-as-csv
8. Provide a more modern interface for log reading as well - Graylog's interface looks fairly modern and easy to use to me:
   
   
9. Be secure in allowing only authenticated members to read a particular tool's logs. - Graylog has access control out-of-the-box, and can integrate with LDAP users and groups: http://docs.graylog.org/en/2.0/pages/users_and_roles/external_auth.html#ldap-active-directory

Graylog has Streams (basically categories for log messages): http://docs.graylog.org/en/2.0/pages/streams.html, alerting based on those streams: http://docs.graylog.org/en/2.0/pages/getting_started/stream_alerts.html, and dashboards: http://docs.graylog.org/en/2.0/pages/dashboards.html.

In T143939#2585688, @zhuyifei1999 wrote:

Doesn't the ssh of Cygwin / MinGW / MSYS (not an expert here) work?

OpenVPN server can be configured to only provide routes to certain IP ranges/hosts

In T143939#2584438, @Platonides wrote:

Sane configuration, not allowing traffic through labs (e.g. user -> labs -> google)

Note that it is possible to configure openvpn so it only provides routes to a subrange (ie. labs hosts). It could be blocked anyway¹ but that would require the user misconfigurating his routes.

¹ Not sure if it is worth it, since labs users already can access the internet through their instances.

In T143939#2584425, @AlexMonk-WMF wrote:

You'd need labsdb/wmnet for database replica access. I'm not quite sure how DNS would work actually - the client would need to be able to resolve these, but then all their own DNS lookups would be going through labs?

OpenVPN

• Using LDAP to authenticate would be easy: https://openvpn.net/index.php/access-server/docs/admin-guides/190-how-to-authenticate-users-with-active-directory.html
• Would only carry traffic to Labs hosts (e.g. *.wmflabs/*.labsdb hosts), and not to internet.
• OpenVPN has clients for many different platforms, including Windows.
• OpenVPN is fast and secure.

But Horizon tells you how much RAM/CPU/storage that flavour gives the instance when you pick the flavour on there

I have a replica.my.cnf in my home dir that was created in January 2016. There has been far more than 6 new users on Tool Labs so far this year.

Perhaps it would make sense to have a separate notice, given that the username isn't as confidential as say, an IP address or similar

In T140486#2494461, @ZhouZ wrote:

Ok in that case, going back to the second question - while a username might not be categorized as private information like IP addresses and passwords per se, should we still treat them with sensitivity in some cases since users might want to keep their association with particular tools confidential? That is even if we don't categorize usernames as information that must be purged or anonymized after 30 days, is this information still sensitive enough to warrant warning the user about the possibility of disclosure as covered in the current "If my tools collect Private Information..." disclaimer notice?

I don't think that's the case. The user using OAuth via a tool will have their username disclosed to the tool though (subject to the usual private info restrictions at present).

In T140486#2494436, @ZhouZ wrote:

Additionally, can someone clarify this statement:

A tool that edit on a user's behalf via OAuth automatically disclose the username to MediaWiki revision table, thus publicly and permanently storing "This username has used this tool", and there's no way for the tool at all to change that fact.

The tool used by a particular user, via OAuth, are publicly available information? If so, is this desired behavior (e.g. are there circumstance where a user might not want that fact they are using a tool to be disclosed)?

@Magnus: T140110: Packages to be installed in Toolforge Kubernetes Images (Tracking) is the tracking task for packages to be installed in Kubernetes containers. You'll need to create a subtask of that.

Java 1.8 is supported on Kubernetes in Tool Labs. If the SGE scripts could be converted or replaced with Kubernetes equivalents, then MerlBot could probably run on that. I recall that not being able to use Java 1.8 was a blocker in getting MerlBot to use HTTPS because it could not use an updated version of a library or additional features provided in Java 1.8 (I might be remembering this wrong. Probably)

Works for me.

In T140486#2474705, @zhuyifei1999 wrote:

IMHO, username shouldn't be private information. Rather, the association of a username and other private information (IP address, User agent) should be private.

Quarry would be non compliant because the ToU classes usernames as private information. The ToU states that you *must* show this disclaimer before collecting the private information (in this case the username):

By using this project, you agree that any private information you give to this project may be made publicly available and not be treated as confidential.
By using this project, you agree that the volunteer administrators of this project will have access to any data you submit. This can include your IP address, your username/password combination for accounts created in Labs services, and any other information that you send. The volunteer administrators of this project are bound by the Wikimedia Labs Terms of Use, and are not allowed to share this information or use it in any non-approved way.
Since access to this information is fundamental to the operation of Wikimedia Labs, these terms regarding use of your data expressly override the Wikimedia Foundation's Privacy Policy as it relates to the use and access of your personal information.

@Dalba

tools.piagetbot@tools-bastion-03:~$ virtualenv ~/venv -p /usr/bin/python3
Running virtualenv with interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /data/project/piagetbot/venv/bin/python3
Also creating executable in /data/project/piagetbot/venv/bin/python
Installing setuptools, pip...done.

(of course replace the "~/venv" with the directory you want the virtualenv to go)

Just typing editor in a shell prompt brings up joe, which on further investigation:

tom29739@tools-bastion-03:~$ type editor
editor is /usr/bin/editor
tom29739@tools-bastion-03:~$ file /usr/bin/editor
/usr/bin/editor: symbolic link to `/etc/alternatives/editor'
tom29739@tools-bastion-03:~$ file /etc/alternatives/editor
/etc/alternatives/editor: symbolic link to `/usr/bin/joe'

I think it's that.

That's weird. It should be back on then...

It doesn't appear to be working. It's not on IRC, so I'd assume it's not working. @yuvipanda did you delete the pid files when you tried to get it working? It crashed last time, so those files need to be removed.

sudo rm -iv /mnt/share/wm-bot/*.pid

Should do that according to the docs.

Or alternatively wait for one of the bot's roots to come along, but that
might take a while.

In T136712#2410709, @yuvipanda wrote:

On running strace for pip -V (which took about 20s), I get:

$ strace -o /tmp/pip-strace-counts -f -c pip -V
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 73.62    1.648355         350      4716      4158 open
 24.50    0.548619         248      2209      2018 stat
  0.45    0.010146         122        83         4 lstat
  0.29    0.006464        3232         2           wait4
  0.17    0.003729           5       728           read
  0.16    0.003644           6       580           close
  0.16    0.003563         210        17           openat 
  0.12    0.002582          70        37        21 access
  0.09    0.002057           5       381           mmap
  0.08    0.001770          89        20           write
  0.07    0.001568           2       882           fstat
  0.07    0.001514           6       275           munmap
  0.06    0.001412         282         5         2 readlink
  0.06    0.001251          15        82           rt_sigaction
  0.04    0.000898          26        34           getdents
  0.02    0.000530           9        56           mprotect
  0.01    0.000307           7        42           brk
  0.00    0.000099          25         4           execve
  0.00    0.000096          16         6         1 ioctl
  0.00    0.000052          13         4           arch_prctl
  0.00    0.000044          22         2           set_robust_list
  0.00    0.000039          13         3           dup2
  0.00    0.000034          17         2           geteuid
  0.00    0.000028          14         2           getpid
  0.00    0.000026          13         2           getppid
  0.00    0.000025           8         3           fcntl
  0.00    0.000021           5         4           uname
  0.00    0.000017           9         2           getrlimit
  0.00    0.000016           8         2         1 futex

Perhaps ulimits are causing it?

In T136712#2410957, @yuvipanda wrote:

Indeed, or even a few months ago. The only significant thing I can think of as having changed since then is the NFS rate limiting, so I'd still like to maybe completely remove that and try testing this again to see if that's the cause.

NFS was much faster about 5-6 months ago, when I joined. It's having an impact on the speed of other things too (e.g. the speed of webservices, especially with PHP).

In T136712#2410687, @zhuyifei1999 wrote:

In T136712#2410605, @tom29739 wrote:

That's another set of results, but note that the version of pip on my tool is pip 1.5.4 when @zhuyifei1999's results were with pip 8.1.2, so this might cause a difference. Pip 1.5.4 appears to be the default version on tool labs (just tested with a new virtualenv with no packages)

It's more likely that I simply have too many packages required to run my tool :)

19s is still a long time when it's supposed to run for less than a second.

@valhallasw looks like it's down again.

It looks like it's spewing out incorrect gzipped data, which the browser can't decode so it downloads it as a file instead.
Maybe the default DocumentRoot is in the wrong place so that the webserver is selecting the wrong file for the user's browser to interpret? That's the only plausible explanation that I can think of where setting the DocumentRoot would fix it.

@yuvipanda Chrome Version 51.0.2704.103 m (32 bit) on Windows 7. download.gz is 707 bytes from the https:// version. WinRAR can open it, there's a file named 'download' inside which is 677 bytes.

@yuvipanda neither 1. or 2. work for me, chrome tries to download a file called 'download.gz'. Both work with curl.

@chasemp it just happens randomly, not every time but still often enough to get annoying. I don't think it's anything that I'm doing because Niharika was having the exact same issue on their project.

A potential problem I foresee is different languages wanting to use the same page name. If wikis A, B and C all want to use page name X, then which wiki's content is used?

I got really confused by this yesterday, nothing worked, because I didn't know about this.

My tools often need different packages to be installed, and sometimes these can be installed easily (binaries available to download) but often the only options for binaries is a custom apt repo, or compiling from source, which is very time consuming and doesn't always work. These packages would make it possible to make a chroot as a non root user, so packages can be installed inside the chroot with apt and can be used.

No longer needed.

That project is no longer needed.

I have requested a new labs project for testing the impact that xdebug has: T138097: Create new labs project tools-xdebug-testing

I discovered that NFS was a huge slowdown of all webservices, and that running a webservice in /tmp, I copied the files to the bastion /tmp and made a symlink, and it worked (strangely, lighttpd could access the bastion's /tmp, because there wasn't anything in the webserver host's /tmp when I checked).
Although NFS did slow down the webservice hugely, other factors (like xdebug) are still affecting the webservice. I'm going to assign myself this task and do some more testing to find out whether xdebug (and maybe other factors) make a difference in the speed of the webservice.