Page MenuHomePhabricator

Bug: client IP is being hashed differently by the different parallel processors {stag} [13 pts]
Closed, ResolvedPublic

Description

This line: https://github.com/wikimedia/mediawiki-extensions-EventLogging/blob/fa0aac1d6e0a0ee0e24f144abe69ec9003ce9f7f/server/bin/eventlogging-processor#L106

is now running on 12? different processors. The parser that's created by each of those processors will hash the same IP address differently. This has caused queries that rely on consistent hashing to fail. We should either tell people that IP addresses are no longer going to be hashed consistently or we should figure out a way to share the hashing function. If we decide to keep it inconsistent, we should find and help people who are relying on consistency.

Related Objects

Event Timeline

Milimetric raised the priority of this task from to Needs Triage.
Milimetric updated the task description. (Show Details)
Milimetric added a project: Analytics-Kanban.
Milimetric added a subscriber: Milimetric.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 15 2015, 7:04 PM
Milimetric triaged this task as High priority.Sep 15 2015, 7:23 PM
Milimetric set Security to None.

Change 238533 had a related patch set uploaded (by Milimetric):
Hack around bad Event Logging IP hash problem

https://gerrit.wikimedia.org/r/238533

Change 238533 merged by Milimetric:
Hack around bad Event Logging IP hash problem

https://gerrit.wikimedia.org/r/238533

By the way, for the https://edit-analysis.wmflabs.org/compare/ dashboard, if you clear cache it looks like the data is back in order. As I mentioned in the notes, there will now be a slight abnormality for any browsers that don't support the crypto functions which generate random session ids. For those, session ids will collide. If you suspect any problems with the data, I'd look into that first. A human can usually tell the collisions from true sessions based on the timestamps of the actions in the session.

Change 238784 had a related patch set uploaded (by Ottomata):
Run 1 EventLogging processor until bug is fixed

https://gerrit.wikimedia.org/r/238784

Change 238784 merged by Ottomata:
Run 1 EventLogging processor until bug is fixed

https://gerrit.wikimedia.org/r/238784

I just restarted eventlogging client side processor running only 1 process. This should fix the problem until we actually fix the bug.

Change 238854 had a related patch set uploaded (by Ottomata):
[POC] add etcd_rotating_key as an option for sharing hash keys between mulitple processes

https://gerrit.wikimedia.org/r/238854

Ottomata renamed this task from Bug: client IP is being hashed differently by the different parallel processors to Bug: client IP is being hashed differently by the different parallel processors {stag} [8 pts].Sep 18 2015, 3:14 PM

Change 240916 had a related patch set uploaded (by Ottomata):
Puppetize etcd use for eventlogging processor

https://gerrit.wikimedia.org/r/240916

Change 240916 merged by Ottomata:
Puppetize etcd use for eventlogging processor

https://gerrit.wikimedia.org/r/240916

Change 242201 had a related patch set uploaded (by Ottomata):
Set default for $ssldir in etcd::ssl::base

https://gerrit.wikimedia.org/r/242201

Change 242201 merged by Ottomata:
Set default for $ssldir in etcd::ssl::base

https://gerrit.wikimedia.org/r/242201

Change 242203 had a related patch set uploaded (by Ottomata):
Remove etcd package requirement from etcd::ssl::base

https://gerrit.wikimedia.org/r/242203

Change 242203 merged by Ottomata:
Remove etcd package requirement from etcd::ssl::base

https://gerrit.wikimedia.org/r/242203

Change 242207 had a related patch set uploaded (by Ottomata):
Ensure /var/lib/etcd exists even if etcd package is not installed

https://gerrit.wikimedia.org/r/242207

Change 242207 merged by Ottomata:
Ensure /var/lib/etcd exists even if etcd package is not installed

https://gerrit.wikimedia.org/r/242207

Change 242213 had a related patch set uploaded (by Ottomata):
Apparently dirname isn't available, set $vardir and ensure it exists for etcd

https://gerrit.wikimedia.org/r/242213

Change 242213 merged by Ottomata:
Apparently dirname isn't available, set $vardir and ensure it exists for etcd

https://gerrit.wikimedia.org/r/242213

Change 242229 had a related patch set uploaded (by Ottomata):
Parameterize owner, group and modes for etcd::ssl::base files

https://gerrit.wikimedia.org/r/242229

Change 242229 merged by Ottomata:
Parameterize owner, group and modes for etcd::ssl::base files

https://gerrit.wikimedia.org/r/242229

Change 238854 merged by Ottomata:
Add SharedRotatingToken to share hash tokens in etcd

https://gerrit.wikimedia.org/r/238854

Change 242251 had a related patch set uploaded (by Ottomata):
Fix eventlogging ssl paths for etcd

https://gerrit.wikimedia.org/r/242251

Change 242251 merged by Ottomata:
Fix eventlogging ssl paths for etcd

https://gerrit.wikimedia.org/r/242251

Change 242270 had a related patch set uploaded (by Ottomata):
Use ca_cert for etcd client in eventlogging processor

https://gerrit.wikimedia.org/r/242270

Change 242270 merged by Ottomata:
Use ca_cert for etcd client in eventlogging processor

https://gerrit.wikimedia.org/r/242270

Change 242338 had a related patch set uploaded (by Ottomata):
eventlogging now requires python-etcd

https://gerrit.wikimedia.org/r/242338

Change 242338 merged by Ottomata:
eventlogging now requires python-etcd

https://gerrit.wikimedia.org/r/242338

Change 242359 had a related patch set uploaded (by Ottomata):
Lower python-etcd version requirement, we run a patched 0.3.3 in prod

https://gerrit.wikimedia.org/r/242359

Change 242359 merged by Ottomata:
Lower python-etcd version requirement, we run a patched 0.3.3 in prod

https://gerrit.wikimedia.org/r/242359

Change 242413 had a related patch set uploaded (by Ottomata):
Comment out python-etcd dependency so we can install on Trusty

https://gerrit.wikimedia.org/r/242413

Change 242413 merged by Ottomata:
Comment out python-etcd dependency so we can install on Trusty

https://gerrit.wikimedia.org/r/242413

Phew, ok, I got this deployed, but I had to do a little hackery.

Our packaged version of python-etcd has a python dependency for python-openssl >= 0.14. Trusty does not have this available as a .deb package, so if eventlogging code specifies the python-etcd dependency, it will refuse to install.

I worked around this by commenting out the python-etcd requirement in eventlogging's setup.py. Puppet installs this in production anyway, and this seems to work just fine with python-openssl 0.13, so now eventlogging can be installed in Trusty.

The downside is, if you attempt to install on Trusty in labs, pip will not auto-install python-etcd.

2 things come out of this:

  1. we should make it so that you can't use eventlogging in labs without the proper deb packages. We should disable the usage of pip there.
  2. We should upgrade all eventlogging servers to Jessie, so things just work properly.

I will make a task for 2.

ggellerman renamed this task from Bug: client IP is being hashed differently by the different parallel processors {stag} [8 pts] to Bug: client IP is being hashed differently by the different parallel processors {stag} [13 pts].Sep 30 2015, 3:35 PM
kevinator closed this task as Resolved.Oct 2 2015, 3:30 PM
kevinator added a subscriber: kevinator.