Page MenuHomePhabricator

Remove logging from labs for schema https://meta.wikimedia.org/wiki/Schema:CommandInvocation
Closed, ResolvedPublic5 Estimated Story Points

Description

Looks like data is still logged for this schema but not used, we should remove logging if so. Thanks!

Event Timeline

Background: T67508 broke logging for this schema. Also it's one of two schemas which don't use the standard JS or PHP logging code so somewhat of a maintenance burden in the long run.

Is this tagged with cloud-services-team in error, or is there something you need from us?

@Andrew : you guys own the script that logs this data (i think it was created by Yuvi a while back) . It doesn't seem like it is used

modules/toollabs/files/log-command-invocation in puppet
T123444

<yuvipanda> Krenair: you should ask some of the cloud team people about it :)

bd808 added a subscriber: bd808.

This schema was for finding out about how people used our jsub and webservice commands. We can shut this down. I don't think anyone has even tried to look at the data for over a year. Removing is two part:

  • make the Puppet managed helper script a no-op
  • remove calls to the helper script from jsub and webserivice

The first would be all that is needed to unblock dropping the schema.

Change 381619 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[labs/toollabs@master] Remove log-command-invocation calls

https://gerrit.wikimedia.org/r/381619

Change 381712 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[operations/puppet@production] wmcs: Make /usr/local/bin/log-command-invocation a no-op

https://gerrit.wikimedia.org/r/381712

Change 381713 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[operations/software/tools-webservice@master] Remove Schema:CommandInvocation EventLogging

https://gerrit.wikimedia.org/r/381713

Let us know if we can also delete the tables this schema was logging to

Let us know if we can also delete the tables this schema was logging to

You can truncate them now if you'd like. I'll update here once we actually get all the callers removed. Until that happens the events will keep coming in.

Change 381712 merged by Andrew Bogott:
[operations/puppet@production] wmcs: Make /usr/local/bin/log-command-invocation a no-op

https://gerrit.wikimedia.org/r/381712

Change 381619 merged by jenkins-bot:
[labs/toollabs@master] Remove log-command-invocation calls

https://gerrit.wikimedia.org/r/381619

@bd808: sounds good, let us know when calls are no loger coming in and we can drop CommandInvocation table, FYI @elukey

Change 381713 merged by jenkins-bot:
[operations/software/tools-webservice@master] Remove Schema:CommandInvocation EventLogging

https://gerrit.wikimedia.org/r/381713

bd808 removed bd808 as the assignee of this task.Oct 12 2017, 11:54 PM
bd808 removed projects: Patch-For-Review, User-bd808.

The code has been rolled out everywhere to stop sending events to the eventlogging beacon endpoint. The webservice script was logging directly (not via the /usr/local/bin/log-command-invocation helper). This script is packaged in the Docker containers that Toolforge Kubernetes apps run in, so the old version of the script is still in the wild for all Kubernetes pods that were started before I rebuilt the Docker images. They will get the new version at the next restart of each pod.

I doubt that there is a lot of running of webservice done from inside a running pod, so it should be fairly safe to drop the CommandInvocation table now.

ping @elukey Can we drop the CommandInvocation table? Thank you

@Nuria Both tables right?

+-------------------------------------+
| Tables_in_log (CommandInvocation%)  |
+-------------------------------------+
| CommandInvocation_15243810          |
| CommandInvocation_15243810_15423246 |
+-------------------------------------+

Yes, pinging @bd808 that those two will be deleted soon.

Yes, pinging @bd808 that those two will be deleted soon.

+1

Mentioned in SAL (#wikimedia-operations) [2017-11-03T08:16:37Z] <elukey> drop CommandInvocation_15243810 and CommandInvocation_15243810_15423246 from analytics dbs (db1046/db1047/db1108/dbstore1002) - data archived on HDFS - T166712

elukey triaged this task as Medium priority.
elukey edited projects, added Analytics-Kanban; removed Analytics.
elukey set the point value for this task to 5.
elukey moved this task from Next Up to Done on the Analytics-Kanban board.