Page MenuHomePhabricator
Create Task
Maniphest T268886

Understand which Phab forms are used how often (by analyzing referer logs?)
Closed, ResolvedPublic
Actions

Description

Coming here from thinking more about T253744: Improve initial Phab task quality: Encourage use of bug report/feature request task subtypes; Review default options in Favorites menu and how to "encourage" setting task subtypes.

Phabricator introduced Forms ~4y ago (in addition to passing URL parameters), in order to create tasks / prefill data in the task creation form.

Since then we've seen a growing number of Phab forms created.
We currently seem to happily create forms to cover specific use cases, but some forms don't seem very discoverable if you don't know about them.
Plus we have no idea which forms are actually used and which once might not be used anymore, as there is no way to find out which task was created by using which form.
Plus some might overlap (or superseded by others without disabling the older ones, and the older ones still linked from somewhere).

I'm wondering if having access to HTTP referer logs and analyzing them could help.
I don't know if this is feasible at all, or if my use case makes enough sense.
Input welcome.

Related Objects
Search...

Event Timeline

Aklapper triaged this task as Lowest priority.Nov 27 2020, 12:23 PM
Aklapper created this task.
Aklapper mentioned this in T253744: Improve initial Phab task quality: Encourage use of bug report/feature request task subtypes; Review default options in Favorites menu.Dec 2 2020, 10:24 PM
• mmodell subscribed.Dec 8 2020, 5:59 PM

I think we have referrer logs on the phabricator server but they aren't retained for very long and they aren't currently analyzed.

Aklapper added a comment.EditedJan 7 2021, 5:29 PM

Guess I'd have to find out if we have a datalake with HTTP referer logs for the Phab domain, and find out whether to access via Hive or Superset or Turnilo or... is best suited - Analytics. Maybe something like https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Traffic/Webrequest

Aklapper added a comment.Jan 9 2021, 5:40 PM

Alright... basically https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Traffic/Webrequest :
Log into Hive on stat1007, then use wmf DB.

Tasks that must have been created yesterday (the T271xxx) via
hive (wmf)> SELECT referer, uri_path FROM webrequest WHERE year = 2021 AND month = 1 AND day = 8 AND uri_host = "phabricator.wikimedia.org" AND uri_path LIKE "/T271%" LIMIT 50; show no results where referer includes a Phab form URL. Hmm, didn't expect that.

hive (wmf)> SELECT referer, uri_path FROM webrequest WHERE year = 2021 AND month = 1 AND day = 8 AND uri_host = "phabricator.wikimedia.org" AND referer LIKE "%/task/edit/for%" LIMIT 50; only had two results (a form with some url params, going to itself without params). I would have expected more results.

So I'm not sure if I understand the system correctly, and I have not yet tried something heavy like
hive (wmf)> SELECT referer, uri_path FROM webrequest WHERE year = 2021 AND referer LIKE "https://phabricator.wikimedia.org/maniphest/task/edit/form/%" AND uri_host = "phabricator.wikimedia.org" AND uri_path LIKE "/T2%" LIMIT 50;
(plus that would welcome a reduce operator anyway).

Anyway, putting this here so I don't reinvent wheels at some point in the future, plus if someone else has ideas.

Aklapper claimed this task.Apr 9 2021, 12:43 PM
Aklapper added a comment.Apr 9 2021, 12:46 PM

I've set up a panel in Logstash for the last 3mo at https://logstash.wikimedia.org/app/dashboards#/view/b20f37b0-992a-11eb-85b7-9d1831ce7631 . Basically: url.domain:"phabricator.wikimedia.org" && (as query DSL) {"query":{"wildcard":{"url.path":"/maniphest/task/edit/form/*"}}}

Curious where folks come from I also tried AND !http.request.referrer:"-" but looking at the data it does not look too useful.

Aklapper added a comment.Apr 9 2021, 12:52 PM

Phab task forms access stats between 2021-01-09 to 2021-04-09 (does not include disabled forms):

P15269 Phab Task Forms access stats for last 3mo (2021-04-09)
1Path Hits Name
2/maniphest/task/edit/form/ 10 (default)
3/maniphest/task/edit/form/1/ 15923 (default)
4/maniphest/task/edit/form/3/ 683 New Task (Advanced)
5/maniphest/task/edit/form/8/ 37 SRE Access Request
6/maniphest/task/edit/form/10/ 139 Edit Task
7/maniphest/task/edit/form/16/ 5 New Private Task (WMF-FR)
8/maniphest/task/edit/form/17/ 11 Request Custom Form
9/maniphest/task/edit/form/18/ 2 Create Prioritized Task
10/maniphest/task/edit/form/22/ 42 Wiki configuration change request
11/maniphest/task/edit/form/23/ 20 New WMF-NDA Task (Private)
12/maniphest/task/edit/form/30/ 3 #wikitech.wikimedia.org user access/permission requests
13/maniphest/task/edit/form/33/ 31 Request repository archiving
14/maniphest/task/edit/form/37/ 161 Edit Deadline
15/maniphest/task/edit/form/43/ 289 New Bug Report (creation form)
16/maniphest/task/edit/form/46/ 171 New Production Error
17/maniphest/task/edit/form/47/ 10 Create Task with a Due Date
18/maniphest/task/edit/form/52/ 14 Server decommission request
19/maniphest/task/edit/form/55/ 15 Hardware Failure / Troubleshoot
20/maniphest/task/edit/form/56/ 12 Feature Request Form
21/maniphest/task/edit/form/59/ 17 Bug Report Form (edit form)
22/maniphest/task/edit/form/60/ 21 Spike Task Form
23/maniphest/task/edit/form/61/ 3 Design Task Form
24/maniphest/task/edit/form/63/ 2 CommRel Private request
25/maniphest/task/edit/form/65/ 8 CommRel Support Request
26/maniphest/task/edit/form/66/ 77 SRE Procurement Request
27/maniphest/task/edit/form/70/ 4 Create Task with a Due Date
28/maniphest/task/edit/form/71/ 11 Request Performance Review
29/maniphest/task/edit/form/72/ 23 Security Request For Service
30/maniphest/task/edit/form/73/ 4 Security Type Advanced
31/maniphest/task/edit/form/74/ 1 Edit Security Issue
32/maniphest/task/edit/form/75/ 121 New Security Issue (Basic)
33/maniphest/task/edit/form/77/ 1 Security Concept Review
34/maniphest/task/edit/form/79/ 8 Security Readiness Review
35/maniphest/task/edit/form/80/ 8 Hardware Racking Request
36/maniphest/task/edit/form/81/ 3 Create Task with a Due Date
37/maniphest/task/edit/form/84/ 5 WMF Editing Team Request
38/maniphest/task/edit/form/86/ 1 WMNO Onboarding
39/maniphest/task/edit/form/90/ 7 Create a new VM request
40/maniphest/task/edit/form/91/ 140 Create New Wiki Form
41/maniphest/task/edit/form/94/ 36 Private Task for Tech-PgM

RhinosF1 subscribed.Apr 9 2021, 1:06 PM
Aklapper mentioned this in T279785: Phab Forms 70 and 81 look like duplicates of form 47.Apr 9 2021, 1:37 PM
Aklapper added a parent task: T253744: Improve initial Phab task quality: Encourage use of bug report/feature request task subtypes; Review default options in Favorites menu.
Aklapper closed this task as Resolved.Apr 9 2021, 1:41 PM
Aklapper added a project: Developer-Advocacy (Apr-Jun 2021).
Aklapper moved this task from To Triage to Misc on the Phabricator board.

Done here; to be continued in T253744.

Aklapper mentioned this in T285939: Evaluate initial Phab form and subtype usage in 09/2021 after changes made in T253744.Jul 1 2021, 11:55 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL