Page MenuHomePhabricator

Add fields to support bug specific information
Closed, ResolvedPublic

Description

The apps team is starting a practice of having required information on a bug ticket to enable our team members to estimate and reproduce bugs more reliably.

We documented these requirements here:
http://www.mediawiki.org/wiki/Wikimedia_Apps/Team/Bug_reporting

To share this with the rest of the organization and to better support the entry of specific data, we would like add some or all of these requirements as separate fields in Phabricator tasks.

The new fields (taken from the document above) would be:

  • Device(s)
  • OS(s)
  • Observed Behavior
  • Expected Behavior
  • Steps to Reproduce

It wold be nice to hear other contributors thoughts on the above document and proposed fields to see if this is something others in the organization would want, and what fields they think should be added.

This task may require that there are different ticket types (feature, bug, story, epic) before it can be implemented. For instance, we may not want these fields visible on feature tickets.

Event Timeline

Fjalapeno raised the priority of this task from to Needs Triage.
Fjalapeno updated the task description. (Show Details)
Fjalapeno added a project: Phabricator.

I am very reluctant to create explicit fields per product. We had lots of fields in previous Bugzilla and (anecdotal impression, not covered by data) they were mostly ignored.
Would using templates be an acceptable workaround? Please see T91538.

http://www.mediawiki.org/wiki/Wikimedia_Apps/Team/Bug_reporting

Most of that looks like duplication of canonical https://www.mediawiki.org/wiki/How_to_report_a_bug ?
Is that filed under /Team because those recommendations can be safely ignored by any non-team bug reporters (users of those mobile apps)?

Aklapper renamed this task from Add fields to support bug related information to Add fields to support bug related information for Wikimedia Apps team.Mar 15 2015, 4:30 PM
Aklapper triaged this task as Low priority.
Aklapper set Security to None.
Aklapper added a subscriber: Qgil.

Sounds like a variation of T91538. Basically, creating custom fields per project is not feasible (as far as I know), but you can create a link that populates the new task with the content you wish.

Should we merge this task?

I am very reluctant to create explicit fields per product. We had lots of fields in previous Bugzilla and (anecdotal impression, not covered by data) they were mostly ignored.

I think an important part of this would be making fields required - no one should be able to file a bug without providing an OS version or steps to complete.

Would using templates be an acceptable workaround? Please see T91538.

A template is a stop gap, but not a solution.
A freeform description field invites people to file poorly written bugs.

http://www.mediawiki.org/wiki/Wikimedia_Apps/Team/Bug_reporting

Most of that looks like duplication of canonical https://www.mediawiki.org/wiki/How_to_report_a_bug ?

We just put this together because we realized we weren't following any format when filing bugs internally. (I don't actually think people on the team either knew the page you reference existed, or at least didn't think it was applicable to our project - I didn't know about it, but I am pretty new)

We do have 2 additional fields that are important for mobile bug triage/diagnosis - OS and Device.

Is that filed under /Team because those recommendations can be safely ignored by any non-team bug reporters (users of those mobile apps)?

It's stored here because thats were we put all of our documentation. We were actually looking for wider adoption - that was part of the reason we filed this ticket - a way to standardize the filing of bugs for the organization in Phabricator. Documentation is nice, but forms are much better. :)

Sounds like a variation of T91538. Basically, creating custom fields per project is not feasible (as far as I know), but you can create a link that populates the new task with the content you wish.

Should we merge this task?

I would consider templates a stop gap. If multiple teams need/want these templates - it would stand to reason that there is a real need for fields.

While templates are good, A freeform description field is limiting. Information is not structured and is less likely to be entered correctly.

A good analogue is the development of Wikidata to supplement the freeform nature of Mediawiki pages - structured data is just much more powerful for requirements, reports, and filtering.

Fields per project may be a bit of a stretch as a first pass, but if the community could standardize on fields for a bug task, I think that would be a big win.

Note that in Bugzilla we had specific fields for "Hardware", "OS", and "Browser", and we decided to get rid of them. While these fields are very relevant for mobile, they were more misused than used for our majority of product and components, which don't depend much on them. You can play with queries at https://old-bugzilla.wikimedia.org/query.cgi?format=advanced

"Observed Behavior", "Expected Behavior", and "Steps to Reproduce" is a pattern for describing bugs, but we are using Phabricator for a lot more things than bugs.

If the initial problem is that you were missing structure in the bugs filed in your own team, a first step is to enforce such structure to yourselves. If you get bug reports poorly described, you can always suggest reporters to improve their descriptions based on http://www.mediawiki.org/wiki/Wikimedia_Apps/Team/Bug_reporting . As said, forcing these fields will not make any sense for users submitting tasks, goals, etc.

Note that in Bugzilla we had specific fields for "Hardware", "OS", and "Browser", and we decided to get rid of them. While these fields are very relevant for mobile, they were more misused than used for our majority of product and components, which don't depend much on them. You can play with queries at https://old-bugzilla.wikimedia.org/query.cgi?format=advanced

"Observed Behavior", "Expected Behavior", and "Steps to Reproduce" is a pattern for describing bugs, but we are using Phabricator for a lot more things than bugs.

If the initial problem is that you were missing structure in the bugs filed in your own team, a first step is to enforce such structure to yourselves. If you get bug reports poorly described, you can always suggest reporters to improve their descriptions based on http://www.mediawiki.org/wiki/Wikimedia_Apps/Team/Bug_reporting . As said, forcing these fields will not make any sense for users submitting tasks, goals, etc.

Is there a task filed to add different ticket types? That way we could have one set of fields for bugs, one for goals, one for features, etc…

Is there a task filed to add different ticket types?

You filed one: T93499.

A template is a stop gap, but not a solution.
A freeform description field invites people to file poorly written bugs.

A bug tracker invites people to file poorly written bugs.
The underlying idea seems to come up with non-freeform fields makes people somehow really spend time to correctly fill out these fields.

I've only seen rare cases where that worked in practice and that was in corporate environments.
I've maintained Bugzilla installations where templates ("Steps to reproduce", "Expected results", etc) were set up. Reporters leave fields empty or enter a random letter to bypass required fields, reporters put the actual results under steps to reproduce, reporters put "It should work!" under "Expected results". (Which is a very valid answer from a reporter point of view and there is not even intention to work around anything; it's just interpreting the field differently.)
In the worst case, reporters just wanted to report a bug in their spare time to help you improve your project, but they spend too much time trying to understand those many fields in your form and gave up and didn't report in the end.

So I encourage you to try a template first if you want to certain information to be present in tasks. :)
And I'm afraid that technical solutions won't fix social "problems" like understanding how to fill in such information in the way that it's useful to the developers.

One example that I have numbers for:

OS(s)

Wikimedia Bugzilla had a "Mobile Platform" field. Between March 15 2013 and March 15 2014, that field got set 58 times. Out of that, 36 times it was set by the reporter (trying to be a good citizen by filling out all those metadata fields). And (not disjunct) 15 times by me triaging tickets. Leaves 11 tickets where the field was either set or reset. 6 were removals of the value. So it was set 5 times in one year by non-reporter/non-bugwrangler (in tasks 39522,55315,62110,62537,62707).
So it basically was not used and if something has changed in Wikimedia Mobile in the last 12 months I'd like to know what. :)

Currently there are Wikipedia-Android-App-Backlog and Wikipedia-iOS-App-Backlog .

chasemp claimed this task.
chasemp added a subscriber: chasemp.

One example that I have numbers for:

OS(s)

Wikimedia Bugzilla had a "Mobile Platform" field. Between March 15 2013 and March 15 2014, that field got set 58 times. Out of that, 36 times it was set by the reporter (trying to be a good citizen by filling out all those metadata fields). And (not disjunct) 15 times by me triaging tickets. Leaves 11 tickets where the field was either set or reset. 6 were removals of the value. So it was set 5 times in one year by non-reporter/non-bugwrangler (in tasks 39522,55315,62110,62537,62707).
So it basically was not used and if something has changed in Wikimedia Mobile in the last 12 months I'd like to know what. :)

Currently there are Wikipedia-Android-App-Backlog and Wikipedia-iOS-App-Backlog .

^ this has been our approach. We have almost no fixed fields because all data points to them not being useful. Post upstream https://secure.phabricator.com/T3820 this could be addressed to where it wouldn't affect the entire organization. As things stand now to add a "Device(s)" field and worse make it mandatory every single task filed by every team would have to fill it in. The best option we have now is templates, and documenting how to file good bugs.

Ignoring the separate issue of submitting bugs from within the app, at the most basic level we need to be able to answer the following questions like the following using our bug tracker:

  • What bugs are only occurring on iOS 6?
  • How many bugs are specific to locale x?
  • etc.

If we can't do this by querying bugs with custom fields, should we assign tags? Parsing the description would probably be less than ideal. What would you recommend?

If you want users to submit bugs within the app, then you probably can construct the URL, and therefore the projects and pre-filled text. One question is how to actually submit the task directly, since we don't allow anonymous users to create tasks.

Ignoring the separate issue of submitting bugs from within the app, at the most basic level we need to be able to answer the following questions like the following using our bug tracker:

  • What bugs are only occurring on iOS 6?
  • How many bugs are specific to locale x?
  • etc.

If we can't do this by querying bugs with custom fields, should we assign tags? Parsing the description would probably be less than ideal. What would you recommend?

Preliminary commentary then if we are ignoring it :) Bugs from within the app (anonymous filings) are really more OTRS's domain? There isn't currently sane support in Phabricator for that kind of process. There is a fledgling upstream effort to make an app oriented towards this called Nuance but at the moment an anonymous RT or OTRS style bug creation work flow is dismal.

For bugs filed by humans now we converted Bugzilla parameters into mostly tags, yes.

Things like:

Browser-Support-Firefox
Browser-Support-Opera

We could do #android5 and #ios6 kind of thing? Having a really concrete idea of what parameters are filter worthy is ideal. Part of how to slice and dice bugs is @Aklapper's domain, but in general tags are a filterable/searchable/malleable/organizationable classification system.

There isn't currently sane support in Phabricator for that kind of process. There is a fledgling upstream effort to make an app oriented towards this called Nuance but at the moment an anonymous RT or OTRS style bug creation work flow is dismal.

I've heard about OTRS, and it's UX is supposedly also dismal. Not sure which is the lesser of two evils.

For bugs filed by humans now we converted Bugzilla parameters into mostly tags, yes.

Looks reasonable, maybe we can adopt this approach.

There isn't currently sane support in Phabricator for that kind of process. There is a fledgling upstream effort to make an app oriented towards this called Nuance but at the moment an anonymous RT or OTRS style bug creation work flow is dismal.

I've heard about OTRS, and it's UX is supposedly also dismal. Not sure which is the lesser of two evils.

For bugs filed by humans now we converted Bugzilla parameters into mostly tags, yes.

Looks reasonable, maybe we can adopt this approach.

FWIW when I used Phab before we had a setup akin to OTRS / Phab. A system for accepting anon bugs was in place and usually it was soem oddity on the user end, or much of the time. If an issue was deemed to be technical in nature, and the purview of engineering then it was escalated to Phab. We had a "script" to do that and it linked the public and the priviate issue. Here both would be public I guess, but yeah flooding Phab with auto-reports from crashing browsers or some such thing will be a bad deal. I think for the moment the two approaches are distinct enough to play off each other.

flooding Phab with auto-reports from crashing browsers or some such thing will be a bad deal.

Wherever they go, it needs to be somewhere that PMs and engineers can de-dupe and triage them easily. If OTRS is acceptable, I don't care as long as:

  • Users have 0 friction to submit quality bug reports
    • e.g. in-app dialogs that upload data to a reporting portal
  • Teams can triage and analyze bugs efficiently
    • scripts, query-able tasks, whatever. weigh trade-offs between tools and get it done

I think this thread has some great points, and I'll be sure to mention it when my team (iOS) decides to tackle bug-reporting in earnest (hopefully soon). We're just getting crash reporting out into the wild now, so this will be another big step in the direction of tightening feedback loops.

@chasemp Reopening this because the template stuff doesn't really solve the problem.

Additionally Team practices has weighed in on T93499, noting that multiple teams have expressed the need for different task types, for instance bugs, which have differentiating fields.

It would be good to ask others in the organization what fields they would like on bug tickets as well once we get figure out how to implement different ticket types.

I should note that it also seems the previous proposed solution of creating tags for every OS and app version is pretty unmanageable in Phab: Users can't even create tags without going through a vetting process to see if the tag is worthwhile to add.

If tags were free form and anyone could new tags, then it might be a solution for some of the items on this ticket (OS versions, platform), but that just doesn't seem to be how the organization ints to manage tags

@releng holds the reigns now, maybe talk to @greg

I should note that it also seems the previous proposed solution of creating tags for every OS and app version is pretty unmanageable in Phab: Users can't even create tags without going through a vetting process to see if the tag is worthwhile to add.

See T97273: Simpler process for Phabricator project creation

Thought: since this discussion seems to be getting somewhat circular in the comments, perhaps a meeting is in order to help move things along?

I see several related aspects mixed in this task.

  1. When reporting a task/bug, having several mandatory/required and partially machine-readable/queryable metadata fields. I've explained in T92708#1140375 why I don't consider that approach feasible. If anyone had positive experience with a "Fill out a bunch of mandatory dropdown and freeform fields to create a task/bug report" approach in an open+public issue tracking system: Please share details and provide an example link. (Because I'd love to finally see an example where that has worked out and try to understand the differences how to make that work.) For the time being and one potential approach, templates have been mentioned.
  1. When searching tasks, being able to search for specific pre-defined metadata values (e.g. implemented via custom fields or projects/tags). Examples brought up in this task were Device(s), OS(s), locale. Based on past experience with custom fields in Bugzilla, I've challenged how often this will be really used in T92708#1140398 and ask what has changed in the meantime so such fields would be actively used this time.
  1. A mechanism to (anonymously? Please see T972) (manually? half-automated?) submit tasks/bug reports (crashers? bugs? enhancement requests? all of them?) into some system (e.g. Phabricator?) from within the Wikimedia mobile app those tasks are about. The UI part of that mechanism on the app could offer mandatory fields and dump those non-queryable fields into the initial task description or such. (What to do with the queryable/machine-readable fields depends on the previous items.) This topic feels out of scope for this task.

Anybody willing / having time to comment on my requests in #1 and #2 in T92708#1418721 ?

@Aklapper I believe that @KLans_WMF wanted to have a hangout conversation on this to clarify. That is probably the best way to bring up your points and have them clarified. I'd be more than willing to join and discus as well.

If tags were free form and anyone could new tags, then it might be a solution for some of the items on this ticket (OS versions, platform), but that just doesn't seem to be how the organization ints to manage tags

I would wholeheartedly support making tags a free-for-all. I don't really see the need for restricting them.

I wouldn't oppose creating projects of the type "tag" (like "OS-Windows-10", "OS-Android-7" etc) as we already have a bunch of Browser-Support-* tags for a bunch of browsers.

The question is how to make reporters aware of setting these tags (not too many people read https://www.mediawiki.org/wiki/How_to_report_a_bug ) and describing when they should be set (as a reporter, always adding a Browser tag when the issue was only tested with a single browser and even if the issue is unrelated to the actual browser used by the reporter pollutes the tag and makes it less useful to filter by).

Still we would run into https://secure.phabricator.com/T4863 (get other projects like such tags shown in cards on workboards) which we (as in WMF) should probably try to push more, in upstream. Somehow™.

And in general I can only repeat that I don't believe in enforcing filling out mandatory fields - based on experience, a number of reporters would set random values or not report a task at all (too complicated; takes too much time). See my previous comments above.

It probably wouldn't be difficult to show tags in workboards, the problem is the space constraint. Maybe we could show the tags as just icons and then expand them to show the actual tag on hover. That seems minimally useful though since you wouldn't be able to quickly scan a column and see which tasks have which tags.

Fjalapeno renamed this task from Add fields to support bug related information for Wikimedia Apps team to Add fields to support bug specific information.Sep 14 2015, 7:44 PM

This is far more reasonable now that phabricator supports custom forms. We can fairly easily make custom task types with their own custom fields, however, I don't want to open a can of worms here - custom fields aren't free, they require setup, maintenance and operational overhead.

Lots of custom forms and custom fields will likely have runtime overhead as well.

I'm not sure if it's related to this, but I'm not able to close T176298; there's no status field. (It's the first time I've seen a 'bug' task, so perhaps I'm missing something; maybe only the assignee can close it or something?)

A collateral comment, once this new task type is reliable, I would put it the first in the list when "Creating a task". Less experienced Phabricator users are likely to come here to report a bug.

I'm not sure if it's related to this, but I'm not able to close T176298; there's no status field. (It's the first time I've seen a 'bug' task, so perhaps I'm missing something; maybe only the assignee can close it or something?)

This problem should be resolved now.

rPHAB1a265de83e15: Change default field visibility to hidden significantly eases one of the major difficulties I was having with adding custom fields. Previously it was very tedious to add new fields because I would have to go back and edit every custom form to hide the fields where they should not be used (so each new field would require editing ~20 forms, one by one, manually)

Once that commit is deployed I can create any custom fields you need for the "Bug" task type.

Upstream just made it possible to edit the type of a task! This is really the last major annoyance with task types. They've also made it possible to bulk-edit them with the batch editor.

Could someone please explain what exactly is left in this task? (@Fjalapeno?)
Nowadays we got Forms and task subtypes.

If something is left, please provide a specific example use case. Thanks!

If something is left, please provide a specific example use case. Thanks!

If Phabricator Forms are not sufficient, please set the status of this report back to "Open" via the Add Action...Change Status dropdown and elaborate. Thanks!