Create conduit method to query the feed and return records with relevant details populated instead of just a bunch of phids
Closed, DeclinedPublic
Actions

Assigned To

None

Authored By

	• mmodell
	Jan 12 2016, 9:16 PM

Description

Currently wikibugs has to do a lot of separate api queries in order to assemble the data needed to provide irc notifications.

The current situation puts a whole lot of load on phabricator and it makes wikibugs code overly complicated. This won't scale forever.

Additionally, there aren't any differential methods to get transaction data the way we are doing for maniphest transactions, so it's not possible for wikibugs to report on differential changes.

I want to build a custom api method that returns the feed with transaction details populated, this way a single api call will return all of the data that wikibugs needs to do it's thing.

Related Objects
Search...

Status	Subtype	Assigned	Task
Declined		None	T116330 Broadcast Differential activity to IRC
Declined		None	T123416 Phabricator does not provide an API to get Differential transaction data, similar to maniphest.gettasktransactions
Resolved	Goal	bd808	T1175 Get rid of screen scraping in Wikibugs
Declined		None	T123417 Create conduit method to query the feed and return records with relevant details populated instead of just a bunch of phids

Event Timeline

• mmodell created this task.Jan 12 2016, 9:16 PM

• mmodell raised the priority of this task from to Needs Triage.

• mmodell updated the task description. (Show Details)

• mmodell added projects: Differential, Gerrit-Migration, Phabricator.

• mmodell subscribed.

Restricted Application added subscribers: StudiesWorld, Luke081515, scfc, Aklapper. · View Herald TranscriptJan 12 2016, 9:16 PM

• mmodell claimed this task.Jan 12 2016, 9:17 PM

• mmodell triaged this task as High priority.

• mmodell added a parent task: T123416: Phabricator does not provide an API to get Differential transaction data, similar to maniphest.gettasktransactions.

greg added a project: OKR-Work.Jan 13 2016, 9:01 PM

greg moved this task from To Triage to Documentation on the Gerrit-Migration board.Jan 20 2016, 12:41 AM

Luke081515 moved this task from To Triage to Doing on the Phabricator board.Jan 21 2016, 9:50 PM

• mmodell mentioned this in T108199: Evaluate Phabricator for its ability to export and import data.Feb 10 2016, 12:24 AM

• mmodell lowered the priority of this task from High to Medium.Feb 15 2016, 3:23 PM

• mmodell mentioned this in T129246: Wikibugs links sometimes to the creation event, not to the mentioned comment.Mar 9 2016, 12:39 AM

greg moved this task from Documentation to Software Changes on the Gerrit-Migration board.Mar 15 2016, 10:59 PM

greg moved this task from To Triage to Backlog on the Differential board.May 12 2016, 10:37 PM

Restricted Application added a subscriber: TerraCodes. · View Herald TranscriptMay 12 2016, 10:37 PM

• mmodell mentioned this in T135327: Decide Phabricator improvements to be funded by WMF Technical Collaboration.May 24 2016, 7:34 AM

• mmodell added a parent task: T1175: Get rid of screen scraping in Wikibugs.

• mmodell mentioned this in T1175: Get rid of screen scraping in Wikibugs.

• RobLa-WMF subscribed.May 24 2016, 6:50 PM

Aklapper moved this task from Doing to Misc on the Phabricator board.Oct 7 2016, 6:41 PM

• mmodell awarded a token.Apr 12 2017, 6:18 PM

• mmodell changed the task status from Open to Stalled.May 12 2017, 1:00 AM

• mmodell added a project: Wikibugs.

greg added a project: Release-Engineering-Team (Kanban).May 16 2017, 3:55 PM

greg removed • mmodell as the assignee of this task.May 17 2017, 2:00 PM

greg edited projects, added Release-Engineering-Team; removed Release-Engineering-Team (Kanban).

greg moved this task from INBOX to Backlog on the Release-Engineering-Team board.May 17 2017, 2:13 PM

greg edited projects, added Release-Engineering-Team (Backlog); removed Release-Engineering-Team.

there aren't any differential methods to get transaction data the way we are doing for maniphest transactions

https://secure.phabricator.com/D18466 and related changes in https://secure.phabricator.com/T5873#231369 might set the basics for this if I get it right (not sure)

@JAufrecht : Perhaps eventually phlogiston could pull data via conduit instead of relying on nightly db dumps?

In the long run, yes, either pulling data from conduit or being replaced by internal reporting features. In the short term, I have heard that Conduit is not ready for use?

"Conduit" is just the Phab app name for the code that manages the API. It is/has been ready for use.

• mmodell claimed this task.Sep 13 2017, 9:52 PM

Restricted Application edited projects, added Release-Engineering-Team (Kanban); removed Release-Engineering-Team (Backlog). · View Herald TranscriptSep 13 2017, 9:52 PM

• mmodell changed the task status from Stalled to Open.Sep 13 2017, 9:52 PM

Added T175872: Pull data from Conduit instead of custom database dump for Phlogiston to switch to Conduit. @ksmith

Aklapper mentioned this in T175872: Pull data from Conduit instead of custom database dump.Sep 14 2017, 9:47 AM

• mmodell moved this task from Backlog to In-progress on the Release-Engineering-Team (Kanban) board.Sep 25 2017, 4:30 PM

I plan to work on this soon, hopefully sometime this quarter.

• mmodell moved this task from In-progress to Backlog on the Release-Engineering-Team (Kanban) board.Oct 11 2017, 4:38 PM

• mmodell changed the task status from Open to Stalled.Nov 27 2017, 5:52 PM

• mmodell removed • mmodell as the assignee of this task.

• mmodell lowered the priority of this task from Medium to Low.

• mmodell removed a project: Release-Engineering-Team (Kanban).Apr 23 2018, 4:30 PM

Aklapper moved this task from Misc to Needs code (in Phab or bot) on the Phabricator board.Aug 29 2018, 6:19 PM

A-NEUN subscribed.Jan 24 2020, 11:49 PM

A-NEUN unsubscribed.Jan 26 2020, 6:38 PM

I don't see who or what (task?) exactly this task is stalled on ("If a report is waiting for further input (e.g. from its reporter or a third party) and can currently not be acted on"). Hence resetting task status.

Aklapper mentioned this in T258305: Rename #Operations Phab project to #SRE.Jan 7 2021, 10:28 AM

Aklapper moved this task from Backlog to To Triage on the Differential board.May 30 2021, 9:40 AM

CC'ing @valhallasw as wikibugs' T1175: Get rid of screen scraping in Wikibugs is listed as a parent task (and probably intended to be main consumer?).

To get the change history of this very task, I currently can

echo '{"filterPHIDs":["PHID-TASK-k5kvgx5ch2dg3ez4jcbl"]}' | /var/www/html/phorge/arcanist/bin/arc call-conduit --conduit-uri https://phabricator.wikimedia.org --conduit-token "api-xxxxxxxxx" feed.query --
echo '{"ids":["123417"]}' | /var/www/html/phorge/arcanist/bin/arc call-conduit --conduit-uri https://phabricator.wikimedia.org --conduit-token "api-xxxxxxxx" maniphest.gettasktransactions --

Could someone please elaborate what exactly is wanted / missing here? TIA!

Wikibugs uses the task page html for

getting the correct url anchor for a given change - https://github.com/wikimedia/labs-tools-wikibugs2/blob/master/wikibugs.py#L178C6-L178C6
getting the tags and their properties -

https://github.com/wikimedia/labs-tools-wikibugs2/blob/master/wikibugs.py#L140

I think the other issue this aimed to address was Wikibugs needing to call maniphest.info for every change (rather than feed.query just containing the relevant details). The "this won't scale forever" seems to not have been a major issue, though :-)

It's possible that the underlying Conduit APIs have gained the required functionality in the last 10 years. At the same time, I don't have a particular interest in resolving T1175 given that the screen scraping worked pretty reliably over the years, nothing is going to change in Phabricator and it's all going over to GitLab at some point.

Thanks *a lot* for elaborating! In that case I'm going to decline this task for the time being. Please reopen if a Wikimedia use case shows up but if there isn't this should go to upstream instead.

I stumbled upon rPHAB586aaa547ade5bf97fa02e2c8e11511b0387b737 which refers to this task.
I do not know if that custom downstream code change ("add html+data for feed.query conduit api") in our Phabricator instance is still needed or not.
Help welcome.

Aklapper mentioned this in T354132: Revert unused downstream rPHAB586aaa547ade5bf97fa02e2c8e11511b0387b737.Dec 31 2023, 11:33 AM

Create conduit method to query the feed and return records with relevant details populated instead of just a bunch of phidsClosed, DeclinedPublicActions

Description

Related ObjectsSearch...

Event Timeline

Create conduit method to query the feed and return records with relevant details populated instead of just a bunch of phids
Closed, DeclinedPublic
Actions

Related Objects
Search...