Specifically written as part of T433683: Make developer metrics related data sets available in the data lake.
As with the git one this description is initially written as an introduction to the dataset, and how it could work, but is naturally expected to change.

Purpose and scope

Collect activity on Wikimedia Phabricator tasks: task metadata and the transactions that record comments, status changes, assignments, title/description changes, and other task actions.
Task discovery is the index used to retrieve transactions, because the transaction API requires task IDs; transactions are the main activity dataset.
This is task-tracking activity, distinct from anything like code commits and code-review activity.

Sources and coverage

The source is https://phabricator.wikimedia.org. The collection targets every task visible to the collection credential, then obtains all visible transactions for each discovered task.

text
Include:
  All visible Maniphest tasks and all visible transactions on those tasks.

Exclude:
  Tasks and transactions not visible to the collection credential.
  Other Phabricator applications, such as repositories, diffs, paste, and wiki content.

The collection credential's visibility determines coverage. If access policy changes, the collection may need to be re-run or reconciled because previously inaccessible tasks may become visible.

Access requirements

The collector should retry transient service/transport errors, and reduce query size when the API rejects expensive searches.

Collection and resumption

Discovery

Use maniphest.search to page through visible tasks. Its purpose is to obtain task IDs: maniphest.gettasktransactions accepts task IDs and does not provide an all-Phabricator transaction feed. For the initial collection, order tasks oldest-first. For ongoing collection, query tasks whose modification time falls within a bounded time window, ordered by modification time, then retrieve transactions for each returned task.

Some task IDs may be restricted or otherwise invisible. If a cursor lands in a dense region of invisible tasks and the API cannot page past it, use targeted ID lookups to find the next visible task, while recording the resulting coverage limitation. Reduce page size when the API reports an overheated query.

Initial backfill

The backfill unit is a page of tasks, followed by transaction retrieval for those task IDs. Process task pages oldest-first; for every task, save its task metadata and all returned transactions. Persist the page cursor only after the records from that page have been written. The task list is therefore both a useful reference extract and the route by which the complete transaction history is reached.

Task metadata is retained as a task-level reference extract. Transactions are the canonical activity extract and are partitioned by their creation month. All transaction types returned by the API are retained.

Incremental collection and resumption

After the initial backfill completes, save a global modification-time watermark. Begin the first incremental pass slightly before that point (for example, seven days earlier), so activity that occurred while the backfill was finishing is collected again; de-duplication makes this overlap safe.

For each later run, fix a target end time at the start of the run and process the interval from the saved watermark to that target in small windows (for example, one day). In each window:

1. Search for tasks modified in that window.
2. Fetch transactions for every returned task ID.
3. Retain transactions created in or immediately adjacent to that window; earlier history was captured by the initial backfill and is ignored on this pass.
4. De-duplicate the retained transactions and write the updated task metadata.
5. Save the page cursor after each page; after the final page, save the next window start.

If the job stops partway through a window, it resumes using that window's fixed start/end and its saved page cursor. Only after every window up to the fixed target is complete does it advance the global watermark to the target. This prevents a moving “now” boundary from skipping changes and makes a repeated page or window harmless.

The task row key is task_id; the transaction row key is transaction_phid. A re-run may repeat a page or window but must not create duplicate keys. Tasks modified after first discovery should be upserted so task metadata remains current; transactions should be inserted/upserted by their immutable transaction PHID.

Schema and identity keys

Task records

Transaction records

Values that are structured in the API are serialised as JSON strings. A maximum field length should be set for large values, documented with the chosen deployment, so a very large task description does not break extraction.

Task titles, descriptions/field values, and comments can contain personal or sensitive information. Identity resolution and any public presentation of text must be handled separately from raw collection.

Representative data

This illustrative data is invented. Replace it with redacted rows from a read-only sample of the deployed collection.

csv
transaction_phid,task_id,transaction_id,transaction_type,published,author_phid,title,comments,old_value,new_value
PHID-XACT-example,12345,67890,comment,2024-01-02 03:04:05,PHID-USER-example,Commented on T12345,I can reproduce this.,,
PHID-XACT-example-status,12345,67891,status,2024-01-02 04:05:06,PHID-USER-example,Closed T12345,,open,resolved

DB access?

I believe all of this data ultimately lives in an SQL DB, so I imagine some sort of direct DB access might be possible if other teams agree to such things

Hooks?

Per webhooks coming up in the git ticket a little on hooks.
I think https://secure.phabricator.com/book/phabricator/article/webhooks/ would be the place to look but this 503s for me currently...
On archive.org https://web.archive.org/web/20250415071251/https://secure.phabricator.com/book/phabricator/article/webhooks/

Webhooks can be triggered in two ways:

Set the hook mode to Firehose. In this mode, your hook will be called for every event.
Set the hook mode to Enabled, then write Herald rules which use the Call webhooks action to choose when the hook is called. This allows you to choose a narrower range of events to be notified about.

I imagine firehose would be an option.

Request format appears to include everything that may be needed? as such, the rest of the content in this ticker might mainly be relevant to backfill?
Ultimatly, these 2 parts...

The object map describes the object which was edited.
The transactions list contains information about the actual changes which triggered the callback.

Limitations, risks, and open decisions

• Coverage is limited to tasks visible to the API credential. Restricted tasks can create gaps in otherwise sequential task-ID discovery.
• Perhaps this should be limited to just meta data, to avoid any content potentially linked to any security issues / things that should be protected that end up ingested.
  • Naturally the usecases for the data increase if we maintain old_value and new_value, comments and title for example.
• The API may reject expensive searches as overheated, particularly near dense regions of invisible tasks. Smaller pages and targeted ID probes mitigate this but may still leave a recorded gap.
• Task metadata changes over time. Decide whether the task extract retains only the current version or versions every change; transactions already provide the activity history.
• Decide the maximum retained length for comments and old/new values, and whether any text fields need redaction, restricted access, or exclusion.