VE Phlogiston report missing a lot of data after 2016-04-27 Phab upgrade
Closed, ResolvedPublic8 Estimated Story Points
Actions

Assigned To

Authored By

	• JAufrecht
	Apr 29 2016, 5:08 PM

Description

Steps to reproduce

Wait for Phabricator to be upgraded ca. 2016-04-27
Wait for Phlogiston to run 2016-04-28
Observe VE Burnup report

Actual results:
Many of the categories are missing data. In particular, Interrupt shows as complete, with zero tasks (open or resolved).

Expected Results
Interrupt has hundreds of tasks, open and resolved.

Differential Diagnosis:

Collab report seems unaffected.
VE partially processed data seems correct - there are many thousands of task-days of Interrupt records. After recategorization, however, there are only 1000 task-days; it looks like the rest of records get incorrectly categorized in general backlog.
No recent changes to Phlogiston code or VE report configuration that seem revelant to this.

Details

	Subject	Repo	Branch	Lines +/-
	Update schema: TYPE_PROJECT_COLUMN is now simply TYPE_COLUMNS	phabricator/tools	master	+1 -1

Customize query in gerrit

Revisions and Commits

rPHDEP Phabricator Deployment
	Audited	rPHDEPb18ba1c67f3d Bugfixes in phabricator tools and misc extensions.

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		• JAufrecht	T125698 Phabricator dump used by Phlogiston should be updated when Phabricator is upgraded
		Resolved		• JAufrecht	T133994 VE Phlogiston report missing a lot of data after 2016-04-27 Phab upgrade

Event Timeline

• JAufrecht created this task.Apr 29 2016, 5:08 PM

Restricted Application added subscribers: TerraCodes, Urbanecm. · View Herald TranscriptApr 29 2016, 5:08 PM

• JAufrecht updated the task description. (Show Details)Apr 29 2016, 5:08 PM

Data is correct through recategorization, but breaks on retroactive categorization. This code:

UPDATE task_history_recat t
   SET category = t0.category
  FROM task_history_recat t0
 WHERE t0.date = (SELECT MAX(date)
                    FROM task_history_recat
                   WHERE scope = scope_prefix)
   AND t0.scope = scope_prefix
   AND t.scope = scope_prefix
   AND t0.id = t.id;

Before this,

phab=# select category, maint_type, count(*) from task_history_recat where scope = 've' group by category, maint_ty
pe order by category;
          category          | maint_type | count  
----------------------------+------------+--------
 Arabic                     | Core       |      1
 Arabic                     | Strategic  |    142
 Backlog                    | Strategic  |   5209
 Backlog                    | Core       | 509382
[...]

 Interrupt                  | Core       |  33521

After this,

phab=# select category, maint_type, count(*) from task_history_recat where scope = 've' group by category, maint_ty
pe order by category;
          category          | maint_type | count  
----------------------------+------------+--------
 Arabic                     | Core       |      1
 Arabic                     | Strategic  |    142
 Backlog                    | Strategic  |  31002
 Backlog                    | Core       | 539016
 [...]
 Interrupt                  | Core       |   1002`

This affects most or all of the categories, although not consistently. Wikivoyages category is unchanged; Interrupt shrinks by tens of thousands and Backlog grows by a similar amount; jQueryIME grows; etc etc.

New information:

Phlog ran an incremental update, so only data for yesterday was imported. Phlogiston for VE uses retroactive categorization. With it turned on, data is scrambled as described. With it turned off, report is normal; turned on, it retroactively incorrectly recategorizes most data.
Looking at a single affected task:

select * from task_history where id = 100275 order by date;

 scope |        date         |   id   |                             title                              |   status   |   project    | projectcolumn  | points | maint_type | priority | category_title 
-------+---------------------+--------+----------------------------------------------------------------+------------+--------------+----------------+--------+------------+----------+----------------
 ve    | 2015-05-26 00:00:00 | 100275 | Highlights don't adequately block link hover effects in Chrome | "open"     | VisualEditor |                |      8 |            | 90       | 
[...]
 ve    | 2016-04-27 00:00:00 | 100275 | Highlights don't adequately block link hover effects in Chrome | "open"     | VisualEditor | TR0: Interrupt |      8 |            | 50       | 
 ve    | 2016-04-28 00:00:00 | 100275 | Highlights don't adequately block link hover effects in Chrome | "open"     | VisualEditor | TR0: Interrupt |      8 |            | 50       | 
 ve    | 2016-04-29 00:00:00 | 100275 | Highlights don't adequately block link hover effects in Chrome | "open"     | VisualEditor |                |      8 |            | 50       |

Tentative hypothesis: Projectcolumns changed in Phabricator's data model, and are not read (or maybe even available?) in the dump, so yesterday's dump and load results in tasks without projectcolumns. This would break VE, which uses projectcolumns in a lot of categorization, and barely affect other teams.

indeed projectcolumns changed. I'll try to figure out how to fix the dump script.

@JAufrecht: This patch should fix it. Note: the TYPE_PROJECTCOLUMN transaction type was renamed to TYPE_COLUMNS

• mmodell added a project: Patch-For-Review.Apr 29 2016, 6:29 PM

Change 286195 had a related patch set uploaded (by Jforrester):
Update schema: TYPE_PROJECT_COLUMN is now simply TYPE_COLUMNS

https://gerrit.wikimedia.org/r/286195

Change 286195 merged by 20after4:
Update schema: TYPE_PROJECT_COLUMN is now simply TYPE_COLUMNS

https://gerrit.wikimedia.org/r/286195

Merged and deployed the new dump script. I'm running it manually now, not sure how long that will take to finish.

Thank you!

• JAufrecht renamed this task from VE Phlogiston report missing a lot of data after Phab upgrade to VE Phlogiston report missing a lot of data after 2016-04-27 Phab upgrade.Apr 29 2016, 7:50 PM

• JAufrecht added a project: User-JAufrecht.Apr 29 2016, 8:02 PM

The dump is complete. @JAufrecht can you confirm that the missing data has been restored? Note: the transaction is renamed so you'll likely have to look for COLUMNS instead of PROJECTCOLUMN

I'm not seeing any change to the dump file:

openssl s_client -connect dumps.wikimedia.org:443
[...]
HEAD /other/misc/phabricator_public.dump HTTP/1.1
host: dumps.wikimedia.org

HTTP/1.1 200 OK
Server: nginx/1.9.4
Date: Fri, 29 Apr 2016 22:43:27 GMT
Content-Type: application/octet-stream
Content-Length: 456641945
Last-Modified: Fri, 29 Apr 2016 02:17:03 GMT

Did the filename change?

I downloaded the file and it's unchanged from the previous download (from last night).

• JAufrecht reassigned this task from • JAufrecht to • mmodell.Apr 29 2016, 10:53 PM

@JAufrecht: It should be updated now, can you check once more?

The columns field is present, but empty in spot checks. Also, I made the one-word change from projectcolumn to columns over the weekend and my test case (VE's backlog) is still missing column data after several nightly runs.

>>> data['task']['100275']['transactions'].keys()                                          
dict_keys(['core:edge', 'reassign', 'columns', 'priority', 'status'])
>>> data['task']['100275']['transactions']['columns']
>>> data['task']['119098']['transactions']['columns']                                      
>>> data['task']['100122']['transactions']['columns']                                      
>>>

phab=# select count(*) from maniphest_transaction where transaction_type = 'columns';
 count 
-------
     0
(1 row)

• mmodell added a commit: rPHDEPb18ba1c67f3d: Bugfixes in phabricator tools and misc extensions..May 2 2016, 7:14 PM

@JAufrecht I'm sorry, this was my mistake. The transaction is core:columns not columns. It should be in the next dump!

Thanks, will check for it tonight.

• mmodell lowered the priority of this task from Unbreak Now! to High.May 2 2016, 9:16 PM

Assign it back to me if there is still a problem I can help with.

• JAufrecht moved this task from Technical Debt to Setting Up Reports on the Phlogiston board.May 2 2016, 9:58 PM

• JAufrecht edited projects, added Phlogiston (Setting Up Reports); removed Phlogiston (Technical Debt).

I see the core:columns data in the dump; processing now.

Ran successfully, charts look okay. Thanks!

• JAufrecht closed this task as Resolved.May 3 2016, 7:15 PM

VE Phlogiston report missing a lot of data after 2016-04-27 Phab upgradeClosed, ResolvedPublic8 Estimated Story PointsActions