Fri, Mar 16
It's much easier to change Phlogiston than the dump, because it runs
outside of any security profile and doesn't require an admin. so I would
vote to do all that in phlogiston.
If you can throw in the new column, I can modify the script to work with
either kind of data.
Thu, Mar 15
For an experiment, I tweaked the load so that it parsed all of the "bad" transactions, treating the list of PHIDs as a list of projects the task belonged to at that date (which was true for at least the sample task 187512). Phlogiston ran to completion and generated a report on dev; I used Reading-Web as the sample:
You probably can't find it in the code because ... it's temporary error-handling code that I added to the dev server and never committed. On production, the load log looks like:
Wed, Mar 14
Steps to Reproduce:
- Download a current dump from https://docs.google.com/spreadsheets/d/1yVx3Fmmxccu2noU18IBztcKs88TDRt2E57XGAFYLb6c/edit#gid=327722724
- Run a standard Phlogiston data loading and reporting run. Alternately, examine the dump file as described below.
Tue, Mar 13
What kind of information would be helpful? I could examine the dump file to give more information over what's above. I could look to see how the apparent 100,000 cutoff manifests in the dump. I could try to define a filter that would cut the number of tasks for reporting below 100,000 to see if the problem then (temporarily) goes away.
Fri, Mar 9
cancelled - no demand.
done in other work threads
Wed, Mar 7
Closed in the absensce of a Team Practices Group.
Fri, Mar 2
Wed, Feb 28
Working with Lynette on Phab basics, we edited the title for clarity and assigned it to Rachel.
All of the current reports are failing every day because they basically have no data and Phlogiston (or R) doesn't have enough error handling to continue functioning. The chart above was captured two weeks after the apparent change but before Phlogiston gave up; you can see that the task counts dropped from ~300 to maybe two or three tasks. When I reported the task count of 179,180, that was a literal count of objects (
) so there are still >100,000 tasks in the dump, but most of the edge data seems to be missing. It's hard to be more precise because I don't have a "before" dump handy, but the first thing that caused Phlogiston to outright crash was that some of the transactional core:edge data started showing up empty. Is that helpful or should I try to provide more precise information about what's missing in the dump?
Tue, Feb 27
I still see 179180 tasks in the dump, which is presumably all of the public tasks. What does seem to be missing in the dumps is some, but not all, of the core:edge transaction content. For the missing ones, there's still a record for the transaction, but the content of the transaction is missing. I haven't spotted other things specifically missing from the dump yet.
Mon, Feb 26
Have dug in deeper; pretty sure this is a change in the data files provided to Phlogiston, and not a code problem introduced recently in Phlogiston alone.
Fri, Feb 23
I also checked in new code on Feb 15, which is a superficial clue. I will continue investigating. Do you think the schema change affected the dumps?
A preliminary inspection suggests that the dumps are still coming, still fresh, but missing all edges (aka which tasks belong to which project). While I investigate further, a question: Did anything change in the dumps around Feb 15th?
Tue, Feb 20
Feb 9 2018
Test task: T182319. Status is blank, but in Phab it is open.
Feb 8 2018
Pivoting since Oct 2018 date is postponed.
Test case T182639 had no parent in get_status_report. This was due to a design conflict. The stored procedure determines parenthood by checking the maniphest_blocked table for the final date of the report. However, the maniphest_blocked table, which is regenerated with each daily load, does not reconstruct historical blocking relationships from transactions. Instead, it gets the current parent state from the task data, and loads that into maniphest_blocked with the load date. So these relationships will only work in Status Reports if the report is run on the most recent day. This was fixed by having the stored procedure get_status_report not filter by date, so that it does get the parent match.
Feb 7 2018
More test data: T182197 has Scope = Unknown bug in report but it does have a status in the report.
Feb 2 2018
Feb 1 2018
Jan 30 2018
Jan 29 2018
Jan 23 2018
Since some sessions are much longer than others (and afternoon sessions are more likely to get impinged upon), we should try to match longer topics to longer sessions.
Jan 20 2018
Jan 17 2018
Jan 16 2018
Jan 8 2018
Jan 4 2018
Jan 3 2018
Dec 22 2017
Thanks. I'll try watching and see how it works. I think 1) I didn't understand about watching, and 2), I'm concerned about spam from watching the whole project as opposed to having some default subscribers that can be edited.
Dec 21 2017
- figure out what time is supposedly reserved for departments. The current published draft (https://office.wikimedia.org/wiki/All_Hands/2018/Schedule) is fully booked; which 3 hours did the C-team have in mind to hand back to Departments? Friday afternoon?
- contact All-hands organizers (Lauren, Danny, Moriel, according to https://office.wikimedia.org/wiki/All_Hands/2018) and see if Victoria + Toby is enough clout to get reserved rooms
- if not, look for other space nearby in the Presidio.
Dec 20 2017
Dec 19 2017
Dec 7 2017
Dec 5 2017
Nov 29 2017
first draft done; followup work will be tracked elsewhere.
Mostly done/partly obsolete given changes in Annual Plan planning.