Page MenuHomePhabricator

Build a background service that implements a queue for saving pages to disk.
Closed, ResolvedPublic5 Estimated Story Points

Description

Right now, the way we save a page is by starting a one-off thread with a specific title to be saved. This will no longer be practical when we implement synchronized collections, and it's already no longer practical for users who refresh a large number of saved pages simultaneously.

The proposed solution is to create a Service that runs in the background, and checks for saved page titles that are "dirty" and need to be written to the filesystem. For example, under our current implementation of saved pages, the solution would be:

  • When the user saves a page, add the title to the db (and nothing else).
  • The Service does a periodic check of the db, and checks whether each title entry is present on disk.
  • For every title that isn't present on disk, the Service queues it up to be fetched and written.
  • (optional) When the user saves a page, or updates a collection in the future, we can kick the dog in the Service, and make it fetch the new saved page(s) as soon as possible.

Event Timeline

Dbrant raised the priority of this task from to Needs Triage.
Dbrant updated the task description. (Show Details)
Dbrant moved this task to Next Sprint on the Wikipedia-Android-App-Backlog board.
Dbrant subscribed.
Dbrant moved this task from Doing to To Do on the Mobile-App-Android-Sprint-76-Osmium board.

Change 271925 had a related patch set uploaded (by Mholloway):
WIP: Implement background saved page syncing

https://gerrit.wikimedia.org/r/271925

Sn1per subscribed.

Sorry, my phone screen glitched. I don't have access to a PC atm so I can't fix the accidental drags, apologies :/

OK, the newest version of this is up. As mentioned during standup I've moved to synchronous Retrofit calls and done away with a lot of the AsyncTask stuff. Note that it's a stub implementation meant to integrate with a DAO and so won't do anything as currently written.

From my back-and-forth with @Niedzielski in Gerrit this task has diverged quite a bit from what's written in the description; specifically, the DAO will be handling a lot of the bookkeeping. Can we all agree on a concrete set of acceptance criteria so this task can stop haunting our sprints? The lack of any is the main reason I haven't been actively working on this.

Is this something you could exercise with Mockito? Maybe build the expected Reading List page DAO interface and mock it. I think you only need three methods:

@NonNull Collection<ReadingListPage> startTransaction();
void completeTransaction(@NonNull ReadingListPage page);
void failTransaction(@NonNull Collection<ReadingListPage> pages)

@Niedzielski Looking at the Mockito docs now. What did you have in mind for the ReadingListPage class? A subclass of PageTitle, perhaps?

@Mholloway, I think PageTitle has strong MW API ties so I wouldn't recommend that. Here's about what I have now:

public class ReadingListPage {
    @NonNull private final String key;
    @NonNull private final Site site;
    @NonNull private final Namespace namespace;
    @NonNull private final String title;
    private final int pageId;
    private final long pageRev;
    private final long mtime;
    private final long atime;
    @Nullable private final String thumbnail;
    @NonNull private final Set<String> listKeys;

    @NonNull public String key() {
        return key;
    }

    @NonNull public Site site() {
        return site;
    }

    @NonNull public Namespace namespace() {
        return namespace;
    }

    @NonNull public String title() {
        return title;
    }

    public int pageId() {
        return pageId;
    }

    public long pageRev() {
        return pageRev;
    }

    public long mtime() {
        return mtime;
    }

    public long atime() {
        return atime;
    }

    @Nullable public String thumbnail() {
        return thumbnail;
    }

    @NonNull public Set<String> listKeys() {
        return Collections.unmodifiableSet(listKeys);
    }
}

*This might change :)

Not a great example, but the only Mockito usage in the code base I know of is in DefaultAvPlayerTest.

Change 271925 merged by jenkins-bot:
Implement background saved page syncing

https://gerrit.wikimedia.org/r/271925

Does this need to be checked using exploratory testing for general function?

Hi @Nicholas.tsg, I've updated the Android Wikipedia App regression testing spreadsheet to reflect the addition of reading lists and the removal of saved pages. Test cases 9-13 from the updated spreadsheet are sufficient QA testing for this task. Feel free to ping me on IRC or here if you have any questions.

Testing on Android 4.4.2 (Samsung SM-T230NU) and Wikipedia app 2.1.144-alpha-2016-05-10. I took instructions from @Mholloway above to test Saved pages/reading lists with and without Flight mode, and results are as expected. So this is fixed.