Page MenuHomePhabricator

Test performance: Consider whether running many tests at once affects the accuracy of performance metadata
Open, LowPublic

Description

Description

As discussed in T363908, in certain situations many tests can be invoked at once (e.g., when a function is modified, all tests for that function will be invoked thereafter). Might the flurry of execution activity substantially affect the accuracy of performance metadata (particularly metadata for duration), and thus affect the validity of implementation ordering choices based on that metadata? If so, is there a good way to space out the invocation of the tests? Would it be worthwhile to at least ensure that the currently connected tests get run independently of the disconnected tests?

Desired behavior/Acceptance criteria

  • ...

Completion checklist

Event Timeline

Some of already occurs issues are:

  • Rate limit - we have a rate limit on function invocation and just viewing some function pages (which run all tests, connected and unconnected, for all implementation, connected and unconnected) may hit this rate limit.
  • When viewing a page tests are invoked via synchronous web requests without using any job system, nor deduplication. First, expensive sync web requests on page view is already an antipattern. Second, user can not immediately see the result of a test, not even a cached one. Third, since request are synchronous, users with unstable internet connection may not see the correct result. Fourth, running tests on every page view of functions, without any backend job system nor deduplication, is prone to DoS or DDoS.