[Feature Request] CI Improvement

[byronz]

The CI job for dash is still about ~20 minutes, we would like to speed up the integration by enabling the headless mode (the race condition has a lot of hardcode sleep, so by adding headless in CI job, we can reduce about ~1 minute, but not by a factor of 50%), and redesign the workflow in a parallel run flavor.

[Marc-Andre-Rivet]

There seems to be two different goals in this issue / could be broken up into two?

(1) optimizing the tests so that they run faster (headless, etc.)
(2) reorganizing the tests structure so that tests significant in context A are run with A, not with B.

@byronz What type of test would you see as generally qualifying for a promotion from a repo test to a dash test? My concern here is if we end up moving tests that could identify issues in the repo but do not anymore because they've been moved to dash.

How does this approach compare to say, identify a subset of tests in the repos that would get tagged as dash-sanity (or whatever) -- with this subset of tests from the repos being run as part of the dash CI?

@byronz @alexcjohnson This request makes me wonder how we feel about the dash+renderer monorepo after a few months and if folding in additional component repos would make sense at this point. With the proper commit triggers / filters we could also gain the same benefits as described in moving the tests above.

[alexcjohnson]

(1) optimizing the tests so that they run faster (headless, etc.)

Short-term this is the more important one to me. We'll all be more productive if tests are done in 10 minutes rather than 20. That said if it takes a week of time to make that happen, that's too much for right now.

(2) reorganizing the tests structure so that tests significant in context A are run with A, not with B.

From our earlier conversations I didn't think we were actually planning to move any tests, rather that we'd tag a subset of tests that live in the component repos and dashR, and run these on CI for the dash repo. I'd propose that we start by identifying 2 or 3 key tests from each component repo and tag them, then make a new job in the dash CI config that runs all of these. Then as time goes on if we see value in this, or if we see other tests breaking due to changes in this repo, we can add more.

I don't see a whole lot of need to go the other way, running dash tests from the component repos (or component tests in other component repos) so I'd hold off adding that until we see a concrete benefit.

makes me wonder how we feel about the dash+renderer monorepo after a few months and if folding in additional component repos would make sense at this point.

Particularly in context of additional back-end languages, I feel like we should keep the current structure. Also a monorepo would exacerbate the test length problem. Running a few select tests across repos feels to me like the best of both worlds.

[Marc-Andre-Rivet]

Alright. Please break this up into 2 issues and let's try to get a sense of the time required for each. Maybe also identifying key tests.

((1) optimizing the tests so that they run faster (headless, etc.)) -- Short-term this is the more important one to me. We'll all be more productive if tests are done in 10 minutes rather than 20.

Agreed.

I'd propose that we start by identifying 2 or 3 key tests from each component repo and tag them, then make a new job in the dash CI config that runs all of these. Then as time goes on if we see value in this, or if we see other tests breaking due to changes in this repo, we can add more.

Sounds good. With the repos currently running against the head and most being updated (semi-)regularly, I think that I've only encountered one situation since the beginning of the year where the tests failed when all builds were triggered just prior to a release. That one time was painful but.. compared with the previous state of affairs, this seems to indicate that we've mitigated most of the issues we can encounter through this imperfect security net.

[byronz]

I will reduce the scope of the on-going PR to only optimizing the CI duration.

I now have a working prototype to have integration tests run on multiple containers, by using the circleci container, it's smart enough to equally select the test subset by test duration.

will target to reduce the whole duration from 20 -> 5-7 minutes