Fix ci coverage & azure

[davidlatwe]

What's changed

This PR fix #558.

The code coverage submission was decoupled from maya test script, and provide a new scheme for future testing on multiple DCC tools or versions.

New code coverage scheme

1. The test script generates a .coverage data file after complete.
2. Rename the .coverage into .coverage.{subject}, where the subject could be DCC tool name and version.
3. After all test scripts completed, run coverage combine to combine all .coverage prefixed data file into one.
4. Submit to coveralls.

[tokejepsen]

Awesome stuff!

[davidlatwe]

About the error message which @tokejepsen mentioned in review:

$ coveralls
Submitting coverage to coveralls.io...
Could not submit coverage: 422 Client Error: Unprocessable Entity for url: https://coveralls.io/api/v1/jobs

This reminds me what happened in #494, looks like it was the same cause.

So the reason to that error was because the COVERALLS_REPO_TOKEN did not get passed, which I am sure it does exists in travis-ci setup.

Turns out the problem was in the Coveralls's python integration module.

I have changed to use coveralls-clients/coveralls-python (pip install coveralls) in commit b17b129, since the original integration module z4r/python-coveralls (pip install python-coveralls) has stopped updating for almost a year..

And with coveralls-clients/coveralls-python integration, it just won't get the coveralls repo token. Because I tried coveralls debug command in 1c83ce1 and the token (should be something like "repo_token": "[secret]") did not exists in the JOSN dump string in submission debug message.

Anyway, I changed back to the z4r/python-coveralls integration, which requires to pin coverage==4.5.4, but here's a new error message:

INFO:coveralls:422
INFO:coveralls:{"message":"Couldn't find a repository matching this job.","error":true}

Maybe it will be gone in the merge commit ? Not sure.

Or, maybe it's time to change to use codecov ? @mottosso

[mottosso]

Maybe it will be gone in the merge commit ? Not sure.

I saw the error, and had a look at coveralls.io, and indeed it does not exist. :O I think GitHub updated the permission features a while back, and it looks like Coveralls got left out. I've added it now and updated the repo token, and expect it will work. For both PRs and merges. Let's give it a try, and if not switch to something else.

[mottosso]

Well, it does work, but not really.

https://coveralls.io/builds/32424978

[davidlatwe]

Looks like there was a problem when combining coverage data

$ coverage combine
Coverage.py warning: Couldn't read data from '/home/travis/build/getavalon/core/.coverage.maya': UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf6 in position 106: invalid start byte
No usable data files

https://travis-ci.org/github/getavalon/core/jobs/713255428#L683

I guess.., it was because we are foced to use outdated coverage==4.5.4 because of we can only use python-coveralls to send coverage data to coveralls.io.

So many pitfall after Python 2.7 being rejected :(

Maybe codecov is the only way out for code coverage. 😭

[davidlatwe]

I guess.., it was because we are foced to use outdated coverage==4.5.4

Ah, sorry, maybe it's because I forgot to pin coverage==4.5.4 back in Dockerfile-maya2016. :S

Which makes the older version coverage out side of the container reading the coverage data that generated from newer version of coverage in side the Maya container.