@ataiasreis has joined the channel
@j.wenjiao has joined the channel
@binarybusiness has joined the channel
@florence does raco cover exit with a nonzero status code? I’m wondering how I can avoid running all my tests twice in CI
@notjack raco cover does do a non-zero exit code on test failure (I think) but it doesn’t have the same notion of failure as raco test. I don’t think I treats stderr as failure. Maybe it should.
@florence I think ideally raco test would provide a way for packages to declare they should be instrumented somehow when being tested, and cover could use that to generate the coverage map. Then the raco cover command would just take the map and upload it or summarize it.
That way it’s not possible for raco test and raco cover to have different notions of test failure
I’m not sure about the implications of doing that… Cover, for instance, is not compatible with running tests in parallel. Also I’m suspicious of this because cover can change a programs behavior
That being said, a unified CI interface would be reaalllyyy nice
@florence maybe combined with an info field to control whether tests run in parallel? or something like that but more general
what sort of programs does cover change test behavior in?
There are a few behavior changes that i don’t understand and are probably bugs, but in general it changes the runtime of programs—both because of all the extra code and the inserted set!s probably mess with the optimizer—which can changes tests with timeouts or might change race condition behavior. It also greatly increases the memory used, which has caused intermittent OOM errors in the past.
Also I’m not sure what effect cover will have on reflection now that I think about it…
Also module loading can deadlock if you change the logger before a dynamic require….
@florence does cover only need to install errortrace during compilation and instrument running that compiled code or does it need to change current-compile in other ways too?
Neither? Cover uses errortrace’s syntax traversal stuff to apply its own annotation function.
ohhh
It does a bunch of stuff differently: it sets up cross phase annotations, it handles #:cross-phase-persistant modules, it uses a different internal representation of coverage to handle multifile coverage more efficiently, etc. It does wind up reimplementing a bunch of stuff in errortrace though but I’m not sure how to get around that
@florence thanks, that clears up my understanding a lot. I’ve been thinking about testing and CI stuff a lot lately so let’s talk more at racketcon, if you like :)
@notjack yes! I thinking about working on the cover issue you logged as a hackathon project
Which one? :P
The most recent one
The other ones are… Hard…
Oh! Yes that’s a great hackathon project and its something I definitely want to do
I’d like to combine that with two testing packages I have to do some neat stuff
@lexi.lambda That sort of fits my theory that, based on observable work output, Matthew is actually identical quadruplets.