@notjack mostly I wish that it was simpler to put together test suites

in TR, we can do pretty much what we need, but it’s complicated

@samth I’m curious: why is it complicated?

@zenspider why is which complicated? TR or rackunit?

I assume both at the same time? rackunit is pretty straightforward. I don’t do TR but I suspect that you don’t find it that complicated. :wink:

the TR test suite code is complicated because it does a lot of different stuff (load a lot of files dynamically as tests, run in parallel, etc)

rackunit is complicated to use when you want to do more than just use check-equal? etc at the top-level of your test module

that use case is very simple (which is great)

can you point me at any of the rackunit complication so I can wrap my head around it? I might be able to help out there a bit.

for example, the distinction between a test case and a test suite

I think this paragraph describes my unhappiness well:

“Test cases can themselves be grouped into test suites. A test suite can contain both test cases and test suites. Unlike a check or test case, a test suite is not immediately run. Instead use one of the functions described in User Interfaces or Programmatically Running Tests and Inspecting Results.”

yeah. I’m not entirely thrilled with that myself. IMHO any time you need to do anything more than just require something to run your tests you’ve failed.

I’m not sure how to fix that in a way that would make “normal” racket devs happy tho. I know how to do it to make me happy but that’s different. :slightly_smiling_face:

something that makes one person happy is better than making 0 people happy

In my opinion, the things I find the most frustrating about rackunit is that test failures are often remarkably useless and “checks” don’t compose at all.

I have spent more time than I would like trying to understand why two things rackunit claims aren’t equal actually aren’t equal.

And writing custom checks that produce decent error messages is a huge chore, mostly because you can’t really implement a new check in terms of an existing one and still produce good check failure messages.

My understanding is that rackunit was just schemeunit, rolled into the main distribution, but I am not sure it’s really the same quality and adheres to the same principles as other libraries that get distributed with racket. :/

@lexi.lambda I’m working on a library to improve rackunit error messages, especially when two data structures are “almost” the same: http://docs.racket-lang.org/expect/index.html?q=expect

It’s not ready for a 0.1 release yet though (mainly due to needing some rackunit features implemented so the check info doesn’t look awful)

@lexi.lambda I published check-sexp-equal for that reason… but since I can’t define any formatting it is still kinda hard to read

here’s an example: > (expect! '(10 12) (expect-list (expect-pred number?) (expect-pred string?))) expected a different kind of value subject: '(10 12) in: item at position 1 expected: value satisfying string? actual: 12

minitest (in ruby) does diffing on assert_equal failures and provides make_my_diffs_pretty to ensure that big things get printed structurally. makes finding the problems a breeze in comparison

It does it that way specifically to find inequalities deep in a nested sexp and it is really really quick to hone in on where the problem is. check-sexp-equal? has to embed #:new and #:old tags to show you where the differences are but the whole sexp is printed flat

just doing a pretty-print of the output would make it much better

@zenspider check-sexp-equal is very nice and was part of the inspiration for expect :)

@notjack I’m not a fan of expectations (at least from the BDD perspective). I imagine that your RHS blows up really quickly on anything non-trivial

thanks! if you have any insight on how to improve its output I’m all ears

there’s “expectation conversion” so that you can write something like (hash 'foo (list 1 2 (vector 'a 'b (expect-pred? number?)))) and it will convert it to an expectation that structurally matches things and reports errors in contexts

the above example I had didn’t do that because expect! does not yet convert its expectation argument automatically

(expect-equal? some-complex-data-with-hashes-and-lists-and-stuff) will do it though

They’re more like junit/hamcrest “matchers” than anything else, but I wanted a more noun-y name

I’m currently working on a test filtering feature for chk and rackunit. The idea is to be able to call raco test my/module -- args to forward and the test lib interprets args to forward as filters for test names, files, lines to run. I’m looking for feedback/feature requests based on the WIP documentation: https://cfinegan.github.io/chk-docs/chk.html#(part._.Filtering_.Tests_with_.Command-line_.Arguments)

Please let me know what you think. Rackunit docs are coming after I figure out what features are the most popular.

@cfinegan instead of relying on command line arguments, would it be cleaner for raco test to define some sort of “filters” concept and pass it along to test submodules? module-level filtering could work with a config submod in the test module

There’s always getenv plus the ability to define an env var on the command-line ¯_(ツ)_/¯

$ cat example.rkt
#lang racket/base

(module test racket/base
  (if (getenv "BIG_TEST")
      (println "Running big tests")
      (println "Skipping big tests")))

$ raco test example.rkt
raco test: (submod "example.rkt" test)
"Skipping big tests"

$ BIG_TEST=1 raco test example.rkt
raco test: (submod "example.rkt" test)
"Running big tests"

Just from the POV of “simplest thing that could possibly work”.

Could still use some macros the sweeten the raw getenv stuff I suppose.

I always imagined doing that kind of thing with a separate submodule

(module+ integration-test ...)

Actually yes I think I’ve done that way, too.

I’m not sure how to write the “run these submods by default unless you’re in the CI environment” logic though

@ben progress is a new info type https://github.com/racket/rackunit/pull/40