You mean for Racket?

I don’t know of any efforts to fuzz Racket tbh.

The biggest blocker to fuzzing racket atm is executions / sec. Racket is just too slow. I need to somehow speed it up or fuzz a few components at a time. I am working on this for JSC too, which shares the same issue so any solutions/ideas I come up with there will be applicable to Racket which is great.

AFL is doing 7 exec/sec of racket CS

Which is very poor. It would be great to do a lot more. In the hundreds if possible, however startup time is quite slow.

This is with AFL++.

I think with a targetted libfuzzer it should be much quicker but I didn’t get there yet.

This is one of the paths of course - finding crashes and security bugs.

The other path to fuzzing is through fuzzing to find semantic differences between the current version and an oracle and we could use XSmith for that.

I note now that the executions / sec went further down. :disappointed: ~5 which is embarassingly low. Last path found was 5 hrs ago… We won’t go very far like this I think.

Anyway, this is just preliminary experiments, I will be working on this for a while.

Here are the fuzzing experiments I know about: 1. I did what @pocmatos is trying once, with a similar lack of success 2. @willghatch is using XSmith to fuzz Racket CS vs BC 3. https://github.com/racket/racket/blob/master/pkgs/racket-test/tests/racket/stress/fuzz.rkt (also by me) which found a lot of bytecode parsing bugs back in the day. I should make it work for Racket CS.

@pocmatos I don’t know what command you’re using, but if you run something like `racket -I racket/kernel/init -f “file” then it will start a lot faster.

the other thing is to add the appropriate fork so that AFL can avoid the startup time.

@sorawee what happened in that run is make failed (http://drdr.racket-lang.org/55400/pkg-src/build/make) so nothing got built so the raco test command didn’t exist

As mentioned this was very preliminary. Didn’t know about racket -I racket/kernel/init -f "file". I was using racket -t .... What’s the difference between these? I assume that if your command line doesn’t load the whole racket/base then I cannot run programs require it, right?

I know about @willghatch work and hope to integrate that into CI as well.

Didn’t know about your fuzz.rkt Will take a look into it as well.

What do you mean about adding the appropriate fork?

racket -t will work fine, but if you start with racket/base then it will be slow to start

i recommend using only the kernel if possible

that’s also not loading modules; it’s calling load

see https://lcamtuf.blogspot.com/2014/10/fuzzing-binaries-without-execve.html and https://lcamtuf.blogspot.com/2015/06/new-in-afl-persistent-mode.html

Thanks @samth I will take a look.

@samth what’s available in racket/kernel/init? do we have documentation on that?

ah found https://docs.racket-lang.org/reference/Kernel_Forms_and_Functions.html?q=racket%2Finit

interesting.

so i guess if i fully expand a program, i can then run it in racket/kernel/init? the idea being finding programs that I can use as initial seeds for fuzzing.

basically nothing is available, but yeah full expansion and then use that

if you want to write modules, then #lang racket/kernel will work

the /init is only there to enable the repl/top-level

but then i need to figure out which forms are available in racket/kernel.

as an idea, do you know if i can use xsmith @willghatch to generate random programs that are valid under racket/kernel and I can then use as seeds for fuzzing?

in particular I was thinking about this: https://gitlab.flux.utah.edu/xsmith/xsmith/-/blob/master/xsmith-examples/racket-kernel-fuzzer/racket-kernel-fuzzer.rkt

yeah maybe it works…

keen to see how many executions I can get per second with this.

but i am late on RN so not today…