@ccshan is there another system where ClinicalTrials or LinearRegression make sense as benchmarks?

Good question… How about another system that performs exact inference? That would be PSI. If it is easy to install (I think it’s written in D and contains those benchmarks or some similar models already programmed). The last I checked, it only handled arrays by unrolling.

mallet’s first update takes 500 ms after that around 200ms each.

naivebayes we do a single update in 25ms but somehow our total sweep takes 40seconds

Cool. Did you recompile mallet and remove the nest of reporting ifs?

this is from the time variable they were using

I just printed that

I see so you rebuilt it with a tiny change

I removed the if’s but I get nullpointer after 9 trial

So you put the ifs back?

yeah with or without if the time is the same

the time is before that if

As for our performance, each sweep is 2000 updates, right? So 25ms * 2000 = 50s, which isn’t so different from the 40s you report, right?

oh yeah

I was rounding 19998 to 10000

So we are 8x as fast as mallet :tada:

for gmmGibbs our startup time from when I enter the command in shell to the point we get the compiled function prog is 1.4 seconds

What command?? Surely not including simplification

no just racket

this includes racket parsing hakaru and doing all the optimizations

I haven’t calculated simplification yet

So how much does the machine know about the data during this 1.4s? Does it know all the array sizes?

it uses runtime information like array size and actual values at the end of the pipeline, theoretically we can do all the stuff before that ahead of time and save it to disk. The compilation using LLVM is around 50ms and which is also mentioned in paper. other than that I think there are two or three recursive pass over the whole hakaru program which use this information and are also cheap.

a lot of it I assume comes from reading from disk parsing and doing the first subsection of optimization.

the runtime stuff in paper is also seperated in a different subsection