@rjnw, I requested some changes to the plots at https://github.iu.edu/ccshan/ppaml/commit/a3ffc51ff34bfc409e6be4b3e218e44665d60f3a#commitcomment-2115

It seems easy to run PSI to beef up our exact-inference evaluation. Is it easy for me to log onto your machine so that I can run on the same hardware as you?

@ccshan the hakaru code for likelihood returns a Prob type and I use log from our LogFloatPrelude. So it is log of likelihood right?

For AugurV2 update time, can I divide sweep time by total updates? (JAGS is calculated the same way)

Yes and yes, thanks!

Augur every 100 sweeps Hakaru every sweep. Jags second sweep is at 500

I like this picture. By “JAGS seconds sweep is at 500”, do you mean it takes 500 seconds for JAGS to do one sweep? Would you please change the horizontal axis label to "Time in seconds” and make another plot for “Log likelihood”?

Yes JAGS takes ~500 seconds for one full update.

A mistake in this graph, when removing warmup time I also removed the 500 seconds for the first update. Or should I count that as warmup? I am not really sure so for now this has outcome of first sweep at time0 for all three.

I’m ok with having “outcome of first sweep at time0 for all three”, but is that the case for LLVM-backend, since you measured updates within a sweep for it?

Oh yeah I forgot about that

Oh in NaiveBayes I don’t do that

It’s only in LDA

Being consistent in this regard (across curves and benchmarks) would help the writing a lot.

The problem with NaiveBayes and JAGS is the 500 second sweep time with ~3hr startup.

How about being consistent in the following way? In every plot, the horizontal axis is seconds after startup. Startup is defined by the table. So “time 0” on each curve is always the time after the startup as listed in the table.

Okay, that sounds good to me. Gmm and NaiveBayes already follow that rule.

I am trying to change the vertical axis range for naive bayes accuracy. If I do 45–85 there is no place for legend.

@ccshan I have psi running on my machine how do you want to run the code you committed?

I tried just giving the file name to psi it’s feels like stuck.

Maybe it’s stuck because it is :notes:slow:notes:

Change n from 100 to 10?

yeah that works.

~/w/h/o/psi > time ./psi ../../testcode/psisrc/clinicalTrial.psi
p(isEffective) = 4199/10007·δ(1)[isEffective]+5808/10007·δ(0)[isEffective]
1.62user 0.02system 0:01.64elapsed 99%CPU (0avgtext+0avgdata 61024maxresident)k
0inputs+0outputs (0major+13175minor)pagefaults 0swaps

So if for each of the two exact inference benchmarks you could make a plot of time in seconds versus data size, just for PSI, that’s be great

Of course there’s trade-off in how many times you run each n (resulting in smaller error bars) vs how large n you get to

And maybe in the spirit of up=better, the horizontal axes should be time in seconds and the vertical axes should be data size.