@ccshan is it possible to do holdouts in gmmGibbs with larger classes and making the accuracy computation faster and simpler? Right now for 12 classes it never finishes

@rjnw Hmm, I think this is our problem: https://en.m.wikipedia.org/wiki/Assignment_problem

So I wonder if we should use https://hackage.haskell.org/package/Munkres-0.1/docs/Data-Algorithm-Munkres.html ?

but this is for efficiently calculating accuracies

is holdout completely out of question?

What do you mean by holdoutm

?

the same way we do naivebayes

keep some classes same as original and run gmmgibbs, and compare directly with original

classes of some points* I think

right now https://github.com/rjnw/hakaru-benchmarks/blob/master/runners/hk/GmmGibbs/Main.hs#L59 vs naivebayes https://github.com/rjnw/hakaru-benchmarks/blob/master/runners/hk/NaiveBayesGibbs/Main.hs#L66

I still don’t understand what you mean by “do holdouts with larger classes”. I know that we have 3 or 6 or 9 or 12 classes, but I don’t know what you mean by doing holdouts with a class. When I think of doing holdout, I think of choosing perhaps 10% of all data points and forgetting their actual classes. These data points come from all classes, so I don’t know what you mean by doing holdouts with larger classes. Larger than what?

I meant for some of the points have the correct classes then at the end compare the output with the original instead of doing factorial. Anyways I did it in racket, it doesn’t work

500 trials, for min 1second and min 100sweeps.

These messages give me the feeling that we have not settled on an agreed design for what it means to run each of our experiments. [I’ve recently published experimental HCI work, so this has been drilled into me]. Could we perhaps write (even if we later comment out) an ‘experimental setup’ section, to get that agreement?

In theory, this would just mean transcribing what the testing scripts say into plain English. So it should not be particularly onerous.

I’ve left a long message at https://github.iu.edu/ccshan/ppaml/commit/a01518e165edf937f33a53adcc8216ffd1c37288 Not sure if everyone here gets github.iu notifies, so I figured crossposting might be useful.