I’m guessing that the new review contains a fundamental misunderstanding that (collapsed) Gibbs samplers is all we do? And that, of course, even when there is no collapse things work [though it is likely true we don’t have any tests for that].

https://docs.google.com/document/d/1YlTKQyJ29trIp7EYTPSpl_M-pdeCCZU7E0WxeuQxk9M/edit

Sam is teaching and I can’t edit the document he shared so I made a copy and I hope everyone here can edit this copy: https://docs.google.com/document/d/1qIKoX6A8lhOUr9HitVDwJJKFa2NppQzE_0IvQvJ7K0U/edit?usp=sharing

I’m going to copy the review into the doc and address the explicit questions. Sam has made a first stab at emphasizing that our techniques are more broadly applicable even though our benchmarks are limited by not being part of a production system.

To intersperse your writing in the doc, use Clear Formatting (Ctrl+)

We need to be very clear (while remaining respectful) right from the start that this reviewer, in his review, seems to have a rather fundamental misunderstanding of our contribution. I kind of understand a B if our stuff was restricted to just collapsed Gibbs samplers!

Please feel free to edit the top of the Google Doc. Please recognize that our system is limited. The broad value of the techniques we introduce in Sections 3–5 is justified by a combination of (1) the system evaluated in Section 6 and (2) the literature rife with manually doing integrating-out and density-recognition and histogramming. But a significant portion of this literature remains uncovered by our system. We hope that our techniques will be incorporated in future systems. But we can’t wait until we have built such a complete system before publishing a paper (which Reviewer A would like even less).

I will - later this afternoon. Off to a meeting in a few minutes.

I get access denied to @ccshan’s Google Doc. Share with please?

you should now have access

Yep, thanks.

@rjnw Would you please point me to the code that produced Figure 11 right-hand side (Naive Bayes “log likelihood”)? Or just confirm that it’s the top-level weight so it’s computed without any use of the 10% held-out classifications?

https://github.com/rjnw/hakaru-benchmarks/blob/master/runners/hk/NaiveBayesGibbs/Likelihood.hs

Ok, I’ve done all the edits that I wanted to the first paragraph (where I think a lot of the points were made in quite obscure ways). I have made the same points in straightforward sentences. [With no snark, to boot.]

I’ve read the rest too, and I think all those other answers are great.

I made a major rewrite. I’m not sure about it. Please (re)read.

(Let’s finalize this response by Thursday.)