Rajan and I just hashed out a plan for LDA evaluation. In the AugurV2 paper, they use predictive log-likelihood (i.e., log-likelihood of held-out data given estimated parameters), which can only be approximated, and they only use it as a correctness check. Given that LDA is unsupervised already, we plan to plot the progress of LDA samplers using log-likelihood (of training data given estimated parameters), which can be computed exactly. MALLET contains this code (cc.mallet.topics.ParallelTopicModel.modelLogLikelihood()), but even better, Hakaru’s simplifier can do the integrals and produce the code as a weight!

Augur had a lot more number of samples, I don’t know if I am sampling it right in terms of sweeps and updates. So for now I just ran it for 20seconds.