Bound Optimization

tight lowerbound $LL(\mathbf{\theta })$ ,

$$Q\left(\mathbf{\theta },{\mathbf{\theta }}^t\right)\le LL(\mathbf{\theta })$$

• ${\mathbf{\theta }}^{t+1}=\underset{\mathbf{\theta }}{\mathrm{argmax}}Q\left(\mathbf{\theta },{\mathbf{\theta }}^t\right)$
• $LL\left({\mathbf{\theta }}^{t+1}\right)\ge Q\left({\mathbf{\theta }}^{t+1},{\mathbf{\theta }}^t\right)\ge Q\left({\mathbf{\theta }}^t,{\mathbf{\theta }}^t\right)=LL\left({\mathbf{\theta }}^t\right)$
• If Q is a quadratic lower bound, the overall method is similar to Newton’s method, which repeatedly fits and then optimizes a quadratic approximation.
• The difference is that optimizing Q is guaranteed to lead to an improvement in the objective, even if it is not convex, whereas Newton-Raphson method may overshoot or lead to a decrease in the objective

B) Related

C) References