Random thoughts on minimization
* Just seen Ecker and Jepson, "Polynomial Shape from Shading"… We often optimize models which are polynomial in the unknowns, e.g. matrix factorization is biquadratic. Line search in such models can be solved in closed form… Don't know if it helps yet.
* If I'm solving a nonlinear least squares problem, e.g. by Levenberg-Marquardt, and the Jacobian is such that multiplication by it can be most easily implemented by recalculating it at every call, and we're doing PCG to solve the augmented system, is there any benefit to be gained by reevaluating the Jacobian at the same point each time?