The chain rule – fundamental to any kind of analytical differentiation - can be applied in various ways to computational
graphs representing vector functions. These variants result in different operations counts for the calculation of the corresponding
Jacobian matrices. The minimization of the number of arithmetic operations required for the calculation of the complete Jacobian
leads to a hard combinatorial optimization problem.
We will describe an approach to the solution of this problem that builds on the idea of optimizing chained matrix products
using dynamic programming techniques. Reductions by a factor of 3 and more are possible regarding the operations count for
the Jacobian accumulation.
After discussing the mathematical basics of Automatic Differentiation we will show how to compute Jacobians by chained sparse
matrix products. These matrix chains can be reordered, must be pruned, and are finally subject to a dynamic programming algorithm
to reduce the number of scalar multiplications performed.
Received: January 17, 2002 / Accepted: May 29, 2002 Published online: February 14, 2003
Key words. chained matrix product – combinatorial optimization – dynamic programming – edge elimination in computational graphs