Algorithmic Differentiation

(Algorithmisches Differenzieren)

Lecturer: Prof. Nicolas R. Gauger, Dr. Max Sagebaum
Date and Place:
Lectures: Wednesdays, 08:20h – 9:50h, SC Seminar Room 32-349 (First Lecture: 30.10.2019)
Exercises: Mondays, 8:15h – 9:45h, SC Seminar Room 32-349 (First Exercise: 11.11.2019)
ECTS: 5 Credits
SWS: V2Ü2
Language: English

Audience:

Master students in mathematics, computer science and engineering. Intermediate level in C/C++ programming is practical.

Objective:

Algorithmic Differentiation (AD), also called automatic differentiation, is the name given to a set of techniques to evaluate the derivative of a function, which is realized as a computer program. AD is based on the simple fact that every computer program, independent of its complexity, executes a sequence of elementary arithmetic operations (+, -, *, etc.) and elementary functions (sin, cos, exp, etc.). By applying the chain rule of calculus repeatedly to these operations, we can generate another computer program that can compute the derivatives of any output with respect to any input of arbitrary order.
In the lecture, the fundamentals of AD will be covered. These include theoretical background as well as the implementation aspects. Special emphasis will be given to AD techniques such as checkpointing or iterative differentiation, which facilitate the usage of AD packages especially for large scale applications. In the exercises, the students are expected to gain more insight and best-practise by applying AD techniques to simple examples selected from scientific computing. Tutorials for the three established AD packages (ADOL-C, CoDiPack and TAPENADE) will be also provided.

Content:

  • Fundamentals of Forward and Reverse Modes of AD
  • Implementation & Software
  • Higher Order Derivatives
  • Reversal Schedules and Checkpointing Schemes
  • Implicit and Iterative Differentiation
  • Memory Issues and Complexity Bounds for AD

Literature:

  • Griewank, A., Walther, A.: Evaluating Derivatives, Principles and Techniques of Algorithmic Differentiation, Second Edition, SIAM
  • Naumann, U.: The Art of Differentiating Computer Programs: An Introduction to Algorithmic Differentiation, SIAM

Lecture Slides and exercise sheets will be available in the OLAT website of the course:
OLAT Link
KIS link