Algorithmic Differentiation

(Algorithmisches Differenzieren)

Lecturer: Prof. Nicolas R. Gauger, Dr. Emre Özkaya
Date and Place:
Lectures: Thursdays, 08:15h – 09:45h, SC Seminar Room 32-349 (First Lecture: 26.10.2017)
Exercises: Fridays, 15:30h – 17:00h, Bldg. 34 Terminal Pool 2 (First Exercise: 03.11.2017, SC Seminar Room 32-349)
ECTS: 5 Credits
SWS: V2Ü2
Language: English

Audience:

Master students in mathematics, computer science and engineering. Intermediate level in C/C++ programming is practical.

Objective:

The Algorithmic Differentiation (AD), also called automatic differentiation, is the name given to a set of techniques to evaluate the derivative of a function, which is realized as a computer program. AD is based on the simple fact that every computer program, independent of its complexity, executes a sequence of elementary arithmetic operations (+, -, *, etc.) and elementary functions (sin, cos, exp, etc.). By applying the chain rule of calculus repeatedly to these operations, we can generate another computer program that can compute the derivatives of any output with respect to any input of arbitrary order.
In the lecture, the fundamentals of AD will be covered. These include theoretical background as well as the implementation aspects. Special emphasis will be given to the AD techniques such as checkpointing or iterative differentiation, which facilitate the usage of AD packages especially for large scale applications. In the exercises, the students are expected to gain more insight and best-practise by applying AD techniques to simple examples selected from scientific computing. Tutorials for the two established AD packages (ADOL-C and TAPENADE) will be also provided.

Content:

  • Fundementals of Forward and Reverse Modes of AD
  • Implementation & Software
  • Higher Order Derivatives
  • Reversal Schedules and Checkpointing Schemes
  • Implicit and Iterative Differentiation
  • Memory Issues and Complexity Bounds for AD

Literature:

  • Griewank, A., Walther, A.: Evaluating Derivatives, Principles and Techniques of Algorithmic Differentiation, Second Edition, SIAM
  • Naumann, U.: The Art of Differentiating Computer Programs: An Introduction to Algorithmic Differentiation, SIAM

Lecture Slides and exercise sheets will be available in the OLAT website of the course:
OLAT Link
KIS link