Timo Sonnenschein, Department of Computer Science, University of Kaiserslautern-Landau (RPTU)

Title: Introduction to Implicit Layers in the context of Deep Equilibrium Models

Abstract:

Training modern neural networks can be challenging with the limited memory of specialized hardware. Reusing parameters and storing only a few intermediate results for backward propagation can reduce memory consumption significantly.

Implicit layers offer a solution by iteratively applying the same function until convergence and calculating the gradients at the converged point without any intermediate results using implicit differentiation. Further, implicit layers allow for various sets of requirements. Due to the separation of problem definition and actual computation, they can utilize several solving methods and adapt the accuracy to the problem. One major shortcoming is the runtime, not only for training but also for inference. Further investigation might solve this problem.

One architectures using implicit layers are the Deep Equilibrium Models. They have comparable accuracy to state-of-the-art models in a variety of fields, including language processing and visual tasks.

In conclusion, this talk explores implicit layers, emphasizing their efficiency, adaptability, and potential contributions to neural network architectures. The discussions on DEQ models aim to unravel their applications in the field.

How to join online

You can join online via Zoom, using the following link:
https://uni-kl-de.zoom.us/j/63123116305?pwd=Yko3WU9ZblpGR3lGUkVTV1kzMCtUUT09