Automatic differentiation (AD) is clearly cool. And it has become useful (as well as cool) recently, because it essentially implements the back-propagation step that is key to learning in neural networks; and it is used extensively in many other machine learning contexts. But as well as being cool and useful, AD is also mysterious. There are forward and backward variants (the latter being mind-bending); there are tapes and mutation, delimited continuations, and more. It all makes me feel stupid.

In this talk I’ll explain a simple and elegant approach to AD from first principles, based on work by Conal Elliott. We’ll end up with an AD algorithm that takes 10 lines of code; that works for forward and backward AD; that works for matrix computations; and that is a source-to-source transformation whose output can be efficiently compiled without needing a runtime trace.

Simon Peyton Jones, MA, MBCS, CEng, graduated from Trinity College Cambridge in 1980. After two years in industry, he spent seven years as a lecturer at University College London, and nine years as a professor at Glasgow University, before moving to Microsoft Research (Cambridge) in 1998. His main research interest is in functional programming languages, their implementation, and their application. He has led a succession of research projects focused around the design and implementation of production-quality functional-language systems for both uniprocessors and parallel machines. He was a key contributor to the design of the now-standard functional language Haskell, and is the lead designer of the widely-used Glasgow Haskell Compiler (GHC). He has written two textbooks about the implementation of functional languages.

More generally, he is interested in language design, rich type systems, software component architectures, compiler technology, code generation, runtime systems, virtual machines, and garbage collection. He is particularly motivated by direct use of principled theory to practical language design and implementation – that’s one reason he loves functional programming so much.