Studying inductive biases of random networks via local volumes

In this post, we will study inductive biases of the parameter-function map of random neural networks using star domain volume estimates. This builds on the ideas introduced in Estimating the Probability of Sampling a Trained Neural Network at Random and Neural Redshift: Random Networks are not Random Functions (henceforth NRS). Inductive biases To understand generalization in deep neural networks, we must understand inductive biases. Given a fixed architecture, some tasks will be easily learnable, while others can take an exponentially long time to learn (see here and here).

Details

The article discusses studying the inductive biases of random neural networks using star domain volume estimates. This builds on previous research that introduced the concept of the parameter-function map and the idea that random neural networks are not actually random functions. The goal is to better understand generalization in deep neural networks by understanding their inductive biases. Given a fixed neural network architecture, some tasks will be easily learnable, while others can take an exponentially long time to learn. The star domain volume estimates are used to quantify these inductive biases and provide insights into the non-randomness of random neural networks.