How Deep Learning Finally Cracked Messy Tables - Frank Hutter

<p>Frank Hutter, co-founder of Prior Labs, talks about TabPFN, a tabular foundation model that makes predictions in a single forward pass, and the research behind it.</p><p><br></p><p>TabPFN is pre-trained on synthetic datasets drawn from a prior over structural causal models, rather than on real data. At prediction time it takes the whole training table as context and outputs an approximation of the Bayesian posterior predictive distribution, without per-dataset training or hyperparameter search. Frank explains how this grew out of his earlier work on AutoML and neural architecture search, how the priors are built and revised, and why tabular data was hard for deep learning for so long.</p><p><br></p><p>The conversation also covers the TabArena benchmark, how the architecture changed from TabPFN v1 to v3, scaling to larger tables, using the model with coding agents, test-time compute, Google&#39;s TabFM, causal inference and interventions, and relational data. At the end, a short update Frank recorded after the interview covers the TabPFN-3.5 release.</p><p><br></p><p>Prior Labs:</p><p>TabPFN-3.5: https://priorlabs.ai/tabpfn-3-5</p><p>https://priorlabs.ai/careers</p><p><br></p><p>TOC:</p><p>00:00 Introduction</p><p>00:44 Welcome and Frank&#39;s background</p><p>02:05 Why tabular data was hard for deep learning</p><p>10:17 Pre-training on synthetic data</p><p>12:52 The TabArena benchmark</p><p>19:28 From AutoML to neural architecture search</p><p>26:34 TabPFN as a learned algorithm</p><p>30:50 Bayesian prediction in one forward pass</p><p>39:37 Scaling to larger tables</p><p>47:48 Using TabPFN with coding agents</p><p>57:47 Output heads and architecture from v1 to v3</p><p>1:05:29 Test-time compute and adaptation</p><p>1:13:32 Google&#39;s TabFM</p><p>1:16:53 How the priors are designed</p><p>1:18:40 Correlation, causation and interventions</p><p>1:35:22 Relational and multimodal data</p><p>1:38:31 Use in organisations</p><p>1:46:38 The open research arm</p><p>1:50:21 Update: TabPFN-3.5</p><p><br></p><p>REFS:</p><p>TabPFN v2, Nature (Hollmann et al., 2025)</p><p>https://www.nature.com/articles/s41586-024-08328-6</p><p>Transformers Can Do Bayesian Inference (Müller et al.)</p><p>https://arxiv.org/abs/2112.10510</p><p>TabArena (Erickson et al.)</p><p>https://arxiv.org/abs/2506.16791</p><p>AutoGluon-Tabular (Erickson et al.)</p><p>https://arxiv.org/abs/2003.06505</p><p>Beyond IID: How General Are Tabular Foundation Models, Really?</p><p>https://arxiv.org/abs/2606.30410</p><p>Neural Architecture Search: A Survey (Elsken, Metzen &amp; Hutter)</p><p>https://arxiv.org/abs/1808.05377</p><p>Auto-WEKA (Thornton et al.)</p><p>https://www.cs.ubc.ca/~hutter/papers/AutoWEKA-KDD2013.pdf</p><p>TabPFN v1 (Hollmann et al., 2022)</p><p>https://arxiv.org/abs/2207.01848</p><p>TabPFN-3 technical report</p><p>https://arxiv.org/abs/2605.13986</p><p>TabPFN-2.5 report</p><p>https://arxiv.org/abs/2511.08667</p><p>CAAFE (Hollmann et al.)</p><p>https://arxiv.org/abs/2305.03403</p><p>TabICL (Qu et al.)</p><p>https://arxiv.org/abs/2502.05564</p><p>TabICLv2 (Qu et al.)</p><p>https://arxiv.org/abs/2602.11139</p><p>Google TabFM</p><p>https://research.google/blog/introducing-tabfm-a-zero-shot-foundation-model-for-tabular-data/</p><p>TALENT benchmark (Ye et al.)</p><p>https://arxiv.org/abs/2407.00956</p><p>Do-PFN (Robertson et al.)</p><p>https://arxiv.org/abs/2506.06039</p><p>CausalPFN (Balazadeh et al.)</p><p>https://arxiv.org/abs/2506.07918</p><p>Causal Foundation Models with Partial Graphs (Reuter et al.)</p><p>https://arxiv.org/abs/2602.14972</p><p>RelBench (Robinson et al.)</p><p>https://arxiv.org/abs/2407.20060</p><p>RelArena-α, TabPFN-Rel and RPI</p><p>https://arxiv.org/abs/2608.16319</p><p>TabPFN on GitHub</p><p>https://github.com/PriorLabs/TabPFN</p><p>TabPFN-3.5 technical report</p><p>https://arxiv.org/abs/2609.17895</p><p>Otto Group Product Classification Challenge (Kaggle, 2015)</p><p>https://www.kaggle.com/competitions/otto-group-product-classification-challenge</p><p><br></p><p>---RESCRIPT:https://app.rescript.info/share/e99676c25ee6189fbf54c9be07eb623e</p>