Exploratory Installation of Unsloth on NVIDIA Jetson AGX Orin 64 GB
This article documents the process of installing and running Unsloth, including Unsloth Studio, on an NVIDIA Jetson AGX Orin 64 GB device using a Docker-based workflow.
Why it matters
This article provides valuable insights for practitioners and the Unsloth team to better understand the current suitability of Unsloth Studio for Jetson-class devices.
Key Points
- 1Successful validation of GPU-accelerated PyTorch and Unsloth's core Python package on Jetson
- 2Substantial friction and incompatibilities in getting Unsloth Studio's full stack to run reliably on the ARM-based edge platform
- 3Detailed technical account to help other practitioners reproduce or avoid the same pitfalls and assess Unsloth Studio's suitability for Jetson-class devices
Details
The article describes the hardware and software environment used for the experiments, including the NVIDIA Jetson AGX Orin 64 GB developer kit, Ubuntu 22.04.5 LTS, JetPack 6.2.2, and various CUDA, cuDNN, and TensorRT versions. It then outlines the process of constructing a custom Docker image based on the 'dustynv/l4t-ml:r36.4.0' base image, which provides PyTorch compiled for Jetson with CUDA and TensorRT integration, as well as JupyterLab and common ML tooling. The article highlights the successful installation of Unsloth's core Python package and the challenges faced in getting Unsloth Studio's full stack, including the Studio backend, frontend, Triton/TorchInductor/TorchAo dependencies, and custom virtual environment, to run reliably on the Jetson platform.
No comments yet
Be the first to comment