December 15, 2022
Scaling PyTorch FSDP for Training Foundation Models on IBM Cloud
Large model training using a cloud native approach is of growing interest for many enterprises given the emergence and success of foundation models. Some AI practitioners may assume that the only way they can achieve high GPU utilization for distributed training jobs is to run them on HPC systems, such as those inter-connected with Infiniband and may not consider Ethernet connected systems. We demonstrate how the latest distributed training technique, Fully Sharded Data Parallel (FSDP) from P...
December 02, 2022
Get Started with PyTorch 2.0 Summary and Overview
Introducing PyTorch 2.0, our first steps toward the next generation 2-series release of PyTorch. Over the last few years we have innovated and iterated from PyTorch 1.0 to the most recent 1.13 and moved to the newly formed PyTorch Foundation, part of the Linux Foundation.
December 02, 2022
Accelerating Hugging Face and TIMM models with PyTorch 2.0
torch.compile() makes it easy to experiment with different compiler backends to make PyTorch code faster with a single line decorator torch.compile(). It works either directly over an nn.Module as a drop-in replacement for torch.jit.script() but without requiring you to make any source code changes. We expect this one line code change to provide you with between 30%-2x training time speedups on the vast majority of models that you’re already running.
November 28, 2022
Optimizing Production PyTorch Models’ Performance with Graph Transformations
1. Introduction
November 21, 2022
Scaling Multimodal Foundation Models in TorchMultimodal with Pytorch Distributed
Introduction
November 17, 2022
Introducing TorchMultimodal - a library for accelerating exploration in Multimodal AI
We are announcing TorchMultimodal Beta, a PyTorch domain library for training SoTA multi-task multimodal models at scale. The library provides composable building blocks (modules, transforms, loss functions) to accelerate model development, SoTA model architectures (FLAVA, MDETR, Omnivore) from published research, training and evaluation scripts, as well as notebooks for exploring these models. The library is under active development, and we’d love to hear your feedback! You can find more det...
