Powered by RND
PodcastsTechnologyPyTorch Developer Podcast
Listen to PyTorch Developer Podcast in the App
Listen to PyTorch Developer Podcast in the App
(7,438)(250,057)
Save favourites
Alarm
Sleep timer

PyTorch Developer Podcast

Podcast PyTorch Developer Podcast
Edward Yang, Team PyTorch
The PyTorch Developer Podcast is a place for the PyTorch dev team to do bite sized (10-20 min) topics about all sorts of internal development topics in PyTorch.

Available Episodes

5 of 83
  • Compiler collectives
    Compiler collectives are a PT2 feature where by compiler instances across multiple ranks use NCCL collectives to communicate information to other instances. This is used to ensure we consistently decide if inputs or static or dynamic across all ranks. See also PR at https://github.com/pytorch/pytorch/pull/130935
    --------  
    16:33
  • TORCH_TRACE and tlparse
    TORCH_TRACE and tlparse are a structured log and log parser for PyTorch 2. It gives useful information about what code was compiled and what the intermediate build products look like.
    --------  
    15:28
  • Higher order operators
    Higher order operators are a special form of operators in torch.ops which have relaxed input argument requirements: in particular, they can accept any form of argument, including Python callables. Their name is based off of their most common use case, which is to represent higher order functions like control flow operators. However, they are also used to implement other variants of basic operators and can also be used to smuggle in Python data that is quite unusual. They are implemented using a Python dispatcher.
    --------  
    17:10
  • Inductor - Post-grad FX passes
    The post-grad FX passes in Inductor run after AOTAutograd has functionalized and normalized the input program into separate forward/backward graphs. As such, they generally can assume that the graph in question is functionalized, except for some mutations to inputs at the end of the graph. At the end of post-grad passes, there are special passes that reintroduce mutation into the graph before going into the rest of Inductor lowering which is generally aware of passes. The post-grad FX passes are varied but are typically domain specific passes making local changes to specific parts of the graph.
    --------  
    24:07
  • CUDA graph trees
    CUDA graph trees are the internal implementation of CUDA graphs used in PT2 when you say mode="reduce-overhead". Their primary innovation is that they allow the reuse of memory across multiple CUDA graphs, as long as they form a tree structure of potential paths you can go down with the CUDA graph. This greatly reduced the memory usage of CUDA graphs in PT2. There are some operational implications to using CUDA graphs which are described in the podcast.
    --------  
    20:50

More Technology podcasts

About PyTorch Developer Podcast

The PyTorch Developer Podcast is a place for the PyTorch dev team to do bite sized (10-20 min) topics about all sorts of internal development topics in PyTorch.
Podcast website

Listen to PyTorch Developer Podcast, Darknet Diaries and many other podcasts from around the world with the radio.net app

Get the free radio.net app

  • Stations and podcasts to bookmark
  • Stream via Wi-Fi or Bluetooth
  • Supports Carplay & Android Auto
  • Many other app features
Social
v7.2.0 | © 2007-2025 radio.de GmbH
Generated: 1/15/2025 - 7:41:02 PM