What are CUDA graphs? How are they implemented? What does it take to actually use them in PyTorch?

Further reading.

• NVIDIA has docs on CUDA graphs https://developer.nvidia.com/blog/cuda-graphs/
• Nuts and bolts implementation PRs from mcarilli: https://github.com/pytorch/pytorch/pull/51436 https://github.com/pytorch/pytorch/pull/46148