If you’ve ever worked with OpenAI’s Triton language, you know how powerful it can be for writing GPU kernel code. But have you ever tried to use it with backpropagation? It’s a bit tricky, especially when you’ve implemented custom operations for your model. That’s why I’m excited to share with you a little proof-of-concept library I’ve created called Triton BWD. It enables automatic differentiation on Triton GPU kernels, making it easier to use backpropagation with custom operations.
I’ve written a blog post explaining my approach in more detail, so be sure to check it out if you’re interested. The library is still in its early stages, but I hope it will be of interest to some of you. Have a nice day!