As a consequence of electric-magnetic duality, partition functions of four-dimensional gauge theories can be expressed in terms of modular forms in many cases. I will discuss new results for the modularity of topologically twisted partition functions of N=2 and N=4 supersymmetric theories, and in particular how these partititon functions may involve (iterated) integrals of modular forms.