Original paper : https://arxiv.org/abs/2005.14165 It uses the same architecture as GPT-2. The largest model uses 170B parameters and trained with a batch size of 3.2 million. (Wow!). Training cost exceeds $12M. “Taking all these runs into account, the researchers estimated that building this model generated over 78,000 pounds of CO2 emissions in total—more than the […]

The post Notes on GPT-3 appeared first on A Blog From Human-engineer-being.