Anton on Twitter: https://twitter.com/atroyn

Moritz on Twitter: https://twitter.com/MoritzW42

Scaling Knowledge: https://scalingknowledge.substack.com

Mentioned Resources

• Unlimiformer: Long-Range Transformers with Unlimited Length Input: arxiv.org/abs/2305.01625
• Memorizing Transformers: arxiv.org/abs/2203.08913
• Chroma: https://www.trychroma.com
• ⁠You and your Research⁠: https://www.cs.virginia.edu/~robins/YouAndYourResearch.html
• Against Method⁠: https://en.wikipedia.org/wiki/Against_Method

Contents:

(00:00) Preview

(01:14) Introduction

(06:13) Edech

(12:12) Airchips and Square Finances

(24:06) Technical Optimism

(25:09) Convenience vs Enabling Technology

(31:13) Chroma and LLM Landscape

(35:37) Minecraft Bot

(37:05) Knowledge Retrieval

(39:38) New Knowledge Retrieval Research and Techniques: (Direct Retrieval Augmentation of Transformers, Conditional Retrieval)

(42:35) AI Native Platforms

(45:43) Theorem Proofing and Generation

(49:40) Do LLMs Reason

(52:18) Computational Complexity and LessWrong

(53:58) Frameworks and Taking Your Work Seriously

(55:53) Job Displacement and Jobs Supply vs Demand Side

(58:29) The Printing Press