[https://i.imgur.com/lG9je1g.png]

[https://i.imgur.com/1hJa51n.png] [https://i.imgur.com/geDXLLG.png] [https://i.imgur.com/zMxBlb0.png]

[https://i.imgur.com/tupgltr.png ][https://i.imgur.com/kNG5for.png]

[https://i.imgur.com/BtzHnUq.png]

[https://i.imgur.com/jaHW2pp.png ]

[https://i.imgur.com/MmtIR5e.png]

[https://i.imgur.com/S1KiiUj.png]

[https://i.imgur.com/9ZqUDO6.png]

[https://i.imgur.com/TT61fRC.png]

[https://i.imgur.com/JGA0KAj.png]

[https://i.imgur.com/ZBG9SXA.png ]

[https://i.imgur.com/IhjRIpN.png ]

[https://i.imgur.com/lsJLMDk.png ]

[https://i.imgur.com/OBmQUKm.png]

Being on Earth when this happens is a big deal, no matter your objectives – you can't hoard pebbles if you're dead! People would feel the loss from anywhere in the cosmos. However, Pebblehoarders wouldn't mind if they weren't in harm's way.

[https://i.imgur.com/zKM6Bt9.png]

[https://i.imgur.com/ZOAeuoe.png]

[https://i.imgur.com/fzLD7kQ.png]

APPENDIX: CONTRIVED OBJECTIVES A natural definitional objection is that a few agents aren't affected by objectively impactful events. If you think every outcome is equally good, then who cares if the meteor hits?

Obviously, our values aren't like this, and any agent we encounter or build is unlikely to be like this (since these agents wouldn't do much). Furthermore, these agents seem contrived in a technical sense (low measure under reasonable distributions in a reasonable formalization), as we'll see later. That is, "most" agents aren't like this.

From now on, assume we aren't talking about this kind of agent. --------------------------------------------------------------------------------

Notes * Eliezer introduced Pebblesorters in the the Sequences [https://www.readthesequences.com/Sorting-Pebbles-Into-Correct-Heaps]; I made them robots here to better highlight how pointless the pebble transformation is to humans. * In informal parts of the sequence, I'll often use "values", "goals", and "objective
https://www.lesswrong.com/posts/TxcYSRQ9giC6zmKov/value-impact
* LESSWRONG * LW * Reframing Impact * TurnTrout * Value Impact * Appendix: Contrived Objectives * No comments * AI Alignment Forum * introduced Pebblesorters in the the Sequences * some people don't have mental imagery