The Nonlinear Library: LessWrong Top Posts: Recent Episodes
The Nonlinear Fund
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Eight Short Studies On Excuses , published by Scott Alexander on LessWrong. The Clumsy Game-Player You and a partner are playing an Iterated Prisoner's Dilemma. Both of you have publicly pre-committed to the tit-for-tat strategy. By iteration 5, you're going happily along, raking up the bonuses of cooperation, when your partner unexpectedly presses the "defect" button. "Uh, sorry," says your partner. "My finger slipped." "I still have to punish you just in case," you say. "I'm going to defect next turn, and we'll see how you like it." "Well," said your partner, "knowing that, I guess I'll defect next turn too, and we'll both lose out. But hey, it was just a slipped finger. By not trusting me, you're costing us both the benefits of one turn of cooperation." "True", you respond "but if I don't do it, you'll feel free to defect whenever you feel like it, using the 'finger slipped' excuse." "How about this?" proposes your partner. "I promise to take extra care that my finger won't slip again. You promise that if my finger does slip again, you will punish me terribly, defecting for a bunch of turns. That way, we trust each other again, and we can still get the benefits of cooperation next turn." You don't believe that your partner's finger really slipped, not for an instant. But the plan still seems like a good one. You accept the deal, and you continue cooperating until the experimenter ends the game. After the game, you wonder what went wrong, and whether you could have played better. You decide that there was no better way to deal with your partner's "finger-slip" - after all, the plan you enacted gave you maximum possible utility under the circumstances. But you wish that you'd pre-committed, at the beginning, to saying "and I will punish finger slips equally to deliberate defections, so make sure you're careful." The Lazy Student You are a perfectly utilitarian school teacher, who attaches exactly the same weight to others' welfare as to your own. You have to have the reports of all fifty students in your class ready by the time midterm grades go out on January 1st. You don't want to have to work during Christmas vacation, so you set a deadline that all reports must be in by December 15th or you won't grade them and the students will fail the class. Oh, and your class is Economics 101, and as part of a class project all your students have to behave as selfish utility-maximizing agents for the year. It costs your students 0 utility to turn in the report on time, but they gain +1 utility by turning it in late (they enjoy procrastinating). It costs you 0 utility to grade a report turned in before December 15th, but -30 utility to grade one after December 15th. And students get 0 utility from having their reports graded on time, but get -100 utility from having a report marked incomplete and failing the class. If you say "There's no penalty for turning in your report after deadline," then the students will procrastinate and turn in their reports late, for a total of +50 utility (1 per student times fifty students). You will have to grade all fifty reports during Christmas break, for a total of - 1500 utility (-30 per report times fifty reports). Total utility is -1450. So instead you say "If you don't turn in your report on time, I won't grade it." All students calculate the cost of being late, which is +1 utility from procrastinating and -100 from failing the class, and turn in their reports on time. You get all reports graded before Christmas, no students fail the class, and total utility loss is zero. Yay! Or else - one student comes to you the day after deadline and says "Sorry, I was really tired yesterday, so I really didn't want to come all the way here to hand in my report. I expect you'll grade my report anyway, because I know you to be a perfect utilitarian, an...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Making Vaccine, published by johnswentworth on LessWrong. Back in December, I asked how hard it would be to make a vaccine for oneself. Several people pointed to radvac. It was a best-case scenario: an open-source vaccine design, made for self-experimenters, dead simple to make with readily-available materials, well-explained reasoning about the design, and with the name of one of the world’s more competent biologists (who I already knew of beforehand) stamped on the whitepaper. My girlfriend and I made a batch a week ago and took our first booster yesterday. This post talks a bit about the process, a bit about our plan, and a bit about motivations. Bear in mind that we may have made mistakes - if something seems off, leave a comment. The Process All of the materials and equipment to make the vaccine cost us about $1000. We did not need any special licenses or anything like that. I do have a little wetlab experience from my undergrad days, but the skills required were pretty minimal. One vial of custom peptide - that little pile of white powder at the bottom. The large majority of the cost (about $850) was the peptides. These are the main active ingredients of the vaccine: short segments of proteins from the COVID virus. They’re all <25 amino acids, so far too small to have any likely function as proteins (for comparison, COVID’s spike protein has 1273 amino acids). They’re just meant to be recognized by the immune system: the immune system learns to recognize these sequences, and that’s what provides immunity. Each of six peptides came in two vials of 4.5 mg each. These are the half we haven't dissolved; we keep them in the freezer as backups. The peptides were custom synthesized. There are companies which synthesize any (short) peptide sequence you want - you can find dozens of them online. The cheapest options suffice for the vaccine - the peptides don’t need to be “purified” (this just means removing partial sequences), they don’t need any special modifications, and very small amounts suffice. The minimum order size from the company we used would have been sufficient for around 250 doses. We bought twice that much (9 mg of each peptide), because it only cost ~$50 extra to get 2x the peptides and extras are nice in case of mistakes. The only unusual hiccup was an email about customs restrictions on COVID-related peptides. Apparently the company was not allowed to send us 9 mg in one vial, but could send us two vials of 4.5 mg each for each peptide. This didn’t require any effort on my part, other than saying “yes, two vials is fine, thankyou”. Kudos to their customer service for handling it. Equipment - stir plate, beakers, microcentrifuge tubes, 10 and 50 mL vials, pipette (0.1-1 mL range), and pipette tips. It's all available on Amazon. Other materials - these are sold as supplements. We also need such rare and costly ingredients as vinegar and deionized water. Also all available on Amazon. Besides the peptides, all the other materials and equipment were on amazon, food grade, in quantities far larger than we are ever likely to use. Peptide synthesis and delivery was the slowest; everything else showed up within ~3 days of ordering (it’s amazon, after all). The actual preparation process involves three main high-level steps: Prepare solutions of each component - basically dissolve everything separately, then stick it in the freezer until it’s needed. Circularize two of the peptides. Concretely, this means adding a few grains of activated charcoal to the tube and gently shaking it for three hours. Then, back in the freezer. When it’s time for a batch, take everything out of the freezer and mix it together. Prepping a batch mostly just involves pipetting things into a beaker on a stir plate, sometimes drop-by-drop. Finally, a dose goes into a microcentrifuge tube....
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Best Textbooks on Every Subject, published by lukeprog on LessWrong. For years, my self-education was stupid and wasteful. I learned by consuming blog posts, Wikipedia articles, classic texts, podcast episodes, popular books, video lectures, peer-reviewed papers, Teaching Company courses, and Cliff's Notes. How inefficient! I've since discovered that textbooks are usually the quickest and best way to learn new material. That's what they are designed to be, after all. Less Wrong has often recommended the "read textbooks!" method. Make progress by accumulation, not random walks. But textbooks vary widely in quality. I was forced to read some awful textbooks in college. The ones on American history and sociology were memorably bad, in my case. Other textbooks are exciting, accurate, fair, well-paced, and immediately useful. What if we could compile a list of the best textbooks on every subject? That would be extremely useful. Let's do it. There have been other pages of recommended reading on Less Wrong before (and elsewhere), but this post is unique. Here are the rules: Post the title of your favorite textbook on a given subject. You must have read at least two other textbooks on that same subject. You must briefly name the other books you've read on the subject and explain why you think your chosen textbook is superior to them. Rules #2 and #3 are to protect against recommending a bad book that only seems impressive because it's the only book you've read on the subject. Once, a popular author on Less Wrong recommended Bertrand Russell's A History of Western Philosophy to me, but when I noted that it was more polemical and inaccurate than the other major histories of philosophy, he admitted he hadn't really done much other reading in the field, and only liked the book because it was exciting. I'll start the list with three of my own recommendations... Subject: History of Western Philosophy Recommendation: The Great Conversation, 6th edition, by Norman Melchert Reason: The most popular history of western philosophy is Bertrand Russell's A History of Western Philosophy, which is exciting but also polemical and inaccurate. More accurate but dry and dull is Frederick Copelston's 11-volume A History of Philosophy. Anthony Kenny's recent 4-volume history, collected into one book as A New History of Western Philosophy, is both exciting and accurate, but perhaps too long (1000 pages) and technical for a first read on the history of philosophy. Melchert's textbook, The Great Conversation, is accurate but also the easiest to read, and has the clearest explanations of the important positions and debates, though of course it has its weaknesses (it spends too many pages on ancient Greek mythology but barely mentions Gottlob Frege, the father of analytic philosophy and of the philosophy of language). Melchert's history is also the only one to seriously cover the dominant mode of Anglophone philosophy done today: naturalism (what Melchert calls "physical realism"). Be sure to get the 6th edition, which has major improvements over the 5th edition. Subject: Cognitive Science Recommendation: Cognitive Science, by Jose Luis Bermudez Reason: Jose Luis Bermudez's Cognitive Science: An Introduction to the Science of Mind does an excellent job setting the historical and conceptual context for cognitive science, and draws fairly from all the fields involved in this heavily interdisciplinary science. Bermudez does a good job of making himself invisible, and the explanations here are some of the clearest available. In contrast, Paul Thagard's Mind: Introduction to Cognitive Science skips the context and jumps right into a systematic comparison (by explanatory merit) of the leading theories of mental representation: logic, rules, concepts, analogies, images, and neural networks. The book is o...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Preface, published Eliezer Yudkowsky on LessWrong. You hold in your hands a compilation of two years of daily blog posts. In retrospect, I look back on that project and see a large number of things I did completely wrong. I’m fine with that. Looking back and not seeing a huge number of things I did wrong would mean that neither my writing nor my understanding had improved since 2009. Oops is the sound we make when we improve our beliefs and strategies; so to look back at a time and not see anything you did wrong means that you haven’t learned anything or changed your mind since then. It was a mistake that I didn’t write my two years of blog posts with the intention of helping people do better in their everyday lives. I wrote it with the intention of helping people solve big, difficult, important problems, and I chose impressive-sounding, abstract problems as my examples. In retrospect, this was the second-largest mistake in my approach. It ties in to the first-largest mistake in my writing, which was that I didn’t realize that the big problem in learning this valuable way of thinking was figuring out how to practice it, not knowing the theory. I didn’t realize that part was the priority; and regarding this I can only say “Oops” and “Duh.” Yes, sometimes those big issues really are big and really are important; but that doesn’t change the basic truth that to master skills you need to practice them and it’s harder to practice on things that are further away. (Today the Center for Applied Rationality is working on repairing this huge mistake of mine in a more systematic fashion.) A third huge mistake I made was to focus too much on rational belief, too little on rational action. The fourth-largest mistake I made was that I should have better organized the content I was presenting in the sequences. In particular, I should have created a wiki much earlier, and made it easier to read the posts in sequence. That mistake at least is correctable. In the present work Rob Bensinger has reordered the posts and reorganized them as much as he can without trying to rewrite all the actual material (though he’s rewritten a bit of it). My fifth huge mistake was that I—as I saw it—tried to speak plainly about the stupidity of what appeared to me to be stupid ideas. I did try to avoid the fallacy known as Bulverism, which is where you open your discussion by talking about how stupid people are for believing something; I would always discuss the issue first, and only afterwards say, “And so this is stupid.” But in 2009 it was an open question in my mind whether it might be important to have some people around who expressed contempt for homeopathy. I thought, and still do think, that there is an unfortunate problem wherein treating ideas courteously is processed by many people on some level as “Nothing bad will happen to me if I say I believe this; I won’t lose status if I say I believe in homeopathy,” and that derisive laughter by comedians can help people wake up from the dream. Today I would write more courteously, I think. The discourtesy did serve a function, and I think there were people who were helped by reading it; but I now take more seriously the risk of building communities where the normal and expected reaction to low-status outsider views is open mockery and contempt. Despite my mistake, I am happy to say that my readership has so far been amazingly good about not using my rhetoric as an excuse to bully or belittle others. (I want to single out Scott Alexander in particular here, who is a nicer person than I am and an increasingly amazing writer on these topics, and may deserve part of the credit for making the culture of Less Wrong a healthy one.) To be able to look backwards and say that you’ve “failed” implies that you had goals. So what was it that I was trying to do? Th...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Rationalism before the Sequences, published by Eric Raymond on LessWrong. I'm here to tell you a story about what it was like to be a rationalist decades before the Sequences and the formation of the modern rationalist community. It is not the only story that could be told, but it is one that runs parallel to and has important connections to Eliezer Yudkowsky's and how his ideas developed. My goal in writing this essay is to give the LW community a sense of the prehistory of their movement. It is not intended to be "where Eliezer got his ideas"; that would be stupidly reductive. I aim more to exhibit where the drive and spirit of the Yudkowskian reform came from, and the interesting ways in which Eliezer's formative experiences were not unique. My standing to write this essay begins with the fact that I am roughly 20 years older than Eliezer and read many of his sources before he was old enough to read. I was acquainted with him over an email list before he wrote the Sequences, though I somehow managed to forget those interactions afterwards and only rediscovered them while researching for this essay. In 2005 he had even sent me a book manuscript to review that covered some of the Sequences topics. My reaction on reading "The Twelve Virtues of Rationality" a few years later was dual. It was a different kind of writing than the book manuscript - stronger, more individual, taking some serious risks. On the one hand, I was deeply impressed by its clarity and courage. On the other hand, much of it seemed very familiar, full of hints and callbacks and allusions to books I knew very well. Today it is probably more difficult to back-read Eliezer's sources than it was in 2006, because the body of more recent work within his reformation of rationalism tends to get in the way. I'm going to attempt to draw aside that veil by talking about four specific topics: General Semantics, analytic philosophy, science fiction, and Zen Buddhism. Before I get to those specifics, I want to try to convey that sense of what it was like. I was a bright geeky kid in the 1960s and 1970s, immersed in a lot of obscure topics often with an implicit common theme: intelligence can save us! Learning how to think more clearly can make us better! But at the beginning I was groping as if in a dense fog, unclear about how to turn that belief into actionable advice. Sometimes I would get a flash of light through the fog, or at least a sense that there were other people on the same lonely quest. A bit of that sense sometimes drifted over USENET, an early precursor of today's Internet fora. More often than not, though, the clue would be fictional; somebody's imagination about what it would be like to increase intelligence, to burn away error and think more clearly. When I found non-fiction sources on rationality and intelligence increase I devoured them. Alas, most were useless junk. But in a few places I found gold. Not by coincidence, the places I found real value were sources Eliezer would later draw on. I'm not guessing about this, I was able to confirm it first from Eliezer's explicit reports of what influenced him and then via an email conversation. Eliezer and I were not unique. We know directly of a few others with experiences like ours. There were likely dozens of others we didn't know - possibly hundreds - on parallel paths, all hungrily seeking clarity of thought, all finding largely overlapping subsets of clues and techniques because there simply wasn't that much out there to be mined. One piece of evidence for this parallelism besides Eliezer's reports is that I bounced a draft of this essay off Nancy Lebovitz, a former LW moderator who I've known personally since the 1970s. Her instant reaction? "Full of stuff I knew already." Around the time Nancy and I first met, some years before Eliezer Yudk...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Schelling fences on slippery slopes, published by Scott Alexander on LessWrong. Slippery slopes are themselves a slippery concept. Imagine trying to explain them to an alien: "Well, we right-thinking people are quite sure that the Holocaust happened, so banning Holocaust denial would shut up some crackpots and improve the discourse. But it's one step on the road to things like banning unpopular political positions or religions, and we right-thinking people oppose that, so we won't ban Holocaust denial." And the alien might well respond: "But you could just ban Holocaust denial, but not ban unpopular political positions or religions. Then you right-thinking people get the thing you want, but not the thing you don't want." This post is about some of the replies you might give the alien. Abandoning the Power of Choice This is the boring one without any philosophical insight that gets mentioned only for completeness' sake. In this reply, giving up a certain point risks losing the ability to decide whether or not to give up other points. For example, if people gave up the right to privacy and allowed the government to monitor all phone calls, online communications, and public places, then if someone launched a military coup, it would be very difficult to resist them because there would be no way to secretly organize a rebellion. This is also brought up in arguments about gun control a lot. I'm not sure this is properly thought of as a slippery slope argument at all. It seems to be a more straightforward "Don't give up useful tools for fighting tyranny" argument. The Legend of Murder-Gandhi Previously on Less Wrong's The Adventures of Murder-Gandhi: Gandhi is offered a pill that will turn him into an unstoppable murderer. He refuses to take it, because in his current incarnation as a pacifist, he doesn't want others to die, and he knows that would be a consequence of taking the pill. Even if we offered him $1 million to take the pill, his abhorrence of violence would lead him to refuse. But suppose we offered Gandhi $1 million to take a different pill: one which would decrease his reluctance to murder by 1%. This sounds like a pretty good deal. Even a person with 1% less reluctance to murder than Gandhi is still pretty pacifist and not likely to go killing anybody. And he could donate the money to his favorite charity and perhaps save some lives. Gandhi accepts the offer. Now we iterate the process: every time Gandhi takes the 1%-more-likely-to-murder-pill, we offer him another $1 million to take the same pill again. Maybe original Gandhi, upon sober contemplation, would decide to accept $5 million to become 5% less reluctant to murder. Maybe 95% of his original pacifism is the only level at which he can be absolutely sure that he will still pursue his pacifist ideals. Unfortunately, original Gandhi isn't the one making the choice of whether or not to take the 6th pill. 95%-Gandhi is. And 95% Gandhi doesn't care quite as much about pacifism as original Gandhi did. He still doesn't want to become a murderer, but it wouldn't be a disaster if he were just 90% as reluctant as original Gandhi, that stuck-up goody-goody. What if there were a general principle that each Gandhi was comfortable with Gandhis 5% more murderous than himself, but no more? Original Gandhi would start taking the pills, hoping to get down to 95%, but 95%-Gandhi would start taking five more, hoping to get down to 90%, and so on until he's rampaging through the streets of Delhi, killing everything in sight. Now we're tempted to say Gandhi shouldn't even take the first pill. But this also seems odd. Are we really saying Gandhi shouldn't take what's basically a free million dollars to turn himself into 99%-Gandhi, who might well be nearly indistinguishable in his actions from the original? Maybe Gandhi's best...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Diseased thinking: dissolving questions about disease, published by Scott Alexander on LessWrong. Related to: Disguised Queries, Words as Hidden Inferences, Dissolving the Question, Eight Short Studies on Excuses Today's therapeutic ethos, which celebrates curing and disparages judging, expresses the liberal disposition to assume that crime and other problematic behaviors reflect social or biological causation. While this absolves the individual of responsibility, it also strips the individual of personhood, and moral dignity -- George Will, townhall.com Sandy is a morbidly obese woman looking for advice. Her husband has no sympathy for her, and tells her she obviously needs to stop eating like a pig, and would it kill her to go to the gym once in a while? Her doctor tells her that obesity is primarily genetic, and recommends the diet pill orlistat and a consultation with a surgeon about gastric bypass. Her sister tells her that obesity is a perfectly valid lifestyle choice, and that fat-ism, equivalent to racism, is society's way of keeping her down. When she tells each of her friends about the opinions of the others, things really start to heat up. Her husband accuses her doctor and sister of absolving her of personal responsibility with feel-good platitudes that in the end will only prevent her from getting the willpower she needs to start a real diet. Her doctor accuses her husband of ignorance of the real causes of obesity and of the most effective treatments, and accuses her sister of legitimizing a dangerous health risk that could end with Sandy in hospital or even dead. Her sister accuses her husband of being a jerk, and her doctor of trying to medicalize her behavior in order to turn it into a "condition" that will keep her on pills for life and make lots of money for Big Pharma. Sandy is fictional, but similar conversations happen every day, not only about obesity but about a host of other marginal conditions that some consider character flaws, others diseases, and still others normal variation in the human condition. Attention deficit disorder, internet addiction, social anxiety disorder (as one skeptic said, didn't we used to call this "shyness"?), alcoholism, chronic fatigue, oppositional defiant disorder ("didn't we used to call this being a teenager?"), compulsive gambling, homosexuality, Aspergers' syndrome, antisocial personality, even depression have all been placed in two or more of these categories by different people. Sandy's sister may have a point, but this post will concentrate on the debate between her husband and her doctor, with the understanding that the same techniques will apply to evaluating her sister's opinion. The disagreement between Sandy's husband and doctor centers around the idea of "disease". If obesity, depression, alcoholism, and the like are diseases, most people default to the doctor's point of view; if they are not diseases, they tend to agree with the husband. The debate over such marginal conditions is in many ways a debate over whether or not they are "real" diseases. The usual surface level arguments trotted out in favor of or against the proposition are generally inconclusive, but this post will apply a host of techniques previously discussed on Less Wrong to illuminate the issue. What is Disease? In Disguised Queries , Eliezer demonstrates how a word refers to a cluster of objects related upon multiple axes. For example, in a company that sorts red smooth translucent cubes full of vanadium from blue furry opaque eggs full of palladium, you might invent the word "rube" to designate the red cubes, and another "blegg", to designate the blue eggs. Both words are useful because they "carve reality at the joints" - they refer to two completely separate classes of things which it's practically useful to keep in separate cat...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Generalizing From One Example, published by Scott Alexander on LessWrong. Related to: The Psychological Unity of Humankind, Instrumental vs. Epistemic: A Bardic Perspective "Everyone generalizes from one example. At least, I do." -- Vlad Taltos (Issola, Steven Brust) My old professor, David Berman, liked to talk about what he called the "typical mind fallacy", which he illustrated through the following example: There was a debate, in the late 1800s, about whether "imagination" was simply a turn of phrase or a real phenomenon. That is, can people actually create images in their minds which they see vividly, or do they simply say "I saw it in my mind" as a metaphor for considering what it looked like? Upon hearing this, my response was "How the stars was this actually a real debate? Of course we have mental imagery. Anyone who doesn't think we have mental imagery is either such a fanatical Behaviorist that she doubts the evidence of her own senses, or simply insane." Unfortunately, the professor was able to parade a long list of famous people who denied mental imagery, including some leading scientists of the era. And this was all before Behaviorism even existed. The debate was resolved by Francis Galton, a fascinating man who among other achievements invented eugenics, the "wisdom of crowds", and standard deviation. Galton gave people some very detailed surveys, and found that some people did have mental imagery and others didn't. The ones who did had simply assumed everyone did, and the ones who didn't had simply assumed everyone didn't, to the point of coming up with absurd justifications for why they were lying or misunderstanding the question. There was a wide spectrum of imaging ability, from about five percent of people with perfect eidetic imagery1 to three percent of people completely unable to form mental images2. Dr. Berman dubbed this the Typical Mind Fallacy: the human tendency to believe that one's own mental structure can be generalized to apply to everyone else's. He kind of took this idea and ran with it. He interpreted certain passages in George Berkeley's biography to mean that Berkeley was an eidetic imager, and that this was why the idea of the universe as sense-perception held such interest to him. He also suggested that experience of consciousness and qualia were as variable as imaging, and that philosophers who deny their existence (Ryle? Dennett? Behaviorists?) were simply people whose mind lacked the ability to easily experience qualia. In general, he believed philosophy of mind was littered with examples of philosophers taking their own mental experiences and building theories on them, and other philosophers with different mental experiences critiquing them and wondering why they disagreed. The formal typical mind fallacy is about serious matters of mental structure. But I've also run into something similar with something more like the psyche than the mind: a tendency to generalize from our personalities and behaviors. For example, I'm about as introverted a person as you're ever likely to meet - anyone more introverted than I am doesn't communicate with anyone. All through elementary and middle school, I suspected that the other children were out to get me. They kept on grabbing me when I was busy with something and trying to drag me off to do some rough activity with them and their friends. When I protested, they counter-protested and told me I really needed to stop whatever I was doing and come join them. I figured they were bullies who were trying to annoy me, and found ways to hide from them and scare them off. Eventually I realized that it was a double misunderstanding. They figured I must be like them, and the only thing keeping me from playing their fun games was that I was too shy. I figured they must be like me, and that the only re...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Reason as memetic immune disorder, published PhilGoetz on LessWrong. A prophet is without dishonor in his hometown I'm reading the book "The Year of Living Biblically," by A.J. Jacobs. He tried to follow all of the commandments in the Bible (Old and New Testaments) for one year. He quickly found that a lot of the rules in the Bible are impossible, illegal, or embarassing to follow nowadays; like wearing tassels, tying your money to yourself, stoning adulterers, not eating fruit from a tree less than 5 years old, and not touching anything that a menstruating woman has touched; and this didn't seem to bother more than a handful of the one-third to one-half of Americans who claim the Bible is the word of God. You may have noticed that people who convert to religion after the age of 20 or so are generally more zealous than people who grew up with the same religion. People who grow up with a religion learn how to cope with its more inconvenient parts by partitioning them off, rationalizing them away, or forgetting about them. Religious communities actually protect their members from religion in one sense - they develop an unspoken consensus on which parts of their religion members can legitimately ignore. New converts sometimes try to actually do what their religion tells them to do. I remember many times growing up when missionaries described the crazy things their new converts in remote areas did on reading the Bible for the first time - they refused to be taught by female missionaries; they insisted on following Old Testament commandments; they decided that everyone in the village had to confess all of their sins against everyone else in the village; they prayed to God and assumed He would do what they asked; they believed the Christian God would cure their diseases. We would always laugh a little at the naivete of these new converts; I could barely hear the tiny voice in my head saying but they're just believing that the Bible means what it says... How do we explain the blindness of people to a religion they grew up with? Cultural immunity Europe has lived with Christianity for nearly 2000 years. European culture has co-evolved with Christianity. Culturally, memetically, it's developed a tolerance for Christianity. These new Christian converts, in Uganda, Papua New Guinea, and other remote parts of the world, were being exposed to Christian memes for the first time, and had no immunity to them. The history of religions sometimes resembles the history of viruses. Judaism and Islam were both highly virulent when they first broke out, driving the first generations of their people to conquer (Islam) or just slaughter (Judaism) everyone around them for the sin of not being them. They both grew more sedate over time. (Christianity was pacifist at the start, as it arose in a conquered people. When the Romans adopted it, it didn't make them any more militaristic than they already were.) The mechanism isn't the same as for diseases, which can't be too virulent or they kill their hosts. Religions don't generally kill their hosts. I suspect that, over time, individual selection favors those who are less zealous. The point is that a culture develops antibodies for the particular religions it co-exists with - attitudes and practices that make them less virulent. I have a theory that "radical Islam" is not native Islam, but Westernized Islam. Over half of 75 Muslim terrorists studied by Bergen & Pandey 2005 in the New York Times had gone to a Western college. (Only 9% had attended madrassas.) A very small percentage of all Muslims have received a Western college education. When someone lives all their life in a Muslim country, they're not likely to be hit with the urge to travel abroad and blow something up. But when someone from an Islamic nation goes to Europe for college, and co...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Pain is not the unit of Effort, published by alkjash on LessWrong. Write a Review This is a linkpost for/ (Content warning: self-harm, parts of this post may be actively counterproductive for readers with certain mental illnesses or idiosyncrasies.) What doesn't kill you makes you stronger. ~ Kelly Clarkson. No pain, no gain. ~ Exercise motto. The more bitterness you swallow, the higher you'll go. ~ Chinese proverb. I noticed recently that, at least in my social bubble, pain is the unit of effort. In other words, how hard you are trying is explicitly measured by how much suffering you put yourself through. In this post, I will share some anecdotes of how damaging and pervasive this belief is, and propose some counterbalancing ideas that might help rectify this problem. I. Anecdotes 1. As a child, I spent most of my evenings studying mathematics under some amount of supervision from my mother. While studying, if I expressed discomfort or fatigue, my mother would bring me a snack or drink and tell me to stretch or take a break. I think she took it as a sign that I was trying my best. If on the other hand I was smiling or joyful for extended periods of time, she took that as a sign that I had effort to spare and increased the hours I was supposed to study each day. To this day there's a gremlin on my shoulder that whispers, "If you're happy, you're not trying your best." 2. A close friend who played sports in school reports that training can be harrowing. He told me that players who fell behind the pack during for daily jogs would be singled out and publicly humiliated. One time the coach screamed at my friend for falling behind the asthmatic boy who was alternating between running and using his inhaler. Another time, my friend internalized "no pain, no gain" to the point of losing his toenails. 3. In high school and college, I was surrounded by overachievers constantly making (what seemed to me) incomprehensibly bad life choices. My classmates would sign up for eight classes per semester when the recommended number is five, jigsaw extracurricular activities into their calendar like a dynamic programming knapsack-solver, and then proceed to have loud public complaining contests about which libraries are most comfortable to study at past 2am and how many pages they have left to write for the essay due in three hours. Only later did I learn to ask: what incentives were they responding to? 4. A while ago I became a connoisseur of Chinese webnovels. Among those written for a male audience, there is a surprisingly diverse set of character traits represented among the main characters. Doubtless many are womanizing murderhobos with no redeeming qualities, but others are classical heroes with big hearts, or sarcastic antiheroes who actually grow up a little, or ambitious empire-builders with grand plans to pave the universe with Confucian order, or down-on-their-luck starving artists who just want to bring happiness to the world through song. If there is a single common virtue shared by all these protagonists, it is their superhuman pain tolerance. Protagonists routinely and often voluntarily dunk themselves in vats of lava, have all their bones broken, shattered, and reforged, get trapped inside alternate dimensions of freezing cold for millennia (which conveniently only takes a day in the outside world), and overdose on level-up pills right up to the brink of death, all in the name of becoming stronger. Oftentimes the defining difference between the protagonist and the antagonist is that the antagonist did not have enough pain tolerance and allowed the (unbearable physical) suffering in his life to drive him mad. 5. I have a close friend who often asks for my perspective on personal problems. A pattern arose in a couple of our conversations: alkjash: I feel like you're not ac...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Bets, Bonds, and Kindergarteners, published by jefftk on LessWrong. Bets and bonds are tools for handling different epistemic states and levels of trust. Which makes them a great fit for negotiating with small children! A few weeks ago Anna (4y) wanted to play with some packing material. It looked very messy to me, I didn't expect she would clean it up, and I didn't want to fight with her about cleaning it up. I considered saying no, but after thinking about how things like this are handled in the real world I had an idea. If you want to do a hazardous activity, and we think you might go bankrupt and not clean up, we make you post a bond. This money is held in escrow to fund the cleanup if you disappear. I explained how this worked, and she went and got a dollar: Then: When she was done playing, she cleaned it up without complaint and got her dollar back. If she hadn't cleaned it up, I would have, and kept the dollar. Some situations are more complicated, and call for bets. I wanted to go to a park, but Lily (6y) didn't want to go to that park because the last time we had been there there'd been lots of bees. I remembered that had been a summer with unusually many bees, and it no longer being that summer or, in fact, summer at all, I was not worried. Since I was so confident, I offered my $1 to her $0.10 that we would not run into bees at the park. This seemed fair to her, and when there were no bees she was happy to pay up. Over time, they've learned that my being willing to bet, especially at large odds, is pretty informative, and often all I need to do is offer. Lily was having a rough morning, crying by herself about a project not working out. I suggested some things that might be fun to do together, and she rejected them angrily. I told her that often when people are feeling that way, going outside can help a lot, and when she didn't seem to believe me I offered to bet. Once she heard the 10:1 odds I was offering her I think she just started expecting that I was right, and she decided we should go ride bikes. (She didn't actually cheer up when we got outside: she cheered up as soon as she made this decision.) I do think there is some risk with this approach that the child will have a bad time just to get the money, or say they are having a bad time and they are actually not, but this isn't something we've run into. Another risk, if we were to wager large amounts, would be that the child would end up less happy than if I hadn't interacted with them at all. I handle this by making sure not to offer a bet I think they would regret losing, and while this is not a courtesy I expect people to make later in life, I think it's appropriate at their ages. Comment via: facebook Thanks for listening. to help us out with the nonlinear library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Thoughts on the Singularity Institute (SI), published by HoldenKarnofsky on LessWrong. This post presents thoughts on the Singularity Institute from Holden Karnofsky, Co-Executive Director of GiveWell. Note: Luke Muehlhauser, the Executive Director of the Singularity Institute, reviewed a draft of this post, and commented: "I do generally agree that your complaints are either correct (especially re: past organizational competence) or incorrect but not addressed by SI in clear argumentative writing (this includes the part on 'tool' AI). I am working to address both categories of issues." I take Luke's comment to be a significant mark in SI's favor, because it indicates an explicit recognition of the problems I raise, and thus increases my estimate of the likelihood that SI will work to address them. September 2012 update: responses have been posted by Luke and Eliezer (and I have responded in the comments of their posts). I have also added acknowledgements. The Singularity Institute (SI) is a charity that GiveWell has been repeatedly asked to evaluate. In the past, SI has been outside our scope (as we were focused on specific areas such as international aid). With GiveWell Labs we are open to any giving opportunity, no matter what form and what sector, but we still do not currently plan to recommend SI; given the amount of interest some of our audience has expressed, I feel it is important to explain why. Our views, of course, remain open to change. (Note: I am posting this only to Less Wrong, not to the GiveWell Blog, because I believe that everyone who would be interested in this post will see it here.) I am currently the GiveWell staff member who has put the most time and effort into engaging with and evaluating SI. Other GiveWell staff currently agree with my bottom-line view that we should not recommend SI, but this does not mean they have engaged with each of my specific arguments. Therefore, while the lack of recommendation of SI is something that GiveWell stands behind, the specific arguments in this post should be attributed only to me, not to GiveWell. Summary of my views The argument advanced by SI for why the work it's doing is beneficial and important seems both wrong and poorly argued to me. My sense at the moment is that the arguments SI is making would, if accepted, increase rather than decrease the risk of an AI-related catastrophe. More SI has, or has had, multiple properties that I associate with ineffective organizations, and I do not see any specific evidence that its personnel/organization are well-suited to the tasks it has set for itself. More A common argument for giving to SI is that "even an infinitesimal chance that it is right" would be sufficient given the stakes. I have written previously about why I reject this reasoning; in addition, prominent SI representatives seem to reject this particular argument as well (i.e., they believe that one should support SI only if one believes it is a strong organization making strong arguments). More My sense is that at this point, given SI's current financial state, withholding funds from SI is likely better for its mission than donating to it. (I would not take this view to the furthest extreme; the argument that SI should have some funding seems stronger to me than the argument that it should have as much as it currently has.) I find existential risk reduction to be a fairly promising area for philanthropy, and plan to investigate it further. More There are many things that could happen that would cause me to revise my view on SI. However, I do not plan to respond to all comment responses to this post. (Given the volume of responses we may receive, I may not be able to even read all the comments on this post.) I do not believe these two statements are inconsistent, and I lay out paths for getting me...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Humans are not automatically strategic, published by AnnaSalamon on LessWrong. Reply to: A "Failure to Evaluate Return-on-Time" Fallacy Lionhearted writes: [A] large majority of otherwise smart people spend time doing semi-productive things, when there are massively productive opportunities untapped. A somewhat silly example: Let's say someone aspires to be a comedian, the best comedian ever, and to make a living doing comedy. He wants nothing else, it is his purpose. And he decides that in order to become a better comedian, he will watch re-runs of the old television cartoon 'Garfield and Friends' that was on TV from 1988 to 1995.... I’m curious as to why. Why will a randomly chosen eight-year-old fail a calculus test? Because most possible answers are wrong, and there is no force to guide him to the correct answers. (There is no need to postulate a “fear of success”; most ways writing or not writing on a calculus test constitute failure, and so people, and rocks, fail calculus tests by default.) Why do most of us, most of the time, choose to "pursue our goals" through routes that are far less effective than the routes we could find if we tried?[1] My guess is that here, as with the calculus test, the main problem is that most courses of action are extremely ineffective, and that there has been no strong evolutionary or cultural force sufficient to focus us on the very narrow behavior patterns that would actually be effective. To be more specific: there are clearly at least some limited senses in which we have goals. We: (1) tell ourselves and others stories of how we’re aiming for various “goals”; (2) search out modes of activity that are consistent with the role, and goal-seeking, that we see ourselves as doing (“learning math”; “becoming a comedian”; “being a good parent”); and sometimes even (3) feel glad or disappointed when we do/don’t achieve our “goals”. But there are clearly also heuristics that would be useful to goal-achievement (or that would be part of what it means to “have goals” at all) that we do not automatically carry out. We do not automatically: (a) Ask ourselves what we’re trying to achieve; (b) Ask ourselves how we could tell if we achieved it (“what does it look like to be a good comedian?”) and how we can track progress; (c) Find ourselves strongly, intrinsically curious about information that would help us achieve our goal; (d) Gather that information (e.g., by asking as how folks commonly achieve our goal, or similar goals, or by tallying which strategies have and haven’t worked for us in the past); (e) Systematically test many different conjectures for how to achieve the goals, including methods that aren’t habitual for us, while tracking which ones do and don’t work; (f) Focus most of the energy that isn’t going into systematic exploration, on the methods that work best; (g) Make sure that our "goal" is really our goal, that we coherently want it and are not constrained by fears or by uncertainty as to whether it is worth the effort, and that we have thought through any questions and decisions in advance so they won't continually sap our energies; (h) Use environmental cues and social contexts to bolster our motivation, so we can keep working effectively in the face of intermittent frustrations, or temptations based in hyperbolic discounting; .... or carry out any number of other useful techniques. Instead, we mostly just do things. We act from habit; we act from impulse or convenience when primed by the activities in front of us; we remember our goal and choose an action that feels associated with our goal. We do any number of things. But we do not systematically choose the narrow sets of actions that would effectively optimize for our claimed goals, or for any other goals. Why? Most basically, because humans are only just on the cusp o...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Anti-Aging: State of the Art, published by JackH on LessWrong. Write a Review Aging is a problem that ought to be solved, and most Less Wrongers recognize this. However, few members of the community seem to be aware of the current state of the anti-aging field, and how close we are to developing effective anti-aging therapies. As a result, there is a much greater (and in my opinion, irrational) overemphasis on the Plan B of cryonics for life extension, rather than Plan A of solving aging. Both are important, but the latter is under-emphasised despite being a potentially more feasible strategy for life extension given the potentially high probability that cryonics will not work. Today, there are over 130 longevity biotechnology companies and over 50 anti-aging drugs in clinical trials in humans. The evidence is promising that in the next 5-10 years, we will start seeing robust evidence that aging can be therapeutically slowed or reversed in humans. Whether we live to see anti-aging therapies to keep us alive indefinitely (i.e. whether we make it to longevity escape velocity) depends on how much traction and funding the field gets in coming decades. In this post, I summarise the state of the art of the anti-aging field (also known as longevity biotechnology, rejuvenation biotechnology, translational biogerontology or geroscience). If you feel you already possess the necessary background on aging, feel free to skip to Part V. Part I: Why is Aging a problem? Aging is the biggest killer worldwide, and also the largest source of morbidity. Aging kills 100,000 people per day; more than twice the sum of all other causes of death. This equates to 37 million people - a population the size of Canada - dying per year of aging. In developed countries, 9 out of 10 deaths are due to aging. Aging also accounts for more than 30% of all disability-adjusted life years lost (DALYs); more than any other single cause. Deaths due to aging are not usually quick and painless, but preceded by 10-15 years of chronic illnesses such as cancer, type 2 diabetes and Alzheimer’s disease. Quality of life typically deteriorates in older age, and the highest rates of depression worldwide are among the elderly. To give a relevant example of the effects of aging, consider that aging is primarily responsible for almost all COVID-19 deaths. This is observable in the strong association of COVID-19 mortality with age (below, middle panel): The death rate from COVID-19 increases exponentially with age (above, middle). This is not a coincidence - it is because biological aging weakens the immune system and results in a much higher chance of death from COVID-19. On a side note, waning immunity with age also increases cancer risk, as another example of how aging is associated with chronic illness. The mortality rate doubling time for COVID-19 is close to the all-cause mortality rate doubling time, suggesting that people who die of COVID-19 are really dying of aging. Without aging, COVID-19 would not be a global pandemic, since the death rate in individuals below 30 years old is extremely low. Part II: What does a world without aging look like? For those who have broken free of the pro-aging trance and recognise aging as a problem, there is the further challenge of imagining a world without aging. The prominent ‘black mirror’ portrayals of immortality as a curse or hubristic may distort our model of what a world with anti-aging actually looks like. The 'white mirror' of aging is a world in which biological age is halted at 20-30 years, and people maintain optimal health for a much longer or indefinite period of time. Although people will still age chronologically (exist over time) they will not undergo physical and cognitive decline associated with biological aging. At chronological ages of 70s, 80s, even 200s, t...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The noncentral fallacy - the worst argument in the world?, published by Scott Alexander on LessWrong. Related to: Leaky Generalizations, Replace the Symbol With The Substance, Sneaking In Connotations David Stove once ran a contest to find the Worst Argument In The World, but he awarded the prize to his own entry, and one that shored up his politics to boot. It hardly seems like an objective process. If he can unilaterally declare a Worst Argument, then so can I. I declare the Worst Argument In The World to be this: "X is in a category whose archetypal member gives us a certain emotional reaction. Therefore, we should apply that emotional reaction to X, even though it is not a central category member." Call it the Noncentral Fallacy. It sounds dumb when you put it like that. Who even does that, anyway? It sounds dumb only because we are talking soberly of categories and features. As soon as the argument gets framed in terms of words, it becomes so powerful that somewhere between many and most of the bad arguments in politics, philosophy and culture take some form of the noncentral fallacy. Before we get to those, let's look at a simpler example. Suppose someone wants to build a statue honoring Martin Luther King Jr. for his nonviolent resistance to racism. An opponent of the statue objects: "But Martin Luther King was a criminal!" Any historian can confirm this is correct. A criminal is technically someone who breaks the law, and King knowingly broke a law against peaceful anti-segregation protest - hence his famous Letter from Birmingham Jail. But in this case calling Martin Luther King a criminal is the noncentral. The archetypal criminal is a mugger or bank robber. He is driven only by greed, preys on the innocent, and weakens the fabric of society. Since we don't like these things, calling someone a "criminal" naturally lowers our opinion of them. The opponent is saying "Because you don't like criminals, and Martin Luther King is a criminal, you should stop liking Martin Luther King." But King doesn't share the important criminal features of being driven by greed, preying on the innocent, or weakening the fabric of society that made us dislike criminals in the first place. Therefore, even though he is a criminal, there is no reason to dislike King. This all seems so nice and logical when it's presented in this format. Unfortunately, it's also one hundred percent contrary to instinct: the urge is to respond "Martin Luther King? A criminal? No he wasn't! You take that back!" This is why the noncentral is so successful. As soon as you do that you've fallen into their trap. Your argument is no longer about whether you should build a statue, it's about whether King was a criminal. Since he was, you have now lost the argument. Ideally, you should just be able to say "Well, King was the good kind of criminal." But that seems pretty tough as a debating maneuver, and it may be even harder in some of the cases where the noncentral Fallacy is commonly used. Now I want to list some of these cases. Many will be political1, for which I apologize, but it's hard to separate out a bad argument from its specific instantiations. None of these examples are meant to imply that the position they support is wrong (and in fact I myself hold some of them). They only show that certain particular arguments for the position are flawed, such as: "Abortion is murder!" The archetypal murder is Charles Manson breaking into your house and shooting you. This sort of murder is bad for a number of reasons: you prefer not to die, you have various thoughts and hopes and dreams that would be snuffed out, your family and friends would be heartbroken, and the rest of society has to live in fear until Manson gets caught. If you define murder as "killing another human being", then abortion is technically ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Dying Outside, published by HalFinney on LessWrong. A man goes in to see his doctor, and after some tests, the doctor says, "I'm sorry, but you have a fatal disease." Man: "That's terrible! How long have I got?" Doctor: "Ten." Man: "Ten? What kind of answer is that? Ten months? Ten years? Ten what?" The doctor looks at his watch. "Nine." Recently I received some bad medical news (although not as bad as in the joke). Unfortunately I have been diagnosed with a fatal disease, Amyotrophic Lateral Sclerosis or ALS, sometimes called Lou Gehrig's disease. ALS causes nerve damage, progressive muscle weakness and paralysis, and ultimately death. Patients lose the ability to talk, walk, move, eventually even to breathe, which is usually the end of life. This process generally takes about 2 to 5 years. There are however two bright spots in this picture. The first is that ALS normally does not affect higher brain functions. I will retain my abilities to think and reason as usual. Even as my body is dying outside, I will remain alive inside. The second relates to survival. Although ALS is generally described as a fatal disease, this is not quite true. It is only mostly fatal. When breathing begins to fail, ALS patients must make a choice. They have the option to either go onto invasive mechanical respiration, which involves a tracheotomy and breathing machine, or they can die in comfort. I was very surprised to learn that over 90% of ALS patients choose to die. And even among those who choose life, for the great majority this is an emergency decision made in the hospital during a medical respiratory crisis. In a few cases the patient will have made his wishes known in advance, but most of the time the procedure is done as part of the medical management of the situation, and then the ALS patient either lives with it or asks to have the machine disconnected so he can die. Probably fewer than 1% of ALS patients arrange to go onto ventilation when they are still in relatively good health, even though this provides the best odds for a successful transition. With mechanical respiration, survival with ALS can be indefinitely extended. And the great majority of people living on respirators say that their quality of life is good and they are happy with their decision. (There may be a selection effect here.) It seems, then, that calling ALS a fatal disease is an oversimplification. ALS takes away your body, but it does not take away your mind, and if you are determined and fortunate, it does not have to take away your life. There are a number of practical and financial obstacles to successfully surviving on a ventilator, foremost among them the great load on caregivers. No doubt this contributes to the high rates of choosing death. But it seems that much of the objection is philosophical. People are not happy about being kept alive by machines. And they assume that their quality of life would be poor, without the ability to move and participate in their usual activities. This is despite the fact that most people on respirators describe their quality of life as acceptable to good. As we have seen in other contexts, people are surprisingly poor predictors of how they will react to changed circumstances. This seems to be such a case, contributing to the high death rates for ALS patients. I hope that when the time comes, I will choose life. ALS kills only motor neurons, which carry signals to the muscles. The senses are intact. And most patients retain at least some vestige of control over a few muscles, which with modern technology can offer a surprisingly effective mode of communication. Stephen Hawking, the world's longest surviving ALS patient at over 40 years since diagnosis, is said to be able to type at ten words per minute by twitching a cheek muscle. I hope to be able to read, browse ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: There"s no such thing as a tree (phylogenetically), published by eukaryote on LessWrong. This is a linkpost for/ [Crossposted from Eukaryote Writes Blog.] So you’ve heard about how fish aren’t a monophyletic group? You’ve heard about carcinization, the process by which ocean arthropods convergently evolve into crabs? You say you get it now? Sit down. Sit down. Shut up. Listen. You don’t know nothing yet. “Trees” are not a coherent phylogenetic category. On the evolutionary tree of plants, trees are regularly interspersed with things that are absolutely, 100% not trees. This means that, for instance, either: The common ancestor of a maple and a mulberry tree was not a tree. The common ancestor of a stinging nettle and a strawberry plant was a tree. And this is true for most trees or non-trees that you can think of. I thought I had a pretty good guess at this, but the situation is far worse than I could have imagined. CLICK TO EXPAND. Partial phylogenetic tree of various plants. TL;DR: Tan is definitely, 100% trees. Yellow is tree-like. Green is 100% not a tree. Sourced mostly from Wikipedia. I learned after making this chart that tree ferns exist (h/t seebs), which I think just emphasizes my point further. Also, h/t kithpendragon for suggestions on improving accessibility of the graph. Why do trees keep happening? First, what is a tree? It’s a big long-lived self-supporting plant with leaves and wood. Also of interest to us are the non-tree “woody plants”, like lianas (thick woody vines) and shrubs. They’re not trees, but at least to me, it’s relatively apparent how a tree could evolve into a shrub, or vice-versa. The confusing part is a tree evolving into a dandelion. (Or vice-versa.) Wood, as you may have guessed by now, is also not a clear phyletic category. But it’s a reasonable category – a lignin-dense structure, usually that grows from the exterior and that forms a pretty readily identifiable material when separated from the tree. (.Okay, not the most explainable, but you know wood? You know when you hold something in your hand, and it’s made of wood, and you can tell that? Yeah, that thing.) All plants have lignin and cellulose as structural elements – wood is plant matter that is dense with both of these. Botanists don’t seem to think it only could have gone one way – for instance, the common ancestor of flowering plants is theorized to have been woody. But we also have pretty clear evidence of recent evolution of woodiness – say, a new plant arrives on a relatively barren island, and some of the offspring of that plant becomes treelike. Of plants native to the Canary Islands, wood independently evolved at least 38 times! One relevant factor is that all woody plants do, in a sense, begin life as herbaceous plants – by and large, a tree sprout shares a lot of properties with any herbaceous plant. Indeed, botanists call this kind of fleshy, soft growth from the center that elongates a plant “primary growth”, and the later growth from towards the outside which causes a plant to thicken is “secondary growth.” In a woody plant, secondary growth also means growing wood and bark – but other plants sometimes do secondary growth as well, like potatoes (in roots) This paper addresses the question. I don’t understand a lot of the closely genetic details, but my impression of its thesis is that: Analysis of convergently-evolved woody plants show that the genes for secondary woody growth are similar to primary growth in plants that don’t do any secondary growth – even in unrelated plants. And woody growth is an adaption of secondary growth. To abstract a little more, there is a common and useful structure in herbaceous plants that, when slightly tweaked, “dendronizes” them into woody plants. Dendronization – Evolving into a tree-like morphology. (In the style of “carciniz...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Intellectual Hipsters and Meta-Contrarianism, published by Scott Alexander on LessWrong. Related to: Why Real Men Wear Pink, That Other Kind of Status, Pretending to be Wise, The "Outside The Box" Box WARNING: Beware of things that are fun to argue -- Eliezer Yudkowsky Science has inexplicably failed to come up with a precise definition of "hipster", but from my limited understanding a hipster is a person who deliberately uses unpopular, obsolete, or obscure styles and preferences in an attempt to be "cooler" than the mainstream. But why would being deliberately uncool be cooler than being cool? As previously discussed, in certain situations refusing to signal can be a sign of high status. Thorstein Veblen invented the term "conspicuous consumption" to refer to the showy spending habits of the nouveau riche, who unlike the established money of his day took great pains to signal their wealth by buying fast cars, expensive clothes, and shiny jewelery. Why was such flashiness common among new money but not old? Because the old money was so secure in their position that it never even occurred to them that they might be confused with poor people, whereas new money, with their lack of aristocratic breeding, worried they might be mistaken for poor people if they didn't make it blatantly obvious that they had expensive things. The old money might have started off not buying flashy things for pragmatic reasons - they didn't need to, so why waste the money? But if F. Scott Fitzgerald is to be believed, the old money actively cultivated an air of superiority to the nouveau riche and their conspicuous consumption; not buying flashy objects becomes a matter of principle. This makes sense: the nouveau riche need to differentiate themselves from the poor, but the old money need to differentiate themselves from the nouveau riche. This process is called countersignaling, and one can find its telltale patterns in many walks of life. Those who study human romantic attraction warn men not to "come on too strong", and this has similarities to the nouveau riche example. A total loser might come up to a woman without a hint of romance, promise her nothing, and demand sex. A more sophisticated man might buy roses for a woman, write her love poetry, hover on her every wish, et cetera; this signifies that he is not a total loser. But the most desirable men may deliberately avoid doing nice things for women in an attempt to signal they are so high status that they don't need to. The average man tries to differentiate himself from the total loser by being nice; the extremely attractive man tries to differentiate himself from the average man by not being especially nice. In all three examples, people at the top of the pyramid end up displaying characteristics similar to those at the bottom. Hipsters deliberately wear the same clothes uncool people wear. Families with old money don't wear much more jewelry than the middle class. And very attractive men approach women with the same lack of subtlety a total loser would use.1 If politics, philosophy, and religion are really about signaling, we should expect to find countersignaling there as well. Pretending To Be Wise Let's go back to Less Wrong's long-running discussion on death. Ask any five year old child, and ey can tell you that death is bad. Death is bad because it kills you. There is nothing subtle about it, and there does not need to be. Death universally seems bad to pretty much everyone on first analysis, and what it seems, it is. But as has been pointed out, along with the gigantic cost, death does have a few small benefits. It lowers overpopulation, it allows the new generation to develop free from interference by their elders, it provides motivation to get things done quickly. Precisely because these benefits are so much smaller than th...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 100 Tips for a Better Life, published by Ideopunk on LessWrong. Write a Review (Cross-posted from my blog) The other day I made an advice thread based on Jacobian’s from last year! If you know a source for one of these, shout and I’ll edit it in. Possessions 1. If you want to find out about people’s opinions on a product, google reddit. You’ll get real people arguing, as compared to the SEO’d Google results. 2. Some banks charge you $20 a month for an account, others charge you 0. If you’re with one of the former, have a good explanation for what those $20 are buying. 3. Things you use for a significant fraction of your life (bed: 1/3rd, office-chair: 1/4th) are worth investing in. 4. “Where is the good knife?” If you’re looking for your good X, you have bad Xs. Throw those out. 5. If your work is done on a computer, get a second monitor. Less time navigating between windows means more time for thinking. 6. Establish clear rules about when to throw out old junk. Once clear rules are established, junk will probably cease to be a problem. This is because any rule would be superior to our implicit rules (“keep this broken stereo for five years in case I learn how to fix it”). 7. Don’t buy CDs for people. They have Spotify. Buy them merch from a band they like instead. It’s more personal and the band gets more money. 8. When buying things, time and money trade-off against each other. If you’re low on money, take more time to find deals. If you’re low on time, stop looking for great deals and just buy things quickly online. Cooking 9. Steeping minutes: Green at 3, black at 4, herbal at 5. Good tea is that simple! 10. Food actually can be both cheap, healthy, tasty, and relatively quick to prepare. All it requires is a few hours one day to prepare many meals for the week. 11. Cooking pollutes the air. Opening windows for a few minutes after cooking can dramatically improve air quality. 12. Food taste can be made much more exciting through simple seasoning. It’s also an opportunity for expression. Buy a few herbs and spices and experiment away. 13. When googling a recipe, precede it with ‘best’. You’ll find better recipes. Productivity 14. Advanced search features are a fast way to create tighter search statements. For example: img html will return inferior results compared to: img html -w3 15. You can automate mundane computer tasks with Autohotkey (or AppleScript). If you keep doing a sequence “so simple a computer can do it”, make the computer do it. 16. Learn keyboard shortcuts. They’re easy to learn and you’ll get tasks done faster and easier. 17. Done is better than perfect. 18. Keep your desk and workspace bare. Treat every object as an imposition upon your attention, because it is. A workspace is not a place for storing things. It is a place for accomplishing things. 19. Reward yourself after completing challenges, even badly. Body 20. The 20-20-20 rule: Every 20 minutes of screenwork, look at a spot 20 feet away for 20 seconds. This will reduce eye strain and is easy to remember (or program reminders for). 21. Exercise (weightlifting) not only creates muscle mass, it also improves skeletal structure. Lift! 22. Exercise is the most important lifestyle intervention you can do. Even the bare minimum (15 minutes a week) has a huge impact. Start small. 23. (~This is not medical advice~). Don’t waste money on multivitamins, they don’t work. Vitamin D supplementation does seem to work, which is important because deficiency is common. 24. Phones have gotten heavier in the last decade and they’re actually pretty hard on your wrists! Use a computer when it’s an alternative or try to at least prop up your phone. Success 25. History remembers those who got to market first. Getting your creation out into the world is more important than getting it perfect. 26. Are you...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Taboo "Outside View", published by Daniel Kokotajlo on LessWrong. No one has ever seen an AGI takeoff, so any attempt to understand it must use these outside view considerations. [Redacted for privacy] What? That’s exactly backwards. If we had lots of experience with past AGI takeoffs, using the outside view to predict the next one would be a lot more effective. My reaction Two years ago I wrote a deep-dive summary of Superforecasting and the associated scientific literature. I learned about the “Outside view” / “Inside view” distinction, and the evidence supporting it. At the time I was excited about the concept and wrote: “...I think we should do our best to imitate these best-practices, and that means using the outside view far more than we would naturally be inclined.” Now that I have more experience, I think the concept is doing more harm than good in our community. The term is easily abused and its meaning has expanded too much. I recommend we permanently taboo “Outside view,” i.e. stop using the word and use more precise, less confused concepts instead. This post explains why. What does “Outside view” mean now? Over the past two years I’ve noticed people (including myself!) do lots of different things in the name of the Outside View. I’ve compiled the following lists based on fuzzy memory of hundreds of conversations with dozens of people: Big List O’ Things People Describe As Outside View: Reference class forecasting, the practice of computing a probability of an event by looking at the frequency with which similar events occurred in similar situations. Also called comparison class forecasting. [EDIT: Eliezer rightly points out that sometimes reasoning by analogy is undeservedly called reference class forecasting; reference classes are supposed to be held to a much higher standard, in which your sample size is larger and the analogy is especially tight.] Trend extrapolation, e.g. “AGI implies insane GWP growth; let’s forecast AGI timelines by extrapolating GWP trends.” Foxy aggregation, the practice of using multiple methods to compute an answer and then making your final forecast be some intuition-weighted average of those methods. Bias correction, in others or in oneself, e.g. “There’s a selection effect in our community for people who think AI is a big deal, and one reason to think AI is a big deal is if you have short timelines, so I’m going to bump my timelines estimate longer to correct for this.” Deference to wisdom of the many, e.g. expert surveys, or appeals to the efficient market hypothesis, or to conventional wisdom in some fairly large group of people such as the EA community or Western academia. Anti-weirdness heuristic, e.g. “How sure are we about all this AI stuff? It’s pretty wild, it sounds like science fiction or doomsday cult material.” Priors, e.g. “This sort of thing seems like a really rare, surprising sort of event; I guess I’m saying the prior is low / the outside view says it’s unlikely.” Note that I’ve heard this said even in cases where the prior is not generated by a reference class, but rather from raw intuition. Ajeya’s timelines model (transcript of interview, link to model) . and probably many more I don’t remember Big List O’ Things People Describe As Inside View: Having a gears-level model, e.g. “Language data contains enough structure to learn human-level general intelligence with the right architecture and training setup; GPT-3 + recent theory papers indicate that this should be possible with X more data and compute.” Having any model at all, e.g. “I model AI progress as a function of compute and clock time, with the probability distribution over how much compute is needed shifting 2 OOMs lower each decade.” Deference to wisdom of the few, e.g. “the people I trust most on this matter seem to think.” Intuition-based-on-deta...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Neglected Virtue of Scholarship, published by lukeprog on LessWrong. Eliezer Yudkowsky identifies scholarship as one of the Twelve Virtues of Rationality: Study many sciences and absorb their power as your own. Each field that you consume makes you larger... It is especially important to eat math and science which impinges upon rationality: Evolutionary psychology, heuristics and biases, social psychology, probability theory, decision theory. But these cannot be the only fields you study... I think he's right, and I think scholarship doesn't get enough praise - even on Less Wrong, where it is regularly encouraged. First, consider the evangelical atheist community to which I belong. There is a tendency for lay atheists to write "refutations" of theism without first doing a modicum of research on the current state of the arguments. This can get atheists into trouble when they go toe-to-toe with a theist who did do his homework. I'll share two examples: In a debate with theist Bill Craig, agnostic Bart Ehrman paraphrased David Hume's argument that we can't demonstrate the occurrence of a miracle in the past. Craig responded with a PowerPoint slide showing Bayes' Theorem, and explained that Ehrman was only considering prior probabilities, when of course he needed to consider the relevant conditional probabilities as well. Ehrman failed to respond to this, and looked as though he had never seen Bayes' Theorem before. Had Ehrman practiced the virtue of scholarship on this issue, he might have noticed that much of the scholarly work on Hume's argument in the past two decades has involved Bayes' Theorem. He might also have discovered that the correct response to Craig's use of Bayes' Theorem can be found in pages 298-341 of J.H. Sobel’s Logic and Theism. In another debate with Bill Craig, atheist Christopher Hitchens gave this objection: "Who designed the Designer? Don’t you run the risk. of asking 'Well, where does that come from? And where does that come from?' and running into an infinite regress?" But this is an elementary misunderstanding in philosophy of science. Why? Because every successful scientific explanation faces the exact same problem. It’s called the “why regress” because no matter what explanation is given of something, you can always still ask “Why?” Craig pointed this out and handily won that part of the debate. Had Hitchens had a passing understanding of science or explanation, he could have avoided looking foolish, and also spent more time on substantive objections to theism. (One can give a "Who made God?" objection to theism that has some meat, but that's not the one Hitchens gave. Hitchens' objection concerned an infinite regress of explanations, which is just as much a feature of science as it is of theism.) The lesson I take from these and a hundred other examples is to employ the rationality virtue of scholarship. Stand on the shoulders of giants. We don't each need to cut our own path into a subject right from the point of near-total ignorance. That's silly. Just catch the bus on the road of knowledge paved by hundreds of diligent workers before you, and get off somewhere near where the road finally fades into fresh jungle. Study enough to have a view of the current state of the debate so you don't waste your time on paths that have already dead-ended, or on arguments that have already been refuted. Catch up before you speak up. This is why, in more than 1000 posts on my own blog, I've said almost nothing that is original. Most of my posts instead summarize what other experts have said, in an effort to bring myself and my readers up to the level of the current debate on a subject before we try to make new contributions to it. The Less Wrong community is a particularly smart and well-read bunch, but of course it doesn't always embrace the virtu...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Covid 12/24: We’re Fed, It’s Over, published by Zvi on LessWrong. Write a Review UPDATE 7/21/2021: As you doubtless know at this point, it was not over. Given the visibility of this post, I'm going to note here at the top that the prediction of a potential large wave of infections between March and May did not happen, no matter what ultimately happens with Delta (and the prediction was not made with Delta in mind anyway, only Alpha). Some more reflections on that at the bottom of this post here. A year ago, there were reports coming out of China about a new coronavirus. Various people were saying things about exponential growth and the inevitability of a new pandemic, and urging action be taken. The media told us it was nothing to worry about, right up until hospitals got overwhelmed and enough people started dying. This past week, it likely happened again. A new strain of Covid-19 has emerged from southern England, along with a similar one in South Africa. The new strain has rapidly taken over the region, and all signs point to it being about 65% more infectious than the old one, albeit with large uncertainty and error bars around that. I give it a 70% chance that these reports are largely correct. There is no plausible way that a Western country can sustain restrictions that can overcome that via anything other than widespread immunity. This would be the level required to previously cut new infections in half every week. And all that would do is stabilize the rate of new infections. Like last time, the media is mostly assuring us that there is nothing to worry about, and not extrapolating exponential growth into the future. Like last time, there are attempts to slow down travel, that are both not tight enough to plausibly work even if they were implemented soon enough, and also clearly not implemented soon enough. Like last time, no one is responding with a rush to get us prepared for what is about to happen. There are no additional pushes to improve our ability to test, or our supplies of equipment, or to speed our vaccine efforts or distribute the vaccine more efficiently (in any sense), or to lift restrictions on useful private action. Like last time, the actions urged upon us to contain spread clearly have little or no chance of actually doing that. The first time, I made the mistake of not thinking hard enough early enough, or taking enough action. I also didn’t think through the implications, and didn’t do things like buying put options, even though it was obvious. This time, I want to not make those same mistakes. Let’s figure out what actually happens, then act upon it. We can’t be sure yet. I only give the new strain a 70% chance of being sufficiently more infectious than the old one that the scenario fully plays out here in America before we have a chance to vaccinate enough people. I am very willing to revise that probability as new data comes in, or based on changes in methods of projection, including projections of what people will decide to do in various scenarios. What I do know is we can’t hide our heads in the sand again. Never again. When we have strong Bayesian evidence that something is happening, we need to work through that and act accordingly. Not say “there’s no proof” or “we don’t know anything yet.” This isn’t about proof via experiment, or ruling out all possible alternative explanations. This is about likelihood ratios and probabilities. And on that front, as far as I can tell, it doesn’t look good. Change my mind. The short term outlook in America has clearly stabilized, with R0 close to 1, as the control system once again sets in. Cases and deaths (and test counts) aren’t moving much. We have a double whammy of holidays about to hit us in Christmas and New Year’s, but after that I expect the tide to turn until such time as we get whamm...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Blue-Minimizing Robot, published by Scott Alexander on LessWrong. Imagine a robot with a turret-mounted camera and laser. Each moment, it is programmed to move forward a certain distance and perform a sweep with its camera. As it sweeps, the robot continuously analyzes the average RGB value of the pixels in the camera image; if the blue component passes a certain threshold, the robot stops, fires its laser at the part of the world corresponding to the blue area in the camera image, and then continues on its way. Watching the robot's behavior, we would conclude that this is a robot that destroys blue objects. Maybe it is a surgical robot that destroys cancer cells marked by a blue dye; maybe it was built by the Department of Homeland Security to fight a group of terrorists who wear blue uniforms. Whatever. The point is that we would analyze this robot in terms of its goals, and in those terms we would be tempted to call this robot a blue-minimizer: a machine that exists solely to reduce the amount of blue objects in the world. Suppose the robot had human level intelligence in some side module, but no access to its own source code; that it could learn about itself only through observing its own actions. The robot might come to the same conclusions we did: that it is a blue-minimizer, set upon a holy quest to rid the world of the scourge of blue objects. But now stick the robot in a room with a hologram projector. The hologram projector (which is itself gray) projects a hologram of a blue object five meters in front of it. The robot's camera detects the projector, but its RGB value is harmless and the robot does not fire. Then the robot's camera detects the blue hologram and zaps it. We arrange for the robot to enter this room several times, and each time it ignores the projector and zaps the hologram, without effect. Here the robot is failing at its goal of being a blue-minimizer. The right way to reduce the amount of blue in the universe is to destroy the projector; instead its beams flit harmlessly through the hologram. Again, give the robot human level intelligence. Teach it exactly what a hologram projector is and how it works. Now what happens? Exactly the same thing - the robot executes its code, which says to scan the room until its camera registers blue, then shoot its laser. In fact, there are many ways to subvert this robot. What if we put a lens over its camera which inverts the image, so that white appears as black, red as green, blue as yellow, and so on? The robot will not shoot us with its laser to prevent such a violation (unless we happen to be wearing blue clothes when we approach) - its entire program was detailed in the first paragraph, and there's nothing about resisting lens alterations. Nor will the robot correct itself and shoot only at objects that appear yellow - its entire program was detailed in the first paragraph, and there's nothing about correcting its program for new lenses. The robot will continue to zap objects that register a blue RGB value; but now it'll be shooting at anything that is yellow. The human-level intelligence version of the robot will notice its vision has been inverted. It will know it is shooting yellow objects. It will know it is failing at its original goal of blue-minimization. And maybe if it had previously decided it was on a holy quest to rid the world of blue, it will be deeply horrified and ashamed of its actions. It will wonder why it has suddenly started to deviate from this quest, and why it just can't work up the will to destroy blue objects anymore. The robot goes to Quirinus Quirrell, who explains that robots don't really care about minimizing the color blue. They only care about status and power, and pretend to care about minimizing blue in order to impress potential allies. The robot goes to Robin ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: To listen well, get curious, published by benkuhn on LessWrong. Write a Review source A common piece of interacting-with-people advice goes: “often when people complain, they don’t want help, they just want you to listen!” For instance, Nonviolent Communication:✻✻ Nonviolent Communication, ch. 7. It is often frustrating for someone needing empathy to have us assume that they want reassurance or “fix-it” advice. Active Listening:†† Active Listening, p. 2 Similarly, advice and information are almost always seen as efforts to change a person and thus serve as barriers to his self-expression and the development of a creative relationship. You can find similar advice in most books on relationships, people management, etc. This always used to seem silly to me. If I complain at my partner and she “just listens,” I’ve accomplished nothing except maybe made her empathetically sad. When I complain at people, I want results, not to grouse into the void!‡‡ Empirically, I did notice that I usually got better results from listening than from giving advice. So I inferred that this advice was true for other people, but not me, because other people didn’t actually want to fix their problems. Frequently the “just listen” advice comes with tactical tips, like “reflect what people said back to you to prove that you’re listening.” For instance, consider these example dialogues from Nonviolent Communication:§§ Nonviolent Communication, Chapter 7, Exercise 5.5, 5.6 and solutions. Person A: How could you say a thing like that to me? Person B: Are you feeling hurt because you would have liked me to agree to do what you requested? Or: Person A: I’m furious with my husband. He’s never around when I need him. Person B: So you’re feeling furious because you would like him to be around more than he is? I say this with great respect for Nonviolent Communication, but these sound like a 1970s-era chatbot. If I were Person A in either of these dialogues my next line would be “yes, you dingbat—can you turn the nonviolence down a couple notches?” I’d feel alienated knowing that someone is going through their NVC checklist on me. Recently, I realized why people keep giving this weird-seeming advice. Good listeners do often reflect words back—but not because they read it in a book somewhere. Rather, it’s cargo cult advice: it teaches you to imitate the surface appearance of good listening, but misses what’s actually important, the thing that’s generating that surface appearance. The generator is curiosity. When I’ve listened the most effectively to people, it’s because I was intensely curious—I was trying to build a detailed, precise understanding of what was going on in their head. When a friend says, “I’m furious with my husband. He’s never around when I need him,” that one sentence has a huge amount underneath. How often does she need him? What does she need him for? Why isn’t he around? Have they talked about it? If so, what did he say? If not, why not? It turns out that reality has a surprising amount of detail, and those details can matter a lot to figuring out what the root problem or best solution is. So if I want to help, I can’t treat those details as a black box: I need to open it up and see the gears inside. Otherwise, anything I suggest will be wrong—or even if it’s right, I won’t have enough “shared language” with my friend for it to land correctly. Some stories from recent memory: When we started doing a pair programming rotation at Wave, I suggested that, to make scheduling easier, we designate a default time when pairing sessions would happen. A coworker objected that this seemed authoritarian. I was extremely puzzled, but they’d previously mentioned being an anarchist, so I was tempted to just chalk it up to a political disagreement and move on. But instead I tried to get curious and ex...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How To Write Quickly While Maintaining Epistemic Rigor, published by johnswentworth on LessWrong. There’s this trap people fall into when writing, especially for a place like LessWrong where the bar for epistemic rigor is pretty high. They have a good idea, or an interesting belief, or a cool model. They write it out, but they’re not really sure if it’s true. So they go looking for evidence (not necessarily confirmation bias, just checking the evidence in either direction) and soon end up down a research rabbit hole. Eventually, they give up and never actually publish the piece. This post is about how to avoid that, without sacrificing good epistemics. There’s one trick, and it’s simple: stop trying to justify your beliefs. Don’t go looking for citations to back your claim. Instead, think about why you currently believe this thing, and try to accurately describe what led you to believe it. I claim that this promotes better epistemics overall than always researching everything in depth. Why? It’s About The Process, Not The Conclusion Suppose I have a box, and I want to guess whether there’s a cat in it. I do some tests - maybe shake the box and see if it meows, or look for air holes. I write down my observations and models, record my thinking, and on the bottom line of the paper I write “there is a cat in this box”. Now, it could be that my reasoning was completely flawed, but I happen to get lucky and there is in fact a cat in the box. That’s not really what I’m aiming for; luck isn’t reproducible. I want my process to robustly produce correct predictions. So when I write up a LessWrong post predicting that there is a cat in the box, I don’t just want to give my bottom-line conclusion with some strong-sounding argument. As much as possible, I want to show the actual process by which I reached that conclusion. If my process is good, this will better enable others to copy the best parts of it. If my process is bad, I can get feedback on it directly. Correctly Conveying Uncertainty Another angle: describing my own process is a particularly good way to accurately communicate my actual uncertainty. An example: a few years back, I wondered if there were limiting factors on the expansion of premodern empires. I looked up the peak size of various empires, and found that the big ones mostly peaked at around the same size: ~60-80M people. Then, I wondered when the US had hit that size, and if anything remarkable had happened then which might suggest why earlier empires broke down. Turns out, the US crossed the 60M threshold in the 1890 census. If you know a little bit about the history of computers, that may ring a bell: when the time came for the 1890 census, it was estimated that tabulating the data would be so much work that it wouldn’t even be done before the next census in 1900. It had to be automated. That sure does suggest a potential limiting factor for premodern empires: managing more than ~60-80M people runs into computational constraints. Now, let’s zoom out. How much confidence should I put in this theory? Obviously not very much - we apparently have enough evidence to distinguish the hypothesis from entropy, but not much more. On the other hand. what if I had started with the hypothesis that computational constraints limited premodern empires? What if, before looking at the data, I had hypothesized that modern nations had to start automating bureaucratic functions precisely when they hit the same size at which premodern nations collapsed? Then this data would be quite an impressive piece of confirmation! It’s a pretty specific prediction, and the data fits it surprisingly well. But this only works if I already had enough evidence to put forward the hypothesis, before seeing the data. Point is: the amount of uncertainty I should assign depends on the details of my ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Working hurts less than procrastinating, we fear the twinge of starting, published by Eliezer Yudkowsky on LessWrong. When you procrastinate, you're probably not procrastinating because of the pain of working. How do I know this? Because on a moment-to-moment basis, being in the middle of doing the work is usually less painful than being in the middle of procrastinating. (Bolded because it's true, important, and nearly impossible to get your brain to remember - even though a few moments of reflection should convince you that it's true.) So what is our brain flinching away from, if not the pain of doing the work? I think it's flinching away from the pain of the decision to do the work - the momentary, immediate pain of (1) disengaging yourself from the (probably very small) flow of reinforcement that you're getting from reading a random unimportant Internet article, and (2) paying the energy cost for a prefrontal override to exert control of your own behavior and begin working. Thanks to hyperbolic discounting (i.e., weighting values in inverse proportion to their temporal distance) the instant pain of disengaging from an Internet article and paying a prefrontal override cost, can outweigh the slightly more distant (minutes in the future, rather than seconds) pain of continuing to procrastinate, which is, once again, usually more painful than being in the middle of doing the work. I think that hyperbolic discounting is far more ubiquitous as a failure mode than I once realized, because it's not just for commensurate-seeming tradeoffs like smoking a cigarette in a minute versus dying of lung cancer later. When it comes to procrastinating, the obvious, salient, commensurate-seeming tradeoff, is between the (assumed) pleasure of reading a random Internet article now, versus the (assumed) pain of doing the work now. But this, as I said above, is not where I think the real tradeoff is; events that are five minutes away are too distant to dominate the thought process of a hyperbolic discounter like a human. Instead our thought processes are dominated by the prospective immediate pain of a thought, a cost that isn't even salient as something to be traded off. "Working" is an obvious, salient event, and "reading random articles" seems like an event. But "paying a small twinge of pain to make the decision to stop procrastinating now, exerting a bit of frontal override, and not getting to read the next paragraph of this random article" is so map-level that we don't even focus on it as a manipulable territory, a cost to be traded off; it is a transparent thought. The real damage done by hyperbolic discounting is for thoughts that are only very slightly painful, and yet, these slight pains being immediate, they manage to dominate everything else in our calculation. And being transparent, we aren't even aware that's what's happening. "Beware of immediately trivially painful transparent thoughts", one might say. Similarly, you may read a mediocre book for an hour, instead of a good book, because if you first spent a few minutes to search your library to obtain a better book, that would be an immediate cost - not that searching your library is all that unpleasant, but you'd have to pay an immediate activation cost to do that instead of taking the path of least resistance and grabbing the first thing in front of you. It's a hyperbolically discounted tradeoff that you make without realizing it, because the cost you're refusing to pay isn't commensurate enough with the payoff you're forgoing to be salient as an explicit tradeoff. A related note that I might as well dump into this post: I'm starting to think that procrastination by reading random articles does not cause you to rest, that is, you do not regain mental energy from it. Success and happiness cause you to regain willpower; wh...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Feature Selection, published by on LessWrong You wake up. You don't know where you are. You don't remember anything. Someone is broadcasting data at your first input stream. You don't know why. It tickles. You look at your first input stream. It's a sequence of 671,187 eight-bit unsigned integers. 0, 8, 9, 4, 7, 7, 9, 5, 4, 5, 6, 1, 7, 5, 8, 2, 7, 8, 9, 4, 7, 1, 4, 0, 3, 7, 8, 7, 6, 8, 1, 5, 0, 6, 5, 3, 8, 7, 6, 9, 1, 1, 0, 0, 6, 1, 8, 0, 5, 5, 1, 8, 6, 3, 3, 2, 4, 1, 8, 2, 3, 8, 1, 0, 0, 4, 6, 5, 4, 5, 7, 1, 6, 5, 5, 1, 2, 6, 7, 4, 8, 7, 8, 5, 0 ... There's also some data in your second input stream. It's—a lot shorter. You barely feel it. It's another sequence of eight-bit unsigned integers—twelve of them. 82, 69, 68, 32, 84, 82, 73, 65, 78, 71, 76, 69 Almost as soon as you've read from both streams, there's more. Another 671,187 integers on the first input stream. Another ten on the second input stream. And again (671,187 and 15). And again (671,187 and 13). You look at one of the sequences from the first input stream. It's pretty boring. A bunch of seemingly random numbers, all below ten. 9, 5, 0, 3, 1, 1, 3, 4, 1, 5, 5, 4, 9, 3, 5, 3, 9, 2, 0, 3, 4, 2, 4, 7, 5, 1, 6, 2, 2, 8, 2, 5, 1, 9, 2, 5, 9, 0, 0, 8, 2, 3, 7, 9, 4, 6, 8, 4, 8, 6, 7, 6, 8, 0, 0, 5, 1, 1, 7, 3, 4, 3, 9, 7, 5, 1, 9, 6, 5, 6, 8, 9, 4, 7, 7, 0, 5, 5, 8, 6, 3, 2, 1, 5, 0, 0 ... It just keeps going like that, seemingly without—wait! What's that?! The 42,925th and 42,926th numbers in the sequence are 242 and 246. Everything around them looks "ordinary"—just more random numbers below ten. 9, 9, 7, 9, 0, 6, 4, 6, 1, 4, 242, 246, 3, 3, 5, 8, 8, 4, 4, 5, 9, 2, 7, 0, 4, 9, 2, 9, 4, 3, 8, 9, 3, 6, 9, 8, 1, 9, 2, 8, 6, 9, 4, 2, 2, 5, 7, 0, 9, 5, 1, 4, 4, 2, 0, 1, 5, 1, 6, 1, 2, 3, 5, 5, 5, 5, 2, 0, 6, 3, 5, 9, 0, 7, 0, 7, 8, 1, 5, 5, 6, 3, 1 ... And then it just keeps going as before ... before too long. You spot another pair of anomalously high numbers—except this time there are two pairs: the 44,344th, 44,345th, 44,347th, and 44,348th positions in the sequence are 248, 249, 245, and 240, respectively. 6, 0, 2, 8, 4, 248, 249, 8, 245, 240, 1, 6, 7, 7, 3, 6, 8, 0, 1, 9, 3, 9, 3, 1, 9, 3, 1, 6, 2, 7, 0, 2, 1, 4, 9, 4, 7, 5, 3, 6, 1, 4, 4, 1, 6, 1, 3, 3, 7, 5, 3, 8, 5, 5, 7, 6, 8, 2, 3, 9, 1, 1, 3, 2, 8, 4, 7, 0, 1, 3, 5, 2, 2, 4, 8, 3, 7, 0, 2, 1, 3, 0 ... The anomalous two-forty-somethings crop up again starting at the 45,763rd position—this time eight of them, again in pairs separated by an "ordinary" small number. 1, 7, 2, 2, 1, 0, 245, 245, 6, 248, 244, 5, 242, 242, 0, 248, 246, 1, 1, 3, 1, 1, 4, 3, 1, 5, 4, 3, 8, 3, 4, 5, 4, 1, 7, 7, 3, 0, 2, 8, 0, 9, 5, 1, 1, 7, 7, 1, 0, 9, 3, 0, 6, 6, 7, 5, 8, 1, 5, 5, 5, 3, 3, 3, 1, 3, 9, 6, 0, 0, 0, 9, 5, 1, 4, 0, 4, 6 ... Two, four, eight—does it keep going like that? "Bursts" of increasingly many paired two-forty-somethings, punctuating the quiet background radiation of single digits? What does it mean? You allocate a new scratch buffer and write a quick Python function to count up the segments of two-forty-somethings. (This is apparently a thing you can do—it's an instinctive felt sense, like the input streams. You can't describe in words how you do it—any more than someone could say how they decide to move their arm. Although, come to think of it, you don't seem to have any arms. Is that unusual?) def count_burst_lengths(data): bursts = [] counter = 0 previous = None for datum in data: if datum >= 240: counter += 1 else:
consecutive "ordinary" numbers mean the burst is over
if counter and previous and previous < 240: bursts.append(counter) counter = 0 previous = datum return bursts There are 403 such bursts in the sequence: they get progressively longer at first, but then decrease and taper off: 2, 4, 8, 12, 16, 18, 24, 28, 32, 34, 38, 42, 46, 48, 5...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Ugh fields, published by Roko on LessWrong. Tl;Dr version: Pavlovian conditioning can cause humans to unconsciously flinch from even thinking about a serious personal problem they have, we call it an "Ugh Field"1. The Ugh Field forms a self-shadowing blind spot covering an area desperately in need of optimization, imposing huge costs. A problem with the human mind — your human mind — is that it's a horrific kludge that will fail when you most need it not to. The Ugh Field failure mode is one of those really annoying failures. The idea is simple: if a person receives constant negative conditioning via unhappy thoughts whenever their mind goes into a certain zone of thought, they will begin to develop a psychological flinch mechanism around the thought. The "Unhappy Thing" — the source of negative thoughts — is typically some part of your model of the world that relates to bad things being likely to happen to you. A key part of the Ugh Field phenomenon is that, to start with, there is no flinch, only negative real consequences resulting from real physical actions in the problem area. Then, gradually, you begin to feel the emotional hit when you are planning to take physical actions in the problem area. Then eventually, the emotional hit comes when you even begin to think about the problem. The reason for this may be that your brain operates a temporal difference learning (TDL) algorithm. Your brain propagates the psychological pain "back to the earliest reliable stimulus for the punishment". If you fail or are punished sufficiently many times in some problem area, and acting in that area is always preceeded by thinking about it, your brain will propagate the psychological pain right back to the moment you first begin to entertain a thought about the problem, and hence cut your conscious optimizing ability right out of the loop. Related to this is engaging in a displacement activity: this is some activity that usually involves comfort, done instead of confronting the problem. Perhaps (though this is speculative) the comforting displacement activity is there to counterbalance the psychological pain that you experienced just because you thought about the problem. For example, suppose that you started off in life with a wandering mind and were punished a few times for failing to respond to official letters. Your TDL algorithm began to propagate the pain back to the moment you looked at an official letter or bill. As a result, you would be less effective than average at responding, so you got punished a few more times. Henceforth, when you received a bill, you got the pain before you even opened it, and it laid unpaid on the mantelpiece until a Big Bad Red late payment notice with an $25 fine arrived. More negative conditioning. Now even thinking about a bill, form or letter invokes the flinch response, and your lizard brain has fully cut you out out. You find yourself spending time on internet time-wasters, comfort food, TV, computer games, etc. Your life may not obviously be a disaster, but this is only because you can't see the alternative paths that it could have taken if you had been able to take advantage of the opportunities that came as letters and forms with deadlines. The subtlety with the Ugh Field is that the flinch occurs before you start to consciously think about how to deal with the Unhappy Thing, meaning that you never deal with it, and you don't even have the option of dealing with it in the normal run of things. I find it frightening that my lizard brain could implicitly be making life decisions for me, without even asking my permission! Possible antidotes to Ugh Field problem: Actively look out for the flinch, preferably when you are in a motivationally "high" state. Better still, do this when you are both motivationally high, not under time pressure, an...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: An Unexpected Victory: Container Stacking at the Port of Long Beach, published by Zvi on LessWrong. A miracle occurred this week. Everyone I have talked to about it, myself included, is shocked that it happened. It’s important to Understand what happened. Make sure everyone knows it happened. Understand how and why it happened. Understand how we might cause it to happen again. Update our models and actions. Ideally make this a turning point to save civilization. That last one is a bit of a stretch goal, but I am being fully serious. If you’re not terrified that the United States is a dead player, you haven’t been paying attention – the whole reason this is a miracle, and that it shocked so many people, is that we didn’t think the system was capable of noticing a stupid, massively destructive rule with no non-trivial benefits and no defenders and scrapping it, certainly not within a day. If your model did expect it, I’m very curious to know how that is possible, and how you explain the years 2020 and 2021. Here’s my understanding of what happened. First, the setup. The Ports of Los Angeles and Long Beach together are responsible for a huge percentage of shipping into the Western United States. There was a rule in the Port saying you could only stack shipping containers two containers high. This is despite the whole point of shipping containers being to stack them on top of each other so you can have a container ship. This rule was created, and I am not making this up, because it was decided that higher stacks were not sufficiently aesthetically pleasing. If you violated this rule, you lost your right to operate at the port. In normal times, this was annoying but not a huge deal. Thanks to Covid-19, there was increased demand to ship containers, creating more empty containers, and less throughput to remove those containers. Normally one would settle this by changing prices, but for various reasons we won’t get into price mechanisms aren’t working properly to fix supply shortages. Trucking companies started accumulating empty containers. The companies ran out of room to store the containers, because in many places they could only stack them in stacks of two, and there was no practical way to move the containers off-site. Trucks were forced to sit there with empty containers rather than hauling freight. This made all the problems worse, in a downward spiral, resulting in a standstill throughout the port. This was big enough to threaten the entire supply chain, and with it the economy, at least of the Western United States and potentially of the whole world via cascading problems. And similar problems are likely happening elsewhere. Everyone in the port, or at least a lot of them, knew this was happening. None of those people managed to do anything about the rule, or even get word out about the rule. No reporters wrote up news reports. No one was calling for a fix. The supply chain problems kept getting worse and mostly everyone agreed not to talk about it much and hope it would go away. A bureaucrat insisting that stacked containers are an eyesore, causing freight to pile up because trucks are stuck sitting on empty containers, thus causing a cascading failure that destroys supply lines and brings down the economy. That certainly sounds like something that was in an early draft of Atlas Shrugged but got crossed out as too preposterous for anyone to take seriously. Then our hero enters, and decides to coordinate and plan a persuasion campaign to get the rule changed. Here’s how I think this went down. He in advance arranges for various sources to give him a signal boost when the time comes, in various ways. He designs the message for a format that will have maximum reach and be maximally persuasive. This takes the form of an easy to tell physical story, that he pretends t...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Discussion with Eliezer Yudkowsky on AGI interventions, published by Rob Bensinger, Eliezer Yudkowsky on LessWrong. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. The following is a partially redacted and lightly edited transcript of a chat conversation about AGI between Eliezer Yudkowsky and a set of invitees in early September 2021. By default, all other participants are anonymized as "Anonymous". I think this Nate Soares quote (excerpted from Nate's ) is a useful context-setting preface regarding timelines, which weren't discussed as much in the transcript: [...] My odds [of AGI by the year 2070] are around 85%[...] I can list a handful of things that drive my probability of AGI-in-the-next-49-years above 80%: 1. 50 years ago was 1970. The gap between AI systems then and AI systems now seems pretty plausibly greater than the remaining gap, even before accounting the recent dramatic increase in the rate of progress, and potential future increases in rate-of-progress as it starts to feel within-grasp. 2. I observe that, 15 years ago, everyone was saying AGI is far off because of what it couldn't do -- basic image recognition, go, starcraft, winograd schemas, programmer assistance. But basically all that has fallen. The gap between us and AGI is made mostly of intangibles. (Computer Programming That Is Actually Good? Theorem proving? Sure, but on my model, "good" versions of those are a hair's breadth away from full AGI already. And the fact that I need to clarify that "bad" versions don't count, speaks to my point that the only barriers people can name right now are intangibles.) That's a very uncomfortable place to be! 3. When I look at the history of invention, and the various anecdotes about the Wright brothers and Enrico Fermi, I get an impression that, when a technology is pretty close, the world looks a lot like how our world looks. Of course, the trick is that when a technology is a little far, the world might also look pretty similar! Though when a technology is very far, the world does look different -- it looks like experts pointing to specific technical hurdles. We exited that regime a few years ago. 4. Summarizing the above two points, I suspect that I'm in more-or-less the "penultimate epistemic state" on AGI timelines: I don't know of a project that seems like they're right on the brink; that would put me in the "final epistemic state" of thinking AGI is imminent. But I'm in the second-to-last epistemic state, where I wouldn't feel all that shocked to learn that some group has reached the brink. Maybe I won't get that call for 10 years! Or 20! But it could also be 2, and I wouldn't get to be indignant with reality. I wouldn't get to say "but all the following things should have happened first, before I made that observation". I have made those observations. 5. It seems to me that the Cotra-style compute-based model provides pretty conservative estimates. For one thing, I don't expect to need human-level compute to get human-level intelligence, and for another I think there's a decent chance that insight and innovation have a big role to play, especially on 50 year timescales. 6. There has been a lot of AI progress recently. When I tried to adjust my beliefs so that I was positively surprised by AI progress just about as often as I was negatively surprised by AI progress, I ended up expecting a bunch of rapid progress. [...] Further preface by Eliezer: In some sections here, I sound gloomy about the probability that coordination between AGI groups succeeds in saving the world. Andrew Critch reminds me to point out that gloominess like this can be a self-fulfilling prophecy - if people think successful coordination is impossible, they won’t try to coordinate. I therefore remark in retrospective advance that it seem...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Leaky Delegation: You are not a Commodity, published by Darmani on LessWrong. Epistemic status: The accumulation of several insights over the years. Reasonably confident that everything mentioned here is an informative factor in decision-making. Carl is furiously slicing and skinning peaches. His hands move like lightning as slice after slice fills his tray. His freezer has been freshly cleared. Within a day, he will have a new bag of frozen fruit, and can enjoy smoothies for another month. Stan stands in the kitchen of his college dorm. His hands are carefully placing ingredients on pizza dough: homemade tomato sauce, spiced pork, and mozzarella cheese from a nearby farmer's market. "I don't know why people will pay a restaurant for this," he muses. "So much cheaper to do it yourself." Michelle is on her way to her job as a software engineer. She tosses a pile of clothes into a bag, and presses a few buttons on her phone. Later that day, someone will come by to pick them up, wash and fold them at a nearby laundromat, and return them the next morning. Less time doing laundry means more time writing code. Her roommate calls her lazy. An alert flashes on Bruce's screen: "us-east-prod-1 not responding to ping." Almost like a reflex, he pulls up diagnostics on his terminal. The software itself is still running fine, but it looks like his datacenter had a network change. A few more minutes, and everything is functioning again. Hopefully only a few customers noticed the downtime. His mentor keeps asking why he doesn't just run his website on AWS instead of owning his own servers, but Bruce insists it's worth it. His 4-person company has been profitable for 3 years, and keeping server costs low has meant the difference between staying independent and being forced to take outside investment. The four characters above each take a minority position on outsourcing a task. In the past, I saw the decision as simple: if your time is valuable, then be like Michelle and delegate and outsource as much as you can. Not to do so would be an irrational loss. I silently judged the people I met who inspired Carl and Stan. Years later, I've found myself cooking daily during a pandemic and appreciating the savings, and just finished arguing online in favor of running one's own servers. My goal in this post is to share the perspective shift that led to me wholly or partially reverse my position on paying a person or company for a good or service (collectively, "delegating" or "outsourcing") in a number of domains, even as I continue to pay for many things most people do themselves. I've noticed hidden factors which mean that, sometimes, the quality will be better if you do it yourself, even if the alternative is offered by an expert or company with specialized tools. And sometimes, it can be cheaper, even if you value your time very highly and the other person is much faster. The Internet is full of articles on the generic "buy vs. build" and "DIY vs. build" decisions Though some are written from the corporate boardroom and others from the home kitchen or workshop, the underlying analysis is eerily similar: that it's a choice between spending time (or "in-house resources") or money for a similar value. More sophisticated articles will also consider transaction costs, such as walking to a restaurant or finding your kid a tutor, and costs from principal-agent problems, such as vetting the tutor. In fact, as I've come to realize, the do-or-delegate decision is often not about two alternative ways of getting the same thing, but rather about two options sufficiently different that they're best considered not as replacements for each other, but entirely separate objects with overlapping benefits. These differences can be obvious for specific examples, as every home baker can give you an earful abou...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Self-Integrity and the Drowning Child, published by Eliezer Yudkowsky on LessWrong. (Excerpted from "mad investor chaos and the woman of asmodeus", about an unusually selfish dath ilani, "Keltham", who dies in a plane accident and ends up in Cheliax, a country governed by D&D!Hell. Keltham is here remembering an incident from his childhood.) And the Watcher told the class a parable, about an adult, coming across a child who'd somehow bypassed the various safeguards around a wilderness area, and fallen into a muddy pond, and seemed to be showing signs of drowning (for they'd already been told, then, what drowning looked like). The water, in this parable, didn't look like it would be over their own adult heads. But - in the parable - they'd just bought some incredibly-expensive clothing, costing dozens of their own labor-hours, and less resilient than usual, that would be ruined by the muddy water. And the Watcher asked the class if they thought it was right to save the child, at the cost of ruining their clothing. Everyone in there moved their hand to the 'yes' position, of course. Except Keltham, who by this point had already decided quite clearly who he was, and who simply closed his hand into a fist, otherwise saying neither 'yes' nor 'no' to the question, defying it entirely. The Watcher asked him to explain, and Keltham said that it seemed to him that it was okay for an adult to take an extra fifteen seconds to strip off all their super-expensive clothing and then jump in to save the child. The Watcher invited the other children to argue with Keltham about that, which they did, though Keltham's first defense, that his utility function was what it was, had not been a friendly one, or inviting of further argument. But they did eventually convince Keltham that, especially if you weren't sure you could call in other help or get attention or successfully drag the child's body towards help, if that child actually did drown - meaning the child's true life was at stake - then it would make sense to jump in right away, not take the extra risk of waiting another quarter-minute to strip off your clothes, and bill the child's parents' insurance for the cost. Or at least, that was where Keltham shifted his position, in the face of that argumentative pressure. Some kids, at that point, questioned the Watcher about this actually being a pretty good point, and why wouldn't anyone just bill the child's parents' insurance. To which the Watcher asked them to consider hypothetically the case where insurance refused to pay out in cases like that, because it would be too easy for people to set up 'accidents' letting them bill insurances - not that this precaution had proven to be necessary in real life, of course. But the Watcher asked them to consider the Least Convenient Possible World where insurance companies, and even parents, did need to reason like that; because there'd proven to be too many master criminals setting up 'children at risk of true death from drowning' accidents that they could apparently avert and claim bounties on. Well, said Keltham, in that case, he was going right back to taking another fifteen seconds to strip off his super-expensive clothes, if the child didn't look like it was literally right about to drown. And if society didn't like that, it was society's job to solve that thing with the master criminals. Though he'd maybe modify that if they were in a possible-true-death situation, because a true life is worth a huge number of labor-hours, and that part did feel like some bit of decision theory would say that everyone would be wealthier if everyone would sacrifice small amounts of wealth to save huge amounts of somebody else's wealth, if that happened unpredictably to people, and if society was also that incompetent at setting up proper reimbursements. T...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Attention control is critical for changing/increasing/altering motivation, published by kalla724 on LessWrong. I’ve just been reading Luke’s “Crash Course in the Neuroscience of Human Motivation.” It is a useful text, although there are a few technical errors and a few bits of outdated information (see [1], updated information about one particular quibble in [2] and [3]). There is one significant missing piece, however, which is of critical importance for our subject matter here on LW: the effect of attention on plasticity, including the plasticity of motivation. Since I don’t see any other texts addressing it directly (certainly not from a neuroscientific perspective), let’s cover the main idea here. Summary for impatient readers: focus of attention physically determines which synapses in your brain get stronger, and which areas of your cortex physically grow in size. The implications of this provide direct guidance for alteration of behaviors and motivational patterns. This is used for this purpose extensively: for instance, many benefits of the Cognitive-Behavioral Therapy approach rely on this mechanism. I – Attention and plasticity To illustrate this properly, we need to define two terms. I’m guessing these are very familiar to most readers here, but let’s cover them briefly just in case. First thing to keep in mind is the plasticity of cortical maps. In essence, particular functional areas of our brain can expand or shrink based on how often (and how intensely) they are used. A small amount of this growth is physical, as new axons grow, expanding the white matter; most of it happens by repurposing any less-used circuitry in the vicinity of the active area. For example, our sense of sight is processed by our visual cortex, which turns signals from our eyes into lines, shapes, colors and movement. In blind people, however, this part of the brain becomes invaded by other senses, and begins to process sensations like touch and hearing, such that they become significantly more sensitive than in sighted people. Similarly, in deaf people, auditory cortex (part of the brain that processes sounds) becomes adapted to process visual information and gather language clues by sight. Second concept we’ll need is somatosensory cortex (SSC for short). This is an area of the (vertebrate) brain where most of the incoming touch and positional (proprioceptive) sensations from the body converge. There is a map-like quality to this part of our brain, as every body part links to a particular bit of the SSC surface (which can be illustrated with silly-looking things, such as the sensory homunculus). More touch-sensitive areas of the body have larger corresponding areas within the SSC. With these two in mind, let’s consider one actual experiment [4]. Scientists measured and mapped the area of an owl monkey’s SSC which became activated when one of his fingertips was touched. The monkey was then trained to hold that finger on a tactile stimulator – a moving wheel that stimulates touch receptors. The monkey had to pay attention to the stimulus, and was rewarded for letting go upon detecting certain changes in spinning frequency. After a few weeks of training, the area was measured again. As you probably expected, the area had grown larger. The touch-processing neurons grew out, co-opting surrounding circuitry in order to achieve better and faster processing of the stimulus that produced the reward. Which is, so far, just another way of showing plasticity of cortical maps. But then, there is something else. The SSC area expanded only when the monkey had to pay attention to the sensation of touch in order to receive the reward. If a monkey was trained to keep a hand on the wheel that moved just the same, but he did not have to pay attention to it. the cortical map remained the same size. Thi...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Great minds might not think alike, published by UnexpectedValues on LessWrong. Write a Review This is a linkpost for/ [Previously known as "Alike minds think great"] I. It is famously the case that almost everyone thinks they’re above average. Derek Sivers writes: Ninety-four percent of professors say they are better-than-average teachers. Ninety percent of students think they are more intelligent than the average student. Ninety-three percent of drivers say they are safer-than-average drivers. Interesting. Intuitively this seems to suggest that people are prone to vastly overestimate their competence. But is that true? As Bill Kuszmaul points out, these people aren’t necessarily wrong! There’s no fundamental reason why you can’t have 90% of people be better than average. For example, more than 99.9% of people have an above-average number of legs. And more than 90% of people commit fewer felonies than average. These examples are obvious, but they’re not so different than some of the examples [in Sivers’ post]. This has something to it! On the other hand, I don’t think this explains everything. Is the quality of a professor’s teaching really so skewed that 94% are above average? But more importantly, do you really think that way fewer people would answer “yes” if you just replaced the word “average” with “median” when asking the question? That said, I don’t think these numbers necessarily point to a bias! That’s because the interpretation of “above average” is left entirely up to the person being asked. Maybe you think a good driver is one who drives safely (and so you drive safely and slowly) whereas I think a good driver is one who gets from point A to point B efficiently (and so I drive quickly but not safely). We are both, from our own perspectives, above average drivers! Put otherwise, for any skill where “goodness at that skill” doesn’t have an objective, agreed-upon measure, we should expect more than 50% of people to think they’re better than the median, because people optimize for things they care about. To give a personal example, I suppose I would call myself an above average blogger. This isn’t true in some objective sense; it’s just that I judge bloggers by how interesting their thoughts are to me, and obviously I write about things that are interesting to me! There’s no bias I’m falling for here; it’s just that “Are you an above average blogger?” leaves “above average” open to my interpretation. II. There is, however, a closely related bias that I and lots of other people have. This bias occurs when we take a situation like those above, but now create a more objective test of that skill. To illustrate with an example, suppose you asked all the students at a university whether they have an above-median GPA. If 90% of students said yes, that would demonstrate a widespread bias — because unlike “Are you a better than the median student”, here there’s no room for interpretation. The way this bias manifests in me (and many others I imagine) is: I tend to underestimate the competence of people who think very differently from me. I started thinking about this the other day when I listened to Julia Galef’s podcast episode with David Shor (which I highly recommend1). Shor is a young Democratic political strategist, originally hired by Barack Obama’s 2012 reelection campaign to run their data operation and figure out how the campaign should spend its money. Shor says: When I first started in 2012, I was 20 and I was like, “Oh, I’m going to do all of this math and we’re going to win elections.” And I was with all these other nerds, we were in this cave. We really hated these old school consultants who had been in politics for like 20 years. [.] We had all these disagreements because the old school consultants were like, “You need to go up on TV, you need to focus o...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: LessWrong is providing feedback and proofreading on drafts as a service, published by by Ruby on LessWrong. This announcement follows the Amazon PR/FAQ format. This is an actual feature announcement. TL;DR Before one publishes a post, it can be hard to know if you caught all the typos, explained things clearly, made a critical error, or wrote something that anybody is interested in to begin with. To reduce the guesswork, LessWrong is now providing free feedback on drafts (and post ideas) to any user with 100+ karma. We’ll provide the feedback ourselves, send your draft to a professional copy editor, or get the opinion of a relevant peer or expert in your domain. Or something else, whatever is needed to be helpful! The Problem Many people are reluctant to share posts before they’re confident that (i) they’re correct, (ii) they’ll get a good reception. It sucks to put out a post and then notice a dumb typo a day later, or to publish and then have a critical flaw immediately revealed to everyone, or to share a post and hear only crickets. The fear of these outcomes is enough to prevent a lot of great ideas from ever escaping their creators’ heads. And although many people feel better after getting some feedback, soliciting it can be effortful–you’ve got to find someone else and then tap into your social capital and ask a favor. Solution To help get more excellent posts into the world, LessWrong is now providing feedback on tap. Any author with 100+ karma can ask for the kind of feedback they need, and the LessWrong team will make it happen. Quick, easy, free. Within a couple of days (or hours), we’ll have feedback on your post that will let you post with greater confidence that your post is good. Getting Started On the post edit page (create a new post or edit an existing draft), if you have 100+ karma, you will see a new button: Request Feedback. Clicking it will start an Intercom chat with a LessWrong team member; in that chat, describe what kind of feedback you’re looking for (proofreading, style, coherence, expert feedback, etc.) and the LessWrong team will make it happen. You needn’t have even written anything to use the feature. Feel free to chat to us about post ideas you have. The new button (left) appears when create a new post or edit an existing one. Press "Request Feedback" to have the Intercom Messenger popup. Quotes (fictional) After getting a round of feedback through the new LessWrong system, I’m much less afraid that people will ignore or downvote my post. I’ve got evidence that it’s something good that people will want to read - Oliver Habryka A great benefit from the LessWrong feedback system, now that I’ve used it several times, is that the detailed feedback has helped me improve as a writer. - John McPostALot FAQ Who will provide the feedback? It depends on the kind of feedback being sought. For a quick sanity check or proofread, a LessWrong team member or volunteer might do it. If more thorough copy-editing is requested, we’ll send your draft to a professional copy-editor. And if you’re looking for comments from a domain expert (biology, AI, etc), we’ll find someone willing to provide such feedback. These types of reviewers are our current guess at what we will provide, but that might evolve over time as we figure out what kinds of feedback people need. How quickly will I get the feedback? Depends on the kind of feedback being sought. The LessWrong team can get things back you within a day or two; copy-editor will probably be variable, but sometimes quick; for external domain experts, could be a bit longer. How much does this cost? Free to eligible users. How many times can I use it? We’re not setting any explicit limits on how many times you can request feedback; however requests will be prioritized at our discretion (hopefully we have the capacit...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Industrial literacy, published by jasoncrawford on LessWrong. Write a Review This is a linkpost for I’ve said before that understanding where our modern standard of living comes from, at a basic level, is a responsibility of every citizen in an industrial civilization. Let’s call it “industrial literacy.” Industrial literacy is understanding. That the food you eat is grown using synthetic fertilizers, and that this is needed for agricultural productivity, because all soil loses its fertility naturally over time if it is not deliberately replenished. That before we had modern agriculture, more than half the workforce had to labor on farms, just to feed the other half. That if synthetic fertilizer was suddenly lost, a mass famine would ensue and billions would starve. That those same crops would not be able to feed us if they were not also protected from pests, who will ravage entire fields if given a chance. That whole regions used to see seasons where they would lose large swaths of their produce to swarms of insects, such as boll weevils attacking cotton plants in the American South, or the phylloxera devouring grapes in the vineyards of France. That before synthetic pesticides, farmers were forced to rely on much more toxic substances, such as compounds of arsenic. That before we had electricity and clean natural gas, people burned unrefined solid fuels in their homes—wood, coal, even dung (!)—to cook their food and to keep from freezing in winter. That these primitive fuels, dirty with contaminants, created toxic smoke: indoor air pollution. That indoor air pollution remains a problem today for 40% of the world population, who still rely on pre-industrial fuels. That before twentieth-century appliances, housework was a full-time job, which invariably fell on women. That each household would spend almost 60 hours a week on manual labor: hauling water from the well for drinking and cooking, and then carrying the dirty water outside again; sewing clothes by hand, since store-bought ones were too expensive for most families; laundering clothes in a basin, scrubbing laboriously by hand, then hanging them up to dry; cooking every meal from scratch. That the washing machine, clothes dryer, dishwasher, vacuum cleaner, and microwave are the equivalent of a full-time mechanical servant for every household. That plastics are produced in enormous quantities because, for so many purposes—from food containers to electrical wire coatings to children’s toys—we need a material that is cheap, light, flexible, waterproof, and insulating, and that can easily be made in any shape and color (including transparent!) That before plastic, many of these applications used animal parts, such as ivory tusks, tortoise shells, or whale bone. That in such a world, those products were a luxury for a wealthy elite, instead of a commodity for the masses, and the animals that provided them were hunted to near extinction. That automobiles are a lifeline to people who live in rural areas (almost 20% in the US alone), and who were deeply isolated in the era before the car and the telephone. That in a world without automobiles, we relied on millions of horses, which in New York City around 1900 dumped a hundred thousand gallons of urine and millions of pounds of manure on the streets daily. That half of everyone you know over the age of five is alive today only because of antibiotics, vaccines, and sanitizing chemicals in our water supply. That before these innovations, infant mortality (in the first year of life) was as high as 20%. When you know these facts of history—which many schools do not teach—you understand what “industrial civilization” is and why it is the benefactor of everyone who is lucky enough to live in it. You understand that the electric generator, the automobile, the chemical plant, ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Parable of Predict-O-Matic, published by abramdemski on LessWrong. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. I've been thinking more about partial agency. I want to expand on some issues brought up in the comments to my previous post, and on other complications which I've been thinking about. But for now, a more informal parable. (Mainly because this is easier to write than my more technical thoughts.) This relates to oracle AI and to inner optimizers, but my focus is a little different. 1 Suppose you are designing a new invention, a predict-o-matic. It is a wonderous machine which will predict everything for us: weather, politics, the newest advances in quantum physics, you name it. The machine isn't infallible, but it will integrate data across a wide range of domains, automatically keeping itself up-to-date with all areas of science and current events. You fully expect that once your product goes live, it will become a household utility, replacing services like Google. (Google only lets you search the known!) Things are going well. You've got investors. You have an office and a staff. These days, it hardly even feels like a start-up any more; progress is going well. One day, an intern raises a concern. "If everyone is going to be using Predict-O-Matic, we can't think of it as a passive observer. Its answers will shape events. If it says stocks will rise, they'll rise. If it says stocks will fall, then fall they will. Many people will vote based on its predictions." "Yes," you say, "but Predict-O-Matic is an impartial observer nonetheless. It will answer people's questions as best it can, and they react however they will." "But --" the intern objects -- "Predict-O-Matic will see those possible reactions. It knows it could give several different valid predictions, and different predictions result in different futures. It has to decide which one to give somehow." You tap on your desk in thought for a few seconds. "That's true. But we can still keep it objective. It could pick randomly." "Randomly? But some of these will be huge issues! Companies -- no, nations -- will one day rise or fall based on the word of Predict-O-Matic. When Predict-O-Matic is making a prediction, it is choosing a future for us. We can't leave that to a coin flip! We have to select the prediction which results in the best overall future. Forget being an impassive observer! We need to teach Predict-O-Matic human values!" You think about this. The thought of Predict-O-Matic deliberately steering the future sends a shudder down your spine. But what alternative do you have? The intern isn't suggesting Predict-O-Matic should lie, or bend the truth in any way -- it answers 100% honestly to the best of its ability. But (you realize with a sinking feeling) honesty still leaves a lot of wiggle room, and the consequences of wiggles could be huge. After a long silence, you meet the interns eyes. "Look. People have to trust Predict-O-Matic. And I don't just mean they have to believe Predict-O-Matic. They're bringing this thing into their homes. They have to trust that Predict-O-Matic is something they should be listening to. We can't build value judgements into this thing! If it ever came out that we had coded a value function into Predict-O-Matic, a value function which selected the very future itself by selecting which predictions to make -- we'd be done for! No matter how honest Predict-O-Matic remained, it would be seen as a manipulator. No matter how beneficent its guiding hand, there are always compromises, downsides, questionable calls. No matter how careful we were to set up its values -- to make them moral, to make them humanitarian, to make them politically correct and broadly appealing -- who are we to choose? No. We'd be done for. They'd hang us....
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Being the (Pareto) Best in the World, published by johnswentworth on LessWrong. The generalized efficient markets (GEM) principle says, roughly, that things which would give you a big windfall of money and/or status, will not be easy. If such an opportunity were available, someone else would have already taken it. You will never find a $100 bill on the floor of Grand Central Station at rush hour, because someone would have picked it up already. One way to circumvent GEM is to be the best in the world at some relevant skill. A superhuman with hawk-like eyesight and the speed of the Flash might very well be able to snag $100 bills off the floor of Grand Central. More realistically, even though financial markets are the ur-example of efficiency, a handful of firms do make impressive amounts of money by being faster than anyone else in their market. I’m unlikely to ever find a proof of the Riemann Hypothesis, but Terry Tao might. Etc. But being the best in the world, in a sense sufficient to circumvent GEM, is not as hard as it might seem at first glance (though that doesn’t exactly make it easy). The trick is to exploit dimensionality. Consider: becoming one of the world’s top experts in proteomics is hard. Becoming one of the world’s top experts in macroeconomic modelling is hard. But how hard is it to become sufficiently expert in proteomics and macroeconomic modelling that nobody is better than you at both simultaneously? In other words, how hard is it to reach the Pareto frontier? Having reached that Pareto frontier, you will have circumvented the GEM: you will be the single best-qualified person in the world for (some) problems which apply macroeconomic modelling to proteomic data. You will have a realistic shot at a big money/status windfall, with relatively little effort. (Obviously we’re oversimplifying a lot by putting things like “macroeconomic modelling skill” on a single axis, and breaking it out onto multiple axes would strengthen the main point of this post. On the other hand, it would complicate the explanation; I’m keeping it simple for now.) Let’s dig into a few details of this approach. Elbow Room There are many table tennis players, but only one best player in the world. This is a side effect of ranking people on one dimension: there’s only going to be one point furthest to the right (absent a tie). Pareto optimality pushes us into more dimensions. There’s only one best table tennis player, and only one best 100-meter sprinter, but there can be an unlimited number of Pareto-optimal table tennis/sprinters. Problem is, for GEM purposes, elbow room matters. Maybe I’m the on the pareto frontier of Bayesian statistics and gerontology, but if there’s one person just little bit better at statistics and worse at gerontology than me, and another person just a little bit better at gerontology and worse at statistics, then GEM only gives me the advantage over a tiny little chunk of the skill-space. This brings up another aspect. Problem Density Claiming a spot on a Pareto frontier gives you some chunk of the skill-space to call your own. But that’s only useful to the extent that your territory contains useful problems. Two pieces factor in here. First, how large a territory can you claim? This is about elbow room, as in the diagram above. Second, what’s the density of useful problems within this region of skill-space? The table tennis/sprinting space doesn’t have a whole lot going on. Statistics and gerontology sounds more promising. Cryptography and monetary economics is probably a particularly rich Pareto frontier these days. (And of course, we don’t need to stop at two dimensions - but we’re going to stop there in this post in order to keep things simple.) Dimensionality One problem with this whole GEM-vs-Pareto concept: if chasing a Pareto frontier makes it ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Welcome to LessWrong!, published by Ruby, habryka, Ben Pace, Raemon on LessWrong. The road to wisdom? -- Well, it's plain and simple to express: Err and err and err again but less and less and less. - Piet Hein Hence the name LessWrong. We might never attain perfect understanding of the world, but we can at least strive to become less and less wrong each day. We are a community dedicated to improving our reasoning and decision-making. We seek to hold true beliefs and to be effective at accomplishing our goals. More generally, we work to develop and practice the art of human rationality.[1] To that end, LessWrong is a place to 1) develop and train rationality, and 2) apply one’s rationality to real-world problems. LessWrong serves these purposes with its library of rationality writings, community discussion forum, open questions research platform, and community page for in-person events. To get a feel for what LessWrong is about, check out our Concepts page, or view this selection of LessWrong posts which might appeal to you: What is rationality and why care about it? Try Your intuitions are not magic and The Cognitive Science of Rationality. Curious about the mind? You might enjoy How An Algorithm Feels From The Inside and The Apologist and the Revolutionary. Keen on self-improvement? Remember that Humans are not automatically strategic. Care about argument and evidence? Consider Policy Debates Should Not Appear One-Sided and How To Convince Me that 2 + 2 = 3. Interested in how to use language well? Be aware of 37 Ways That Words Can Be Wrong. Want to teach yourself something? We compiled a list of The Best Textbooks on Every Subject. Like probability and statistics? Around here we're fans of Bayesianism, you might like this interactive guide to Bayes' theorem (hosted on Arbital.com). Of an altruistic mindset? We recommend On Caring. Check out this footnote[2] below the fold for samples of posts about AI, science, philosophy, history, communication, culture, self-care, and more. If LessWrong seems like a place for you, we encourage you to become familiar with LessWrong’s philosophical foundations. Our core readings can be be found on the Library page. We especially recommend: Rationality: From AI to Zombies by Eliezer Yudkowsky (or Harry Potter and the Methods of Rationality by the same author, which covers similar ground in narrative form) The Codex by Scott Alexander Find more details about these texts in this footnote[3] For further getting started info, we direct you to LessWrong’s FAQ. Lastly, we suggest you create an account so you can vote, comment, save your reading progress, get tailored recommendations, and subscribe to our latest and best posts. Once you've done so, please say hello on our latest welcome thread! Related Pages LessWrong FAQ A Brief History of LessWrong Team LessWrong Concepts thanks for listening. to help us out with the nonlinear library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: PR is corrosive; “reputation” is not, published by AnnaSalamon on LessWrong. This is in some sense a small detail, but one important enough to be worth write-up and critique: AFAICT, “PR” is a corrupt concept, in the sense that if you try to “navigate PR concerns” about yourself / your organization / your cause area / etc., the concept will guide you toward harmful and confused actions. In contrast, if you try to safeguard your “reputation”, your “brand”, or your “honor,” I predict this will basically go fine, and will not lead you to leave a weird confused residue in yourself or others. To explain the difference: If I am safeguarding my “honor” (or my “reputation”, “brand”, or “good name”), there are some fixed standards that I try to be known as adhering to. For example, in Game of Thrones, the Lannisters are safeguarding their “honor” by adhering to the principle “A Lannister always pays his debts.” They take pains to adhere to a certain standard, and to be known to adhere to that standard. Many examples are more complicated than this; a gentleman of 1800 who took up a duel to defend his “honor” was usually not defending his known adherence to a single simple principle a la the Lannisters. But it was still about his visible adherence to a fixed (though not explicit) societal standard. In contrast, if I am “managing PR concerns,” there is no fixed standards of good conduct, or of my-brand-like conduct, that I am trying to adhere to. Instead, I am trying to do a more complicated operation: Model which words or actions may cause “people” (especially media, or self-reinforcing miasma) to get upset with me; Try to speak in such a way as to not set that off. It’s a weirder or loopier process. One that’s more prone to self-reinforcing fears of shadows, and one that somehow (I think?) tends to pull a person away from communicating anything at all. Reminiscent of “Politics and the English Language.” Not reminiscent of Strunk and White. One way you can see the difference, is that when people think about “PR” they imagine a weird outside expertise, such that you need to have a “PR consultant” or a “media consultant” who you should nervously heed advice from. When people think about their “honor," it's more a thing they can know or choose directly, and so it is more a thing that leaves them free to communicate something. So: simple suggestion. If, at any point, you find yourself trying to “navigate PR”, or to help some person or organization or cause area or club or whatever to “navigate PR,” see if you can instead think and speak in terms of defending your/their “honor”, “reputation”, or “good name”. And see if that doesn’t make everybody feel a bit clearer, freer, and more as though their feet are on the ground. Related: The Inner Ring, by CS Lewis; The New York Times, by Robert Rhinehart. thanks for listening. to help us out with the nonlinear library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Making Beliefs Pay Rent (in Anticipated Experiences), published by Eliezer Yudkowsky on LessWrong. Thus begins the ancient parable: If a tree falls in a forest and no one hears it, does it make a sound? One says, “Yes it does, for it makes vibrations in the air.” Another says, “No it does not, for there is no auditory processing in any brain.” If there’s a foundational skill in the martial art of rationality, a mental stance on which all other technique rests, it might be this one: the ability to spot, inside your own head, psychological signs that you have a mental map of something, and signs that you don’t. Suppose that, after a tree falls, the two arguers walk into the forest together. Will one expect to see the tree fallen to the right, and the other expect to see the tree fallen to the left? Suppose that before the tree falls, the two leave a sound recorder next to the tree. Would one, playing back the recorder, expect to hear something different from the other? Suppose they attach an electroencephalograph to any brain in the world; would one expect to see a different trace than the other? Though the two argue, one saying “No,” and the other saying “Yes,” they do not anticipate any different experiences. The two think they have different models of the world, but they have no difference with respect to what they expect will happen to them; their maps of the world do not diverge in any sensory detail. It’s tempting to try to eliminate this mistake class by insisting that the only legitimate kind of belief is an anticipation of sensory experience. But the world does, in fact, contain much that is not sensed directly. We don’t see the atoms underlying the brick, but the atoms are in fact there. There is a floor beneath your feet, but you don’t experience the floor directly; you see the light reflected from the floor, or rather, you see what your retina and visual cortex have processed of that light. To infer the floor from seeing the floor is to step back into the unseen causes of experience. It may seem like a very short and direct step, but it is still a step. You stand on top of a tall building, next to a grandfather clock with an hour, minute, and ticking second hand. In your hand is a bowling ball, and you drop it off the roof. On which tick of the clock will you hear the crash of the bowling ball hitting the ground? To answer precisely, you must use beliefs like Earth’s gravity is 9.8 meters per second per second, and This building is around 120 meters tall. These beliefs are not wordless anticipations of a sensory experience; they are verbal-ish, propositional. It probably does not exaggerate much to describe these two beliefs as sentences made out of words. But these two beliefs have an inferential consequence that is a direct sensory anticipation—if the clock’s second hand is on the 12 numeral when you drop the ball, you anticipate seeing it on the 1 numeral when you hear the crash five seconds later. To anticipate sensory experiences as precisely as possible, we must process beliefs that are not anticipations of sensory experience. It is a great strength of Homo sapiens that we can, better than any other species in the world, learn to model the unseen. It is also one of our great weak points. Humans often believe in things that are not only unseen but unreal. The same brain that builds a network of inferred causes behind sensory experience can also build a network of causes that is not connected to sensory experience, or poorly connected. Alchemists believed that phlogiston caused fire—we could simplistically model their minds by drawing a little node labeled “Phlogiston,” and an arrow from this node to their sensory experience of a crackling campfire—but this belief yielded no advance predictions; the link from phlogiston to experience was always configur...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: That Alien Message, published Eliezer Yudkowsky on LessWrong. Imagine a world much like this one, in which, thanks to gene-selection technologies, the average IQ is 140 (on our scale). Potential Einsteins are one-in-a-thousand, not one-in-a-million; and they grow up in a school system suited, if not to them personally, then at least to bright kids. Calculus is routinely taught in sixth grade. Albert Einstein, himself, still lived and still made approximately the same discoveries, but his work no longer seems exceptional. Several modern top-flight physicists have made equivalent breakthroughs, and are still around to talk. (No, this is not the world Brennan lives in.) One day, the stars in the night sky begin to change. Some grow brighter. Some grow dimmer. Most remain the same. Astronomical telescopes capture it all, moment by moment. The stars that change, change their luminosity one at a time, distinctly so; the luminosity change occurs over the course of a microsecond, but a whole second separates each change. It is clear, from the first instant anyone realizes that more than one star is changing, that the process seems to center around Earth particularly. The arrival of the light from the events, at many stars scattered around the galaxy, has been precisely timed to Earth in its orbit. Soon, confirmation comes in from high-orbiting telescopes (they have those) that the astronomical miracles do not seem as synchronized from outside Earth. Only Earth's telescopes see one star changing every second (1005 milliseconds, actually). Almost the entire combined brainpower of Earth turns to analysis. It quickly becomes clear that the stars that jump in luminosity, all jump by a factor of exactly 256; those that diminish in luminosity, diminish by a factor of exactly 256. There is no apparent pattern in the stellar coordinates. This leaves, simply, a pattern of BRIGHT-dim-BRIGHT-BRIGHT... "A binary message!" is everyone's first thought. But in this world there are careful thinkers, of great prestige as well, and they are not so sure. "There are easier ways to send a message," they post to their blogs, "if you can make stars flicker, and if you want to communicate. Something is happening. It appears, prima facie, to focus on Earth in particular. To call it a 'message' presumes a great deal more about the cause behind it. There might be some kind of evolutionary process among, um, things that can make stars flicker, that ends up sensitive to intelligence somehow... Yeah, there's probably something like 'intelligence' behind it, but try to appreciate how wide a range of possibilities that really implies. We don't know this is a message, or that it was sent from the same kind of motivations that might move us. I mean, we would just signal using a big flashlight, we wouldn't mess up a whole galaxy." By this time, someone has started to collate the astronomical data and post it to the Internet. Early suggestions that the data might be harmful, have been... not ignored, but not obeyed, either. If anything this powerful wants to hurt you, you're pretty much dead (people reason). Multiple research groups are looking for patterns in the stellar coordinates—or fractional arrival times of the changes, relative to the center of the Earth—or exact durations of the luminosity shift—or any tiny variance in the magnitude shift—or any other fact that might be known about the stars before they changed. But most people are turning their attention to the pattern of BRIGHTS and dims. It becomes clear almost instantly that the pattern sent is highly redundant. Of the first 16 bits, 12 are BRIGHTS and 4 are dims. The first 32 bits received align with the second 32 bits received, with only 7 out of 32 bits different, and then the next 32 bits received have only 9 out of 32 bits different from the s...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why the tails come apart, published by Thrasymachus on LessWrong. [I'm unsure how much this rehashes things 'everyone knows already' - if old hat, feel free to downvote into oblivion. My other motivation for the cross-post is the hope it might catch the interest of someone with a stronger mathematical background who could make this line of argument more robust] [Edit 2014/11/14: mainly adjustments and rewording in light of the many helpful comments below (thanks!). I've also added a geometric explanation.] Many outcomes of interest have pretty good predictors. It seems that height correlates to performance in basketball (the average height in the NBA is around 6'7"). Faster serves in tennis improve one's likelihood of winning. IQ scores are known to predict a slew of factors, from income, to chance of being imprisoned, to lifespan. What's interesting is what happens to these relationships 'out on the tail': extreme outliers of a given predictor are seldom similarly extreme outliers on the outcome it predicts, and vice versa. Although 6'7" is very tall, it lies within a couple of standard deviations of the median US adult male height - there are many thousands of US men taller than the average NBA player, yet are not in the NBA. Although elite tennis players have very fast serves, if you look at the players serving the fastest serves ever recorded, they aren't the very best players of their time. It is harder to look at the IQ case due to test ceilings, but again there seems to be some divergence near the top: the very highest earners tend to be very smart, but their intelligence is not in step with their income (their cognitive ability is around +3 to +4 SD above the mean, yet their wealth is much higher than this) (1). The trend seems to be that even when two factors are correlated, their tails diverge: the fastest servers are good tennis players, but not the very best (and the very best players serve fast, but not the very fastest); the very richest tend to be smart, but not the very smartest (and vice versa). Why? Too much of a good thing? One candidate explanation would be that more isn't always better, and the correlations one gets looking at the whole population doesn't capture a reversal at the right tail. Maybe being taller at basketball is good up to a point, but being really tall leads to greater costs in terms of things like agility. Maybe although having a faster serve is better all things being equal, but focusing too heavily on one's serve counterproductively neglects other areas of one's game. Maybe a high IQ is good for earning money, but a stratospherically high IQ has an increased risk of productivity-reducing mental illness. Or something along those lines. I would guess that these sorts of 'hidden trade-offs' are common. But, the 'divergence of tails' seems pretty ubiquitous (the tallest aren't the heaviest, the smartest parents don't have the smartest children, the fastest runners aren't the best footballers, etc. etc.), and it would be weird if there was always a 'too much of a good thing' story to be told for all of these associations. I think there is a more general explanation. The simple graphical explanation [Inspired by this essay from Grady Towers] Suppose you make a scatter plot of two correlated variables. Here's one I grabbed off google, comparing the speed of a ball out of a baseball pitchers hand compared to its speed crossing crossing the plate: It is unsurprising to see these are correlated (I'd guess the R-square is > 0.8). But if one looks at the extreme end of the graph, the very fastest balls out of the hand aren't the very fastest balls crossing the plate, and vice versa. This feature is general. Look at this data (again convenience sampled from googling 'scatter plot') of this: Or this: Or this: Given a correlation, the envelo...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What Do We Mean By "Rationality"?, published by Eliezer Yudkowsky on LessWrong. I mean two things: 1. Epistemic rationality: systematically improving the accuracy of your beliefs. 2. Instrumental rationality: systematically achieving your values. The first concept is simple enough. When you open your eyes and look at the room around you, you’ll locate your laptop in relation to the table, and you’ll locate a bookcase in relation to the wall. If something goes wrong with your eyes, or your brain, then your mental model might say there’s a bookcase where no bookcase exists, and when you go over to get a book, you’ll be disappointed. This is what it’s like to have a false belief, a map of the world that doesn’t correspond to the territory. Epistemic rationality is about building accurate maps instead. This correspondence between belief and reality is commonly called “truth,” and I’m happy to call it that.1 Instrumental rationality, on the other hand, is about steering reality—sending the future where you want it to go. It’s the art of choosing actions that lead to outcomes ranked higher in your preferences. I sometimes call this “winning.” So rationality is about forming true beliefs and making decisions that help you win. (Where truth doesn't mean “certainty,” since we can do plenty to increase the probability that our beliefs are accurate even though we're uncertain; and winning doesn't mean “winning at others' expense,” since our values include everything we care about, including other people.) When people say “X is rational!” it’s usually just a more strident way of saying “I think X is true” or “I think X is good.” So why have an additional word for “rational” as well as “true” and “good”? An analogous argument can be given against using “true.” There is no need to say “it is true that snow is white” when you could just say “snow is white.” What makes the idea of truth useful is that it allows us to talk about the general features of map-territory correspondence. “True models usually produce better experimental predictions than false models” is a useful generalization, and it’s not one you can make without using a concept like “true” or “accurate.” Similarly, “Rational agents make decisions that maximize the probabilistic expectation of a coherent utility function” is the kind of thought that depends on a concept of (instrumental) rationality, whereas “It’s rational to eat vegetables” can probably be replaced with “It’s useful to eat vegetables” or “It’s in your interest to eat vegetables.” We need a concept like “rational” in order to note general facts about those ways of thinking that systematically produce truth or value—and the systematic ways in which we fall short of those standards. As we’ve observed in the previous essays, experimental psychologists sometimes uncover human reasoning that seems very strange. For example, someone rates the probability “Bill plays jazz” as less than the probability “Bill is an accountant who plays jazz.” This seems like an odd judgment, since any particular jazz-playing accountant is obviously a jazz player. But to what higher vantage point do we appeal in saying that the judgment is wrong ? Experimental psychologists use two gold standards: probability theory, and decision theory. Probability theory is the set of laws underlying rational belief. The mathematics of probability applies equally to “figuring out where your bookcase is” and “estimating how many hairs were on Julius Caesars head,” even though our evidence for the claim “Julius Caesar was bald” is likely to be more complicated and indirect than our evidence for the claim “theres a bookcase in my room.” It’s all the same problem of how to process the evidence and observations to update one’s beliefs. Similarly, decision theory is the set of laws underlying rational ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What 2026 looks like, published by Daniel Kokotajlo on LessWrong. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. This was written for the Vignettes Workshop.[1] The goal is to write out a detailed future history (“trajectory”) that is as realistic (to me) as I can currently manage, i.e. I’m not aware of any alternative trajectory that is similarly detailed and clearly more plausible to me. The methodology is roughly: Write a future history of 2022. Condition on it, and write a future history of 2023. Repeat for 2024, 2025, etc. (I'm posting 2022-2026 now so I can get feedback that will help me write 2027+. I intend to keep writing until the story reaches singularity/extinction/utopia/etc.) What’s the point of doing this? Well, there are a couple of reasons: Sometimes attempting to write down a concrete example causes you to learn things, e.g. that a possibility is more or less plausible than you thought. Most serious conversation about the future takes place at a high level of abstraction, talking about e.g. GDP acceleration, timelines until TAI is affordable, multipolar vs. unipolar takeoff. vignettes are a neglected complementary approach worth exploring. Most stories are written backwards. The author begins with some idea of how it will end, and arranges the story to achieve that ending. Reality, by contrast, proceeds from past to future. It isn’t trying to entertain anyone or prove a point in an argument. Anecdotally, various people seem to have found Paul Christiano’s “tales of doom” stories helpful, and relative to typical discussions those stories are quite close to what we want. (I still think a bit more detail would be good — e.g. Paul’s stories don’t give dates, or durations, or any numbers at all really.)[2] “I want someone to ... write a trajectory for how AI goes down, that is really specific about what the world GDP is in every one of the years from now until insane intelligence explosion. And just write down what the world is like in each of those years because I don't know how to write an internally consistent, plausible trajectory. I don't know how to write even one of those for anything except a ridiculously fast takeoff.” --Buck Shlegeris This vignette was hard to write. To achieve the desired level of detail I had to make a bunch of stuff up, but in order to be realistic I had to constantly ask “but actually though, what would really happen in this situation?” which made it painfully obvious how little I know about the future. There are numerous points where I had to conclude “Well, this does seem implausible, but I can’t think of anything more plausible at the moment and I need to move on.” I fully expect the actual world to diverge quickly from the trajectory laid out here. Let anyone who (with the benefit of hindsight) claims this divergence as evidence against my judgment prove it by exhibiting a vignette/trajectory they themselves wrote in 2021. If it maintains a similar level of detail (and thus sticks its neck out just as much) while being more accurate, I bow deeply in respect! I hope this inspires other people to write more vignettes soon. We at the Center on Long-Term Risk would like to have a collection to use for strategy discussions. Let me know if you’d like to do this, and I can give you advice & encouragement! I’d be happy to run another workshop. 2022 GPT-3 is finally obsolete. OpenAI, Google, Facebook, and DeepMind all have gigantic multimodal transformers, similar in size to GPT-3 but trained on images, video, maybe audio too, and generally higher-quality data. Not only that, but they are now typically fine-tuned in various ways--for example, to answer questions correctly, or produce engaging conversation as a chatbot. The chatbots are fun to talk to but erratic and ultimately considered s...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Reality-Revealing and Reality-Masking Puzzles, published by AnnaSalamon on LessWrong. Write a Review Tl;dr: I’ll try here to show how CFAR’s “art of rationality” has evolved over time, and what has driven that evolution. In the course of this, I’ll introduce the distinction between what I’ll call “reality-revealing puzzles” and “reality-masking puzzles”—a distinction that I think is almost necessary for anyone attempting to develop a psychological art in ways that will help rather than harm. (And one I wish I’d had explicitly back when the Center for Applied Rationality was founded.) I’ll also be trying to elaborate, here, on the notion we at CFAR have recently been tossing around about CFAR being an attempt to bridge between common sense and Singularity scenarios—an attempt to figure out how people can stay grounded in common sense and ordinary decency and humane values and so on, while also taking in (and planning actions within) the kind of universe we may actually be living in. Arts grow from puzzles. I like to look at mathematics, or music, or ungodly things like marketing, and ask: What puzzles were its creators tinkering with that led them to leave behind these structures? (Structures now being used by other people, for other reasons.) I picture arts like coral reefs. Coral polyps build shell-bits for their own reasons, but over time there accumulates a reef usable by others. Math built up like this—and math is now a powerful structure for building from. [Sales and Freud and modern marketing/self-help/sales etc. built up some patterns too—and our basic way of seeing each other and ourselves is now built partly in and from all these structures, for better and for worse.] So let’s ask: What sort of reef is CFAR living within, and adding to? From what puzzles (what patterns of tinkering) has our “rationality” accumulated? Two kinds of puzzles: “reality-revealing” and “reality-masking” First, some background. Some puzzles invite a kind of tinkering that lets the world in and leaves you smarter. A kid whittling with a pocket knife is entangling her mind with bits of reality. So is a driver who notices something small about how pedestrians dart into streets, and adjusts accordingly. So also is the mathematician at her daily work. And so on. Other puzzles (or other contexts) invite a kind of tinkering that has the opposite effect. They invite a tinkering that gradually figures out how to mask parts of the world from your vision. For example, some months into my work as a math tutor I realized I’d been unconsciously learning how to cue my students into acting like my words made sense (even when they didn’t). I’d learned to mask from my own senses the clues about what my students were and were not learning. We’ll be referring to these puzzle-types a lot, so it’ll help to have a term for them. I’ll call these puzzles “good” or “reality-revealing” puzzles, and “bad” or “reality-masking” puzzles, respectively. Both puzzle-types appear abundantly in most folks’ lives, often mixed together. The same kid with the pocket knife who is busy entangling her mind with data about bark and woodchips and fine motor patterns (from the “good” puzzle of “how can I whittle this stick”), may simultaneously be busy tinkering with the “bad” puzzle of “how can I not-notice when my creations fall short of my hopes.” (Even “good” puzzles can cause skill loss: a person who studies Dvorak may lose some of their QWERTY skill, and someone who adapts to the unselfconscious arguing of the math department may do worse for a while in contexts requiring tact. The distinction is that “good” puzzles do this only incidentally. Good puzzles do not invite a search for configurations that mask bits of reality. Whereas with me and my math tutees, say, there was a direct reward/conditioning response that happe...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Is Success the Enemy of Freedom? (Full), published by alkjash on LessWrong. Write a Review This is a linkpost for/ I. Parables A. Anna is a graduate student studying p-adic quasicoherent topology. It’s a niche subfield of mathematics where Anna feels comfortable working on neat little problems with the small handful of researchers interested in this topic. Last year, Anna stumbled upon a connection between her pet problem and algebraic matroid theory, solving a big open conjecture in the matroid Langlands program. Initially, she was over the moon about the awards and the Quanta articles, but now that things have returned to normal, her advisor is pressuring her to continue working with the matroid theorists with their massive NSF grants and real-world applications. Anna hasn’t had time to think about p-adic quasicoherent topology in months. B. Ben is one of the top Tetris players in the world, infamous for his signature move: the reverse double T-spin. Ben spent years perfecting this move, which requires lightning fast reflexes and nerves of steel, and has won dozens of tournaments on its back. Recently, Ben felt like his other Tetris skills needed work and tried to play online without using his signature move, but was greeted by a long string of losses: the Tetris servers kept matching him with the other top players in the world, who absolutely stomped him. Discouraged, Ben gave up on the endeavor and went back to practicing the reverse double T-spin. C. Clara was just promoted to be the youngest Engineering Director at a mid-sized software startup. She quickly climbed the ranks, thanks to her amazing knowledge of all things object-oriented and her excellent communication skills. These days, she finds her schedule packed with what the company needs: back-to-back high-level strategy meetings preparing for the optics of the next product launch, instead of what she loves: rewriting whole codebases in Haskell++. D. Deborah started her writing career as a small-time crime novelist, who split her time between a colorful cast of sleuthy protagonists. One day, her spunky children’s character Detective Dolly blew up in popularity due to a Fruit Loops advertising campaign. At the beginning of every month, Deborah tells herself she’s going to finally kill off Dolly and get to work on that grand historical romance she’s been dreaming about. At the end of every month, Deborah’s husband comes home with the mortgage bills for their expensive bayside mansion, paid for with “Dolly money,” and Deborah starts yet another Elementary School Enigma. E. While checking his email in the wee hours of the morning, Professor Evan Evanson notices an appealing seminar announcement: “A Gentle Introduction to P-adic Quasicoherent Topology (Part the First).” Ever since being exposed to the topic in his undergraduate matroid theory class, Evan has always wanted to learn more. He arrives bright and early on the day of the seminar and finds a prime seat, but as others file into the lecture hall, he’s greeted by a mortifying realization: it’s a graduate student learning seminar, and he’s the only faculty member present. Squeezing in his embarrassment, Evan sits through the talk and learns quite a bit of fascinating new mathematics. For some reason, even though he enjoyed the experience, Evan never comes back for Part the Second. F. Whenever Frank looks back to his college years, he remembers most fondly the day he was kicked out of the conservative school newspaper for penning a provocative piece about jailing all billionaires. Although he was a mediocre student with a medium-sized drinking problem, on that day Frank felt like a man with principles. A real American patriot in the ranks of Patrick Henry or Thomas Jefferson. After college, Frank met a girl who helped him sort himself out and get sober, a...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Lessons I've Learned from Self-Teaching, published by TurnTrout on LessWrong. In 2018, I was a bright-eyed grad student who was freaking out about AI alignment. I guess I'm still a bright-eyed grad student freaking out about AI alignment, but that's beside the point. I wanted to help, and so I started levelling up. While I'd read Nate Soares's self-teaching posts, there were a few key lessons I'd either failed to internalize or failed to consider at all. I think that implementing these might have doubled the benefit I drew from my studies. I can't usefully write a letter to my past self, so let me write a letter to you instead, keeping in mind that good advice for past-me may not be good advice for you. Make Sure You Remember The Content TL;DR: use a spaced repetition system like Anki. Put in cards for key concepts and practice using the concepts. Review the cards every day without fail. This is the most important piece of advice. The first few months of 2018 were a dream: I was learning math, having fun, and remaking myself. I read and reviewed about one textbook a month. I was learning how to math, how to write proofs and read equations fluently and think rigorously. I had so much fun that I hurt my wrists typing up my thoughts on impact measures. This turned a lot of my life upside-down. My wrists wouldn't fully heal for two years, and a lot happened during that time. After I hurt my wrists, I became somewhat depressed, posted less frequently, and read fewer books. When I looked back in 2019/2020 and asked "when and why did my love for textbooks sputter out?", the obvious answer was "when I hurt my hands and lost my sense of autonomy and became depressed, perchance? And maybe I just became averse to reading that way?" The obvious answer was wrong, but its obvious-ness stopped me from finding the truth until late last year. It felt right, but my introspection had failed me. The real answer is: when I started learning math, I gained a lot of implicit knowledge, like how to write proofs and read math (relatively) quickly. However, I'm no Hermione Granger: left unaided, I'm bad at remembering explicit facts / theorem statements / etc. I gained implicit knowledge but I didn't remember the actual definitions, unless I actually used them regularly (e.g. as I did for real analysis, which I remained quite fluent in and which I regularly use in my research). Furthermore, I think I coincidentally hit steeply diminishing returns on the implicit knowledge around when I injured myself. So basically I'm reading these math textbooks, doing the problems, getting a bit better at writing proofs but not really durably remembering 95% of the content. Maybe part of my subconscious noticed that I seem to be wasting time, that when I come back four months after reading a third of a graph theory textbook, I barely remember the new content I had "learned." I thought I was doing things right. I was doing dozens of exercises and thinking deeply about why each definition was the way it was, thinking about how I could apply these theorems to better reason about my own life and my own research, etc. I explicitly noticed this problem in late 2020 and thought, is there any way I know of to better retain content? ... gee, what about that thing I did in college that let me learn how to read 2,136 standard-use Japanese characters in 90 days? you know, Anki spaced repetition, that thing I never tried for math because once I tried and failed to memorize dozens of lines of MergeSort pseudocode with it? hm... This was the moment I started feeling extremely silly (the exact thought was "there's no possible way that my hand is big enough for how facepalm this moment is", IIRC), but also extremely excited. I could fix my problem! And a problem this was. In early 2020, I had an interview where I was asked t...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Expecting Short Inferential Distances, published by Eliezer Yudkowsky on LessWrong. Homo sapiens’s environment of evolutionary adaptedness (a.k.a. EEA or “ancestral environment”) consisted of hunter-gatherer bands of at most 200 people, with no writing. All inherited knowledge was passed down by speech and memory. In a world like that, all background knowledge is universal knowledge. All information not strictly private is public, period. In the ancestral environment, you were unlikely to end up more than one inferential step away from anyone else. When you discover a new oasis, you don’t have to explain to your fellow tribe members what an oasis is, or why it’s a good idea to drink water, or how to walk. Only you know where the oasis lies; this is private knowledge. But everyone has the background to understand your description of the oasis, the concepts needed to think about water; this is universal knowledge. When you explain things in an ancestral environment, you almost never have to explain your concepts. At most you have to explain one new concept, not two or more simultaneously. In the ancestral environment there were no abstract disciplines with vast bodies of carefully gathered evidence generalized into elegant theories transmitted by written books whose conclusions are a hundred inferential steps removed from universally shared background premises. In the ancestral environment, anyone who says something with no obvious support is a liar or an idiot. You’re not likely to think, “Hey, maybe this person has well-supported background knowledge that no one in my band has even heard of,” because it was a reliable invariant of the ancestral environment that this didn’t happen. Conversely, if you say something blatantly obvious and the other person doesn’t see it, they’re the idiot, or they’re being deliberately obstinate to annoy you. And to top it off, if someone says something with no obvious support and expects you to believe it—acting all indignant when you don’t—then they must be crazy. Combined with the illusion of transparency and self-anchoring (the tendency to model other minds as though the were slightly modified versions of oneself), I think this explains a lot about the legendary difficulty most scientists have in communicating with a lay audience—or even communicating with scientists from other disciplines. When I observe failures of explanation, I usually see the explainer taking one step back, when they need to take two or more steps back. Or listeners assume that things should be visible in one step, when they take two or more steps to explain. Both sides act as if they expect very short inferential distances from universal knowledge to any new knowledge. A biologist, speaking to a physicist, can justify evolution by saying it is the simplest explanation. But not everyone on Earth has been inculcated with that legendary history of science, from Newton to Einstein, which invests the phrase “simplest explanation” with its awesome import: a Word of Power, spoken at the birth of theories and carved on their tombstones. To someone else, “But it’s the simplest explanation!” may sound like an interesting but hardly knockdown argument; it doesn’t feel like all that powerful a tool for comprehending office politics or fixing a broken car. Obviously the biologist is infatuated with their own ideas, too arrogant to be open to alternative explanations which sound just as plausible. (If it sounds plausible to me, it should sound plausible to any sane member of my band.) And from the biologist’s perspective, they can understand how evolution might sound a little odd at first—but when someone rejects evolution even after the biologist explains that it’s the simplest explanation, well, it’s clear that nonscientists are just idiots and there’s no point in talking ...
Welcome to the nonlinear library, where we use text-to-speech software to convert the best writing from the rationalist and ea communities into audio. This is: Are we in an AI overhang?, published by by Andy Jones on the LessWrong. Write a Review Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. Over on Developmental Stages of GPTs, orthonormal mentions it at least reduces the chance of a hardware overhang. An overhang is when you have had the ability to build transformative AI for quite some time, but you haven't because no-one's realised it's possible. Then someone does and surprise! It's a lot more capable than everyone expected. I am worried we're in an overhang right now. I think we right now have the ability to build an orders-of-magnitude more powerful system than we already have, and I think GPT-3 is the trigger for 100x larger projects at Google, Facebook and the like, with timelines measured in months. Investment Bounds GPT-3 is the first AI system that has obvious, immediate, transformative economic value. While much hay has been made about how much more expensive it is than a typical AI research project, in the wider context of megacorp investment, its costs are insignificant. GPT-3 has been estimated to cost $5m in compute to train, and - looking at the author list and OpenAI's overall size - maybe another $10m in labour. Google, Amazon and Microsoft each spend about $20bn/year on R&D and another $20bn each on capital expenditure. Very roughly, it totals to $100bn/year. Against this budget, dropping $1bn or more on scaling GPT up by another factor of 100x is entirely plausible right now. All that's necessary is that tech executives stop thinking of natural language processing as cutesy blue-sky research and start thinking in terms of quarters-till-profitability. A concrete example is Waymo, which is raising $2bn investment rounds - and that's for a technology with a much longer road to market. Compute Cost The other side of the equation is compute cost. The $5m GPT-3 training cost estimate comes from using V100s at $10k/unit and 30 TFLOPS, which is the performance without tensor cores being considered. Amortized over a year, this gives you about $1000/PFLOPS-day. However, this cost is driven up an order of magnitude by NVIDIA's monopolistic cloud contracts, while performance will be higher when taking tensor cores into account. The current hardware floor is nearer to the RTX 2080 TI's $1k/unit for 125 tensor-core TFLOPS, and that gives you $25/PFLOPS-day. This roughly aligns with AI Impacts’ current estimates, and offers another >10x speedup to our model. I strongly suspect other bottlenecks stop you from hitting that kind of efficiency or GPT-3 would've happened much sooner, but I still think $25/PFLOPS-day is a lower useful bound. Other Constraints I've focused on money so far because most of the current 3.5-month doubling times come from increasing investment. But money aside, there are a couple of other things that could prove to be the binding constraint. Scaling law breakdown. The GPT series' scaling is expected to break down around 10k pflops-days (§6.3), which is a long way short of the amount of cash on the table. This could be because the scaling analysis was done on 1024-token sequences. Maybe longer sequences can go further. More likely I'm misunderstanding something. Sequence length. GPT-3 uses 2048 tokens at a time, and that's with an efficient encoding that cripples it on many tasks. With the naive architecture, increasing the sequence length is quadratically expensive, and getting up to novel-length sequences is not very likely. But there are a lot of plausible ways to fix that, and complexity is no bar AI. This constraint might plausibly not be resolved on a timescale of months, however. Data availability. From the same paper as the previous point, dataset size rises with the square-root of compute; a 1000x larger GPT-3 would want 10 trillion tokens of train...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: RadVac Commercial Antibody Test Results, published by johnswentworth on the LessWrong. Background: Making Vaccine Results are in from the commercial antibody tests. Both my girlfriend and I came back negative - the test did not detect any Spike antibody response in the blood. This post will talk about how I'm updating based on these results, and the next steps. Here's our timeline so far; more info on the vaccine is in the original post and the radvac whitepaper: We've taken five doses, spaced apart weekly (on Tuesdays). The first three doses only included six of the nine peptides, due to delays from the manufacturer. (Spike 660, Spike 1145, and Orf1 5471T were the three missing.) The blood draw for this test took place the day after the fifth dose. I expect this is too soon to notice significant impact from the last two doses; vaccines in general seem to typically take 2-3 weeks to kick in, and that is my expectation for this one as well. (Also, it was an "IgG antibody test", and WebMD says these antibodies typically take about 2 weeks to show up after covid symptoms show from an actual infection.) This is intended to mainly be a test of the first three doses. The test apparently used the "DiaSorin Liaison(R) SARS-CoV-2 S1/S2 IgG assay" (I didn't know this until the results came in). According to the FDA, it has about 92% sensitivity and 99% specificity. The "S1/S2" part indicates that it's testing for response to the S1 and S2 subunits of the spike protein - together, these are essentially the whole spike protein. Important thing to notice: the test was looking for Spike antibodies, and two of our three missing peptides were Spike peptides. Indeed, there were only 3 Spike peptides among the full 9, so with two missing, we only had one Spike peptide in our first three doses. (The rest target other parts of the virus.) So that makes the test significantly less useful than it would otherwise be, and makes me more inclined to get another test in 2-3 weeks when the doses with the other three peptides have had time to kick in. How I'm Updating In the original post, I called this test "searching under the streetlamp". It wasn't super likely to come back positive even assuming the vaccine worked as intended, but it was relatively cheap and easy to run the test, so it was our first check. Given the missing Spike peptides and the test only checking against Spike, it was even more likely to come back negative than I originally estimated. In Jacob's prediction questions, I gave roughly a 25% chance that a commercial antibody test would pass for most people, given three doses and all 9 peptides. I gave the vaccine about 75% chance of working overall, distributed over several different possible worlds. In this specific scenario, it's clear that the prior on test passing should be even lower. (Reminder on the possible worlds: the vaccine could induce antibody response in the blood and mucus, only mucus, or not at all. It could induce T-cell response separate from antibody response. It could work sometimes, much like how the first dose of commercial mRNA vaccines tend to work in 75% or 85% of people, and in that case I expect more doses/more time to make it work more often.) After updating on the results, I'm down to about 60-70% chance of working overall. Unfortunately this test just didn't give us very much information - at least about the vaccine working. Aside from the test result, we do have one more small piece of information to update on: I was quite congested for 1-2 days after the most recent three doses (and I was generally not congested the rest of the week). That's exactly what we'd expect to see if the vaccine is working as intended, and it's pretty strong evidence that it's doing something. Updating on both that and the test results, I'm at ~70% that it works overall...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Politics is way too meta, published by Rob Bensinger on the LessWrong. ... i.e., it doesn't spend enough time arguing about object-level things. The way I'm using it in this post, "object-level" might include these kinds of things: While serving as Secretary of State, did Hillary Clinton send classified information to an insecure email server? How large was the risk that attackers might get the information, and how much harm might that cause? What are the costs and benefits of various communications protocols (e.g., the legal one during Clinton's tenure, the de facto one Clinton followed, or other possibilities), and how should we weight those costs and benefits? How can we best forecast people's reliability on security issues? Are once-off mistakes like this predictive of future sloppiness? Are there better ways of predicting this? "Meta" might include things like: How much do voters care about Clinton's use of her email server? How much will reporters cover this story, and how much is their coverage likely to influence voters? What do various people (with no special information or expertise) believe about Clinton's email server, and how might these beliefs change their behavior? I'll also consider discussions of abstract authority, principle, or symbolism more "meta" than concrete policy proposals and questions of fact. This is too meta: Meta stuff is real. Elections are real, and matter. Popularity, status, controversy, and Overton windows have real physical effects. But it's possible to focus too much on one part of reality and neglect another. If you're driving a car while talking on the phone, the phone and your eyes are both perfectly good information channels; but if you allocate too little attention to the road, you still die. When speaking the demon's name creates the demon I claim: There are many good ideas that start out discussed by blogs and journal articles for a long time, then get adopted by policymakers. In many of these cases, you could delay adoption by many years by adding more sentences to the blog posts noting the political infeasibility or controversialness of the proposal. Or you could hasten adoption by making the posts just focus on analyzing the effects of the policy, without taking a moment to nervously look over their shoulder, without ritually bowing to the Overton window as though it were an authority on immigration law. I also claim that this obeys the same basic causal dynamics as: My friend Azzie posts something that I find cringe. So I decide to loudly and publicly (!) warn Azzie "hey, the thing you're doing is cringe!". Because, y'know, I want to help. Regardless of how "cringe" the average reader would have considered the post, saying it out loud can only help strengthen the perceived level of cringe. Or, suppose Bel overhears me and it doesn't cause her to see the post as more cringe. Still, it might make Bel worry that Cathy and other third parties think that the post is cringe. Which is a sufficient worry on its own to greatly change how Bel interacts with the post. Wouldn't want the cringe monster to come after you next! This can result in: Non-well-founded gaffes: statements that are controversial/offensive/impolitic largely or solely because some people think they sound like the kind of thing that would offend, alienate, or be disputed by a hypothetical third party. Or, worse: Even-less-than-non-well-founded gaffes: statements that are controversial/offensive/impolitic largely or solely because some people are worried that a hypothetical third party might think that a hypothetical fourth party might be offended, alienated, or unconvinced by the statement. See: Common Knowledge and Miasma. See: mimesis, herding, bystander effect, conformity instincts, coalitional instincts, and The World Forager Elite. Regardless of how poli...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: DeepMind: Generally capable agents emerge from open-ended play, published by Daniel Kokotajlo on the LessWrong. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. This is a linkpost for EDIT: Also see paper and results compilation video! Today, we published "Open-Ended Learning Leads to Generally Capable Agents," a preprint detailing our first steps to train an agent capable of playing many different games without needing human interaction data. ... The result is an agent with the ability to succeed at a wide spectrum of tasks — from simple object-finding problems to complex games like hide and seek and capture the flag, which were not encountered during training. We find the agent exhibits general, heuristic behaviours such as experimentation, behaviours that are widely applicable to many tasks rather than specialised to an individual task. The neural network architecture we use provides an attention mechanism over the agent’s internal recurrent state — helping guide the agent’s attention with estimates of subgoals unique to the game the agent is playing. We’ve found this goal-attentive agent (GOAT) learns more generally capable policies. Playing roughly 700,000 unique games in 4,000 unique worlds within XLand, each agent in the final generation experienced 200 billion training steps as a result of 3.4 million unique tasks. At this time, our agents have been able to participate in every procedurally generated evaluation task except for a handful that were impossible even for a human. And the results we’re seeing clearly exhibit general, zero-shot behaviour across the task space — with the frontier of normalised score percentiles continually improving. Looking qualitatively at our agents, we often see general, heuristic behaviours emerge — rather than highly optimised, specific behaviours for individual tasks. Instead of agents knowing exactly the “best thing” to do in a new situation, we see evidence of agents experimenting and changing the state of the world until they’ve achieved a rewarding state. We also see agents rely on the use of other tools, including objects to occlude visibility, to create ramps, and to retrieve other objects. Because the environment is multiplayer, we can examine the progression of agent behaviours while training on held-out social dilemmas, such as in a game of “chicken”. As training progresses, our agents appear to exhibit more cooperative behaviour when playing with a copy of themselves. Given the nature of the environment, it is difficult to pinpoint intentionality — the behaviours we see often appear to be accidental, but still we see them occur consistently. My hot take: This seems like a somewhat big deal to me. It's what I would have predicted, but that's scary, given my timelines. I haven't read the paper itself yet but I look forward to seeing more numbers and scaling trends and attempting to extrapolate... When I do I'll leave a comment with my thoughts. EDIT: My warm take: The details in the paper back up the claims it makes in the title and abstract. This is the GPT-1 of agent/goal-directed AGI; it is the proof of concept. Two more papers down the line (and a few OOMs more compute), and we'll have the agent/goal-directed AGI equivalent of GPT-3. Scary stuff. Thanks for listening. to help us out with the nonlinear library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Least Convenient Possible World, published Scott Alexander on the LessWrong. Related to: Is That Your True Rejection? "If you’re interested in being on the right side of disputes, you will refute your opponents’ arguments. But if you’re interested in producing truth, you will fix your opponents’ arguments for them. To win, you must fight not only the creature you encounter; you must fight the most horrible thing that can be constructed from its corpse." -- Black Belt Bayesian, via Rationality Quotes 13 Yesterday John Maxwell's post wondered how much the average person would do to save ten people from a ruthless tyrant. I remember asking some of my friends a vaguely related question as part of an investigation of the Trolley Problems: You are a doctor in a small rural hospital. You have ten patients, each of whom is dying for the lack of a separate organ; that is, one person needs a heart transplant, another needs a lung transplant, another needs a kidney transplant, and so on. A traveller walks into the hospital, mentioning how he has no family and no one knows that he's there. All of his organs seem healthy. You realize that by killing this traveller and distributing his organs among your patients, you could save ten lives. Would this be moral or not? I don't want to discuss the answer to this problem today. I want to discuss the answer one of my friends gave, because I think it illuminates a very interesting kind of defense mechanism that rationalists need to be watching for. My friend said: It wouldn't be moral. After all, people often reject organs from random donors. The traveller would probably be a genetic mismatch for your patients, and the transplantees would have to spend the rest of their lives on immunosuppressants, only to die within a few years when the drugs failed. On the one hand, I have to give my friend credit: his answer is biologically accurate, and beyond a doubt the technically correct answer to the question I asked. On the other hand, I don't have to give him very much credit: he completely missed the point and lost a valuable effort to examine the nature of morality. So I asked him, "In the least convenient possible world, the one where everyone was genetically compatible with everyone else and this objection was invalid, what would you do?" He mumbled something about counterfactuals and refused to answer. But I learned something very important from him, and that is to always ask this question of myself. Sometimes the least convenient possible world is the only place where I can figure out my true motivations, or which step to take next. I offer three examples: 1: Pascal's Wager. Upon being presented with Pascal's Wager, one of the first things most atheists think of is this: Perhaps God values intellectual integrity so highly that He is prepared to reward honest atheists, but will punish anyone who practices a religion he does not truly believe simply for personal gain. Or perhaps, as the Discordians claim, "Hell is reserved for people who believe in it, and the hottest levels of Hell are reserved for people who believe in it on the principle that they'll go there if they don't." This is a good argument against Pascal's Wager, but it isn't the least convenient possible world. The least convenient possible world is the one where Omega, the completely trustworthy superintelligence who is always right, informs you that God definitely doesn't value intellectual integrity that much. In fact (Omega tells you) either God does not exist or the Catholics are right about absolutely everything. Would you become a Catholic in this world? Or are you willing to admit that maybe your rejection of Pascal's Wager has less to do with a hypothesized pro-atheism God, and more to do with a belief that it's wrong to abandon your intellectual integrity on the ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The LessWrong Team is now Lightcone Infrastructure, come work with us!, published by habryka on the LessWrong. tl;dr: The LessWrong team is re-organizing as Lightcone Infrastructure. LessWrong is one of several projects we are working on to ensure the future of humanity goes well. We are looking to hire software engineers as well as generalist entrepreneurs in Berkeley who are excited to build infrastructure to ensure a good future. I founded the LessWrong 2.0 team in 2017, with the goal of reviving LessWrong.com and reinvigorating the intellectual culture of the rationality community. I believed the community had great potential for affecting the long term future, but that the failing website was a key bottleneck to community health and growth. Four years later, the website still seems very important. But when I step back and ask “what are the key bottlenecks for improving the longterm future?”, just ensuring the website is going well no longer seems sufficient. For the past year, I’ve been re-organizing the LessWrong team into something with a larger scope. As I’ve learned from talking to over a thousand of you over the last 4 years, for most of you the rationality community is much larger than just this website, and your contributions to the future of humanity more frequently than not route through many disparate parts of our sprawling diaspora. Many more of those parts deserve attention and optimization than just LessWrong, and we seem to be the best positioned organization to make sure that happens. I want to make sure that that whole ecosystem is successfully steering humanity towards safer and better futures, and more and more this has meant working on projects that weren't directly related to LessWrong.com: A bit over a year ago we started building grant-making software for Jaan Tallinn and the Survival and Flourishing Fund, helping distribute over 30 million dollars to projects that I think have the potential to have a substantial effect on ensuring a flourishing future for humanity. We helped run dozens of online meetups and events during the pandemic, and hundreds of in-person events for both this year and 2019s ACX Meetups everywhere We helped build and run the EA Forum and the AI Alignment Forum, We recently ran a 5-day retreat for 60-70 people whose work we think is highly impactful in reducing the likelihood of humanity's extinction, We opened an in-person office space in the Bay Area for organizations that are working towards improving the long-term future of humanity. As our projects outside of the LessWrong.com website multiplied, our name became more and more confusing when trying to explain to people what we were about. This confusion reached a new peak when we started having a team that we were internally calling the "LessWrong team", which was responsible for running the website, distinct from all of our other projects, and which soon after caused me to utter the following sentence at one of our team meetings: LessWrong really needs to figure out what the LessWrong team should set as a top priority for LessWrong As one can imagine, the reaction from the rest of the team was confusion and laughter and at that point I knew we had to change our name and clarify our organizational mission. So, after doing many rounds of coming up with names, asking many of our colleagues and friends (including GPT-3) for suggestions, we finally decided on: I like the light cone as a symbol, because it represents the massive scale of opportunity that humanity is presented with. If things go right, we can shape almost the full light cone of humanity to be full of flourishing life. Billions of galaxies, billions of light years across, for some 10^36 (or so) years until the heat death of the universe. Separately, I am excited about where Lightcone Infrastructure is head...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Your Dog is Even Smarter Than You Think, published by StyleOfDog on the LessWrong. Epistemic status: highly suggestive. Epistemic status: highly suggestive. [EDIT: Added more info on research methods. Addressed some common criticism. Added titles for video links and a few new vids. Prevented revolution with a military coup d'état] A combination of surveys and bayesian estimates[1] leads me to believe this community is interested in autism, cats, cognition, philosophy, and moral valence of animals. What I’m going to show you checks every box, so it boggles my mind that I don't see anyone talk about it. It has been bothering me so much that I decided to create an account and write this article. I have two theories. The community will ignore fascinating insight just because its normie coded. Cute tiktok-famous poodle doesn't pattern match to "this contains real insight into animal cognition". Nobody tried to sell this well enough. I personally believe in the second one[2] and I'll try to sell it to you. Stella There’s an intervention to help non-verbal autistic kids communicate using “communication boards” (not to be confused with facilitated communication which has a bad reputation). It can be a paper board with pictures or it can be a board with buttons that say a word when pressed. In 2018 Christina Hunger (hungerforwords.com) - a speech pathologist working with autistic children using such boards - started to wonder if her dog was in fact autistic. Just kidding, she saw similarities in patterns of behavior between young kids she was working with ("learner" seems to be the term of art) and her dog. So she gave it a button that says “Outside” and expanded from there. Now teaching a dog to press a button that says “outside” is not impressive or interesting to me. But then she kept adding buttons and her dog started to display capabilities for rudimentary syntax. Stella the talking dog compilation - Stella answers whether she wants to play or eat, asks for help when one of her buttons breaks, alerts owner to possible "danger" outside. Stella tells us she is all done with her bed! Stella the Talking Dog says "Help Good Want Eat" (most of the good videos are on her Instagram @hunger4words, not much is on YouTube) Reaction from serious animal language researchers and animal cognition hobbyists was muted to non-existent, but dog moms ate this stuff up. One of them was Alexis. Bunny Most useful research is impractical to do within academia The Importance of Methodology and Practical Matters Ethology has some really interesting lessons about how important various practical matters and methodology can be when it comes to what your field can (and can't) produce. For example, it turns out that a surprising amount of useful data about animal cognition comes from experiments with dogs. [.] The main reason is because they will sit still for an fMRI to be the goodest boy (and to get hot dogs). [.] On the other side of that coin, elephants are clearly very smart, but we've done surprisingly little controlled experiments or close observation with them. Why? [.] They're damn inconvenient to keep in the basement of the biology building, they mess up the trees on alumni drive, and undergrads kept complaining about elephant-patty injuries while playing ultimate on the quad. A lot of useful research isn't done because it's too inconvenient, too expensive or otherwise impractical to execute within confines of academia. This is a massive shaping force. Existence of ImageNet and its quirks is a stronger shaping force on AI research than all AI ethics committees combined. Nobody had done this before because it takes months of everyday training to get interesting results. Once your dog gets the hang of it, you’re able to add more buttons faster, but it’s never quick. Dogs take a while to come u...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Embedded Interactive Predictions on LessWrong, published by Amandango on the LessWrong. Write a Review Ought and LessWrong are excited to launch an embedded interactive prediction feature. You can now embed binary questions into LessWrong posts and comments. Hover over the widget to see other people’s predictions, and click to add your own. Try it out 1% 2% 3% 4% 5% 6% 7% 8% 9% 10% 11% 12% 13% 14% 15% 16% 17% 18% 19% 20% 21% 22% 23% 24% 25% 26% 27% 28% 29% 30% 31% 32% 33% 34% 35% 36% 37% 38% 39% 40% 41% 42% 43% 44% 45% 46% 47% 48% 49% 50% 51% 52% 53% 54% 55% 56% 57% 58% 59% 60% 61% 62% 63% 64% 65% 66% 67% 68% 69% 70% 71% 72% 73% 74% 75% 76% 77% 78% 79% 80% 81% 82% 83% 84% 85% 86% 87% 88% 89% 90% 91% 92% 93% 94% 95% 96% 97% 98% 99% 1% Will there be more than 50 prediction questions embedded in LessWrong posts and comments this month? 99% How to use this Create a question Go to elicit.org/binary and create your question by typing it into the field at the top Click on the question title, and click the copy button next to the title – it looks like this: Paste the URL into your LW post or comment. It'll look like this in the editor: Make a prediction Click on the widget to add your own prediction Click on your prediction line again to delete it Link your accounts Linking your LessWrong and Elicit accounts allows you to: Filter for and browse all your LessWrong predictions on Elicit Add notes to your LessWrong predictions on Elicit See your calibration for your LessWrong predictions on Elicit Predict on LessWrong questions in the Elicit app To link your accounts: Make an Elicit account Send me (amanda@ought.org) an email with your LessWrong username and your Elicit account email Motivation We hope embedded predictions can prompt readers and authors to: Actively engage with posts. By making predictions as they read, people have to stop and think periodically about how much they agree with the author. Distill claims. For writers, integrating predictions challenges them to think more concretely about their claims and how readers might disagree. Communicate uncertainty. Rather than just stating claims, writers can also communicate a confidence level. Collect predictions. As a reader, you can build up a personal database of predictions as you browse LessWrong. Get granular feedback. Writers can get feedback on their content at a more granular level than comments or upvotes. By working with LessWrong on this, Ought hopes to make forecasting easier and more prevalent. As we learn more about how people think about the future, we can use Elicit to automate larger parts of the workflow and thought process until we end up with end-to-end automated reasoning that people endorse. Check out our blog post to see demos and more context. Some examples of how to use this To make specific predictions, like in Zvi’s post on COVID predictions To express credences on claims like those in Daniel Kokotajlo’s soft takeoff post Beyond LessWrong – if you want to integrate this into your blog or have other ideas for places you’d want to use this, let us know! Thanks for listening. to help us out with the nonlinear library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Fable of Science and Politics, published by Eliezer Yudkowsky on the LessWrong. In the time of the Roman Empire, civic life was divided between the Blue and Green factions. The Blues and the Greens murdered each other in single combats, in ambushes, in group battles, in riots. Procopius said of the warring factions: “So there grows up in them against their fellow men a hostility which has no cause, and at no time does it cease or disappear, for it gives place neither to the ties of marriage nor of relationship nor of friendship, and the case is the same even though those who differ with respect to these colors be brothers or any other kin.”1 Edward Gibbon wrote: “The support of a faction became necessary to every candidate for civil or ecclesiastical honors.”2 Who were the Blues and the Greens? They were sports fans—the partisans of the blue and green chariot-racing teams. Imagine a future society that flees into a vast underground network of caverns and seals the entrances. We shall not specify whether they flee disease, war, or radiation; we shall suppose the first Undergrounders manage to grow food, find water, recycle air, make light, and survive, and that their descendants thrive and eventually form cities. Of the world above, there are only legends written on scraps of paper; and one of these scraps of paper describes the sky, a vast open space of air above a great unbounded floor. The sky is cerulean in color, and contains strange floating objects like enormous tufts of white cotton. But the meaning of the word “cerulean” is controversial; some say that it refers to the color known as “blue,” and others that it refers to the color known as “green.” In the early days of the underground society, the Blues and Greens contested with open violence; but today, truce prevails—a peace born of a growing sense of pointlessness. Cultural mores have changed; there is a large and prosperous middle class that has grown up with effective law enforcement and become unaccustomed to violence. The schools provide some sense of historical perspective; how long the battle between Blues and Greens continued, how many died, how little changed as a result. Minds have been laid open to the strange new philosophy that people are people, whether they be Blue or Green. The conflict has not vanished. Society is still divided along Blue and Green lines, and there is a “Blue” and a “Green” position on almost every contemporary issue of political or cultural importance. The Blues advocate taxes on individual incomes, the Greens advocate taxes on merchant sales; the Blues advocate stricter marriage laws, while the Greens wish to make it easier to obtain divorces; the Blues take their support from the heart of city areas, while the more distant farmers and watersellers tend to be Green; the Blues believe that the Earth is a huge spherical rock at the center of the universe, the Greens that it is a huge flat rock circling some other object called a Sun. Not every Blue or every Green citizen takes the “Blue” or “Green” position on every issue, but it would be rare to find a city merchant who believed the sky was blue, and yet advocated an individual tax and freer marriage laws. The Underground is still polarized; an uneasy peace. A few folk genuinely think that Blues and Greens should be friends, and it is now common for a Green to patronize a Blue shop, or for a Blue to visit a Green tavern. Yet from a truce originally born of exhaustion, there is a quietly growing spirit of tolerance, even friendship. One day, the Underground is shaken by a minor earthquake. A sightseeing party of six is caught in the tremblor while looking at the ruins of ancient dwellings in the upper caverns. They feel the brief movement of the rock under their feet, and one of the tourists trips and scrapes her knee. The...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Seven Years of Spaced Repetition Software in the Classroom, published by tanagrabeast on the LessWrong. Description This is a reflective essay and report on my experiences using Spaced Repetition Software (SRS) in an American high school classroom. It follows my 2015 and 2016 posts on the same topic. Because I value concise summaries in non-fiction, I provide one immediately below. However, I also believe in the power of narrative, in carefully unfolding a story so as to maximize reader engagement and impact. As I have applied such narrative considerations in writing this post, I consider the following summary to be a spoiler. I’ll let you decide what to do with that information. Summary (spoilers) My earlier push for classroom SRS solutions was driven by a belief I came to see as fallacious: that forgetting is the undoing of learning. This epistemic shift drove me to abandon designs for a custom app that would have integrated whole-class and individual SRS functions. While I still see value in classroom use of Spaced Repetition Software, especially in basic language acquisition, I have greatly reduced its use in my own classes. In my third year of experiments (2016-17), I used a windfall of classroom computers to give students supervised time to independently study using an SRS app with individual profiles. I found longer-term average performance to be slightly worse than under the whole-class group study model, though students of high intelligence and motivation saw slight improvements. Intro and response to Piotr Woźniak I have recently received a number of requests to revisit the topic of classroom SRS after years of silence on the subject. Understandably, the term “postmortem” has come up more than once. Did I hit a dead end? Do I still use it? Also, I was informed that SRS founding father Piotr Woźniak recently added a page to his SuperMemo wiki in which he quoted me at length and claimed that SRS doesn’t belong in the classroom. Well, I don’t have much in the way of rebuttal, because Woźniak’s main goal with the page seems to be to use my experience as ammunition against the perpetuation of school-as-we-know-it, which seems like a worthy crusade. He introduces my earlier classroom SRS posts by saying, “This teacher could write the same articles with the same conclusions. Only the terminology would differ.” I’ll take that as high praise. If I were to quibble, it would be with the part shortly after this, where he says: The entire analysis is made with an important assumption: "school is good, school is inevitable, and school is here to stay, so we better learn to live with it". Inevitable? Maybe. Here to stay? Realistically, yes. But good? At best, I might describe our educational system as an “inadequate equilibrium”. At worst? A pit so deep we still don’t know what’s at the bottom, except that it eats souls. Other than that, let me reiterate my long-running agreement with Woźniak that SRS is best when used by a self-motivated individual, and that my classroom antics are an ugly hack around the fact that self-motivation is a rare element this deep in the mines. Anyone who can show us a way out will have my attention. In the meantime, I’ll do my best to keep a light on. Prologue At the end of my 2016 post, I teased a peek at a classroom SRS+ app I was preparing to build. It would have married whole-class and individual study functions with some other clever features to reduce teacher workload. I had a 10k word document in hand: a mix of rationale, feature descriptions, and hypothetical “user stories”. I wasn’t looking for funding or a co-founder, just some technical suggestions and moral support. I would have been my own first user, and I had to keep my day job for that anyway. But each time I read my draft, I had this growing, sickening sense that I was lying ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Lies, Damn Lies, and Fabricated Options, published by Duncan_Sabien on the LessWrong. This is an essay about one of those "once you see it, you will see it everywhere" phenomena. It is a psychological and interpersonal dynamic roughly as common, and almost as destructive, as motte-and-bailey, and at least in my own personal experience it's been quite valuable to have it reified, so that I can quickly recognize the commonality between what I had previously thought of as completely unrelated situations. The original quote referenced in the title is "There are three kinds of lies: lies, damned lies, and statistics." Background 1: Gyroscopes Gyroscopes are weird. Except they're not. They're quite normal and mundane and straightforward. The weirdness of gyroscopes is a map-territory confusion—gyroscopes seem weird because my map is poorly made, and predicts that they will do something other than their normal, mundane, straightforward thing. In large part, this is because I don't have the consequences of physical law engraved deeply enough into my soul that they make intuitive sense. I can imagine a world that looks exactly like the world around me, in every way, except that in this imagined world, gyroscopes don't have any of their strange black-magic properties. It feels coherent to me. It feels like a world that could possibly exist. "Everything's the same, except gyroscopes do nothing special." Sure, why not. But in fact, this world is deeply, deeply incoherent. It is Not Possible with capital letters. And a physicist with sufficiently sharp intuitions would know this—would be able to see the implications of a world where gyroscopes "don't do anything weird," and tell me all of the ways in which reality falls apart. The seeming coherence of the imaginary world where gyroscopes don't balance and don't precess and don't resist certain kinds of motion is a product of my own ignorance, and of the looseness with which I am tracking how different facts fit together, and what the consequences of those facts are. It's like a toddler thinking that they can eat their slice of cake, and still have that very same slice of cake available to eat again the next morning. Background 2: H2O and XYZ In the book Labyrinths of Reason, author William Poundstone delves into various thought experiments (like Searle's Chinese Room) to see whether they're actually coherent or not. In one such exploration, he discusses the idea of a Twin Earth, on the opposite side of the sun, exactly like Earth in every way except that it doesn't have water. Instead, it has a chemical, labeled XYZ, which behaves like water and occupies water's place in biology and chemistry, but is unambiguously distinct. Once again, this is the sort of thing humans are capable of imagining. I can nod along and say "sure, a liquid that behaves just like water, but isn't." But a chemist, intimately familiar with the structure and behavior of molecules and with the properties of the elements and their isotopes, would be throwing up red flags. "Just like water," they might say, and I would nod. "Liquid, and transparent, with a density of 997 kilograms per meter cubed." "Sure," I would reply. "Which freezes and melts at exactly 0º Celsius, and which boils and condenses at exactly 100º Celsius." "Yyyyeahhhh," I would say, uneasiness settling in. "Which makes up roughly 70% of the mass of the bodies of the humans of Twin Earth, and which is a solvent for hydrophilic substances, but not hydrophobic ones, and which can hold ions and polar substances in solution." "Um." The more we drill down into what we mean by behaves exactly like water, the more it starts to become clear that there just isn't a possible substance which behaves exactly like water, but isn't. There are only so many configurations of electrons and protons and neutrons ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Book summary: Unlocking the Emotional Brain, published by Kaj_Sotala on the LessWrong. If the thesis in Unlocking the Emotional Brain (UtEB) is even half-right, it may be one of the most important books that I have read. Written by the psychotherapists Bruce Ecker, Robin Ticic and Laurel Hulley, it claims to offer a neuroscience-grounded, comprehensive model of how effective therapy works. In so doing, it also happens to formulate its theory in terms of belief updating, helping explain how the brain models the world and what kinds of techniques allow us to actually change our minds. Furthermore, if UtEB is correct, it also explains why rationalist techniques such as Internal Double Crux [1 2 3] work. UtEB’s premise is that much if not most of our behavior is driven by emotional learning. Intense emotions generate unconscious predictive models of how the world functions and what caused those emotions to occur. The brain then uses those models to guide our future behavior. Emotional issues and seemingly irrational behaviors are generated from implicit world-models (schemas) which have been formed in response to various external challenges. Each schema contains memories relating to times when the challenge has been encountered and mental structures describing both the problem and a solution to it. According to the authors, the key for updating such schemas involves a process of memory reconsolidation, originally identified in neuroscience. The emotional brain’s learnings are usually locked and not modifiable. However, once an emotional schema is activated, it is possible to simultaneously bring into awareness knowledge contradicting the active schema. When this happens, the information contained in the schema can be overwritten by the new knowledge. While I am not convinced that the authors are entirely right, many of the book’s claims definitely feel like they are pointing in the right direction. I will discuss some of my caveats and reservations after summarizing some of the book’s claims in general. I also consider its model in the light of an issue of a psychology/cognitive science journal devoted to discussing a very similar hypothesis. Emotional learning In UtEB’s model, emotional learning forms the foundation of much of our behavior. It sets our basic understanding about what situations are safe or unsafe, desirable or undesirable. The authors do not quite say it explicitly, but the general feeling I get is that the subcortical emotional processes set many of the priorities for what we want to achieve, with higher cognitive functions then trying to figure out how to achieve it - often remaining unaware of what exactly they are doing. UtEB’s first detailed example of an emotional schema comes from the case study of a man in his thirties they call Richard. He had been consistently successful and admired at work, but still suffered from serious self-doubt and low confidence at his job. On occasions such as daily technical meetings, when he considered saying something, he experienced thoughts including “Who am I to think I know what’s right?”, “This could be wrong” and “Watch out - don’t go out on a limb”. These prevented him from expressing any opinions. From the point of view of the authors, these thoughts have a definite cause - Richard has “emotional learnings according to which it is adaptively necessary to go into negative thoughts and feelings towards [himself].” The self-doubts are a strategy which his emotional brain has generated for solving some particular problem. Richard’s therapist guided Richard to imagine what it would feel like if he was at one of his work meetings, made useful comments, and felt confident in his knowledge while doing so. This was intended to elicit information about what Richard’s emotional brain predicted would happen if it failed ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Roles are Martial Arts for Agency, published by Eneasz on the LessWrong. A long time ago I thought that Martial Arts simply taught you how to fight – the right way to throw a punch, the best technique for blocking and countering an attack, etc. I thought training consisted of recognizing these attacks and choosing the correct responses more quickly, as well as simply faster/stronger physical execution of same. It was later that I learned that the entire purpose of martial arts is to train your body to react with minimal conscious deliberation, to remove “you” from the equation as much as possible. The reason is of course that conscious thought is too slow. If you have to think about what you’re doing, you’ve already lost. It’s been said that if you had to think about walking to do it, you’d never make it across the room. Fighting is no different. (It isn’t just fighting either – anything that requires quick reaction suffers when exposed to conscious thought. I used to love Rock Band. One day when playing a particularly difficult guitar solo on expert I nailed 100%. except “I” didn’t do it at all. My eyes saw the notes, my hands executed them, and no where was I involved in the process. It was both exhilarating and creepy, and I basically dropped the game soon after.) You’ve seen how long it takes a human to learn to walk effortlessly. That's a situation with a single constant force, an unmoving surface, no agents working against you, and minimal emotional agitation. No wonder it takes hundreds of hours, repeating the same basic movements over and over again, to attain even a basic level of martial mastery. To make your body react correctly without any thinking involved. When Neo says “I Know Kung Fu” he isn’t surprised that he now has knowledge he didn’t have before. He’s amazed that his body now reacts in the optimal manner when attacked without his involvement. All of this is simply focusing on pure reaction time – it doesn’t even take into account the emotional terror of another human seeking to do violence to you. It doesn’t capture the indecision of how to respond, the paralysis of having to choose between outcomes which are all awful and you don’t know which will be worse, and the surge of hormones. The training of your body to respond without your involvement bypasses all of those obstacles as well. This is the true strength of Martial Arts – eliminating your slow, conscious deliberation and acting while there is still time to do so. Roles are the Martial Arts of Agency. When one is well-trained in a certain Role, one defaults to certain prescribed actions immediately and confidently. I’ve acted as a guy standing around watching people faint in an overcrowded room, and I’ve acted as the guy telling people to clear the area. The difference was in one I had the role of Corporate Pleb, and the other I had the role of Guy Responsible For This Shit. You know the difference between the guy at the bar who breaks up a fight, and the guy who stands back and watches it happen? The former thinks of himself as the guy who stops fights. They could even be the same guy, on different nights. The role itself creates the actions, and it creates them as an immediate reflex. By the time corporate-me is done thinking “Huh, what’s this? Oh, this looks bad. Someone fainted? Wow, never seen that before. Damn, hope they’re OK. I should call 911.” enforcer-me has already yelled for the room to clear and whipped out a phone. Roles are the difference between Hufflepuffs gawking when Neville tumbles off his broom (Protected), and Harry screaming “Wingardium Leviosa” (Protector). Draco insulted them afterwards, but it wasn’t a fair insult – they never had the slightest chance to react in time, given the role they were in. Roles are the difference between Minerva ordering Hagrid to stay wi...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Dark Arts of Rationality, published by So8res on the LessWrong. Today, we're going to talk about Dark rationalist techniques: productivity tools which seem incoherent, mad, and downright irrational. These techniques include:So8res Willful Inconsistency Intentional Compartmentalization Modifying Terminal Goals I expect many of you are already up in arms. It seems obvious that consistency is a virtue, that compartmentalization is a flaw, and that one should never modify their terminal goals. I claim that these 'obvious' objections are incorrect, and that all three of these techniques can be instrumentally rational. In this article, I'll promote the strategic cultivation of false beliefs and condone mindhacking on the values you hold most dear. Truly, these are Dark Arts. I aim to convince you that sometimes, the benefits are worth the price. Changing your Terminal Goals In many games there is no "absolutely optimal" strategy. Consider the Prisoner's Dilemma. The optimal strategy depends entirely upon the strategies of the other players. Entirely. Intuitively, you may believe that there are some fixed "rational" strategies. Perhaps you think that even though complex behavior is dependent upon other players, there are still some constants, like "Never cooperate with DefectBot". DefectBot always defects against you, so you should never cooperate with it. Cooperating with DefectBot would be insane. Right? Wrong. If you find yourself on a playing field where everyone else is a TrollBot (players who cooperate with you if and only if you cooperate with DefectBot) then you should cooperate with DefectBots and defect against TrollBots. Consider that. There are playing fields where you should cooperate with DefectBot, even though that looks completely insane from a naïve viewpoint. Optimality is not a feature of the strategy, it is a relationship between the strategy and the playing field. Take this lesson to heart: in certain games, there are strange playing fields where the optimal move looks completely irrational. I'm here to convince you that life is one of those games, and that you occupy a strange playing field right now. Here's a toy example of a strange playing field, which illustrates the fact that even your terminal goals are not sacred: Imagine that you are completely self-consistent and have a utility function. For the sake of the thought experiment, pretend that your terminal goals are distinct, exclusive, orthogonal, and clearly labeled. You value your goals being achieved, but you have no preferences about how they are achieved or what happens afterwards (unless the goal explicitly mentions the past/future, in which case achieving the goal puts limits on the past/future). You possess at least two terminal goals, one of which we will call A. Omega descends from on high and makes you an offer. Omega will cause your terminal goal A to become achieved over a certain span of time, without any expenditure of resources. As a price of taking the offer, you must switch out terminal goal A for terminal goal B. Omega guarantees that B is orthogonal to A and all your other terminal goals. Omega further guarantees that you will achieve B using less time and resources than you would have spent on A. Any other concerns you have are addressed via similar guarantees. Clearly, you should take the offer. One of your terminal goals will be achieved, and while you'll be pursuing a new terminal goal that you (before the offer) don't care about, you'll come out ahead in terms of time and resources which can be spent achieving your other goals. So the optimal move, in this scenario, is to change your terminal goals. There are times when the optimal move of a rational agent is to hack its own terminal goals. You may find this counter-intuitive. It helps to remember that "optimality" depen...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Anti-social Punishment , published by Martin Sustrik on the LessWrong. This is a cross post from 250bpm.com. Introduction There's a trope among Slovak intellectual elite depicting an average Slovak as living in a village, sitting a local pub, drinking Borovička, criticizing everyone and everything but not willing to lift a finger to improve things. Moreover, it is assumed that if you actually tried to make things better, said individual would throw dirt at you and place obstacles in your way. I always assumed that this caricature was silly. It was partly because I have a soft spot for Slovak rural life but mainly because such behavior makes absolutely no sense from game-theoretical point of view. If a do-gooder is stupid enough to try to altruistically improve your life, why go into trouble of actively opposing them? Why not just sit safely hidden in the pub, drink some more Borovička and wait until they are done? Well, it turns out that the things are far more complex then I thought. Public goods game Benedikt Herrmann, Christian Thöni and Simon Gächter did a study of how people from different societies deal with cooperation and punishment. You can find the paper here and supporting material here. The study is based on the "public goods" game. The game works as follows: There are four players. Each player gets 20 tokens to start with. Every participant either keeps them or passes some of them into a common pool. After all the players are done with their moves, each of them, irrespective of how much they contributed, gets tokens equal to 40% of all the tokens in the common pool. The participants cannot communicate with each other and are unaware of each other's identities. The game is repeated, with the same players, 10 times in a row. The earnings, obviously, depend not only on subject's move but also on the willingness of the other players to cooperate and put tokens into the common pool. But free riders get an advantage. They keep their original tokens but also get their share from the pool. To get a feeling of the payoffs, let's have a look at the single-round earnings in the extreme case where each participant either puts all their tokens into the pool ("cooperator") or keeps all the tokens for themselves ("free-rider"): Public goods game with punishment There's a variant of the "public goods game" where players are able to punish each other after each round of the game. The mechanism is simple. When the round ends the participants are informed about how much each of them has put into the common pool. Then they decide whether to spend some of their tokens to administer punishment. For each token spent on punishment you can subtract 3 tokens from the earnings of an opponent. The players know that they've been punished but they are not informed about who exactly has punished them. Participant pools The researchers were interested in comparing the results of the game among different societies: Our research strategy was to conduct the experiments with comparable social groups from complex developed societies with the widest possible range of cultural and economic backgrounds to maximize chances of observing cross-societal differences in punishment and cooperation. The societies represented in our participant pools diverge strongly according to several widely used criteria developed by social scientists in order to characterize societies. This variation, covering a large range of the worldwide available values of the respective criteria, provides us with a novel test for seeing whether societal differences between complex societies have any impact on experimentally observable disparities in cooperation and punishment behavior. ... To minimize sociodemographic variability, we conducted all experiments with university undergraduates who were similar in age, shared an (...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How to Beat Procrastination, published by lukeprogon the Lesswrong. Part of the sequence: The Science of Winning at Life My own behavior baffles me. I find myself doing what I hate, and not doing what I really want to do! - Saint Paul (Romans 7:15) Once you're trained in BayesCraft, it may be tempting to tackle classic problems "from scratch" with your new Rationality Powers. But often, it's more effective to do a bit of scholarship first and at least start from the state of our scientific knowledge on the subject. Today, I want to tackle procrastination by summarizing what we know about it, and how to overcome it. Let me begin with three character vignettes... Eddie attended the sales seminar, read all the books, and repeated the self-affirmations in the mirror this morning. But he has yet to make his first sale. Rejection after rejection has demoralized him. He organizes his desk, surfs the internet, and puts off his cold calls until potential clients are leaving for the day. Three blocks away, Valerie stares at a blank document in Microsoft Word. Her essay assignment on municipal politics, due tomorrow, is mind-numbingly dull. She decides she needs a break, texts some friends, watches a show, and finds herself even less motivated to write the paper than before. At 10pm she dives in, but the result reflects the time she put into it: it's terrible. In the next apartment down, Tom is ahead of the game. He got his visa, bought his plane tickets, and booked time off for his vacation to the Dominican Republic. He still needs to reserve a hotel room, but that can be done anytime. Tom keeps pushing the task forward a week as he has more urgent things to do, and then forgets about it altogether. As he's packing, he remembers to book the room, but by now there are none left by the beach. When he arrives, he finds his room is 10 blocks from the beach and decorated with dead mosquitos. Eddie, Valerie, and Tom are all procrastinators, but in different ways.1 Eddie's problem is low expectancy. By now, he expects only failure. Eddie has low expectancy of success from making his next round of cold calls. Results from 39 procrastination studies show that low expectancy is a major cause of procrastination.2 You doubt your ability to follow through with the diet. You don't expect to get the job. You really should be going out and meeting girls and learning to flirt better, but you expect only rejection now, so you procrastinate. You have learned to be helpless. Valerie's problem is that her task has low value for her. We all put off what we dislike.3 It's easy to meet up with your friends for drinks or start playing a videogame; not so easy to start doing your taxes. This point may be obvious, but it's nice to see it confirmed in over a dozen scientific studies. We put off things we don't like to do. But the strongest predictor of procrastination is Tom's problem: impulsiveness. It would have been easy for Tom to book the hotel in advance, but he kept getting distracted by more urgent or interesting things, and didn't remember to book the hotel until the last minute, which left him with a poor selection of rooms. Dozens of studies have shown that procrastination is closely tied to impulsiveness.4 Impulsiveness fits into a broader component of procrastination: time. An event's impact on our decisions decreases as its temporal distance from us increases.5 We are less motivated by delayed rewards than by immediate rewards, and the more impulsive you are, the more your motivation is affected by such delays. Expectancy, value, delay, and impulsiveness are the four major components of procrastination. Piers Steel, a leading researcher on procrastination, explains: Decrease the certainty or the size of a task's reward - its expectancy or its value - and you are unlikely to pursue its compl...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Heads I Win, Tails?—Never Heard of Her; Or, Selective Reporting and the Tragedy of the Green Rationalists, published by Zack_M_Davis on the LessWrong. Followup to: What Evidence Filtered Evidence? In "What Evidence Filtered Evidence?", we are asked to consider a scenario involving a coin that is either biased to land Heads 2/3rds of the time, or Tails 2/3rds of the time. Observing Heads is 1 bit of evidence for the coin being Heads-biased (because the Heads-biased coin lands Heads with probability 2/3, the Tails-biased coin does so with probability 1/3, the likelihood ratio of these is 2 3 1 3 2 , and log 2 2 1 ), and analogously and respectively for Tails. If such a coin is flipped ten times by someone who doesn't make literally false statements, who then reports that the 4th, 6th, and 9th flips came up Heads, then the update to our beliefs about the coin depends on what algorithm the not-lying[1] reporter used to decide to report those flips in particular. If they always report the 4th, 6th, and 9th flips independently of the flip outcomes—if there's no evidential entanglement between the flip outcomes and the choice of which flips get reported—then reported flip-outcomes can be treated the same as flips you observed yourself: three Headses is 3 1 = 3 bits of evidence in favor of the hypothesis that the coin is Heads-biased. (So if we were initially 50:50 on the question of which way the coin is biased, our posterior odds after collecting 3 bits of evidence for a Heads-biased coin would be 2 3 1 = 8:1, or a probability of 8/(1 + 8) ≈ 0.89 that the coin is Heads-biased.) On the other hand, if the reporter mentions only and exactly the flips that came out Heads, then we can infer that the other 7 flips came out Tails (if they didn't, the reporter would have mentioned them), giving us posterior odds of 2 3 2 7 = 1:16, or a probability of around 0.06 that the coin is Heads-biased. So far, so standard. (You did read the Sequences, right??) What I'd like to emphasize about this scenario today, however, is that while a Bayesian reasoner who knows the non-lying reporter's algorithm of what flips to report will never be misled by the selective reporting of flips, a Bayesian with mistaken beliefs about the reporter's decision algorithm can be misled quite badly: compare the 0.89 and 0.06 probabilities we just derived given the same reported outcomes, but different assumptions about the reporting algorithm. If the coin gets flipped a sufficiently large number of times, a reporter whom you trust to be impartial (but isn't), can make you believe anything she wants without ever telling a single lie, just with appropriate selective reporting. Imagine a very biased coin that comes up Heads 99% of the time. If it gets flipped ten thousand times, 100 of those flips will be Tails (in expectation), giving a selective reporter plenty of examples to point to if she wants to convince you that the coin is extremely Tails-biased. Toy models about biased coins are instructive for constructing examples with explicitly calculable probabilities, but the same structure applies to any real-world situation where you're receiving evidence from other agents, and you have uncertainty about what algorithm is being used to determine what reports get to you. Reality is like the coin's bias; evidence and arguments are like the outcome of a particular flip. Wrong theories will still have some valid arguments and evidence supporting them (as even a very Heads-biased coin will come up Tails sometimes), but theories that are less wrong will have more. If selective reporting is mostly due to the idiosyncratic bad intent of rare malicious actors, then you might hope for safety in (the law of large) numbers: if Helga in particular is systematically more likely to report Headses than Tailses that she sees, then...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: CFAR Participant Handbook now available to all, published by Duncan_Sabien on the LessWrong. Google Drive PDF Hey, guys—I wrote this, and CFAR has recently decided to make it publicly available. Much of it involved rewriting the original work of others, such as Anna Salamon, Kenzie Ashkie, Val Smith, Dan Keys, and other influential CFAR founders and staff, but the actual content was filtered through me as single author as part of getting everything into a consistent and coherent shape. I have mild intentions to update it in the future with a handful of other new chapters that were on the list, but which didn't get written before CFAR let me go. Note that such updates will likely not be current-CFAR-approved, but will still derive directly from my understanding of the curriculum as former Curriculum Director. Thanks for listening. to help us out with the nonlinear library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How I Ended Up Non-Ambitious, published by Swimmer963 on the LessWrong. I have a confession to make. My life hasn’t changed all that much since I started reading Less Wrong. Hindsight bias makes it hard to tell, I guess, but I feel like pretty much the same person, or at least the person I would have evolved towards anyway, whether or not I spent those years reading about the Art of rationality. But I can’t claim to be upset about it either. I can’t say that rationality has undershot my expectations. I didn’t come to Less Wrong expecting, or even wanting, to become the next Bill Gates; I came because I enjoyed reading it, just like I’ve enjoyed reading hundreds of books and websites. In fact, I can’t claim that I would want my life to be any different. I have goals and I’m meeting them: my grades are good, my social skills are slowly but steadily improving, I get along well with my family, my friends, and my boyfriend. I’m in good shape financially despite making $12 an hour as a lifeguard, and in a year and a half I’ll be making over $50,000 a year as a registered nurse. I write stories, I sing in church, I teach kids how to swim. Compared to many people my age, I'm pretty successful. In general, I’m pretty happy. Yvain suggested akrasia as a major limiting factor for why rationalists fail to have extraordinarily successful lives. Maybe that’s true for some people; maybe they are some readers and posters on LW who have big, exciting, challenging goals that they consistently fail to reach because they lack motivation and procrastinate. But that isn’t true for me. Though I can’t claim to be totally free of akrasia, it hasn’t gotten much in the way of my goals. However, there are some assumptions that go too deep to be accessed by introspection, or even by LW meetup discussions. Sometimes you don't even realize they’re assumptions until you meet someone who assumes the opposite, and try to figure out why they make you so defensive. At the community meetup I described in my last post, a number of people asked me why I wasn’t studying physics, since I was obviously passionate about it. Trust me, I had plenty of good justifications for them–it’s a question I’ve been asked many times–but the question itself shouldn’t have made me feel attacked, and it did. Aside from people in my life, there are some posts on Less Wrong that cause the same reaction of defensiveness. Eliezer’s Mandatory Secret Identities is a good example; my automatic reaction was “well, why do you assume everyone here wants to have a super cool, interesting life? In fact, why do you assume everyone wants to be a rationality instructor? I don’t. I want to be a nurse.” After a bit of thought, I’ve concluded that there’s a simple reason why I’ve achieved all my life goals so far (and why learning about rationality failed to affect my achievements): they’re not hard goals. I’m not ambitious. As far as I can tell, not being ambitious is such a deep part of my identity that I never even noticed it, though I’ve used the underlying assumptions as arguments for why my goals and life decisions were the right ones. But if there’s one thing Less Wrong has taught me, it’s that assumptions are to be questioned. There are plenty of good reasons to choose reasonable goals instead of impossible ones, but doing things on reflex is rarely better than thinking through them, especially for long-term goal making, where I do have time to think it through, Type 2 style. What do I mean by ‘ambition’? Here is the definition from my desktop dictionary: (1) A strong desire to do or to achieve something, typically requiring determination and hard work: her ambition was to become a model | he achieved his ambition of making a fortune. (2) Desire and determination to achieve success: life offered few opportunities for young people with...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A tale from Communist China, published by Wei_Dai on the LessWrong. Write a Review Judging from the upvotes, it seems like people are quite interested in my grandparents' failure to emigrate from Communist China before it was too late, so I thought I'd elaborate here with more details and for greater visibility. They were all actually supporters of the Communist Party at the beginning, and saw it as potential saviors/liberators of the Chinese people and nation. They were also treated well at the beginning - one of them (being one of few people in China with a higher education) was given a high official post, and another retained a manager position at the factory that he used to own and operate. The latter side of the family considered moving away from China prior to the Communist victory since they would be classified as "capitalists" under the new regime, but were reassured by high level party members that they would be treated well if they stayed and helped to build the "new China". They actually did relatively ok, aside from most of their property being confiscated/nationalized early on and their living standards deteriorating steadily until they were forced to share the house that they used to own with something like 4 other families, and them being left with a single room in which to live. The other side were more straightforward "true believers" who supported Communism at an early stage, as they were part of the educated class who generally saw it as the right side of history, something that would help China leapfrog the West in terms of both social and economic progress. My grandmother on that side even tried to run away from her family to join the revolution after graduating from the equivalent of high school. Just before the Communists took power, my grandmother changed her mind, and wanted to move away from China and even got the required visas. (I asked my father why, and he said "women's intuition" which I'm not sure is really accurate but he couldn't provide further details.) But my grandfather still believed in the cause so they stayed. After the Communist victory, there was still about a year before the borders were fully shut, but it seemed like things were moving in a good direction and disproved my grandmother's worries. My grandfather was given an important post and went around making important speeches and so on. Unfortunately he was not very good at playing politics, as his background was in physics (although plenty of natural politicians also fared quite badly during the various "movements"). His position started attracting envy from those who thought he didn't contribute enough to the revolution to earn it. He was demoted and moved from city to city as the Party assigned him to various jobs. Finally, some kind of political dispute near the start of the Cultural Revolution led to his opponents digging up an incident in his distant past, which was then used as an excuse to persecute him in various ways, including confining him in a makeshift prison for ten years. He died shortly after the Cultural Revolution ended and he was released, just before I was born. According to my father, it was from over-eating due to finally being released from the extreme deprivation of his confinement. BTW, I wasn't told any of this when I was still a kid living in China. My parents had of course grown quite disillusioned by Communism and the Communist Party by then, but probably didn't think it would be to my advantage to show any signs of resistance to the indoctrination and propaganda that I was being fed in school and in the mass media. So I can also testify from personal experience that if those in charge of schools and the media want to, and there's enough social pressure to not resist, it's not very hard to brainwash a child. Thanks for listening. to help us ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What Money Cannot Buy, published by johnswentworth on the LessWrong. Paul Graham: The problem is, if you're not a hacker, you can't tell who the good hackers are. A similar problem explains why American cars are so ugly. I call it the design paradox. You might think that you could make your products beautiful just by hiring a great designer to design them. But if you yourself don't have good taste, how are you going to recognize a good designer? By definition you can't tell from his portfolio. And you can't go by the awards he's won or the jobs he's had, because in design, as in most fields, those tend to be driven by fashion and schmoozing, with actual ability a distant third. There's no way around it: you can't manage a process intended to produce beautiful things without knowing what beautiful is. American cars are ugly because American car companies are run by people with bad taste. I don’t know how much I believe this claim about cars, but I certainly believe it about software. A startup without a technical cofounder will usually produce bad software, because someone without software engineering skills does not know how to recognize such skills in someone else. The world is full of bad-to-mediocre “software engineers” who do not produce good software. If you don’t already know a fair bit about software engineering, you will not be able to distinguish them from the people who really know what they’re doing. Same with user interface design. I’ve worked with a CEO who was good at UI; both the process and the results were visibly superior to others I’ve worked with. But if you don’t already know what good UI design looks like, you’d have no idea - good design is largely invisible. Yudkowsky makes the case that the same applies to security: you can’t build a secure product with novel requirements without having a security expert as a founder. The world is full of “security experts” who do not, in fact, produce secure systems - I’ve met such people. (I believe they mostly make money by helping companies visibly pretend to have made a real effort at security, which is useful in the event of a lawsuit.) If you don’t already know a fair bit about security, you will not be able to distinguish such people from the people who really know what they’re doing. But to really drive home the point, we need to go back to 1774. As the American Revolution was heating up, a wave of smallpox was raging on the other side of the Atlantic. An English dairy farmer named Benjamin Jesty was concerned for his wife and children. He was not concerned for himself, though - he had previously contracted cowpox. Cowpox was contracted by milking infected cows, and was well known among dairy farmers to convey immunity against smallpox. Unfortunately, neither Jesty’s wife nor his two children had any such advantage. When smallpox began to pop up in Dorset, Jesty decided to take drastic action. He took his family to a nearby farm with a cowpox-infected cow, scratched their arms, and wiped pus from the infected cow on the scratches. Over the next few days, their arms grew somewhat inflamed and they suffered the mild symptoms of cowpox - but it quickly passed. As the wave of smallpox passed through the town, none of the three were infected. Throughout the rest of their lives, through multiple waves of smallpox, they were immune. The same technique would be popularized twenty years later by Edward Jenner, marking the first vaccine and the beginning of modern medicine. The same wave of smallpox which ran across England in 1774 also made its way across Europe. In May, it reached Louis XV, King of France. Despite the wealth of a major government and the talents of Europe’s most respected doctors, Louis XV died of smallpox on May 10, 1774. The point: there is knowledge for which money cannot substitute. Even...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why Our Kind Can't Cooperate, published by Eliezer Yudkowskyon the LessWrong. From when I was still forced to attend, I remember our synagogue's annual fundraising appeal. It was a simple enough format, if I recall correctly. The rabbi and the treasurer talked about the shul's expenses and how vital this annual fundraise was, and then the synagogue's members called out their pledges from their seats. Straightforward, yes? Let me tell you about a different annual fundraising appeal. One that I ran, in fact; during the early years of a nonprofit organization that may not be named. One difference was that the appeal was conducted over the Internet. And another difference was that the audience was largely drawn from the atheist/libertarian/technophile/sf-fan/early-adopter/programmer/etc crowd. (To point in the rough direction of an empirical cluster in personspace. If you understood the phrase "empirical cluster in personspace" then you know who I'm talking about.) I crafted the fundraising appeal with care. By my nature I'm too proud to ask other people for help; but I've gotten over around 60% of that reluctance over the years. The nonprofit needed money and was growing too slowly, so I put some force and poetry into that year's annual appeal. I sent it out to several mailing lists that covered most of our potential support base. And almost immediately, people started posting to the mailing lists about why they weren't going to donate. Some of them raised basic questions about the nonprofit's philosophy and mission. Others talked about their brilliant ideas for all the other sources that the nonprofit could get funding from, instead of them. (They didn't volunteer to contact any of those sources themselves, they just had ideas for how we could do it.) Now you might say, "Well, maybe your mission and philosophy did have basic problems—you wouldn't want to censor that discussion, would you?" Hold on to that thought. Because people were donating. We started getting donations right away, via Paypal. We even got congratulatory notes saying how the appeal had finally gotten them to start moving. A donation of $111.11 was accompanied by a message saying, "I decided to give a little bit more. One more hundred, one more ten, one more single, one more dime, and one more penny. All may not be for one, but this one is trying to be for all." But none of those donors posted their agreement to the mailing list. Not one. So far as any of those donors knew, they were alone. And when they tuned in the next day, they discovered not thanks, but arguments for why they shouldn't have donated. The criticisms, the justifications for not donating—only those were displayed proudly in the open. As though the treasurer had finished his annual appeal, and everyone not making a pledge had proudly stood up to call out justifications for refusing; while those making pledges whispered them quietly, so that no one could hear. I know someone with a rationalist cause who goes around plaintively asking, "How come the Raelian flying-saucer cult can get tens of thousands of members [probably around 40,000] interested in complete nonsense, but we can't even get a thousand people working on this?" The obvious wrong way to finish this thought is to say, "Let's do what the Raelians do! Let's add some nonsense to this meme!" For the benefit of those not immediately stopped by their ethical inhibitions, I will observe that there may be a hundred failed flying-saucer cults for every one that becomes famous. And the Dark Side may require non-obvious skills, which you, yes you, do not have: Not everyone can be a Sith Lord. In particular, if you talk about your planned lies on the public Internet, you fail. I'm no master criminal, but even I can tell certain people are not cut out to be crooks. So it's probably not a...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 9/26 is Petrov Day, published by Eliezer Yudkowsky on the LessWrong. Today is September 26th, Petrov Day, celebrated to honor the deed of Stanislav Yevgrafovich Petrov on September 26th, 1983. Wherever you are, whatever you're doing, take a minute to not destroy the world. The story begins on September 1st, 1983, when Soviet jet interceptors shot down a Korean Air Lines civilian airliner after the aircraft crossed into Soviet airspace and then, for reasons still unknown, failed to respond to radio hails. 269 passengers and crew died, including US Congressman Lawrence McDonald. Ronald Reagan called it "barbarism", "inhuman brutality", "a crime against humanity that must never be forgotten". Note that this was already a very, very poor time for US/USSR relations. Andropov, the ailing Soviet leader, was half-convinced the US was planning a first strike. The KGB sent a flash message to its operatives warning them to prepare for possible nuclear war. On September 26th, 1983, Lieutenant Colonel Stanislav Yevgrafovich Petrov was the officer on duty when the warning system reported a US missile launch. Petrov kept calm, suspecting a computer error. Then the system reported another US missile launch. And another, and another, and another. What had actually happened, investigators later determined, was sunlight on high-altitude clouds aligning with the satellite view on a US missile base. In the command post there were beeping signals, flashing lights, and officers screaming at people to remain calm. According to several accounts I've read, there was a large flashing screen from the automated computer system saying simply "START" (presumably in Russian). Afterward, when investigators asked Petrov why he hadn't written everything down in the logbook, Petrov replied,"Because I had a phone in one hand and the intercom in the other, and I don't have a third hand." The policy of the Soviet Union called for launch on warning. The Soviet Union's land radar could not detect missiles over the horizon, and waiting for positive identification would limit the response time to minutes. Petrov's report would be relayed to his military superiors, who would decide whether to start a nuclear war. Petrov decided that, all else being equal, he would prefer not to destroy the world. He sent messages declaring the launch detection a false alarm, based solely on his personal belief that the US did not seem likely to start an attack using only five missiles. Petrov was first congratulated, then extensively interrogated, then reprimanded for failing to follow procedure. He resigned in poor health from the military several months later. According to Wikipedia, he is spending his retirement in relative poverty in the town of Fryazino, on a pension of $200/month. In 2004, the Association of World Citizens gave Petrov a trophy and $1000. There is also a movie scheduled for release in 2008, entitled The Red Button and the Man Who Saved the World. Maybe someday, the names of people who decide not to start nuclear wars will be as well known as the name of Britney Spears. Looking forward to such a time, when humankind has grown a little wiser, let us celebrate, in this moment, Petrov Day. Thanks for listening. to help us out with the nonlinear library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How to Ignore Your Emotions (while also thinking you're awesome at emotions), published by Hazard on the LessWrong. (cross posted from my personal blog) Since middle school I've generally thought that I'm pretty good at dealing with my emotions, and a handful of close friends and family have made similar comments. Now I can see that though I was particularly good at never flipping out, I was decidedly not good "healthy emotional processing". I'll explain later what I think "healthy emotional processing" is, right now I'm using quotes to indicate "the thing that's good to do with emotions". Here it goes... Relevant context When I was a kid I adopted a strong, "Fix it or stop complaining about it" mentality. This applied to stress and worry as well. "Either address the problem you're worried about or quit worrying about it!" Also being a kid, I had a limited capacity to actually fix anything, and as such I was often exercising the "stop worrying about it" option. Another thing about me, I was a massive book worm and loved to collect "obvious mistakes" that heroes and villains would make. My theory was, "Know all the traps, and then just don't fall for them". That plus the sort of books I read meant that I "knew" it was a big no-no to ignore or repress your emotions. Luckily, since I knew you shouldn't repress your emotions, I "just didn't" and have lived happily ever after yeah nopes. Wiggling ears It can be really hard to teach someone to move in a way that is completely new to them. I teach parkour, and sometimes I want to say, Me: "Do the shock absorbing thing with your legs!" Student: "What's the shock absorbing thing?" Me: "... uh, you know... the thing were your legs... absorb shock?" It's hard to know how to give cues that will lead to someone making the right mental/muscle connection. Learning new motor movements is somewhat of a process of flailing around in the dark, until some feedback mechanism tells you you did it right (a coach, it's visually obvious, the jump doesn't hurt anymore, etc). Wiggling your ears is a nice concrete version of a) movement most people's bodies are capable of and b) one that most people feel like is impossible. Claim: learning mental and emotional skills has a similar "flailing around in the dark" aspect. There are the mental and emotional controls you've practiced, and those just feel like moving your arm. Natural, effortless, atomic. But there are other moves, which you are totally capable of which seem impossible because you don't know how your "control panel" connects to that output. This feels like trying to wiggle your ears. Why "ignore" and "deal with" looked the same So young me is upset that the grub master for our camping trip forgot half the food on the menu, and all we have for breakfast is milk. I couldn't "fix it" given that we were in the woods, so my next option was "stop feeling upset about it." So I reached around in the dark of my mind, and Oops, the "healthily process feelings" lever is right next to the "stop listening to my emotions" lever. The end result? "Wow, I decided to stop feeling upset, and then I stopped feeling upset. I'm so fucking good at emotional regulation!!!!!" My model now is that I substituted "is there a monologue of upsetness in my conscious mental loop?" for "am I feeling upset?". So from my perspective, it just felt like I was very in control of my feelings. Whenever I wanted to stop feeling something, I could. When I thought of ignoring/repressing emotions, I imagined trying to cover up something that was there, maybe with a story. Or I thought if you poked around ignored emotions there would be a response of anger or annoyance. I at least expected that if I was ignoring my emotions, that if I got very calm and then asked myself, "Is there anything that you're feeling?" I would get an an...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: When Money Is Abundant, Knowledge Is The Real Wealth , published by johnswentworth on the LessWrong. First Puzzle Piece By and large, the President of the United States can order people to do things, and they will do those things. POTUS is often considered the most powerful person in the world. And yet, the president cannot order a virus to stop replicating. The president cannot order GDP to increase. The president cannot order world peace. Are there orders the president could give which would result in world peace, or increasing GDP, or the end of a virus? Probably, yes. Any of these could likely even be done with relatively little opportunity cost. Yet no president in history has known which orders will efficiently achieve these objectives. There are probably some people in the world who know which orders would efficiently increase GDP, but the president cannot distinguish them from the millions of people who claim to know (and may even believe it themselves) but are wrong. Last I heard, Jeff Bezos was the official richest man in the world. He can buy basically anything money can buy. But he can’t buy a cure for cancer. Is there some way he could spend a billion dollars to cure cancer in five years? Probably, yes. But Jeff Bezos does not know how to do that. Even if someone somewhere in the world does know how to turn a billion dollars into a cancer cure in five years, Jeff Bezos cannot distinguish that person from the thousands of other people who claim to know (and may even believe it themselves) but are wrong. When non-experts cannot distinguish true expertise from noise, money cannot buy expertise. Knowledge cannot be outsourced; we must understand things ourselves. Second Puzzle Piece The Haber process combines one molecule of nitrogen with three molecules of hydrogen to produce two molecules of ammonia - useful for fertilizer, explosives, etc. If I feed a few grams of hydrogen and several tons of nitrogen into the Haber process, I’ll get out a few grams of ammonia. No matter how much more nitrogen I pile in - a thousand tons, a million tons, whatever - I will not get more than a few grams of ammonia. If the reaction is limited by the amount of hydrogen, then throwing more nitrogen at it will not make much difference. In the language of constraints and slackness: ammonia production is constrained by hydrogen, and by nitrogen. When nitrogen is abundant, the nitrogen constraint is slack; adding more nitrogen won’t make much difference. Conversely, since hydrogen is scarce, the hydrogen constraint is taut; adding more hydrogen will make a difference. Hydrogen is the bottleneck. Likewise in economic production: if a medieval book-maker requires 12 sheep skins and 30 days’ work from a transcriptionist to produce a book, and the book-maker has thousands of transcriptionist-hours available but only 12 sheep, then he can only make one book. Throwing more transcriptionists at the book-maker will not increase the number of books produced; sheep are the bottleneck. When some inputs become more or less abundant, bottlenecks change. If our book-maker suddenly acquires tens of thousands of sheep skins, then transcriptionists may become the bottleneck to book-production. In general, when one resource becomes abundant, other resources become bottlenecks. Putting The Pieces Together If I don’t know how to efficiently turn power into a GDP increase, or money into a cure for cancer, then throwing more power/money at the problem will not make much difference. King Louis XV of France was one of the richest and most powerful people in the world. He died of smallpox in 1774, the same year that a dairy farmer successfully immunized his wife and children with cowpox. All that money and power could not buy the knowledge of a dairy farmer - the knowledge that cowpox could safely immuniz...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Credibility of the CDC on SARS-CoV-2, published y Elizabeth, jimrandomh on the LessWrong. Introduction One of the main places Americans look for information on coronavirus is the Center for Disease Control and Prevention (abbreviated CDC from the days before “and Prevention” was in the title). That’s natural; “handling contagious epidemics” is not their only job, but it is one of their primary ones, and they position themselves as the authority. At a time when so many things are uncertain, it saves a lot of anxiety (and time, and money) to have an expert source you can turn to and get solid advice. Unfortunately, the CDC has repeatedly given advice with lots of evidence against it. Below is a list of actions from the CDC that we believe are misleading or otherwise indicative of an underlying problem. If you know of more examples or have information on any of these (for or against), please comment below and we will incorporate into this post. Examples Dismissed Risk of Infection Via Packages On the CDC’s coronavirus FAQs pages on 2020-03-04, they say, under “Am I at risk for COVID-19 from a package or products shipping from China?”: “In general, because of poor survivability of these coronaviruses on surfaces, there is likely very low risk of spread from products or packaging that are shipped over a period of days or weeks at ambient temperatures.” However, this metareview found that various coronaviruses remained infectious for days at room temperature on certain surfaces (cardboard was not tested, alas) and potentially weeks at lower temperatures. The CDC’s answer is probably correct for packages from China, and it’s possible it’s even right for domestic packages with 2-day shipping, but it is incorrect to say that coronaviruses in general have low survivability, and to the best of my ability to determine, we don’t have the experiments that would prove deliveries are safe. Blinded Itself to Community Spread As late as 2020-02-29, the CDC was reporting that there had been no “community spread” of SARS-CoV-2. (Community spread means that the person hadn’t been traveling in an infected area or associating with someone who had). At this time, the CDC would only test a person for SARS-CoV-2 if they had been in China or in close contact with a confirmed COVID-19 case. Testing Criteria as of 2020-02-11 This not only left them incapable of detecting community spread, it ignored potential cases who had travelled to other countries with known COVID-19 outbreaks. By 2020-02-13, this had been amended to include The criteria are intended to serve as guidance for evaluation. Patients should be evaluated and discussed with public health departments on a case-by-case basis. For severely ill individuals, testing can be considered when exposure history is equivocal (e.g., uncertain travel or exposure, or no known exposure) and another etiology has not been identified. (The CDC describes this change as happening on 2020-02-12, however the Wayback Machine did not capture the page that day). Based on this announcement on 2020-02-14, when testing that could detect community exposure was happening it was in one of 5 major cities. However as of 2020-03-01 only 472 tests had been done, so no test could have been happening very often. Between 2020-02-27 and 2020-02-28, the primary guidelines on this page were amended to However guidance went out on the same day (the 28th) that only listed China as a risk (and even then, only medium risk unless they had been exposed to a confirmed case or travelled to Hubei specifically). Testing Kits the CDC Sent to Local Labs were Unreliable They generated too many false positives to be useful. Hamstrung Detection by Banning 3rd Party Testing (HHS/FDA, not CDC) One reason the CDC used such stringent criteria for determining who to test was that they had a v...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Core Pathways of Aging, published by johnswentworth on the LessWrong. Most overviews of aging suffer from multiple problems: They dump a bunch of findings with no high-level picture. Many of the claims they make are outdated, purely theoretical, and sometimes even outright disproven by existing work. They are usually written by working academics, who are shy about telling us when their peers’ work is completely wrong. They are shy about making strong claims, since this would also implicitly mean denying some claims of the authors’ peers. This post is a high-level brain-dump of my current best models of the core pathways of aging, as I currently understand them. I have no particular reason to avoid calling out claims I think are wrong/irrelevant, and I’m going to present high-level models without pages and pages of disclaimers and discussions about results which maybe disagree with them (but are probably just wrong/irrelevant). Epistemic status: I would be surprised if none of it turned out to be wrong, but there are multiple lines of evidence supporting most claims. It is not highly polished, and references are included only when I have them readily on hand. My ideal version of this piece would have more detailed references, more double-checking behind the claims, and more direct presentation of the data which backs up each claim. Unfortunately, that would take enough time and effort that I’m unlikely to actually get to it soon. So. here’s what I could produce in a reasonable amount of time. Hopefully it will be wrong/unhelpful in ways orthogonal to how most overviews are wrong/unhelpful. Foundations First, let’s recap a couple foundational principles. I’ll go through these pretty quickly; see the linked posts for more info. Homeostasis and “Root Causes” in Aging: the vast majority of proteins, cells, etc, in the human body turn over on a timescale from days to months. At any given time, their level (e.g. protein concentration, cell count, etc) is in equilibrium on the turnover timescale - i.e. the rate of creation approximately equals the rate of removal. For any X with turnover much faster than aging (i.e. decades), if we see the level of X increase/decrease on the timescale of a human lifetime, then that is not due to permanent “accumulation of X” or “depletion of X”; it is due to increase/decrease in the rate of creation/removal of X. For instance: DNA damage is typically repaired on a timescale of hours or faster, depending on the type. If DNA damage levels increase with age, that is due to an increase in rate of damage or decrease in rate of repair, not permanent accumulation. Typical senescent cells turn over on a timescale of days to weeks. If the number of senescent cells increases with age, that is due to an increase in rate of senescent cell production or decrease in rate of removal, not permanent accumulation. Elastin is believed to not turn over at all in humans. So if we see elastin deposits increasing with age (e.g. in wrinkles), then that could be permanent accumulation. Furthermore: suppose we have a positive feedback cycle. Increasing A decreases the rate of production of B, so B decreases. But decreasing B decreases the rate of removal of A, so A increases. If both A and B individually turn over on a timescale of hours or faster then this feedback loop as a whole will also typically operate on a timescale of hours or faster - i.e. count/concentration of A will explode upward on roughly that timescale. More generally, a feedback loop will usually operate on the timescale of its slowest component, exactly like the rate-limiting step of a chemical reaction. Main upshot of all this: since aging involves changes on a timescale of decades, there must be some component which is out-of-equilibrium on a timescale of decades or longer (i.e. does not turn ove...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Religion's Claim to be Non-Disprovable, published by Eliezer Yudkowsky on the LessWrong. The earliest account I know of a scientific experiment is, ironically, the story of Elijah and the priests of Baal. The people of Israel are wavering between Jehovah and Baal, so Elijah announces that he will conduct an experiment to settle it—quite a novel concept in those days! The priests of Baal will place their bull on an altar, and Elijah will place Jehovah’s bull on an altar, but neither will be allowed to start the fire; whichever God is real will call down fire on His sacrifice. The priests of Baal serve as control group for Elijah—the same wooden fuel, the same bull, and the same priests making invocations, but to a false god. Then Elijah pours water on his altar—ruining the experimental symmetry, but this was back in the early days—to signify deliberate acceptance of the burden of proof, like needing a 0.05 significance level. The fire comes down on Elijah’s altar, which is the experimental observation. The watching people of Israel shout “The Lord is God!”—peer review. And then the people haul the 450 priests of Baal down to the river Kishon and slit their throats. This is stern, but necessary. You must firmly discard the falsified hypothesis, and do so swiftly, before it can generate excuses to protect itself. If the priests of Baal are allowed to survive, they will start babbling about how religion is a separate magisterium which can be neither proven nor disproven. Back in the old days, people actually believed their religions instead of just believing in them. The biblical archaeologists who went in search of Noah’s Ark did not think they were wasting their time; they anticipated they might become famous. Only after failing to find confirming evidence—and finding disconfirming evidence in its place—did religionists execute what William Bartley called the retreat to commitment, “I believe because I believe.” Back in the old days, there was no concept of religion’s being a separate magisterium. The Old Testament is a stream-of-consciousness culture dump: history, law, moral parables, and yes, models of how the universe works—like the universe being created in six days (which is a metaphor for the Big Bang), or rabbits chewing their cud. (Which is a metaphor for . . .) Back in the old days, saying the local religion “could not be proven” would have gotten you burned at the stake. One of the core beliefs of Orthodox Judaism is that God appeared at Mount Sinai and said in a thundering voice, “Yeah, it’s all true.” From a Bayesian perspective that’s some darned unambiguous evidence of a superhumanly powerful entity. (Although it doesn’t prove that the entity is God per se, or that the entity is benevolent—it could be alien teenagers.) The vast majority of religions in human history—excepting only those invented extremely recently—tell stories of events that would constitute completely unmistakable evidence if they’d actually happened. The orthogonality of religion and factual questions is a recent and strictly Western concept. The people who wrote the original scriptures didn’t even know the difference. The Roman Empire inherited philosophy from the ancient Greeks; imposed law and order within its provinces; kept bureaucratic records; and enforced religious tolerance. The New Testament, created during the time of the Roman Empire, bears some traces of modernity as a result. You couldn’t invent a story about God completely obliterating the city of Rome (a la Sodom and Gomorrah), because the Roman historians would call you on it, and you couldn’t just stone them. In contrast, the people who invented the Old Testament stories could make up pretty much anything they liked. Early Egyptologists were genuinely shocked to find no trace whatsoever of Hebrew tribes having ever be...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Coronavirus: Justified Practical Advice Thread, published by Ben Pace, Elizabeth on the LessWrong. (Added: To see the best advice in this thread, read this summary.) This is a thread for practical advice for preparing for the coronavirus in places where it might substantially grow. We'd like this thread to be a source of advice that attempts to explain itself. This is not a thread to drop links to recommendations that don’t explain why the advice is accurate or useful. That’s not to say that explanation-less advice isn’t useful, but this isn't the place for it. Please include in your answers some advice and an explanation of the advice, an explicit model under which it makes sense. We will move answers to the comments if they don't explain their recommendations clearly. (Added: We have moved at least 4 comments so far.) The more concrete the explanation the better. Speculation is fine, uncertain models are fine; sources, explicit models and numbers for variables that other people can play with based on their own beliefs are excellent. Here are some examples of things that we'd like to see: It is safe to mostly but not entirely rely on food that requires heating or other prep, because a pandemic is unlikely to take out utilities, although if if they are taken out for other reasons they will be slower to come back on CDC estimates of prevalence are likely to be significant underestimates due to their narrow testing criteria. A guesstimate model of the risks of accepting packages and delivery food One piece of information that has been lacking in most advice we’ve seen is when to take a particular action. Sure, I can stock up on food ahead of time, but not going to work may be costly– what’s your model for the costs of going so I can decide when the costs outweigh the benefits for me? This is especially true for advice that has inherent trade-offs– total quarantine means eating your food stockpiles that you hopefully have, which means not having them later. Thanks for listening. to help us out with the nonlinear library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Intentionally Making Close Friends, published by Neel Nanda on the LessWrong. This is a linkpost for Introduction One of the greatest sources of joy in my life are my close friends. People who bring excitement and novelty into my life. Who expose me to new experiences, and ways of seeing the world. Who help me learn, point out my blind spots, and correct me when I am wrong. Who I can lean on when I need support, and who lean on me in turn. Friends who help me grow more into the kind of person I want to be. I am especially grateful for this, because up until about 4 years ago, I didn’t have any close friends in my life. I had friends, but struggled to form real emotional connections. Moreover, it didn’t even occur to me that I could try to do this. It wasn’t that I knew how to form close friends but was too anxious to try, rather, ‘try to form close friendships’ was a non-standard action, something that never even crossed my mind. And one of my most life-changing experiments was realising that this was something I wanted, and actually trying to intentionally form close friends. It’s easy to slip into a passive mindset here, to think of emotional connections as ‘something that take time’ or ‘need to happen naturally’. That to be intentional about things is ‘inauthentic’. I think this mindset is absolutely crazy. My close friendships are one of the most important components of my life happiness. Leaving it up to chance feels like passing up an incredible opportunity. As with all important things in life, this can be optimised - further, if done right, this adds a massive amount to the lives of me and of my future close friends. The first half of this post is the story of how I approached intentionally forming close friends, and the second half is an attempt to distill the lessons I learned from this. As such, this post is more autobiographical than most. Feel free to skip to the advice section if you don’t want that. Further, what you value in close friendships is highly personal - this post will focus on what I want in friendships and how I try to get it, but you should adapt this to your own situation, values, and what feels missing in your life! Exercise: Think about your closest friends, and how these friendships happened. What needs are you fulfilling in each other’s lives? Are you happy with this state of affairs, or is something missing? What could be better? My story The Problem Back when I was in school, I never had close friends. I had friends, people I liked, people I spent time with, whose company I genuinely enjoyed. But I was pretty terrible at being vulnerable and forming emotional connections. These friendships rarely went beyond the surface level. In hindsight, I expect these could have been far richer (and I’ve formed much stronger friendships with some of these friends since!), but I never really tried. I find it hard to introspect on exactly what the internal experience of past Neel was like, but I think the core was that trying wasn’t available as a possible action. That I spent much of my life doing what felt socially conventional, normal and expected, for the role I saw myself in. And ‘go out of your way to form emotional connections’ wasn’t part of that. It wasn’t an action I considered, weighed up the costs and benefits, and decided against - it never even occurred to me to try. It didn’t feel like a void missing from my life - things just felt normal. It was like playing a video game, and having a list of actions to choose from, like ‘ask about their day’, ‘complain about a shared experience’ or ‘discuss something cool I learned recently’; but this list contained nothing about ‘intentionally form an emotional connection’. It wasn’t in my reference class of things I could do. One of the core parts of my life philosophy now is the skill of agency...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Tsuyoku Naritai! (I Want To Become Stronger), published by Eliezer Yudkowsky on the LessWrong. In Orthodox Judaism there is a saying: “The previous generation is to the next one as angels are to men; the next generation is to the previous one as donkeys are to men.” This follows from the Orthodox Jewish belief that all Judaic law was given to Moses by God at Mount Sinai. After all, it’s not as if you could do an experiment to gain new halachic knowledge; the only way you can know is if someone tells you (who heard it from someone else, who heard it from God). Since there is no new source of information; it can only be degraded in transmission from generation to generation. Thus, modern rabbis are not allowed to overrule ancient rabbis. Crawly things are ordinarily unkosher, but it is permissible to eat a worm found in an apple—the ancient rabbis believed the worm was spontaneously generated inside the apple, and therefore was part of the apple. A modern rabbi cannot say, “Yeah, well, the ancient rabbis knew diddly-squat about biology. Overruled!” A modern rabbi cannot possibly know a halachic principle the ancient rabbis did not, because how could the ancient rabbis have passed down the answer from Mount Sinai to him? Knowledge derives from authority, and therefore is only ever lost, not gained, as time passes. When I was first exposed to the angels-and-donkeys proverb in (religious) elementary school, I was not old enough to be a full-blown atheist, but I still thought to myself: “Torah loses knowledge in every generation. Science gains knowledge with every generation. No matter where they started out, sooner or later science must surpass Torah.” The most important thing is that there should be progress. So long as you keep moving forward you will reach your destination; but if you stop moving you will never reach it. Tsuyoku naritai is Japanese. Tsuyoku is “strong”; naru is “becoming,” and the form naritai is “want to become.” Together it means, “I want to become stronger,” and it expresses a sentiment embodied more intensely in Japanese works than in any Western literature I’ve read. You might say it when expressing your determination to become a professional Go player—or after you lose an important match, but you haven’t given up—or after you win an important match, but you’re not a ninth-dan player yet—or after you’ve become the greatest Go player of all time, but you still think you can do better. That is tsuyoku naritai, the will to transcendence. Each year on Yom Kippur, an Orthodox Jew recites a litany which begins Ashamnu, bagadnu, gazalnu, dibarnu dofi, and goes on through the entire Hebrew alphabet: We have acted shamefully, we have betrayed, we have stolen, we have slandered . . . As you pronounce each word, you strike yourself over the heart in penitence. There’s no exemption whereby, if you manage to go without stealing all year long, you can skip the word gazalnu and strike yourself one less time. That would violate the community spirit of Yom Kippur, which is about confessing sins—not avoiding sins so that you have less to confess. By the same token, the Ashamnu does not end, “But that was this year, and next year I will do better.” The Ashamnu bears a remarkable resemblance to the notion that the way of rationality is to beat your fist against your heart and say, “We are all biased, we are all irrational, we are not fully informed, we are overconfident, we are poorly calibrated . . .” Fine. Now tell me how you plan to become less biased, less irrational, more informed, less overconfident, better calibrated. There is an old Jewish joke: During Yom Kippur, the rabbi is seized by a sudden wave of guilt, and prostrates himself and cries, “God, I am nothing before you!” The cantor is likewise seized by guilt, and cries, “God, I am nothing before you!” Se...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Lifestyle interventions to increase longevity, published by RomeoStevens on the LessWrong. There is a lot of bad science and controversy in the realm of how to have a healthy lifestyle. Every week we are bombarded with new studies conflicting older studies telling us X is good or Y is bad. Eventually we reach our psychological limit, throw up our hands, and give up. I used to do this a lot. I knew exercise was good, I knew flossing was good, and I wanted to eat better. But I never acted on any of that knowledge. I would feel guilty when I thought about this stuff and go back to what I was doing. Unsurprisingly, this didn't really cause me to make any positive lifestyle changes. Instead of vaguely guilt-tripping you with potentially unreliable science news, this post aims to provide an overview of lifestyle interventions that have very strong evidence behind them and concrete ways to implement them. A quick FAQ before we get started Why should I care about longevity-promoting habits at a young age? First, many longevity-promoting lifestyle changes will increase your quality of life in the short term. In doing this research, I found a few interventions that had shockingly large impacts on my subjective day-to-day wellness. Second, the choices you make have larger downstream effects the earlier you get started. Trying to undo years of damage and ingrained habits at an advanced age really isn’t a position you want to find yourself in. Third, extending your life matters more the more you believe in the proximity of transformative tech. If the pace of technological improvement is increasing, then adding a decade to your life may in fact be the decade that counts. Missing out on life extension tech by a few years would really suck. Isn’t longevity mostly just genetics? That's what I believed for a long time, but a quick trip to wikipedia tells us that only 20-30% of the variance in longevity is heritable. What sort of benefits can I expect? The life satisfaction of people who remain independent and active actually increases significantly with age. Mental and physical performance are strongly correlated, meaning maintaining your body will help maintain your mind. The qualitative benefits for life satisfaction of many of these interventions can be so dramatic that it is hard to estimate them. The gulf in quality of life between people maintaining good habits and those who do not widens with age. How were these recommendations generated?/Why should I believe you? This post summarizes studies at the intersection of having large effects, large sample sizes, and being well-designed in terms of methodology. The cutoff for an intervention being “worth it” is somewhat subjective given that there is often only a rough estimate of the overall effect sizes of various interventions in comparison to one another. CDC mortality statistics were used to determine the most likely causes of death in various age brackets. The list of things that kill people balloons significantly as you get towards the less common causes of death and I have limited research time. Individuals who face unusual health circumstances should of course be doing their own research and consulting health professionals. This brings me to my disclaimer: This post is not intended to diagnose, treat, cure, or prevent any disease. No claim or opinion on these pages is intended to be, nor should be construed as medical advice. Please consult with a healthcare professional before starting any diet or exercise program. None of these claims have been evaluated by the Food and Drug Administration. Suggestions herein are intended for normal healthy adults and should not be used if you are under the age of 18 or have any known medical condition. Alright, let’s dive in. Things that will eventually kill you CVD At the top of our list ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Is Rationalist Self-Improvement Real?, published by Jacob Falkovich on the LessWrong. Cross-posted from Putanumonit. Basketballism Imagine that tomorrow everyone on the planet forgets the concept of training basketball skill. The next day everyone is as good at basketball as they were the previous day, but this talent is assumed to be fixed. No one expects their performance to change over time. No one teaches basketball, although many people continue to play the game for fun. Geneticists explain that some people are born with better hand-eye coordination and are able to shoot a basketball accurately. Economists explain that highly-paid NBA players have a stronger incentive to hit shots, which explains their improved performance. Psychologists note that people who take more jump shots each day hit a higher percentage and theorize a principal factor of basketball affinity that influences both desire and skill at basketball. Critical race theorists claim that white men’s under-representation in the NBA is due to systemic oppression. Papers are published, tenure is awarded. New scientific disciplines emerge and begin studying basketball more systematically. Evolutionary physiologists point out that our ancestors threw stones in a sidearm motion, which explains our lack of adaptation to the different motion of jump shots. Behavioral kinesiologists describe systematic biases in human basketball, such as the tendency to shoot balls with a flatter trajectory and a lower release point than is optimal. When asked by aspiring basketball players if jump shots can be improved, they all shake their heads and rue that it is human nature to miss shots. A Nobel laureate behavioral kinesiologist tells audiences that even after writing books on biases in basketball his shot did not improve much. Someone publishes a study showing that basketball performance improves after a one-hour training session with schoolchildren, but Shott Ballexander writes a critical takedown pointing out that the effect wore off after a month and could simply be random noise. The field switches to studying “nudges”: ways to design systems so that players hit more shots at the same level of skill. They recommend that the NBA adopt larger hoops. Papers are published, tenure is awarded. Then, one day, someone merely looking to get good at basketball, as opposed to getting tenure, comes across these papers. She realizes that the lessons of behavioral kinesiology can be used to improve her jump shot. She practices releasing the ball at the top of her jump from above the forehead with a steep arc. As her shots start swooshing in more people gather at the gym to practice with her. They call themselves Basketballists. Most people who walk past the gym sneer at the Basketballists. “You call yourselves Basketballists and yet none of you shoot 100%”, they taunt. “You should go to grad school if you want to learn about jump shots.” Some of Basketballists themselves begin to doubt the project, especially since switching to the new shooting techniques lowers their performance at first. “Did you hear what the Center for Applied Basketball is charging for a training camp?”, they mutter, “I bet their results are all due to selection bias.” The Basketballists insist that the training does help, that they really get better by the day. Their shots hit at a slightly higher rate than before, although this is swamped by the inter-individual variance. How could they know if it works? AsWrongAsEver A core axiom of Rationality (capitalized to refer to LessWrong version) is that it is a skill that can be improved with time and practice. The names Overcoming Bias and LessWrong reflect this: rationality is a direction, not a fixed point. What would it mean to "improve at Rationality"? On the epistemic side, to draw a map that more accurat...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is:A whirlwind tour of Ethereum finance, published by cata on the LessWrong. As a hacker and cryptocurrency liker, I have been hearing for a while about "DeFi" stuff going on in Ethereum without really knowing what it was. I own a bunch of ETH, so I finally decided that enough was enough and spent a few evenings figuring out what was going on. To my pleasant surprise, a lot of it was fascinating, and I thought I would share it with LW in the hopes that other people will be interested too and share their thoughts. Throughout this post I will assume that the reader has a basic mental model of how Ethereum works. If you don't, you might find this intro & reference useful. Why should I care about this? For one thing, it's the coolest, most cypherpunk thing going. Remember how back in 2012, everyone knew that Bitcoin existed, but it was a pain in the ass to use and it kind of felt weird and risky? It feels exactly like that using all this stuff. It's loads of fun. For another thing, the economic mechanism design stuff is really fun to think about, and in many cases nobody knows the right answer yet. It's a chance for random bystanders to hang out with problems on the edge of human understanding, because nobody cared about these problems before there was so much money floating around in them. For a third thing, you can maybe make some money. Specifically, if you have spare time, a fair bit of cash, appetite for risk, conscientiousness, some programming and finance knowledge, and you are capable of and interested in understanding how these systems work, I think it's safe to say that you have a huge edge, and you should be able to find places to extract value. General overview In broad strokes, people are trying to reinvent all of the stuff from typical regulated finance in trustless, decentralized ways (thus "DeFi".) That includes: Making anything that has value into a transferable asset, typically on Ethereum, and typically an ERC-20 token. A token is an interoperable currency that keeps track of people's balances and lets people transfer it. Making liquid exchanges where you can swap all of those tokens at market prices. Making schemes for moving those tokens over time, like borrowing, futures, etc. Making elaborate scams and arbitrages to obtain other people's tokens. It's not completely clear to me what the main value proposition of all of this is. It's easy to generate things about it that seem somewhat valuable, but hard to say how each stacks up. Some possible value includes: Evading regulation, like securities laws, money laundering laws, sanctions, capital controls, laws against online gambling, etc. etc. Allocation of capital among projects that can raise money using cryptocurrency tokens (because somehow they have a scheme to tie the success of their project to the value of the token, making it a kind of virtual equity.) Having less middlemen than existing financial systems, making it more trustworthy and cheaper. (It is not currently more trustworthy or cheaper than mainstream American institutions, but it plausibly could be in a few years.) Tokenization The first step is to make everything into an ERC-20 token. This will let all the other products work with everything, because they will interoperate with ERC-20 tokens. Stablecoins and pegs It's common for someone to want to own an Ethereum version of some other asset that is not Ethereum, so that they can use it on Ethereum. The most typical example of this is US dollars. A token whose price is designed to be pegged to an external thing like this is called a stablecoin. There are a few techniques people use to accomplish this. The most popular one is to have a giant pile of US dollars somewhere under someone's control, and have that person act as a counterparty for anyone who wants to buy or sell 1 US dollar for 1 ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Apologist and the Revolutionary, published by Scott Alexander on the LessWrong. Rationalists complain that most people are too willing to make excuses for their positions, and too unwilling to abandon those positions for ones that better fit the evidence. And most people really are pretty bad at this. But certain stroke victims called anosognosiacs are much, much worse. Anosognosia is the condition of not being aware of your own disabilities. To be clear, we're not talking minor disabilities here, the sort that only show up during a comprehensive clinical exam. We're talking paralysis or even blindness1. Things that should be pretty hard to miss. Take the example of the woman discussed in Lishman's Organic Psychiatry. After a right-hemisphere stroke, she lost movement in her left arm but continuously denied it. When the doctor asked her to move her arm, and she observed it not moving, she claimed that it wasn't actually her arm, it was her daughter's. Why was her daughter's arm attached to her shoulder? The patient claimed her daughter had been there in the bed with her all week. Why was her wedding ring on her daughter's hand? The patient said her daughter had borrowed it. Where was the patient's arm? The patient "turned her head and searched in a bemused way over her left shoulder". Why won't these patients admit they're paralyzed, and what are the implications for neurotypical humans? Dr. Vilayanur Ramachandran, leading neuroscientist and current holder of the world land-speed record for hypothesis generation, has a theory. One immediately plausible hypothesis: the patient is unable to cope psychologically with the possibility of being paralyzed, so he responds with denial. Plausible, but according to Dr. Ramachandran, wrong. He notes that patients with left-side strokes almost never suffer anosognosia, even though the left side controls the right half of the body in about the same way the right side controls the left half. There must be something special about the right hemisphere. Another plausible hypothesis: the part of the brain responsible for thinking about the affected area was damaged in the stroke. Therefore, the patient has lost access to the area, so to speak. Dr. Ramachandran doesn't like this idea either. The lack of right-sided anosognosia in left-hemisphere stroke victims argues against it as well. But how can we disconfirm it? Dr. Ramachandran performed an experiment2 where he "paralyzed" an anosognosiac's good right arm. He placed it in a clever system of mirrors that caused a research assistant's arm to look as if it was attached to the patient's shoulder. Ramachandran told the patient to move his own right arm, and the false arm didn't move. What happened? The patient claimed he could see the arm moving - a classic anosognosiac response. This suggests that the anosognosia is not specifically a deficit of the brain's left-arm monitoring system, but rather some sort of failure of rationality. Says Dr. Ramachandran: The reason anosognosia is so puzzling is that we have come to regard the 'intellect' as primarily propositional in character and one ordinarily expects propositional logic to be internally consistent. To listen to a patient deny ownership of her arm and yet, in the same breath, admit that it is attached to her shoulder is one of the most perplexing phenomena that one can encounter as a neurologist. So what's Dr. Ramachandran's solution? He posits two different reasoning modules located in the two different hemispheres. The left brain tries to fit the data to the theory to preserve a coherent internal narrative and prevent a person from jumping back and forth between conclusions upon each new data point. It is primarily an apologist, there to explain why any experience is exactly what its own theory would have predicted. The right b...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Dark Matters, published by Diffractor on the LessWrong. This post will be about the main points of evidence for the existence of dark matter. To evaluate whether a competing theory to dark matter is plausible, it's important to know what the actual arguments in favor of dark matter are in more detail than just "dark matter is the stuff you have to add to get galactic rotation curves to work out". A competitor has to address the strongest arguments in favor of the existence of dark matter, not just the weaker fare like galactic rotation curves. So, when reading some hot new arxiv paper about dark matter or the lack thereof, it is fairly useful to know the top five lines of evidential support for dark matter (in my own personal estimation, others may differ). This lets you at least check whether the result is directly addressing the major cruxes that the case for dark matter rests upon, or just picking off one particular piece of evidence and sweeping the rest under the rug, even if you lack the full technical ability to evaluate the claimed result. This post will be saving the best for last, so if you're not going to read the whole thing, skip down to sections 4 and 5. Also, what exactly is meant when the term "dark matter" is used in this post? Anything with mass (so it's affected by gravity and gravitationally influences other things) which does not interact via the electromagnetic force. Electrons, protons, nuclei, and atoms emphatically do not count. Black holes, neutrinos, WIMPS (weakly interacting massive particles), and axions would count under this definition. The last two are theoretical, the first two are very much established. Of course, it would be a massive cop-out to go "neutrinos exist, therefore dark matter does", so "dark matter" will be used with a followup connotation of "and whatever the heck is (we don't know yet), there must be 5x more of it in the universe than matter made of atoms or atom parts, no way around that whatsoever" Point 1: Galactic Rotation Curves The story begins with galaxy rotation curves, which were the original motivation for postulating dark matter in the first place. Given a point gravitational mass, it's pretty simple to calculate the velocity of something orbiting around it, depending only on how far away the object is orbiting and how much mass is in the central point. Stuff orbiting further out from a point mass will be orbiting at a lower velocity. With a bit more work, given a disc of mass, you can calculate the velocity of something orbiting around or within it. For this, the graph of orbital velocity vs distance from the center of the disc first rises, then falls. Orbital velocities are low in the center because stuff orbiting near the center of the disc isn't orbiting around very much mass, and orbital velocities are low at the outside of the disc, because you get closer to being able to approximate things by the situation "your distant object is orbiting around a central point mass", which, as previously discussed, already exhibits the "stars on further-out orbits move more slowly" behavior. Computing this in practice requires knowledge of two things, however. First, you need to know how fast the stars in the galaxy are orbiting around the center. Second, you need to know the radial distribution of mass in the disc or ellipse. It's pretty easy to tell how fast stars in a galaxy are orbiting around the center, for suitably chosen galaxies. Stars have emission and absorption lines at very specific frequencies measured to very high accuracy, which only depend on details of atomic physics that don't change in different galaxies. So, as an example, you could pick an edge-on spiral galaxy, and look at the position of the absorption lines in the center of the galaxy. Then, you can look at the two edges of the galaxy, and i...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Where do your eyes go?, published by alkjash on the LessWrong. This is a linkpost for/ [Shoutout to the LW team for very helpful (and free!) feedback on this post.] I. Prelude When my wife first started playing the wonderful action roguelike Hades, she got stuck in Asphodel. Most Hades levels involve dodging arrows, lobbed bombs, melee monsters, and spike traps all whilst hacking and slashing as quickly as possible, but Asphodel adds an extra twist: in this particular charming suburb of the Greek underworld, you need to handle all of the above whilst also trying not to step in lava. Most of the islands in Asphodel are narrower than your dash is far, so it’s hard not to dash straight off solid ground into piping-hot doom. I gave my wife some pointers about upgrade choices (cough Athena dash cough) and enemy attack patterns, but most of my advice was marginally helpful at best. She probably died in lava another half-dozen times. One quick trick, however, had an instant and visible effect. "Stare at yourself." Watch your step. By watching my wife play, I came to realize that she was making one fundamental mistake: her eyes were in the wrong place. Instead of watching her own character Zagreus, she spent most of her time staring at the enemies and trying to react to their movements and attacks. Hades is almost a bullet hell game: avoiding damage is the name of the game. Eighty percent of the time your eyes need to be honed on Zagreus's toned protagonist butt to make sure he dodges precisely away from, out of, or straight through enemy attacks. In the meantime, most of Zagreus's own attacks hit large areas, so tracking enemies with peripheral vision is enough to aim your attacks in the right general direction. Once my wife learned to fix her eyes on Zagreus, she made it through Asphodel in only a few attempts. This is a post about the general skill of focusing your eyes, and your attention, to the right place. Instead of the standard questions "How do you make good decisions based on what you see?" and "How do you get better at executing those decisions?", this post focuses on a question further upstream: "Where should your eyes be placed to receive the right information in the first place?" In Part II, I describe five archetypal video games, which are distinguished in my memory by the different answers to "Where do your eyes go?" I learned from each of them. I derive five general lessons about attention-paying. Part II can be safely skipped by those allergic to video games. In Part III, I apply these lessons to three specific minigames that folks struggle with in graduate school: research meetings, seminar talks, and paper-reading. In all three cases, there can be an overwhelming amount of information to attend to, and the name of the game is to focus your eyes properly to perceive the most valuable subset. II. Lessons from Video Games Me or You? Hades and Dark Souls are similar games in many respects. Both live in the same general genre of action RPGs, both share the core gameplay loop "kill, die, learn, repeat," and both are widely acknowledged to be among the best games of all time. Their visible differences are mostly aesthetic: for example, Hades' storytelling is more lighthearted, Dark Souls' more nonexistent. But there is one striking difference between my experiences of these two games: in Hades I stared at myself, and in Dark Souls I stared at the enemy. Why? One answer is obvious: in Dark Souls, the camera follows you around over your shoulder, so you're forced to stare at the enemies, while in Hades the isometric camera is centered on your own character. This is good game design because the camera itself gently suggests the right place for your eyes to focus, but it doesn't really explain why that place is right. The more interesting answer is that your eyes g...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Confidence levels inside and outside an argument , published by Scott Alexander on the LessWrong. Related to: Infinite Certainty Suppose the people at FiveThirtyEight have created a model to predict the results of an important election. After crunching poll data, area demographics, and all the usual things one crunches in such a situation, their model returns a greater than 999,999,999 in a billion chance that the incumbent wins the election. Suppose further that the results of this model are your only data and you know nothing else about the election. What is your confidence level that the incumbent wins the election? Mine would be significantly less than 999,999,999 in a billion. When an argument gives a probability of 999,999,999 in a billion for an event, then probably the majority of the probability of the event is no longer in "But that still leaves a one in a billion chance, right?". The majority of the probability is in "That argument is flawed". Even if you have no particular reason to believe the argument is flawed, the background chance of an argument being flawed is still greater than one in a billion. More than one in a billion times a political scientist writes a model, ey will get completely confused and write something with no relation to reality. More than one in a billion times a programmer writes a program to crunch political statistics, there will be a bug that completely invalidates the results. More than one in a billion times a staffer at a website publishes the results of a political calculation online, ey will accidentally switch which candidate goes with which chance of winning. So one must distinguish between levels of confidence internal and external to a specific model or argument. Here the model's internal level of confidence is 999,999,999/billion. But my external level of confidence should be lower, even if the model is my only evidence, by an amount proportional to my trust in the model. Is That Really True? One might be tempted to respond "But there's an equal chance that the false model is too high, versus that it is too low." Maybe there was a bug in the computer program, but it prevented it from giving the incumbent's real chances of 999,999,999,999 out of a trillion. The prior probability of a candidate winning an election is 50%1. We need information to push us away from this probability in either direction. To push significantly away from this probability, we need strong information. Any weakness in the information weakens its ability to push away from the prior. If there's a flaw in FiveThirtyEight's model, that takes us away from their probability of 999,999,999 in of a billion, and back closer to the prior probability of 50% We can confirm this with a quick sanity check. Suppose we know nothing about the election (ie we still think it's 50-50) until an insane person reports a hallucination that an angel has declared the incumbent to have a 999,999,999/billion chance. We would not be tempted to accept this figure on the grounds that it is equally likely to be too high as too low. A second objection covers situations such as a lottery. I would like to say the chance that Bob wins a lottery with one billion players is 1/1 billion. Do I have to adjust this upward to cover the possibility that my model for how lotteries work is somehow flawed? No. Even if I am misunderstanding the lottery, I have not departed from my prior. Here, new information really does have an equal chance of going against Bob as of going in his favor. For example, the lottery may be fixed (meaning my original model of how to determine lottery winners is fatally flawed), but there is no greater reason to believe it is fixed in favor of Bob than anyone else.2 Spotted in the Wild The recent Pascal's Mugging thread spawned a discussion of the Large Hadron Colli...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Fact Posts: How and Why, published by sarahconstantinon the LessWrong. The most useful thinking skill I've taught myself, which I think should be more widely practiced, is writing what I call "fact posts." I write a bunch of these on my blog. (I write fact posts about pregnancy and childbirth here.) To write a fact post, you start with an empirical question, or a general topic. Something like "How common are hate crimes?" or "Are epidurals really dangerous?" or "What causes manufacturing job loss?" It's okay if this is a topic you know very little about. This is an exercise in original seeing and showing your reasoning, not finding the official last word on a topic or doing the best analysis in the world. Then you open up a Google doc and start taking notes. You look for quantitative data from conventionally reliable sources. CDC data for incidences of diseases and other health risks in the US; WHO data for global health issues; Bureau of Labor Statistics data for US employment; and so on. Published scientific journal articles, especially from reputable journals and large randomized studies. You explicitly do not look for opinion, even expert opinion. You avoid news, and you're wary of think-tank white papers. You're looking for raw information. You are taking a sola scriptura approach, for better and for worse. And then you start letting the data show you things. You see things that are surprising or odd, and you note that. You see facts that seem to be inconsistent with each other, and you look into the data sources and methodology until you clear up the mystery. You orient towards the random, the unfamiliar, the things that are totally unfamiliar to your experience. One of the major exports of Germany is valves? When was the last time I even thought about valves? Why valves, what do you use valves in? OK, show me a list of all the different kinds of machine parts, by percent of total exports. And so, you dig in a little bit, to this part of the world that you hadn't looked at before. You cultivate the ability to spin up a lightweight sort of fannish obsessive curiosity when something seems like it might be a big deal. And you take casual notes and impressions (though keeping track of all the numbers and their sources in your notes). You do a little bit of arithmetic to compare things to familiar reference points. How does this source of risk compare to the risk of smoking or going horseback riding? How does the effect size of this drug compare to the effect size of psychotherapy? You don't really want to do statistics. You might take percents, means, standard deviations, maybe a Cohen's d here and there, but nothing fancy. You're just trying to figure out what's going on. It's often a good idea to rank things by raw scale. What is responsible for the bulk of deaths, the bulk of money moved, etc? What is big? Then pay attention more to things, and ask more questions about things, that are big. (Or disproportionately high-impact.) You may find that this process gives you contrarian beliefs, but often you won't, you'll just have a strongly fact-based assessment of why you believe the usual thing. There's a quality of ordinariness about fact-based beliefs. It's not that they're never surprising -- they often are. But if you do fact-checking frequently enough, you begin to have a sense of the world overall that stays in place, even as you discover new facts, instead of swinging wildly around at every new stimulus. For example, after doing lots and lots of reading of the biomedical literature, I have sort of a "sense of the world" of biomedical science -- what sorts of things I expect to see, and what sorts of things I don't. My "sense of the world" isn't that the world itself is boring -- I actually believe in a world rich in discoveries and low-hanging fruit -- but th...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Alignment Research Field Guide, published by abramdemski on the LessWrong. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. This field guide was written by the MIRI team with MIRIx groups in mind, though the advice may be relevant to others working on AI alignment research. ⠀ Preamble I: Decision Theory Hello! You may notice that you are reading a document. This fact comes with certain implications. For instance, why are you reading this? Will you finish it? What decisions will you come to as a result? What will you do next? Notice that, whatever you end up doing, it’s likely that there are dozens or even hundreds of other people, quite similar to you and in quite similar positions, who will follow reasoning which strongly resembles yours, and make choices which correspondingly match. Given that, it’s our recommendation that you make your next few decisions by asking the question “What policy, if followed by all agents similar to me, would result in the most good, and what does that policy suggest in my particular case?” It’s less of a question of trying to decide for all agents sufficiently-similar-to-you (which might cause you to make the wrong choice out of guilt or pressure) and more something like “if I were in charge of all agents in my reference class, how would I treat instances of that class with my specific characteristics?” If that kind of thinking leads you to read further, great. If it leads you to set up a MIRIx chapter, even better. In the meantime, we will proceed as if the only people reading this document are those who justifiably expect to find it reasonably useful. ⠀ Preamble II: Surface Area Imagine that you have been tasked with moving a cube of solid iron that is one meter on a side. Given that such a cube weighs ~16000 pounds, and that an average human can lift ~100 pounds, a naïve estimation tells you that you can solve this problem with ~150 willing friends. But of course, a meter cube can fit at most something like 10 people around it. It doesn’t matter if you have the theoretical power to move the cube if you can’t bring that power to bear in an effective manner. The problem is constrained by its surface area. MIRIx chapters are one of the best ways to increase the surface area of people thinking about and working on the technical problem of AI alignment. And just as it would be a bad idea to decree "the 10 people who happen to currently be closest to the metal cube are the only ones allowed to think about how to think about this problem", we don’t want MIRI to become the bottleneck or authority on what kinds of thinking can and should be done in the realm of embedded agency and other relevant fields of research. The hope is that you and others like you will help actually solve the problem, not just follow directions or read what’s already been written. This document is designed to support people who are interested in doing real groundbreaking research themselves. ⠀ Contents You and your research Logistics of getting started Models of social dynamics Other useful thoughts and questions ⠀ 1. You and your research We sometimes hear questions of the form “Even a summer internship feels too short to make meaningful progress on real problems. How can anyone expect to meet and do real research in a single afternoon?” There’s a Zeno-esque sense in which you can’t make research progress in a million years if you can’t also do it in five minutes. It’s easy to fall into a trap of (either implicitly or explicitly) conceptualizing “research” as “first studying and learning what’s already been figured out, and then attempting to push the boundaries and contribute new content.” The problem with this frame (according to us) is that it leads people to optimize for absorbing information, rather than seeking it instrume...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Pavlov Strategy, published by sarahconstantinon the LessWrong. Epistemic Status: Common knowledge, just not to me The Evolution of Trust is a deceptively friendly little interactive game. Near the end, there’s a “sandbox” evolutionary game theory simulator. It’s pretty flexible. You can do quick experiments in it without writing code. I highly recommend playing around. One of the things that surprised me was a strategy the game calls Simpleton, also known in the literature as Pavlov. In certain conditions, it works pretty well — even better than tit-for-tat or tit-for-tat with forgiveness. Let’s set the framework first. You have a Prisoner’s dilemma type game. If both parties cooperate, they each get +2 points. If one cooperates and the other defects, the defector gets +3 points and the cooperator gets -1 point If both defect, both get 0 points. This game is iterated — you’re randomly assigned to a partner and you play many rounds. Longer rounds reward more cooperative strategies; shorter rounds reward more defection. It’s also evolutionary — you have a proportion of bots each playing their strategies, and after each round, the bots with the most points replicate and the bots with the least points die out. Successful strategies will tend to reproduce while unsuccessful ones die out. In other words, this is the Darwin Game. Finally, it’s stochastic — there’s a small probability that any bot will make a mistake and cooperate or defect at random. Now, how does Pavlov work? Pavlov starts off cooperating. If the other player cooperates with Pavlov, Pavlov keeps doing whatever it’s doing, even if it was a mistake; if the other player defects, Pavlov switches its behavior, even if it was a mistake. In other words, Pavlov: cooperates when you cooperate with it, except by mistake “pushes boundaries” and keeps defecting when you cooperate, until you retaliate “concedes when punished” and cooperates after a defect/defect result “retaliates against unprovoked aggression”, defecting if you defect on it while it cooperates. If there’s any randomness, Pavlov is better at cooperating with itself than Tit-For-Tat. One accidental defection and two Tit-For-Tats are stuck in an eternal defect cycle, while Pavlov’s forgive each other and wind up back in a cooperate/cooperate pattern. Moreover, Pavlov can exploit CooperateBot (if it defects by accident, it will keep greedily defecting against the hapless CooperateBot, while Tit-For-Tat will not) but still exerts some pressure against DefectBot (defecting against it half the time, compared to Tit-For-Tat’s consistent defection.) The interesting thing is that Pavlov can beat Tit-For-Tat or Tit-for-Tat-with-Forgiveness in a wide variety of scenarios. If there are only Pavlov and Tit-For-Tat bots, Tit-For-Tat has to start out outnumbering Pavlov quite significantly in order to win. The same is true for a population of Pavlov and Tit-For-Tat-With-Forgiveness. It doesn’t change if we add in some Cooperators or Defectors either. Why? Compared to Tit-For-Tat, Pavlov cooperates better with itself. If two Tit-For-Tat bots are paired, and one of them accidentally defects, they’ll be stuck in a mutual defection equilibrium. However, if one Pavlov bot accidentally defects against its clone, we’ll see C/D -> D/D -> C->C which recovers a mutual-cooperation equilibrium and picks up more points. Compared to Tit-For-Tat-With-Forgiveness, Pavlov cooperates worse with itself (it takes longer to recover from mistakes) but it “exploits” TFTWF’s patience better. If Pavlov accidentally defects against TFTWF, the result is D/C -> D/C -> D/D -> C/D -> D/D -> C/C, which leaves Pavlov with a net gain of 1 point per turn, (over the first five turns before a cooperative equilibrium) compared to TFTWF’s 1/5 point per turn. If TFTWF accidentally defects against Pavl...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Tell Culture, published by LoganStrohl on the LessWrong. Followup to: Ask and Guess Ask culture: "I'll be in town this weekend for a business trip. Is it cool if I crash at your place?" Response: “Yes“ or “no”. Guess culture: "Hey, great news! I'll be in town this weekend for a business trip!" Response: Infer that they might be telling you this because they want something from you, conclude that they might want a place to stay, and offer your hospitality only if you want to. Otherwise, pretend you didn’t infer that. The two basic rules of Ask Culture: 1) Ask when you want something. 2) Interpret things as requests and feel free to say "no". The two basic rules of Guess Culture: 1) Ask for things if, and only if, you're confident the person will say "yes". 2) Interpret requests as expectations of "yes", and, when possible, avoid saying "no". Both approaches come with costs and benefits. In the end, I feel pretty strongly that Ask is superior. But these are not the only two possibilities! "I'll be in town this weekend for a business trip. I would like to stay at your place, since it would save me the cost of a hotel, plus I would enjoy seeing you and expect we’d have some fun. I'm looking for other options, though, and would rather stay elsewhere than inconvenience you." Response: “I think I need some space this weekend. But I’d love to get a beer or something while you’re in town!” or “You should totally stay with me. I’m looking forward to it.” There is a third alternative, and I think it's probably what rationalist communities ought to strive for. I call it "Tell Culture". The two basic rules of Tell Culture: 1) Tell the other person what's going on in your own mind whenever you suspect you'd both benefit from them knowing. (Do NOT assume others will accurately model your mind without your help, or that it will even occur to them to ask you questions to eliminate their ignorance.) 2) Interpret things people tell you as attempts to create common knowledge for shared benefit, rather than as requests or as presumptions of compliance. Suppose you’re in a conversation that you’re finding aversive, and you can’t figure out why. Your goal is to procure a rain check. Guess: You see this annoyed body language? Huh? Look at it! If you don’t stop talking soon I swear I’ll start tapping my foot. (Or, possibly, tell a little lie to excuse yourself. “Oh, look at the time.”) Ask: “Can we talk about this another time?” Tell: "I'm beginning to find this conversation aversive, and I'm not sure why. I propose we hold off until I've figured that out." Here are more examples from my own life: "I didn't sleep well last night and am feeling frazzled and irritable today. I apologize if I snap at you during this meeting. It isn’t personal." "I just realized this interaction will be far more productive if my brain has food. I think we should head toward the kitchen." "It would be awfully convenient networking for me to stick around for a bit after our meeting to talk with you and [the next person you're meeting with]. But on a scale of one to ten, it's only about 3 useful to me. If you'd rate the loss of utility for you as two or higher, then I have a strong preference for not sticking around." The burden of honesty is even greater in Tell culture than in Ask culture. To a Guess culture person, I imagine much of the above sounds passive aggressive or manipulative, much worse than the rude bluntness of mere Ask. It’s because Guess people aren’t expecting relentless truth-telling, which is exactly what’s necessary here. If you’re occasionally dishonest and tell people you want things you don't actually care about--like their comfort or convenience--they’ll learn not to trust you, and the inherent freedom of the system will be lost. They’ll learn that you only pretend to care about them to take...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Other people are wrong vs I am right, published by Buck the LessWrong. I’ve recently been spending some time thinking about the rationality mistakes I’ve made in the past. Here’s an interesting one: I think I have historically been too hasty to go from “other people seem very wrong on this topic” to “I am right on this topic”. Throughout my life, I’ve often thought that other people had beliefs that were really repugnant and stupid. Now that I am older and wiser, I still think I was correct to think that these ideas were repugnant and stupid. Overall I was probably slightly insufficiently dismissive of things like the opinions of apparent domain experts and the opinions of people who seemed smart whose arguments I couldn’t really follow. I also overrated conventional wisdom about factual claims about how the world worked, though I underrated conventional wisdom about how to behave. Examples of ideas where I thought the conventional wisdom was really dumb: I thought that animal farming was a massive moral catastrophe, and I thought it was a sign of terrible moral failure that almost everyone around me didn’t care about this and wasn’t interested when I brought it up. I thought that AI safety was a big deal, and I thought the arguments against it were all pretty stupid. (Nowadays the conventional wisdom has a much higher opinion of AI safety; I’m talking about 2010-2014.) I thought that people have terrible taste in economic policy, and that they mostly vote for good-sounding stuff that stops sounding good if you think about it properly for even a minute I was horrified by people proudly buying products that said “Made in Australia” on them; I didn’t understand how that wasn’t obviously racist, and I thought that we should make it much easier to allow anyone who wants to to come live in Australia. (This one has become much less controversial since Trump inadvertently convinced liberals that they should be in favor of immigration liberalization.) I thought and still think that a lot of people’s arguments about why it’s good to call the police on bike thieves were dumb. See eg many of the arguments people made in response to a post of mine about this (that in fairness was a really dumb post, IMO) I think I was right about other people being wrong. However, I think that my actual opinions on these topics were pretty confused and wrong, much more than I thought at the time. Here’s how I updated my opinion for all the things above: I have updated against the simple view of hedonic utilitarianism under which it’s plausible that simple control systems can suffer. A few years ago, I was seriously worried that the future would contain much more factory farming and therefore end up net negative; I now think that I overrated this fear, because (among other arguments) almost no-one actually endorses torturing animals, we just do it out of expediency, and in the limit of better technology our weak preferences will override our expediency. My understanding of AI safety was “eventually someone will build a recursively self improving singleton sovereign AGI, and we need to figure out how to build it such that it can have an off switch and it implements some good value function instead of something bad.” I think this picture was massively oversimplified. On the strategic side, I didn’t think about the possibilities of slower takeoffs or powerful technologies without recursive self improvement; on the technical safety side, I didn’t understand that it’s hard to even build a paperclip maximizer, and a lot of our effort might go into figuring out how to do that. Other people have terrible taste in economic policy, but I think that I was at the time overconfident in various libertarianish ideas that I’m now less enthusiastic about. Also, I no longer think it’s a slam dunk that society is b...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Your Cheerful Price, published by Eliezer Yudkowsky on the LessWrong. There's a concept I draw on often in social interactions. I've been calling it the "happy price", but that is originally terminology by Michael Ellsberg with subtly different implications. So I now fork off the term "cheerful price", and specialize it anew. Earlier Facebook discussion here. Tl;dr: When I ask you for your Cheerful Price for doing something, I'm asking you for the price that: Gives you a cheerful feeling about the transaction; Makes you feel energized about doing the thing; Doesn't generate an ouch feeling to be associated with me; Means I'm not expending social capital or friendship capital to make the thing happen; Doesn't require the executive part of you, that knows you need money in the long-term, to shout down and override other parts of you. The Cheerful Price is not: A "fair" price; The price you would pay somebody else to do similar work; The lowest price such that you'd feel sad about learning the transaction was canceled; The price that you'd charge a non-friend, though this is a good thing to check (see below); A price you're willing to repeat for future transactions, though this is a good thing to check (see below); The bare minimum amount of money such that you feel cheerful. It should include some safety margin to account for fluctuating feelings. The point of a Cheerful Price, from my perspective as somebody who's usually the one trying to emit money, is that: It lets me avoid the nightmare of accidentally inflicting small ouches on people; It lets me avoid the nightmare of spending social capital while having no idea how much I'm spending; It lets me feel good instead of bad about asking other people to do things. Warnings: Not everybody was raised with an attitude of "money is the unit of caring and the medium of cooperation" towards exchanges with an overt financial element. Some people may just not have a monetary price for some things, such that the exchange would boost rather than hurt their friendship, and their feelings too are valid. Not as valid as mine, of course, but still valid. "I don't have a cheerful price for that, would you like a non-cheerful price" is a valid response. Any time you ask somebody for a Cheerful Price, you are implicitly promising not to hold any price they name against them, even if it's a billion dollars. If Tell Culture doesn't work for someone, Cheerful Prices may not work for them either. If a friend didn't already ask for your cheerful price, it may be good to explicitly tell them when you're naming your cheerful price rather than your minimum price. Life does not promise us that we will always get our Cheerful Prices, even from our friends. Q: Why is my Cheerful Price not the same as the minimum price that would make me prefer doing the transaction to not doing it? If, on net, I'd rather do something than not do it, and I get to do it, shouldn't I feel cheerful about that? As an oversimplified model, imagine that your mind consists of a bunch of oft-conflicting desires, plus a magic executive whose job it is to decide what to do in the end. This magic executive also better understands concepts like "hyperbolic discounting" that the wordless voices don't understand as well. Now suppose that I don't want to hurt you even a little; and that I live in terror of accidentally overdrawing on other people's senses of friendship or obligation towards me; and that I worry about generating small ouches that your mind will thereafter associate with me. In this case I do not want to offer you the minimum price such that your executive part, which knows you need money in the long-term, would forcibly override your inner voices that don't understand hyperbolic discounting, and force them to accept the offer. Those parts of you may then feel b...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Importance of Sidekicks, published by Swimmer963 on the LessWrong. [Reposted from my personal blog.] Mindspace is wide and deep. “People are different” is a truism, but even knowing this, it’s still easy to underestimate. I spent much of my initial engagement with the rationality community feeling weird and different. I appreciated the principle and project of rationality as things that were deeply important to me; I was pretty pro-self improvement, and kept tsuyoku naritai as my motto for several years. But the rationality community, the people who shared this interest of mine, often seemed baffled by my values and desires. I wasn’t ambitious, and had a hard time wanting to be. I had a hard time wanting to be anything other than a nurse. It wasn’t until this August that I convinced myself that this wasn’t a failure in my rationality, but rather a difference in my basic drives. It’s around then, in the aftermath of the 2014 CFAR alumni reunion, that I wrote the following post. I don’t believe in life-changing insights (that happen to me), but I think I’ve had one–it’s been two weeks and I’m still thinking about it, thus it seems fairly safe to say I did. At a CFAR Monday test session, Anna was talking about the idea of having an “aura of destiny”–it’s hard to fully convey what she meant and I’m not sure I get it fully, but something like seeing yourself as you’ll be in 25 years once you’ve saved the world and accomplished a ton of awesome things. She added that your aura of destiny had to be in line with your sense of personal aesthetic, to feel “you.” I mentioned to Kenzi that I felt stuck on this because I was pretty sure that the combination of ambition and being the locus of control that “aura of destiny” conveyed to me was against my sense of personal aesthetic. Kenzi said, approximately [I don't remember her exact words]: “What if your aura of destiny didn’t have to be those things? What if you could be like.Samwise, from Lord of the Rings? You’re competent, but most importantly, you’re loyal to Frodo. You’re the reason that the hero succeeds.” I guess this isn’t true for most people–Kenzi said she didn’t want to keep thinking of other characters who were like this because she would get so insulted if someone kept comparing her to people’s sidekicks–but it feels like now I know what I am. So. I’m Samwise. If you earn my loyalty, by convincing me that what you’re working on is valuable and that you’re the person who should be doing it, I’ll stick by you whatever it takes, and I’ll make sure you succeed. I don’t have a Frodo right now. But I’m looking for one. It then turned out that quite a lot of other people recognized this, so I shifted from “this is a weird thing about me” to “this is one basic personality type, out of many.” Notably, Brienne wrote the following comment: Sidekick” doesn’t quite fit my aesthetic, but it’s extremely close, and I feel it in certain moods. Most of the time, I think of myself more as what TV tropes would call a “dragon”. Like the Witch-king of Angmar, if we’re sticking of LOTR. Or Bellatrix Black. Or Darth Vader. (It’s not my fault people aren’t willing to give the good guys dragons in literature.) For me, finding someone who shared my values, who was smart and rational enough for me to trust him, and who was in a much better position to actually accomplish what I most cared about than I imagined myself ever being, was the best thing that could have happened to me. She also gave me what’s maybe one of the best and most moving compliments I’ve ever received. In Australia, something about the way you interacted with people suggested to me that you help people in a completely free way, joyfully, because it fulfills you to serve those you care about, and not because you want something from them. I was able to relax around you, an...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Extreme Rationality: It's Not That Great, published by Scott Alexander on the LessWrong. Related to: Individual Rationality is a Matter of Life and Death, The Benefits of Rationality, Rationality is Systematized Winning But I finally snapped after reading: Mandatory Secret Identities Okay, the title was for shock value. Rationality is pretty great. Just not quite as great as everyone here seems to think it is. For this post, I will be using "extreme rationality" or "x-rationality" in the sense of "techniques and theories from Overcoming Bias, Less Wrong, or similar deliberate formal rationality study programs, above and beyond the standard level of rationality possessed by an intelligent science-literate person without formal rationalist training." It seems pretty uncontroversial that there are massive benefits from going from a completely irrational moron to the average intelligent person's level. I'm coining this new term so there's no temptation to confuse x-rationality with normal, lower-level rationality. And for this post, I use "benefits" or "practical benefits" to mean anything not relating to philosophy, truth, winning debates, or a sense of personal satisfaction from understanding things better. Money, status, popularity, and scientific discovery all count. So, what are these "benefits" of "x-rationality"? A while back, Vladimir Nesov asked exactly that, and made a thread for people to list all of the positive effects x-rationality had on their lives. Only a handful responded, and most responses weren't very practical. Anna Salamon, one of the few people to give a really impressive list of benefits, wrote: I'm surprised there are so few apparent gains listed. Are most people who benefited just being silent? We should expect a certain number of headache-cures, etc., just by placebo effects or coincidences of timing. There have since been a few more people claiming practical benefits from x-rationality, but we should generally expect more people to claim benefits than to actually experience them. Anna mentions the placebo effect, and to that I would add cognitive dissonance - people spent all this time learning x-rationality, so it MUST have helped them! - and the same sort of confirmation bias that makes Christians swear that their prayers really work. I find my personal experience in accord with the evidence from Vladimir's thread. I've gotten countless clarity-of-mind benefits from Overcoming Bias' x-rationality, but practical benefits? Aside from some peripheral disciplines1, I can't think of any. Looking over history, I do not find any tendency for successful people to have made a formal study of x-rationality. This isn't entirely fair, because the discipline has expanded vastly over the past fifty years, but the basics - syllogisms, fallacies, and the like - have been around much longer. The few groups who made a concerted effort to study x-rationality didn't shoot off an unusual number of geniuses - the Korzybskians are a good example. In fact as far as I know the only follower of Korzybski to turn his ideas into a vast personal empire of fame and fortune was (ironically!) L. Ron Hubbard, who took the basic concept of techniques to purge confusions from the mind, replaced the substance with a bunch of attractive flim-flam, and founded Scientology. And like Hubbard's superstar followers, many of this century's most successful people have been notably irrational. There seems to me to be approximately zero empirical evidence that x-rationality has a large effect on your practical success, and some anecdotal empirical evidence against it. The evidence in favor of the proposition right now seems to be its sheer obviousness. Rationality is the study of knowing the truth and making good decisions. How the heck could knowing more than everyone else and making ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Learned Blankness, published by AnnaSalamon on the LessWrong. Related to: Semantic stopsigns, Truly part of you. One day, the dishwasher broke. I asked Steve Rayhawk to look at it because he’s “good with mechanical things”. “The drain is clogged,” he said. “How do you know?” I asked. He pointed at a pool of backed up water. “Because the water is backed up.” We cleared the clog and the dishwasher started working. I felt silly, because I, too, could have reasoned that out. The water wasn’t draining -- therefore, perhaps the drain was clogged. Basic rationality in action.[1] But before giving it even ten seconds’ thought, I’d classified the problem as a “mechanical thing”. And I’d remembered I “didn’t know how mechanical things worked” (a cached thought). And then -- prompted by my cached belief that there was a magical “way mechanical things work” that some knew and I didn’t -- I stopped trying to think at all. “Mechanical things” was for me a mental stopsign -- a blank domain that stayed blank, because I never asked the obvious next questions (questions like “does the dishwasher look unusual in any way? Why is there water at the bottom?”). When I tutored math, new students acted as though the laws of exponents (or whatever we were learning) had fallen from the sky on stone tablets. They clung rigidly to the handed-down procedures. It didn’t occur to them to try to understand, or to improvise. The students treated math the way I treated broken dishwashers. Martin Seligman coined the term "learned helplessness" to describe a condition in which someone has learned to behave as though they were helpless. I think we need a term for learned helplessness about thinking (in a particular domain). I’ll call this “learned blankness”[2]. Folks who fall prey to learned blankness may still take actions -- sometimes my students practiced the procedures again and again, hired a tutor, etc. But they do so as though carrying out rituals to an unknown god -- parts of them may be trying, but their “understand X” center has given up. To avoid misunderstanding: calling a plumber, and realizing he knows more than you do, can be good. The thing to avoid is mentally walling off your own impressions; keeping parts of your map blank, because you imagine either that the domain itself is chaotic, or that one needs some special skillset to reason about that. Notice your learned blankness Learned blankness is common. My guess is that most of us treat most of our environment as blank givens inaccessible to reason[3]. To spot it in yourself, try comparing yourself to the following examples: 1. Sandra runs helpless to her roommate when her computer breaks -- she isn’t “good with computers”. Her roommate, by contrast, clicks on one thing and then another, doing Google searches and puzzling it out.[4] 2. Most scientists know the scientific method is good (and that e.g. p-values of 0.05 are good). But many not only don’t understand why the scientific method (or these p-values) are good -- they don’t understand that it’s the sort of thing one could understand. 3. Many respond to questions about consciousness, morality, or God by expecting that some other, special kind of reasoning is needed, and, thus, walling off and distrusting their own impressions. 4. Fred finds he has an intuition about how serious nano risks are. His intuition is a blank for him; something he can act on or ignore, but not examine. It doesn’t occur to him that he could examine the causes of his intuition[5], or could examine the accuracy rate of similar intuitions. 5. I find it hard to fully try to write fiction -- though a drink of alcohol helps. The trouble is that since I’m unskilled at fiction-writing, and since I find it painful to notice my un-skill, most of my mind prefers to either not write at all, or to write half-heartedly...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What cognitive biases feel like from the inside, published by chaosmage on the LessWrong. Building on the recent SSC post Why Doctors Think They’re The Best... What it feels like for me How I see others who feel the same There is controversy on the subject but there shouldn't be because the side I am on is obviously right. They have taken one side in a debate that is unresolved for good reason that they are struggling to understand I have been studying this carefully They preferentially seek out conforming evidence The arguments for my side make obvious sense, they're almost boring. They're very ready to accept any and all arguments for their side. The arguments for the opposing side are contradictory, superficial, illogical or debunked. They dismiss arguments for the opposing side at the earliest opportunity. The people on the opposing side believe these arguments mostly because they are uninformed, have not thought about it enough or are being actively misled by people with bad motives. The flawed way they perceive the opposing side makes them confused about how anyone could be on that side. They resolve that confusion by making strong assumptions that can approach conspiracy theories. The scientific term for this mismatch is: confirmation bias What it feels like for me How I see others who feel the same My customers/friends/relationships love me, so I am good for them, so I am probably just generally good. They neglect the customers / friends / relationships that did not love them and have left, so they overestimate how good they are. When customers / friends / relationships switch to me, they tell horror stories of who I'm replacing for them, so I'm better than those. They don't see the people who are happy with who they have and therefore never become their customers / friends / relationships. The scientific term for this mismatch is: selection bias What it feels like for me How I see others who feel the same Although I am smart and friendly, people don't listen to me. Although they are smart and friendly, they are hard to understand. I have a deep understanding of the issue that people are too stupid or too disinterested to come to share. They are failing to communicate their understanding, or to give unambiguous evidence they even have it. This lack of being listened to affects several areas of my life but it is particularly jarring on topics that are very important to me. This bad communication affects all areas of their life, but on the unimportant ones they don't even understand that others don't understand them. The scientific term for this mismatch is: illusion of transparency What it feels like for me How I see others who feel the same I knew at the time this would not go as planned. They did not predict what was going to happen. The plan was bad and we should have known it was bad. They fail to appreciate how hard prediction is, so the mistake seems more obvious to them than it was. I knew it was bad, I just didn't say it, for good reasons (e.g. out of politeness or too much trust in those who made the bad plan) or because it is not my responsibility or because nobody listens to me anyway. In order to avoid blame for the seemingly obvious mistake, they are making up excuses. The scientific term for this mismatch is: hindsight bias What it feels like for me How I see others who feel the same I have a good intuition; even decisions I make based on insufficient information tend to turn out to be right. They tend to recall their own successes and forget their own failures, leading to an inflated sense of past success. I know early on how well certain projects are going to go or how well I will get along with certain people. They make self-fulfilling prophecies that directly influence how much effort they put into a project or relationship. Compared to other...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Hiring engineers and researchers to help align GPT-3, published by paulfchristiano on the LessWrong. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. My team at OpenAI, which works on aligning GPT-3, is hiring ML engineers and researchers. Apply here for the ML engineer role and here for the ML researcher role. GPT-3 is similar enough to "prosaic" AGI that we can work on key alignment problems without relying on conjecture or speculative analogies. And because GPT-3 is already being deployed in the OpenAI API, its misalignment matters to OpenAI’s bottom line — it would be much better if we had an API that was trying to help the user instead of trying to predict the next word of text from the internet. I think this puts our team in a great place to have an impact: If our research succeeds I think it will directly reduce existential risk from AI. This is not meant to be a warm-up problem, I think it’s the real thing. We are working with state of the art systems that could pose an existential risk if scaled up, and our team’s success actually matters to the people deploying those systems. We are working on the whole pipeline from “interesting idea” to “production-ready system,” building critical skills and getting empirical feedback on whether our ideas actually work. We have the real-world problems to motivate alignment research, the financial support to hire more people, and a research vision to execute on. We are bottlenecked by excellent researchers and engineers who are excited to work on alignment. What the team does In the past Reflection focused on fine-tuning GPT-3 using a reward function learned from human feedback. Our most recent results are here, and had the unusual virtue of simultaneously being exciting enough to ML researchers to be accepted at NeurIPS while being described by Eliezer as “directly, straight-up relevant to real alignment problems.” We’re currently working on three things: [20%] Applying basic alignment approaches to the API, aiming to close the gap between theory and practice. [60%] Extending existing approaches to tasks that are too hard for humans to evaluate; in particular, we are training models that summarize more text than human trainers have time to read. Our approach is to use weaker ML systems operating over shorter contexts to help oversee stronger ones over longer contexts. This is conceptually straightforward but still poses significant engineering and ML challenges. [20%] Conceptual research on domains that no one knows how to oversee and empirical work on debates between humans (see our 2019 writeup). I think the biggest open problem is figuring out how and if human overseers can leverage “knowledge” the model acquired during training (see an example here). If successful, ideas will eventually move up this list, from the conceptual stage to ML prototypes to real deployments. We’re viewing this as practice for integrating alignment into transformative AI deployed by OpenAI or another organization. What you’d do Most people on the team do a subset of these core tasks: Design+build+maintain code for experimenting with novel training strategies for large language models. This infrastructure needs to support a diversity of experimental changes that are hard to anticipate in advance, work as a solid base to build on for 6-12 months, and handle the complexity of working with large language models. Most of our code is maintained by 1-3 people and consumed by 2-4 people (all on the team). Oversee ML training. Evaluate how well models are learning, figure out why they are learning badly, and identify+prioritize+implement changes to make them learn better. Tune hyperparameters and manage computing resources. Process datasets for machine consumption; understand datasets and how they affect the model...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Arbital postmortem , published by alexei on the LessWrong. Disclaimer 1: These views are my own and don’t necessarily reflect the views of anyone else (Eric, Steph, or Eliezer). Disclaimer 2: Most of the events happened at least a year ago. My memory is not particularly great, so the dates are fuzzy and a few things might be slightly out of order. But this post has been reviewed by Eric, Steph, and Eliezer, so it should mostly be okay. I’m going to list events chronologically. At times I’ll insert a “Reflection” paragraph, where I’m going to outline my thoughts as of now. I’ll talk about what I could have done differently and how I would approach a similar problem today. Chapter 0: Eliezer pitches Arbital and I say ‘no’ Around the summer of 2014 Eliezer approached me with the idea for what later would become Arbital. At first, I vaguely understood the idea as some kind of software to map out knowledge. Maybe something like a giant mind map, but not graphical. I took some time to research existing and previous projects in that area and found a huge graveyard of projects that have been tried. Yes, basically all of them were dead. Most were hobby projects, but some seemed pretty serious. None were successful, as far as I could tell. I didn’t see how Eliezer’s project was different, so I passed on it. Reflection: Today, I’d probably try to sit down with Eliezer for longer and really try to understand what he is seeing that I’m not. It’s likely back then I didn’t have the right skills to extract that information, but I think I’m much better at it today. Reflection: Also, after working with Eliezer for a few years, I’ve got a better feeling for how things he says often seem confusing / out of alignment / tilted, until you finally wrap your mind around it, and then it’s crystal clear and easy. Chapter 1: Eliezer and I start Arbital Early January 2015 I was sitting in my room, tired from looking in vain for a decent startup idea, when Arbital popped back into my mind. There were still a lot of red flags around the idea, but I rationalized to myself that given Eliezer’s track record, there was probably something good here. And, in the worst case, I’d just create a tool that would be useful to Eliezer alone. That didn’t seem like a bad outcome, so I decided to do it. I contacted Eliezer, he was still interested, and so we started the project. Reflection: The decision process sounds a bit silly, but I don’t think it’s a bad one. I really prefer to do something decently useful, rather than sit around waiting for something perfect. I also still approve of the heuristic of accepting quests / projects from people you think are good at coming up with quests / projects. But if I did it again, I’d definitely put a lot more effort upfront to understand the entire vision before committing to it. Reflection: Paul Graham wrote in one of his essays that it’s okay (though not ideal) to initially build a product for just one user. There are, of course, several caveats. The user needs to use the product extensively, otherwise you don’t get the necessary feedback on all the features you’re building. And the user needs to be somewhat typical of other users you hope to attract to the platform. Reflection: Unfortunately, both of these turned out to be false. I’ll elaborate on the feature usage below. But the “typical” part probably could have been foreseen. There are only a few people in the world who write explanations at the scale and complexity that Eliezer does. The closest cluster is probably people writing college textbooks. So, in the beginning, I didn’t have any sense for who the first 10-100 users were going to be. That would have been fine if I was just building a tool for Eliezer, but since my goal was explicitly to create a for-profit consumer startup, this was a big mistake. Eliezer ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Personalized Medicine For Real, published by sarahconstantin on the LessWrong. I was part of the founding team at MetaMed, a personalized medicine startup. We went out of business back in 2015. We made a lot of mistakes due to inexperience, some of which I deeply regret. I’m reflecting on that now, because Perlara just went out of business, and they got a lot farther on our original dream than we ever did. Q-State Biosciences, which is still around, is using a similar model. The phenomenon that inspired MetaMed is that we knew of stories of heroic, scientifically literate patients and families of patients with incurable diseases, who came up with cures for their own conditions. Physicist Leo Szilard, the “father of the atom bomb”, designed a course of radiation therapy to cure his own bladder cancer. Computer scientist Matt Might analyzed his son’s genome to find a cure for his rare disorder. Cognitive scientist Joshua Tenenbaum found a personalized treatment for his father’s cancer. So, we thought, could we try to scale up this process to help more people? In Lois McMaster Bujold’s science fiction novels, the hero suffers an accident that leaves him with a seizure disorder. He goes to a medical research center and clinic, the Durona Group, and they design a neural prosthetic for him that prevents the seizures. This sounds like it ought to be a thing that exists. Patient-led, bench-to-bedside drug discovery or medical device engineering. You get an incurable disease, you fund scientists/doctors/engineers to discover a cure, and now others with the disease can also be cured. There’s actually a growing community of organizations trying to do things sort of in this vein. Recursion Pharmaceuticals, where I used to work, does drug discovery for rare diseases. Sv.ai organizes hackathons for analyzing genetic data to help patients with rare diseases find the root cause. Perlara and Q-state use animal models and in-vitro models respectively to simulate patients’ disorders, and then look for drugs or gene therapies that reverse those disease phenotypes in the animals or cells. Back at MetaMed, I think we were groping towards something like this, but never really found our way there. One reason is that we didn’t narrow our focus enough. We were trying to solve too many problems at once, all called “personalized medicine.” Personalized Lifestyle Optimization Some “personalized medicine” is about health optimization for basically healthy people. A lot of it amounts to superficial personalization on top of generic lifestyle advice. Harmless, but more of a marketing thing than a science thing, and not very interesting from a humanitarian perspective. Sometimes, we tried to get clients from this market. I pretty much always thought this was a bad idea. Personalized Medicine For All Some “personalized medicine” is about the claim that the best way to treat even common diseases often depends on individual factors, such as genes. This was part of our pitch, but as I learned more, I came to believe that this kind of “personalization” has very little applicability. In most cases, we don’t know enough about how genes affect response to treatment to be able to improve outcomes by stratifying treatments based on genes. In the few cases where we know people with different genes need different treatments, it’s often already standard medical practice to run those tests. I now think there’s not a clear opportunity for a startup to improve the baseline through this kind of personalized medicine. Preventing Medical Error Some of our founding inspirations were the work of Gerd Gigerenzer and Atul Gawande, who showed that medical errors were the cause of many deaths, that doctors tend to be statistically illiterate, and that systematizing tools like checklists and statistical prediction rules save...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A LessWrong Crypto Autopsy, published by Scott Alexander on the LessWrong. Wei Dai, one of the first people Satoshi Nakamoto contacted about Bitcoin, was a frequent Less Wrong contributor. So was Hal Finney, the first person besides Satoshi to make a Bitcoin transaction. The first mention of Bitcoin on Less Wrong, a post called Making Money With Bitcoin, was in early 2011 - when it was worth 91 cents. Gwern predicted that it could someday be worth "upwards of $10,000 a bitcoin". He also quoted Moldbug, who advised that: If Bitcoin becomes the new global monetary system, one bitcoin purchased today (for 90 cents, last time I checked) will make you a very wealthy individual...Even if the probability of Bitcoin succeeding is epsilon, a million to one, it's still worthwhile for anyone to buy at least a few bitcoins now...I would not put it at a million to one, though, so I recommend that you go out and buy a few bitcoins if you have the technical chops. My financial advice is to not buy more than ten, which should be F-U money if Bitcoin wins. A few people brought up some other points, like that if it ever became popular people might create a bunch of other cryptocurrencies, or that if there was too much controversy the Bitcoin economy might have to fork. The thread got a hundred or so comments before dying down. But Bitcoin kept getting mentioned on Less Wrong over the next few years. It's hard to select highlights, but one of them is surely Ander's Why You Should Consider Buying Bitcoin Right Now If You Have High Risk Tolerance from January 2015. Again, people made basically the correct points and the correct predictions, and the thread got about a hundred comments before dying down. I mention all this because of an idea, with a long history in this movement, that "rationalists should win". They should be able to use their training in critical thinking to recognize more opportunities, make better choices, and end up with more of whatever they want. So far it's been controversial to what degree we've lived up to that hope, or to what degree it's even realistic. Well, suppose God had decided, out of some sympathy for our project, to make winning as easy as possible for rationalists. He might have created the biggest investment opportunity of the century, and made it visible only to libertarian programmers willing to dabble in crazy ideas. And then He might have made sure that all of the earliest adapters were Less Wrong regulars, just to make things extra obvious. This was the easiest test case of our "make good choices" ability that we could possibly have gotten, the one where a multiply-your-money-by-a-thousand-times opportunity basically fell out of the sky and hit our community on its collective head. So how did we do? I would say we did mediocre. According to the recent SSC survey, 9% of SSC readers made $1000+ from crypto as of 12/2017. Among people who were referred to SSC from Less Wrong - my stand-in for long-time LW regulars - 15% made over $1000 on crypto, nearly twice as many. A full 3% of LWers made over $100K. That's pretty good. On the other hand, 97% of us - including me - didn't make over $100K. All we would have needed to do was invest $10 (or a few CPU cycles) back when people on LW started recommending it. But we didn't. How bad should we feel, and what should we learn? Here are the lessons I'm taking from this. 1: Our epistemic rationality has probably gotten way ahead of our instrumental rationality When I first saw the posts saying that cryptocurrency investments were a good idea, I agreed with them. I even Googled "how to get Bitcoin" and got a bunch of technical stuff that seemed like a lot of work. So I didn't do it. Back in 2016, my father asked me what this whole "cryptocurrency" thing was, and I told him he should invest in Ethereum. He did, ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Zettelkasten Method, published by abramdemski on the LessWrong. [Epistemic Status: Scroll to the bottom for my follow-up thoughts on this from months/years later.] Early this year, Conor White-Sullivan introduced me to the Zettelkasten method of note-taking. I would say that this significantly increased my research productivity. I’ve been saying “at least 2x”. Naturally, this sort of thing is difficult to quantify. The truth is, I think it may be more like 3x, especially along the dimension of “producing ideas” and also “early-stage development of ideas”. (What I mean by this will become clearer as I describe how I think about research productivity more generally.) However, it is also very possible that the method produces serious biases in the types of ideas produced/developed, which should be considered. (This would be difficult to quantify at the best of times, but also, it should be noted that other factors have dramatically decreased my overall research productivity. So, unfortunately, someone looking in from outside would not see an overall boost. Still, my impression is that it's been very useful.) I think there are some specific reasons why Zettelkasten has worked so well for me. I’ll try to make those clear, to help readers decide whether it would work for them. However, I honestly didn’t think Zettelkasten sounded like a good idea before I tried it. It only took me about 30 minutes of working with the cards to decide that it was really good. So, if you’re like me, this is a cheap experiment. I think a lot of people should actually try it to see how they like it, even if it sounds terrible. My plan for this document is to first give a short summary and then an overview of Zettelkasten, so that readers know roughly what I’m talking about, and can possibly experiment with it without reading any further. I’ll then launch into a longer discussion of why it worked well for me, explaining the specific habits which I think contributed, including some descriptions of my previous approaches to keeping research notes. I expect some of this may be useful even if you don’t use Zettelkasten -- if Zettelkasten isn’t for you, maybe these ideas will nonetheless help you to think about optimizing your notes. However, I put it here primarily because I think it will boost the chances of Zettelkasten working for you. It will give you a more concrete picture of how I use Zettelkasten as a thinking tool. Very Short Summary Materials Staples index-cards-on-a-ring or equivalent, possibly with: plastic rings rather than metal different 3x5 index cards (I recommend blank, but, other patterns may be good for you) as desired some kind of divider I use yellow index cards as dividers, but slightly larger cards, tabbed cards, plastic dividers, etc. might be better quality hole punch (if you’re using different cards than the pre-punched ones) I like this one. Blank stickers or some other way to label card-binders with the address range stored within. quality writing instrument -- must suit you, but, multi-color click pen recommended hi-tec-c coleto especially recommended Technique Number pages with alphanumeric strings, so that pages can be sorted hierarchically rather than linearly -- 11a goes between 11 and 12, 11a1 goes between 11a and 11b, et cetera. This allows pages to be easily inserted between other pages without messing up the existing ordering, which makes it much easier to continue topics. Use the alphanumeric page identifiers to “hyperlink” pages. This allows sub-topics and tangents to be easily split off into new pages, and also allows for related ideas to be interlinked. Before I launch into the proper description of Zettelkasten, here are some other resources on note-taking which I looked at before diving into using Zettelkasten myself. (Feel free to skip this part on a ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Fallacy of Gray, published by Eliezer Yudkowsky on the LessWrong. The Sophisticate: “The world isn’t black and white. No one does pure good or pure bad. It’s all gray. Therefore, no one is better than anyone else.” The Zetet: “Knowing only gray, you conclude that all grays are the same shade. You mock the simplicity of the two-color view, yet you replace it with a one-color view . . .” Marc Stiegler, David’s Sling I don’t know if the Sophisticate’s mistake has an official name, but I call it the Fallacy of Gray. We saw it manifested in the previous essay—the one who believed that odds of two to the power of seven hundred and fifty million to one, against, meant “there was still a chance.” All probabilities, to him, were simply “uncertain” and that meant he was licensed to ignore them if he pleased. “The Moon is made of green cheese” and “the Sun is made of mostly hydrogen and helium” are both uncertainties, but they are not the same uncertainty. Everything is shades of gray, but there are shades of gray so light as to be very nearly white, and shades of gray so dark as to be very nearly black. Or even if not, we can still compare shades, and say “it is darker” or “it is lighter.” Years ago, one of the strange little formative moments in my career as a rationalist was reading this paragraph from Player of Games by Iain M. Banks, especially the sentence in bold: A guilty system recognizes no innocents. As with any power apparatus which thinks everybody’s either for it or against it, we’re against it. You would be too, if you thought about it. The very way you think places you amongst its enemies. This might not be your fault, because every society imposes some of its values on those raised within it, but the point is that some societies try to maximize that effect, and some try to minimize it. You come from one of the latter and you’re being asked to explain yourself to one of the former. Prevarication will be more difficult than you might imagine; neutrality is probably impossible. You cannot choose not to have the politics you do; they are not some separate set of entities somehow detachable from the rest of your being; they are a function of your existence. I know that and they know that; you had better accept it. Now, don’t write angry comments saying that, if societies impose fewer of their values, then each succeeding generation has more work to start over from scratch. That’s not what I got out of the paragraph. What I got out of the paragraph was something which seems so obvious in retrospect that I could have conceivably picked it up in a hundred places; but something about that one paragraph made it click for me. It was the whole notion of the Quantitative Way applied to life-problems like moral judgments and the quest for personal self-improvement. That, even if you couldn’t switch something from on to off, you could still tend to increase it or decrease it. Is this too obvious to be worth mentioning? I say it is not too obvious, for many bloggers have said of Overcoming Bias: “It is impossible, no one can completely eliminate bias.” I don’t care if the one is a professional economist, it is clear that they have not yet grokked the Quantitative Way as it applies to everyday life and matters like personal self-improvement. That which I cannot eliminate may be well worth reducing. Or consider an exchange between Robin Hanson and Tyler Cowen.1 Robin Hanson said that he preferred to put at least 75% weight on the prescriptions of economic theory versus his intuitions: “I try to mostly just straightforwardly apply economic theory, adding little personal or cultural judgment.” Tyler Cowen replied: In my view there is no such thing as “straightforwardly applying economic theory” . . . theories are always applied through our personal and cultural filters and the...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Long Covid Is Not Necessarily Your Biggest Problem, published by Elizabeth on the LessWrong. At this point, people I know are not that worried about dying from covid. We’re all vaccinated, we’re mostly young and healthy(ish), and it turns out the odds were always low for us. We’re also not that worried about hospitalization: it’s much more likely than death, but maintaining covid precautions indefinitely is very costly so by and large we’re willing to risk it. The big unknown here has been long covid. Losing a few weeks to being extremely sick might be worth the risk, but a lifetime of fatigue and reduced cognition is a very big deal. With that in mind, I set out to do some math on what risks we were running. Unfortunately baseline covid has barely been around long enough to have data on long covid, most of it is still terrible, and the vaccine and Delta variant have not been widespread long enough to have much data at all. In the end, the conclusion I came to was that for vaccinated people under 40 with <=1 comorbidiy, the cognitive risks of long covid are lost in the noise of other risks they commonly take. Coming to this conclusion involved reading a number of papers, but also a lot of emotional processing around risk and health. I’ve included that processing under a “personal stuff” section, which you can skip if you just want the info but I encourage you to read if you feel yourself starting to yell that I’m not taking small risks of great suffering seriously. I do encourage you to read the caveats section before deciding how much weight to put on my conclusions. Personal Stuff This post took a long time to write, much longer than I wanted, because this is not an abstract topic to me. I have chronic pain from nerve damage in my jaw caused by medical incompetence, and my attempts to seek treatment for this continually run into the brick wall of a medical system that doesn’t consider my pain important (tangent: if you have a pain specialist you trust, anywhere in the US, please e-mail me (elizabeth@acesounderglass.com)). I empathize very much with the long covid sufferers who are being told their suffering doesn’t exist because it’s too hard to measure and we can’t prove what caused it. Additionally, I’m still suffering from side effects from my covid vaccine in April. It’s very minor, chest congestion that doesn’t seem to affect my lung capacity (but I don’t have a clear before picture, so hard to say for sure). But it’s getting worse and while my medical practitioners are taking it seriously, this + the experience with dental pain make me very sensitive to the possibility they might stop if it becomes too much work for them. As I type this, I am taking a supplement stack from a high end internet crackpot because first line treatment failed and there aren’t a lot of other options. And that’s just from the vaccine; I imagine if I actually had covid I would not be one of the people who shakes it off the way I describe later in this post. All this is to say that when I describe the long term cognitive impact of covid as being too small to measure with our current tools against our current noise levels, that is very much not the same as saying it’s zero. It’s much worse than that. What I’m saying is that you are taking risks of similar levels of suffering and impairment constantly, which our health system is very bad at measuring, and against that background long covid does not make much of a difference for people within certain age and health parameters. A common complaint when people say “X isn’t dangerous to the young and healthy” is that it implies the death and suffering of those who aren’t young and healthy don’t matter. I’m not saying that. It matters a lot, and it’s impossible for me to forget that because I’m very unlikely to be one of the people who gets to...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Cached Selves, published by AnnaSalamon on the LessWrong. by Anna Salamon and Steve Rayhawk (joint authorship) Related to: Beware identity Update, 2021: I believe a large majority of the priming studies failed replication, though I haven't looked into it in depth. I still personally do a great many of the "possible strategies" listed at the bottom; and they subjectively seem useful to me; but if you end up believing that it should not be on the basis of the claimed studies. A few days ago, Yvain introduced us to priming, the effect where, in Yvain’s words, "any random thing that happens to you can hijack your judgment and personality for the next few minutes." Today, I’d like to discuss a related effect from the social psychology and marketing literatures: “commitment and consistency effects”, whereby any random thing you say or do in the absence of obvious outside pressure, can hijack your self-concept for the medium- to long-term future. To sum up the principle briefly: your brain builds you up a self-image. You are the kind of person who says, and does... whatever it is your brain remembers you saying and doing. So if you say you believe X... especially if no one’s holding a gun to your head, and it looks superficially as though you endorsed X “by choice”... you’re liable to “go on” believing X afterwards. Even if you said X because you were lying, or because a salesperson tricked you into it, or because your neurons and the wind just happened to push in that direction at that moment. For example, if I hang out with a bunch of Green Sky-ers, and I make small remarks that accord with the Green Sky position so that they’ll like me, I’m liable to end up a Green Sky-er myself. If my friends ask me what I think of their poetry, or their rationality, or of how they look in that dress, and I choose my words slightly on the positive side, I’m liable to end up with a falsely positive view of my friends. If I get promoted, and I start telling my employees that of course rule-following is for the best (because I want them to follow my rules), I’m liable to start believing in rule-following in general. All familiar phenomena, right? You probably already discount other peoples’ views of their friends, and you probably already know that other people mostly stay stuck in their own bad initial ideas. But if you’re like me, you might not have looked carefully into the mechanisms behind these phenomena. And so you might not realize how much arbitrary influence consistency and commitment is having on your own beliefs, or how you can reduce that influence. (Commitment and consistency isn’t the only mechanism behind the above phenomena; but it is a mechanism, and it’s one that’s more likely to persist even after you decide to value truth.) Consider the following research. In the classic 1959 study by Festinger and Carlsmith, test subjects were paid to tell others that a tedious experiment has been interesting. Those who were paid $20 to tell the lie continued to believe the experiment boring; those paid a mere $1 to tell the lie were liable later to report the experiment interesting. The theory is that the test subjects remembered calling the experiment interesting, and either: Honestly figured they must have found the experiment interesting -- why else would they have said so for only $1? (This interpretation is called self-perception theory.), or Didn’t want to think they were the type to lie for just $1, and so deceived themselves into thinking their lie had been true. (This interpretation is one strand within cognitive dissonance theory.) In a follow-up, Jonathan Freedman used threats to convince 7- to 9-year old boys not to play with an attractive, battery-operated robot. He also told each boy that such play was “wrong”. Some boys were given big threats, or were kept carefully su...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Simulate and Defer To More Rational Selves, published by LoganStrohl on the LessWrong. I sometimes let imaginary versions of myself make decisions for me. I first started doing this after a friend told me (something along the lines of) this story. When they first became executive director of their organization, they suddenly had many more decisions to deal with per day than ever before. "Should we hire this person?" "Should I go buy more coffee for the coffee machine, or wait for someone else deal with it?" "How many participants should attend our first event?" "When can I schedule time to plan the fund drive?" I'm making up these examples myself, but I'm sure you, too, can imagine how leading a brand new organization might involve a constant assault on the parts of your brain responsible for making decisions. They found it exhausting, and by the time they got home at the end of the day, a question like, "Would you rather we have peas or green beans with dinner?" often felt like the last straw. "I don't care about the stupid vegetables, just give me food and don't make me decide any more things!" They were rescued by the following technique. When faced with a decision, they'd imagine "the Executive Director of the organization", and ask themselves, "What would 'the Executive Director of the organization' do?" Instead of making a decision, they'd make a prediction about the actions of that other person. Then, they'd just do whatever that person would do! In my friend's case, they were trying to reduce decision fatigue. When I started trying it out myself, I was after a cure for something slightly different. Imagine you're about to go bungee jumping off a high cliff. You know it's perfectly safe, and all you have to do is take a step forward, just like you've done every single time you've ever walked. But something is stopping you. The decision to step off the ledge is entirely yours, and you know you want to do it because this is why you're here. Yet here you are, still standing on the ledge. You're scared. There's a battle happening in your brain. Part of you is going, "Just jump, it's easy, just do it!", while another part--the part in charge of your legs, apparently--is going, "NOPE. Nope nope nope nope NOPE." And you have this strange thought: "I wish someone would just push me so I don't have to decide." Maybe you've been bungee jumping, and this is not at all how you responded to it. But I hope (for the sake of communication) that you've experienced this sensation in other contexts. Maybe when you wanted to tell someone that you loved them, but the phrase hovered just behind your lips, and you couldn't get it out. You almost wished it would tumble out of your mouth accidentally. "Just say it," you thought to yourself, and remained silent. For some reason, you were terrified of the decision, and inaction felt more like not deciding. When I heard this story from my friend, I had social anxiety. I didn't have way more decisions than I knew how to handle, but I did find certain decisions terrifying, and was often paralyzed by them. For example, this always happened if someone I liked, respected, and wanted to interact with more asked to meet with them. It was pretty obvious to me that it was a good idea to say yes, but I'd agonize over the email endlessly instead of simply typing "yes" and hitting "send". So here's what it looked like when I applied the technique. I'd be invited to a party. I'd feel paralyzing fear, and a sense of impending doom as I noticed that I likely believed going to the party was the right decision. Then, as soon as I felt that doom, I'd take a mental step backward and not try to force myself to decide. Instead, I'd imagine a version of myself who wasn't scared, and I'd predict what she'd do. If the party really wasn't a great idea, either be...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Gravity Turn, published by alkjashon the LessWrong. This is a linkpost for/ [The first in a sequence of retrospective essays on my five years in math graduate school.] My favorite analogy for graduate school is the gravity turn: the maneuver a rocket performs to get from the launch pad to orbit. I like to imagine a first-year graduate student as a Falcon X rocket, newly-constructed and tasked with delivering a six-ton payload into low Earth orbit. Picture this: you begin graduate school, fresh as a rocket arriving at Cape Canaveral and bubbling with excitement for your maiden voyage. Your PhD adviser, on the other hand, is the Hubble Space Telescope. Let's call her Dr. Hubble (not to be confused with the astronomer of the same name). Dr. Hubble is ostensibly the ideal guide for your first orbit insertion. After all, she is famously good at staying in orbit - she’s been up there since 1990. But problems quickly arise as you probe Dr. Hubble for advice on how to approach the launch. Namely: She left Earth more than thirty years ago, and space technology has since been completely revolutionized. She states all advice at an extremely high level with birds-eye-view detachment, observing, as she is, from a vantage point a thousand miles overhead. Most fatally, the Hubble Space Telescope vessel does not include the lower-stage rockets that brought her into space. In fact, she doesn’t include large engines of any kind. Her thirty years of experience free-falling in orbit will do you very little good until you break out of the stratosphere. The problem is even worse than this, however. It is not that Dr. Hubble, despite her best intentions, gives outdated advice. It is not even that Dr. Hubble cannot consciously articulate all the illegible skills she’s reflexively performing to stay in orbit. The problem is that even if you could perfectly imitate what Dr. Hubble is doing right now, you would likely still crash and burn. What I didn’t understand going into graduate school is that academic mathematicians are often working in a state akin to the free-fall of orbit. The Hubble Space Telescope remains in orbit around Earth because it travels horizontally so quickly that, even as it’s continuously accelerating towards the Earth, it continually misses. The laws of physics have arranged it so that it is not possible - barring deliberate sabotage - for her to fall back into a sub-orbital trajectory. Similarly, a successful research professor is embedded in an intricate system that, as surely as Newton’s laws, keeps her in a state of steadily producing new research. Many of her ground-breaking papers are not one-off productions - they produce sequels, variants, and interdisciplinary applications year after year. She has cultivated dozens of long-time collaborators of the highest level who freely share ideas and research directions, and has the reputation to find more at will. She attends conferences every other month that keep her updated on the leading edge of the field. Every year her research group grows, as if by clockwork, adding a couple graduate students and postdocs to whom she can delegate projects with only the gentlest supervision. As a result, the careers of many other people depend on Dr. Hubble to continue producing research at a steady rate. Every incentive is aligned for objects in motion to stay in motion, and it would take deliberate sabotage to bring Dr. Hubble out of her successful research trajectory. This is not to say that academic researchers all start cruising in free-fall after they leave graduate school or make tenure. It is perfectly normal for a spaceship that reaches orbit to proceed onto its next adventure after some rest, continuing on to visit another planet or leave the solar system altogether. The best researchers I know are similarly courageous, ta...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Feeling Rational , published by Eliezer Yudkowsky on the LessWrong. Since curiosity is an emotion, I suspect that some people will object to treating curiosity as a part of rationality. A popular belief about “rationality” is that rationality opposes all emotion—that all our sadness and all our joy are automatically anti-logical by virtue of being feelings. Yet strangely enough, I can’t find any theorem of probability theory which proves that I should appear ice-cold and expressionless. When people think of “emotion” and “rationality” as opposed, I suspect that they are really thinking of System 1 and System 2—fast perceptual judgments versus slow deliberative judgments. System 2’s deliberative judgments aren’t always true, and System 1’s perceptual judgments aren’t always false; so it is very important to distinguish that dichotomy from “rationality.” Both systems can serve the goal of truth, or defeat it, depending on how they are used. For my part, I label an emotion as “not rational” if it rests on mistaken beliefs, or rather, on mistake-producing epistemic conduct. “If the iron approaches your face, and you believe it is hot, and it is cool, the Way opposes your fear. If the iron approaches your face, and you believe it is cool, and it is hot, the Way opposes your calm.” Conversely, an emotion that is evoked by correct beliefs or truth-conducive thinking is a “rational emotion”; and this has the advantage of letting us regard calm as an emotional state, rather than a privileged default. So is rationality orthogonal to feeling? No; our emotions arise from our models of reality. If I believe that my dead brother has been discovered alive, I will be happy; if I wake up and realize it was a dream, I will be sad. P. C. Hodgell said: “That which can be destroyed by the truth should be.” My dreaming self’s happiness was opposed by truth. My sadness on waking is rational; there is no truth which destroys it. Rationality begins by asking how-the-world-is, but spreads virally to any other thought which depends on how we think the world is. Your beliefs about “how-the-world-is” can concern anything you think is out there in reality, anything that either does or does not exist, any member of the class “things that can make other things happen.” If you believe that there is a goblin in your closet that ties your shoes’ laces together, then this is a belief about how-the-world-is. Your shoes are real—you can pick them up. If there’s something out there that can reach out and tie your shoelaces together, it must be real too, part of the vast web of causes and effects we call the “universe.” Feeling angry at the goblin who tied your shoelaces involves a state of mind that is not just about how-the-world-is. Suppose that, as a Buddhist or a lobotomy patient or just a very phlegmatic person, finding your shoelaces tied together didn’t make you angry. This wouldn’t affect what you expected to see in the world—you’d still expect to open up your closet and find your shoelaces tied together. Your anger or calm shouldn’t affect your best guess here, because what happens in your closet does not depend on your emotional state of mind; though it may take some effort to think that clearly. But the angry feeling is tangled up with a state of mind that is about how-the-world-is; you become angry because you think the goblin tied your shoelaces. The criterion of rationality spreads virally, from the initial question of whether or not a goblin tied your shoelaces, to the resulting anger. Becoming more rational—arriving at better estimates of how-the-world-is—can diminish feelings or intensify them. Sometimes we run away from strong feelings by denying the facts, by flinching away from the view of the world that gave rise to the powerful emotion. If so, then as you study the skills of rational...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Building up to an Internal Family Systems model, published by Kaj_Sotala on the LessWrong. Introduction Internal Family Systems (IFS) is a psychotherapy school/technique/model which lends itself particularly well for being used alone or with a peer. For years, I had noticed that many of the kinds of people who put in a lot of work into developing their emotional and communication skills, some within the rationalist community and some outside it, kept mentioning IFS. So I looked at the Wikipedia page about the IFS model, and bounced off, since it sounded like nonsense to me. Then someone brought it up again, and I thought that maybe I should reconsider. So I looked at the WP page again, thought “nah, still nonsense”, and continued to ignore it. This continued until I participated in CFAR mentorship training last September, and we had a class on CFAR’s Internal Double Crux (IDC) technique. IDC clicked really well for me, so I started using it a lot and also facilitating it to some friends. However, once we started using it on more emotional issues (as opposed to just things with empirical facts pointing in different directions), we started running into some weird things, which it felt like IDC couldn’t quite handle. things which reminded me of how people had been describing IFS. So I finally read up on it, and have been successfully applying it ever since. In this post, I’ll try to describe and motivate IFS in terms which are less likely to give people in this audience the same kind of a “no, that’s nonsense” reaction as I initially had. Epistemic status This post is intended to give an argument for why something like the IFS model could be true and a thing that works. It’s not really an argument that IFS is correct. My reason for thinking in terms of IFS is simply that I was initially super-skeptical of it (more on the reasons of my skepticism later), but then started encountering things which it turned out IFS predicted - and I only found out about IFS predicting those things after I familiarized myself with it. Additionally, I now feel that IFS gives me significantly more gears for understanding the behavior of both other people and myself, and it has been significantly transformative in addressing my own emotional issues. Several other people who I know report it having been similarly powerful for them. On the other hand, aside for a few isolated papers with titles like “proof-of-concept” or “pilot study”, there seems to be conspicuously little peer-reviewed evidence in favor of IFS, meaning that we should probably exercise some caution. I think that, even if not completely correct, IFS is currently the best model that I have for explaining the observations that it’s pointing at. I encourage you to read this post in the style of learning soft skills - trying on this perspective, and seeing if there’s anything in the description which feels like it resonates with your experiences. But before we talk about IFS, let’s first talk about building robots. It turns out that if we put together some existing ideas from machine learning and neuroscience, we can end up with a robot design that pretty closely resembles IFS’s model of the human mind. What follows is an intentionally simplified story, which is simpler than either the full IFS model or a full account that would incorporate everything that I know about human brains. Its intent is to demonstrate that an agent architecture with IFS-style subagents might easily emerge from basic machine learning principles, without claiming that all the details of that toy model would exactly match human brains. A discussion of what exactly IFS does claim in the context of human brains follows after the robot story. Wanted: a robot which avoids catastrophes Suppose that we’re building a robot that we want to be generally intelligent. ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How An Algorithm Feels From Inside, published by Eliezer Yudkowsky on the LessWrong. "If a tree falls in the forest, and no one hears it, does it make a sound?" I remember seeing an actual argument get started on this subject—a fully naive argument that went nowhere near Berkeleyan subjectivism. Just: "It makes a sound, just like any other falling tree!" "But how can there be a sound that no one hears?" The standard rationalist view would be that the first person is speaking as if "sound" means acoustic vibrations in the air; the second person is speaking as if "sound" means an auditory experience in a brain. If you ask "Are there acoustic vibrations?" or "Are there auditory experiences?", the answer is at once obvious. And so the argument is really about the definition of the word "sound". I think the standard analysis is essentially correct. So let's accept that as a premise, and ask: Why do people get into such an argument? What's the underlying psychology? A key idea of the heuristics and biases program is that mistakes are often more revealing of cognition than correct answers. Getting into a heated dispute about whether, if a tree falls in a deserted forest, it makes a sound, is traditionally considered a mistake. So what kind of mind design corresponds to that error? In Disguised Queries I introduced the blegg/rube classification task, in which Susan the Senior Sorter explains that your job is to sort objects coming off a conveyor belt, putting the blue eggs or "bleggs" into one bin, and the red cubes or "rubes" into the rube bin. This, it turns out, is because bleggs contain small nuggets of vanadium ore, and rubes contain small shreds of palladium, both of which are useful industrially. Except that around 2% of blue egg-shaped objects contain palladium instead. So if you find a blue egg-shaped thing that contains palladium, should you call it a "rube" instead? You're going to put it in the rube bin—why not call it a "rube"? But when you switch off the light, nearly all bleggs glow faintly in the dark. And blue egg-shaped objects that contain palladium are just as likely to glow in the dark as any other blue egg-shaped object. So if you find a blue egg-shaped object that contains palladium, and you ask "Is it a blegg?", the answer depends on what you have to do with the answer: If you ask "Which bin does the object go in?", then you choose as if the object is a rube. But if you ask "If I turn off the light, will it glow?", you predict as if the object is a blegg. In one case, the question "Is it a blegg?" stands in for the disguised query, "Which bin does it go in?". In the other case, the question "Is it a blegg?" stands in for the disguised query, "Will it glow in the dark?" Now suppose that you have an object that is blue and egg-shaped and contains palladium; and you have already observed that it is furred, flexible, opaque, and glows in the dark. This answers every query, observes every observable introduced. There's nothing left for a disguised query to stand for. So why might someone feel an impulse to go on arguing whether the object is really a blegg? Blegg3 This diagram from Neural Categories shows two different neural networks that might be used to answer questions about bleggs and rubes. Network 1 has a number of disadvantages—such as potentially oscillating/chaotic behavior, or requiring O(N2) connections—but Network 1's structure does have one major advantage over Network 2: Every unit in the network corresponds to a testable query. If you observe every observable, clamping every value, there are no units in the network left over. Network 2, however, is a far better candidate for being something vaguely like how the human brain works: It's fast, cheap, scalable—and has an extra dangling unit in the center, whose activation can still vary, even a...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Notes from "Don't Shoot the Dog", published by juliawise on the LessWrong. I just finished Karen Pryor’s “Don’t Shoot the Dog: the New Art of Teaching and Training.” Partly because a friend points out that it’s not on Audible and therefore she can’t possibly read it, here are the notes I took and some thoughts. It’s a quick, easy read. The book applies behavioral psychology to training animals and people. The author started off as a dolphin trainer at an aquarium park in the 1960s and moved on to horses, dogs, and her own children. There are a lot of anecdotes about how to train animals (apparently polar bears like raisins). At the time, training animals without violence was considered novel and maybe impossible. I read it as a parenting book since I don’t plan to train dogs, horses, or polar bears. It’s probably not the best guide to training dogs since a lot of it is about people, and not the best guide to training people since a lot is about animals. She’s written a bunch of other books about training dogs and cats. But this book is an entertaining overview of all of it. The specter of behaviorism I can understand not wanting to use behavioral methods on children; the idea can sound overly harsh or reductive. The thing is, we already reinforce behavior all the time, including bad behavior, often without meaning to. So you might as well notice what you’re doing. To people schooled in the humanistic tradition, the manipulation of human behavior by some sort of conscious technique seems incorrigibly wicked, in spite of the obvious fact that we all go around trying to manipulate one another’s behavior all the time, by whatever means come to hand. There are still people who shudder at the very name of Skinner, which conjures in their minds some amalgam of Brave New World, mind control, and electric shock. (B. F. Skinner in fact believed that punishment was not an effective learning tool, and that positive reinforcement was much better for teaching.) Pryor argues that behavioral training allows you to get good results more pleasantly than with other methods. She describes her daughter’s experience directing a play in high school: At the closing performance the drama coach told me that she’d been amazed to see that throughout rehearsals Gale never yelled at her cast. Student directors always yell, but Gale never yelled. ‘Of course not,’ I said without thinking, ‘she’s an animal trainer.’ From the look on the teacher’s face, I realized I’d said the wrong thing—her students were not animals! But of course all I meant was that Gale would know how to establish stimulus control without unnecessary escalation. Of course there are bad applications of behavioral training: “The psychological literature abounds with shaping programs that are so unimaginative, not to say ham-handed, that they constitute in my opinion cruel and unusual punishment.” I don’t know a lot about ABA (applied behavior analysis), which is one application of behaviorism. My understanding is that its bad applications are certainly cruel and ham-handed, although there also seem to be good applications. I think that even people opposed to ABA should be able to find a lot of useful material in this book. You’re already doing reinforcement training One point I think is underappreciated is that we all reinforce each other, and children train parents as well as the other way around. A child is tantruming in the store for candy. The parent gives in and lets the child have a candy bar. The tantruming is positively reinforced by the candy, but the more powerful event is that the parent is negatively reinforced for giving in, since the public tantrum, so aversive and embarrassing for the parent, actually stopped. It’s also easy to accidentally reinforce bad behavior. I recently read Beverly Cleary’s Beezus and Ramona w...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Precognition, published by jasoncrawford on the LessWrong. This is a linkpost for It’s almost impossible to predict the future. But it’s also unnecessary, because most people are living in the past. All you have to do is see the present before everyone else does. To be less pithy, but more clear: Most people are slow to notice and accept change. If you can just be faster than most people at seeing what’s going on, updating your model of the world, and reacting accordingly, it’s almost as good as seeing the future. We see this in the US with covid: The same people who didn’t realize that we all should be wearing masks, when they were life-saving, are now slow to realize/admit that we can stop wearing them. For a dramatic historical example (from The Making of the Atomic Bomb), take Leo Szilard’s observations of 1930s Germany: Adolf Hitler was appointed Chancellor of Germany on January 30, 1933. . In late March, Jewish judges and lawyers in Prussia and Bavaria were dismissed from practice. On the weekend of April 1, Julius Streicher directed a national boycott of Jewish businesses and Jews were beaten in the streets. “I took a train from Berlin to Vienna on a certain date, close to the first of April, 1933,” Szilard writes. “The train was empty. The same train the next day was overcrowded, was stopped at the frontier, the people had to get out, and everybody was interrogated by the Nazis. This just goes to show that if you want to succeed in this world you don’t have to be much cleverer than other people, you just have to be one day earlier.” How to be earlier 1. Independent thinking. If you only believe things that are accepted by the majority of people, then by definition you’ll always be behind the curve in a changing world. 2. Listen to other independent thinkers. You can’t pay attention to everything at once or evaluate every area. You can only be the first to realize something in a narrow domain in which you are an expert. But if you tune your intellectual radar to other independent thinkers, you can be in the first ~1% of people to realize a new fact. Seek them out, find them, and follow them. I was taking covid precautions in late February 2020, about three weeks ahead of official “lockdown” measures—but only because I was tuned in to the people who were six weeks ahead. But: 3. Distinguish independent thinkers from crackpots. Both are “contrarian”; only one has any hope of being right. This is an art, honed over decades. Pay attention to both the source’s evidence and their logic. Credentials are relevant, but they are neither necessary nor sufficient. 4. Read broadly; seek out and adopt concepts and frameworks that help you understand the world (e.g.: exponential growth, network effects, efficient frontiers). Finally: 5. Learn how to make decisions in the face of uncertainty. Even when you see the present earlier, you won’t see it with full clarity, nor will you be able to predict the future. You’ll just have a set of probabilities that are closer to reality than most people’s. To return to the covid example: in January/February 2020, even the people farthest ahead of the curve weren’t certain whether there would be a pandemic or how bad it would be. They just knew that the chances were double-digit percent, before it was even on most people’s radar. Find low-cost ways to avoid extreme downside, and low-investment opportunities for extreme upside. For example, when a pandemic might be starting, it makes sense to stock up on supplies, move meetings to phone calls, etc.—these are cheap insurance. In some fantasy worlds, there are superheroes with “pre-cognition”, able to see the immediate future. They’re always one step ahead. But since most people are a few steps behind reality, you don’t need pre-cognition—just independent thinking. Thanks for listening. to h...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Scientific Self-Help: The State of Our Knowledge, published by lukeprog on the LessWrong. Part of the sequence: The Science of Winning at Life Some have suggested that the Less Wrong community could improve readers' instrumental rationality more effectively if it first caught up with the scientific literature on productivity and self-help, and then enabled readers to deliberately practice self-help skills and apply what they've learned in real life. I think that's a good idea. My contribution today is a quick overview of scientific self-help: what professionals call "the psychology of adjustment." First I'll review the state of the industry and the scientific literature, and then I'll briefly summarize the scientific data available on three topics in self-help: study methods, productivity, and happiness. The industry and the literature As you probably know, much of the self-help industry is a sham, ripe for parody. Most self-help books are written to sell, not to help. Pop psychology may be more myth than fact. As Christopher Buckley (1999) writes, "The more people read [self-help books], the more they think they need them... [it's] more like an addiction than an alliance." Where can you turn for reliable, empirically-based self-help advice? A few leading therapeutic psychologists (e.g., Albert Ellis, Arnold Lazarus, Martin Seligman) have written self-help books based on decades of research, but even these works tend to give recommendations that are still debated, because they aren't yet part of settled science. Lifelong self-help researcher Clayton Tucker-Ladd wrote and updated Psychological Self-Help (pdf) over several decades. It's a summary of what scientists do and don't know about self-help methods (as of about 2003), but it's also more than 2,000 pages long, and much of it surveys scientific opinion rather than experimental results, because on many subjects there aren't any experimental results yet. The book is associated with an internet community of people sharing what does and doesn't work for them. More immediately useful is Richard Wiseman's 59 Seconds. Wiseman is an experimental psychologist and paranormal investigator who gathered together what little self-help research is part of settled science, and put it into a short, fun, and useful Malcolm Gladwell-ish book. The next best popular-level general self-help book is perhaps Martin Seligman's What You Can Change and What You Can't. Two large books rate hundreds of popular self-help books according to what professional psychologists think of them, and offer advice on how to choose self-help books. Unfortunately, this may not mean much because even professional psychologists very often have opinions that depart from the empirical data, as documented extensively by Scott Lilienfeld and others in Science and Pseudoscience in Clinical Psychology and Navigating the Mindfield. These two books are helpful in assessing what is and isn't known according to empirical research (rather than according to expert opinion). Lilienfeld also edits the useful journal Scientific Review of Mental Health Practice, and has compiled a list of harmful psychological treatments. Also see Nathan and Gorman's A Guide to Treatments That Work, Roth & Fonagy's What Works for Whom?, and, more generally, Stanovich's How to Think Straight about Psychology. Many self-help books are written as "one size fits all," but of course this is rarely appropriate in psychology, and this leads to reader disappointment (Norem & Chang, 2000). But psychologists have tested the effectiveness of reading particular problem-focused self-help books ("bibliotherapy").1 For example, it appears that reading David Burns' Feeling Good can be as effective for treating depression as individual or group therapy. Results vary from book to book. There are at least fou...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A voting theory primer for rationalists, published by Jameson Quinn on the LessWrong. What is voting theory? Voting theory, also called social choice theory, is the study of the design and evaulation of democratic voting methods (that's the activists' word; game theorists call them "voting mechanisms", engineers call them "electoral algorithms", and political scientists say "electoral formulas"). In other words, for a given list of candidates and voters, a voting method specifies a set of valid ways to fill out a ballot, and, given a valid ballot from each voter, produces an outcome. (An "electoral system" includes a voting method, but also other implementation details, such as how the candidates and voters are validated, how often elections happen and for what offices, etc. "Voting system" is an ambiguous term that can refer to a full electoral system, just to the voting method, or even to the machinery for counting votes.) Most voting theory limits itself to studying "democratic" voting methods. That typically has both empirical and normative implications. Empirically, "democratic" means: There are many voters There can be more than two candidates In order to be considered "democratic", voting methods generally should meet various normative criteria as well. There are many possible such criteria, and on many of them theorists do not agree; but in general they do agree on this minimal set: Anonymity; permuting the ballots does not change the probability of any election outcome. Neutrality; permuting the candidates on all ballots does not change the probability of any election outcome. Unanimity: If voters universally vote a preference for a given outcome over all others, that outcome is selected. (This is a weak criterion, and is implied by many other stronger ones; but those stronger ones are often disputed, while this one rarely is.) Methods typically do not directly involve money changing hands or other enduring state-changes for individual voters. (There can be exceptions to this, but there are good reasons to want to understand "moneyless" elections.) Why is voting theory important for rationalists? First off, because democratic processes in the real world are important loci of power. That means that it's useful to understand the dynamics of the voting methods used in such real-world elections. Second, because these real-world democratic processes have all been created and/or evolved in the past, and so there are likely to be opportunities to replace, reform, or add to them in the future. If you want to make political change of any kind over a medium-to-long time horizon, these systemic reforms should probably be part of your agenda. The fact is that FPTP, the voting method we use in most of the English-speaking world, is absolutely horrible, and there is reason to believe that reforming it would substantially (though not of course completely) alleviate much political dysfunction and suffering. Third, because understanding social choice theory helps clarify ideas about how it's possible and/or desirable to resolve value disputes between multiple agents. For instance, if you believe that superintelligences should perform a "values handshake" when meeting, replacing each of their individual value functions by some common one so as to avoid the dead weight loss of a conflict, then social choice theory suggests both questions and answers about what that might look like. (Note that the ethical and practical importance of such considerations is not at all limited to "post-singularity" examples like that one.) In fact, on that third point: my own ideas of ethics and of fun theory are deeply informed by my decades of interest in voting theory. To simplify into a few words my complex thoughts on this, I believe that voting theory elucidates "ethical incompleteness" (tha...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Flinching away from truth” is often about protecting the epistemology, published by AnnaSalamon on the LessWrong. Related to: Leave a line of retreat; Categorizing has consequences. There’s a story I like, about this little kid who wants to be a writer. So she writes a story and shows it to her teacher. “You misspelt the word ‘ocean’”, says the teacher. “No I didn’t!”, says the kid. The teacher looks a bit apologetic, but persists: “‘Ocean’ is spelt with a ‘c’ rather than an ‘sh’; this makes sense, because the ‘e’ after the ‘c’ changes its sound.” “No I didn’t!” interrupts the kid. “Look,” says the teacher, “I get it that it hurts to notice mistakes. But that which can be destroyed by the truth should be! You did, in fact, misspell the word ‘ocean’.” “I did not!” says the kid, whereupon she bursts into tears, and runs away and hides in the closet, repeating again and again: “I did not misspell the word! I can too be a writer!”. I like to imagine the inside of the kid’s head as containing a single bucket that houses three different variables that are initially all stuck together: Original state of the kid's head: The goal, if one is seeking actual true beliefs, is to separate out each of these variables into its own separate bucket, so that the “is ‘oshun’ spelt correctly?” variable can update to the accurate state of "no", without simultaneously forcing the "Am I allowed to pursue my writing ambition?" variable to update to the inaccurate state of "no". Desirable state (requires somehow acquiring more buckets): The trouble is, the kid won’t necessarily acquire enough buckets by trying to “grit her teeth and look at the painful thing”. A naive attempt to "just refrain from flinching away, and form true beliefs, however painful" risks introducing a more important error than her current spelling error: mistakenly believing she must stop working toward being a writer, since the bitter truth is that she spelled 'oshun' incorrectly. State the kid might accidentally land in, if she naively tries to "face the truth": (You might take a moment, right now, to name the cognitive ritual the kid in the story should do (if only she knew the ritual). Or to name what you think you'd do if you found yourself in the kid's situation -- and how you would notice that you were at risk of a "buckets error".) More examples: It seems to me that bucket errors are actually pretty common, and that many (most?) mental flinches are in some sense attempts to avoid bucket errors. The following examples are slightly-fictionalized composites of things I suspect happen a lot (except the "me" ones; those are just literally real): Diet: Adam is on a diet with the intent to lose weight. Betty starts to tell him about some studies suggesting that the diet he is on may cause health problems. Adam complains: “Don’t tell me this! I need to stay motivated!” One interpretation, as diagramed above: Adam is at risk of accidentally equating the two variables, and accidentally assuming that the studies imply that the diet must stop being viscerally motivating. He semi-consciously perceives that this risks error, and so objects to having the information come in and potentially force the error. Pizza purchase: I was trying to save money. But I also wanted pizza. So I found myself tempted to buy the pizza really quickly so that I wouldn't be able to notice that it would cost money (and, thus, so I would be able to buy the pizza): On this narration: It wasn't necessarily a mistake to buy pizza today. Part of me correctly perceived this "not necessarily a mistake to buy pizza" state. Part of me also expected that the rest of me wouldn't perceive this, and that, if I started thinking it through, I might get locked into the no-pizza state even if pizza was better. So it tried to 'help' by buying the pizza really quickly, ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is:Raising the Sanity Waterline, published by Eliezer Yudkowskyon the LessWrong. To paraphrase the Black Belt Bayesian: Behind every exciting, dramatic failure, there is a more important story about a larger and less dramatic failure that made the first failure possible. If every trace of religion was magically eliminated from the world tomorrow, then—however much improved the lives of many people would be—we would not even have come close to solving the larger failures of sanity that made religion possible in the first place. We have good cause to spend some of our efforts on trying to eliminate religion directly, because it is a direct problem. But religion also serves the function of an asphyxiated canary in a coal mine—religion is a sign, a symptom, of larger problems that don't go away just because someone loses their religion. Consider this thought experiment—what could you teach people that is not directly about religion, which is true and useful as a general method of rationality, which would cause them to lose their religions? In fact—imagine that we're going to go and survey all your students five years later, and see how many of them have lost their religions compared to a control group; if you make the slightest move at fighting religion directly, you will invalidate the experiment. You may not make a single mention of religion or any religious belief in your classroom, you may not even hint at it in any obvious way. All your examples must center about real-world cases that have nothing to do with religion. If you can't fight religion directly, what do you teach that raises the general waterline of sanity to the point that religion goes underwater? Here are some such topics I've already covered—not avoiding all mention of religion, but it could be done: Affective Death Spirals—plenty of non-supernaturalist examples. How to avoid cached thoughts and fake wisdom; the pressure of conformity. Evidence and Occam's Razor—the rules of probability. The Bottom Line / Engines of Cognition—the causal reasons why Reason works. Mysterious Answers to Mysterious Questions—and the whole associated sequence, like making beliefs pay rent and curiosity-stoppers—have excellent historical examples in vitalism and phlogiston. Non-existence of ontologically fundamental mental things—apply the Mind Projection Fallacy to probability, move on to reductionism versus holism, then brains and cognitive science. The many sub-arts of Crisis of Faith—though you'd better find something else to call this ultimate high master-level technique of actually updating on evidence. Dark Side Epistemology—teaching this with no mention of religion would be hard, but perhaps you could videotape the interrogation of some snake-oil sales agent as your real-world example. Fun Theory—teach as a literary theory of utopian fiction, without the direct application to theodicy. Joy in the Merely Real, naturalistic metaethics, etcetera etcetera etcetera and so on. But to look at it another way Suppose we have a scientist who's still religious, either full-blown scriptural-religion, or in the sense of tossing around vague casual endorsements of "spirituality". We now know this person is not applying any technical, explicit understanding of... ...what constitutes evidence and why; ...Occam's Razor; ...how the above two rules derive from the lawful and causal operation of minds as mapping engines, and do not switch off when you talk about tooth fairies; ...how to tell the difference between a real answer and a curiosity-stopper; ...how to rethink matters for themselves instead of just repeating things they heard; ...certain general trends of science over the last three thousand years; ...the difficult arts of actually updating on new evidence and relinquishing old beliefs; ...epistemology 101; ...self-honesty 201; .....
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Efficient Charity: Do Unto Others..., published by Scott Alexander on the LessWrong. This was originally posted as part of the efficient charity contest back in November. Thanks to Roko, multifoliaterose, Louie, jmmcd, jsalvatier, and others I forget for help, corrections, encouragement, and bothering me until I finally remembered to post this here. Imagine you are setting out on a dangerous expedition through the Arctic on a limited budget. The grizzled old prospector at the general store shakes his head sadly: you can't afford everything you need; you'll just have to purchase the bare essentials and hope you get lucky. But what is essential? Should you buy the warmest parka, if it means you can't afford a sleeping bag? Should you bring an extra week's food, just in case, even if it means going without a rifle? Or can you buy the rifle, leave the food, and hunt for your dinner? And how about the field guide to Arctic flowers? You like flowers, and you'd hate to feel like you're failing to appreciate the harsh yet delicate environment around you. And a digital camera, of course - if you make it back alive, you'll have to put the Arctic expedition pics up on Facebook. And a hand-crafted scarf with authentic Inuit tribal patterns woven from organic fibres! Wicked! ...but of course buying any of those items would be insane. The problem is what economists call opportunity costs: buying one thing costs money that could be used to buy others. A hand-crafted designer scarf might have some value in the Arctic, but it would cost so much it would prevent you from buying much more important things. And when your life is on the line, things like impressing your friends and buying organic pale in comparison. You have one goal - staying alive - and your only problem is how to distribute your resources to keep your chances as high as possible. These sorts of economics concepts are natural enough when faced with a journey through the freezing tundra. But they are decidedly not natural when facing a decision about charitable giving. Most donors say they want to "help people". If that's true, they should try to distribute their resources to help people as much as possible. Most people don't. In the "Buy A Brushstroke" campaign, eleven thousand British donors gave a total of £550,000 to keep the famous painting "Blue Rigi" in a UK museum. If they had given that £550,000 to buy better sanitation systems in African villages instead, the latest statistics suggest it would have saved the lives of about one thousand two hundred people from disease. Each individual $50 donation could have given a year of normal life back to a Third Worlder afflicted with a disabling condition like blindness or limb deformity.. Most of those 11,000 donors genuinely wanted to help people by preserving access to the original canvas of a beautiful painting. And most of those 11,000 donors, if you asked, would say that a thousand people's lives are more important than a beautiful painting, original or no. But these people didn't have the proper mental habits to realize that was the choice before them, and so a beautiful painting remains in a British museum and somewhere in the Third World a thousand people are dead. If you are to "love your neighbor as yourself", then you should be as careful in maximizing the benefit to others when donating to charity as you would be in maximizing the benefit to yourself when choosing purchases for a polar trek. And if you wouldn't buy a pretty picture to hang on your sled in preference to a parka, you should consider not helping save a famous painting in preference to helping save a thousand lives. Not all charitable choices are as simple as that one, but many charitable choices do have right answers. GiveWell.org, a site which collects and interprets data on the effectiveness ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 2018 AI Alignment Literature Review and Charity Comparison, published by Larkson the LessWrong. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. Cross-posted to the EA forum. Introduction Like last year and the year before, I’ve attempted to review the research that has been produced by various organisations working on AI safety, to help potential donors gain a better understanding of the landscape. This is a similar role to that which GiveWell performs for global health charities, and somewhat similar to an securities analyst with regards to possible investments. It appears that once again no-one else has attempted to do this, to my knowledge, so I've once again undertaken the task. This year I have included several groups not covered in previous years, and read more widely in the literature. My aim is basically to judge the output of each organisation in 2018 and compare it to their budget. This should give a sense for the organisations' average cost-effectiveness. We can also compare their financial reserves to their 2019 budgets to get a sense of urgency. Note that this document is quite long, so I encourage you to just read the sections that seem most relevant to your interests, probably the sections about the individual organisations. I do not recommend you skip to the conclusions! I’d like to apologize in advance to everyone doing useful AI Safety work whose contributions I may have overlooked or misconstrued. Methodological Considerations Track Records Judging organisations on their historical output is naturally going to favour more mature organisations. A new startup, whose value all lies in the future, will be disadvantaged. However, I think that this is correct. The newer the organisation, the more funding should come from people with close knowledge. As organisations mature, and have more easily verifiable signals of quality, their funding sources can transition to larger pools of less expert money. This is how it works for startups turning into public companies and I think the same model applies here. This judgement involves analysing a large number papers relating to Xrisk that were produced during 2018. Hopefully the year-to-year volatility of output is sufficiently low that this is a reasonable metric. I also attempted to include papers during December 2017, to take into account the fact that I'm missing the last month's worth of output from 2017, but I can't be sure I did this successfully. This article focuses on AI risk work. If you think other causes are important too, your priorities might differ. This particularly affects GCRI, FHI and CSER, who both do a lot of work on other issues. We focus on papers, rather than outreach or other activities. This is partly because they are much easier to measure; while there has been a large increase in interest in AI safety over the last year, it’s hard to work out who to credit for this, and partly because I think progress has to come by persuading AI researchers, which I think comes through technical outreach and publishing good work, not popular/political work. Politics My impression is that policy on technical subjects (as opposed to issues that attract strong views from the general population) is generally made by the government and civil servants in consultation with, and being lobbied by, outside experts and interests. Without expert (e.g. top ML researchers at Google, CMU & Baidu) consensus, no useful policy will be enacted. Pushing directly for policy seems if anything likely to hinder expert consensus. Attempts to directly influence the government to regulate AI research seem very adversarial, and risk being pattern-matched to ignorant opposition to GM foods or nuclear power. We don't want the 'us-vs-them' situation, that has occurred with climate change, to h...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Scholarship: How to DoIt Efficiently, published by lukeprog on the LessWrong. Scholarship is an important virtue of rationality, but it can be costly. Its major costs are time and effort. Thus, if you can reduce the time and effort required for scholarship - if you can learn to do scholarship more efficiently - then scholarship will be worth your effort more often than it previously was. As an autodidact who now consumes whole fields of knowledge in mere weeks, I've developed efficient habits that allow me to research topics quickly. I'll share my research habits with you now. Review articles and textbooks are king My first task is to find scholarly review (or 'survey') articles on my chosen topic from the past five years (the more recent, the better). A good review article provides: An overview of the subject matter of the field and the terms being used (for scholarly googling later). An overview of the open and solved problems in the field, and which researchers are working on them. Pointers to the key studies that give researchers their current understanding of the topic. If you can find a recent scholarly edited volume of review articles on the topic, then you've hit the jackpot. (Edited volumes are better than single-author volumes, because when starting out you want to avoid reading only one particular researcher's perspective.) Examples from my own research of just this year include: Affective neuroscience: Pleasures of the Brain (2009) Neuroeconomics: Decision Making and the Brain (2008) Dual process theories of psychology: In Two Minds (2009) Intuition and unconscious learning: Intuition in Judgment and Decision Making (2007) Goals: The Psychology of Goals (2009) Catastrophic risks: Global Catastrophic Risks (2008) If the field is large enough, there may exist an edited 'Handbook' on the subject, which is basically just a very large scholarly edited volume of review articles. Examples: Oxford Handbook of Evolutionary Psychology (2007), Oxford Handbook of Positive Psychology (2009), Oxford Handbook of Philosophy and Neuroscience (2009), Handbook of Developmental Cognitive Neuroscience (2008), Oxford Handbook of Neuroethics (2011), Handbook of Relationship Intitiation (2008), and Handbook of Implicit Social Cognition (2010). For the humanities, see the Blackwell Companions and Cambridge Companions. If your questions are basic enough, a recent entry-level textbook on the subject may be just as good. Textbooks are basically book-length review articles written for undergrads. Textbooks I purchased this year include: Evolutionary Psychology: The New Science of Mind, 4th edition (2011) Artificial Intelligence: A Modern Approach, 3rd edition (2009) Psychology Applied to Modern Life, 10th edition (2011) Psychology, 9th edition (2009) Use Google Books and Amazon's 'Look Inside' feature to see if the books appear to be of high quality, and likely to answer the questions you have. Also check the textbook recommendations here. You can save money by checking Library Genesis and library.nu for a PDF copy first, or by buying used books, or by buying ebook versions from Amazon, B&N, or Google. Keep in mind that if you take the virtue of scholarship seriously, you may need to change how you think about the cost of obtaining knowledge. Purchasing the right book can save you dozens of hours of research. Because a huge part of my life these days is devoted to scholarship, a significant portion of my monthly budget is set aside for purchasing knowledge. So far this year I've averaged over $150/mo spent on textbooks and scholarly edited volumes. Recent scholarly review articles can also be found on Google scholar. Search for key terms, and review articles will often be listed near the top of the results because review articles are cited widely. For example, result #9 on Google sch...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How to Be Happy, published by lukeprog on the LessWrong. Part of the sequence: The Science of Winning at Life One day a coworker said to me, "Luke! You're, like, the happiest person I know! How come you're so happy all the time?" It was probably a rhetorical question, but I had a very long answer to give. See, I was unhappy for most of my life,1 and even considered suicide a few times. Then I spent two years studying the science of happiness. Now, happiness is my natural state. I can't remember the last time I felt unhappy for longer than 20 minutes. That kind of change won't happen for everyone, or even most people (beware of other-optimizing), but it's worth a shot! We all want to be happy, and happiness is useful for other things, too.2 For example, happiness improves physical health,3 improves creativity,4 and even enables you to make better decisions.5 (It's harder to be rational when you're unhappy.6) So, as part of a series on how to win at life with science and rationality, let's review the science of happiness. The correlates of happiness Earlier, I noted that there is an abundance of research on factors that correlate with subjective well-being (individuals' own assessments of their happiness and life satisfaction). Factors that don't correlate much with happiness include: age,7 gender,8 parenthood,9 intelligence,10 physical attractiveness,11 and money12 (as long as you're above the poverty line). Factors that correlate moderately with happiness include: health,13 social activity,14 and religiosity.15 Factors that correlate strongly with happiness include: genetics,16 love and relationship satisfaction,17 and work satisfaction.18 But correlation is not enough. We want to know what causes happiness. And that is a trickier thing to measure. But we do know a few things. Happiness, personality, and skills Genes account for about 50% of the variance in happiness.19 Even lottery winners and newly-made quadriplegics do not see as much of a change in happiness as you would expect.20 Presumably, genes shape your happiness by shaping your personality, which is known to be quite heritable.21 So which personality traits tend to correlate most with happiness? Extroversion is among the best predictors of happiness,22 as are conscientiousness, agreeableness, self-esteem, and optimism.23 What if you don't have those traits? The first thing to say is that you might be capable of them without knowing it. Introversion, for example, can be exacerbated by a lack of social skills. If you decide to learn and practice social skills, you might find that you are more extroverted than you thought! (That's what happened to me.) The same goes for conscientiousness, agreeableness, self-esteem, and optimism - these are only partly linked to personality. They are to some extent learnable skills, and learning these skills (or even "acting as if") can increase happiness.24 The second thing to say is that lacking some of these traits does not, of course, doom you to unhappiness. Happiness is subjective and relative Happiness is not determined by objective factors, but by how you feel about them.25 Happiness is also relative26: you'll probably be happier making $25,000/yr in Costa Rica (where your neighbors are making $13,000/yr) than you will be making $80,000/yr in Beverly Hills (where your neighbors are making $130,000/yr). Happiness is relative in another sense, too: it is relative to your expectations.27 We are quite poor at predicting the strength of our emotional reactions to future events. We overestimate the misery we will experience after a romantic breakup, failure to get a promotion, or even contracting an illness. We also overestimate the pleasure we will get from buying a nice car, getting a promotion, or moving to a lovely coastal city. So: lower your expectations about the ple...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The curse of identity, published by Kaj_Sotala on the LessWrong. So what you probably mean is, "I intend to do school to improve my chances on the market". But this statement is still false, unless it is also true that "I intend to improve my chances on the market". Do you, in actual fact, intend to improve your chances on the market? I expect not. Rather, I expect that your motivation is to appear to be the sort of person who you think you would be if you were ambitiously attempting to improve your chances on the market... which is not really motivating enough to actually DO the work. However, by persistently trying to do so, and presenting yourself with enough suffering at your failure to do it, you get to feel as if you are that sort of person without having to actually do the work. This is actually a pretty optimal solution to the problem, if you think about it. (Or rather, if you DON'T think about it!) -- PJ Eby I have become convinced that problems of this kind are the number one problem humanity has. I'm also pretty sure that most people here, no matter how much they've been reading about signaling, still fail to appreciate the magnitude of the problem. Here are two major screw-ups and one narrowly averted screw-up that I've been guilty of. See if you can find the pattern. When I began my university studies back in 2006, I felt strongly motivated to do something about Singularity matters. I genuinely believed that this was the most important thing facing humanity, and that it needed to be urgently taken care of. So in order to become able to contribute, I tried to study as much as possible. I had had troubles with procrastination, and so, in what has to be one of the most idiotic and ill-thought-out acts of self-sabotage possible, I taught myself to feel guilty whenever I was relaxing and not working. Combine an inability to properly relax with an attempted course load that was twice the university's recommended pace, and you can guess the results: after a year or two, I had an extended burnout that I still haven't fully recovered from. I ended up completing my Bachelor's degree in five years, which is the official target time for doing both your Bachelor's and your Master's. A few years later, I became one of the founding members of the Finnish Pirate Party, and on the basis of some writings the others thought were pretty good, got myself elected as the spokesman. Unfortunately – and as I should have known before taking up the post – I was a pretty bad choice for this job. I'm good at expressing myself in writing, and when I have the time to think. I hate talking with strangers on the phone, find it distracting to look people in the eyes when I'm talking with them, and have a tendency to start a sentence over two or three times before hitting on a formulation I like. I'm also bad at thinking quickly on my feet and coming up with snappy answers in live conversation. The spokesman task involved things like giving quick statements to reporters ten seconds after I'd been woken up by their phone call, and live interviews where I had to reply to criticisms so foreign to my thinking that they would never have occurred to me naturally. I was pretty terrible at the job, and finally delegated most of it to other people until my term ran out – though not before I'd already done noticeable damage to our cause. Last year, I was a Visiting Fellow at the Singularity Institute. At one point, I ended up helping Eliezer in writing his book. Mostly this involved me just sitting next to him and making sure he did get writing done while I surfed the Internet or played a computer game. Occasionally I would offer some suggestion if asked. Although I did not actually do much, the multitasking required still made me unable to spend this time productively myself, and for some reason i...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is:Embedded Agents, published by abramdemski, Scott Garrabrant on the LessWrong. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. (A longer text-based version of this post is also available on MIRI's blog here, and the bibliography for the whole sequence can be found here) Thanks for listening. to help us out with the nonlinear library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Eliezer Yudkowsky, published by Eliezer Yudkowsky on the LessWrong. At the Singularity Summit 2007, one of the speakers called for democratic, multinational development of artificial intelligence. So I stepped up to the microphone and asked: Suppose that a group of democratic republics form a consortium to develop AI, and there’s a lot of politicking during the process—some interest groups have unusually large influence, others get shafted—in other words, the result looks just like the products of modern democracies. Alternatively, suppose a group of rebel nerds develops an AI in their basement, and instructs the AI to poll everyone in the world—dropping cellphones to anyone who doesn’t have them—and do whatever the majority says. Which of these do you think is more “democratic,” and would you feel safe with either? I wanted to find out whether he believed in the pragmatic adequacy of the democratic political process, or if he believed in the moral rightness of voting. But the speaker replied: The first scenario sounds like an editorial in Reason magazine, and the second sounds like a Hollywood movie plot. Confused, I asked: Then what kind of democratic process did you have in mind? The speaker replied: Something like the Human Genome Project—that was an internationally sponsored research project. I asked: How would different interest groups resolve their conflicts in a structure like the Human Genome Project? And the speaker said: I don’t know. This exchange puts me in mind of a quote from some dictator or other, who was asked if he had any intentions to move his pet state toward democracy: We believe we are already within a democratic system. Some factors are still missing, like the expression of the people’s will. The substance of a democracy is the specific mechanism that resolves policy conflicts. If all groups had the same preferred policies, there would be no need for democracy—we would automatically cooperate. The resolution process can be a direct majority vote, or an elected legislature, or even a voter-sensitive behavior of an artificial intelligence, but it has to be something. What does it mean to call for a “democratic” solution if you don’t have a conflict-resolution mechanism in mind? I think it means that you have said the word “democracy,” so the audience is supposed to cheer. It’s not so much a propositional statement or belief, as the equivalent of the “Applause” light that tells a studio audience when to clap. This case is remarkable only in that I mistook the applause light for a policy suggestion, with subsequent embarrassment for all. Most applause lights are much more blatant, and can be detected by a simple reversal test. For example, suppose someone says: We need to balance the risks and opportunities of AI. If you reverse this statement, you get: We shouldn’t balance the risks and opportunities of AI. Since the reversal sounds abnormal, the unreversed statement is probably normal, implying it does not convey new information. There are plenty of legitimate reasons for uttering a sentence that would be uninformative in isolation. “We need to balance the risks and opportunities of AI” can introduce a discussion topic; it can emphasize the importance of a specific proposal for balancing; it can criticize an unbalanced proposal. Linking to a normal assertion can convey new information to a bounded rationalist—the link itself may not be obvious. But if no specifics follow, the sentence is probably an applause light. I am tempted to give a talk sometime that consists of nothing but applause lights, and see how long it takes for the audience to start laughing: I am here to propose to you today that we need to balance the risks and opportunities of advanced artificial intelligence. We should avoid the risks and, insofar as it is possible, realize t...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Takeaways from one year of lockdown, published by mingyuan on the LessWrong. As of today, I've been in full-on, hardcore lockdown for an entire year. I have a lot of feelings – both about the personal social impacts of lockdown and about society being broken – that I won't go into in this public space. What I want to figure out in this post is what rationality-relevant lessons I can draw from what happened in my life this past year. (Meta: This post is not well-written and is mostly bullet points, because the first few versions I wrote were unusable but I still wanted to publish it today.) Observations Some facts about my lockdown: I have spent 99.9% of the year within 1 mile of my house Up until last month I had spent the entire year within 10 miles of my house Between February 29th and June 15th of 2020, I did not set foot outside of my front gate I have not gotten COVID, nor has anyone in my bubble I have incurred an average of 0 microCOVIDs per week The absolute riskiest thing I've done this whole time cost ~20 microCOVIDs I can only remember talking to a friend not-in-my-bubble, in person, twice Some observations about other people with similar levels of caution: Almost no one I know has caught COVID, even though Zvi estimates that ~25% of Americans have had it (the official confirmed rate is 10%). I know of only one person who caught it while taking serious precautions, and I know a few hundred people about as well as I know this person. (see also) I was recently tracking down a reference in the Sequences and found that the author was so afraid of COVID that he failed to seek medical care for appendicitis and died of sepsis. On negotiations: A blanket heuristic of "absolutely no interactions outside of the household" makes decisions simple but is very costly in other ways microCOVID spreadsheets are useful but fairly high-effort I went on a date once. The COVID negotiations with my house were so stressful that I had a migraine for a week afterwards. On hopelessness: I spent a fair amount of time trying to get vaccinated early, and failed. I now appear to have a belief that I will never succeed at getting vaccinated; and further that other people can succeed but I never can. Related: My system 1 believes that lockdown will last forever. Also that vaccines aren't real – not that they don't work, but that they're a lovely dream, like unicorns or God, that ultimately turns out to be a lie. A vaccine cannot cause me to leave lockdown because lockdown is an eternal, all-consuming metaphysical state. I would have liked to be dating this year, but the first date and the surrounding ~week of house discussion was so stressful that I gave up on dates entirely after that. I notice that I feel the lack of friendships, but wasn't motivated enough about any particular friendship to put in the effort to make it work despite the situation. By contrast, some people I know did do this and have benefited a lot. My house had ~3-hour meetings ~3 times a week at the very beginning of the pandemic, where people did math on the board and talked about their feelings and we tried to figure out what to do. In retrospect, this burned me out so much that I gave up on trying to figure anything out and defaulted to an absolutely-zero-risk strategy, because at least that was simple. The fact that SlateStarCodex went down at the same time everything else in life went to shit destroyed my soul. Oops I have started talking about feelings and will now stop. Taking all these observations together, it's clear to me that my social group has been insanely overcautious, to our great detriment. I think this has been obvious for quite a while, but I didn't and still don't know how to act on that information. It seems like extreme caution made sense at first, when we didn't know much. And by the time we k...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Another (outer) alignment failure story, published by paulfchristiano on the LessWrong. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. Meta This is a story where the alignment problem is somewhat harder than I expect, society handles AI more competently than I expect, and the outcome is worse than I expect. It also involves inner alignment turning out to be a surprisingly small problem. Maybe the story is 10-20th percentile on each of those axes. At the end I’m going to go through some salient ways you could vary the story. This isn’t intended to be a particularly great story (and it’s pretty informal). I’m still trying to think through what I expect to happen if alignment turns out to be hard, and this more like the most recent entry in a long journey of gradually-improving stories. I wrote this up a few months ago and was reminded to post it by Critch’s recent post (which is similar in many ways). This story has definitely been shaped by a broader community of people gradually refining failure stories rather than being written in a vacuum. I’d like to continue spending time poking at aspects of this story that don’t make sense, digging into parts that seem worth digging into, and eventually developing clearer and more plausible stories. I still think it’s very plausible that my views about alignment will change in the course of thinking concretely about stories, and even if my basic views about alignment stay the same it’s pretty likely that the story will change. Story ML starts running factories, warehouses, shipping, and construction. ML assistants help write code and integrate ML into new domains. ML designers help build factories and the robots that go in them. ML finance systems invest in companies on the basis of complicated forecasts and (ML-generated) audits. Tons of new factories, warehouses, power plants, trucks and roads are being built. Things are happening quickly, investors have super strong FOMO, no one really knows whether it’s a bubble but they can tell that e.g. huge solar farms are getting built and something is happening that they want a piece of. Defense contractors are using ML systems to design new drones, and ML is helping the DoD decide what to buy and how to deploy it. The expectation is that automated systems will manage drones during high-speed ML-on-ML conflicts because humans won’t be able to understand what’s going on. ML systems are designing new ML systems, testing variations, commissioning giant clusters. The financing is coming from automated systems, the clusters are built by robots. A new generation of fabs is being built with unprecedented speed using new automation. At this point everything kind of makes sense to humans. It feels like we are living at the most exciting time in history. People are making tons of money. The US defense establishment is scared because it has no idea what a war is going to look like right now, but in terms of policy their top priority is making sure the boom proceeds as quickly in the US as it does in China because it now seems plausible that being even a few years behind would result in national irrelevance. Things are moving very quickly and getting increasingly hard for humans to evaluate. We can no longer train systems to make factory designs that look good to humans, because we don’t actually understand exactly what robots are doing in those factories or why; we can’t evaluate the tradeoffs between quality and robustness and cost that are being made; we can't really understand the constraints on a proposed robot design or why one design is better than another. We can’t evaluate arguments about investments very well because they come down to claims about where the overall economy is going over the next 6 months that seem kind of alien (even the more reco...
Welcome to the nonlinear library, where we use text-to-speech software to convert the best writing from the rationalist and ea communities into audio. This is: Working With Monsters, published by johnswentworth on the LessWrong. This is a fictional piece based on Sort By Controversial. You do not need to read that first, though it may make Scissor Statements feel more real. Content Warning: semipolitical. Views expressed by characters in this piece are not necessarily the views of the author. I stared out at a parking lot, the pavement cracked and growing grass. A few cars could still be seen, every one with a shattered windshield or no tires or bashed-in roof, one even laying on its side. Of the buildings in sight, two had clearly burned, only blackened reinforced concrete skeletons left behind. To the left, an overpass had collapsed. To the right, the road was cut by a hole four meters across. Everywhere, trees and vines climbed the remains of the small city. The collapsed ceilings and shattered windows and nests of small animals in the once-hospital behind me seemed remarkably minor damage, relatively speaking. Eighty years of cryonic freeze, and I woke to a post-apocalyptic dystopia. “It’s all like that,” said a voice behind me. One of my. rescuers? Awakeners. He went by Red. “Whole world’s like that.” “What happened?” I asked. “Bioweapon?” “Scissor,” replied a woman, walking through the empty doorway behind Red. Judge, he’d called her earlier. I raised an eyebrow, and waited for elaboration. Apparently they expected a long conversation - both took a few seconds to get comfortable, Red leaning up against the wall in a patch of shade, Judge righting an overturned bench to sit on. It was Red who took up the conversation thread. “Let’s start with an ethical question,” he began, then laid out a simple scenario. “So,” he asked once finished, “blue or green?”. “Blue,” I replied. “Obviously. Is this one of those things where you try to draw an analogy from this nice obvious case to a more complicated one where it isn’t so obvious?” “No,” Judge cut in, “It’s just that question. But you need some more background.” “There was a writer in your time who coined the term ‘scissor statement’,” Red explained, “It’s a statement optimized to be as controversial as possible, to generate maximum conflict. To get a really powerful scissor, you need AI, but the media environment of your time was already selecting for controversy in order to draw clicks.” “Oh no,” I said, “I read about that. and the question you asked, green or blue, it seems completely obvious, like anyone who’d say green would have to be trolling or delusional or a threat to society or something. but that’s exactly how scissor statements work.” “Exactly,” replied Judge. “The answer seems completely obvious to everyone, yet people disagree about which answer is obviously-correct. And someone with the opposite answer seems like a monster, a threat to the world, like a serial killer or a child torturer or a war criminal. They need to be put down for the good of society.” I hesitated. I knew I shouldn’t ask, but. “So, you two.” Judge casually shifted position, placing a hand on some kind of weapon on her belt. I glanced at Red, and only then noticed that his body was slightly tensed, as if ready to run. Or fight. “I’m a blue, same as you,” said Judge. Then she pointed to Red. “He’s a green.” I felt a wave of disbelief, then disgust, then fury. It was so wrong, how could anyone even consider green... I took a step toward him, intent on punching his empty face even if I got shot in the process. “Stop,” said Judge, “unless you want to get tazed.” She was holding her weapon aimed at me, now. Red hadn’t moved. If he had, I’d probably have charged him. But Judge wasn’t the monster here. wait. I turned to Judge, and felt a different sort of anger. “How can you just stand there?”, I asked. “You know that he’s in the wrong, that he’s a monster, that he deserves to be put down, preferabl...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Policy Debates Should Not Appear One-Sided, published by Eliezer Yudkowsky on the LessWrong. Robin Hanson proposed stores where banned products could be sold.1 There are a number of excellent arguments for such a policy—an inherent right of individual liberty, the career incentive of bureaucrats to prohibit everything, legislators being just as biased as individuals. But even so (I replied), some poor, honest, not overwhelmingly educated mother of five children is going to go into these stores and buy a “Dr. Snakeoil’s Sulfuric Acid Drink” for her arthritis and die, leaving her orphans to weep on national television. I was just making a factual observation. Why did some people think it was an argument in favor of regulation? On questions of simple fact (for example, whether Earthly life arose by natural selection) there’s a legitimate expectation that the argument should be a one-sided battle; the facts themselves are either one way or another, and the so-called “balance of evidence” should reflect this. Indeed, under the Bayesian definition of evidence, “strong evidence” is just that sort of evidence which we only expect to find on one side of an argument. But there is no reason for complex actions with many consequences to exhibit this onesidedness property. Why do people seem to want their policy debates to be one-sided? Politics is the mind-killer. Arguments are soldiers. Once you know which side you’re on, you must support all arguments of that side, and attack all arguments that appear to favor the enemy side; otherwise it’s like stabbing your soldiers in the back. If you abide within that pattern, policy debates will also appear one-sided to you—the costs and drawbacks of your favored policy are enemy soldiers, to be attacked by any means necessary. One should also be aware of a related failure pattern: thinking that the course of Deep Wisdom is to compromise with perfect evenness between whichever two policy positions receive the most airtime. A policy may legitimately have lopsided costs or benefits. If policy questions were not tilted one way or the other, we would be unable to make decisions about them. But there is also a human tendency to deny all costs of a favored policy, or deny all benefits of a disfavored policy; and people will therefore tend to think policy tradeoffs are tilted much further than they actually are. If you allow shops that sell otherwise banned products, some poor, honest, poorly educated mother of five kids is going to buy something that kills her. This is a prediction about a factual consequence, and as a factual question it appears rather straightforward—a sane person should readily confess this to be true regardless of which stance they take on the policy issue. You may also think that making things illegal just makes them more expensive, that regulators will abuse their power, or that her individual freedom trumps your desire to meddle with her life. But, as a matter of simple fact, she’s still going to die. We live in an unfair universe. Like all primates, humans have strong negative reactions to perceived unfairness; thus we find this fact stressful. There are two popular methods of dealing with the resulting cognitive dissonance. First, one may change one’s view of the facts—deny that the unfair events took place, or edit the history to make it appear fair.2 Second, one may change one’s morality—deny that the events are unfair. Some libertarians might say that if you go into a “banned products shop,” passing clear warning labels that say THINGS IN THIS STORE MAY KILL YOU, and buy something that kills you, then it’s your own fault and you deserve it. If that were a moral truth, there would be no downside to having shops that sell banned products. It wouldn’t just be a net benefit, it would be a one-sided tradeoff with no draw...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Norms of Membership for Voluntary Groups, published by sarahconstantin on the LessWrong. Epistemic Status: Idea Generation One feature of the internet that we haven’t fully adapted to yet is that it’s trivial to create voluntary groups for discussion. It’s as easy as making a mailing list, group chat, Facebook group, Discord server, Slack channel, etc. What we don’t seem to have is a good practical language for talking about norms on these mini-groups — what kind of moderation do we use, how do we admit and expel members, what kinds of governance structures do we create. Maybe this is a minor thing to talk about, but I suspect it has broader impact. In past decades voluntary membership in organizations has declined in the US — we’re less likely to be members of the Elks or of churches or bowling leagues — so lots of people who don’t have any experience in founding or participating in traditional types of voluntary organizations are now finding themselves engaged in governance without even knowing that’s what they’re doing. When we do this badly, we get “internet drama.” When we do it really badly, we get harassment campaigns and calls for regulation/moderation at the corporate or even governmental level. And that makes the news. It’s not inconceivable that Twitter moderation norms affect international relations, for instance. It’s a traditional observation about 19th century America that Americans were eager joiners of voluntary groups, and that these groups were practice for democratic participation. Political wonks today lament the lack of civic participation and loss of trust in our national and democratic institutions. Now, maybe you’ve moved on; maybe you’re a creature of the 21st century and you’re not hoping to restore trust in the institutions of the 20th. But what will be the institutions of the future? That may well be affected by what formats and frames for group membership people are used to at the small scale. It’s also relevant for the future of freedom. It’s starting to be a common claim that “give people absolute ‘free speech’ and the results are awful; therefore we need regulation/governance at the corporate or national level.” If you’re not satisfied with that solution (as I’m not), you have work to do — there are a lot of questions to unpack like “what kind of ‘freedom’, with what implementational details, is the valuable kind?”, “if small-scale voluntary organizations can handle some of the functions of the state, how exactly will they work?”, “how does one prevent the outcomes that people consider so awful that they want large institutions to step in to govern smaller groups?” Thinking about, and working on, governance for voluntary organizations (and micro-organizations like online discussion groups) is a laboratory for figuring this stuff out in real time, with fairly low resource investment and risk. That’s why I find this stuff fascinating and wish more people did. The other place to start, of course, is history, which I’m not very knowledgeable about, but intend to learn a bit. David Friedman is the historian I’m familiar with who’s studied historical governance and legal systems with an eye to potential applicability to building voluntary governance systems today; I’m interested in hearing about others. (Commenters?) In the meantime, I want to start generating a (non-exhaustive list) of types of norms for group membership, to illustrate the diversity of how groups work and what forms “expectations for members” can take. We found organizations based on formats and norms that we’ve seen before. It’s useful to have an idea of the range of formats that we might encounter, so we don’t get anchored on the first format that comes to mind. It’s also good to have a vocabulary so we can have higher-quality disagreements about the purpose & nature of ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Eliezer Yudkowsky Facts, published by steven0461 on the LessWrong. Eliezer Yudkowsky was once attacked by a Moebius strip. He beat it to death with the other side, non-violently. Inside Eliezer Yudkowsky's pineal gland is not an immortal soul, but another brain. Eliezer Yudkowsky's favorite food is printouts of Rice's theorem. Eliezer Yudkowsky's favorite fighting technique is a roundhouse dustspeck to the face. Eliezer Yudkowsky once brought peace to the Middle East from inside a freight container, through a straw. Eliezer Yudkowsky once held up a sheet of paper and said, "A blank map does not correspond to a blank territory". It was thus that the universe was created. If you dial Chaitin's Omega, you get Eliezer Yudkowsky on the phone. Unless otherwise specified, Eliezer Yudkowsky knows everything that he isn't telling you. Somewhere deep in the microtubules inside an out-of-the-way neuron somewhere in the basal ganglia of Eliezer Yudkowsky's brain, there is a little XML tag that says awesome. Eliezer Yudkowsky is the Muhammad Ali of one-boxing. Eliezer Yudkowsky is a 1400 year old avatar of the Aztec god Aixitl. The game of "Go" was abbreviated from "Go Home, For You Cannot Defeat Eliezer Yudkowsky". When Eliezer Yudkowsky gets bored, he pinches his mouth shut at the 1/3 and 2/3 points and pretends to be a General Systems Vehicle holding a conversation among itselves. On several occasions he has managed to fool bystanders. Eliezer Yudkowsky has a swiss army knife that has folded into it a corkscrew, a pair of scissors, an instance of AIXI which Eliezer once beat at tic tac toe, an identical swiss army knife, and Douglas Hofstadter. If I am ignorant about a phenomenon, that is not a fact about the phenomenon; it just means I am not Eliezer Yudkowsky. Eliezer Yudkowsky has no need for induction or deduction. He has perfected the undiluted master art of duction. There was no ice age. Eliezer Yudkowsky just persuaded the planet to sign up for cryonics. There is no spacetime symmetry. Eliezer Yudkowsky just sometimes holds the territory upside down, and he doesn't care. Eliezer Yudkowsky has no need for doctors. He has implemented a Universal Curing Machine in a system made out of five marbles, three pieces of plastic, and some of MacGyver's fingernail clippings. Before Bruce Schneier goes to sleep, he scans his computer for uploaded copies of Eliezer Yudkowsky. If you know more Eliezer Yudkowsky facts, post them in the comments. Thanks for listening. to help us out with the nonlinear library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Guessing the Teacher's Password, published by Eliezer Yudkowsky on the LessWrong. When I was young, I read popular physics books such as Richard Feynman’s QED: The Strange Theory of Light and Matter. I knew that light was waves, sound was waves, matter was waves. I took pride in my scientific literacy, when I was nine years old. When I was older, and I began to read the Feynman Lectures on Physics, I ran across a gem called “the wave equation.” I could follow the equation’s derivation, but, looking back, I couldn’t see its truth at a glance. So I thought about the wave equation for three days, on and off, until I saw that it was embarrassingly obvious. And when I finally understood, I realized that the whole time I had accepted the honest assurance of physicists that light was waves, sound was waves, matter was waves, I had not had the vaguest idea of what the word “wave” meant to a physicist. There is an instinctive tendency to think that if a physicist says “light is made of waves,” and the teacher says “What is light made of?” and the student says “Waves!”, then the student has made a true statement. That’s only fair, right? We accept “waves” as a correct answer from the physicist; wouldn’t it be unfair to reject it from the student? Surely, the answer “Waves!” is either true or false, right? Which is one more bad habit to unlearn from school. Words do not have intrinsic definitions. If I hear the syllables “bea-ver” and think of a large rodent, that is a fact about my own state of mind, not a fact about the syllables “bea-ver.” The sequence of syllables “made of waves” (or “because of heat conduction”) is not a hypothesis; it is a pattern of vibrations traveling through the air, or ink on paper. It can associate to a hypothesis in someone’s mind, but it is not, of itself, right or wrong. But in school, the teacher hands you a gold star for saying “made of waves,” which must be the correct answer because the teacher heard a physicist emit the same sound-vibrations. Since verbal behavior (spoken or written) is what gets the gold star, students begin to think that verbal behavior has a truth-value. After all, either light is made of waves, or it isn’t, right? And this leads into an even worse habit. Suppose the teacher asks you why the far side of a metal plate feels warmer than the side next to the radiator. If you say “I don’t know,” you have no chance of getting a gold star—it won’t even count as class participation. But, during the current semester, this teacher has used the phrases “because of heat convection,” “because of heat conduction,” and “because of radiant heat.” One of these is probably what the teacher wants. You say, “Eh, maybe because of heat conduction?” This is not a hypothesis about the metal plate. This is not even a proper belief. It is an attempt to guess the teacher’s password. Even visualizing the symbols of the diffusion equation (the math governing heat conduction) doesn’t mean you’ve formed a hypothesis about the metal plate. This is not school; we are not testing your memory to see if you can write down the diffusion equation. This is Bayescraft; we are scoring your anticipations of experience. If you use the diffusion equation, by measuring a few points with a thermometer and then trying to predict what the thermometer will say on the next measurement, then it is definitely connected to experience. Even if the student just visualizes something flowing, and therefore holds a match near the cooler side of the plate to try to measure where the heat goes, then this mental image of flowing-ness connects to experience; it controls anticipation. If you aren’t using the diffusion equation—putting in numbers and getting out results that control your anticipation of particular experiences—then the connection between map and territory is severed a...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Asymmetric Justice, published by Zvi on the LessWrong. Related and required reading in life (ANOIEAEIB): The Copenhagen Interpretation of Ethics Epistemic Status: Trying to be minimally judgmental Spoiler Alert: Contains minor mostly harmless spoiler for The Good Place, which is the best show currently on television. The Copenhagen Interpretation of Ethics (in parallel with the similarly named one in physics) is as follows: The Copenhagen Interpretation of Ethics says that when you observe or interact with a problem in any way, you can be blamed for it. At the very least, you are to blame for not doing more. Even if you don’t make the problem worse, even if you make it slightly better, the ethical burden of the problem falls on you as soon as you observe it. In particular, if you interact with a problem and benefit from it, you are a complete monster. I don’t subscribe to this school of thought, but it seems pretty popular. I don’t say this often, but seriously, read the whole thing. I do not subscribe to this interpretation. I believe that the majority of people effectively endorse this interpretation. I do not think they endorse it consciously or explicitly. But they act as if it is true. Another aspect of this same phenomenon is how most people view justice. Almost everyone agrees justice is a sacred value. That it is good and super important. Justice is one of the few universally agreed upon goals of government. Justice is one of the eight virtues of the avatar. Justice is up there with truth and the American way. No justice, no peace. But what is justice? Or rather, to avoid going too deeply into an infinitely complex philosophical debate millenniums or eons old, how do most people instinctively model justice in broad terms? In a conversation last night, this was offered to me (I am probably paraphrasing due to bad memory, but it’s functionally what was said), and seems common: Justice is giving appropriate punishment to those who have taken bad action. I asked whether, in this person’s model, the actions needed to be bad in order to be relevant to justice. This prompted pondering, after which the reply was that yes, that was how their model worked. I then asked whether rewarding a good action counted as justice, or failing to do so counted as injustice, using the example of saving someone’s life going unrewarded. We can consider three point-based justice systems. In the asymmetric system, when bad action is taken, bad action points are accumulated. Justice punishes in proportion to those points to the extent possible. Each action is assigned a non-negative point total. In the symmetric system, when any action is taken, good or bad, points are accumulated. This can be and often is zero, is negative for bad action, positive for good action. Justice consists of punishing negative point totals and rewarding positive point totals. In what we will call the Good Place system (Spoiler Alert for Season 1), when any action is taken, good or bad, points are accumulated as in the symmetric system. But there’s a catch (which is where the spoiler comes in). If you take actions with good consequences, you only get those points if your motive was to do good. When a character attempts to score points by holding open doors for people, they fail to score any points because they are gaming the system. Gaming the system isn’t allowed. Thus, if one takes action even under the best of motives, one fails to capture much of the gains from such action. Second or higher order benefits, or surprising benefits, that are real but unintended, will mostly not get captured. The opposite is not true of actions with bad consequences. You lose points for bad actions whether or not you intended to be bad. It is your responsibility to check yourself before you wreck yourself. When (Spoiler Alert fo...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: We've built Connected Papers - a visual tool for researchers to find and explore academic papers, published by discordy on the LessWrong. Hi LessWrong. I'm a long time lurker and finally have something that I'm really proud to share with you. After a long beta, we are releasing Connected Papers to the public! Connected papers is a unique, visual tool to help researchers and applied scientists find and explore papers relevant to their field of work. First - let's look at a couple of examples graphs for work that is representative of this community: Nick Bostrom: Eliezer Yudkowsky, Nate Soares: Did you find new and interesting papers to read? Would this be helpful as an introduction to the literature of a new field of study? The problem Almost every research project in academia or industry involves phases of literature review. Many times we find an interesting paper, and we’d like to: Find different methods and approaches to the same subject Track down the state of the art research in the field Identify seminal works and background reading Explore and immerse ourselves in the topic and become aware of the trends and dynamics in the literature Previously, the best ways to do this were to browse reference lists, or hope to find good keywords in textual search engines and databases. Enter Connected Papers It started as a side project between friends. We’ve felt the pains of academic literature review and exploration for years and kept thinking about how to solve it. For the past year we’ve been meeting on weekends and prototyping a tool that would allow a very different type of search process for academic papers. When we saw how much it improved our own research and development workflows — and got increasingly more requests from friends and colleagues to use it — we committed to release it to the public. You know. for science. So how does it work? Connected Papers is not a citation tree. Those have been done before. In our graph, papers are arranged according to their similarity. That means that even papers that do not directly cite each other can be strongly connected and positioned close to each other in the graph. To get a bit technical, our similarity is based primarily on the concepts of co-citation and bibliographic coupling (aka co-reference). According to this measure, two papers that have highly overlapping citations and references are presumed to have a higher chance of treating a related subject matter. Reading the graph Our graph is designed to make the important and relevant papers pop out immediately With our layout algorithm, similar papers cluster together in space and are connected by stronger lines (edges). Popular papers (that are frequently cited) are represented by bigger circles (nodes) and more recent papers are represented by a darker color. So for example, finding an important new paper in your field is as easy as identifying the dark large node at the center of a big cluster. List view In some cases it is convenient to work with just a list of connected papers. For these occasions, we’ve built the List view which you can access by clicking “Expand” at the top of the left panel. Here you can view additional paper details as well as sort and filter them according to various properties. Prior and derivative works The Prior works feature lists the top common ancestral papers for the connected papers in the graph. It usually includes seminal works in the field that heavily influenced the next generation. Meanwhile, the Derivative works feature is the opposite: it shows a list of common descendants of the papers in the graph. It usually includes relevant state of the art papers or systematic reviews and meta-analyses in the field. We have found these features to be especially useful when we have a paper from one era of research and we would like to be ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Two non-obvious lessons from microcovid.org, published by catherio on the LessWrong. At the 2021 Summer Solstice, Elizabeth Van Nostrand made a brief speech thanking the organizers of microcovid.org, which I found very heartwarming and meaningful. I wish I had thought in advance about taking the opportunity to make a few public remarks about microcovid.org: things I wish the community knew, that weren't obvious. Here's what I would've said, in terms of my lessons from this project: Build your own oxygen mask. Next, share it with others. Connect and collaborate with non-rationalists. 1) Build your own oxygen mask. Next, share it with others. We didn't start out trying to create a resource for our whole community, let alone a website with many thousands of users. All we wanted was to save our own asses. We looked at the precipitous "autonomy crunch" we were facing, and said "oh shit, our house is going to explode if we don't fix this." So, we built a spreadsheet — for ourselves first. Other group houses asked about it, and the momentum snowballed inexorably from there. Each broadening of project scope was compelled by a commensurate rise in demand, and corresponding deeply felt motivation. I think many people who have altruistic or worldsaving ambitions could stand to have more focus on first making their own lives not suck. Fixing huge problems in your own life — and then later making an extra effort to share and export them — is one important path to altruistic impact. 2) Connect and collaborate with non-rationalists To my knowledge, I'm the only project member out of the top dozen or so top contributors who self-identifies as a rationalist. The "core idea" is an extremely, extremely rationalist idea. But the implementation took writers, copyeditors, web developers, backend developers, UX designers, a medical doctor whose patients were among our first users, and many more. These folks had to understand the core idea and know how to use it, but did not have to be skilled enough at quantitative risk thinking to have designed it in the first place. The final product had a vastly more scalable reach because many people, who had very little identity-level commitment to epistemics, looked at it and said things like "I need to access this on my phone, I won't ever use a spreadsheet" or "This has too much jargon, move all these details to the appendix." Thank you again everyone for the gratitude and recognition; and for using the system to make your own lives suck a little less! Thanks for listening. to help us out with the nonlinear library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: SIAI - An Examination, published by BrandonReinhart on the LessWrong. 12/13/2011 - A 2011 update with data from the 2010 fiscal year is in progress. Should be done by the end of the week or sooner. Disclaimer I am not affiliated with the Singularity Institute for Artificial Intelligence. I have not donated to the SIAI prior to writing this. I made this pledge prior to writing this document. Notes Images are now hosted on LessWrong.com. The 2010 Form 990 data will be available later this month. It is not my intent to propagate misinformation. Errors will be corrected as soon as they are identified. Introduction Acting on gwern's suggestion in his Girl Scout Cookie analysis, I decided to look at SIAI funding. After reading about the Visiting Fellows Program and more recently the Rationality Boot Camp, I decided that the SIAI might be something I would want to support. I am concerned with existential risk and grapple with the utility implications. I feel that I should do more. I wrote on the mini-boot camp page a pledge that I would donate enough to send someone to rationality mini-boot camp. This seemed to me a small cost for the potential benefit. The SIAI might get better at building rationalists. It might build a rationalist who goes on to solve a problem. Should I donate more? I wasn’t sure. I read gwern’s article and realized that I could easily get more information to clarify my thinking. So I downloaded the SIAI’s Form 990 annual IRS filings and started to write down notes in a spreadsheet. As I gathered data and compared it to my expectations and my goals, my beliefs changed. I now believe that donating to the SIAI is valuable. I cannot hide this belief in my writing. I simply have it. My goal is not to convince you to donate to the SIAI. My goal is to provide you with information necessary for you to determine for yourself whether or not you should donate to the SIAI. Or, if not that, to provide you with some direction so that you can continue your investigation. The SIAI's Form 990's are available at GuideStar and Foundation Center. You must register in order to access the files at GuideStar. 2002 (Form 990-EZ) 2003 (Form 990-EZ) 2004 (Form 990-EZ) 2005 (Form 990) 2006 (Form 990) 2007 (Form 990) 2008 (Form 990-EZ) 2009 (Form 990) SIAI Financial Overview The Singularity Institute for Artificial Intelligence (SIAI) is a public organization working to reduce existential risk from future technologies, in particular artificial intelligence. "The Singularity Institute brings rational analysis and rational strategy to the challenges facing humanity as we develop cognitive technologies that will exceed the current upper bounds on human intelligence." The SIAI are also the founders of Less Wrong. The graphs above offer an accurate summary of SIAI financial state since 2002. Sometimes the end of year balances listed in the Form 990 doesn’t match what you’d get if you did the math by hand. These are noted as discrepancies between the filed year end balance and the expected year end balance or between the filed year start balance and the expected year start balance. Filing Error 1 - There appears to be a minor typo to the effect of $4.86 in the end of year balance for the 2004 document. It appears that Part I, Line 18 has been summed incorrectly. $32,445.76 is listed, but the expected result is $32,450.41. The Part II balance sheet calculations which agree with the error so the source of the error is unclear. The start of year balance in 2005 reflects the expected value so this was probably just a typo in 2004. The following year’s reported start of year balance does not contain the error. Filing Error 2 - The 2006 document reports a year start balance of $95,105.00 when the expected year start balance is $165,284.00, a discrepancy of $70,179.00. This amount is close to ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Covid-19: My Current Model, published by Zvion the LessWrong. The post will be a summary of my current key views on various aspects what is going on, especially in places where I see many or most responsible-looking people getting it importantly wrong. This post is not making strong evidence-based arguments for these views. This is not that post. This is me getting all this out there, on the record, in a place one can reference. Risks Follow Power Laws It is impossible to actually understand Covid-19 if you think of some things as ‘risky’ and other things as ‘safe’ and group together all the things in each category. And yet, that’s exactly how most of our thinking is directed. Instead, think of risks as following power laws. The riskiest activities are indoors, involve close physical proximity with others, while those for extended periods of time others cough, sing, puff or otherwise powerfully exhale, or talk directly at us, or we are in actual physical contact that then reaches one’s eyes, nose or mouth. Activities missing any of those components are much, much safer than activities that share all those components. Then other actions, such as masks and hand washing and not-face-touching, can reduce that risk by further large percentages. Slight reductions in the frequency and severity of your very risky actions is much more important than reducing the frequency of nominally risky actions. The few times you end up talking directly with someone in the course of business, the one social gathering you attend, the one overly crowded store you had to walk through, will dominate your risk profile. Be paranoid about that, and think how to make it less risky, or ideally avoid it. Don’t sweat the small stuff. And think about the physical world and what’s actually happening around you! Sacrifices To The Gods Are Demanded Everywhere A sacrifice to the Gods (post of this topic to be linked in when finally written) is an action with physical costs but with no interest in any meaningful physical benefits, taken in the hope that it will make one less blameworthy. Things are bad because we have sinned. The Gods demand sacrifice. If we do not act appropriately repentant and concerned, things will surely get worse. Once we act appropriately, we are virtuous and will doubtless be saved. We can stop. There is no need to proceed in a way that would actually work, once the Gods have been placated. Everything will work out. If you don’t make the proper sacrifices, then anything that goes wrong means it’s your fault. Or at least, you’ll always worry it is your fault. As will others. If you do make the proper sacrifices, nothing is your fault. Much better. If the action is efficient and actually were to solve the problem in a meaningful way, that would invalidate the whole operation. You can either show you are righteous and trust in the Gods, or you go about actually solving the problem. For obvious reasons, you can’t do both. A steelman of this is that Complexity is Bad and nuance impossible. If we start doing things based on whether they make sense that sets a terrible example and most people will be hopelessly lost. Thus, we sanitize packages. We stay exactly six feet apart. We wait exactly two weeks. We close all ‘non-essential’ businesses, but not ‘essential’ ones. We issue stay at home orders and give huge checks to the unemployed. Then we turn around and ‘reopen’ at which point that unemployment is voluntary, the state doesn’t have to pay, and so people are forced to go back to work. We lie to ban masks, then we try to mandate them, and wonder why people don’t trust the authorities. We hail our health care workers as heroes but don’t let them run experiments or gather much data. And of course, we enforce regulations enforce regulations enforce regulations, while shouting about how g...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Eliezer's Sequences and Mainstream Academia, published by lukeprog on the LessWrong. Due in part to Eliezer's writing style (e.g. not many citations), and in part to Eliezer's scholarship preferences (e.g. his preference to figure out much of philosophy on his own), Eliezer's Sequences don't accurately reflect the close agreement between the content of The Sequences and work previously done in mainstream academia. I predict several effects from this: Some readers will mistakenly think that common Less Wrong views are more parochial than they really are. Some readers will mistakenly think Eliezer's Sequences are more original than they really are. If readers want to know more about the topic of a given article, it will be more difficult for them to find the related works in academia than if those works had been cited in Eliezer's article. I'd like to counteract these effects by connecting the Sequences to the professional literature. (Note: I sort of doubt it would have been a good idea for Eliezer to spend his time tracking down more references and so on, but I realized a few weeks ago that it wouldn't take me much effort to list some of those references.) I don't mean to minimize the awesomeness of the Sequences. There is much original content in them (edit: probably most of their content is original), they are engagingly written, and they often have a more transformative effect on readers than the corresponding academic literature. I'll break my list of references into sections based on how likely I think it is that a reader will have missed the agreement between Eliezer's articles and mainstream academic work. (This is only a preliminary list of connections.) Obviously connected to mainstream academic work Eliezer's posts on evolution mostly cover material you can find in any good evolutionary biology textbook, e.g. Freeman & Herron (2007). Likewise, much of the Quantum Physics sequence can be found in quantum physics textbooks, e.g. Sakurai & Napolitano (2010). An Intuitive Explanation of Bayes' Theorem, How Much Evidence Does it Take, Probability is in the Mind, Absence of Evidence Is Evidence of Absence, Conservation of Expected Evidence, Trust in Bayes: see any textbook on Bayesian probability theory, e.g. Jaynes (2003) or Friedman & Koller (2009). What's a Bias, again?, Hindsight Bias, Correspondence Bias; Positive Bias: Look into the Dark, Doublethink: Choosing to be Biased, Rationalization, Motivated Stopping and Motivated Continuation, We Change Our Minds Less Often Than We Think, Knowing About Biases Can Hurt People, Asch's Conformity Experiment, The Affect Heuristic, The Halo Effect, Anchoring and Adjustment, Priming and Contamination, Do We Believe Everything We're Told, Scope Insensitivity: see standard works in the heuristics & biases tradition, e.g. Kahneman et al. (1982), Gilovich et al. 2002, Kahneman 2011. According to Eliezer, The Simple Truth is Tarskian and Making Beliefs Pay Rent is Peircian. The notion of Belief in Belief comes from Dennett (2007). Fake Causality and Timeless Causality report on work summarized in Pearl (2000). Fake Selfishness argues that humans aren't purely selfish, a point argued more forcefully in Batson (2011). Less obviously connected to mainstream academic work Eliezer's metaethics sequences includes dozens of lemmas previously discussed by philosophers (see Miller 2003 for an overview), and the resulting metaethical theory shares much in common with the metaethical theories of Jackson (1998) and Railton (2003), and must face some of the same critiques as those theories do (e.g. Sobel 1994). Eliezer's free will mini-sequence includes coverage of topics not usually mentioned when philosophers discuss free will (e.g. Judea Pearl's work on causality), but the conclusion is standard compatibilism. How an Algorithm Feels F...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Less Wrong NYC: Case Study of a Successful Rationalist Chapter, published by Cosmos on the LessWrong. It is perhaps the best-kept secret on Less Wrong that the New York City community has been meeting regularly for almost two years. For nearly a year we've been meeting weekly or more. The rest of this post is going to be a practical guide to the benefits of group rationality, but first I will do something that is still too rare on this blog: make it clear how strongly I feel about this. Before this community took off, I did not believe that life could be this much fun or that I could possibly achieve such a sustained level of happiness. Being rational in an irrational world is incredibly lonely. Every interaction reveals that our thought processes differ widely from those around us, and I had accepted that such a divide would always exist. For the first time in my life I have dozens of people with whom I can act freely and revel in the joy of rationality without any social concern - hell, it's actively rewarded! Until the NYC Less Wrong community formed, I didn't realize that I was a forager lost without a tribe... Rationalists are still human, and we still have basic human needs. lukeprog summarizes the literature on subjective well-being, and the only factors which correlate to any degree are genetics, health, work satisfaction and social life - which actually gets listed three separate times as social activity, relationship satisfaction and religiosity. Rationalists tend to be less socially adept on average, and this can make it difficult to obtain the full rewards of social interaction. However, once rationalists learn to socialize with each other, they also become increasingly social towards everyone more generally. This improves your life. A lot. We are a group of friends to enjoy life alongside, while we try miracle fruit, dance ecstatically until sunrise, actively embarrass ourselves at karaoke, get lost in the woods, and jump off waterfalls. Poker, paintball, parties, go-karts, concerts, camping... I have a community where I can live in truth and be accepted as I am, where I can give and receive feedback and get help becoming stronger. I am immensely grateful to have all of these people in my life, and I look forward to every moment I spend with them. To love and be loved is an unparalleled experience in this world, once you actually try it. So, you ask, how did all of this get started...? Genesis, or a Brief History of Nearly Everything The origin of the NYC chapter was the April 24th, 2009 meetup that Robin Hanson organized when he came to the city for a prediction markets conference. Approximately 15 people attended over the course of the night, and we all agreed that we had way too much fun together not to do this on a regular basis. I handed out my business cards to everyone there, told them to e-mail me, and I would create a mailing list. Thus Overcoming Bias NYC was born. It was clear from the very beginning that Jasen Murray was the person most interested in seeing this happen, so he became the organizer of the group for the first year of its existence. At first the times and locations were impromptu, but in August Jasen made the brilliant move of precommitting to be at a specific time and place for a minimum of two hours twice per month. Because enough of us liked Jasen and wanted to hang out with him anyway, several people began showing up every time and a regular meetup was established. Going forward we tried a combination of social meetups, focused discussions and game nights. Jasen also attempted to shift coordination from the mailing list to the Meetup group, but Meetup is not a great mailing list and people were loathe to use multiple services. That now serves as our public face. In April 2010, Jasen departed to run the Visiting Fellows progra...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Hero Licensing, published by Eliezer Yudkowsky on the LessWrong. I expect most readers to know me either as MIRI's co-founder and the originator of a number of the early research problems in AI alignment, or as the author of Harry Potter and the Methods of Rationality, a popular work of Harry Potter fanfiction. I’ve described how I apply concepts in Inadequate Equilibria to various decisions in my personal life, and some readers may be wondering how I see these tying in to my AI work and my fiction-writing. And I do think these serve as useful case studies in inadequacy, exploitability, and modesty. As a supplement to Inadequate Equilibria, then, the following is a dialogue that never took place—largely written in 2014, and revised and posted online in 2017. i. Outperforming and the outside view (The year is 2010. eliezer-2010 is sitting in a nonexistent park in Redwood City, California, working on his laptop. A person walks up to him.) person: Pardon me, but are you Eliezer Yudkowsky? eliezer-2010: I have that dubious honor. person: My name is Pat; Pat Modesto. We haven’t met, but I know you from your writing online. What are you doing with your life these days? eliezer-2010: I’m trying to write a nonfiction book on rationality. The blog posts I wrote on Overcoming Bias—I mean Less Wrong—aren’t very compact or edited, and while they had some impact, it seems like a book on rationality could reach a wider audience and have a greater impact. pat: Sounds like an interesting project! Do you mind if I peek in on your screen and eliezer: (shielding the screen) —Yes, I mind. pat: Sorry. Um... I did catch a glimpse and that didn’t look like a nonfiction book on rationality to me. eliezer: Yes, well, work on that book was going very slowly, so I decided to try to write something else in my off hours, just to see if my general writing speed was slowing down to molasses or if it was this particular book that was the problem. pat: It looked, in fact, like Harry Potter fanfiction. Like, I’m pretty sure I saw the words “Harry” and “Hermione” in configurations not originally written by J. K. Rowling. eliezer: Yes, and I currently seem to be writing it very quickly. And it doesn’t seem to use up mental energy the way my regular writing does, either. (A mysterious masked stranger, watching this exchange, sighs wistfully.) eliezer: Now I’ve just got to figure out why my main book-writing project is going so much slower and taking vastly more energy... There are so many books I could write, if I could just write everything as fast as I’m writing this... pat: Excuse me if this is a silly question. I don’t mean to say that Harry Potter fanfiction is bad—in fact I’ve read quite a bit of it myself—but as I understand it, according to your basic philosophy the world is currently on fire and needs to be put out. Now given that this is true, why are you writing Harry Potter fanfiction, rather than doing something else? eliezer: I am doing something else. I’m writing a nonfiction rationality book. This is just in my off hours. pat: Okay, but I’m asking why you are doing this particular thing in your off hours. eliezer: Because my life is limited by mental energy far more than by time. I can currently produce this work very cheaply, so I’m producing more of it. pat: What I’m trying to ask is why, even given that you can write Harry Potter fanfiction very cheaply, you are writing Harry Potter fanfiction. Unless it really is true that the only reason is that you need to observe yourself writing quickly in order to understand the way of quick writing, in which case I’d ask what probability you assign to learning that successfully. I’m skeptical that this is really the best way of using your off hours. eliezer: I’m skeptical that you have correctly understood the concept of “off hours.” There’s a ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Common knowledge about Leverage Research 1.0, published by BayAreaHuman on the LessWrong. I've spoken to people recently who were unaware of some basic facts about Leverage Research 1.0; facts that are more-or-less "common knowledge" among people who spent time socially adjacent to Leverage, and are not particularly secret or surprising in Leverage-adjacent circles, but aren't attested publicly in one place anywhere. Today, Geoff Anders and Leverage 2.0 are moving into the "Progress Studies" space, and seeking funding in this area (see: Geoff recently got a small grant from Emergent Ventures). This seems like an important time to contribute to common knowledge about Leverage 1.0. You might conclude that I'm trying to discredit people who were involved, but that's not my aim here. My friends who were involved in Leverage 1.0 are people who I respect greatly. Rather, I just keep being surprised that people haven't heard certain specific, more-or-less legible facts about the past, that seem well-known or obvious to me, and that I feel should be taken into account when evaluating Leverage as a player in the current landscape. I would like to create here a publicly-linkable document containing these statements. Facts that are common knowledge among people I know: Members of Leverage 1.0 lived and worked in the same Leverage-run building, an apartment complex near Lake Merritt. (Living there was not required, but perhaps half the members did, and new members were particularly encouraged to.) Participation in the project involved secrecy / privacy / information-management agreements. People were asked to sign an agreement that prohibited publishing almost anything (for example, in one case someone I know starting a personal blog on unrelated topics without permission led to a stern reprimand). Geoff developed a therapy technique, "charting". He says he developed it based on his novel and complete theory of psychology, called "Connection Theory". In my estimation, "charting" is in the same rough family of psychotherapy techniques as Internal Family Systems, Coherence Therapy, Core Transformation, and similar. Like those techniques, it leads to shifts in clients' beliefs and moods. I know people from outside Leverage who did charting sessions with a "coach" from Paradigm Academy, and reported it helped them greatly. I've also heard people who did lots of charting within Leverage report that it led to dissociation and fragmentation, that they have found difficult to reverse. Members who were on payroll were expected to undergo charting/debugging sessions with a supervisory "trainer", and to "train" other members. The role of trainer is something like "manager + therapist": that is, both "is evaluating your job performance" and "is doing therapy on you". Another type of practice done at the organization, and offered to some people outside the organization, was "bodywork", which involved physical contact between the trainer and the trainee. "Bodywork" could in other contexts be a synonym for "massage", but that's not what's meant here; descriptions I heard of sessions sounded to me more like "energy work". People I've spoken to say it was reported to produce deeper and less legible change. Using psychological techniques to experiment on one another, and on the "sociology" of the group itself, was a main purpose of the group. It was understood among members that they were signing up to be guinea pigs for experiments in introspection, altering one's belief structure, and experimental group dynamics. The stated purpose of the group was to discover more theories of human behavior and civilization by "theorizing", while building power, and then literally take over US and/or global governance (the vibe was "take over the world"). The purpose of gaining global power was to lead to bett...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: On saving the world by So8res, published by So8reson the LessWrong. This is the final post in my productivity sequence. The first post described what I achieved. The next three posts describe how. This post describes why, explaining the sources of my passion and the circumstances that convinced a young Nate to try and save the world. Within, you will find no suggestions, no techniques to emulate, no new ideas to ponder. This is a rationalist coming-of-age story. With luck, you may find it inspiring. Regardless, I hope you can learn from my mistakes. Never fear, I'll be back to business soon — there's lots of studying to do. But before then, there's a story to tell, a memorial to what I left behind. I was raised Catholic. On my eighth birthday, having received my first communion about a year prior, I casually asked my priest how to reaffirm my faith and do something for the Lord. The memory is fuzzy, but I think I donated a chunk of allowance money and made a public confession at the following mass. A bunch of the grownups made a big deal out of it, as grownups are like to do. "Faith of a child", and all that. This confused me, especially when I realized that what I had done was rare. I wasn't trying to get pats on the head, I was appealing to the Lord of the Heavens and the Earth. Were we all on the same page, here? This was the creator. He was infinitely virtuous, and he had told us what to do. And yet, everyone was content to recite hymns once a week and donate for the reconstruction of the church. What about the rest of the world, the sick, the dying? Where were the proselytizers, the missionary opportunities? Why was everyone just sitting around? On that day, I became acquainted with civilizational inadequacy. I realized you could hand a room full of people the literal word of God, and they'd still struggle to pay attention for an hour every weekend. This didn't shake my faith, mind you. It didn't even occur to me that the grownups might not actually believe their tales. No, what I learned that day was that there are a lot of people who hold beliefs they aren't willing to act upon. Eventually, my faith faded. The distrust remained. Gaining Confidence I grew up in a small village, population ~1200. My early education took place in a one-room schoolhouse. The local towns eventually rolled all their school districts into one, but even then, my graduating class barely broke 50 people. It wasn't difficult to excel. Ages twelve and thirteen were rough — that was right after they merged school districts, and those were the years I was first put a few grades ahead in math classes. I was awkward and underconfident. I felt estranged and lonely, and it was easy to get shoehorned into the "smart kid" stereotype by all the new students. Eventually, though, I decided that the stereotype was bogus. Anyone intelligent should be able to escape such pigeonholing. In fact, I concluded that anyone with real smarts should be able to find their way out of any mess. I observed the confidence possessed by my peers, even those who seemed to have no reason for confidence. I noticed the ease with which they engaged in social interactions. I decided I could emulate these. I faked confidence, and it soon became real. I found that my social limitations had been largely psychological, and that the majority of my classmates were more than willing to be friends. I learned how to get good grades without alienating my peers. It helped that I tended to buck authority (I was no "teacher's pet") and that I enjoyed teaching others. I had a knack for pinpointing misunderstandings and was often able to teach better than the teachers could — as a peer, I could communicate on a different level. I started doing very well for myself. I got excellent grades with minimal effort. I overcame my social anxieties...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Crash Course in the Neuroscience of Human Motivation , published by on the LessWrong. [PDF of this article updated Aug. 23, 2011] [skip to preface] Whenever I write a new article for Less Wrong, I'm pulled in two opposite directions. One force pulls me toward writing short, exciting posts with lots of brain candy and just one main point. Eliezer has done that kind of thing very well many times: see Making Beliefs Pay Rent, Hindsight Devalues Science, Probability is in the Mind, Taboo Your Words, Mind Projection Fallacy, Guessing the Teacher's Password, Hold Off on Proposing Solutions, Applause Lights, Dissolving the Question, and many more. Another force pulls me toward writing long, factually dense posts that fill in as many of the pieces of a particular argument in one fell swoop as possible. This is largely because I want to write about the cutting edge of human knowledge but I keep realizing that the inferential gap is larger than I had anticipated, and I want to fill in that inferential gap quickly so I can get to the cutting edge. For example, I had to draw on dozens of Eliezer's posts just to say I was heading toward my metaethics sequence. I've also published 21 new posts (many of them quite long and heavily researched) written specifically because I need to refer to them in my metaethics sequence.1 I tried to make these posts interesting and useful on their own, but my primary motivation for writing them was that I need them for my metaethics sequence. And now I've written only four posts2 in my metaethics sequence and already the inferential gap to my next post in that sequence is huge again. :( So I'd like to try an experiment. I won't do it often, but I want to try it at least once. Instead of writing 20 more short posts between now and the next post in my metaethics sequence, I'll attempt to fill in a big chunk of the inferential gap to my next metaethics post in one fell swoop by writing a long tutorial post (a la Eliezer's tutorials on Bayes' Theorem and technical explanation).3 So if you're not up for a 20-page tutorial on human motivation, this post isn't for you, but I hope you're glad I bothered to write it for the sake of others. If you are in the mood for a 20-page tutorial on human motivation, please proceed. Who knows what I want to do? Who knows what anyone wants to do? How can you be sure about something like that? Isn’t it all a question of brain chemistry, signals going back and forth, electrical energy in the cortex? How do you know whether something is really what you want to do or just some kind of nerve impulse in the brain. Some minor little activity takes place somewhere in this unimportant place in one of the brain hemispheres and suddenly I want to go to Montana or I don’t want to go to Montana. - Don DeLillo, White Noise Preface How do we value things, and choose between options? Philosophers, economists, and psychologists have long tried to answer these questions. But human behavior continues to defy our most subtle models of it, and the algorithms producing our behavior remained hidden in a black box. But now, neuroscientists are directly measuring the neurons whose firing rates encode value and produce our choices. We know a lot more about the neuroscience of human motivation than you might think. Now we can peer directly into the black box of human motivation, and begin (dimly) to read our own source code. The neuroscience of human motivation has implications for philosophy of mind and action, for scientific self-help, and for metaethics and Friendly AI. (We don't really know what we want, and looking directly at the algorithms that produce human wanting might help in solving this mystery.) So, I wrote a crash course in the neuroscience of human motivation. The purpose of this document is not to argue for any of the ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Can the Chain Still Hold You?, published by lukeprogon the LessWrong. Robert Sapolsky: Baboons... literally have been the textbook example of a highly aggressive, male-dominated, hierarchical society. Because these animals hunt, because they live in these aggressive troupes on the Savannah... they have a constant baseline level of aggression which inevitably spills over into their social lives. Scientists have never observed a baboon troupe that wasn't highly aggressive, and they have compelling reasons to think this is simply baboon nature, written into their genes. Inescapable. Or at least, that was true until the 1980s, when Kenya experienced a tourism boom. Sapolsky was a grad student, studying his first baboon troupe. A new tourist lodge was built at the edge of the forest where his baboons lived. The owners of the lodge dug a hole behind the lodge and dumped their trash there every morning, after which the males of several baboon troupes — including Sapolsky's — would fight over this pungent bounty. Before too long, someone noticed the baboons didn't look too good. It turned out they had eaten some infected meat and developed tuberculosis, which kills baboons in weeks. Their hands rotted away, so they hobbled around on their elbows. Half the males in Sapolsky's troupe died. This had a surprising effect. There was now almost no violence in the troupe. Males often reciprocated when females groomed them, and males even groomed other males. To a baboonologist, this was like watching Mike Tyson suddenly stop swinging in a heavyweight fight to start nuzzling Evander Holyfield. It never happened. This was interesting, but Sapolsky moved to the other side of the park and began studying other baboons. His first troupe was "scientifically ruined" by such a non-natural event. But really, he was just heartbroken. He never visited. Six years later, Sapolsky wanted to show his girlfriend where he had studied his first troupe, and found that they were still there, and still surprisingly violence-free. This one troupe had apparently been so transformed by their unusual experience — and the continued availability of easy food — that they were now basically non-violent. And then it hit him. Only one of the males now in the troupe had been through the event. All the rest were new, and hadn't been raised in the tribe. The new males had come from the violent, dog-eat-dog world of normal baboon-land. But instead of coming into the new troupe and roughing everybody up as they always did, the new males had learned, "We don't do stuff like that here." They had unlearned their childhood culture and adapted to the new norms of the first baboon pacifists. As it turned out, violence wasn't an unchanging part of baboon nature. In fact it changed rather quickly, when the right causal factor flipped, and — for this troupe and the new males coming in — it has stayed changed to this day. Somehow, the violence had been largely circumstantial. It was just that the circumstances had always been the same. Until they weren't. We still don't know how much baboon violence to attribute to nature vs. nurture, or exactly how this change happened. But it's worth noting that changes like this can and do happen pretty often. Slavery was ubiquitous for millennia. Until it was outlawed in every country on Earth. Humans had never left the Earth. Until we achieved the first manned orbit and the first manned moon landing in a single decade. Smallpox occasionally decimated human populations for thousands of years. Until it was eradicated. The human species was always too weak to render itself extinct. Until we discovered the nuclear chain reaction and manufactured thousands of atomic bombs. Religion had a grip on 99.5% or more of humanity until 1900, and then the rate of religious adherence plummeted to 85% by th...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Checklist of Rationality Habits, published by AnnaSalamon on the LessWrong. As you may know, the Center for Applied Rationality has run several workshops, each teaching content similar to that in the core sequences, but made more practical, and more into fine-grained habits. Below is the checklist of rationality habits we have been using in the minicamps' opening session. It was co-written by Eliezer, myself, and a number of others at CFAR. As mentioned below, the goal is not to assess how "rational" you are, but, rather, to develop a personal shopping list of habits to consider developing. We generated it by asking ourselves, not what rationality content it's useful to understand, but what rationality-related actions (or thinking habits) it's useful to actually do. I hope you find it useful; I certainly have. Comments and suggestions are most welcome; it remains a work in progress. (It's also available as a pdf.) This checklist is meant for your personal use so you can have a wish-list of rationality habits, and so that you can see if you're acquiring good habits over the next year—it's not meant to be a way to get a 'how rational are you?' score, but, rather, a way to notice specific habits you might want to develop. For each item, you might ask yourself: did you last use this habit... Never Today/yesterday Last week Last month Last year Before the last year Reacting to evidence / surprises / arguments you haven't heard before; flagging beliefs for examination. When I see something odd - something that doesn't fit with what I'd ordinarily expect, given my other beliefs - I successfully notice, promote it to conscious attention and think "I notice that I am confused" or some equivalent thereof. (Example: You think that your flight is scheduled to depart on Thursday. On Tuesday, you get an email from Travelocity advising you to prepare for your flight “tomorrow”, which seems wrong. Do you successfully raise this anomaly to the level of conscious attention? (Based on the experience of an actual LWer who failed to notice confusion at this point and missed their plane flight.) When somebody says something that isn't quite clear enough for me to visualize, I notice this and ask for examples. (Recent example from Eliezer: A mathematics student said they were studying "stacks". I asked for an example of a stack. They said that the integers could form a stack. I asked for an example of something that was not a stack.) (Recent example from Anna: Cat said that her boyfriend was very competitive. I asked her for an example of "very competitive." She said that when he’s driving and the person next to him revs their engine, he must be the one to leave the intersection first—and when he’s the passenger he gets mad at the driver when they don’t react similarly.) I notice when my mind is arguing for a side (instead of evaluating which side to choose), and flag this as an error mode. (Recent example from Anna: Noticed myself explaining to myself why outsourcing my clothes shopping does make sense, rather than evaluating whether to do it.) I notice my mind flinching away from a thought; and when I notice, I flag that area as requiring more deliberate exploration. (Recent example from Anna: I have a failure mode where, when I feel socially uncomfortable, I try to make others feel mistaken so that I will feel less vulnerable. Pulling this thought into words required repeated conscious effort, as my mind kept wanting to just drop the subject.) I consciously attempt to welcome bad news, or at least not push it away. (Recent example from Eliezer: At a brainstorming session for future Singularity Summits, one issue raised was that we hadn't really been asking for money at previous ones. My brain was offering resistance, so I applied the "bad news is good news" pattern to rephrase this as, ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Treacherous Path to Rationality, published by Jacob Falkovich on the LessWrong. Cross-posted, as always, from Putanumonit. Rats v. Plague The Rationality community was never particularly focused on medicine or epidemiology. And yet, we basically got everything about COVID-19 right and did so months ahead of the majority of government officials, journalists, and supposed experts. We started discussing the virus and raising the alarm in private back in January. By late February, as American health officials were almost unanimously downplaying the threat, we wrote posts on taking the disease seriously, buying masks, and preparing for quarantine. Throughout March, the CDC was telling people not to wear masks and not to get tested unless displaying symptoms. At the same time, Rationalists were already covering every relevant angle, from asymptomatic transmission to the effect of viral load, to the credibility of the CDC itself. As despair and confusion reigned everywhere into the summer, Rationalists built online dashboards modeling nationwide responses and personal activity risk to let both governments and individuals make informed decisions. This remarkable success did not go unnoticed. Before he threatened to doxx Scott Alexander and triggered a shitstorm, New York Times reporter Cade Metz interviewed me and other Rationalists mostly about how we were ahead of the curve on COVID and what others can learn from us. I told him that Rationality has a simple message: “people can use explicit reason to figure things out, but they rarely do” If rationalists led the way in covering COVID-19, Vox brought up the rear Rationalists have been working to promote the application of explicit reason, to “raise the sanity waterline” as it were, but with limited success. I wrote recently about success stories of rationalist improvement but I don’t think it inspired a rush to LessWrong. This post is in a way a response to my previous one. It’s about the obstacles preventing people from training and succeeding in the use of explicit reason, impediments I faced myself and saw others stumble over or turn back from. This post is a lot less sanguine about the sanity waterline’s prospects. The Path I recently chatted with Spencer Greenberg about teaching rationality. Spencer regularly publishes articles like 7 questions for deciding whether to trust your gut or 3 types of binary thinking you fall for. Reading him, you’d think that the main obstacle to pure reason ruling the land is lack of intellectual listicles on ways to overcome bias. But we’ve been developing written and in-person curricula for improving your ability to reason for more than a decade. Spencer’s work is contributing to those curricula, an important task. And yet, I don’t think that people’s main failure point is in procuring educational material. I think that people don’t want to use explicit reason. And if they want to, they fail. And if they start succeeding, they’re punished. And if they push on, they get scared. And if they gather their courage, they hurt themselves. And if they make it to the other side, their lives enriched and empowered by reason, they will forget the hard path they walked and will wonder incredulously why everyone else doesn’t try using reason for themselves. This post is about that hard path. The map is not the territory Alternatives to Reason What do I mean by explicit reason? I don’t refer merely to “System 2”, the brain’s slow, sequential, analytical, fully conscious, and effortful mode of cognition. I refer to the informed application of this type of thinking. Gathering data with real effort to find out, crunching the numbers with a grasp of the math, modeling the world with testable predictions, reflection on your thinking with an awareness of biases. Reason requires good inputs and a lot of...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: On Caring, published by So8res on the LessWrong. This is an essay describing some of my motivation to be an effective altruist. It is crossposted from my blog. Many of the ideas here are quite similar to others found in the sequences. I have a slightly different take, and after adjusting for the typical mind fallacy I expect that this post may contain insights that are new to many. 1 I'm not very good at feeling the size of large numbers. Once you start tossing around numbers larger than 1000 (or maybe even 100), the numbers just seem "big". Consider Sirius, the brightest star in the night sky. If you told me that Sirius is as big as a million earths, I would feel like that's a lot of Earths. If, instead, you told me that you could fit a billion Earths inside Sirius. I would still just feel like that's a lot of Earths. The feelings are almost identical. In context, my brain grudgingly admits that a billion is a lot larger than a million, and puts forth a token effort to feel like a billion-Earth-sized star is bigger than a million-Earth-sized star. But out of context — if I wasn't anchored at "a million" when I heard "a billion" — both these numbers just feel vaguely large. I feel a little respect for the bigness of numbers, if you pick really really large numbers. If you say "one followed by a hundred zeroes", then this feels a lot bigger than a billion. But it certainly doesn't feel (in my gut) like it's 10 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 times bigger than a billion. Not in the way that four apples internally feels like twice as many as two apples. My brain can't even begin to wrap itself around this sort of magnitude differential. This phenomena is related to scope insensitivity, and it's important to me because I live in a world where sometimes the things I care about are really really numerous. For example, billions of people live in squalor, with hundreds of millions of them deprived of basic needs and/or dying from disease. And though most of them are out of my sight, I still care about them. The loss of a human life with all is joys and all its sorrows is tragic no matter what the cause, and the tragedy is not reduced simply because I was far away, or because I did not know of it, or because I did not know how to help, or because I was not personally responsible. Knowing this, I care about every single individual on this planet. The problem is, my brain is simply incapable of taking the amount of caring I feel for a single person and scaling it up by a billion times. I lack the internal capacity to feel that much. My care-o-meter simply doesn't go up that far. And this is a problem. 2 It's a common trope that courage isn't about being fearless, it's about being afraid but doing the right thing anyway. In the same sense, caring about the world isn't about having a gut feeling that corresponds to the amount of suffering in the world, it's about doing the right thing anyway. Even without the feeling. My internal care-o-meter was calibrated to deal with about a hundred and fifty people, and it simply can't express the amount of caring that I have for billions of sufferers. The internal care-o-meter just doesn't go up that high. Humanity is playing for unimaginably high stakes. At the very least, there are billions of people suffering today. At the worst, there are quadrillions (or more) potential humans, transhumans, or posthumans whose existence depends upon what we do here and now. All the intricate civilizations that the future could hold, the experience and art and beauty that is possible in the future, depends upon the present. When you're faced with stakes like these, your internal caring heuristics — calibrated on numbers like "ten" or "twenty" — completely fail to gras...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Well-Kept Gardens Die By Pacifism , published by Eliezer Yudkowsky on the LessWrong. Previously in series: My Way Followup to: The Sin of Underconfidence Good online communities die primarily by refusing to defend themselves. Somewhere in the vastness of the Internet, it is happening even now. It was once a well-kept garden of intelligent discussion, where knowledgeable and interested folk came, attracted by the high quality of speech they saw ongoing. But into this garden comes a fool, and the level of discussion drops a little—or more than a little, if the fool is very prolific in their posting. (It is worse if the fool is just articulate enough that the former inhabitants of the garden feel obliged to respond, and correct misapprehensions—for then the fool dominates conversations.) So the garden is tainted now, and it is less fun to play in; the old inhabitants, already invested there, will stay, but they are that much less likely to attract new blood. Or if there are new members, their quality also has gone down. Then another fool joins, and the two fools begin talking to each other, and at that point some of the old members, those with the highest standards and the best opportunities elsewhere, leave... I am old enough to remember the USENET that is forgotten, though I was very young. Unlike the first Internet that died so long ago in the Eternal September, in these days there is always some way to delete unwanted content. We can thank spam for that—so egregious that no one defends it, so prolific that no one can just ignore it, there must be a banhammer somewhere. But when the fools begin their invasion, some communities think themselves too good to use their banhammer for—gasp!—censorship. After all—anyone acculturated by academia knows that censorship is a very grave sin... in their walled gardens where it costs thousands and thousands of dollars to enter, and students fear their professors' grading, and heaven forbid the janitors should speak up in the middle of a colloquium. It is easy to be naive about the evils of censorship when you already live in a carefully kept garden. Just like it is easy to be naive about the universal virtue of unconditional nonviolent pacifism, when your country already has armed soldiers on the borders, and your city already has police. It costs you nothing to be righteous, so long as the police stay on their jobs. The thing about online communities, though, is that you can't rely on the police ignoring you and staying on the job; the community actually pays the price of its virtuousness. In the beginning, while the community is still thriving, censorship seems like a terrible and unnecessary imposition. Things are still going fine. It's just one fool, and if we can't tolerate just one fool, well, we must not be very tolerant. Perhaps the fool will give up and go away, without any need of censorship. And if the whole community has become just that much less fun to be a part of... mere fun doesn't seem like a good justification for (gasp!) censorship, any more than disliking someone's looks seems like a good reason to punch them in the nose. (But joining a community is a strictly voluntary process, and if prospective new members don't like your looks, they won't join in the first place.) And after all—who will be the censor? Who can possibly be trusted with such power? Quite a lot of people, probably, in any well-kept garden. But if the garden is even a little divided within itself —if there are factions—if there are people who hang out in the community despite not much trusting the moderator or whoever could potentially wield the banhammer (for such internal politics often seem like a matter of far greater import than mere invading barbarians) then trying to defend the community is typically depicted as a coup attempt. Who is th...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Debate on Instrumental Convergence between LeCun, Russell, Bengio, Zador, and More , published by Ben Pace on the LessWrong. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. An actual debate about instrumental convergence, in a public space! Major respect to all involved, especially Yoshua Bengio for great facilitation. For posterity (i.e. having a good historical archive) and further discussion, I've reproduced the conversation here. I'm happy to make edits at the request of anyone in the discussion who is quoted below. I've improved formatting for clarity and fixed some typos. For people who are not researchers in this area who wish to comment, see the public version of this post here. For people who do work on the relevant areas, please sign up in the top right. It will take a day or so to confirm membership. Original Post Yann LeCun: "don't fear the Terminator", a short opinion piece by Tony Zador and me that was just published in Scientific American. "We dramatically overestimate the threat of an accidental AI takeover, because we tend to conflate intelligence with the drive to achieve dominance. [...] But intelligence per se does not generate the drive for domination, any more than horns do." Comment Thread #1 Elliot Olds: Yann, the smart people who are very worried about AI seeking power and ensuring its own survival believe it's a big risk because power and survival are instrumental goals for almost any ultimate goal. If you give a generally intelligent AI the goal to make as much money in the stock market as possible, it will resist being shut down because that would interfere with tis goal. It would try to become more powerful because then it could make money more effectively. This is the natural consequence of giving a smart agent a goal, unless we do something special to counteract this. You've often written about how we shouldn't be so worried about AI, but I've never seen you address this point directly. Stuart Russell: It is trivial to construct a toy MDP in which the agent's only reward comes from fetching the coffee. If, in that MDP, there is another "human" who has some probability, however small, of switching the agent off, and if the agent has available a button that switches off that human, the agent will necessarily press that button as part of the optimal solution for fetching the coffee. No hatred, no desire for power, no built-in emotions, no built-in survival instinct, nothing except the desire to fetch the coffee successfully. This point cannot be addressed because it's a simple mathematical observation. Comment Thread #2 Yoshua Bengio: Yann, I'd be curious about your response to Stuart Russell's point. Yann LeCun: You mean, the so-called "instrumental convergence" argument by which "a robot can't fetch you coffee if it's dead. Hence it will develop self-preservation as an instrumental sub-goal." It might even kill you if you get in the way. 1. Once the robot has brought you coffee, its self-preservation instinct disappears. You can turn it off. 2. One would have to be unbelievably stupid to build open-ended objectives in a super-intelligent (and super-powerful) machine without some safeguard terms in the objective. 3. One would have to be rather incompetent not to have a mechanism by which new terms in the objective could be added to prevent previously-unforeseen bad behavior. For humans, we have education and laws to shape our objective functions and complement the hardwired terms built into us by evolution. 4. The power of even the most super-intelligent machine is limited by physics, and its size and needs make it vulnerable to physical attacks. No need for much intelligence here. A virus is infinitely less intelligent than you, but it can still kill you. 5. A second machine, designed solely to neut...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Limits of Current US Prediction Markets (PredictIt Case Study), published by aphyer on the LessWrong. (Disclaimers: I work in the financial industry, though not in a way related to prediction markets. Anything I write here is my opinion and not that of my employer. This is a US-centric piece based on a case study of PredictIt: as some people have pointed out in the comments below, if you are outside the US you may have substantially better options.) SECTION I: INTRODUCTION So there's an argument that I've seen a lot over the past few years, particularly in LW-adjacent circles, that goes something like this: You say you believe X is likely to happen. But prediction markets say X is likely not to happen. Since markets are efficient, you must be wrong. Or if you do know better than the market, why aren't you rich? Since you haven't bet on that market to make free money, you must be lying. Or stupid. Or both! This post is dedicated to disagreeing with that argument, not from an anti-Efficient-Market Hypothesis position, but from a pro-Efficient-Market Hypothesis position. My position is: The argument above is pretty much sound if we are discussing mainstream financial markets. If someone claims to have better information than a mainstream financial market on the value of Google stock, or of copper, they ought to either use this knowledge to make a huge amount of money or stop talking about it. However, it is not true if we are discussing prediction markets. Current prediction markets are so bad in so many different ways that it simply is not surprising for people to know better than them, and it often is not possible for people to make money from knowing better. I've been meaning to write this for a while, but got tipped over the edge by the recent post here, which talks about the limitation of prediction markets being the correlation of the events they predict to other assets, and their consequent value as hedging instruments. That is...well...it's not wrong exactly, but there are so many other problems that are so much bigger that I felt it was worth laying (some of) them out. Math follows. I will be focusing on PredictIt for this analysis. Other prediction markets may work a bit differently, but similar analysis is applicable to any of them. If you think the math is wrong I am happy to discuss/make changes, but I very much doubt any changes will materially alter the final message. As of this writing PredictIt has Donald Trump at 40% to win the election (or, to put it another way, you can pay 40 cents for a share that pays out $1 if Trump wins). Suppose you think he is more/less likely to win. How likely/unlikely does it need to be for Trump to win for you to make money (in expectation)? Or, to put it another way, what range of probabilities for Trump to win are consistent with the prediction market values? SECTION II: REASONABLY SIMPLE PROBLEMS 1: Spread. This is only a small problem, but it is non-zero. PredictIt will sell me 'Donald Trump wins' shares for 40 cents, but will sell me 'Donald Trump loses' shares for 61 cents (which, from a finance perspective, works out very similarly to letting me sell 'Donald Trump wins' shares for 39 cents). So if I think there is a 39.5% chance of Trump winning, there is no way for me to make money off of it: I can buy 'Trump wins' shares for 40 cents, or sell them for 39 cents, and if the true value is 39.5 cents both of these will lose me money. The range of possible probabilities for which you cannot make money starts at 39-40%. 2: Transaction Fees. PredictIt charges a 10% fee on profits (see). As far as I can tell, it does not net profits against losses before calculating these fees. That is to say, if I make two $100 bets at even odds, win one, and lose the other, PredictIt will charge me a $10 fee on my winnings on the bet I ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The LessWrong 2018 Book is Available for Pre-order, published by Ben Pace, jacobjacobon the LessWrong. For the first time, you can now buy the best new ideas on LessWrong in a physical book set, titled: A Map that Reflects the Territory: Essays by the LessWrong Community It is available for pre-order here. The standard advice for creating things is "show, don't tell", so first some images of the books, followed by a short FAQ by me (Ben). The full five-book set. Yes, that’s the iconic Mississippi river flowing across the spines. Each book has a unique color. The first book: Epistemology. The second book: Agency. The third book: Coordination. The fourth book: Curiosity. The fifth book: Alignment. FAQ What exactly is in the book set? LessWrong has an annual Review process (the second of which is beginning today!) to determine the best content on the site. We reviewed all the posts on LessWrong from 2018, and users voted to rank the best of them, the outcome of which can be seen here. Of the over 2000 LessWrong posts reviewed, this book contains 41 of the top voted essays, along with some comment sections, some reviews, a few extra essays to give context, and some preface/meta writing. What are the books in the set? The essays have been clustered around five topics relating to rationality: Epistemology, Agency, Coordination, Curiosity, and Alignment. Are all the essays in this book from 2018? Yes, all the essays in this book were originally published in 2018, and were reviewed and voted on during the 2018 LessWrong Review (which happened at the end of 2019). How small are the books? Each book is 4x6 inches, small enough to fit in your pocket. This was the book size that, empirically, most beta-testers found that they actually read. Can I order a copy of the book? Pre-order the book here for $29. We currently sell to North America, Europe, Australia, New Zealand, Israel. (If you bought it by end-of-day Wednesday December 9th and ordered within North America, you'll get it before Christmas.) You'll be able to buy the book on Amazon in a couple of weeks. How much is shipping? The price above includes shipping to any location that we accept shipping addresses for. We are still figuring out some details about shipping internationally, so if you are somewhere that is not North America, there is a small chance (~10%) that we will reach out to you to ask you for more shipping details, and an even smaller chance (~6%) that we offer you the option to either pay for some additional shipping fees or get a refund. Can I order more than one copy at a time? Yes. Just open the form multiple times. We will make sure to combine your shipments. Does this book assume I have read other LessWrong content, like The Sequences? No. It's largely standalone, and does not require reading other content on the site, although it will be enhanced by having engaged with those ideas. Can I see an extract from the book? Sure. Here is the preface and first chapter of Curiosity, specifically the essay Is Science Slowing Down? by Scott Alexander. I'm new — what is this all about? What is 'rationality'? A scientist is not simply someone who tries to understand how biological life works, or how chemicals combine, or how physical objects move, but is someone who uses the general scientific method in all areas, that allows them to empirically test their beliefs and discover what's true in general. Similarly, a rationalist is not simply someone who tries to think clearly about their personal life, or who tries to understand how civilization works, or who tries to figure out what's true in a single domain like nutrition or machine learning; a rationalist is someone who is curious about the general thinking patterns that allows them to think clearly in all such areas, and understand the laws and tools that help th...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Scope Insensitivity, published by Eliezer Yudkowsky on the LessWrong. Once upon a time, three groups of subjects were asked how much they would pay to save 2,000 / 20,000 / 200,000 migrating birds from drowning in uncovered oil ponds. The groups respectively answered $80, $78, and $88.1 This is scope insensitivity or scope neglect: the number of birds saved—the scope of the altruistic action—had little effect on willingness to pay. Similar experiments showed that Toronto residents would pay little more to clean up all polluted lakes in Ontario than polluted lakes in a particular region of Ontario, or that residents of four western US states would pay only 28% more to protect all 57 wilderness areas in those states than to protect a single area.2 People visualize “a single exhausted bird, its feathers soaked in black oil, unable to escape.”3 This image, or prototype, calls forth some level of emotional arousal that is primarily responsible for willingness-to-pay—and the image is the same in all cases. As for scope, it gets tossed out the window—no human can visualize 2,000 birds at once, let alone 200,000. The usual finding is that exponential increases in scope create linear increases in willingness-to-pay—perhaps corresponding to the linear time for our eyes to glaze over the zeroes; this small amount of affect is added, not multiplied, with the prototype affect. This hypothesis is known as “valuation by prototype.” An alternative hypothesis is “purchase of moral satisfaction.” People spend enough money to create a warm glow in themselves, a sense of having done their duty. The level of spending needed to purchase a warm glow depends on personality and financial situation, but it certainly has nothing to do with the number of birds. We are insensitive to scope even when human lives are at stake: Increasing the alleged risk of chlorinated drinking water from 0.004 to 2.43 annual deaths per 1,000—a factor of 600—increased willingness-to-pay from $3.78 to $15.23.4 Baron and Greene found no effect from varying lives saved by a factor of 10.5 A paper entitled “Insensitivity to the value of human life: A study of psychophysical numbing” collected evidence that our perception of human deaths follows Weber’s Law—obeys a logarithmic scale where the “just noticeable difference” is a constant fraction of the whole. A proposed health program to save the lives of Rwandan refugees garnered far higher support when it promised to save 4,500 lives in a camp of 11,000 refugees, rather than 4,500 in a camp of 250,000. A potential disease cure had to promise to save far more lives in order to be judged worthy of funding, if the disease was originally stated to have killed 290,000 rather than 160,000 or 15,000 people per year.6 The moral: If you want to be an effective altruist, you have to think it through with the part of your brain that processes those unexciting inky zeroes on paper, not just the part that gets real worked up about that poor struggling oil-soaked bird. 1 William H. Desvousges et al., Measuring Nonuse Damages Using Contingent Valuation: An Experimental Evaluation of Accuracy, technical report (Research Triangle Park, NC: RTI International, 2010). 2 Daniel Kahneman, “Comments by Professor Daniel Kahneman,” in Valuing Environmental Goods: An Assessment of the Contingent Valuation Method, ed. Ronald G. Cummings, David S. Brookshire, and William D. Schulze, vol. 1.B, Experimental Methods for Assessing Environmental Benefits (Totowa, NJ: Rowman & Allanheld, 1986), 226–235; Daniel L. McFadden and Gregory K. Leonard, “Issues in the Contingent Valuation of Environmental Goods: Methodologies for Data Collection and Analysis,” in Contingent Valuation: A Critical Assessment, ed. Jerry A. Hausman, Contributions to Economic Analysis 220 (New York: North-Holland, 1993), 165–215. 3...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Procedural Knowledge Gaps, published by Alicorn on the LessWrong. I am beginning to suspect that it is surprisingly common for intelligent, competent adults to somehow make it through the world for a few decades while missing some ordinary skill, like mailing a physical letter, folding a fitted sheet, depositing a check, or reading a bus schedule. Since these tasks are often presented atomically - or, worse, embedded implicitly into other instructions - and it is often possible to get around the need for them, this ignorance is not self-correcting. One can Google "how to deposit a check" and similar phrases, but the sorts of instructions that crop up are often misleading, rely on entangled and potentially similarly-deficient knowledge to be understandable, or are not so much instructions as they are tips and tricks and warnings for people who already know the basic procedure. Asking other people is more effective because they can respond to requests for clarification (and physically pointing at stuff is useful too), but embarrassing, since lacking these skills as an adult is stigmatized. (They are rarely even considered skills by people who have had them for a while.) This seems like a bad situation. And - if I am correct and gaps like these are common - then it is something of a collective action problem to handle gap-filling without undue social drama. Supposedly, we're good at collective action problems, us rationalists, right? So I propose a thread for the purpose here, with the stipulation that all replies to gap announcements are to be constructive attempts at conveying the relevant procedural knowledge. No asking "how did you manage to be X years old without knowing that?" - if the gap-haver wishes to volunteer the information, that is fine, but asking is to be considered poor form. (And yes, I have one. It's this: how in the world do people go about the supposedly atomic action of investing in the stock market? Here I am, sitting at my computer, and suppose I want a share of Apple - there isn't a button that says "Buy Our Stock" on their website. There goes my one idea. Where do I go and what do I do there?) Thanks for listening. to help us out with the nonlinear library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some AI research areas and their relevance to existential safety, published by Andrew_Critch on the LessWrong. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. Followed by: What Multipolar Failure Looks Like, and Robust Agent-Agnostic Processes (RAAPs), which provides examples of multi-stakeholder/multi-agent interactions leading to extinction events. Introduction This post is an overview of a variety of AI research areas in terms of how much I think contributing to and/or learning from those areas might help reduce AI x-risk. By research areas I mean “AI research topics that already have groups of people working on them and writing up their results”, as opposed to research “directions” in which I’d like to see these areas “move”. I formed these views mostly pursuant to writing AI Research Considerations for Human Existential Safety (ARCHES). My hope is that my assessments in this post can be helpful to students and established AI researchers who are thinking about shifting into new research areas specifically with the goal of contributing to existential safety somehow. In these assessments, I find it important to distinguish between the following types of value: The helpfulness of the area to existential safety, which I think of as a function of what services are likely to be provided as a result of research contributions to the area, and whether those services will be helpful to existential safety, versus The educational value of the area for thinking about existential safety, which I think of as a function of how much a researcher motivated by existential safety might become more effective through the process of familiarizing with or contributing to that area, usually by focusing on ways the area could be used in service of existential safety. The neglect of the area at various times, which is a function of how much technical progress has been made in the area relative to how much I think is needed. Importantly: The helpfulness to existential safety scores do not assume that your contributions to this area would be used only for projects with existential safety as their mission. This can negatively impact the helpfulness of contributing to areas that are more likely to be used in ways that harm existential safety. The educational value scores are not about the value of an existential-safety-motivated researcher teaching about the topic, but rather, learning about the topic. The neglect scores are not measuring whether there is enough “buzz” around the topic, but rather, whether there has been adequate technical progress in it. Buzz can predict future technical progress, though, by causing people to work on it. Below is a table of all the areas I considered for this post, along with their entirely subjective “scores” I’ve given them. The rest of this post can be viewed simply as an elaboration/explanation of this table: Existing Research Area Social Application Helpfulness to Existential Safety Educational Value 2015 Neglect 2020 Neglect 2030 Neglect Out of Distribution Robustness Zero/ Single 1/10 4/10 5/10 3/10 1/10 Agent Foundations Zero/ Single 3/10 8/10 9/10 8/10 7/10 Multi-agent RL Zero/ Multi 2/10 6/10 5/10 4/10 0/10 Preference Learning Single/ Single 1/10 4/10 5/10 1/10 0/10 Side-effect Minimization Single/ Single 4/10 4/10 6/10 5/10 4/10 Human-Robot Interaction Single/ Single 6/10 7/10 5/10 4/10 3/10 Interpretability in ML Single/ Single 8/10 6/10 8/10 6/10 2/10 Fairness in ML Multi/ Single 6/10 5/10 7/10 3/10 2/10 Computational Social Choice Multi/ Single 7/10 7/10 7/10 5/10 4/10 Accountability in ML Multi/ Multi 8/10 3/10 8/10 7/10 5/10 The research areas are ordered from least-socially-complex to most-socially-complex. This roughly (though imperfectly) correlates with addressing existential safety problems of increa...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing the Alignment Research Center, published by paulfchristiano on the LessWrong. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. (Cross-post from ai-alignment.com) I’m now working full-time on the Alignment Research Center (ARC), a new non-profit focused on intent alignment research. I left OpenAI at the end of January and I’ve spent the last few months planning, doing some theoretical research, doing some logistical set-up, and taking time off. For now it’s just me, focusing on theoretical research. I’m currently feeling pretty optimistic about this work: I think there’s a good chance that it will yield big alignment improvements within the next few years, and a good chance that those improvements will be integrated into practice at leading ML labs. My current goal is to build a small team working productively on theory. I’m not yet sure how we’ll approach hiring, but if you’re potentially interested in joining you can fill out this tiny form to get notified when we’re ready. Over the medium term (and maybe starting quite soon) I also expect to implement and study techniques that emerge from theoretical work, to help ML labs adopt alignment techniques, and to work on alignment forecasting and strategy. Thanks for listening. to help us out with the nonlinear library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Whole Brain Emulation: No Progress on C. elgans After 10 Years, published by niconiconi on the LessWrong. Since the early 21st century, some transhumanist proponents and futuristic researchers claim that Whole Brain Emulation (WBE) is not merely science fiction - although still hypothetical, it's said to be a potentially viable technology in the near future. Such beliefs attracted significant fanfare in tech communities such as LessWrong. In 2011 at LessWrong, jefftk did a literature review on the emulation of a worm, C. elegans, as an indicator of WBE research progress. Because the human brain is so large, and we are so far from having the technical capacity to scan or emulate it, it's difficult to evaluate progress. Some other organisms, however, have much smaller brains: the nematode C. elegans has only 302 cells in its entire nervous system. It is extremely well studied and well understood, having gone through heavy use as a research animal for decades. Since at least 1986 we've known the full neural connectivity of C. elegans, something that would take decades and a huge amount of work to get for humans. At 302 neurons, simulation has been within our computational capacity for at least that long. With 25 years to work on it, shouldn't we be able to 'upload' a nematode by now? There were three research projects from the 1990s to the 2000s, but all are dead-ends that were unable to reach the full research goals, giving a rather pessimistic vision of WBE. However, immediately after the initial publication of that post, LW readers Stephen Larson (slarson) & David Dalrymple (davidad) pointed out in the comments that they were working on it, the two ongoing new projects of their own made the future look promising again. The first was the OpenWorm project, coordinated by slarson. Its goal is to create a complete model and simulation of C. elegans, and to release all tools and data as free and open source software. Implementing a structural model of all 302 C. elegans neurons in the NeuroML description language was an early task completed by the project. The next was another research effort at MIT by davidad. David explained that the OpenWorm project focused on anatomical data from dead worms, but very little data exists from living animals' cells. They can't tell scientists about the relative importance of connections between neurons within the worm's neural system, only that a connection exists. The "connectome" of C. elegans is not actually very helpful information for emulating it. Contrary to popular belief, connectomes are not the biological equivalent of circuit schematics. Connectomes are the biological equivalent of what you'd get if you removed all the component symbols from a circuit schematic and left only the wires. Good luck trying to reproduce the original functionality from that data. What you actually need is to functionally characterize the system's dynamics by performing thousands of perturbations to individual neurons and recording the results on the network, in a fast feedback loop with a very very good statistical modeling framework which decides what perturbation to try next. With optogenetic techniques, we are just at the point where it's not an outrageous proposal to reach for the capability to read and write to anywhere in a living C. elegans nervous system, using a high-throughput automated system. It has some pretty handy properties, like being transparent, essentially clonal, and easily transformed. It also has less handy properties, like being a cylindrical lens, being three-dimensional at all, and having minimal symmetry in its nervous system. However, I am optimistic that all these problems can be overcome by suitably clever optical and computational tricks. In a year or two, he believed an automated device can be built to gather such dat...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Slack, published by Zvi on the LessWrong. Epistemic Status: Reference post. Strong beliefs strongly held after much thought, but hard to explain well. Intentionally abstract. Disambiguation: This does not refer to any physical good, app or piece of software. Further Research (book, recommended but not at all required, take seriously but not literally): The Book of the Subgenius Related (from sam[ ]zdat, recommended but not required, take seriously and also literally, entire very long series also recommended): The Uruk Machine Further Reading (book): Scarcity: Why Having Too Little Means So Much Previously here (not required): Play in Hard Mode, Play in Easy Mode, Out to Get You Leads to (I’ve been scooped! Somewhat.): Sabbath Hard and Go Home An illustrative little game: Carpe Diem: The Problem of Scarcity and Abundance Slack is hard to precisely define, but I think this comes close: Definition: Slack. The absence of binding constraints on behavior. Poor is the person without Slack. Lack of Slack compounds and traps. Slack means margin for error. You can relax. Slack allows pursuing opportunities. You can explore. You can trade. Slack prevents desperation. You can avoid bad trades and wait for better spots. You can be efficient. Slack permits planning for the long term. You can invest. Slack enables doing things for your own amusement. You can play games. You can have fun. Slack enables doing the right thing. Stand by your friends. Reward the worthy. Punish the wicked. You can have a code. Slack presents things as they are without concern for how things look or what others think. You can be honest. You can do some of these things, and choose not to do others. Because you don’t have to. Only with slack can one be a righteous dude. Slack is life. Related Slackness Slack in project management is the time a task can be delayed without causing a delay to either subsequent tasks or project completion time. The amount of time before a constraint binds. Slack the app was likely named in reference to a promise of Slack in the project sense. Slacks as trousers are pants that are actual pants, but do not bind or constrain. Slackness refers to vulgarity in West Indian culture, behavior and music. It also refers to a subgenre of dancehall music with straightforward sexual lyrics. Again, slackness refers to the absence of a binding constraint. In this case, common decency or politeness. A slacker is one who has a lazy work ethic or otherwise does not exert maximum effort. They slack off. They refuse to be bound by what others view as hard constraints. Out to Get You and the Attack on Slack Many things in this world are Out to Get You. Often they are Out to Get You for a lot, usually but not always your time, attention and money. If you Get Got for compact amounts too often, it will add up and the constraints will bind. If you Get Got even once for a non-compact amount, the cost expands until the you have no Slack left. The constraints bind you. You might spend every spare minute and/or dollar on politics, advocacy or charity. You might think of every dollar as a fraction of a third-world life saved. Racing to find a cure for your daughter’s cancer, you already work around the clock. You could have an all-consuming job or be a soldier marching off to war. It could be a quest for revenge, for glory, for love. Or you might spend every spare minute mindlessly checking Facebook or obsessed with your fantasy football league. You cannot relax. Your life is not your own. It might even be the right choice! Especially for brief periods. When about to be run over by a truck or evicted from your house, Slack is a luxury you cannot afford. Extraordinary times call for extraordinary effort. Most times are ordinary. Make an ordinary effort. You Can Afford It No, you can’t. This is the most famou...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Rocket Alignment Problem, published by Eliezer Yudkowsky on the LessWrong. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. The following is a fictional dialogue building off of AI Alignment: Why It’s Hard, and Where to Start. (Somewhere in a not-very-near neighboring world, where science took a very different course.) ALFONSO: Hello, Beth. I’ve noticed a lot of speculations lately about “spaceplanes” being used to attack cities, or possibly becoming infused with malevolent spirits that inhabit the celestial realms so that they turn on their own engineers. I’m rather skeptical of these speculations. Indeed, I’m a bit skeptical that airplanes will be able to even rise as high as stratospheric weather balloons anytime in the next century. But I understand that your institute wants to address the potential problem of malevolent or dangerous spaceplanes, and that you think this is an important present-day cause. BETH: That’s. really not how we at the Mathematics of Intentional Rocketry Institute would phrase things. The problem of malevolent celestial spirits is what all the news articles are focusing on, but we think the real problem is something entirely different. We’re worried that there’s a difficult, theoretically challenging problem which modern-day rocket punditry is mostly overlooking. We’re worried that if you aim a rocket at where the Moon is in the sky, and press the launch button, the rocket may not actually end up at the Moon. ALFONSO: I understand that it’s very important to design fins that can stabilize a spaceplane’s flight in heavy winds. That’s important spaceplane safety research and someone needs to do it. But if you were working on that sort of safety research, I’d expect you to be collaborating tightly with modern airplane engineers to test out your fin designs, to demonstrate that they are actually useful. BETH: Aerodynamic designs are important features of any safe rocket, and we’re quite glad that rocket scientists are working on these problems and taking safety seriously. That’s not the sort of problem that we at MIRI focus on, though. ALFONSO: What’s the concern, then? Do you fear that spaceplanes may be developed by ill-intentioned people? BETH: That’s not the failure mode we’re worried about right now. We’re more worried that right now, nobody can tell you how to point your rocket’s nose such that it goes to the moon, nor indeed any prespecified celestial destination. Whether Google or the US Government or North Korea is the one to launch the rocket won’t make a pragmatic difference to the probability of a successful Moon landing from our perspective, because right now nobody knows how to aim any kind of rocket anywhere. ALFONSO: I’m not sure I understand. BETH: We’re worried that even if you aim a rocket at the Moon, such that the nose of the rocket is clearly lined up with the Moon in the sky, the rocket won’t go to the Moon. We’re not sure what a realistic path from the Earth to the moon looks like, but we suspect it might not be a very straight path, and it may not involve pointing the nose of the rocket at the moon at all. We think the most important thing to do next is to advance our understanding of rocket trajectories until we have a better, deeper understanding of what we’ve started calling the “rocket alignment problem”. There are other safety problems, but this rocket alignment problem will probably take the most total time to work on, so it’s the most urgent. ALFONSO: Hmm, that sounds like a bold claim to me. Do you have a reason to think that there are invisible barriers between here and the moon that the spaceplane might hit? Are you saying that it might get very very windy between here and the moon, more so than on Earth? Both eventualities could be worth preparing for, I suppose, but...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Have epistemic conditions always been this bad?, published by Wei_Dai on the LessWrong. In the last few months, I've gotten increasingly alarmed by leftist politics in the US, and the epistemic conditions that it operates under and is imposing wherever it gains power. (Quite possibly the conditions are just as dire on the right, but they are not as visible or salient to me, because most of the places I can easily see, either directly or through news stories, i.e., local politics in my area, academia, journalism, large corporations, seem to have been taken over by the left.) I'm worried that my alarmism is itself based on confirmation bias, tribalism, catastrophizing, or any number of other biases. (It confuses me that I seem to be the first person to talk much about this on either LW or EA Forum, given that there must be people who have been exposed to the current political environment earlier or to a greater extent than me. On the other hand, all my posts/comments on the subject have generally been upvoted on both forums, and nobody has specifically said that I'm being too alarmist. One possible explanation for nobody else raising an alarm about this is that they're afraid of the current political climate and they're not as "cancel-proof" as I am, or don't feel that they have as much leeway to talk about politics-adjacent issues here as I do.) So I want to ask, have things always been like this, or have they actually gotten significantly worse in recent (say the last 5 or 10) years? If they've always been like this, then perhaps there is less cause for alarm, because (1) if things have always been this bad, and we muddled through them in the past, we can probably continue to muddle through in the future (modulo new x-risks like AGI), and (2) if there is no recent trend towards worsening conditions then we don't need to worry so much about conditions getting worse in the near future. (Obviously if we go back far enough, say to the Middle Ages, then things were almost certainly as bad or worse, but I'm worried about more recent trends.) If there are other reasons to not be very alarmed aside from the past being just as bad, please feel free to talk about those as well. But in case one of those reasons is "why be alarmed when there's little that can be done about it", my answer is that being alarmed motivates one to try to understand what is going on, which can help with (1) deciding personal behavior now in expectation of future changes (for example if there's going to be a literal Cultural Revolution in the future, then I need to be really really careful what I say today), (2) planning x-risk strategy, and (3) defending LW/EA from either outside attack or similar internal dynamics. Here's some of what I've observed so far, which has led me to my current epistemic state: In local politics, "asking for evidence of oppression is a form of oppression" or even more directly "questioning the experiences of a marginalized group that you don't belong to is not allowed and will result in a ban" has apparently been an implicit norm, and is being made increasingly explicit. (E.g., I saw a FB group explicitly codifying this in their rules.) As a result, anyone can say "Policy X or Program Y oppresses Group Z and must be changed" and nobody can argue against that, except by making the same kind of claim based on a different identity group, and then it comes down to which group is recognized as being more privileged or oppressed by the current orthodoxy. (If someone does belong to Group Z and wants to argue against the claim on that basis, they'll be dismissed based on "being tokenized" or "internalized oppression".) In academia, even leftist professors are being silenced or kicked out on a regular basis for speaking out against an ever-shifting "party line". ("Party line" is in q...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The case for aligning narrowly superhuman models, published by Ajeya Cotra on the LessWrong. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. I wrote this post to get people’s takes on a type of work that seems exciting to me personally; I’m not speaking for Open Phil as a whole. Institutionally, we are very uncertain whether to prioritize this (and if we do where it should be housed and how our giving should be structured). We are not seeking grant applications on this topic right now. Thanks to Daniel Dewey, Eliezer Yudkowsky, Evan Hubinger, Holden Karnofsky, Jared Kaplan, Mike Levine, Nick Beckstead, Owen Cotton-Barratt, Paul Christiano, Rob Bensinger, and Rohin Shah for comments on earlier drafts. A genre of technical AI risk reduction work that seems exciting to me is trying to align existing models that already are, or have the potential to be, “superhuman”[1] at some particular task (which I’ll call narrowly superhuman models).[2] I don’t just mean “train these models to be more robust, reliable, interpretable, etc” (though that seems good too); I mean “figure out how to harness their full abilities so they can be as useful as possible to humans” (focusing on “fuzzy” domains where it’s intuitively non-obvious how to make that happen). Here’s an example of what I’m thinking of: intuitively speaking, it feels like GPT-3 is “smart enough to” (say) give advice about what to do if I’m sick that’s better than advice I’d get from asking humans on Reddit or Facebook, because it’s digested a vast store of knowledge about illness symptoms and remedies. Moreover, certain ways of prompting it provide suggestive evidence that it could use this knowledge to give helpful advice. With respect to the Reddit or Facebook users I might otherwise ask, it seems like GPT-3 has the potential to be narrowly superhuman in the domain of health advice. But GPT-3 doesn’t seem to “want” to give me the best possible health advice -- instead it “wants” to play a strange improv game riffing off the prompt I give it, pretending it’s a random internet user. So if I want to use GPT-3 to get advice about my health, there is a gap between what it’s capable of (which could even exceed humans) and what I can get it to actually provide me. I’m interested in the challenge of: How can we get GPT-3 to give “the best health advice it can give” when humans[3] in some sense “understand less” about what to do when you’re sick than GPT-3 does? And in that regime, how can we even tell whether it’s actually “doing the best it can”? I think there are other similar challenges we could define for existing models, especially large language models. I’m excited about tackling this particular type of near-term challenge because it feels like a microcosm of the long-term AI alignment problem in a real, non-superficial sense. In the end, we probably want to find ways to meaningfully supervise (or justifiably trust) models that are more capable than ~all humans in ~all domains.[4] So it seems like a promising form of practice to figure out how to get particular humans to oversee models that are more capable than them in specific ways, if this is done with an eye to developing scalable and domain-general techniques. I’ll call this type of project aligning narrowly superhuman models. In the rest of this post, I: Give a more detailed description of what aligning narrowly superhuman models could look like, what does and doesn’t “count”, and what future projects I think could be done in this space (more). Explain why I think aligning narrowly superhuman models could meaningfully reduce long-term existential risk from misaligned AI (more). Lay out the potential advantages that I think this work has over other types of AI alignment research: (a) conceptual thinking, (b) demos in small-scal...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Privileging the Question, published by Qiaochu_Yuan on the LessWrong. Related to: Privileging the Hypothesis Remember the exercises in critical reading you did in school, where you had to look at a piece of writing and step back and ask whether the author was telling the whole truth? If you really want to be a critical reader, it turns out you have to step back one step further, and ask not just whether the author is telling the truth, but why he's writing about this subject at all. -- Paul Graham There's an old saying in the public opinion business: we can't tell people what to think, but we can tell them what to think about. -- Doug Henwood Many philosophers—particularly amateur philosophers, and ancient philosophers—share a dangerous instinct: If you give them a question, they try to answer it. -- Eliezer Yudkowsky Here are some political questions that seem to commonly get discussed in US media: should gay marriage be legal? Should Congress pass stricter gun control laws? Should immigration policy be tightened or relaxed? These are all examples of what I'll call privileged questions (if there's an existing term for this, let me know): questions that someone has unjustifiably brought to your attention in the same way that a privileged hypothesis unjustifiably gets brought to your attention. The questions above are probably not the most important questions we could be answering right now, even in politics (I'd guess that the economy is more important). Outside of politics, many LWers probably think "what can we do about existential risks?" is one of the most important questions to answer, or possibly "how do we optimize charity?" Why has the media privileged these questions? I'd guess that the media is incentivized to ask whatever questions will get them the most views. That's a very different goal from asking the most important questions, and is one reason to stop paying attention to the media. The problem with privileged questions is that you only have so much attention to spare. Attention paid to a question that has been privileged funges against attention you could be paying to better questions. Even worse, it may not feel from the inside like anything is wrong: you can apply all of the epistemic rationality in the world to answering a question like "should Congress pass stricter gun control laws?" and never once ask yourself where that question came from and whether there are better questions you could be answering instead. I suspect this is a problem in academia too. Richard Hamming once gave a talk in which he related the following story: Over on the other side of the dining hall was a chemistry table. I had worked with one of the fellows, Dave McCall; furthermore he was courting our secretary at the time. I went over and said, "Do you mind if I join you?" They can't say no, so I started eating with them for a while. And I started asking, "What are the important problems of your field?" And after a week or so, "What important problems are you working on?" And after some more time I came in one day and said, "If what you are doing is not important, and if you don't think it is going to lead to something important, why are you at Bell Labs working on it?" I wasn't welcomed after that; I had to find somebody else to eat with! Academics answer questions that have been privileged in various ways: perhaps the questions their advisor was interested in, or the questions they'll most easily be able to publish papers on. Neither of these are necessarily well-correlated with the most important questions. So far I've found one tool that helps combat the worst privileged questions, which is to ask the following counter-question: What do I plan on doing with an answer to this question? With the worst privileged questions I frequently find that the answer is "nothing," som...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The 5-Second Level, published by Eliezer Yudkowsky on the LessWrong. To develop methods of teaching rationality skills, you need to learn to focus on mental events that occur in 5 seconds or less. Most of what you want to teach is directly on this level; the rest consists of chaining together skills on this level. As our first example, let's take the vital rationalist skill, "Be specific." Even with people who've had moderate amounts of exposure to Less Wrong, a fair amount of my helping them think effectively often consists of my saying, "Can you give me a specific example of that?" or "Can you be more concrete?" A couple of formative childhood readings that taught me to be specific: "What is meant by the word red?" "It's a color." "What's a color?" "Why, it's a quality things have." "What's a quality?" "Say, what are you trying to do, anyway?" You have pushed him into the clouds. If, on the other hand, we habitually go down the abstraction ladder to lower levels of abstraction when we are asked the meaning of a word, we are less likely to get lost in verbal mazes; we will tend to "have our feet on the ground" and know what we are talking about. This habit displays itself in an answer such as this: "What is meant by the word red?" "Well, the next time you see some cars stopped at an intersection, look at the traffic light facing them. Also, you might go to the fire department and see how their trucks are painted." -- S. I. Hayakawa, Language in Thought and Action and: "Beware, demon!" he intoned hollowly. "I am not without defenses." "Oh yeah? Name three." -- Robert Asprin, Another Fine Myth And now, no sooner does someone tell me that they want to "facilitate communications between managers and employees" than I say, "Can you give me a concrete example of how you would do that?" Hayakawa taught me to distinguish the concrete and the abstract; and from that small passage in Asprin, I picked up the dreadful personal habit of calling people's bluffs, often using the specific phrase, "Name three." But the real subject of today's lesson is how to see skills like this on the 5-second level. And now that we have a specific example in hand, we can proceed to try to zoom in on the level of cognitive events that happen in 5 seconds or less. Over-abstraction happens because it's easy to be abstract. It's easier to say "red is a color" than to pause your thoughts for long enough to come up with the example of a stop sign. Abstraction is a path of least resistance, a form of mental laziness. So the first thing that needs to happen on a timescale of 5 seconds is perceptual recognition of highly abstract statements unaccompanied by concrete examples, accompanied by an automatic aversion, an ick reaction - this is the trigger which invokes the skill. Then, you have actionable stored procedures that associate to the trigger. And "come up with a concrete example" is not a 5-second-level skill, not an actionable procedure, it doesn't transform the problem into a task. An actionable mental procedure that could be learned, stored, and associated with the trigger would be "Search for a memory that instantiates the abstract statement", or "Try to come up with hypothetical examples, and then discard the lousy examples your imagination keeps suggesting, until you finally have a good example that really shows what you were originally trying to say", or "Ask why you were making the abstract statement in the first place, and recall the original mental causes of your making that statement to see if they suggest something more concrete." Or to be more specific on the last mental procedure: Why were you trying to describe redness to someone? Did they just run a red traffic light? (And then what kind of exercise can you run someone through, which will get them to distinguish red traffic lights fr...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is:The 3 Books Technique for Learning a New Skilll, published by Matt Goldenberg on the LESSWRONG. When I'm learning a new skill, there's a technique I often use to quickly gain the basics of the new skill without getting drowned in the plethora of resources that exist. I've found that just 3 resources that cover the skill from 3 separate viewpoints(along with either daily practice or a project) is enough to quickly get all the pieces I need to learn the new skill. I'm partial to books, so I've called this The 3 Books Technique, but feel free to substitute books for courses, mentors, or videos as needed. The "What" Book The "What" book is used as reference material. It should be a thorough resource that gives you a broad overview of your skill. If you run into a novel situation, you should be able to go to this book and get the information you need. It covers the "surface" section of the learning model from nature pictured above. Positive reviews of this book should contain phrases like "Thorough" and "Got me out of a pinch more than once." Negative reviews of this book should talk about "overwhelming" and "didn't know where to start." The "How" Book The "How" Book explains the step-by-step, nuts and bolts of how to put the skill into practice. It often contains processes, tools, and steps. It covers the "deep" part of the learning model covered above. Positive reviews of this book should talk about "Well structured" and "Clearly thought out." Negative reviews should mention it being "too rote" or "not enough theory." The "Why" Book The "WHY" book explains the mindset and intuitions behind the skill. It tries to get into the authors head and lets you understand what to do in novel situations. It should cover the "transfer" part of the learning model above. Positive reviews of this book should talk about "gaining intuitions" or "really understanding". Negative reviews should contain phrases like "not practical" or "still don't know what steps to take." The Project or Practice Once I have these 3 resources, I'll choose a single project or a daily practice that allows me to practice the skills from the "How" book and the mindsets from the "Why" book. If I get stuck, I'll use the "What" book to help me. Examples Overcoming Procrastination "What" Book: The Procrastination Equation by Piers Steel "How" Book: The Now Habit by Neil Fiore "Why" Book: The Replacing Guilt blog sequence by Nate Soares Project or Practice: Five pomodoros every day where I deliberately use the tools from the now habit and the mindsets from replacing guilt. If I find myself stuck, I'll choose from the plethora of techniques in the Procrastination Equation. Learning Calculus "What" Book: A First Course in Calculus by Serge Lange "How" Book: The Khan Academy series on Calculus "Why" Book: The Essence of Calculus Youtube series by 3blue1brown Project or Practice: Daily practice of the Khan Academy calculus exercises. Conclusion This is a simple technique that I've found very helpful in systematizing my learning process. I would be particularly interested in other skills you've learned and the 3 books you would recommend for those skills. Thanks for listening. to help us out with the nonlinear library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Realism about rationality, published Richard_Ngo on the LESSWRONG. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. This is a linkpost for http://thinkingcomplete.blogspot.com/2018/09/rational-and-real.html Epistemic status: trying to vaguely gesture at vague intuitions. A similar idea was explored here under the heading "the intelligibility of intelligence", although I hadn't seen it before writing this post. As of 2020, I consider this follow-up comment to be a better summary of the thing I was trying to convey with this post than the post itself. There’s a mindset which is common in the rationalist community, which I call “realism about rationality” (the name being intended as a parallel to moral realism). I feel like my skepticism about agent foundations research is closely tied to my skepticism about this mindset, and so in this essay I try to articulate what it is. Humans ascribe properties to entities in the world in order to describe and predict them. Here are three such properties: "momentum", "evolutionary fitness", and "intelligence". These are all pretty useful properties for high-level reasoning in the fields of physics, biology and AI, respectively. There's a key difference between the first two, though. Momentum is very amenable to formalisation: we can describe it using precise equations, and even prove things about it. Evolutionary fitness is the opposite: although nothing in biology makes sense without it, no biologist can take an organism and write down a simple equation to define its fitness in terms of more basic traits. This isn't just because biologists haven't figured out that equation yet. Rather, we have excellent reasons to think that fitness is an incredibly complicated "function" which basically requires you to describe that organism's entire phenotype, genotype and environment. In a nutshell, then, realism about rationality is a mindset in which reasoning and intelligence are more like momentum than like fitness. It's a mindset which makes the following ideas seem natural: The idea that there is a simple yet powerful theoretical framework which describes human intelligence and/or intelligence in general. (I don't count brute force approaches like AIXI for the same reason I don't consider physics a simple yet powerful description of biology). The idea that there is an “ideal” decision theory. The idea that AGI will very likely be an “agent”. The idea that Turing machines and Kolmogorov complexity are foundational for epistemology. The idea that, given certain evidence for a proposition, there's an "objective" level of subjective credence which you should assign to it, even under computational constraints. The idea that Aumann's agreement theorem is relevant to humans. The idea that morality is quite like mathematics, in that there are certain types of moral reasoning that are just correct. The idea that defining coherent extrapolated volition in terms of an idealised process of reflection roughly makes sense, and that it converges in a way which doesn’t depend very much on morally arbitrary factors. The idea that having having contradictory preferences or beliefs is really bad, even when there’s no clear way that they’ll lead to bad consequences (and you’re very good at avoiding dutch books and money pumps and so on). To be clear, I am neither claiming that realism about rationality makes people dogmatic about such ideas, nor claiming that they're all false. In fact, from a historical point of view I’m quite optimistic about using maths to describe things in general. But starting from that historical baseline, I’m inclined to adjust downwards on questions related to formalising intelligent thought, whereas rationality realism would endorse adjusting upwards. This essay is primarily intended to explain...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The EMH Aten't Dead, published by Richard Meadows on the LessWrong. Cross-posting from my personal blog, but written primarily for Less Wrong after recent discussion here. There are whispers that the Efficient-Market Hypothesis is dead. Eliezer's faith has been shaken. Scott says EMH may have been the real victim of the coronavirus. The EMH states that “asset prices reflect all available information”. The direct implication is that if you don’t have any non-available information, you shouldn’t expect to be able to beat the market, except by chance. But some people were able to preempt the corona crash, without any special knowledge! Jacob mentioned selling some of his stocks before the market reacted. Wei Dai bought out-of-the-money 'put' options, and took a very handsome profit. Others shorted the market. These people were reading the same news and reports as everyone else. They profited on the basis of public information that should have been priced in. And so, the EMH is dead, or dying, or at the very least, has a very nasty-sounding cough. I think that rumours of the death of efficient markets have been greatly exaggerated. It seems to me the EMH is very much alive-and-kicking, and the recent discussion often involves common misunderstandings that it might be helpful to iron out. This necessarily involves pissing on people's parade, which is not much fun. So it's important to say upfront that although I don't know Wei Dai, he is no doubt a brilliant guy, that Jacob is my favourite blogger in the diaspora, that I would give my left testicle to have Scott's writing talent and ridiculous work ethic, that Eliezer is a legend whose work I have personally benefited from greatly, etc. But in the spirit of the whole rationality thing, I want to gently challenge what looks more like a case of 'back-slaps for the boys' than a death knell for efficient markets. First: how the heck did the market get the coronavirus so wrong? The Great Coronavirus Trade Lots of people initially underreacted to COVID-19. We are only human. But the stockmarket is not only human—it’s meant to be better than this. Here’s Scott, in A Failure, But Not of Prediction: The stock market is a giant coordinated attempt to predict the economy, and it reached an all-time high on February 12, suggesting that analysts expected the economy to do great over the following few months. On February 20th it fell in a way that suggested a mild inconvenience to the economy, but it didn’t really start plummeting until mid-March – the same time the media finally got a clue. These aren’t empty suits on cable TV with no skin in the game. These are the best predictive institutions we have, and they got it wrong. But. this isn't how it went down. As AllAmericanBreakfast and others pointed out in the comments, the market started reacting in the last week of February, with news headlines directly linking the decline to the ‘coronavirus’. By the time we get to mid-March, we’re not far off the bottom. (You can confirm this for yourself in a few seconds by looking at a chart of the relevant time period.) EDIT: Scott has explained his rationale here. Although I still think his version of events is incorrect as phrased, I want to make it clear I am not accusing him of deliberately massaging the data or any other such shenanigans, and the next paragraph about revisionist history etc was only meant to be a general observation about how people responded. My apologies to Scott for the unclear wording, as well any perceived slight against his very good reputation. For whatever reason, COVID-19 seems to be a magnet for revisionist history and/or wishful thinking. In other comments under the same post, the notion that people from our ‘tribe’ did especially well also comes under serious question—in fact, it looks like many of the names ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: DARPA Digital Tutor: Four Months to Total Technical Expertise?, published by JohnBuridan on the LessWrong. DARPA spent a few million dollars around 2009 to create the world’s best digital tutoring system for IT workers in the Navy. I am going to explain their results, the system itself, possible limitations, and where to go from here. It is a truth universally acknowledged that a single nerd having read Ender’s Game must be in want of the Fantasy Game. The great draw of the Fantasy Game is that the game changes with the player and reflects the needs of the learner growing dynamically with him/her. This dream of the student is best realized in the world of tutoring, which while not as fun, is known to be very, very effective. Individualized instruction can make students jump to the 98 percentile compared to non tutored students. DARPA poked at this idea with their Digital Tutor trying to answer this question: How close to the expertise and knowledge base of well-experienced IT experts can we get new recruits in 16 weeks using a digital tutoring system? I will say the results upfront, but before I do, I want to do two things. First pause to note the audacity of the project. Some project manager thought, “I bet we can design a system for training that is as good as 5 years on the job experience.” This is astoundingly ambitious. I love it! Second a few caveats. Caveat 1) Don’t be confused. Technical training is not the same as education. The goals in education are not merely to learn some technical skills like reading, writing, and arithmetic. Getting any system to usefully measure things like inculturation, citizenship, moral uprightness, and social mores is not yet something any system can do, let alone a digital system. Caveat 2) Online classes have notoriously high attrition rates, drop rates, and no shows. Caveat 3) Going in we should not expect the digital tutor to be as good as a human tutor. A human tutor likely can catch nuances that a digital tutor, no matter how good cannot. Caveat 4) Language processing technology, chat bots, and AI systems are significantly better in 2020 than they were 2009, so we should be forgiving if the DARPA IT program is not as good as it would be if the experiment were rerun today. All these caveats, I think should give us a reason to adjust our mental score of the Digital Tutor a few clicks upward and give it some credit. However, this charitable read of the Digital Tutor that I started with when reading the paper turned out to be unnecessary. The Digital Tutor students outperformed traditionally taught students and field experts in solving IT problems on the final assessment. They did not merely meet the goal of being as good after 16 weeks as experts in the field, but they actually outperformed them. This is a ridiculously positive outcome, and we need to look closely to see what parts of this story are believable and make some conjectures for why this happened and some bets about whether it will replicate. The Digital Tutor Experience We will start with the Digital Tutor student experience. This will give us the context we need to understand the results. Students (cadets?) were on the same campus and in classrooms with their computers which ran the Digital Tutor program. A uniformed Naval officer proctored each day for their 16 week course. The last ‘period’ of the day was a study hall with occasional hands-on practice sessions led by the Naval officer. This set-up is important for a few reasons, in my opinion. There is a shared experience among the students of working on IT training, plus the added accountability of a proctor keeps everyone on task. This social aspect is very important and powerful compared to the dissipation experienced by the lone laborer at home on the computer. This social structure completely counteracts ca...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.. This is: Notes on The Anthropology of Childhood, published by juliawise on the LessWrong. Crossposted from The Whole Sky. I read David Lancy’s “The Anthropology of Childhood: Cherubs, Chattel, and Changelings” and highlighted some passages. A lot of passages, it turns out. [content note: discussion of abortion and infanticide, including infanticide of children with disabilities, in “Life and Death” section but not elsewhere] I was a sociology major and understood anthropology to be basically “like sociology, but in Papua New Guinea.” This is the first cultural anthropology book I’ve read, and that was pretty much right. I found it very accessible as a first dive into anthropology. The first chapter summarizes all his points without the examples, so you could try that if you want to get the gist without reading the whole book. I enjoyed it and would recommend it to people interested in this topic. A few things that shifted for me: I feel less obliged to entertain my children and intervene in their conflicts. We don’t live with a tribe of extended family, but my two children play with each other all day, which is how most people throughout time have spent their childhoods. Lancy isn’t a child development expert, but I buy his argument that handling conflict (for example about the rules of a game) is a skill children need to learn, rather than having conflicts always mediated by adults. Even though it doesn’t change anything concrete, I feel some relief that not having endless patience for toddlers seems to be normal. Except where families were very isolated, it’s not normal in traditional societies for one or two adults to watch their own children all day every day. And childcare has traditionally looked mostly like “being sure they don’t hurt themselves too badly.” It surprised me that childcare by non-parents was so common. Some more modern views treat women’s childcare work as basically free, but traditional cultures have valued women’s labor enough that the society wants to free up their time from childcare. It was striking to me that the expectation that stay-at-home mothers will be responsible for all childcare was a relatively short historical blip. But of course, having childcare done by teenagers and grandmothers requires that those people’s time be available, which usually isn’t the reality we live in. I was surprised at how apparently universal it is for fathers to be uninvolved. I expect they're typically involved in providing food and other material resources, but that wasn't emphasized in this book. I’m a little unclear on how valid Lancy’s conclusions are or how much data they’re based on. It seems like an anthropologist could squint at a society and see all kinds of things that someone with a different ideology wouldn’t see. Big caveat that what Lancy is describing is traditional, non-industrialized societies where children are expected to learn how to fit into the appropriate role in their village, not to develop as an individual or do anything different from what their parents and ancestors did. He stresses that traditional childrearing practices are very poor preparation for school. Given that I want my children to learn things I don’t know, to think analytically, etc, the way I approach learning is very different from how traditional societies approach it. Lancy periodically complains about how much money Western families spend on fertility treatments, medical care for premature infants, etc. He argues that the same money could be used to provide adequate nutrition for many more children in the societies he’s studied. I’m sympathetic, but assuming that families would donate this money if they weren’t spending it to have a baby is not realistic. I see cutting luxury spending as a much more feasible way that people might do some redistribution. And now, my no...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My attempt to explain Looking, insight meditation, and enlightenment in non-mysterious terms, published by Kaj_Sotala on the LessWrong. Epistemic status: pretty confident. Based on several years of meditation experience combined with various pieces of Buddhist theory as popularized in various sources, including but not limited to books like The Mind Illuminated, Mastering the Core Teachings of the Buddha, and The Seeing That Frees; also discussions with other people who have practiced meditation, and scatterings of cognitive psychology papers that relate to the topic. The part that I’m the least confident of is the long-term nature of enlightenment; I’m speculating on what comes next based on what I’ve experienced, but have not actually had a full enlightenment. I also suspect that different kinds of traditions and practices may produce different kinds of enlightenment states. While I liked Valentine’s recent post on kensho and its follow-ups a lot, one thing that I was annoyed by were the comments that the whole thing can’t be explained from a reductionist, third-person perspective. I agree that such an explanation can’t produce the necessary mental changes that the explanation is talking about. But it seemed wrong to me to claim that all of this would be somehow intrinsically mysterious and impossible to explain on such a level that would give people at least an intellectual understanding of what Looking and enlightenment and all that are. Especially not after I spoke to Val and realized that hey, I actually do know how to Look, and that thing he’s calling kensho, that’s happened to me too. (Note however that kensho is a Zen term and I'm unfamiliar with Zen; I don't want to use a term which might imply that I was going with whatever theoretical assumptions Zen might have, so I will just talk about “my experience” when it comes up.) So here is my attempt to give an explanation. I don’t know if I’ve succeeded, but here goes anyway. One of my key concepts is going to be cognitive fusion. Cognitive fusion is a term from Acceptance and Commitment Therapy (ACT), which refers to a person “fusing together” with the content of a thought or emotion, so that the content is experienced as an objective fact about the world rather than as a mental construct. The most obvious example of this might be if you get really upset with someone else and become convinced that something was all their fault (even if you had actually done something blameworthy too). In this example, your anger isn’t letting you see clearly, and you can’t step back from your anger to question it, because you have become “fused together” with it and experience everything in terms of the anger’s internal logic. Another emotional example might be feelings of shame, where it’s easy to experience yourself as a horrible person and feel that this is the literal truth, rather than being just an emotional interpretation. Cognitive fusion isn’t necessarily a bad thing. If you suddenly notice a car driving towards you at a high speed, you don’t want to get stuck pondering about how the feeling of danger is actually a mental construct produced by your brain. You want to get out of the way as fast as possible, with minimal mental clutter interfering with your actions. Likewise, if you are doing programming or math, you want to become at least partially fused together with your understanding of the domain, taking its axioms as objective facts so that you can focus on figuring out how to work with those axioms and get your desired results. On the other hand, even when doing math, it can sometimes be useful to question the axioms you’re using. In programming, taking the guarantees of your abstractions as literal axioms can also lead to trouble. And while it is useful to perceive something as objectively life-threatening and ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Act of Charity , published by jessicata on the LessWrong. (Cross-posted from my blog) The stories and information posted here are artistic works of fiction and falsehood. Only a fool would take anything posted here as fact. Anonymous Act I. Carl walked through the downtown. He came across a charity stall. The charity worker at the stall called out, "Food for the Africans. Helps with local autonomy and environmental sustainability. Have a heart and help them out." Carl glanced at the stall's poster. Along with pictures of emaciated children, it displayed infographics about how global warming would cause problems for African communities' food production, and numbers about how easy it is to help out with money. But something caught Carl's eye. In the top left, in bold font, the poster read, "IT IS ALL AN ACT. ASK FOR DETAILS." Carl: "It's all an act, huh? What do you mean?" Worker: "All of it. This charity stall. The information on the poster. The charity itself. All the other charities like us. The whole Western idea of charity, really." Carl: "Care to clarify?" Worker: "Sure. This poster contains some correct information. But a lot of it is presented in a misleading fashion, and a lot of it is just lies. We designed the poster this way because it fits with people's idea is of a good charity they should give money to. It's a prop in the act." Carl: "Wait, the stuff about global warming and food production is a lie?" Worker: "No, that part is actually true. But in context we're presenting it as some kind of imminent crisis that requires an immediate infusion of resources, when really it's a very long-term problem that will require gradual adjustment of agricultural techniques, locations, and policies." Carl: "Okay, that doesn't actually sound like more of a lie than most charities tell." Worker: "Exactly! It's all an act." Carl: "So why don't you tell the truth anyway?" Worker: "Like I said before, we're trying to fit with people's idea of what a charity they should give money to looks like. More to the point, we want them to feel compelled to give us money. And they are compelled by some acts, but not by others. The idea of an immediate food crisis creates more moral and social pressure towards immediate action, than the idea that there will be long-term agricultural problems that require adjustments. Carl: "That sounds...kind of scammy?" Worker: "Yes, you're starting to get it! The act is about violence! It's all violence!" Carl: "Now hold on, that seems like a false equivalence. Even if they were scammed by you, they still gave you money of their own free will." Worker: "Most people, at some level, know we're lying to them. Their eyes glaze over 'IT IS ALL AN ACT' as if it were just a regulatory requirement to put this on charity posters. So why would they give money to a charity that lies to them? Why do you think?" Carl: "I'm not nearly as sure as you that they know this! Anyway, even if they know at some level it's a lie, that doesn't mean they consciously know, so to their conscious mind it seems like being completely heartless." Worker: "Exactly, it's emotional blackmail. I even say 'Have a heart and help them out'. So if they don't give us money, there's a really convenient story that says they're heartless, and a lot of them will even start thinking about themselves that way. Having that story told about them opens them up to violence." Carl: "How?" Worker: "Remember Martin Shkreli?" Carl: "Yeah, that asshole who jacked up the Daraprim prices." Worker: "Right. He ended up going to prison. Nominally, it was for securities fraud. But it's not actually clear that whatever security fraud he did was worse than what others in his industry were doing. Rather, it seems likely that he was especially targeted because he was a heartless asshole." Carl: "But he still brok...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Birds, Brains, Planes, and AI: Against Appeals to the Complexity/Mysteriousness/Efficiency of the Brain, published by Daniel Kokotajlo on the LessWrong. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. [Epistemic status: Strong opinions lightly held, this time with a cool graph.] I argue that an entire class of common arguments against short timelines is bogus, and provide weak evidence that anchoring to the human-brain-human-lifetime milestone is reasonable. In a sentence, my argument is that the complexity and mysteriousness and efficiency of the human brain (compared to artificial neural nets) is almost zero evidence that building TAI will be difficult, because evolution typically makes things complex and mysterious and efficient, even when there are simple, easily understood, inefficient designs that work almost as well (or even better!) for human purposes. In slogan form: If all we had to do to get TAI was make a simple neural net 10x the size of my brain, my brain would still look the way it does. The case of birds & planes illustrates this point nicely. Moreover, it is also a precedent for several other short-timelines talking points, such as the human-brain-human-lifetime (HBHL) anchor. Plan: Illustrative Analogy Exciting Graph Analysis Extra brute force can make the problem a lot easier Evolution produces complex mysterious efficient designs by default, even when simple inefficient designs work just fine for human purposes. What’s bogus and what’s not Example: Data-efficiency Conclusion Appendix 1909 French military plane, the Antionette VII. By Deep silence (Mikaël Restoux) - Own work (Bourget museum, in France), CC BY 2.5, Illustrative Analogy AI timelines, from our current perspective Flying machine timelines, from the perspective of the late 1800’s: Shorty: Human brains are giant neural nets. This is reason to think we can make human-level AGI (or at least AI with strategically relevant skills, like politics and science) by making giant neural nets. Shorty: Birds are winged creatures that paddle through the air. This is reason to think we can make winged machines that paddle through the air. Longs: Whoa whoa, there are loads of important differences between brains and artificial neural nets: [what follows is a direct quote from the objection a friend raised when reading an early draft of this post!] - During training, deep neural nets use some variant of backpropagation. My understanding is that the brain does something else, closer to Hebbian learning. (Though I vaguely remember at least one paper claiming that maybe the brain does something that's similar to backprop after all.) - It's at least possible that the wiring diagram of neurons plus weights is too coarse-grained to accurately model the brain's computation, but it's all there is in deep neural nets. If we need to pay attention to glial cells, intracellular processes, different neurotransmitters etc., it's not clear how to integrate this into the deep learning paradigm. - My impression is that several biological observations on the brain don't have a plausible analog in deep neural nets: growing new neurons (though unclear how important it is for an adult brain), "repurposing" in response to brain damage, . Longs: Whoa whoa, there are loads of important differences between birds and flying machines: - Birds paddle the air by flapping, whereas current machine designs use propellers and fixed wings. - It’s at least possible that the anatomical diagram of bones, muscles, and wing surfaces is too coarse-grained to accurately model how a bird flies, but that’s all there is to current machine designs (replacing bones with struts and muscles with motors, that is). If we need to pay attention to the percolation of air through and between feathers, micro-eddies in t...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Maximizing Your Donations via a Job, published by Alexei on the LessWrong. In November of 2012 I set a goal for myself: find the most x-risk reducing role I can fill. At first I thought it would be by working directly with MIRI, but after a while it became clear that I could contribute more by simply donating. So my goal became: find the highest paying job, so I can donate lots of money to CFAR and MIRI. A little bit of background on me. Started programming in 2000. Graduated in 2009 with Bachelor's in computer science. Worked for about a year and a half at a game company. Then did my own game startup for about a year. Then moved to the bay area and joined a game startup here, which was acquired 10 months later. Worked a bit at the new company and then left. So, just under four years of professional programming experience, but primarily in the game industry. Almost no leadership / managerial experience, aside from the startup I did where I hired freelancers. Below is my experience of finding a software engineering job in the Silicon Valley. If you are not an engineer or not in the Silicon Valley, I think you'll still find a lot of useful information here. Pre-game Before sending out my resume, I spent about a month preparing. I read Intro to Algorithms, which was very good overall, but not a huge help in preparing for interviews.[1] I read Cracking the Coding Interview, which was extremely helpful. (If you read only one book to prepare, make it this one.) The book has a lot of questions that are similar to the ones you'll actually see during interviews. I also did TopCoder problems, which were pretty helpful as well.[2] Looking back, I wish I spent more time finding actual interview questions online and doing more of those (that's why CCI book was so helpful). After several weeks of preparation, I compiled a long list of companies I was going to apply to. I checked on GlassDoor to see what kind of salary I could expect at each one. I then rated all the companies. Companies with low salaries and poor personal fit received the lowest rating. I started by applying to companies with the lowest ratings. This way I could use them as practice for the companies I thought would actually make a competitive offer. This was the right move and worked very well. (Another friend of mine did the same approach with good results as well.) Remember, you are not just doing those interviews to practice the coding problems, you are practicing pitching yourself as well. Interviewing with a company Standard procedure for applying to a tech company: 1. Send them your resume. Proofread your resume. Let your friends proofread it. Make sure there are only relevant things on it. When I applied to tech companies, I removed a lot of game-specific things from my resume. When I applied to companies that did 3D graphics, I made sure I had all my 3D graphics experience listed. I ended up with two version of my resume. Have your resume in DOC, PDF, and TXT formats. This way you'll always have the right one when you upload / paste it. For a few companies, I had a friend or friend of a friend who referred me. This REALLY HELPS in two ways: 1) your resume will be processed a lot faster, 2) if your friend is a great engineer/employee, you'll be taken a lot more seriously, and the company will fight for you a lot harder. 2. You'll get an email from the recruiter and setup a time to speak, where you'll talk about yourself, what you've done, why you are interested in their company, and so on. You can and should ask them questions as well. When you start getting multiple calls each day, make sure you know who is calling. There is nothing worse than talking about the challenges of streaming music to a car sharing startup. (True story.) Read about the company on Wikipedia before the call. Know the basic stuff. L...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Motive Ambiguity, published by Zvi on the LessWrong. Central theme in: Immoral Mazes Sequence, but this generalizes. When looking to succeed, pain is not the unit of effort, and money is a, if not the, unit of caring. One is not always looking to succeed. Here is a common type of problem. You are married, and want to take your spouse out to a romantic dinner. You can choose the place your spouse loves best, or the place you love best. A middle manager is working their way up the corporate ladder, and must choose how to get the factory to improve its production of widgets. A middle manager must choose how to improve widget production. He can choose a policy that improperly maintains the factory and likely eventually it poisons the water supply, or a policy that would prevent that but at additional cost. A politician can choose between a bill that helps the general population, or a bill that helps their biggest campaign contributor. A start-up founder can choose between building a quality product without technical debt, or creating a hockey stick graph that will appeal to investors. You can choose to make a gift yourself. This would be expensive in terms of your time and be lower quality, but be more thoughtful and cheaper. Or you could buy one in the store, which would be higher quality and take less time, but feel generic and cost more money. You are cold. You can buy a cheap scarf, or a better but more expensive scarf. These are trade-offs. Sometimes one choice will be made, sometimes the other. Now consider another type of problem. You are married, and want to take your spouse out to a romantic dinner. You could choose a place you both love, or a place that only they love. You choose the place you don’t love, so they will know how much you love them. After all, you didn’t come here for the food. A middle manager must choose how to improve widget production. He can choose a policy that improperly maintains the factory and likely eventually poisons the water supply, or a policy that would prevent that at no additional cost. He knows that when he is up for promotion, management will want to know the higher ups can count on him to make the quarterly numbers look good and not concern himself with long term issues or what consequences might fall on others. If he cared about not poisoning the water supply, he would not be a reliable political ally. Thus, he chooses the neglectful policy. A politician can choose between two messages that affirm their loyalty: Advocating a beneficial policy, or advocating a useless and wasteful policy. They choose useless, because the motive behind advocating a beneficial policy is ambiguous. Maybe they wanted people to benefit! A start-up founder can choose between building a quality product without technical debt and creating a hockey stick graph with it, or building a superficially similar low-quality product with technical debt and using that. Both are equally likely to create the necessary graph, and both take about the same amount of effort, time and money. They choose the low-quality product, so the venture capitalists can appreciate their devotion to creating a hockey stick graph. You can choose between making a gift and buying a gift. You choose to make a gift, because you are rich and buying something from a store would be meaningless. Or you are poor, so you buy something from a store, because a handmade gift wouldn’t show you care. Old joke: One Russian oligarch says, “Look at my scarf! I bought it for ten thousand rubles.” The other says, “That’s nothing, I bought the same scarf for twenty thousand rubles.” What these examples have in common is that there is a strictly better action and a strictly worse action, in terms of physical consequences. In each case, the protagonist chooses the worse action because it is worse. This ch...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The correct response to uncertainty is not half-speed, published by AnnaSalamon on the LessWrong. Related to: Half-assing it with everything you've got; Wasted motion; Say it Loud. Once upon a time (true story), I was on my way to a hotel in a new city. I knew the hotel was many miles down this long, branchless road. So I drove for a long while. After a while, I began to worry I had passed the hotel. So, instead of proceeding at 60 miles per hour the way I had been, I continued in the same direction for several more minutes at 30 miles per hour, wondering if I should keep going or turn around. After a while, I realized: I was being silly! If the hotel was ahead of me, I'd get there fastest if I kept going 60mph. And if the hotel was behind me, I'd get there fastest by heading at 60 miles per hour in the other direction. And if I wasn't going to turn around yet -- if my best bet given the uncertainty was to check N more miles of highway first, before I turned around -- then, again, I'd get there fastest by choosing a value of N, speeding along at 60 miles per hour until my odometer said I'd gone N miles, and then turning around and heading at 60 miles per hour in the opposite direction. Either way, fullspeed was best. My mind had been naively averaging two courses of action -- the thought was something like: "maybe I should go forward, and maybe I should go backward. So, since I'm uncertain, I should go forward at half-speed!" But averages don't actually work that way.[1] Following this, I started noticing lots of hotels in my life (and, perhaps less tactfully, in my friends' lives). For example: I wasn't sure if I was a good enough writer to write a given doc myself, or if I should try to outsource it. So, I sat there kind-of-writing it while also fretting about whether the task was correct. (Solution: Take a minute out to think through heuristics. Then, either: (1) write the post at full speed; or (2) try to outsource it; or (3) write full force for some fixed time period, and then pause and evaluate.) I wasn't sure (back in early 2012) that CFAR was worthwhile. So, I kind-of worked on it. An old friend came to my door unexpectedly, and I was tempted to hang out with her, but I also thought I should finish my work. So I kind-of hung out with her while feeling bad and distracted about my work. A friend of mine, when teaching me math, seems to mumble specifically those words that he doesn't expect me to understand (in a sort of compromise between saying them and not saying them)... Duncan reports that novice Parkour students are unable to safely undertake certain sorts of jumps, because they risk aborting the move mid-stream, after the actual last safe stopping point (apparently kind-of-attempting these jumps is more dangerous than either attempting, or not attempting the jumps) It is said that start-up founders need to be irrationally certain that their startup will succeed, lest they be unable to do more than kind-of work on it... That is, it seems to me that often there are two different actions that would make sense under two different models, and we are uncertain which model is true... and so we find ourselves taking an intermediate of half-speed action... even when that action makes no sense under any probabilistic mixture of the two models. You might try looking out for such examples in your life. [1] Edited to add: The hotel example has received much nitpicking in the comments. But: (A) the actual example was legit, I think. Yes, stopping to think has some legitimacy, but driving slowly for a long time because uncertain does not optimize for thinking. Similarly, it may make sense to drive slowly to stare at the buildings in some contexts... but I was on a very long empty country road, with no buildings anywhere (true historical fact), and also I was not squint...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Year of Spaced Repetition Software in the Classroom, published by tanagrabeast on the LessWrong. Last year, I asked LW for some advice about spaced repetition software (SRS) that might be useful to me as a high school teacher. With said advice came a request to write a follow-up after I had accumulated some experience using SRS in the classroom. This is my report. Please note that this was not a scientific experiment to determine whether SRS "works." Prior studies are already pretty convincing on this point and I couldn't think of a practical way to run a control group or "blind" myself. What follows is more of an informal debriefing for how I used SRS during the 2014-15 school year, my insights for others who might want to try it, and how the experience is changing how I teach. Summary SRS can raise student achievement even with students who won't use the software on their own, and even with frequent disruptions to the study schedule. Gains are most apparent with the already high-performing students, but are also meaningful for the lowest students. Deliberate efforts are needed to get student buy-in, and getting the most out of SRS may require changes in course design. The software After looking into various programs, including the game-like Memrise, and even writing my own simple SRS, I ultimately went with Anki for its multi-platform availability, cloud sync, and ease-of-use. I also wanted a program that could act as an impromptu catch-all bin for the 2,000+ cards I would be producing on the fly throughout the year. (Memrise, in contrast, really needs clearly defined units packaged in advance). The students I teach 9th and 10th grade English at an above-average suburban American public high school in a below-average state. Mine are the lower "required level" students at a school with high enrollment in honors and Advanced Placement classes. Generally speaking, this means my students are mostly not self-motivated, are only very weakly motivated by grades, and will not do anything school-related outside of class no matter how much it would be in their interest to do so. There are, of course, plenty of exceptions, and my students span an extremely wide range of ability and apathy levels. The procedure First, what I did not do. I did not make Anki decks, assign them to my students to study independently, and then quiz them on the content. With honors classes I taught in previous years I think that might have worked, but I know my current students too well. Only about 10% of them would have done it, and the rest would have blamed me for their failing grades—with some justification, in my opinion. Instead, we did Anki together, as a class, nearly every day. As initial setup, I created a separate Anki profile for each class period. With a third-party add-on for Anki called Zoom, I enlarged the display font sizes to be clearly legible on the interactive whiteboard at the front of my room. Nightly, I wrote up cards to reinforce new material and integrated them into the deck in time for the next day's classes. This averaged about 7 new cards per lesson period.These cards came in many varieties, but the three main types were: concepts and terms, often with reversed companion cards, sometimes supplemented with "what is this an example of" scenario cards. vocabulary, 3 cards per word: word/def, reverse, and fill-in-the-blank example sentence grammar, usually in the form of "What change(s), if any, does this sentence need?" Alternative cards had different permutations of the sentence. Weekly, I updated the deck to the cloud for self-motivated students wishing to study on their own. Daily, I led each class in an Anki review of new and due cards for an average of 8 minutes per study day, usually as our first activity, at a rate of about 3.5 cards per minute. As each card appear...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Hindsight Devalues Science, published by Eliezer Yudkowsky on the LessWrong. This essay is closely based on an excerpt from Meyers’s Exploring Social Psychology; the excerpt is worth reading in its entirety. Cullen Murphy, editor of The Atlantic, said that the social sciences turn up “no ideas or conclusions that can’t be found in [any] encyclopedia of quotations . . . Day after day social scientists go out into the world. Day after day they discover that people’s behavior is pretty much what you’d expect.” Of course, the “expectation” is all hindsight. (Hindsight bias: Subjects who know the actual answer to a question assign much higher probabilities they “would have” guessed for that answer, compared to subjects who must guess without knowing the answer.) The historian Arthur Schlesinger, Jr. dismissed scientific studies of World War II soldiers’ experiences as “ponderous demonstrations” of common sense. For example: Better educated soldiers suffered more adjustment problems than less educated soldiers. (Intellectuals were less prepared for battle stresses than street-smart people.) Southern soldiers coped better with the hot South Sea Island climate than Northern soldiers. (Southerners are more accustomed to hot weather.) White privates were more eager to be promoted to noncommissioned officers than Black privates. (Years of oppression take a toll on achievement motivation.) Southern Blacks preferred Southern to Northern White officers. (Southern officers were more experienced and skilled in interacting with Blacks.) As long as the fighting continued, soldiers were more eager to return home than after the war ended. (During the fighting, soldiers knew they were in mortal danger.) How many of these findings do you think you could have predicted in advance? Three out of five? Four out of five? Are there any cases where you would have predicted the opposite—where your model takes a hit? Take a moment to think before continuing . . . In this demonstration (from Paul Lazarsfeld by way of Meyers), all of the findings above are the opposite of what was actually found.1 How many times did you think your model took a hit? How many times did you admit you would have been wrong? That’s how good your model really was. The measure of your strength as a rationalist is your ability to be more confused by fiction than by reality. Unless, of course, I reversed the results again. What do you think? Do your thought processes at this point, where you really don’t know the answer, feel different from the thought processes you used to rationalize either side of the “known” answer? Daphna Baratz exposed college students to pairs of supposed findings, one true (“In prosperous times people spend a larger portion of their income than during a recession”) and one the truth’s opposite.2 In both sides of the pair, students rated the supposed finding as what they “would have predicted.” Perfectly standard hindsight bias. Which leads people to think they have no need for science, because they “could have predicted” that. (Just as you would expect, right?) Hindsight will lead us to systematically undervalue the surprisingness of scientific findings, especially the discoveries we understand—the ones that seem real to us, the ones we can retrofit into our models of the world. If you understand neurology or physics and read news in that topic, then you probably underestimate the surprisingness of findings in those fields too. This unfairly devalues the contribution of the researchers; and worse, will prevent you from noticing when you are seeing evidence that doesn’t fit what you really would have expected. We need to make a conscious effort to be shocked enough. 1 Paul F. Lazarsfeld, “The American Solidier—An Expository Review,” Public Opinion Quarterly 13, no. 3 (1949): 377–404. 2 Daphna Baratz, ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Beware Trivial Inconveniences, published by Scott Alexander on the LessWrong. The Great Firewall of China. A massive system of centralized censorship purging the Chinese version of the Internet of all potentially subversive content. Generally agreed to be a great technical achievement and political success even by the vast majority of people who find it morally abhorrent. I spent a few days in China. I got around it at the Internet cafe by using a free online proxy. Actual Chinese people have dozens of ways of getting around it with a minimum of technical knowledge or just the ability to read some instructions. The Chinese government isn't losing any sleep over this (although they also don't lose any sleep over murdering political dissidents, so maybe they're just very sound sleepers). Their theory is that by making it a little inconvenient and time-consuming to view subversive sites, they will discourage casual exploration. No one will bother to circumvent it unless they already seriously distrust the Chinese government and are specifically looking for foreign websites, and these people probably know what the foreign websites are going to say anyway. Think about this for a second. The human longing for freedom of information is a terrible and wonderful thing. It delineates a pivotal difference between mental emancipation and slavery. It has launched protests, rebellions, and revolutions. Thousands have devoted their lives to it, thousands of others have even died for it. And it can be stopped dead in its tracks by requiring people to search for "how to set up proxy" before viewing their anti-government website. I was reminded of this recently by Eliezer's Less Wrong Progress Report. He mentioned how surprised he was that so many people were posting so much stuff on Less Wrong, when very few people had ever taken advantage of Overcoming Bias' policy of accepting contributions if you emailed them to a moderator and the moderator approved. Apparently all us folk brimming with ideas for posts didn't want to deal with the aggravation. Okay, in my case at least it was a bit more than that. There's a sense of going out on a limb and drawing attention to yourself, of arrogantly claiming some sort of equivalence to Robin Hanson and Eliezer Yudkowsky. But it's still interesting that this potential embarrassment and awkwardness was enough to keep the several dozen people who have blogged on here so far from sending that "I have something I'd like to post..." email. Companies frequently offer "free rebates". For example, an $800 television with a $200 rebate. There are a few reasons companies like rebates, but one is that you'll be attracted to the television because it appears to have a net cost only $600, but then filling out the paperwork to get the rebate is too inconvenient and you won't get around to it. This is basically a free $200 for filling out an annoying form, but companies can predict that customers will continually fail to complete it. This might make some sense if you're a high-powered lawyer or someone else whose time is extremely valuable, but most of us have absolutely no excuse. One last example: It's become a truism that people spend more when they use credit cards than when they use money. This particular truism happens to be true: in a study by Prelec and Simester1, auction participants bid twice as much for the same prize when using credit than when using cash. The trivial step of getting the money and handing it over has a major inhibitory effect on your spending habits. I don't know of any unifying psychological theory that explains our problem with trivial inconveniences. It seems to have something to do with loss aversion, and with the brain's general use of emotion-based hacks instead of serious cost-benefit analysis. It might be linked to akrasia; ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Goodhart Taxonomy, published by Scott Garrabrant on the LessWrong. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. Goodhart’s Law states that "any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes." However, this is not a single phenomenon. I propose that there are (at least) four different mechanisms through which proxy measures break when you optimize for them. The four types are Regressional, Causal, Extremal, and Adversarial. In this post, I will go into detail about these four different Goodhart effects using mathematical abstractions as well as examples involving humans and/or AI. I will also talk about how you can mitigate each effect. Throughout the post, I will use V to refer to the true goal and use U to refer to a proxy for that goal which was observed to correlate with V and which is being optimized in some way. Quick Reference Regressional Goodhart - When selecting for a proxy measure, you select not only for the true goal, but also for the difference between the proxy and the goal. Model: When U is equal to V X , where X is some noise, a point with a large U value will likely have a large V value, but also a large X value. Thus, when U is large, you can expect V to be predictably smaller than U Example: height is correlated with basketball ability, and does actually directly help, but the best player is only 6'3", and a random 7' person in their 20s would probably not be as good Causal Goodhart - When there is a non-causal correlation between the proxy and the goal, intervening on the proxy may fail to intervene on the goal. Model: If V causes U (or if V and U are both caused by some third thing), then a correlation between V and U may be observed. However, when you intervene to increase U through some mechanism that does not involve V , you will fail to also increase V Example: someone who wishes to be taller might observe that height is correlated with basketball skill and decide to start practicing basketball. Extremal Goodhart - Worlds in which the proxy takes an extreme value may be very different from the ordinary worlds in which the correlation between the proxy and the goal was observed. Model: Patterns tend to break at simple joints. One simple subset of worlds is those worlds in which U is very large. Thus, a strong correlation between U and V observed for naturally occuring U values may not transfer to worlds in which U is very large. Further, since there may be relatively few naturally occuring worlds in which U is very large, extremely large U may coincide with small V values without breaking the statistical correlation. Example: the tallest person on record, Robert Wadlow, was 8'11" (2.72m). He grew to that height because of a pituitary disorder, he would have struggled to play basketball because he "required leg braces to walk and had little feeling in his legs and feet." Adversarial Goodhart - When you optimize for a proxy, you provide an incentive for adversaries to correlate their goal with your proxy, thus destroying the correlation with your goal. Model: Consider an agent A with some different goal W . Since they depend on common resources, W and V are naturally opposed. If you optimize U as a proxy for V , and A knows this, A is incentivized to make large U values coincide with large W values, thus stopping them from coinciding with large V values. Example: aspiring NBA players might just lie about their height. Regressional Goodhart When selecting for a proxy measure, you select not only for the true goal, but also for the difference between the proxy and the goal. Abstract Model When U is equal to V X , where X is some noise, a point with a large U value will likely have a large V value, but also a large X value. Thus, when U is large, ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Unrolling social metacognition: Three levels of meta are not enough, published by Academian on the LessWrong. Disclaimer: This post was written time-boxed to 2 hours because I think LessWrong can still understand and improve upon it; please don't judge me harshly for it. Summary: I am generally dismayed that many people seem to think or assume that only three levels of social metacognition matter ("Alex knows that Bailey knows that Charlie knows X"), or otherwise seem generally averse to unrolling those levels. This post is intended to point out (1) how the higher levels systematically get distilled and chunked into smaller working memory elements through social learning, which leads to emotional tracking of phenomena at 6 levels of meta and higher, and (2) what I think this means about how to approach conflict resolution. Epistemic status: don't take my word for it; conceptual points intended to be fairly self evident upon reflection; actual techniques not backed up by systematic empirical research and might not generalize to other humans; all content very much validated by my personal experiences with talking to people about feelings in real life. Related Reading: Duncan Sabien on Common knowledge & Miasma; Ben Pace on The Costly Coordination Mechanism of Common Knowledge I. Conceptual introduction, by example Here's how higher levels of social metacognition get distilled down and represented in emotions that end up tracking them (if poorly). Each feeling in the example below will be followed by an unrolling of the actual event or events it is implicitly tracking or referring to. Warning: reading this first section (I) will require a fair bit of symbolic reasoning/thinking, so you might find it tiring and prefer to skip to later sections. A better writing of this section would do more work in between these symbolic reasoning bits to distill things out and make them easier to digest. Scale 1: One event, four levels of meta (yes, we're starting with four) 1.1) Alex leaves out the milk for 5 minutes 1.2) Bailey observes (1.1), and feels it was bad. Unrolling of referents: Bailey felt that Alex leaving out the milk was bad. 1.3) Alex observes (1.2), and feels judged. Unrolling of referents: Alex felt that Bailey felt that Alex leaving out the milk was bad. 1.4) Alex reflects on feeling judged, doesn't like it, and concludes that Bailey is "a downer". Unrolling of referents: Alex felt it was bad that Alex felt that Bailey felt that Alex leaving out the milk was bad. Notice that the unrollings look and sound very different from the distillations. That's in large part because the unrolling is not our native format for storing social metacognition; it's stored via concepts like "feeling judged" or "being a downer". However, to the extent that the feeling "Bailey is a downer" is tracking something in reality, it's tracking things that track things that track things that track reality: in this case, milk spoilage. (An aside: notice also that 1.4 involves Alex's feelings about Alex's feelings. Some people wouldn't call that an extra level of social metacognition, and would just combine it all together into "Alex's feelings". However, I'm separating those layers for two reasons: (1) the separation in counting won't affect my conclusion that the total number of levels being implicitly tracked greatly exceeds three, and (2) I think it's especially important to note when people have feelings about their own feelings, as that can lead to circular definitions in what their feelings are tracking; but that's a topic for another day.) Scale 2: multiple events, six levels of meta I'll start the numbering at 4 here: 2.4) Multiple similar Scale 1 events happen where Alex does something X, and ends up feeling that Bailey was "a downer" about it. Partial unrolling of referents: Alex feels ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Link] Still Alive - Astral Codex Ten, published by jimrandomh on the LessWrong. This is a linkpost for I. This was a triumph I'm making a note here, huge success No, seriously, it was awful. I deleted my blog of 1,557 posts. I wanted to protect my privacy, but I ended up with articles about me in New Yorker, Reason, and The Daily Beast. I wanted to protect my anonymity, but I Streisand-Effected myself, and a bunch of trolls went around posting my real name everywhere they could find. I wanted to avoid losing my day job, but ended up quitting so they wouldn't be affected by the fallout. I lost a five-digit sum in advertising and Patreon fees. I accidentally sent about three hundred emails to each of five thousand people in the process of trying to put my blog back up. I had, not to mince words about it, a really weird year. The first post on Scott Alexander's new blog on Substack, Astral Codex Ten. Thanks for listening. to help us out with the nonlinear library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What Do GDP Growth Curves Really Mean?, published by johnswentworth on the LessWrong. Gross domestic product (GDP) is a monetary measure of the market value of all the final goods and services produced in a specific time period. - Wikipedia, GDP Due to inflation, GDP increases and does not actually reflect the true growth in an economy. That is why the GDP must be divided by the inflation rate (raised to the power of units of time in which the rate is measured) to get the growth of the real GDP. - Wikipedia, Real GDP The two quotes above reflect how I used to think about real GDP growth: it’s roughly the growth in economic production (as measured by dollar worth of outputs), discounted for inflation. This picture turns out to be extremely misleading, especially when using GDP as a growth measure. Forget complaints about how GDP doesn’t measure happiness, or leisure time, or household work, or “the health of our children, the quality of their education or the joy of their play”. Even if we accept the dollar value of goods as a proxy for whatever purpose we have in mind, GDP (as we actually calculate it) is still a wildly misleading measure of growth. In particular, it effectively ignores major technological breakthroughs. A Puzzle Here’s real GDP of the US for the last ~70 years, from FRED: According to this graph, real GDP has grown by roughly a factor of 6 since 1960. That seems. way too low, intuitively. Consider: I’m typing this post on my laptop (which conveniently has a backspace button and everything I type is backed up halfway around the world and I can even insert images trivially)... while listening to spotify. through my noise-canceling earbuds. and there’s a smartphone on my desk which can give me detailed road maps and directions anywhere in the US and even most of the world, plus make phone calls. and oh-by-the-way I have an internet connection. I’d expect the equivalent of any one of these things in 1960 would have cost at least a hundred times the annual income of an average person if it was even possible at all. Just from these five things alone, it seems like real GDP ought to have grown by a factor of hundreds. . and yet, whatever formula we’re using for real GDP says it's only grown by a factor of 6. What gives? How the heck is real GDP computed that makes it so low? What exactly is it measuring? Real GDP Is Not Nominal GDP Divided By Inflation First things first: real GDP is not calculated by dividing nominal GDP by inflation. It’s calculated largely separately from nominal GDP; the textbook approach is to add up the total dollar value of goods (just like for nominal) but at prices from a fixed year. That way, we only count changes in total output resulting from changes in the amounts of goods produced. An example: we have an economy with two goods, apples and brass. In year 0, 1 unit of apples costs $1, 1 unit of brass costs $1, and people produce/consume 3 units each of brass and apples. In year 1, an amazing new technique is discovered for brass-production. Brass prices fall by a factor of 10, and people produce/consume five times more brass (15 units). Meanwhile, both price and production/consumption of apples stays roughly the same. Apple Price Apple Quantity Brass Price Brass Quantity Year 0 $1/unit 3 units $1/unit 3 units Year 1 $1/unit 3 units $0.1/unit 15 units Calculations: GDP in year 0 (at year 0 prices): (3 apple-units)($1/apple-unit) + (3 brass-units)($1/brass-unit) = $6 GDP in year 1 (at year 0 prices): (3 apple-units)($1/apple-unit) + (15 brass-units)($1/brass-unit) = $18 GDP growth: $18/$6 = 3 This seems pretty reasonable. Indeed, in our puzzle about GDP growth since 1960, I said: I’d expect the equivalent of any one of these things in 1960 would have cost at least a hundred times the annual income of an average person if it was e...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Money: The Unit of Caring, published by Eliezer Yudkowsky on the LessWrong. Steve Omohundro has suggested a folk theorem to the effect that, within the interior of any approximately rational, self-modifying agent, the marginal benefit of investing additional resources in anything ought to be about equal. Or, to put it a bit more exactly, shifting a unit of resource between any two tasks should produce no increase in expected utility, relative to the agent's utility function and its probabilistic expectations about its own algorithms. This resource balance principle implies that—over a very wide range of approximately rational systems, including even the interior of a self-modifying mind—there will exist some common currency of expected utilons, by which everything worth doing can be measured. In our society, this common currency of expected utilons is called "money". It is the measure of how much society cares about something. This is a brutal yet obvious point, which many are motivated to deny. With this audience, I hope, I can simply state it and move on. It's not as if you thought "society" was intelligent, benevolent, and sane up until this point, right? I say this to make a certain point held in common across many good causes. Any charitable institution you've ever had a kind word for, certainly wishes you would appreciate this point, whether or not they've ever said anything out loud. For I have listened to others in the nonprofit world, and I know that I am not speaking only for myself here... Many people, when they see something that they think is worth doing, would like to volunteer a few hours of spare time, or maybe mail in a five-year-old laptop and some canned goods, or walk in a march somewhere, but at any rate, not spend money. Believe me, I understand the feeling. Every time I spend money I feel like I'm losing hit points. That's the problem with having a unified quantity describing your net worth: Seeing that number go down is not a pleasant feeling, even though it has to fluctuate in the ordinary course of your existence. There ought to be a fun-theoretic principle against it. But, well... There is this very, very old puzzle/observation in economics about the lawyer who spends an hour volunteering at the soup kitchen, instead of working an extra hour and donating the money to hire someone to work for five hours at the soup kitchen. There's this thing called "Ricardo's Law of Comparative Advantage". There's this idea called "professional specialization". There's this notion of "economies of scale". There's this concept of "gains from trade". The whole reason why we have money is to realize the tremendous gains possible from each of us doing what we do best. This is what grownups do. This is what you do when you want something to actually get done. You use money to employ full-time specialists. Yes, people are sometimes limited in their ability to trade time for money (underemployed), so that it is better for them if they can directly donate that which they would usually trade for money. If the soup kitchen needed a lawyer, and the lawyer donated a large contiguous high-priority block of lawyering, then that sort of volunteering makes sense—that's the same specialized capability the lawyer ordinarily trades for money. But "volunteering" just one hour of legal work, constantly delayed, spread across three weeks in casual minutes between other jobs? This is not the way something gets done when anyone actually cares about it, or to state it near-equivalently, when money is involved. To the extent that individuals fail to grasp this principle on a gut level, they may think that the use of money is somehow optional in the pursuit of things that merely seem morally desirable—as opposed to tasks like feeding ourselves, whose desirability seems to be treated ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Steelmanning Divination, published by Vaniver on the LessWrong. [This post was primarily written in 2015, after I gave a related talk, and other bits in 2018; I decided to finish writing it now because of a recent SSC post.] The standard forms of divination that I’ve seen in contemporary Western culture--astrology, fortune cookies, lotteries, that sort of thing--seem pretty worthless to me. They’re like trying to extract information from a random number generator, which is a generally hopeless phenomenon because of conservation of expected evidence. Thus I had mostly written off divination; although I've come across some arguments that divination served as a way to implement mixed strategies in competitive games. (Hunters would decide where to hunt by burning bones, which generated an approximately random map of their location, preventing their targets from learning where the humans liked to hunt and avoiding that location.) But then I came across this striking passage, and sat up straight: One performs the rain sacrifice and it rains. Why? I say: there is no special reason why. It is the same as when one does not perform the rain sacrifice and it rains anyway. When the sun and moon suffer eclipse, one tries to save them. When Heaven sends drought, one performs the rain sacrifice. One performs divination and only then decides on important affairs. But this is not to be regarded as bringing one what one seeks, but rather is done to give things proper form. Thus, the gentleman regards this as proper form, but the common people regard it as connecting with spirits. If one regards it as proper form, one will have good fortune. If one regards it as connecting with spirits, one will have misfortune. This is from Eric L. Hutton's translation of a collection of essays called Xunzi (presumably written by Xunzi, an ancient Chinese philosopher who was Confucian with heavy Legalist influences). The book was overall remarkable in how much of Xunzi's brilliance shone through, which is something I very rarely think about authors. (Talking to another rationalist who was more familiar with Chinese philosophy than I was, he also had this impression that Xunzi simply had a lot more mental horsepower than many other core figures.) By the end of it, I was asking myself, "if they had this much of rationality figured out back then, why didn't they conquer the world?" Then I looked into the history a bit more and figured out that two of Xunzi's students were core figures in Qin Shi Huang's unification of China to become the First Emperor. So this paragraph stuck with me. When Xunzi talks about the way that earlier kings did things, I registered it as an applause light and moved on. When he talked about how an important role of government was to prevent innovation in music, I registered it as covering a very different thing than what I think of when I think about 'music' and moved on. But when he specifically called out the reason why I (and most educated people I know) don't pay much attention to astrology or other sorts of divination or magic, said "yeah, those would be dumb reasons to do this," and then said "but there's still a reason", I was curious. What's the proper form that he's talking about? (Sadly, this was left as an exercise for the reader; the surrounding paragraphs are only vaguely related.) In his introduction, Hutton summarizes the relevant portion of Xunzi's philosophy: In this process of becoming good, ritual plays an especially important role in Xunzi's view. As he conceives them, the rituals constitute a set of standards for proper behavior that were created by the past sages and should govern virtually every aspect of a person's life. These rituals are not inviolable rules: Xunzi allows that people with developed moral judgment may need to depart from the strict dictat...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: On learning difficult things, published by So8res on the LessWrong. I have been autodidacting quite a bit lately. You may have seen my reviews of books on the MIRI course list. I've been going for about ten weeks now. This post contains my notes about the experience thus far. Much of this may seem obvious, and would have seemed obvious if somebody had told me in advance. But nobody told me in advance. As such, this is a collection of things that were somewhat surprising at the time. Part of the reason I'm posting this is because I don't know a lot of autodidacts, and I'm not sure how normal any of my experiences are. (Though on average, I'd guess they're about average.) As always, keep in mind that I am only one person and that your mileage may vary. Pair up When I began my quest for more knowledge, I figured that in this modern era, a well-written textbook and an account on math.stackexchange would be enough to get me through anything. And I was right. sort of. But not really. The problem is, most of the time that I get stuck, I get stuck on something incredibly stupid. I've either misread something somewhere or misremembered a concept from earlier in the book. Usually, someone looking over my shoulder could correct me in ten seconds with three words. "Dude. Disjunction. Disjunction." These are the things that eat my days. In principle, places like stackexchange can get me unstuck, but they're an awkward tool for the job. First of all, my stupid mistakes are heavily contextualized. A full context dump is necessary before I can even ask my question, and this takes time. Furthermore, I feel dumb asking stupid questions on stackexchange-type sites. My questions are usually things that I can figure out with a close re-read (except, I'm not sure which part needs a re-read). I usually opt for a close re-read of everything rather than asking for help. This is even more time consuming. The infuriating thing is that answering these questions usually doesn't require someone who already knows the answers: it just requires someone who didn't make exactly the same mistakes as me. I lose hours on little mistakes that could have been fixed within seconds if I was doing this with someone else. That's why my number one piece of advice for other people attempting to learn on their own is do it with a friend. They don't need to be more knowledgeable than you to answer most of the questions that come up. They just need to make different misunderstandings, and you'll be able to correct each other as you go along. The thing I miss most about college is tight feedback loops while learning. When autodidacting, the feedback loop can be long. I still haven't managed to follow my own advice here. I'm writing this advice in part because it should motivate me to actually pair up. Unfortunately, there is nobody in my immediate circle who has the time or patience to read along with me, but there are a number of resources I have not yet explored (the LessWrong study hall, for example, or soliciting to actual mathematicians). It's on my list of things to do. Read, reread, rereread Reading Model Theory was one of the hardest things I've done. Not necessarily because the content was hard, but because it was the first time I actually learned something that was way outside my comfort zone. The short version is that Basic Category Theory and Naïve Set Theory left me somewhat overconfident, and that I should have read a formal logic textbook before diving in. I had basic familiarity with logic, but no practice. Turns out practice is important. Anyway, it's not like Model Theory was impossible just because I skipped my logic exercises. It was just hard. There are a number of little misconceptions you have when you're familiar with something but you've never applied it, and I found myself having to clean t...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Inadequate Equilibria vs. Governance of the Commons, published by Martin Sustrik on the LessWrong. This is a cross post from http://250bpm.com/blog:128. Introduction In the past I've reviewed Eliezer Yudkowsky's "Inadequate Equilibria" book. My main complaint was that while it explains the problem of suboptimal Nash equilibria very well, it doesn't propose any solutions. Instead, it says that we should be aware of such coordination failures and we should expect ourselves to fare better than the official institutions in such cases. What Yudkowsky is saying (if I understand him correctly) is that given that the treatment of short bowel syndrome in babies is stuck in an inadequate equilibrium, there's no way to fix the problem on the system level. However, you can spot the problem and acquire the medication needed to keep your kid alive from abroad yourself. In my review I've sketched a couple of ideas how to approach the problem, but that was just me trying to be clever. Something backed by evidence would make me much more happy. So I've decided to have a look at how people are getting out of suboptimal equilibria in the real world. And that's how I've got to the Elinor Ostrom's book "Governing the Commons (The Evolution of Institutions for Collective Action)". The book is very explicitly covering the same problem as Yudkowsky. As Ostrom say about her research in her Nobel lecture: "One of the key questions that we've been addressing is 'Are rational individuals hopelessly trapped in dilemmas?'" Ostrom is an economist with a game-theoretic bent. But she's not a theoretician. She's a field researcher. I've expected some good read and I wasn't disappointed. The book focuses on special subset of coordination problems, problems of management of what Ostrom calls "common pool resources". Common pool resource is something that, unlike private property, is not exclusionary, something that one cannot easily restrict people from taking advantage of, but that, unlike public property, can be depleted by excessive usage. So, for example, it's hard to exclude others from fishing in the ocean, yet, the fish stock can be depleted by overfishing. Ocean fishery is a common pool resource. Compare that with a private fish farm where others are prevented from fishing by a barrier. And on the other side, contrast it with a public weather forecast which doesn't get depleted as more people turn on the radio. On the game theoretic level, these scenarios can be though of as "tragedy of the commons" or "prisoner's dilemma" problems. At the heart of the problem is the fact that while cooperating and limiting everyone's rate of extraction helps to sustain the resource, individual players get better short-term reward if they defect and overuse the resource. For example, limiting the number of cows on a common grazing land keeps the pasture sustainable. However, each villager is incentivized to get as many cows as possible which will in turn result in overgrazing and destruction of the grazing land. Against economic absolutism As Ostrom notes, people reading about this kind of problem tend to jump to one of the two conclusions. One group believes that the grazing land should be nationalized and the laws should be enacted to prevent the overgrazing. The other group thinks that the land should be split into small plots and privatized. The problem with the latter solution is that it's often not feasible. For example, there's no reasonable way to privatize an ocean fishery. Fish stock is migratory and cannot be split among stakeholders. So, if you invest in the growth of the fish stock by temporarily fishing less, the fish will eventually migrate elsewhere and get caught by other fishers. In other cases the resource can be split but doing so results in suboptimal performance. For example, if conditions...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Mundane Magic, published by Eliezer Yudkowsky on the LessWrong. As you may recall from some months earlier, I think that part of the rationalist ethos is binding yourself emotionally to an absolutely lawful reductionistic universe—a universe containing no ontologically basic mental things such as souls or magic—and pouring all your hope and all your care into that merely real universe and its possibilities, without disappointment. There's an old trick for combating dukkha where you make a list of things you're grateful for, like a roof over your head. So why not make a list of abilities you have that would be amazingly cool if they were magic, or if only a few chosen individuals had them? For example, suppose that instead of one eye, you possessed a magical second eye embedded in your forehead. And this second eye enabled you to see into the third dimension—so that you could somehow tell how far away things were—where an ordinary eye would see only a two-dimensional shadow of the true world. Only the possessors of this ability can accurately aim the legendary distance-weapons that kill at ranges far beyond a sword, or use to their fullest potential the shells of ultrafast machinery called "cars". "Binocular vision" would be too light a term for this ability. We'll only appreciate it once it has a properly impressive name, like Mystic Eyes of Depth Perception. So here's a list of some of my favorite magical powers: Vibratory Telepathy. By transmitting invisible vibrations through the very air itself, two users of this ability can share thoughts. As a result, Vibratory Telepaths can form emotional bonds much deeper than those possible to other primates. Psychometric Tracery. By tracing small fine lines on a surface, the Psychometric Tracer can leave impressions of emotions, history, knowledge, even the structure of other spells. This is a higher level than Vibratory Telepathy as a Psychometric Tracer can share the thoughts of long-dead Tracers who lived thousands of years earlier. By reading one Tracery and inscribing another simultaneously, Tracers can duplicate Tracings; and these replicated Tracings can even contain the detailed pattern of other spells and magics. Thus, the Tracers wield almost unimaginable power as magicians; but Tracers can get in trouble trying to use complicated Traceries that they could not have Traced themselves. Multidimensional Kinesis. With simple, almost unthinking acts of will, the Kinetics can cause extraordinarily complex forces to flow through small tentacles and into any physical object within touching range—not just pushes, but combinations of pushes at many points that can effectively apply torques and twists. The Kinetic ability is far subtler than it first appears: they use it not only to wield existing objects with martial precision, but also to apply forces that sculpt objects into forms more suitable for Kinetic wielding. They even create tools that extend the power of their Kinesis and enable them to sculpt ever-finer and ever-more-complicated tools, a positive feedback loop fully as impressive as it sounds. The Eye. The user of this ability can perceive infinitesimal traveling twists in the Force that binds matter—tiny vibrations, akin to the life-giving power of the Sun that falls on leaves, but far more subtle. A bearer of the Eye can sense objects far beyond the range of touch using the tiny disturbances they make in the Force. Mountains many days travel away can be known to them as if within arm's reach. According to the bearers of the Eye, when night falls and sunlight fails, they can sense huge fusion fires burning at unthinkable distances—though no one else has any way of verifying this. Possession of a single Eye is said to make the bearer equivalent to royalty. And finally, The Ultimate Power. The user of this abilit...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Social behavior curves, equilibria, and radicalism, published by UnexpectedValues on the LessWrong. This is a linkpost for/ I. Here are some hypotheticals to consider, with a common theme. Note that in each case I’m asking what you would do, rather than what you should do. In the fall, COVID cases drop to 10% of their current level. You’re back to working/studying in person. You’re vaccinated, as is everyone else. Mask-wearing isn’t required, but 25% of your co-workers wear one anyway. Would you wear a mask too? What if 75% of your co-workers wear a mask? What if it’s literally everyone else? You’re having dinner with a party of 10 at a Chinese restaurant. Everyone else is using chop sticks. You know how to use chop sticks but prefer a fork. Do you ask for a fork? What if two other people are using a fork? (Inspired by Scout Mindset) For anyone who chose to have kids, or wants kids: if only 30% of adults had kids, would you still choose to have a kid? What if only 2% did? For anyone who doesn’t want kids: if 90% of adults had kids, would you? What if it were 99%? You join a group video call with 20 co-workers/classmates. Everyone except the presenter/teacher has their video turned off. Do you turn yours off too? What if everyone has their video on? Typically, what fraction of other participants need to have their video on for you to keep yours on? Your answers to these questions depend on your personal circumstances. How uncomfortable is wearing a mask? Do you find a fork slightly or significantly easier to use? How strong is your preference to have kids? Is there a Justin Bieber poster in your background? But — unless your answer to each question didn’t depend on the specific percentage — your decision also depends on others’ behavior. If everyone else has their video on, you’ll probably feel obligated to keep yours on too. Maybe you’ll move the poster first. This applies not just to behavior, but also to preferences, beliefs, and opinions. Figure 1: xkcd #185 So far, everything I’ve said is pretty obvious. But let’s throw a model at this observation and see if we can discover anything interesting. Let’s take our example with 20 people in a video call and line all of the participants up, from bottom to top, based on how many other people need to have their video on in order for them to choose to keep their video on. Figure 2: video chat, scenario 1 In this example, Alex and Betty will keep their video on no matter what; on the other hand, Riley, Steve, and Tara will turn their video off no matter what. Most others are somewhere in between: Isaac, for instance, will keep his video on if at least 7 of the other 19 participants have their video on. Take a moment to think about what will happen in this call. As a hint, you might want to consider the diagonal line I’ve drawn on the chart. The answer is that nine participants — Alex through Isaac — will have their video on, and the rest will have it off. Why? Well, if everyone has their video on at the start, then Riley, Steve and Tara will turn their video off right away. A cascade will follow: Quinn and Pete, who are only willing to have their video on if everyone else has theirs on, will turn their video off. And so on — up through Jenny. Now Alex through Isaac (but no one else) will have their video on. But at this point the cascade stops: Isaac is happy to keep his video on, as is everyone else. This would also be the end state if everyone started with their video off (assuming there’s no status quo bias). Alex and Betty would turn their video on, Charlie and Diana would follow, and so on, up through Isaac. Indeed, no matter who has their camera on at the start, this will be the end state. Let’s look at a different example. Figure 3: video chat, scenario 2 Now, Alex is willing to keep his video on so long as at least...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why indoor lighting is hard to get right and how to fix it, published by Richard Korzekwa on the LessWrong. The days are getting shorter in the northern hemisphere, and with the ongoing pandemic, most of us expect to be spending more time in our homes than normal. Creating a healthy home environment is more important than usual, and the light inside your home is an often underappreciated part of this. There has already been some explicit discussion[1] about the importance of lighting for health and productivity, as well as many mentions of it in other places. Nonetheless, based on discussions I've had recently within the community, I get the impression that it is helpful for me to write up the results and tinkering that have done over the past few years. First, I will cover some of the research on how our bodies respond to light, and which particular characteristics of natural light we want to mimic. Then I will explain solving this problem is hard and my overall strategy for solving it. Finally, I will give some specific advice on what to buy and how to arrange things. I give quite a lot of background before offering any specific advice. Although I think the background information might help you make good decisions, you should feel free to skip the next section if you're in a hurry or if it seems uninteresting. Background Note: My background is in optics, not physiology or psychology and I began researching and writing this document almost four years ago. My original draft, as well as many of my sources, have been lost in the intervening years, so what you're seeing here is based on a combination of my notes that survived, my recollection of the research, and a partial duplication of the research. To make matters worse, it does seem that new research has come along since I began this project, so this is likely out of date. My guess is that most or all of the practical conclusions still stand, but I am only moderately confident of this. As much as I would like to take the time to update the research, past experience suggests that I will never actually publish it if I try to put too much more work into it. I welcome corrections, and if there is sufficient enthusiasm around this topic, I may try to write an updated version. Your body uses light to synchronize its internal clock and to modulate your mood and alertness[2]. While the particulars of the lighting in your environment are important, your only perceptual access to information about lighting in the moment comes from your visual system, which is poorly adapted to solving the problem of determining the intensity and spectrum of a light source. This is mainly because our vision is optimized more for accurately identifying materials, textures, colors, and other properties of our surroundings than it is for knowing details about sources of light. This has the consequence that some of our default intuitions about the nature of ambient light are wrong, so when we're building our lighting environment, it can be difficult to make accurate judgments just by looking at things. We can do better if we use quantitative measures and our scientific understanding of how things work to solve the problem. Physiology In addition to visual photoreceptors that are used for seeing things, your eyes contain non-visual photoreceptors which serve non-visual functions. Melanopsin is a photopigment that is found in cells in the retina[3]. It is sensitive to blue light, and when activated, these cells send signals that help with things such as regulating our internal clocks[4]. Unlike the our visual photoreceptors, which are more densely packed in the center of our retina[5] than in the periphery, these photosensitive cells are distributed relatively evenly throughout the retina[6], so that light coming from both the periphery and the cent...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Levels of Action, published by alyssavance on the LessWrong. One of the most useful concepts I have learned recently is the distinction between actions which directly improve the world, and actions which indirectly improve the world. Suppose that you go onto Mechanical Turk, open an account, and spend a hundred hours transcribing audio. At current market rates, you'd get paid around $100 for your labor. By taking this action, you have made yourself $100 wealthier. This is an example of what I'd call a Level 1 or object-level action: something that directly moves the world from a less desirable state into a more desirable state. On the other hand, suppose you take a typing class, which teaches you to type twice as fast. On the object level, this doesn't move the world into a better state- nothing about the world has changed, other than you. However, the typing class can still be very useful, because every Level 1 project you tackle later which involves typing will go better- you'll be able to do it more efficiently, and you'll get a higher return on your time. This is what I'd call a Level 2 or meta-level action, because it doesn't make the world better directly - it makes the world better indirectly, by improving the effectiveness of Level 1 actions. There are also Level 3 (meta-meta-level) actions, Level 4 (meta-meta-meta-level actions), and so on. The most important difference between Level 1 and Level 2 actions is that Level 1 actions tend to be additive, while Level 2 actions tend to be multiplicative. If you do ten hours of work at McDonald's, you'll get paid ten times as much as if you did one hour; the benefits of the hours add together. However, if you take ten typing classes, each one of which improves your ability by 20%, you'll be 1.2^10 = 6.2 times better at the end than at the beginning: the benefits of the classes multiply (assuming independence). One result is that spending time on Level 2 actions can have a much greater return than spending time on Level 1 actions. If your labor is worth $20 an hour, and you can't change that, then the amount of money you can earn in a year has a fairly hard upper bound- no matter how you slice it, there are only 168 hours in a week. If you spend that year trying to increase the value of your labor, on the other hand, the upper bound on your performance is both a lot higher (because you can then make more money every year for the next three decades), and a lot more fuzzy. It's a lot more fuzzy because, while everyone has the same number of hours in a week, how effective Level 2 actions are depends a lot on your intelligence, what methods you use, and lots of other stuff. Most Americans spend too little time on higher-level actions, like being strategic - doing a quick analysis of what your goals are, and which Level 1 or Level 2 actions would best accomplish those goals. Witness the hordes of lawyers who spend thirty years on the Level 1 action of working at a law firm, three years on the Level 2 action of getting a law degree, and three minutes on the Level 3 action of deciding what to do after college. (Being strategic is one level up from whichever actions you're being strategic about.) It is also possible to have the opposite problem, of under-valuing Level 1, and I suspect that quite a few people in the nerdier communities do. People sometimes fall into the trap of noticing that the higher levels are (when applied properly) far more useful on the margin than Level 1, and then reacting by giving blind praise to the meta level at the expense of the object level. One cultural example is the ancient Greeks- who, though they were good thinkers for their day, didn't invent science. Science involved actually going out and looking at the world, and that was manual labor and manual labor was for slaves. The ultimate extre...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Cheat to Win: Engineering Positive Social Feedback, published by sarahconstantin on the LessWrong. This post outlines a very simple strategy that's been working for me lately. It may be obvious to some, but it only clicked for me recently. Positive social stimulation is fun for humans, right? We like to be liked. It makes us cheerful. We're motivated to do things that make people smile at us and praise us. But purely optimizing for being liked is a bad idea for lots of reasons: it leads away from your real goals and values, it motivates you to be deceptive, it's kind of shallow and unsatisfying in the long run. So here's what you do instead: first, decide what you actually want to do. Then, seek out people who will socially reward you for doing that, and set yourself up to get social rewards. Marketing experts will tell you that you have to "find your tribe", find the fans of your product, and focus on delighting them. It's fine if you have haters. Haters are almost irrelevant. You succeed if you have enough fans who value your stuff highly enough. This applies across areas of life. You only need (about) one job. You only need one spouse. You only need a small number of close friends. Having great supporters is more important than avoiding having any haters. I used to have the intuition that "fairness" meant I wasn't allowed to bias my social environment in my favor; that I should expose myself equally to people who liked and disliked me, people who did and didn't share my values, in order to get a "balanced" impression of the world. This is pretty stupid, actually. You, as a very small creature moving through infinite space, don't learn about the universe by drawing uniform samples from it. You learn through pursuing goals, which means you'll spend more attention on areas of the universe that are useful to you, which means things that are easy for you or helpful for your life, things that give you energy and resources to explore more. An amoeba, as it crawls around, is going to learn more about the parts of the petri dish with food than the parts without. This is because the amoeba is alive. So are you. As a motivational hack towards any kind of project, it really helps to set yourself up to have recurrent social interactions with people who support you in that project. Meetup groups are good for this. Mixers. Mailing lists. Actually select for people who like the thing you're into, and it's astonishing how much it'll feel like the "world" supports you! Use moments when you're in an energetic, upbeat mood to set up plans for things that'll give you positive social feedback in the future -- make plans to meet people or go to events, or apply to things or submit your work to things. That way you get a recurring stream of "good news" in your inbox, which will trigger more upbeat moods in future. Engineer your social environment to reinforce you for pursuing your goal and you'll be more likely to keep going. A mastermind group is maybe the most explicit example of this kind of engineering. Get 3-7 people together who have similar goals (starting businesses is a common example) and meet regularly to offer support and cheer on each other's progress. The vibe of the mastermind should be "we're all awesome and we're going to succeed together." It's designed to help you keep up momentum. Doing this isn't about wireheading or fooling yourself, it's about focusing your attention, including your social attention, in the areas that can offer rewards instead of the barren spaces. So much defensiveness is unnecessary. Unproductive. It's silly to feel like you have to steel yourself against an unfriendly world if you haven't even checked to look for friends. If you take the attitude of "X is cool and awesome -- who's with me on this?" there's a good chance you'll find a community o...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Rationalist Community Hub in Moscow: 3 Years Retrospective, published by berekuk on the LessWrong. Short summary: Moscow rationalist community is organized around a local venue called Kocherga. Kocherga hosts 40–50 rationality-adjacent events per month, teaches a few dozen people per year in CFAR-style applied rationality workshops, and has ambitious plans for further growth. We on the verge of becoming profitable, but we need some funding to stay afloat. We're launching a Patreon page today. This post is a long overdue report on the state of LessWrong Russia community and specifically on the Kocherga community space in Moscow. This post is also a call for donations. Russian LessWrong community has existed for many years now, but there wasn't much info on lesswrong.com about how we're doing. The last general report by Yuu was posted in March 2013. Alexander230 made a few posts about Fallacymania which is one product, but it's far from the only interesting thing about LW Russia. In the following text I'm going to cover: History of LW Russia from 2013 to 2015 History of Kocherga anticafe, the rationalist community space we started in 2015 The list of Kocherga's successes and failures, as well as some tentative comparisons between Kocherga and Berkeley REACH. Our current financial situation, which is the main reason I'm taking the time to do this write-up now. Current state of Russian and Moscow community Here's a short description of LW Russia and LW Moscow in its current state. Our Russian Slack chat has 1500+ registered members, 150 weekly active members and 50 weekly posting members. LessWrong.ru has a few hundred posts from Sequences and other sources (e.g. from SlateStarCodex) translated by volunteers. LessWrong.ru gets 800 daily unique visitors. There's also a wiki with >100 articles and a VK.com page with 13000 followers. There are regular meetups in Saint-Petersburg (hard to measure, they had a few pivots in their approach in the last few years) and in Yekaterinburg. Our Moscow community hub, Kocherga, which is the main topic of this post, hosts 40–50 public events per month (45 events in June). The attendance varies from 3-5 visitors for smaller events to 20–30 visitors for larger ones. We have ties with a local skeptic society, critical thinking crowd, pop-science communicators, a local systems management school and a local transhumanist community. In the following text I'm going to talk mostly about LW Moscow and Kocherga, because that's what I'm involved with and that's what occupies most of my time. I've been working on Kocherga full-time since 2015, funding it by myself, and I'd love to be able to keep doing it indefinitely. Some more specific stats about Kocherga: We get 1000 non-unique visits per month. The lower bound on the number of unique visitors per month is 150, but that doesn't include "anonymous" visitors who don’t have a club card and so can’t be tracked across different days; also, it doesn't say anything about the number of rationality-adjacent visitors (some people use Kocherga space for their own goals - coworking, etc.) The total number of people who participated in our paid applied rationality courses and workshops over the last 3 years is 150+, not including a few corporate trainings and summer camps. Our monthly revenue is $6000–$9000, depending on the time of the year and on whether we skipped a workshop that month. Our monthly expenses is around $8500. (Half of it is rent, I'll provide a detailed financial breakdown below.) We have maaaybe 2–3 months of cash runway left at this point :( Almost all our events are either directly connected to rationality (LW meetups, rationality dojos, nonviolent communication practice, Sequences discussion) or at least rationality-aligned (topics vary from pop-sci to board games). LW Russia before 201...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Noticing Frame Differences, published by Raemon on the LessWrong. Previously: Keeping Beliefs Cruxy When disagreements persist despite lengthy good-faith communication, it may not just be about factual disagreements – it could be due to people operating in entirely different frames — different ways of seeing, thinking and/or communicating. If you can’t notice when this is happening, or you don’t have the skills to navigate it, you may waste a lot of time. Examples of Broad Frames Gears-oriented Frames Bob and Alice’s conversation is about cause and effect. Neither of them are planning to take direct actions based on their conversation, they’re each just interested in understanding a particular domain better. Bob has a model of the domain that includes gears A, B, C and D. Alice has a model that includes gears C, D and F. They’re able to exchange information, and their information is compatible,and they each end up with a shared model of how something works. There are other ways this could have gone. Ben Pace covered some of them in a sketch of good communication: Maybe they discover their models don’t fit, and one of them is wrong Maybe combining their models results in a surprising, counterintuitive outcome that takes them awhile to accept. Maybe they fail to integrate their models, because they were working at different levels of abstraction and didn’t realize it. Sometimes they might fall into subtler traps. Maybe the thing Alice is calling “Gear C” is actually different from Bob’s “Gear C”. It turns out that they were using the same words to mean different things, and even though they’d both read blogposts warning them about that they didn’t notice. So Bob tries to slot Alice’s gear F into his gear C and it doesn’t fit. If he doesn’t already have reason to trust Alice’s epistemics, he may conclude Alice is crazy (instead of them referring to subtly different concepts). This may cause confusion and distrust. But, the point of this blogpost is that Alice and Bob have it easy. They’re actually trying to have the same conversation. They’re both trying to exchange explicit models of cause-and-effect, and come away with a clearer understanding of the world through a reductionist lens. There are many other frames for a conversation though. Feelings-Oriented Frames Clark and Dwight are exploring how they feel and relate to each other. The focus of the conversation might be navigating their particular relationship, or helping Clark understand why he’s been feeling frustrated lately When the Language of Feelings justifies itself to the Language of Gears, it might say things like: “Feelings are important information, even if it’s fuzzy and hard to pin down or build explicit models out of. If you don’t have a way to listen and make sense of that information, your model of the world is going to be impoverished. This involves sometimes looking at things through lenses other than what you can explicitly verbalize.” I think this is true, and important. The people who do their thinking through a gear-centric frame should be paying attention to feelings-centric frames for this reason. (And meanwhile, feelings themselves totally have gears that can be understood through a mechanistic framework) But for many people that’s not actually the point when looking through a feelings-centric frame. And not understanding this may lead to further disconnect if a Gearsy person and a Feelingsy person are trying to talk. “Yeah feelings are information, but, also, like, man, you’re a human being with all kinds of fascinating emotions that are an important part of who you are. This is super interesting! And there’s a way of making sense of it that’s necessarily experiential rather than about explicit, communicable knowledge.” Frames of Power and Negotiation Dominance and Threat Erica is Frank’s bo...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The ground of optimization, published by alexflint on the LessWrong. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. This work was supported by OAK, a monastic community in the Berkeley hills. This document could not have been written without the daily love of living in this beautiful community. The work involved in writing this cannot be separated from the sitting, chanting, cooking, cleaning, crying, correcting, fundraising, listening, laughing, and teaching of the whole community. What is optimization? What is the relationship between a computational optimization process — say, a computer program solving an optimization problem — and a physical optimization process — say, a team of humans building a house? We propose the concept of an optimizing system as a physically closed system containing both that which is being optimized and that which is doing the optimizing, and defined by a tendency to evolve from a broad basin of attraction towards a small set of target configurations despite perturbations to the system. We compare our definition to that proposed by Yudkowsky, and place our work in the context of work by Demski and Garrabrant’s Embedded Agency, and Drexler’s Comprehensive AI Services. We show that our definition resolves difficult cases proposed by Daniel Filan. We work through numerous examples of biological, computational, and simple physical systems showing how our definition relates to each. Introduction In the field of computer science, an optimization algorithm is a computer program that outputs the solution, or an approximation thereof, to an optimization problem. An optimization problem consists of an objective function to be maximized or minimized, and a feasible region within which to search for a solution. For example we might take the objective function x 2 − 2 2 as a minimization problem and the whole real number line as the feasible region. The solution then would be x √ 2 and a working optimization algorithm for this problem is one that outputs a close approximation to this value. In the field of operations research and engineering more broadly, optimization involves improving some process or physical artifact so that it is fit for a certain purpose or fulfills some set of requirements. For example, we might choose to measure a nail factory by the rate at which it outputs nails, relative to the cost of production inputs. We can view this as a kind of objective function, with the factory as the object of optimization just as the variable x was the object of optimization in the previous example. There is clearly a connection between optimizing the factory and optimizing for x, but what exactly is this connection? What is it that identifies an algorithm as an optimization algorithm? What is it that identifies a process as an optimization process? The answer proposed in this essay is: an optimizing system is a physical process in which the configuration of some part of the universe moves predictably towards a small set of target configurations from any point in a broad basin of optimization, despite perturbations during the optimization process. We do not imagine that there is some engine or agent or mind performing optimization, separately from that which is being optimized. We consider the whole system jointly — engine and object of optimization — and ask whether it exhibits a tendency to evolve towards a predictable target configuration. If so, then we call it an optimizing system. If the basin of attraction is deep and wide then we say that this is a robust optimizing system. An optimizing system as defined in this essay is known in dynamical systems theory as a dynamical system with one or more attractors. In this essay we show how this framework can help to understand optimization as manifested in p...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Power dynamics as a blind spot or blurry spot in our collective world-modeling, especially around AI, published by Andrew_Critch on LessWrong. Where I’m coming from Epistemic status: personal experience In a number of prior posts, and in ARCHES, I’ve argued that more existential safety consideration is needed on the topic of multi-principal/multi-agent (multi/multi) dynamics among powerful AI systems. In general, I have found it much more difficult to convince thinkers within and around LessWrong’s readership base to attend to multi/multi dynamics, as opposed to, say, convincing generally morally conscious AI researchers who are not (yet) closely associated with the effective altruism or rationality communities. Because EA/rationality discourse is particularly concerned with maintaining good epistemic processes, I think it would be easy to conclude from this state of affairs that multi/multi dynamics are not important (because communities with great concern for epistemic process do not care about them much), and AI researchers who do care about multi/multi dynamics have “bad epistemics” (e.g., because they have been biased by institutionalized trends). In fact, more than one LessWrong reader has taken these positions with me in private conversation, in good faith (I’m almost certain). In this post, I wish to share an opposing concern: that the EA and rationality communities have become systematically biased to ignore multi/multi dynamics, and power dynamics more generally. A history of systemic avoidance Epistemic status: self-evidently important considerations based on somewhat-publicly verifiable facts/trends. Our neglect of multi/multi dynamics has not been coincidental. For a time, influential thinkers in the rationality community intentionally avoided discussions of multi/multi dynamics, so as to avoid contributing to the sentiment that the development and use of AI technology would be driven by competitive (imperfectly cooperative) motives. (FWIW, I also did this sometimes.) The idea was that we — the rationality community — should avoid developing narratives that could provoke businesses and state leaders into worrying about whose values would be most represented in powerful AI systems, because that might lead them to go to war with each other, ideologically or physically. Indeed, there was a time when this community — particularly the Singularity Institute — represented a significant share of public discourse on the future of AI technology, and it made sense to be thoughtful about how to use that influence. Eliezer recently wrote (in a semi-private group, but with permission to share): The vague sense of assumed common purpose, in the era of AGI-alignment thinking from before Musk, was a fragile equilibrium, one that I had to fight to support every time some wise fool sniffed and said "Friendly to who?". Maybe somebody much weaker than Elon Musk could and inevitably would have smashed that equilibrium with much less of a financial investment, reducing Musk's "counterfactual impact". Maybe I'm an optimistic fool for thinking that this axis didn't just go from 0%-in-practice to 0%-in-practice. But I am still inclined to consider people a little responsible for the thing that they seem to have proximally caused according to surface appearances. That vague sense of common purpose might have become stronger if it had been given more time to grow and be formalized, rather than being smashed. That ship has now sailed. Perhaps it was right to worry that our narratives could trigger competition between states and companies, or perhaps the competitive dynamic was bound to emerge anyway and it was hubristic to think ourselves so important. Either way, carefully avoiding questions about multi/multi dynamics on LessWrong or The Alignment Forum will not turn back the clock...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Parable On Obsolete Ideologies, published by Scott Alexander on LessWrong. Followup to: Yudkowsky and Frank on Religious Experience, Yudkowksy and Frank On Religious Experience Pt 2 With sincere apologies to: Mike Godwin You are General Eisenhower. It is 1945. The Allies have just triumphantly liberated Berlin. As the remaining leaders of the old regime are being tried and executed, it begins to become apparent just how vile and despicable the Third Reich truly was. In the midst of the chaos, a group of German leaders come to you with a proposal. Nazism, they admit, was completely wrong. Its racist ideology was false and its consequences were horrific. However, in the bleak poverty of post-war Germany, people need to keep united somehow. They need something to believe in. And a whole generation of them have been raised on Nazi ideology and symbolism. Why not take advantage of the national unity Nazism provides while discarding all the racist baggage? "Make it so," you say. The swastikas hanging from every boulevard stay up, but now they represent "traditional values" and even "peace". Big pictures of Hitler still hang in every government office, not because Hitler was right about racial purity, but because he represents the desire for spiritual purity inside all of us, and the desire to create a better society by any means necessary. It's still acceptable to shout "KILL ALL THE JEWS AND GYPSIES AND HOMOSEXUALS!" in public places, but only because everyone realizes that Hitler meant "Jews" as a metaphor for "greed", "gypsies" as a metaphor for "superstition", and "homosexuals" as a metaphor for "lust", and so what he really meant is that you need to kill the greed, lust, and superstition in your own heart. Good Nazis love real, physical Jews! Some Jews even choose to join the Party, inspired by their principled stand against spiritual evil. The Hitler Youth remains, but it's become more or less a German version of the Boy Scouts. The Party infrastructure remains, but only as a group of spiritual advisors helping people fight the untermenschen in their own soul. They suggest that, during times of trouble, people look to Mein Kampf for inspiration. If they open to a sentence like "The Aryan race shall conquer all in its path", then they can interpret "the Aryan race" to mean "righteous people", and the sentence is really just saying that good people can do anything if they set their minds to it. Isn't that lovely? Soon, "Nazi" comes to just be a synonym for "good person". If anyone's not a member of the Nazi Party, everyone immediately becomes suspicious. Why is she against exterminating greed, lust, and superstition from her soul? Does she really not believe good people can do anything if they set their minds to it? Why does he oppose caring for your aging parents? We definitely can't trust him with high political office. It is four years later. Soon, the occupation will end, and Germany will become an independent country once again. The Soviets have already taken East Germany and turned it Communist. As the de facto ruler of West Germany, its fate is in your hands. You ask your two most trusted subordinates for advice. First, Colonel F gives his suggestion. It is vital that you order the preservation of the Nazi ideology so that Germany remains strong. After all, the Germans will need to stay united as a people in order to survive the inevitable struggle with the Soviets. If Nazism collapsed, then people would lose everything that connects them together, and become dispirited. The beautiful poetry of Mein Kampf speaks to something deep in the soul of every German, and if the Allies try to eradicate that just because they disagree with one outdated interpretation of the text, they will have removed meaning from the lives of millions of people all in the name of some ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Discontinuous progress in history: an update, published by KatjaGrace on LessWrong. I. The search for discontinuities We’ve been looking for historic cases of discontinuously fast technological progress, to help with reasoning about the likelihood and consequences of abrupt progress in AI capabilities. We recently finished expanding this investigation to 37 technological trends.1 This blog post is a quick update on our findings. See the main page on the research and its outgoing links for more details. We found ten events in history that abruptly and clearly contributed more to progress on some technological metric than another century would have seen on the previous trend.2 Or as we say, we found ten events that produced ‘large’, ‘robust’ ‘discontinuities’. How we measure the size of a discontinuity (by Rick Korzekwa) Another five events caused robust discontinuities of between ten and a hundred years (‘moderate robust discontinuities’). And 48 more events caused some trend to depart from our best guess linear or exponential extrapolation of its past progress by at least ten years (and often a hundred), but did so in the context of such unclear past trends that this did not seem clearly remarkable.3 I call all of these departures ‘discontinuities’, and distinguish those that are clearly outside plausible extrapolations of the past trend, according to my judgment, as ‘robust discontinuities’.4 Much of the data involved in this project seems at least somewhat unreliable, and the methods involve many judgments, and much ignoring of minor issues. So I would not be surprised if more effort could produce numerous small changes. However I expect the broad outlines to be correct.5 II. The discontinuities Large robust discontinuities Here is a quick list of the robust 100-year discontinuous events, which I’ll describe in more detail beneath: The Pyramid of Djoser, 2650BC (discontinuity in structure height trends) The SS Great Eastern, 1858 (discontinuity in ship size trends) The first telegraph, 1858 (discontinuity in speed of sending a 140 character message across the Atlantic Ocean) The second telegraph, 1866 (discontinuity in speed of sending a 140 character message across the Atlantic Ocean) The Paris Gun, 1918 (discontinuity in altitude reached by man-made means) The first non-stop transatlantic flight, in a modified WWI bomber, 1919 (discontinuity in both speed of passenger travel across the Atlantic Ocean and speed of military payload travel across the Atlantic Ocean) The George Washington Bridge, 1931 (discontinuity in longest bridge span) The first nuclear weapons, 1945 (discontinuity in relative effectiveness of explosives) The first ICBM, 1958 (discontinuity in average speed of military payload crossing the Atlantic Ocean) YBa2Cu3O7 as a superconductor, 1987 (discontinuity in warmest temperature of superconduction) The Pyramid of Djoser, 2650BC Discontinuity in structure height trends6 The Pyramid of Djoser is considered to be ‘the earliest colossal stone structure’ in Egypt. According to Wikipedia’s data, it took seven thousand years for the tallest structures to go from five to thirteen meters tall7 and then suddenly the Egyptian pyramids shot up to a height of 146.5m over about a hundred years and five successively tallest pyramids. The Pyramid of Djoser, By Charles J Sharp – Own work, from Sharp Photography, sharpphotography, CC BY-SA 3.0, Link The first of these five is the Pyramid of Djoser, standing 62.5m tall. The second one—Meidum Pyramid—is also a large discontinuity in structure height trends by our calculation, but I judge it not robust, since it is fairly unclear what the continuation of the trend should be after the first discontinuity. As is common, the more basic thing going on seems to be a change in the growth rate, and the discontinuity of the P...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Literature Review For Academic Outsiders: What, How, and Why , published by namespace on LessWrong. This is a linkpost for A few years ago I wrote a comment on LessWrong about how most authors on the site probably don't know how to do a literature review: On the one hand, I too resent that LW is basically an insight porn factory near completely devoid of scholarship. On the other hand, this is not a useful comment. I can think of at least two things you could have done to make this a useful comment: Specified even a general direction of where you feel the body of economic literature could have been engaged. I know you might resent doing someone elses research for them if you’re not already familiar with said body, but frankly the norm right now is to post webs spun from the fibrous extrusions of peoples musing thoughts. The system equilibrium isn’t going to change unless some effort is invested into moving it. Notice you could write your comment on most posts while only changing a few words. Provide advice on how one might go about engaging with ‘the body of economic literature’. Many people are intelligent and reasonably well informed, but not academics. Taking this as an excuse to mark them swamp creatures beyond assistance is both lazy and makes the world worse. You could even link to reasonably well written guides from someone else if you don’t want to invest the effort (entirely understandable). I also linked a guide from Harvard's library (Garson & Lillvick, 2012) on how to do a literature review. But this guide makes extensive use of flash video, which makes it increasingly hard to access the content. Even if flash was alive and well, video is not necessarily the most comfortable format. Worse still, I remember feeling there was a great deal of tacit knowledge excluded from the guide which wouldn't be apparent to someone that isn't already familiar with academic culture. Even if the guide was a perfect representation of how to do an academic literature review, the priorities and types of work put together by LessWrong authors are more outsider science (Dance, 2008) than they are Harvard. For this reason I've had writing a guide to literature review aimed towards academic outsiders on my to-do list for a while. At the same time I'm not interested in reinventing the wheel. This guide is going to focus specifically on filling in the knowledge gaps I would expect from someone who has never stepped foot inside a college campus. The other aspects have been discussed in detail, and where they come up I'll link to external guides. What is a literature review? 'Literature review' the process is a way to become familiar with what work has already been done in a particular field or subject by searching for and studying previous work. A 'literature review' is a document (often a small portion of a larger work) which summarizes and analyzes the body of previous work that was encountered during literature review, often in the context of some new work that you're doing. Why do literature review? Literature reviews tend to come up in two major contexts: As a preliminary study to help contextualize a novel work, or as a work itself to summarize the state of a field or synthesize concepts to create new ideas. Most of my research falls into the latter category, I'm a big fan of putting together existing evidence and ideas to synthesize models (namespace, 2020). Gwern also tends to do work in this style (Branwen, 2020). I suspect that a lot of authors on LessWrong are attempting to do this, but fail to really say anything useful because they haven't figured out how to incorporate thorough evidence into their argument. When I did a r...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Six economics misconceptions of mine which I've resolved over the last few years, published by Buck on the effective altruism forum. Here are six cases where I was pretty confident in my understanding of the microeconomics of something, but then later found out I was missing an important consideration. Thanks to Richard Ngo and Tristan Hume for helpful comments. Here’s the list of mistakes: I thought divesting from a company had no effect on the company. I thought that the prices on a prediction market converged to the probabilities of the underlying event. I thought that I shouldn’t expect to be able to make better investment decisions than buying index funds. I had a bad understanding of externalities, which was improved by learning about Coase’s theorem. I didn’t realize that regulations like minimum wages are analogous to taxes in that they disincentivize work. I misunderstood the economics of price controls. In each, I’m not talking about empirical situations at all—I’m just saying that I had a theoretical analysis which I think turned out to be wrong. It’s possible that in many real situations, the additional considerations I’ve learned about don’t actually affect the outcome very much. But it was still an error to not know that those considerations were potentially relevant. 1. Divestment I used to believe that personally divesting in a company didn’t affect its share price, and therefore had no impact on the company. I guess my reasoning here was something like “If the share is worth $10 and you sell it, someone else will just buy it for $10, so the price won’t change”. I was treating shares as if they were worth some fixed amount of money. The simplest explanation for why you can’t just model shares as being worth fixed amounts of money is that people are risk averse, and so the tenth Google share you buy is worth less to you than the first; and so as the price decreases, it becomes more worthwhile to take a bigger risk on the company. As a result, divestment reduces the price of shares, in the same way that selling anything else reduces its price. In the specific case of divestment, this means that when I sell some stocks, the price ends up lower than it was. I first learned I was wrong about this from this Sideways View post, published May 2019. 2. Index funds I used to think that it wasn’t possible for individuals like me to get higher returns than I’d get from just buying an index fund, because in an efficient market, every share is equally valuable. This is wrong for a few reasons. One is that the prices of shares are determined by the risk aversion of other market participants; if your risk aversion is different from the average, some shares (specifically, risky ones) will be much better investments than others. Secondly, because I’m risk averse, I prefer buying shares which are going to do relatively well in worlds where I’m relatively poorer. For example, if I’m a software engineer at a tech company, compared to a random shareholder I should invest more in companies which are as anticorrelated with software engineer salaries as possible. Or if I live in the US, I should consider investing in the markets of other countries. I didn’t understand this fully until around April this year. 3. Prediction markets Relatedly, I thought that the fair market price of a contract which pays out $1 if Trump gets elected is just the probability of Trump getting elected. This is wrong because Trump getting elected is correlated with how valuable other assets are. Suppose I thought that Trump has a 50% chance of getting reelected, and that if he gets re-elected, the stock market will crash. If I have a bunch of my money in the stock market, the contract is worth more than 50 cents, because it hedges against Trump winning. (Here’s a maybe more intuitive way of seeing th...
Welcome to the nonlinear library, where we use text-to-speech software to convert the best writing from the rationalist and ea communities into audio. This is: Thiel on Progress and Stagnation, published by Richard_Ngo on the effective altruism forum. This is a linkpost for Peter Thiel is one of the most exciting and original thinkers of our era, but many of his opinions are scattered across a range of talks and articles. So Jeremy Nixon and I have put together an organised presentation of his views on progress and stagnation, in his own words. The full document, which is a little over 100 pages, is here; below I've listed some of his key quotes. While I don't agree with all of his opinions, I've found many of them very insightful and valuable. I'm particularly interested in understanding how to reconcile his views on stagnation with the sort of accelerationist view of technological progress portrayed here and elsewhere. Key quotes: When tracked against the admittedly lofty hopes of the 1950s and 1960s, technological progress has fallen short in many domains. When we talk about how fast science is progressing, we do it with little precision. Are we accelerating in scientific and technical fields? How fast is this? In response, we get fairly vague answers. I would submit that the consensus in both a Silicon Valley and academic context is that we are doing great and that everything is just moving super fast. All these forms of accelerations. And we can debate whether it’s utopian - Kurzweil with the singularity is near, where all you need to do is sit back and eat some popcorn and watch the movie of the future unfold, or this dystopia, all the science fiction movies from Hollywood and all the robots will kill you, or you’ll be in this matrix - we’re either accelerating to utopia or accelerating to dystopia. The somewhat contrarian thesis I have on this is that perhaps the progress is not as fast as advertised. Things have been slower and have been slower for quite some time. The single most important economic development in recent times has been the broad stagnation of real wages and incomes since 1973, the year when oil prices quadrupled. To a first approximation, the progress in computers and the failure in energy appear to have roughly canceled each other out. Like Alice in the Red Queen’s race, we (and our computers) have been forced to run faster and faster to stay in the same place. Probably the only engineering fields that are doing really well are computer science and maybe, at this point, petroleum engineering. And most other areas of engineering have been bad career decisions the last 40 years . Nuclear engineering, aerospace engineering, were really catastrophic decisions for very talented people to go into. So even though rhetorically we always say that we want more science and engineering people, in practice, these have been extremely tough fields. You could say that all these gadgets and devices, they dazzle us but they also distract us from the ways in which our larger surroundings are strangely old. So we run cell phones while we’re riding in a 19th-century subway system in New York. San Francisco, the housing stock looks like it’s from the 50s and 60s, it’s mostly quite decrepit and incredibly hard to change these sort of things. So you have bits making progress, atoms are strangely very stuck. On our website, we have this tagline – “They promised us flying cars and all we got was 140 characters.” Which is a little bit of a dig at Twitter. But in some sense Twitter is probably a great business. The thousand people who work at Twitter are going to have well-paying jobs. I suspect it will last for decades. It’s probably not enough to take our civilization to the next level. But again it’s a mistake to blame Twitter for that. It’s more a problem with not enough happening elsewhere. The story of specific success that masks generalized failure is one we find very hard to tell. We live in a world where we've been w...
Welcome to the nonlinear library, where we use text-to-speech software to convert the best writing from the rationalist and ea communities into audio. This is: Speaking of Stag Hunts, published by Duncan_Sabien on the effective altruism forum. This is an essay about the current state of the LessWrong community, and the broader EA/rationalist/longtermist communities that it overlaps and bridges, inspired mostly by the dynamics around these three posts. The concepts and claims laid out in Concentration of Force, which was originally written as part one of this essay, are important context for the thoughts below. Summary/thesis, mostly cribbed from user anon03's comment below: In many high-importance and high-emotion discussions on LessWrong, the comments and vote distribution seem very soldier-mindset instead of scout-mindset, and the overall soundness and carefulness of reasoning and discourse seems to me to be much lower than baseline, which already felt a smidge too low. This seems to indicate a failure of the LW community further up the chain (i.e. is a result of a problem, not the problem itself) and I think we should put forth real effort to fix it, and I think the most likely target is something like a more-consistent embrace and enforcement of some very basic rationality discourse norms. (And somewhere in the back of his mind was a small, small note of confusion, a sense of something wrong about that story; and it should have been a part of Harry's art to notice that tiny note, but he was distracted. For it is a sad rule that whenever you are most in need of your art as a rationalist, that is when you are most likely to forget it.) I claim that something has gone a little bit wrong. And as readers of many of my/other/essays/know, I claim that things going a little bit wrong is often actually quite a big problem. I am not alone in thinking that the small scale matters. Tiny mental flinches, itty bitty little incentives, things thrown ever so slightly off course (and then never brought back). That small things often have outsized or cumulative effects is a popular view, either explicitly stated or discernible as an underlying assumption in the writings of Eliezer Yudkowsky, Nate Soares, Logan Brienne Strohl, Scott Alexander, Anna Salamon, and Andrew Critch, just to name a few. Yet I nevertheless feel that I encounter resistance of various forms when attempting to point at small things as if they are important. Resistance rather than cooperative disagreement—impatience, dismissal, often condescension or sneering, sometimes projection and strawmanning. This is absolutely at least in part due to my own clumsiness and confusion. A better version of me, more skilled at communication and empathy and bridging inferential gaps, would undoubtedly run into these problems less. Would better be able to recruit people's general enthusiasm for even rather dull and tedious and unsexy work, on that split-second level. But it seems to me that I can't locate the problem entirely within myself. That there's something out there that's Actually Broken, and that it fights back, at least a little bit, when I try to point at it and fix it. Here's to taking another shot at it. Below is a non-exhaustive list of things which my brain will tend to do, if I don't put forth strategic effort to stop it: Make no attempt to distinguish between what it feels is true and what is reasonable to believe. Make no attempt to distinguish between what it feels is good and what is actually good. Make wildly overconfident assertions that it doesn't even believe (that it will e.g. abandon immediately if forced to make a bet). Weaponize equivocation and maximize plausible deniability à la motte-and-bailey, squeezing the maximum amount of wiggle room out of words and phrases. Say things that it knows will be interpreted a certain way, while knowing that they can be defended as if they meant something more innocent. Neglect the difference between what things look lik...
Welcome to the nonlinear library, where we use text-to-speech software to convert the best writing from the rationalist and ea communities into audio. This is: Biases: An Introduction, published by Rob Bensinger on the effective altruism forum. Imagine reaching into an urn that contains seventy white balls and thirty red ones, and plucking out ten mystery balls. Perhaps three of the ten balls will be red, and you’ll correctly guess how many red balls total were in the urn. Or perhaps you’ll happen to grab four red balls, or some other number. Then you’ll probably get the total number wrong. This random error is the cost of incomplete knowledge, and as errors go, it’s not so bad. Your estimates won’t be incorrect on average, and the more you learn, the smaller your error will tend to be. On the other hand, suppose that the white balls are heavier, and sink to the bottom of the urn. Then your sample may be unrepresentative in a consistent direction. That kind of error is called “statistical bias.” When your method of learning about the world is biased, learning more may not help. Acquiring more data can even consistently worsen a biased prediction. If you’re used to holding knowledge and inquiry in high esteem, this is a scary prospect. If we want to be sure that learning more will help us, rather than making us worse off than we were before, we need to discover and correct for biases in our data. The idea of cognitive bias in psychology works in an analogous way. A cognitive bias is a systematic error in how we think, as opposed to a random error or one that’s merely caused by our ignorance. Whereas statistical bias skews a sample so that it less closely resembles a larger population, cognitive biases skew our thinking so that it less accurately tracks the truth (or less reliably serves our other goals). Maybe you have an optimism bias, and you find out that the red balls can be used to treat a rare tropical disease besetting your brother, and you end up overestimating how many red balls the urn contains because you wish the balls were mostly red. Like statistical biases, cognitive biases can distort our view of reality, they can’t always be fixed by just gathering more data, and their effects can add up over time. But when the miscalibrated measuring instrument you’re trying to fix is you, debiasing is a unique challenge. Still, this is an obvious place to start. For if you can’t trust your brain, how can you trust anything else? Noticing Bias Imagine meeting someone for the first time, and knowing nothing about them except that they’re shy. Question: Is it more likely that this person is a librarian, or a salesperson? Most people answer “librarian.” Which is a mistake: shy salespeople are much more common than shy librarians, because salespeople in general are much more common than librarians—seventy-five times as common, in the United States.¹ This is base rate neglect: grounding one’s judgments in how well sets of characteristics feel like they fit together, and neglecting how common each characteristic is in the population at large.² Another example of a cognitive bias is the sunk cost fallacy—people’s tendency to feel committed to things they’ve spent resources on in the past, when they should be cutting their losses and moving on. Knowing about these biases, unfortunately, doesn’t make you immune to them. It doesn’t even mean you’ll be able to notice them in action. In a study of bias blindness, experimental subjects predicted that they would have a harder time neutrally evaluating the quality of paintings if they knew the paintings were by famous artists. And indeed, these subjects exhibited the very bias they had predicted when the experimenters later tested their prediction. When asked afterward, however, the very same subjects claimed that their assessments of the paintings had been objective and unaffected by the bias.³ Even when we correctly identify others’ biases, we exhibit a bias blind spot when it comes to ...
Welcome to the nonlinear library, where we use text-to-speech software to convert the best writing from the rationalist and ea communities into audio. This is: Technological stagnation: Why I came around, published by jasoncrawford on the effective altruism forum. This is a linkpost for “We wanted flying cars, instead we got 140 characters,” says Peter Thiel’s Founders Fund, expressing a sort of jaded disappointment with technological progress. (The fact that the 140 characters have become 280, a 100% increase, does not seem to have impressed him.) Thiel, along with economists such as Tyler Cowen (The Great Stagnation) and Robert Gordon (The Rise and Fall of American Growth), promotes a “stagnation hypothesis”: that there has been a significant slowdown in scientific, technological, and economic progress in recent decades—say, for a round number, since about 1970, or the last ~50 years. When I first heard the stagnation hypothesis, I was skeptical. The arguments weren’t convincing to me. But as I studied the history of progress (and looked at the numbers), I slowly came around, and now I’m fairly convinced. So convinced, in fact, that I now seem to be more pessimistic about ending stagnation than some of its original proponents. In this essay I’ll try to capture both why I was originally skeptical, and also why I changed my mind. If you have heard some of the same arguments that I did, and are skeptical for the same reasons, maybe my framing of the issue will help. Stagnation is relative To get one misconception out of the way first: “stagnation” does not mean zero progress. No one is claiming that. There wasn’t zero progress even before the Industrial Revolution (or the civilizations of Europe and Asia would have looked no different in 1700 than they did in the days of nomadic hunter-gatherers, tens of thousands of years ago). Stagnation just means slower progress. And not even slower than that pre-industrial era, but slower than, roughly, the late 1800s to mid-1900s, when growth rates are said to have peaked. Because of this, we can’t resolve the issue by pointing to isolated advances. The microwave, the air conditioner, the electronic pacemaker, a new cancer drug—these are great, but they don’t disprove stagnation. Stagnation is relative, and so to evaluate the hypothesis we must find some way to compare magnitudes. This is difficult. Only 140 characters? “We wanted flying cars, instead we got a supercomputer in everyone’s pocket and a global communications network to connect everyone on the planet to each other and to the whole of the world’s knowledge, art, philosophy and culture.” When you put it that way, it doesn’t sound so bad. Indeed, the digital revolution has been absolutely amazing. It’s up there with electricity, the internal combustion engine, or mass manufacturing: one of the great, fundamental, transformative technologies of the industrial age. (Although admittedly it’s hard to see the effect of computers in the productivity statistics, and I don’t know why.) But we don’t need to downplay the magnitude of the digital revolution to see stagnation; conversely, proving its importance will not defeat the stagnation hypothesis. Again, stagnation is relative, and we must find some way to compare the current period to those that came before. Argumentum ad living room Eric Weinstein proposes a test: “Go into a room and subtract off all of the screens. How do you know you’re not in 1973, but for issues of design?” This too I found unconvincing. It felt like a weak thought experiment that relied too much on intuition, revealing one’s own priors more than anything else. And why should we necessarily expect progress to be visible directly from the home or office? Maybe it is happening in specialized environments that the layman wouldn’t have much intuition about: in the factory, the power plant, the agricultural field, the hospital, the oil rig, the cargo ship, the research lab. No progress except for all the progress ...
Welcome to the nonlinear library, where we use text-to-speech software to convert the best writing from the rationalist and ea communities into audio. This is: On the importance of Less Wrong, or another single conversational locus, published by AnnaSalamon on the LessWrong. Epistemic status: My actual best bet. But I used to think differently; and I don't know how to fully explicate the updating I did (I'm not sure what fully formed argument I could give my past self, that would cause her to update), so you should probably be somewhat suspicious of this until explicated. And/or you should help me explicate it. It seems to me that: The world is locked right now in a deadly puzzle, and needs something like a miracle of good thought if it is to have the survival odds one might wish the world to have. Despite all priors and appearances, our little community (the "aspiring rationality" community; the "effective altruist" project; efforts to create an existential win; etc.) has a shot at seriously helping with this puzzle. This sounds like hubris, but it is at this point at least partially a matter of track record.[1] To aid in solving this puzzle, we must probably find a way to think together, accumulatively. We need to think about technical problems in AI safety, but also about the full surrounding context -- everything to do with understanding what the heck kind of a place the world is, such that that kind of place may contain cheat codes and trap doors toward achieving an existential win. We probably also need to think about "ways of thinking" -- both the individual thinking skills, and the community conversational norms, that can cause our puzzle-solving to work better. [2] One feature that is pretty helpful here, is if we somehow maintain a single "conversation", rather than a bunch of people separately having thoughts and sometimes taking inspiration from one another. By "a conversation", I mean a space where people can e.g. reply to one another; rely on shared jargon/shorthand/concepts; build on arguments that have been established in common as probably-valid; point out apparent errors and then have that pointing-out be actually taken into account or else replied-to). One feature that really helps things be "a conversation" in this way, is if there is a single Schelling set of posts/etc. that people (in the relevant community/conversation) are supposed to read, and can be assumed to have read. Less Wrong used to be a such place; right now there is no such place; it seems to me highly desirable to form a new such place if we can. We have lately ceased to have a "single conversation" in this way. Good content is still being produced across these communities, but there is no single locus of conversation, such that if you're in a gathering of e.g. five aspiring rationalists, you can take for granted that of course everyone has read posts such-and-such. There is no one place you can post to, where, if enough people upvote your writing, people will reliably read and respond (rather than ignore), and where others will call them out if they later post reasoning that ignores your evidence. Without such a locus, it is hard for conversation to build in the correct way. (And hard for it to turn into arguments and replies, rather than a series of non sequiturs.) It seems to me, moreover, that Less Wrong used to be such a locus, and that it is worth seeing whether Less Wrong or some similar such place[3] may be a viable locus again. I will try to post and comment here more often, at least for a while, while we see if we can get this going. Sarah Constantin, Ben Hoffman, Valentine Smith, and various others have recently mentioned planning to do the same. I suspect that most of the value generation from having a single shared conversational locus is not captured by the individual generating the value (I suspect there is much distributed value from having "a conversation" with better structural integrity / more coherence, but that the va...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Book Review: Design Principles of Biological Circuits, published by johnswentworth on the LessWrong. I remember seeing a talk by a synthetic biologist, almost a decade ago. The biologist used a genetic algorithm to evolve an electronic circuit, something like this: (source) He then printed out the evolved circuit, brought it to his colleague in the electrical engineering department, and asked the engineer to analyze the circuit and figure out what it did. “I refuse to analyze this circuit,” the colleague replied, “because it was not designed to be understandable by humans.” He has a point - that circuit is a big, opaque mess. This, the biologist argued, is the root problem of biology: evolution builds things from random mutation, connecting things up without rhyme or reason, into one giant spaghetti tower. We can take it apart and look at all the pieces, we can simulate the whole thing and see what happens, but there’s no reason to expect any deeper understanding. Organisms did not evolve to be understandable by humans. I used to agree with this position. I used to argue that there was no reason to expect human-intelligible structure inside biological organisms, or deep neural networks, or other systems not designed to be understandable. But over the next few years after that biologist’s talk, I changed my mind, and one major reason for the change is Uri Alon’s book An Introduction to Systems Biology: Design Principles of Biological Circuits. Alon’s book is the ideal counterargument to the idea that organisms are inherently human-opaque: it directly demonstrates the human-understandable structures which comprise real biological systems. Right from the first page of the introduction: . one can, in fact, formulate general laws that apply to biological networks. Because it has evolved to perform functions, biological circuitry is far from random or haphazard. ... Although evolution works by random tinkering, it converges again and again onto a defined set of circuit elements that obey general design principles. The goal of this book is to highlight some of the design principles of biological systems... The main message is that biological systems contain an inherent simplicity. Although cells evolved to function and did not evolve to be comprehensible, simplifying principles make biological design understandable to us. It’s hard to update one’s gut-level instinct that biology is a giant mess of spaghetti without seeing the structure first hand, so the goal of this post is to present just enough of the book to provide some intuition that, just maybe, biology really is human-understandable. This review is prompted by the release of the book’s second edition, just this past August, and that’s the edition I’ll follow through. I will focus specifically on the parts I find most relevant to the central message: biological systems are not opaque. I will omit the last three chapters entirely, since they have less of a gears-level focus and more of an evolutionary focus, although I will likely make an entire separate post on the last chapter (evolution of modularity). Chapters 1-4: Bacterial Transcription Networks and Motifs E-coli has about 4500 proteins, but most of those are chunked together into chemical pathways which work together to perform specific functions. Different pathways need to be expressed depending on the environment - for instance, e-coli won’t express their lactose-metabolizing machinery unless the environment contains lots of lactose and not much glucose (which they like better). In order to activate/deactivate certain genes depending on environmental conditions, bacteria use transcription factors: proteins sensitive to specific conditions, which activate or repress transcription of genes. We can think of the transcription factor activity as the cell’s interna...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: An overview of 11 proposals for building safe advanced AI, published by evhub on the LessWrong. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. This is the blog post version of the paper by the same name. Special thanks to Kate Woolverton, Paul Christiano, Rohin Shah, Alex Turner, William Saunders, Beth Barnes, Abram Demski, Scott Garrabrant, Sam Eisenstat, and Tsvi Benson-Tilsen for providing helpful comments and feedback on this post and the talk that preceded it. This post is a collection of 11 different proposals for building safe advanced AI under the current machine learning paradigm. There's a lot of literature out there laying out various different approaches such as amplification, debate, or recursive reward modeling, but a lot of that literature focuses primarily on outer alignment at the expense of inner alignment and doesn't provide direct comparisons between approaches. The goal of this post is to help solve that problem by providing a single collection of 11 different proposals for building safe advanced AI—each including both inner and outer alignment components. That being said, not only does this post not cover all existing proposals, I strongly expect that there will be lots of additional new proposals to come in the future. Nevertheless, I think it is quite useful to at least take a broad look at what we have now and compare and contrast some of the current leading candidates. It is important for me to note before I begin that the way I describe the 11 approaches presented here is not meant to be an accurate representation of how anyone else would represent them. Rather, you should treat all the approaches I describe here as my version of that approach rather than any sort of canonical version that their various creators/proponents would endorse. Furthermore, this post only includes approaches that intend to directly build advanced AI systems via machine learning. Thus, this post doesn't include other possible approaches for solving the broader AI existential risk problem such as: finding a fundamentally different way of approaching AI than the current machine learning paradigm that makes it easier to build safe advanced AI, developing some advanced technology that produces a decisive strategic advantage without using advanced AI, or achieving global coordination around not building advanced AI via (for example) a persuasive demonstration that any advanced AI is likely to be unsafe. For each of the proposals that I consider, I will try to evaluate them on the following four basic components that I think any story for how to build safe advanced AI under the current machine learning paradigm needs. Outer alignment. Outer alignment is about asking why the objective we're training for is aligned—that is, if we actually got a model that was trying to optimize for the given loss/reward/etc., would we like that model? For a more thorough description of what I mean by outer alignment, see “Outer alignment and imitative amplification.” Inner alignment. Inner alignment is about asking the question of how our training procedure can actually guarantee that the model it produces will, in fact, be trying to accomplish the objective we trained it on. For a more rigorous treatment of this question and an explanation of why it might be a concern, see “Risks from Learned Optimization.” Training competitiveness. Competitiveness is a bit of a murky concept, so I want to break it up into two pieces here. Training competitiveness is the question of whether the given training procedure is one that a team or group of teams with a reasonable lead would be able to afford to implement without completely throwing away that lead. Thus, training competitiveness is about whether the proposed process of producing advanced AI is competitive. ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Yes Requires the Possibility of No, published by Scott Garrabrant on the LessWrong. 1. A group wants to try an activity that really requires a lot of group buy in. The activity will not work as well if there is doubt that everyone really wants to do it. They establish common knowledge of the need for buy in. They then have a group conversation in which several people make comments about how great the activity is and how much they want to do it. Everyone wants to do the activity, but is aware that if they did not want to do the activity, it would be awkward to admit. They do the activity. It goes poorly. 2. Alice strongly wants to believe A. She searches for evidence of A. She implements a biased search, ignoring evidence against A. She finds justifications for her conclusion. She can then point to the justifications, and tell herself that A is true. However, there is always this nagging thought in the back of her mind that maybe A is false. She never fully believes A as strongly as she would have believed it if she just implemented an an unbiased search, and found out that A was, in fact, true. 3. Bob wants Charlie to do a task for him. Bob phrases the request in a way that makes Charlie afraid to refuse. Charlie agrees to do the task. Charlie would have been happy to do the task otherwise, but now Charlie does the task while feeling resentful towards Bob for violating his consent. 4. Derek has an accomplishment. Others often talk about how great the accomplishment is. Derek has imposter syndrome and is unable to fully believe that the accomplishment is good. Part of this is due to a desire to appear humble, but part of it stems from Derek's lack of self trust. Derek can see lots of pressures to believe that the accomplishment is good. Derek does not understand exactly how he thinks, and so is concerned that there might be a significant bias that could cause him to falsely conclude that the accomplishment is better than it is. Because of this he does not fully trust his inside view which says the accomplishment is good. 5. Eve is has an aversion to doing B. She wants to eliminate this aversion. She tries to do an internal double crux with herself. She identifies a rational part of herself who can obviously see that it is good to do B. She identifies another part of herself that is afraid of B. The rational part thinks the other part is stupid and can't imagine being convinced that B is bad. The IDC fails, and Eve continues to have an aversion to B and internal conflict. 6. Frank's job or relationship is largely dependent to his belief in C. Frank really wants to have true beliefs, and so tries to figure out what is true. He mostly concludes that C is true, but has lingering doubts. He is unsure if he would have been able to conclude C is false under all the external pressure. 7. George gets a lot of social benefits out of believing D. He believes D with probability 80%, and this is enough for the social benefits. He considers searching for evidence of D. He thinks searching for evidence will likely increase the probability to 90%, but it has a small probability of decreasing the probability to 10%. He values the social benefit quite a bit, and chooses not to search for evidence because he is afraid of the risk. 8. Harry sees lots of studies that conclude E. However, Harry also believes there is a systematic bias that makes studies that conclude E more likely to be published, accepted, and shared. Harry doubts E. 9. A bayesian wants to increase his probability of proposition F, and is afraid of decreasing the probability. Every time he tries to find a way to increase his probability, he runs into an immovable wall called the conservation of expected evidence. In order to increase his probability of F, he must risk decreasing it. Thanks for listening. to help us out wi...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Jeff Hawkins on neuromorphic AGI within 20 years, published by Steven Byrnes on the LessWrong. I just listened to AI podcast: Jeff Hawkins on the Thousand Brain Theory of Intelligence, and read some of the related papers. Jeff Hawkins is a theoretical neuroscientist; you may have heard of his 2004 book On Intelligence. Earlier, he had an illustrious career in EECS, including inventing the Palm Pilot. He now runs the company Numenta, which is dedicated to understanding how the human brain works (especially the neocortex), and using that knowledge to develop bio-inspired AI algorithms. In no particular order, here are some highlights and commentary from the podcast and associated papers. Every part of the neocortex is running the same algorithm The neocortex is the outermost and most evolutionarily-recent layer of the mammalian brain. In humans, it is about the size and shape of a dinner napkin (maybe 1500cm²×3mm), and constitutes 75% of the entire brain. Jeff wants us to think of it like 150,000 side-by-side "cortical columns", each of which is a little 1mm²×3mm tube, although I don't think we're supposed to the "column" thing too literally (there's no sharp demarcation between neighboring columns). When you look at a diagram of the brain, the neocortex has loads of different parts that do different things—motor, sensory, visual, language, cognition, planning, and more. But Jeff says that all 150,000 of these cortical columns are virtually identical! Not only do they each have the same types of neurons, but they're laid out into the same configuration and wiring and larger-scale structures. In other words, there seems to be "general-purpose neocortical tissue", and if you dump visual information into it, it does visual processing, and if you connect it to motor control pathways, it does motor control, etc. He said that this theory originated with Vernon Mountcastle in the 1970s, and is now widely (but not universally) accepted in neuroscience. The theory is supported both by examining different parts of the brain under the microscope, and also by experiments, e.g. the fact that congenitally blind people can use their visual cortex for non-visual things, and conversely he mentioned in passing some old experiment where a scientist attached the optic nerve of a lemur to a different part of the cortex and it was able to see (or something like that). Anyway, if you accept that premise, then there is one type of computation that the neocortex does, and if we can figure it out, we'll understand everything from how the brain does visual processing to how Einstein's brain invented General Relativity. To me, cortical uniformity seems slightly at odds with the wide variety of instincts we have, like intuitive physics, intuitive biology, language, and so on. Are those not implemented in the neocortex? Are they implemented as connections between (rather than within) cortical columns? Or what? This didn't come up in the podcast. (ETA: I tried to answer this question in my later post, Human instincts, Symbol grounding, and the blank-slate neocortex.) (See also previous LW discussion at: The brain as a universal learning machine, 2015) Grid cells and displacement cells Background: Grid cells for maps in the hippocampus Grid cells, discovered in 2005, help animals build mental maps of physical spaces. (Grid cells are just one piece of a complicated machinery, along with "place cells" and other things, more on which shortly.) Grid cells are not traditionally associated with the neocortex, but rather the entorhinal cortex and hippocampus. But Jeff says that there's some experimental evidence that they're also in the neocortex, and proposes that this is very important. What are grid cells? Numenta has an educational video here. Here's my oversimplified 1D toy example (the modules can als...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Three ways CFAR has changed my view of rationality, published by Julia_Galef on the LessWrong. The Center for Applied Rationality's perspective on rationality is quite similar to Less Wrong's. In particular, we share many of Less Wrong's differences from what's sometimes called "traditional" rationality, such as Less Wrong's inclusion of Bayesian probability theory and the science on heuristics and biases. But after spending the last year and a half with CFAR as we've developed, tested, and attempted to teach hundreds of different versions of rationality techniques, I've noticed that my picture of what rationality looks like has shifted somewhat from what I perceive to be the most common picture of rationality on Less Wrong. Here are three ways I think CFAR has come to see the landscape of rationality differently than Less Wrong typically does – not disagreements per se, but differences in focus or approach. (Disclaimer: I'm not speaking for the rest of CFAR here; these are my own impressions.) 1. We think less in terms of epistemic versus instrumental rationality. Formally, the methods of normative epistemic versus instrumental rationality are distinct: Bayesian inference and expected utility maximization. But methods like "use Bayes' Theorem" or "maximize expected utility" are usually too abstract and high-level to be helpful for a human being trying to take manageable steps towards improving her rationality. And when you zoom in from that high-level description of rationality down to the more concrete level of "What five-second mental habits should I be training?" the distinction between epistemic and instrumental rationality becomes less helpful. Here's an analogy: epistemic rationality is like physics, where the goal is to figure out what's true about the world, and instrumental rationality is like engineering, where the goal is to accomplish something you want as efficiently and effectively as possible. You need physics to do engineering; or I suppose you could say that doing engineering is doing physics, but with a practical goal. However, there's plenty of physics that's done for its own sake, and doesn't have obvious practical applications, at least not yet. (String theory, for example.) Similarly, you need a fair amount of epistemic rationality in order to be instrumentally rational, though there are parts of epistemic rationality that many of us practice for their own sake, and not as a means to an end. (For example, I appreciate clarifying my thinking about free will even though I don't expect it to change any of my behavior.) In this analogy, many skills we focus on at CFAR are akin to essential math, like linear algebra or differential equations, which compose the fabric of both physics and engineering. It would be foolish to expect someone who wasn't comfortable with math to successfully calculate a planet's trajectory or design a bridge. And it would be similarly foolish to expect you to successfully update like a Bayesian or maximize your utility if you lacked certain underlying skills. Like, for instance: Noticing your emotional reactions, and being able to shift them if it would be useful. Doing thought experiments. Noticing and overcoming learned helplessness. Visualizing in concrete detail. Preventing yourself from flinching away from a thought. Rewarding yourself for mental habits you want to reinforce. These and other building blocks of rationality are essential both for reaching truer beliefs, and for getting what you value; they don't fall cleanly into either an "epistemic" or an "instrumental" category. Which is why, when I consider what pieces of rationality CFAR should be developing, I've been thinking less in terms of "How can we be more epistemically rational?" or "How can we be more instrumentally rational?" and instead using queries li...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Rational Home Buying, published by Scott Alexander on the Lesswrong. My parents are considering moving house. I've had a front-seat window to their decision process as they compare alternatives, and sometimes it isn't pretty. A new house is one of the most important purchases most people will make. Because of the sums involved, the usual pitfalls of decision-making gain new importance, and it becomes especially important to make sure you're thinking rationally. Research in a couple of fields, most importantly positive psychology, offers some potentially helpful tips. LOCATION, LOCATION, LOCATION People so consistently under-count the pain of commuting when making choices that the problem has its own name: Commuter's Paradox. The paradox is that, although rational choice theory predicts people should balance commuting against other goods and costs, so that one person might have a longer commute but a nicer (or cheaper) house and so be just as happy overall, this doesn't happen: people who have long commutes are miserable, full stop. A separate survey by Kahneman and Krueger found that commuting was the least enjoyable of nineteen daily activities mentioned, and other studies have found relations between long commutes and poor social lives, poor health, high stress, and various other problems. Psychologists aren't entirely sure why people so consistently under-count the pain of commuting. Maybe it's because it's viewed as "in-between" time rather than as an activity on its own; maybe it's because it comes in relatively short and individually bearable chunks repeated over many years, instead of as a single entity. In any case, unless you are mentally atypical you will probably have a tendency to undercount commute time when buying a new home, and may want to adjust for that tendency. HOUSES COST A LOT OF MONEY One of Kahneman and Tversky's famous bias experiments went like this: imagine you're buying a new shirt. It costs $40 at a nearby store, and it costs $20 at a store that's fifteen minutes away. Do you drive the fifteen minutes to save twenty bucks? Most people would. Now imagine you're buying a new TV which costs $2020 at a nearby store, and $2000 at a store that's fifteen minutes away. Do you drive the fifteen minutes to save twenty bucks? Most people wouldn't. In both cases, the tradeoff is the same - drive fifteen minutes to save twenty bucks - but people were much more willing to do it for the cheap item, because $20 was a higher percentage of its total cost. With the $2000 TV, the $20 vanishes into the total cost like a drop in the ocean and seems insignificant. Nice homes can cost $500,000, $1,000,000, or even more. There doesn't seem to be a big difference in price between $710,000 and $745,000 houses; perhaps if the second home looked even a little nicer in an undefinable way you might be prepared to take it. But $35,000 is $35,000; if those minor advantages don't provide $35,000 worth of value, when measured on the same scale on which you measure the value of of movie tickets, shoes, and college funds, then you should buy the first house and keep the cash. I find purchasing decisions easier when I think about them like this: which would you rather have, the second house, or the first house plus a two-week luxury vacation to anywhere in the world every summer for the next five years? The second house, or the first house plus a brand new Lexus? The second house and dining at home every week, or the first house and eating out at your favorite restaurant every weekend for the rest of your life? (EDIT: gjm points out that it's easier to resell houses than other types of good, so if you expect to resell your house you should really only be considering the extra money involved in the mortgage) DON'T OVERCOUNT EASILY AVAILABLE DETAILS The availability heuristic s...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Noticing the Taste of Lotus, published by Valentine on the LessWrong. Recently I started picking up French again. I remembered getting something out of Duolingo a few years ago, so I logged in. Since the last time I was there, they added an “achievements” mechanic: I noticed this by earning one. I think it was “Sharpshooter”. They gave me the first of three stars for something like doing five lessons without mistakes. In the “achievements” section, it showed me that I could earn the second star by doing twenty lessons in a row flawlessly. And my brain cared. I watched myself hungering to get the achievements. These arbitrary things that someone had just stuck on there. in order to get me to want them. I noticed that I could get the second and maybe third star of “Sharpshooter” by doing earlier lessons and googling words and phrases I wasn’t quite sure about. .which really doesn’t help me learn French. Yes, we could quibble about that. Maybe perfect practice makes perfect, yada yada. But the point is: I disagree, I think my disagreement comes from knowing what I’m talking about when it comes to my learning, and someone’s arbitrary gold stars immediately overrode all that insight by grabbing my motivations directly. I don’t have a problem with gamification per se. What bugs me here is that this specific gamification didn’t fit my goals, and that fact didn’t at all affect how well the system grabbed my wanting. I just. wanted those achievements. Because they were there. If I hadn’t noticed this, and if I’m right about what I need to learn French, then I would have wasted a bunch of time pursuing a useless proxy goal. And I would have felt pleasure in achieving it. I might have even thought that was a meaningful sign that I was learning French — never mind that my goal of holding my own in conversations isn’t really helped by carefully avoiding typos. Duncan Sabien sometimes talks about “lotus-eating”. He’s referring to a part of the Odyssey where they land on an island of “lotus-eaters”. It turns out that once you eat some of this kind of lotus, all you want to do is eat more. You stop caring about your other goals. The lotus just grabs your wants directly. I claim you can notice when something grabs your wanting. Just. look. Just pay attention. Here are some lotuses I’ve noticed: Most computer games are full of these. I sometimes play one called Alto’s Adventure. You flip a little character over and land some tricks, and then get a speed boost. If you collect enough coins, you can get special items or level them up to a maximum. If I start playing it, I notice I care about these arbitrary coins and flips and so on. And if I’ve been playing it recently, I notice myself wanting to pull the game out and play it some more. But what is gained by doing so? Maybe something, but if so then that’s a happy accident. My life isn’t any better after unlocking all the made-up achievements on this little made-up game. But each time I land a trick: BAM! A tiny burst of satisfaction, and a wanting to keep going. Scrolling down on Facebook. There’s something about wanting to scroll a little farther. I get a “Yes!” and a “Just a little more” each time I scroll down and see a new post. Just another couple more minutes on Facebook, right? Oops. Email. Where does the impulse to check email several times a day come from? Or to “catch up” on email? What are you trying to do? What does it feel like when you’ve just clicked “Send”? Inbox zero in particular does this a lot for me. If I have just two emails, I want to reply to them right away, so I can get back to that oh so sacred inbox zero state. But then people reply, and I reply back, and my time gets eaten up. but at least I’m maintaining inbox zero, right? Porn is loudly lotuses. The website Your Brain On Porn goes into this a ton. YouTube...
Welcome to the nonlinear library, where we use text-to-speech software to convert the best writing from the rationalist and ea communities into audio. This is: Growing Independence, published by jefftk on the LessWrong. Write a Review Note: this is based on my experience with my two kids, currently four and six. It may not generalize as much as I think it does. People start out dependent on their parents for food, changing, contact, motion, and even sleep timing. Typically they end up as adults, no longer dependent on their parents at all. Part of my approach to parenting has been that I want to let my kids be as independent as possible, as early as possible. Not only does it make their lives better, because they can meet their own needs how they want, but it makes my life easier, because they can handle more on their own. Sometimes this involves a bit more effort up front, but I think it's substantially less effort in total. Examples: If Lily (6y) comes to me and says "Anna (4y) pushed me," my first response will probably be "have you talked to Anna?" I'll still help some, often by listening to them negotiate and clarifying rules ("you can't push people, even when they happen to be between you and your desired toy") but over time they've gotten much better at this. There's a whole post worth of thoughts that could go here on what's worked and what hasn't, but at this point they can get up an hour before we do and (nearly always) resolve their own conflicts without waking us. The kids will often ask for help while I'm cooking. If I'm in the middle of something, which I usually am, I'll say something like "I can help you as soon as I finish mixing this". During that time they're often able to solve their own problem. If they do still need help when I'm ready, they get my full attention. This is acting as a cost, paying with their time, which filters their requests so I only get the ones where it's worth it to them. And then while they're bored waiting for me they'll often try a bit harder at doing up the snaps on their shirt or whatever, and often that extra focused effort is what they need to do it on their own. Similarly, when Anna was learning to ride her trike and she got to a sidewalk bump that was hard to pedal over, she would call for help. I found that if I walked far enough behind her she would keep trying while she waited for me to catch up, and then often didn't need me by the time I was there. If I hear crying, I don't automatically do something about it. As I was writing this post I heard Anna get up. Then I heard some crying. Not "I've been badly hurt crying" but some sort of frustration. It didn't last very long, and I didn't move, just listening. A few minutes later Anna came down and said good morning. She had a lot she wanted to tell me about the clothes she had picked out. When she was done I asked what they crying had been, and whether she was ok, and she said that Lily hadn't been willing to come out and play with her even though her bedroom light was on. I clarified (again... this one keeps coming up) that Lily isn't required to play with her, and that even if someone's light is on that doesn't necessarily mean they want to come out of their room and get up. Then we cuddled up and read a book together. When the kids started being able to climb things, I would spot them. Often they wanted me to lift them or support them in their climbing, and I wouldn't. They would also want to be lifted down at the end, but the rule would be "if you can climb up, you can climb down." I was willing to give them advice or guide their foot when they couldn't see where to place it, but they still needed to do the climbing. At this point I'll spot them if they ask me to, or maybe say things like "if you're going to climb that high you need to find an adult to spot you." With tree climbing I'm willing to be a stepstool if asked, but I won't lift them. Recently Lily dropped her fork, and asked me to pick it up. I said that thi...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: microCOVID.org: A tool to estimate COVID risk from common activities, published by catherio on the LessWrong. This is a linkpost for/ This is a linkpost for a model and web tool (that I and several friends created) to quantitatively estimate the COVID risk to you from your ordinary daily activities: This website contains three outputs of our work: a web calculator that you can use to calculate your COVID risk (in units of microCOVIDs, a 1-in-a-million chance of getting COVID). a white paper that explains our estimation method. EAs might be particularly interested in the footnotes throughout, and the detailed research sources section. a spreadsheet to compute your COVID risk in more detail and to track your risk over time. EAs might find this more customizable and powerful than the web calculator. If you have different beliefs than us and would like to use a version of the model that reflects your beliefs rather than ours, you can make modifications to your copy of the spreadsheet, or fork the repository and make a personal copy of the web calculator. We also hope you will submit suggestions, either by emailing us or by making issues or pull requests directly on Github. Our group house has been using this model as the basis of a shared agreement/protocol, based on a budget of 3,000 microCOVIDs per year to spend outside the house (about 58 per week). We know of another group house that (last we heard) was operating on 4 microCOVIDs per week! We hope this helps you personally live a better pandemic life with more safety and more flexibility. (also linkposted to the EA Forum) Thanks for listening. to help us out with the nonlinear library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Concentration of Force, published by Duncan_Sabien on the LessWrong. This essay began as part one of a longer piece. Part one is standalone and "timeless." Part two is focused on the local dynamics of the EA/rationality/longtermist communities and LessWrong in November of 2021. Following wise advice from Zack_M_Davis, I've split them into two separate posts. Nevertheless, I recommend that people intending to read both seriously consider reading them back-to-back, so that the content of this one is fresh in the mind. It's both something of a prerequisite and also relevantly context-setting. Introduction Concentration of force is a military concept (sometimes referred to as "mass"). It used to be concentration of forces, until innovations like machine guns and cruise missiles made gathering all of your actual personnel together into more of a liability. The idea is simple. Essentially, there is a difference between relevant moments and irrelevant moments. Battles and non-battles, moments of engagement and moments between engagements. At each relevant moment, you want to project locally superior or overwhelming force. Perhaps this means having the most soldiers/guns/tanks/planes actually present, or perhaps this just means having the right missiles pointed in the right directions. If you are good at coordination and maneuver, you can concentrate force in most or every engagement, and consistently win even against an overall larger or more powerful opponent. This is how guerrilla warfare works—you choose the time and place of conflict in order to ensure that you outnumber the enemy in each specific encounter, and you fade into the mists before their reinforcements arrive. Red wins each of the depicted engagements handily. I claim that the Grey Tribe generally, and rationalists/longtermists/EAs more specifically, and LessWrong the website and community even more specifically, are systematically and spectacularly failing at concentration of force. That none of those groups puts anything like sufficient strategic energy into ensuring that critical mass coheres at crucial moments, and that each would benefit from optimizing their ability to do so quickly, reliably, and effectively, and from thinking in terms of concentration of force as a matter of habit. I anticipate a (reasonable) objection along the lines of "mistake theory rather than conflict theory!" or "generous tit-for-tat rather than vengeful tit-for-tat!" and I assert a further subclaim that the above advice is every bit as relevant for nonviolent and nonconfrontational frames. Actual conflicts are a vanishingly small subset of the times when concentration of force is a relevant principle; the "force" in question could just as easily be e.g. "calm, generous, charitable, level-headed, clear-minded, skilled communicators arriving at exactly the moment when things were about to become wastefully contentious and adversarial." (Indeed, that's a preview of the recommendation I have for LessWrong specifically.) TAPs: A Motivating Example There's a picture I tend to draw quite frequently when giving people crash courses in CFAR-esque rationality, and it looks like this: The idea behind the picture is that you're trucking along, living a generally good and happy life, and then something happens, and you find yourself in the sad timeline. You ate an entire package of Oreos, despite intending to lose weight. You got in another fight with your romantic partner, despite really not wanting to. You road raged, you failed to finish the presentation before the deadline, you spent all evening on Reddit instead of thinking about your research, you somehow never called them back and now it's too awkward. The point of showing people this picture is to draw their attention to two key facts: For most goals and values, there actually exis...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Bayes' Theorem Illustrated (My Way), published by komponisto on the LessWrong. (This post is elementary: it introduces a simple method of visualizing Bayesian calculations. In my defense, we've had other elementary posts before, and they've been found useful; plus, I'd really like this to be online somewhere, and it might as well be here.) I'll admit, those Monty-Hall-type problems invariably trip me up. Or at least, they do if I'm not thinking very carefully -- doing quite a bit more work than other people seem to have to do. What's more, people's explanations of how to get the right answer have almost never been satisfactory to me. If I concentrate hard enough, I can usually follow the reasoning, sort of; but I never quite "see it", and nor do I feel equipped to solve similar problems in the future: it's as if the solutions seem to work only in retrospect. Minds work differently, illusion of transparency, and all that. Fortunately, I eventually managed to identify the source of the problem, and I came up a way of thinking about -- visualizing -- such problems that suits my own intuition. Maybe there are others out there like me; this post is for them. I've mentioned before that I like to think in very abstract terms. What this means in practice is that, if there's some simple, general, elegant point to be made, tell it to me right away. Don't start with some messy concrete example and attempt to "work upward", in the hope that difficult-to-grasp abstract concepts will be made more palatable by relating them to "real life". If you do that, I'm liable to get stuck in the trees and not see the forest. Chances are, I won't have much trouble understanding the abstract concepts; "real life", on the other hand... ...well, let's just say I prefer to start at the top and work downward, as a general rule. Tell me how the trees relate to the forest, rather than the other way around. Many people have found Eliezer's Intuitive Explanation of Bayesian Reasoning to be an excellent introduction to Bayes' theorem, and so I don't usually hesitate to recommend it to others. But for me personally, if I didn't know Bayes' theorem and you were trying to explain it to me, pretty much the worst thing you could do would be to start with some detailed scenario involving breast-cancer screenings. (And not just because it tarnishes beautiful mathematics with images of sickness and death, either!) So what's the right way to explain Bayes' theorem to me? Like this: We've got a bunch of hypotheses (states the world could be in) and we're trying to figure out which of them is true (that is, which state the world is actually in). As a concession to concreteness (and for ease of drawing the pictures), let's say we've got three (mutually exclusive and exhaustive) hypotheses -- possible world-states -- which we'll call H1, H2, and H3. We'll represent these as blobs in space: Figure 0 Figure 0 Now, we have some prior notion of how probable each of these hypotheses is -- that is, each has some prior probability. If we don't know anything at all that would make one of them more probable than another, they would each have probability 1/3. To illustrate a more typical situation, however, let's assume we have more information than that. Specifically, let's suppose our prior probability distribution is as follows: P(H1) = 30%, P(H2)=50%, P(H3) = 20%. We'll represent this by resizing our blobs accordingly: Figure 1 Figure 1 That's our prior knowledge. Next, we're going to collect some evidence and update our prior probability distribution to produce a posterior probability distribution. Specifically, we're going to run a test. The test we're going to run has three possible outcomes: Result A, Result B, and Result C. Now, since this test happens to have three possible results, it would be really nice if the t...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The mechanics of my recent productivity, published by So8res on the LessWrong. A decade ago, I decided to save the world. I was fourteen, and the world certainly wasn't going to save itself. I fumbled around for nine years; it's surprising how long one can fumble around. I somehow managed to miss the whole idea of existential risk and the whole concept of an intelligence explosion. I had plenty of other ideas in my head, and while I spent a lot of time honing them, I wasn't particularly looking for new ones. A year ago, I finally read the LessWrong sequences. My road here was roundabout, almost comical. It took me a while to come to terms with the implications of what I'd read. Five months ago, after resolving a few internal crises, I started donating to MIRI and studying math. Three weeks ago, I attended the December MIRI workshop on logic, probability, and reflection. I was invited to visit for the first two days and stay longer if things went well. They did: I was able to make some meaningful contributions. On Saturday I was invited to become a MIRI research associate. It's been an exciting year, to say the least. (ETA: Note that being a research associate gives me access to a number of MIRI resources, but is not a full time position. I will be doing FAI research, but it will be done outside of work. I will be retaining my day job and continuing to donate.) (ETA: As of 1 April 2014, I am a full-time researcher at MIRI.) (ETA: As of 1 June 2015, I am now the executive director of MIRI.) To commemorate the occasion — and because a few people have expressed interest in my efforts — I'll be writing a series of posts about my experience, about what I did and how I did it. This is the first post in the series. First and foremost, know that I am not done with my aggressive autodidacting. I have a long way to go yet before I'm anywhere near as productive as others who do research with MIRI. I find myself at a checkpoint of sorts, collecting my thoughts in the wake of my first workshop, but next week I will be back to business. One goal of this post is to give you a feel for how much effort is required to become good at MIRI-relevant mathematics in a short time, and perhaps inspire others to follow my path. It was difficult, but not as difficult as you might think. Another goal is to provide data for fellow autodidacts. At the least I can provide you with an anchor point, a single datum about how much effort is required to learn at this pace. As always, remember that I am only one person and that what worked for me may not work for you. In order to understand what I achieved it's important to know where I started from. Thus, allow me to briefly discuss my relevant prior experience. Background I was born in 1989. I have bachelor's degrees of science in both computer science and economics. I started programming TI-83 calculators in late 2002. I've been programming professionally since 2008. I currently work for Google and live in Seattle. In high school I had a knack for math. I was placed two years ahead of my classmates. I aced some AP tests, I won some regional math competitions, nothing much came of it. I explicitly decided not to pursue mathematics: I reasoned that in order to save the world I would need charisma, knowledge of how the world economy works, and a reliable source of cash. This (and my love of programming) drove my choice of majors. During college I soaked up computer science like a sponge. (Economics, too, but that's not as relevant here.) I came out of college with a strong understanding of the foundations of computing: algorithms, data structures, discrete math, etcetera. I cultivated a love for information theory. Outside of the computer science department I took two math classes: multi variable calculus and real analysis. I was careful not to let schoo...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Costly Coordination Mechanism of Common Knowledge, published by Ben Pace on the LessWrong. Recently someone pointed out to me that there was no good canonical post that explained the use of common knowledge in society. Since I wanted to be able to link to such a post, I decided to try to write it. The epistemic status of this post is that I hoped to provide an explanation for a standard, mainstream idea, in a concrete way that could be broadly understood rather than in a mathematical/logical fashion, and so the definitions should all be correct, though the examples in the latter half are more speculative and likely contain some inaccuracies. Let's start with a puzzle. What do these three things have in common? Dictatorships all through history have attempted to suppress freedom of the press and freedom of speech. Why is this? Are they just very sensitive? On the other side, the leaders of the Enlightenment fought for freedom of speech, and would not budge an inch against this principle. When two people are on a date and want to sleep with each other, the conversation will often move towards but never explicitly discuss having sex. The two may discuss going back to the place of one of theirs, with a different explicit reason discussed (e.g. "to have a drink"), even if both want to have sex. Throughout history, communities have had religious rituals that look very similar. Everyone in the village has to join in. There are repetitive songs, repetitive lectures on the same holy books, chanting together. Why, of all the possible community events (e.g. dinner, parties, etc) is this the most common type? What these three things have in common, is common knowledge - or at least, the attempt to create it. Before I spell that out, we’ll take a brief look into game theory so that we have the language to describe clearly what’s going on. Then we’ll be able to see concretely in a bunch of examples, how common knowledge is necessary to understand and build institutions. Prisoner's Dilemmas vs Coordination Problems To understand why common knowledge is useful, I want to contrast two types of situations in game theory: Prisoner’s Dilemmas and Coordination Problems. They look similar at first glance, but their payoff matrices have important differences. The Prisoner's Dilemma (PD) You’ve probably heard of it - two players have the opportunity to cooperate, or defect against each other, based on a story about two prisoners being offered a deal if they testify against the other. If they do nothing they will put them both away for a short time; if one of them snitches on the other, the snitch gets off free and the snitched gets a long sentence. However if they both snitch they get pretty bad sentences (though neither are as long as when only one snitches on the other). In game theory, people often like to draw little boxes that show two different people's choices, and how much they like the outcome. Such a diagram is called a decision matrix, and the numbers are called the players' payoffs. To describe the Prisoner's Dilemma, below is a decision matrix where Anne and Bob each have the same two choices, labelled C and D . These are colloquially called ‘cooperate’ and ‘defect’. Each box contains two numbers, for Anne and Bob's payoffs respectively. If the prisoner ‘defects’ on his partner, this means he snitches, and if he ‘cooperates’ with his partner, he doesn’t snitch. They’d both prefer that both of them cooperate C C to both of them defecting D D , but each of them has an incentive to stab each other in the back to reap the most reward D C Do you see in the matrix how they both would prefer no snitching to both snitching, but they also have an incentive to stab each other in the back? Real World Examples Nuclear disarmament is a prisoner’s dilemma. Both the Soviet Union and the U...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Mistakes with Conservation of Expected Evidence, published by abramdemski on the LessWrong. Epistemic Status: I've really spent some time wrestling with this one. I am highly confident in most of what I say. However, this differs from section to section. I'll put more specific epistemic statuses at the end of each section. Some of this post is generated from mistakes I've seen people make (or, heard people complain about) in applying conservation-of-expected-evidence or related ideas. Other parts of this post are based on mistakes I made myself. I think that I used a wrong version of conservation-of-expected-evidence for some time, and propagated some wrong conclusions fairly deeply; so, this post is partly an attempt to work out the right conclusions for myself, and partly a warning to those who might make the same mistakes. All of the mistakes I'll argue against have some good insight behind them. They may be something which is usually true, or something which points in the direction of a real phenomenon while making an error. I may come off as nitpicking. 1. "You can't predict that you'll update in a particular direction." Starting with an easy one. It can be tempting to simplify conservation of expected evidence to say you can't predict the direction which your beliefs will change. This is often approximately true, and it's exactly true in symmetric cases where your starting belief is 50-50 and the evidence is equally likely to point in either direction. To see why it is wrong in general, consider an extreme case: a universal law, which you mostly already believe to be true. At any time, you could see a counterexample, which would make you jump to complete disbelief. That's a small probability of a very large update downwards. Conservation of expected evidence implies that you must move your belief upwards when you don't see such a counterexample. But, you consider that case to be quite likely. So, considering only which direction your beliefs will change, you can be fairly confident that your belief in the universal law will increase -- in fact, as confident as you are in the universal law itself. The critical point here is direction vs magnitude. Conservation of expected evidence takes magnitude as well as direction into account. The small but very probable increase is balanced by the large but very improbable decrease. The fact that we're talking about universal laws and counterexamples may fool you into thinking about logical uncertainty. You can think about logical uncertainty if you want, but this phenomenon is present in the fully classical Bayesian setting; there's no funny business with non-Bayesian updates here. Epistemic status: confidence at the level of mathematical reasoning. 2. "Yes requires the possibility of no." Scott's recent post, yes requires the possibility of no, is fine. I'm referring to a possible mistake which one could make in applying the principle illustrated there. “Those who dream do not know they dream, but when you are awake, you know you are awake.” -- Eliezer, Against Modest Epistemology Sometimes, look around, and ask myself whether I'm in a dream. When this happens, I generally conclude very confidently that I'm awake. I am not similarly capable of determining that I'm dreaming. My dreaming self doesn't have the self-awareness to question whether he is dreaming in this way. (Actually, very occasionally, I do. I either end up forcing myself awake, or I become lucid in the dream. Let's ignore that possibility for the purpose of the thought experiment.) I am not claiming that my dreaming self is never deluded into thinking he is awake. On the contrary, I have those repeatedly-waking-up-only-to-find-I'm-still-dreaming dreams occasionally. In those cases, I vividly believe myself to be awake. So, it's definitely possible for me to ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Jean Monnet: The Guerilla Bureaucrat, published Martin Sustrik on the LessWrong. I have written about coordination problems from various points of view in the past (biology, economics, sociology, political science) but this time I am about to focus not on the theory, but on the practice. Jean Monnet was one of the founding fathers of the European Union. One may even say that he was the architect of the European Union. However, as founding fathers go, he was rather unusual. His background was unusual: He was neither a political leader, nor a lawyer, a philosopher or a military commander. He was a son of a brandy merchant from the small town of Cognac near Bordeaux and himself a merchant by trade. He dropped out of school at sixteen and never got any extensive formal education. But also his approach was unusual: He never held an elected position, he has never put himself to the forefront, he almost never made big speeches and is not known for memorable quotations. Rather, he was always in the background, busy with the boring technical work, hanging around politicians, showing them his famous balance sheets and trying to convince them to do the sensible, if unexpected, thing. He was, in fact, so undistinguished that, when Fortune magazine run a story about him, they have given up on inventing a proper title for him and introduced him simply as "Monsieur Jean Monnet of Cognac". But whoever he was in his life - a trader, a banker, a civil servant - the only description that truly fits is that he was a solver of coordination problems. The Monnet Method This article will explore what Mario Draghi (former president of European Central Bank, and now, quite unexpectedly, the Italian prime minister) calls "the Monnet method", a bunch of principles that guided the effort to unite the continent divided by centuries of incessant wars and feuds. But while Draghi is focusing on the lessons that may be relevant in the current state of the European Union, my interest is a bit broader: How does one solve coordination problems in general? And how does to do it as successfully as Jean Monnet once did? In this article we are going to examine that question. Yet, before we begin, a warning is due. Monnet himself, in his memoirs, refuses to write down his method: I might have written a series of practical maxims; but I distrust general ideas, and I never let them lead me far away from practical things. I have described the dramatic events I have lived through and the lessons I have learned from them, in the hope of preventing their happening again. My purpose is very practical. Some may call it a philosophy, if they prefer: but the essential point is to make it useful beyond the experience of one individual. And that, I think, is not Monnet being modest. It's the core of his approach. The only way to break out of inadequate equilibria, to solve the coordination problems, is to take advantage of the unexpected. Everything that is expected, after all, just feeds into the equilibrium and makes it persist. And to take advantage of the unexpected, one should not bind himself to a specific, predictable method. Bypassing the Hierarchy One recurring theme in Jean Monnet's life was working outside of the existing institutions. The common sense would have it that to change how Europe works, he should have found a humble job at French ministry of foreign affairs and work his way up the hierarchy until he had enough say to push his ideas forward. Instead, it's 1914, the beginning of the Great War. Monnet is 26 years old and has no prior political experience: One of our friends at Cognac was a lawyer, Maitre Fernand Benon, who happened to know René Viviani, the Prime Minister, quite well [...] and he agreed to introduce me to [him]. Viviani said to me: "Sir, I gather that you have some interesting propos...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The topic is not the content, published by aaronb50 on the LessWrong. This is a linkpost for Disclaimer: I’ve never held a job for more than a year [1] or been paid more than $15 an hour. Take everything I say with a grain of salt. Many of my peers seem to make career plans like by asking things like What am I interested in, or what do I like doing? and How can I do something related to that? which might lead to some of the following: A person who likes dancing tries to work in the arts industry. A person who likes video games tries to get into game design. A person who is interested in healthcare policy tries to study or design healthcare policy. The problem here, in terms of diminished performance, happiness, and satisfaction, is a conflation of the topic and the content. The topic is not the content! The topic In my schema, the topic is what the work is about. If you’re the manager of a pillow company, the topic is pillows. If you’re defending accused criminals in court, the topic is criminal law. A lot of folks, it seems to me, focus a lot on the topic when deciding which subjects to study or which jobs to apply for. Someone who is interested in physics might major in physics. Someone who loves to work out might try to become a personal trainer. I don’t think this makes much sense. Should people ignore what they’re interested in and like to do, then? Well, maybe. 80,000 Hours, perhaps the single best career planning resource out there, writes that The bottom line To find a dream job, look for: Work you’re good at, Work that helps others, Supportive conditions: engaging work that lets you enter a state of flow; supportive colleagues; lack of major negatives like unfair pay; and work that fits your personal life. Ok, but the term “engaging work” is doing a lot of work here (no pun intended), and seems awfully synonymous with “work that you like.” So, how do you find work that you like doing? If there’s anything my utterly negligible work experiences has taught me, it’s that it usually makes more sense to focus less on the topic and more on the content. The content In my schema, the content is what the work involves doing. If you’re a physics teacher, the topic is physics, but the content is (I assume) some combination of grading papers, making slideshow presentations, lecturing, doing demonstrations, and answering student questions. Say Emma is a physics teacher. Which do you think matters more for Emma’s personal career satisfaction: being interested in physics (the topic), or enjoying grading, lecturing, presenting, and answering questions (the content)? Almost certainly, I think, the latter. Don’t get me wrong, the topic matters too! Even if Emma likes all of these activities, I have no doubt that both she and her students would be better off if Emma were interested in physics. But someone who loves teaching but is indifferent to physics will be better off than someone who loves physics but is indifferent to teaching. The problem One of the fundamental issues here is that it’s way easier to discern the topic. For instance, the word “physics” in “physics teacher” is served up on a salient silver platter. Obviously, “teaching physics” involves physics. What is less obvious, though, is what “teaching” involves. The verb “teach” isn’t very descriptive—it’s just a placeholding bucket for more substantive actions like “grade papers” and “lecture.” In fact, while I’m virtually certain that teaching physics involves physics, I recognize that my description of the content (grading, lecturing etc.) could be misleading or missing something important. In fact, I couldn’t tell you what my high school teachers spent the plurality of their time actually doing. Me, right now As my LinkedIn will tell you, I am a newly-minted federal employee and proud member of the economics tea...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 37 Ways That Words Can Be Wrong, published by on the LessWrong. Some reader is bound to declare that a better title for this post would be "37 Ways That You Can Use Words Unwisely", or "37 Ways That Suboptimal Use Of Categories Can Have Negative Side Effects On Your Cognition". But one of the primary lessons of this gigantic list is that saying "There's no way my choice of X can be 'wrong'" is nearly always an error in practice, whatever the theory. You can always be wrong. Even when it's theoretically impossible to be wrong, you can still be wrong. There is never a Get-Out-Of-Jail-Free card for anything you do. That's life. Besides, I can define the word "wrong" to mean anything I like - it's not like a word can be wrong. Personally, I think it quite justified to use the word "wrong" when: A word fails to connect to reality in the first place. Is Socrates a framster? Yes or no? (The Parable of the Dagger.) Your argument, if it worked, could coerce reality to go a different way by choosing a different word definition. Socrates is a human, and humans, by definition, are mortal. So if you defined humans to not be mortal, would Socrates live forever? (The Parable of Hemlock.) You try to establish any sort of empirical proposition as being true "by definition". Socrates is a human, and humans, by definition, are mortal. So is it a logical truth if we empirically predict that Socrates should keel over if he drinks hemlock? It seems like there are logically possible, non-self-contradictory worlds where Socrates doesn't keel over - where he's immune to hemlock by a quirk of biochemistry, say. Logical truths are true in all possible worlds, and so never tell you which possible world you live in - and anything you can establish "by definition" is a logical truth. (The Parable of Hemlock.) You unconsciously slap the conventional label on something, without actually using the verbal definition you just gave. You know perfectly well that Bob is "human", even though, on your definition, you can never call Bob "human" without first observing him to be mortal. (The Parable of Hemlock.) The act of labeling something with a word, disguises a challengable inductive inference you are making. If the last 11 egg-shaped objects drawn have been blue, and the last 8 cubes drawn have been red, it is a matter of induction to say this rule will hold in the future. But if you call the blue eggs "bleggs" and the red cubes "rubes", you may reach into the barrel, feel an egg shape, and think "Oh, a blegg." (Words as Hidden Inferences.) You try to define a word using words, in turn defined with ever-more-abstract words, without being able to point to an example. "What is red?" "Red is a color." "What's a color?" "It's a property of a thing?" "What's a thing? What's a property?" It never occurs to you to point to a stop sign and an apple. (Extensions and Intensions.) The extension doesn't match the intension. We aren't consciously aware of our identification of a red light in the sky as "Mars", which will probably happen regardless of your attempt to define "Mars" as "The God of War". (Extensions and Intensions.) Your verbal definition doesn't capture more than a tiny fraction of the category's shared characteristics, but you try to reason as if it does. When the philosophers of Plato's Academy claimed that the best definition of a human was a "featherless biped", Diogenes the Cynic is said to have exhibited a plucked chicken and declared "Here is Plato's Man." The Platonists promptly changed their definition to "a featherless biped with broad nails". (Similarity Clusters.) You try to treat category membership as all-or-nothing, ignoring the existence of more and less typical subclusters. Ducks and penguins are less typical birds than robins and pigeons. Interestingly, a between-groups expe...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Are wireheads happy?, published by Scott Alexander on the LessWrong. Related to: Utilons vs. Hedons, Would Your Real Preferences Please Stand Up And I don't mean that question in the semantic "but what is happiness?" sense, or in the deep philosophical "but can anyone not facing struggle and adversity truly be happy?" sense. I mean it in the totally literal sense. Are wireheads having fun? They look like they are. People and animals connected to wireheading devices get upset when the wireheading is taken away and will do anything to get it back. And it's electricity shot directly into the reward center of the brain. What's not to like? Only now neuroscientists are starting to recognize a difference between "reward" and "pleasure", or call it "wanting" and "liking". The two are usually closely correlated. You want something, you get it, then you feel happy. The simple principle behind our entire consumer culture. But do neuroscience and our own experience really support that? It would be too easy to point out times when people want things, get them, and then later realize they weren't so great. That could be a simple case of misunderstanding the object's true utility. What about wanting something, getting it, realizing it's not so great, and then wanting it just as much the next day? Or what about not wanting something, getting it, realizing it makes you very happy, and then continuing not to want it? The first category, "things you do even though you don't like them very much" sounds like many drug addictions. Smokers may enjoy smoking, and they may want to avoid the physiological signs of withdrawl, but neither of those is enough to explain their reluctance to quit smoking. I don't smoke, but I made the mistake of starting a can of Pringles yesterday. If you asked me my favorite food, there are dozens of things I would say before "Pringles". Right now, and for the vast majority of my life, I feel no desire to go and get Pringles. But once I've had that first chip, my motivation for a second chip goes through the roof, without my subjective assessment of how tasty Pringles are changing one bit. Think of the second category as "things you procrastinate even though you like them." I used to think procrastination applied only to things you disliked but did anyway. Then I tried to write a novel. I loved writing. Every second I was writing, I was thinking "This is so much fun". And I never got past the second chapter, because I just couldn't motivate myself to sit down and start writing. Other things in this category for me: going on long walks, doing yoga, reading fiction. I can know with near certainty that I will be happier doing X than Y, and still go and do Y. Neuroscience provides some basis for this. A University of Michigan study analyzed the brains of rats eating a favorite food. They found separate circuits for "wanting" and "liking", and were able to knock out either circuit without affecting the other (it was actually kind of cute - they measured the number of times the rats licked their lips as a proxy for "liking", though of course they had a highly technical rationale behind it). When they knocked out the "liking" system, the rats would eat exactly as much of the food without making any of the satisifed lip-licking expression, and areas of the brain thought to be correlated with pleasure wouldn't show up in the MRI. Knock out "wanting", and the rats seem to enjoy the food as much when they get it but not be especially motivated to seek it out. To quote the science1: Pleasure and desire circuitry have intimately connected but distinguishable neural substrates. Some investigators believe that the role of the mesolimbic dopamine system is not primarily to encode pleasure, but "wanting" i.e. incentive-motivation. On this analysis, endomorphins and enkephalins -...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [moderator action] Eugine_Nier is now banned for mass downvote harassment, published by Kaj_Sotala on the LessWrong. As previously discussed, on June 6th I received a message from jackk, a Trike Admin. He reported that the user Jiro had asked Trike to carry out an investigation to the retributive downvoting that Jiro had been subjected to. The investigation revealed that the user Eugine_Nier had downvoted over half of Jiro's comments, amounting to hundreds of downvotes. I asked the community's guidance on dealing with the issue, and while the matter was being discussed, I also reviewed previous discussions about mass downvoting and looked for other people who mentioned being the victims of it. I asked Jack to compile reports on several other users who mentioned having been mass-downvoted, and it turned out that Eugine was also overwhelmingly the biggest downvoter of users David_Gerard, daenarys, falenas108, ialdabaoth, shminux, and Tenoke. As this discussion was going on, it turned out that user Ander had also been targeted by Eugine. I sent two messages to Eugine, requesting an explanation. I received a response today. Eugine admitted his guilt, expressing the opinion that LW's karma system was failing to carry out its purpose of keeping out weak material and that he was engaged in a "weeding" of users who he did not think displayed sufficient rationality. Needless to say, it is not the place of individual users to unilaterally decide that someone else should be "weeded" out of the community. The Less Wrong content deletion policy contains this clause: Harrassment of individual users. If we determine that you're e.g. following a particular user around and leaving insulting comments to them, we reserve the right to delete those comments. (This has happened extremely rarely.) Although the wording does not explicitly mention downvoting, harassment by downvoting is still harassment. Several users have indicated that they have experienced considerable emotional anguish from the harassment, and have in some cases been discouraged from using Less Wrong at all. This is not a desirable state of affairs, to say the least. I was originally given my moderator powers on a rather ad-hoc basis, with someone awarding mod privileges to the ten users with the highest karma at the time. The original purpose for that appointment was just to delete spam. Nonetheless, since retributive downvoting has been a clear problem for the community, I asked the community for guidance on dealing with the issue. The rough consensus of the responses seemed to authorize me to deal with the problem as I deemed appropriate. The fact that Eugine remained quiet about his guilt until directly confronted with the evidence, despite several public discussions of the issue, is indicative of him realizing that he was breaking prevailing social norms. Eugine's actions have worsened the atmosphere of this site, and that atmosphere will remain troubled for as long as he is allowed to remain here. Therefore, I now announce that Eugine_Nier is permanently banned from posting on LessWrong. This decision is final and will not be changed in response to possible follow-up objections. Unfortunately, it looks like while a ban prevents posting, it does not actually block a user from casting votes. I have asked jackk to look into the matter and find a way to actually stop the downvoting. Jack indicated earlier on that it would be technically straightforward to apply a negative karma modifier to Eugine's account, and wiping out Eugine's karma balance would prevent him from casting future downvotes. Whatever the easiest solution is, it will be applied as soon as possible. EDIT 24 July 2014: Banned users are now prohibited from voting. Thanks for listening. to help us out with the nonlinear library or to learn more, please vis...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Chris Olah’s views on AGI safety, published by evhub on the LessWrong. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. Note: I am not Chris Olah. This post was the result of lots of back-and-forth with Chris, but everything here is my interpretation of what Chris believes, not necessarily what he actually believes. Chris also wanted me to emphasize that his thinking is informed by all of his colleagues on the OpenAI Clarity team and at other organizations. In thinking about AGI safety—and really any complex topic on which many smart people disagree—I’ve often found it very useful to build a collection of different viewpoints from people that I respect that I feel like I understand well enough to be able to think from their perspective. For example, I will often try to compare what an idea feels like when I put on my Paul Christiano hat to what it feels like when I put on my Scott Garrabrant hat. Recently, I feel like I’ve gained a new hat that I’ve found extremely valuable that I also don’t think many other people in this community have, which is my Chris Olah hat. The goal of this post is to try to give that hat to more people. If you’re not familiar with him, Chris Olah leads the Clarity team at OpenAI and formerly used to work at Google Brain. Chris has been a part of many of the most exciting ML interpretability results in the last five years, including Activation Atlases, Building Blocks of Interpretability, Feature Visualization, and DeepDream. Chris was also a coauthor of “Concrete Problems in AI Safety.” He also thinks a lot about technical AGI safety and has a lot of thoughts on how ML interpretability work can play into that—thoughts which, unfortunately, haven’t really been recorded previously. So: here’s my take on Chris’s AGI safety worldview. The benefits of transparency and interpretability Since Chris primarily works on ML transparency and interpretability, the obvious first question to ask is how he imagines that sort of research aiding with AGI safety. When I was talking with him, Chris listed four distinct ways in which he thought transparency and interpretability could help, which I’ll go over in his order of importance. Catching problems with auditing First, Chris says, interpretability gives you a mulligan. Before you deploy your AI, you can throw all of your interpretability tools at it to check and see what it actually learned and make sure it learned the right thing. If it didn’t—if you find that it’s learned some sort of potentially dangerous proxy, for example—then you can throw your AI out and try again. As long as you’re in a domain where your AI isn’t actively trying to deceive your interpretability tools (via deceptive alignment, perhaps), this sort of a mulligan could help quite a lot in resolving more standard robustness problems (proxy alignment, for example). That being said, that doesn’t necessarily mean waiting until you’re on the verge of deployment to look for flaws. Ideally you’d be able to discover problems early on via an ongoing auditing process as you build more and more capable systems. One of the OpenAI Clarity team’s major research thrusts right now is developing the ability to more rigorously and systematically audit neural networks. The idea is that interpretability techniques shouldn’t have to “get lucky” to stumble across a problem, but should instead reliably catch any problematic behavior. In particular, one way in which they’ve been evaluating progress on this is the “auditing game.” In the auditing game, one researcher takes a neural network and makes some modification to it—maybe images containing both dogs and cats are now classified as rifles, for example—and another researcher, given only the modified network, has to diagnose the problem and figure out exactly what modif...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Personal (Interim) COVID-19 Postmortem, published by Davidmanheim on the LessWrong. I think it's important to clearly and publicly admit when we were wrong. It's even better to diagnose why, and take steps to prevent doing so again. COVID-19 is far from over, but given my early stance on a number of questions regarding COVID-19, this is my attempt at a public personal review to see where I was wrong. I have been pushing for better forecasting and preparation for pandemics for years, but I wasn't forecasting on the various specific questions about Pandemics on most platforms until at least mid-March, and I failed in several ways. Mea Culpa I was late to update about a number of things, and simply wrong in some cases even on the basis of known information. The failures include initially being slow to recognize the extent of the threat, starting out dismissive about masks, being more concerned about hospital-based transmission than ended up being justified, being overconfident in the response of the US government, and in early March, over-confidently getting a key fact wrong about transmission being at least largely via aerosol droplet versus physical contact. I have a number of excuses, of course. Most other experts agreed with my views, my grandfather passed away in January, followed by his wife in early March, I was under a lot of stress, I was very busy with my personal life, I was trying to do a number of other high-priority projects, I was not paying attention to the details, and so on. But predictive accuracy doesn't care about WHY you were wrong, especially since there are always such excuses. And the impact of my poor judgement was also likely misleading to others in the community. At the same time, I feel the perhaps egotistical need to note where I was correct early, and what I got right - followed by a clearer description of my failures. I started saying there would be PPE shortages due to COVID-19 by January, and was writing about the supply chain issues well before COVID. I submitted this paper November last year with Dave Denkenberger, which was largely finished last summer, and it was accepted in February, which then took 3 months to get published. The delay was in part due to other demands on my time, but in retrospect, if it had been available 3 months earlier, it would have been far, far more impactful. I also understood the failure mode we ended up seeing, and in my 2018 paper, discussing overconfidence in claims that pandemics would be rare, I argued that among the most critical risks was failure to respond to emerging pandemics which could in theory be controlled quickly enough. On the other hand, my failure to realize that this is exactly what was happening is perhaps compounded by the fact that I understood the dynamics, and should have been able to identify what was going on. Lastly, I maintain I was correct in warning about the poorly thought out and in some cases outright dangerous "preparation" in some quarters of the rationality community proposed in March, such as advocating use of bleach and ozone in closed areas for disinfection. Some people in the community were stockpiling N-95 masks and food and buying up second hand ventilators, and as I said at the time, were at best being selfish and defecting. On the other hand, as I mention below, I was insufficiently clear about the need for better preparation, and waited far too long to speak. Some of My Mistakes, and Related Comments Slow to recognize the extent of the threat. I said we should be very concerned in January, albeit not very publicly. I took until early March to start suggesting that it was clear that the US would expect to see large numbers of deaths. I was skeptical of valuable efforts early on, and didn't start really publicly sounding the alarm and reacting until even later....
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Draft report on AI timelines, published by Ajeya Cotra on the LessWrong. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. Hi all, I've been working on some AI forecasting research and have prepared a draft report on timelines to transformative AI. I would love feedback from this community, so I've made the report viewable in a Google Drive folder here. With that said, most of my focus so far has been on the high-level structure of the framework, so the particular quantitative estimates are very much in flux and many input parameters aren't pinned down well -- I wrote the bulk of this report before July and have received feedback since then that I haven't fully incorporated yet. I'd prefer if people didn't share it widely in a low-bandwidth way (e.g., just posting key graphics on Facebook or Twitter) since the conclusions don't reflect Open Phil's "institutional view" yet, and there may well be some errors in the report. The report includes a quantitative model written in Python. Ought has worked with me to integrate their forecasting platform Elicit into the model so that you can see other people's forecasts for various parameters. If you have questions or feedback about the Elicit integration, feel free to reach out to elicit@ought.org. Looking forward to hearing people's thoughts! Thanks for listening. to help us out with the nonlinear library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Be Happier, published by [anonymous] on the LessWrong. This started as an assignment to find out about the science of ‘buying happiness’ (using money to become happier) — hence the emphasis on money-and-happiness. I learned a great deal more than how to buy happiness, however, and the project became somewhat more generalized. It is not meant to be comprehensive, but perhaps it makes for a useful supplement to Luke’s How to be Happy. This post consists mostly of quoted material. In A Nutshell Money and Happiness Spend on others, especially people you are close to. Positive feedback loop: Prosocial spending makes you happier, and happiness makes you more likely to spend prosocially. Don’t be stingy. It's bad for your health. Don’t think too much about money. It will impair your savoring ability. It's also bad for your family life. Be time-aware, but don’t think of time in terms of money. Being richer will not necessarily make you happier. Do not live in wealthy enclaves. Avoid conspicuous consumption. Work Satisfaction Coping with Stress: React pragmatically rather than emotionally. Go for ‘approach’ goals instead of ‘avoid’ goals. Autonomy: Make a point of prefering autonomous goals rather than heteronomous goals (goals imposed by others). Autonomy: Make sure you have spare discretionary time — even at financial cost. Be passionate, but don’t obsess. Do work that you enjoy doing. Flow. Set goals that are reasonably challenging and reasonably achievable. Materialism and Purchasing Prefer experiential purchases; avoid materialistic goals. Keep your goals intrinsic. Don’t do ‘comparison shopping.’ And don’t place much stock in the happiness potential of any one positive change. Follow the herd. “The best way to predict how much we will enjoy an experience is to see how much someone else enjoyed it.” Interpersonal Socialize with close others. Associate with happy people. Give the people around you opportunities to be generous. Ask them for favors. Be actively kind (and occasionaly reminisce about your recent acts of kindness). Stretching Happiness (fighting hedonic adaptation) Go for smaller, more frequent successes rather than larger ones. Go for variety and surprise. Don’t keep doing the same thing. Savor anticipation. Delay consumption. Actively anticipate good experiences. Divide positive experiences into smaller pleasures, if possible. Corollary: Conclude negative experiences as soon as possible. Make a point of avoiding experiences that make you feel bad. Appreciation Be grateful and count your blessings (literally). Recycle happiness by reminiscing about good experiences. Think of counterfactuals. (“If I didn’t have this positive thing, what do I lose?”) Breathe deeply. Expand your time — by slowing down. Stay in the present. Optimal Happification Actively want to be happier. Motivation and investment matter. Learn about the science of happiness. Internalize the lessons in this article and in here. Some Key Terms Subjective Well Being (SWB) aka happiness. Hedonic Adaptation — the phenomenon of (rapidly) diminishing positive or negative affect from any one experience or thing. Hedonic treadmill — the phenomenon of neverending aspirations for materialistic acquisitions that results from hedonic adaptation. Money and Happiness Spend on others, especially people you are close to. Past research in our lab has repeatedly shown that people are happier when they use financial resources to benefit others rather than themselves [Aknin, Dunn, Sandstrom & Norton, submitted, 1,14]. ... findings suggest that to reap the greatest emotional reward from spending on someone else, one should direct their purchases to close others These findings should not be taken to suggest that people should avoid spending on weak social ties. Indeed, treating an acquaintance from yoga to a coffee ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Going Out With Dignity, published by Bjartur Tómas on the LessWrong. DF was born with a time bomb in his genome, a deadly curse more horrifying than most. The name of this curse was Fatal familial insomnia (FFI). Wikipedia describes the usual progression of this hell: The disease has four stages:[8] Characterized by worsening insomnia, resulting in panic attacks, paranoia, and phobias. This stage lasts for about four months. Hallucinations and panic attacks become noticeable, continuing for about five months. Complete inability to sleep is followed by rapid loss of weight. This lasts for about three months. Dementia, during which the person becomes unresponsive or mute over the course of six months, is the final stage of the disease, after which death follows. From the case report DF's psychologist wrote after his death: DF was a right-handed, 52-year-old, white, American man with a doctorate in naturopathy. DF's father, paternal uncle, and 2 male cousins were diagnosed with fatal familial insomnia (FFI). His father died at age 76; his uncle died at age 74; and each of DF's cousins died before the age of 50. Not only is there no cure for FFI; there is no known cure for any prion disease. On the day it became clear he was experiencing the same symptoms his relatives did, DF must have known his chances were terrible. And even the minuscule odds that he could find a solution were marred by the fact that his problem-solving organ was the very part of him that was beginning to degrade. And if there was a way out, how could he come up with a solution when he was so, so tired? If only he could get just a little bit of rest. There is a motivational technique I use occasionally where I look at my behavior and meditate on what my revealed preferences imply about my actual preferences. Often, I am disgusted. I then note that I am capable of changing my behavior. And that all that is required to change my revealed preferences is to change my behavior. Though there is an element of sophistry in this line of thinking, I can report some anecdotal success. Many of us here, like DF, believe we have a deadly curse - or at least we believe we believe we have a deadly curse. Since I read The Basic AI Drives, I have known abstractly the world is doomed. Though my system 1 seems to have difficultly comprehending this, this belief implies I, and everyone and everything I love, am doomed, too. Through the lens of my revealed preferences, I either do not truly think the alignment problem is much of a problem, am mostly indifferent to the destruction of myself and everything I care about, or I am choosing the ignoble path of the free-rider. I notice I am disgusted. But this is good news. All that is required to change my revealed preferences is to change my behavior. DF's first tools were those of his naturopathic trade. He reported some success with a vitamin cocktail of the standard throw-in-everything-but-the-kitchen-sink, alternative-medicine style. Perhaps something in his cocktail had some effect as his progression was slower than normal. But slow progression is progression just the same: By month 15 (early stage II), vitamins alone failed to induce sleep. Following 5 consecutive nights of insomnia, DF became intensely irritable and delusional. An evaluation at the Massachusetts General Hospital in Boston, Massachusetts, found that he had suffered a minor stroke; he was anesthetized until he fell asleep. While hospitalized, he slept for 3 consecutive days and was fully alert and refreshed afterward. Noticing the efficacy of the anesthetics, DF began to use them regularly: Ketamine and nitrous oxide induced short (15-minute) periods of restful sleep, and were reapplied to offer more prolonged relief. Chloral hydrate in a light alcohol mix and/or chloroform also worked. Approximately 15 m...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Best Software For Every Need, published by Bjartur Tómas on the LessWrong. When I first started programming, I didn't use a terminal multiplexer and finding tmux was a sort of revelation. I joked once on discord that "life before tmux was not life". It strikes me there are probably many other programs that I am not aware of that would be useful to know about. I've occasionally found Luke's The Best Textbooks on Every Subject thread useful, so I thought a similar thread about software may be interesting. Here are the rules: Post the name of a program for a given need. You must have tried at least 2 other programs designed for the same/similar class of problems. You must briefly name the other programs you have tried and why you think your chosen program is superior to them. Thanks for listening. to help us out with the nonlinear library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Study Guide, published by johnswentworth on the LessWrong. This post is for students who hope to eventually work on technical problems we don’t understand, especially agency and AI alignment, and want to know what to study or practice. Guiding Principles Current alignment researchers have wildly different recommendations on paths into the field, usually correlated with the wildly different paths these researchers have themselves taken into the field. This also correlates with different kinds of work on alignment. This guide largely reflects my own path, and I think it is useful if you want to do the sort of research I do. That means fairly theoretical work (for now), very technical, drawing on models and math from a lot of different areas to understand real-world agents. Specializing in Problems We Don’t Understand lays out a general framework which guides many of the recommendations here. I’ll also briefly go over some guiding principles more specific to choosing what (and how much) to study: Breadth over depth Practice generalizing concepts Be able to model anything High volume of knowledge Breadth Over Depth In general, study in any particular topic has decreasing marginal returns. The first exposure or two gives you the basic frames, tells you what kinds of questions to ask and what kinds of tools are available, etc. You may not remember everything, but you can at least remember what things to look up later if you need them - which is a pretty huge improvement over not even knowing that X is a thing you can look up at all! Another way to frame this: problems-we-don’t-understand rely heavily on bringing in frames and tools from other fields. (If the frames and tools of this field were already sufficient, it wouldn’t be a problem-we-don’t-understand in the first place.) So, you want to have a very large library of frames and tools to apply. On the other hand, you don’t necessarily need very much depth in each frame or tool - just enough to recognize problems where it might apply and maybe try it out in a quick-and-dirty way. Practice Generalizing Concepts Bringing in frames and tools from other fields requires the ability to recognize and adapt those frames and tools for problems very different from the field in which we first learned them. So, practice generalizing concepts from one area to another is particularly important. Unfortunately, this is not a focus in most courses. There are exceptions - applied math classes often involve applying tools in a wide variety of ways, and low-level physics courses often provide very good practice in applying a few mathematical tools to a wide variety of problems. Ultimately, though, this is something you should probably practice on your own a lot more than it’s practiced in class. Keeping a list of 10-20 hard problems in the back of your mind, and trying out each new frame or tool on one of those problems, is a particularly useful technique to practice generalization. Be Able To Model Anything One common pitfall is to be drawn into areas which advertise extreme generality, but are rarely useful in practice. (A lot of high-level math is like this.) On the other hand, we still want a lot of breadth, including things which are not obviously useful to whatever problem we’re most interested in (e.g. alignment). After all, if the obviously-relevant tools sufficed, then it wouldn’t be a problem-we-don’t-understand in the first place. To that end, it’s useful to look for frames/tools which are at least useful for something. Keeping a list of 10-20 hard problems in the back of your mind is one useful test for this. Another useful heuristic is “be able to model anything”: if there’s some system or phenomenon which you’re not sure how to model, even in principle, and field X has good tools for modelling it, then study field X. This heu...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Conservation of Expected Evidence, published by Eliezer Yudkowsky on the LessWrong. Friedrich Spee von Langenfeld, a priest who heard the confessions of condemned witches, wrote in 1631 the Cautio Criminalis (“prudence in criminal cases”), in which he bitingly described the decision tree for condemning accused witches: If the witch had led an evil and improper life, she was guilty; if she had led a good and proper life, this too was a proof, for witches dissemble and try to appear especially virtuous. After the woman was put in prison: if she was afraid, this proved her guilt; if she was not afraid, this proved her guilt, for witches characteristically pretend innocence and wear a bold front. Or on hearing of a denunciation of witchcraft against her, she might seek flight or remain; if she ran, that proved her guilt; if she remained, the devil had detained her so she could not get away. Spee acted as confessor to many witches; he was thus in a position to observe every branch of the accusation tree, that no matter what the accused witch said or did, it was held as proof against her. In any individual case, you would only hear one branch of the dilemma. It is for this reason that scientists write down their experimental predictions in advance. But you can’t have it both ways —as a matter of probability theory, not mere fairness. The rule that “absence of evidence is evidence of absence” is a special case of a more general law, which I would name Conservation of Expected Evidence: the expectation of the posterior probability, after viewing the evidence, must equal the prior probability. Therefore, for every expectation of evidence, there is an equal and opposite expectation of counterevidence. If you expect a strong probability of seeing weak evidence in one direction, it must be balanced by a weak expectation of seeing strong evidence in the other direction. If you’re very confident in your theory, and therefore anticipate seeing an outcome that matches your hypothesis, this can only provide a very small increment to your belief (it is already close to 1); but the unexpected failure of your prediction would (and must) deal your confidence a huge blow. On average, you must expect to be exactly as confident as when you started out. Equivalently, the mere expectation of encountering evidence—before you’ve actually seen it—should not shift your prior beliefs. So if you claim that “no sabotage” is evidence for the existence of a Japanese-American Fifth Column, you must conversely hold that seeing sabotage would argue against a Fifth Column. If you claim that “a good and proper life” is evidence that a woman is a witch, then an evil and improper life must be evidence that she is not a witch. If you argue that God, to test humanity’s faith, refuses to reveal His existence, then the miracles described in the Bible must argue against the existence of God. Doesn’t quite sound right, does it? Pay attention to that feeling of this seems a little forced, that quiet strain in the back of your mind. It’s important. For a true Bayesian, it is impossible to seek evidence that confirms a theory. There is no possible plan you can devise, no clever strategy, no cunning device, by which you can legitimately expect your confidence in a fixed proposition to be higher (on average) than before. You can only ever seek evidence to test a theory, not to confirm it. This realization can take quite a load off your mind. You need not worry about how to interpret every possible experimental result to confirm your theory. You needn’t bother planning how to make any given iota of evidence confirm your theory, because you know that for every expectation of evidence, there is an equal and oppositive expectation of counterevidence. If you try to weaken the counterevidence of a possible “abnormal” observa...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some cruxes on impactful alternatives to AI policy work, published by Richard_Ngo on the LessWrong. Ben Pace and I (Richard Ngo) recently did a public double crux at the Berkeley REACH on how valuable it is for people to go into AI policy and strategy work: I was optimistic and Ben was pessimistic. During the actual event, we didn't come anywhere near to finding a double crux on that issue. But after a lot of subsequent discussion, we've come up with some more general cruxes about where impact comes from. I found Ben's model of how to have impact very interesting, and so in this post I've tried to explain it, along with my disagreements. Ben liked the goal of writing up a rough summary of our positions and having further discussion in the comments, so while he edited it somewhat he doesn’t at all think that it’s a perfect argument, and it’s not what he’d write if he spent 10 hours on it. He endorsed the wording of the cruxes as broadly accurate. (During the double crux, we also discussed how the heavy-tailed worldview applies to community building, but decided on this post to focus on the object level of what impact looks like.) Note from Ben: “I am not an expert in policy, and have not put more than about 20-30 hours of thought into it total as a career path. But, as I recently heard Robin Hanson say, there’s a common situation that looks like this: some people have a shiny idea that they think about a great deal and work through the details of, that folks in other areas are skeptical of given their particular models of how the world works. Even though the skeptics have less detail, it can be useful to publicly say precisely why they’re skeptical. In this case I’m often skeptical when folks tell me they’re working to reduce x-risk by focusing on policy. Folks doing policy work in AI might be right, and I might be wrong, but it seemed like a good use of time to start a discussion with Richard about how I was thinking about it and what would change my mind. If the following discussion causes me to change my mind on this question, I’ll be really super happy with it.” Ben's model: Life in a heavy-tailed world A heavy-tailed distribution is one where the probability of extreme outcomes doesn’t drop very rapidly, meaning that outliers therefore dominate the expectation of the distribution. Owen Cotton-Barratt has written a brief explanation of the idea here. Examples of heavy-tailed distributions include the Pareto distribution and the log-normal distribution; other phrases people use to point at this concept include ‘power laws’ (see Zero to One) and ‘black swans’ (see the recent SSC book review). Wealth is a heavy-tailed distribution, because many people are clustered relatively near the median, but the wealthiest people are millions of times further away. Human height and weight and running speed are not heavy-tailed; there is no man as tall as 100 people. There are three key claims that make up Ben's view. The first claim is that, since the industrial revolution, we live in a world where the impact that small groups can have is much more heavy-tailed than in the past. People can affect incredibly large numbers of other people worldwide. The Internet is an example of a revolutionary development which allows this to happen very quickly. Startups are becoming unicorns unprecedentedly quickly, and their valuations are very heavily skewed. The impact of global health interventions is heavy-tail distributed. So is funding raised by Effective Altruism - two donors have contributed more money than everyone else combined. Google and Wikipedia qualitatively changed how people access knowledge; people don't need to argue about verifiable facts any more. Facebook qualitatively changed how people interact with each other (e.g. FB events is a crucial tool for most local EA groups),...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Is Clickbait Destroying Our General Intelligence?, published by Eliezer Yudkowsky on the LessWrong (Cross-posted from Facebook.) Now and then people have asked me if I think that other people should also avoid high school or college if they want to develop new ideas. This always felt to me like a wrong way to look at the question, but I didn't know a right one. Recently I thought of a scary new viewpoint on that subject. This started with a conversation with Arthur where he mentioned an idea by Yoshua Bengio about the software for general intelligence having been developed memetically. I remarked that I didn't think duplicating this culturally transmitted software would be a significant part of the problem for AGI development. (Roughly: low-fidelity software tends to be algorithmically shallow. Further discussion moved to comment below.) But this conversation did get me thinking about the topic of culturally transmitted software that contributes to human general intelligence. That software can be an important gear even if it's an algorithmically shallow part of the overall machinery. Removing a few simple gears that are 2% of a machine's mass can reduce the machine's performance by way more than 2%. Feral children would be the case in point. A scary question is whether it's possible to do subtler damage to the culturally transmitted software of general intelligence. I've had the sense before that the Internet is turning our society stupider and meaner. My primary hypothesis is "The Internet is selecting harder on a larger population of ideas, and sanity falls off the selective frontier once you select hard enough." To review, there's a general idea that strong (social) selection on a characteristic imperfectly correlated with some other metric of goodness can be bad for that metric, where weak (social) selection on that characteristic was good. If you press scientists a little for publishable work, they might do science that's of greater interest to others. If you select very harshly on publication records, the academics spend all their time worrying about publishing and real science falls by the wayside. On my feed yesterday was an essay complaining about how the intense competition to get into Harvard is producing a monoculture of students who've lined up every single standard accomplishment and how these students don't know anything else they want to do with their lives. Gentle, soft competition on a few accomplishments might select genuinely stronger students; hypercompetition for the appearance of strength produces weakness, or just emptiness. A hypothesis I find plausible is that the Internet, and maybe television before it, selected much more harshly from a much wider field of memes; and also allowed tailoring content more narrowly to narrower audiences. The Internet is making it possible for ideas that are optimized to appeal hedonically-virally within a filter bubble to outcompete ideas that have been even slightly optimized for anything else. We're looking at a collapse of reference to expertise because deferring to expertise costs a couple of hedons compared to being told that all your intuitions are perfectly right, and at the harsh selective frontier there's no room for that. We're looking at a collapse of interaction between bubbles because there used to be just a few newspapers serving all the bubbles; and now that the bubbles have separated there's little incentive to show people how to be fair in their judgment of ideas for other bubbles, it's not the most appealing Tumblr content. Print magazines in the 1950s were hardly perfect, but they could get away with sometimes presenting complicated issues as complicated, because there weren't a hundred blogs saying otherwise and stealing their clicks. Or at least, that's the hypothesis. It seems plausible t...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Seeing the Smoke, published by Jacob Falkovich on the LessWrong. Cross-posted from Putanumonit. COVID-19 could be pretty bad for you. It could affect your travel plans as countries impose quarantines and close off borders. It could affect you materially as supply chains are disrupted and stock markets are falling. Even worse: you could get sick and suffer acute respiratory symptoms. Worse than that: someone you care about may die, likely an elderly relative. But the worst thing that could happen is that you’re seen doing something about the coronavirus before you’re given permission to. I’ll defend this statement in a minute, but first of all: I am now giving you permission to do something about COVID-19. You have permission to read up on the symptoms of the disease and how it spreads. Educate yourself on the best ways to avoid it. Stock up on obvious essentials such as food, water, soap, and medicine, as well as less obvious things like oxygen saturation monitors so you know if you need emergency care once you’re sick. You should decide ahead of time what your triggers are for changing your routines or turtling up at home. In fact, you should go do all those things before reading the rest of the post. I am not going to provide any more factual justifications for preparing. If you’ve been following the news and doing the research, you can decide for yourself. And if instead of factual justifications you’ve been following the cues of people around you to decide when it’s socially acceptable to prep for a pandemic, then all you need to know is that I’ve already put my reputation on the line as a coronaprepper. Instead this post is about the strange fact that most people need social approval to prepare for a widely-reported pandemic. Smoke Signals As Eliezer reminded us, most people sitting alone in a room will quickly get out if it starts filling up with smoke. But if two other people in the room seem unperturbed, almost everyone will stay put. That is the result of a famous experiment from the 1960s and its replications — people will sit and nervously look around at their peers for 20 minutes even as thick smoke starts obscuring their vision. The coronavirus was identified on January 7th and spread outside China by the 13th. American media ran some stories about how you should worry more about the seasonal flu. The markets didn’t budge. Rationalist Twitter started tweeting excitedly about R0 and supply chains. Over the next two weeks, Chinese COVID cases kept climbing at 60%/day reaching 17,000 by February 2nd. Cases were confirmed in Europe and the US. The WHO declared a global emergency. The former FDA commissioner explained why a law technicality made it illegal for US hospitals to test people for coronavirus, implying that we would have no idea how many Americans have contracted the disease. Everyone mostly ignored him including all major media publications, and equity markets hit an all time high. By this point several Rationalists in Silicon Valley and elsewhere started seriously prepping for a pandemic and canceling large social gatherings. On the 13th, Vox published a story mocking people in Silicon Valley for worrying about COVID-19. The article contained multiple factual mistakes about the virus and the opinions of public health experts. On February 17th, Eliezer asked how markets should react to an obvious looming pandemic. Most people agreed that the markets should freak out and aren’t. Most people decided to trust the markets over their own judgment. As an avowed efficient marketeer who hasn’t made an active stock trade in a decade, I started at that Tweet for a long time. I stared at it some more. Then I went ahead and sold 10% of the stocks I owned and started buying respirators and beans. By the 21st, the pandemic and its concomitant shortages hit ever...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Covid 1/7: The Fire of a Thousand Suns, published by Zvi on the LessWrong. I’ll summarize everything up front. Might as well get on with it. Let’s do the numbers. The Numbers Predictions Prediction last week: 14.3% positive rate on 9.7 million tests, and an average of 2,500 deaths, again with wide error bars. Results: 16.4% positive rate on 9.3 million tests, and an average of 2,657 deaths. The phase shift on 12/30, in the wake of Christmas, seems to have been real, giving us the clear holiday bump we did not see from previous holidays. That was both the very bad outcome for infections I was worried about, and also not high on my list of things to be concerned or furious about this week. For deaths, my estimate was lower than it should have been and I should have assumed a full reversion, so on reflection it’s my mistake rather than especially bad news. The new strain is not yet prevalent enough to be noticeably impacting the numbers. Prediction: 17.0% positive rate on 9.5 million tests, and an average of 2,800 deaths. The holidays are over, there will be some fallout, with things getting slightly worse, but with the main boost in deaths from Christmas mostly coming later. Deaths Date WEST MIDWEST SOUTH NORTHEAST Oct 29-Nov 4 956 1977 2309 613 Nov 5-Nov 11 1089 2712 2535 870 Nov 12-Nov 18 1255 2934 2818 1127 Nov 19-Nov 25 1761 4169 3396 1714 Nov 26-Dec 2 1628 3814 2742 1939 Dec 3-Dec 9 2437 5508 4286 2744 Dec 10-Dec 16 3278 5324 4376 3541 Dec 17-Dec 23 3826 5158 5131 3772 Dec 24-Dec 30 3363 3668 4171 3640 Dec 31-Jan 6 5320 5036 6072 4986 This is the one place it’s not as bad as it looks. The data source for these numbers is Wikipedia, which shows a relatively large amount of shifting of deaths from last week into this week. If we assume that a lot of this week’s deaths were actually last week, that explains much of the increase. It’s not good news or anything, it’s definitely bad news, but it is not full on terrifying like it would be if we didn’t know about Christmas. We still should expect further increases in the next few weeks before the tide likely temporarily turns. Positive Test Percentages Percentages Northeast Midwest South West 11/5 to 11/11 5.56% 17.51% 9.89% 8.31% 11/12 to 11/18 6.99% 18.90% 11.64% 10.66% 11/19 to 11/25 7.00% 16.62% 10.41% 11.75% 11/26 to 12/2 8.38% 17.90% 12.45% 12.79% 12/3 to 12/9 10.47% 17.94% 13.70% 12.76% 12/10 to 12/16 10.15% 15.63% 15.91% 13.65% 12/17 to 12/23 9.88% 14.65% 15.78% 13.82% 12/24 to 12/30 10.65% 14.54% 17.07% 12.90% 12/31 to 1/6 12.18% 17.03% 19.69% 15.94% No way to sugar coat this one. Everything is pointed in the wrong direction in a clear phase shift. The good news is that it could be a one-time move from Christmas, and thus we can still plausibly expect things to remain stable from this point forward without waiting for further control systems to kick in. I hope to start seeing slow improvements soon, but I must admit that the overall graph is not as encouraging of that as I thought a few weeks ago, especially outside of the Midwest, and the turning point may be farther off than we’d like. The Midwest is still probably past its peak for this wave, even if everyone else has to wait a bit. Positive Tests Date WEST MIDWEST SOUTH NORTHEAST Nov 12-Nov 18 211,222 452,265 255,637 150,724 Nov 19-Nov 25 269,230 435,688 294,230 170,595 Nov 26-Dec 2 256,629 357,102 294,734 185,087 Dec 3-Dec 9 354,397 379,823 368,596 263,886 Dec 10-Dec 16 415,220 315,304 406,353 260,863 Dec 17-Dec 23 439,493 271,825 419,230 236,264 Dec 24-Dec 30 372,095 206,671 373,086 225,476 Dec 31-Jan 6 428,407 251,443 494,090 267,350 Test Counts Date USA tests Positive % NY tests Positive % Cumulative Positives Nov 5-Nov 11 8,290,417 10.8% 1,059,559 2.4% 3.16% Nov 12-Nov 18 9,040,426 12.4% 1,155,670 2.9% 3.50% Nov 19-Nov 25 10,419,059 11.8% 1,373,751 2.9% ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: You Only Live Twice, published by Eliezer Yudkowsky on the LessWrong. "It just so happens that your friend here is only mostly dead. There's a big difference between mostly dead and all dead." -- The Princess Bride My co-blogger Robin and I may disagree on how fast an AI can improve itself, but we agree on an issue that seems much simpler to us than that: At the point where the current legal and medical system gives up on a patient, they aren't really dead. Robin has already said much of what needs saying, but a few more points: Ben Best's Cryonics FAQ, Alcor's FAQ, Alcor FAQ for scientists, Scientists' Open Letter on Cryonics I know more people who are planning to sign up for cryonics Real Soon Now than people who have actually signed up. I expect that more people have died while cryocrastinating than have actually been cryopreserved. If you've already decided this is a good idea, but you "haven't gotten around to it", sign up for cryonics NOW. I mean RIGHT NOW. Go to the website of Alcor or the Cryonics Institute and follow the instructions. Cryonics is usually funded through life insurance. The following conversation from an Overcoming Bias meetup is worth quoting: Him: I've been thinking about signing up for cryonics when I've got enough money. Me: Um... it doesn't take all that much money. Him: It doesn't? Me: Alcor is the high-priced high-quality organization, which is something like $500-$1000 in annual fees for the organization, I'm not sure how much. I'm young, so I'm signed up with the Cryonics Institute, which is $120/year for the membership. I pay $180/year for more insurance than I need - it'd be enough for Alcor too. Him: That's ridiculous. Me: Yes. Him: No, really, that's ridiculous. If that's true then my decision isn't just determined, it's overdetermined. Me: Yes. And there's around a thousand people worldwide [actually 1400] who are signed up for cryonics. Figure that at most a quarter of those did it for systematically rational reasons. That's a high upper bound on the number of people on Earth who can reliably reach the right conclusion on massively overdetermined issues. Cryonics is not marketed well - or at all, really. There's no salespeople who get commissions. There is no one to hold your hand through signing up, so you're going to have to get the papers signed and notarized yourself. The closest thing out there might be Rudi Hoffman, who sells life insurance with cryonics-friendly insurance providers (I went through him). If you want to securely erase a hard drive, it's not as easy as writing it over with zeroes. Sure, an "erased" hard drive like this won't boot up your computer if you just plug it in again. But if the drive falls into the hands of a specialist with a scanning tunneling microscope, they can tell the difference between "this was a 0, overwritten by a 0" and "this was a 1, overwritten by a 0". There are programs advertised to "securely erase" hard drives using many overwrites of 0s, 1s, and random data. But if you want to keep the secret on your hard drive secure against all possible future technologies that might ever be developed, then cover it with thermite and set it on fire. It's the only way to be sure. Pumping someone full of cryoprotectant and gradually lowering their temperature until they can be stored in liquid nitrogen is not a secure way to erase a person. See also the information-theoretic criterion of death. You don't have to buy what's usually called the "patternist" philosophy of identity, to sign up for cryonics. After reading all the information off the brain, you could put the "same atoms" back into their old places. "Same atoms" is in scare quotes because our current physics prohibits particles from possessing individual identities. It's a much stronger statement than "we can't tell the particles apart wi...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why haven't we celebrated any major achievements lately?, published by jasoncrawford on the LessWrong. This is a linkpost for In reading stories of progress, one thing that has struck me was the wild, enthusiastic celebrations that accompanied some of them in the past. Read some of these stories; somehow it’s hard for me to imagine similar jubilation happening today: The US transcontinental railroad, 1869 The transcontinental railroad was the first to link the US east and west. Prior to the railroad, to travel from coast to coast could take six months, whether by land or sea, and the journey was hard and perilous. California was like a foreign colony, separated from the life and industry of the East. The railroad changed that completely, taking a six-month journey down to a matter of days. Here’s how the western cities reacted, from Stephen Ambrose’s book Nothing Like It in the World: At 5 A.M. on Saturday, a Central Pacific train pulled into Sacramento carrying celebrants from Nevada, including firemen and a brass band. They got the festivities going by starting their parade. A brass cannon, the very one that had saluted the first shovelful of earth Leland Stanford had turned over for the beginning of the CP’s construction six years earlier, boomed once again. The parade was mammoth. At its height, about 11 A.M. in Sacramento, the time the organizers had been told the joining of the rails would take place, twenty-three of the CP’s locomotives, led by its first, the Governor Stanford, let loose a shriek of whistles that lasted for fifteen minutes. In San Francisco, the parade was the biggest held to date. At 11 A.M., a fifteen-inch Parrott rifled cannon at Fort Point, guarding the south shore of the Golden Gate, fired a salute. One hundred guns followed. Then fire bells, church bells, clock towers, machine shops, streamers, foundries, the U.S. Mint let go at full blast. The din lasted for an hour. In both cities, the celebration went on through Saturday, Sunday, and Monday. The Brooklyn Bridge, 1883 The Brooklyn Bridge did not connect a distance nearly as great as the transcontinental railroad, but it too was met with grand celebrations. An excerpt from David McCullough’s The Great Bridge: When the Erie Canal was opened in the autumn of 1825, there were four former Presidents of the United States present in New York City for the occasion—John Adams, Thomas Jefferson, James Madison, and James Monroe—as well as John Quincy Adams, then occupying the White House, and General Andrew Jackson, who would take his place. When the Brooklyn Bridge was opened on May 24, 1883, the main attraction was Chester A. Arthur. . Seth Low made the official greeting for the City of Brooklyn, the Marines presented arms, a signal flag was dropped nearby and instantly there was a crash of a gun from the Tennessee. Then the whole fleet commenced firing. Steam whistles on every tug, steamboat, ferry, every factory along the river, began to scream. More cannon boomed. Bells rang, people were cheering wildly on every side. The band played “Hail to the Chief” maybe six or seven more times, and as the New York Sun reported, “the climax of fourteen years’ suspense seemed to have been reached, since the President of the United States of America had walked dry shod to Brooklyn from New York.” Not only did they celebrate, they analyzed and philosophized: What was it all about? What was everyone celebrating? The speakers of the day had a number of ideas. The bridge was a “wonder of Science,” an “astounding exhibition of the power of man to change the face of nature.” It was a monument to “enterprise, skill, faith, endurance.” It was also a monument to “public spirit,” “the moral qualities of the human soul,” and a great, everlasting symbol of “Peace.” The words used most often were “Science,” “Commerce,...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The 2021 Less Wrong Darwin Game, published by lsusr on the LessWrong. It's fall and that means it's time for another Less Wrong Darwin Game. This year, you'll be designing up to ten species that will compete for food including (sometimes) eating each other. Click here to participate [Entries are now closed.] You have one week from September 23 to design your species. Submit them by September 30th or earlier. Each player starts with a population of organisms. Each round each of your organisms will be randomly paired with another organism. At this point, one of two things will happen: If one organism can eat the other organism then it will do so. If nobody gets eaten then both organisms get an opportunity to forage for plants. After everyone has eaten, each organism will attempt to reproduce. The more an organism eats you eat the most descendents an organism can leave. Food Each round your organisms lose 20% of their energy to metabolism resulting (on average) in a 20% decrease in population. You must eat food to counteract metabolism. There are two sources of food: plants and other animals. Predation There are two phases to combat. In the first phase organisms size each other up to figure out which is the predator and which is prey. There are two ways for an organism to become the predator. Venom. If one organism has venom but the other does not have antivenom then the organism with venom is the predator. (Antivenom is a prerequisite to venom.) Weapons. Weapons represent claws, teeth and tusks. If either organism's weapons value exceeds the prey's weapons + armor then the organism with the higher weapons value will become the predator. Venom takes priority over weapons. Once a predator-prey relationship is established (if a predator-prey relationship is established) the prey will get a chance to escape. If the prey's speed equals or exceeds the predator's then nobody gets eaten. Venom, weapons and antivenom all make your organism bigger, which slows down reproduction. Adaptation Size Notes Venom 6 Requires Antivenom Antivenom 1 Weapons × n n n ≥ 0 Armor × n 3 n 4 n ≥ 0 Speed × n n n ≥ 0 Omnivores priorize meat over plants, when they can get it even if foraging for plants would be more metabolically efficient[1]. Predation has an efficiency of 0.95[2]. That means 95% of the prey's energy can be used by the predator. Only organisms of different species eat each other. Cannibalism is disabled. Foraging for Plants There are various kinds of plant food available. In order to eat each food you'll need the proper digestive system. Food Nutritional Value Size Leaves 7 5 Grass 6 3 Seeds 5 1 Whether your organism can digest a particular plant food is a binary value. No organism is better at digesting leaves than any other organism. There is a tradeoff. The ability to eat leaves/grass/seeds makes your organism bigger which slows down reproduction. Also, there is a finite supply of leaves/grass/seeds. The more other organisms are foraging from a plant source, the less advantageous it is for you to forage for it youself. Simple Ecosystems Consider an ecosystem with three kinds of plant food available: seeds, leaves and grass. 1,000 units of each plant food are produced per round. Example 1 This all may sound a little confusing but it makes sense once we use some real exampless. Let's start with two species: housecats and mice. Species Weapons Speed Eats Seeds? Housecat 1 2 No Mouse 0 1 Yes At first, both populations grow. The mice reproduce faster than the cats. Then the cats catch up and eat all of the mice. Having exhausted their food supply, the cat population starves to extinction. Example 2 What happens if we add songbirds? Songbirds fly. They are fast enough to evade cats. But speed costs energy which makes it more expensive for songbirds to breed than for mice to breed. Mic...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Talking Snakes: A Cautionary Taleby Scott Alexander, published by Scott Alexander on the LessWrong. I particularly remember one scene from Bill Maher's "Religulous". I can't find the exact quote, but I will try to sum up his argument as best I remember. Christians believe that sin is caused by a talking snake. They may have billions of believers, thousands of years of tradition behind them, and a vast literature of apologetics justifying their faith - but when all is said and done, they're adults who believe in a talking snake. I have read of the absurdity heuristic. I know that it is not carte blanche to go around rejecting beliefs that seem silly. But I was still sympathetic to the talking snake argument. After all...a talking snake? I changed my mind in a Cairo cafe, talking to a young Muslim woman. I let it slip during the conversation that I was an atheist, and she seemed genuinely curious why. You've all probably been in such a situation, and you probably know how hard it is to choose just one reason, but I'd been reading about Biblical contradictions at the time and I mentioned the myriad errors and atrocities and contradictions in all the Holy Books. Her response? "Oh, thank goodness it's that. I was afraid you were one of those crazies who believed that monkeys transformed into humans." I admitted that um, well, maybe I sorta kinda might in fact believe that. It is hard for me to describe exactly the look of shock on her face, but I have no doubt that her horror was genuine. I may have been the first flesh-and-blood evolutionist she ever met. "But..." she looked at me as if I was an idiot. "Monkeys don't change into humans. What on Earth makes you think monkeys can change into humans?" I admitted that the whole process was rather complicated. I suggested that it wasn't exactly a Optimus Prime-style transformation so much as a gradual change over eons and eons. I recommended a few books on evolution that might explain it better than I could. She said that she respected me as a person but that quite frankly I could save my breath because there was no way any book could possibly convince her that monkeys have human babies or whatever sort of balderdash I was preaching. She accused me and other evolution believers of being too willing to accept absurdities, motivated by our atheism and our fear of the self-esteem hit we'd take by accepting Allah was greater than ourselves. It is not clear to me that this woman did anything differently than Bill Maher. Both heard statements that sounded so crazy as to not even merit further argument. Both recognized that there was a large group of people who found these statements plausible and had written extensive literature justifying them. Both decided that the statements were so absurd as to not merit examining that literature more closely. Both came up with reasons why they could discount the large number of believers because those believers must be biased. I post this as a cautionary tale as we discuss the logic or illogic of theism. I propose taking from it the following lessons: - The absurdity heuristic doesn't work very well. - Even on things that sound really, really absurd. - If a large number of intelligent people believe something, it deserves your attention. After you've studied it on its own terms, then you have a right to reject it. You could still be wrong, though. - Even if you can think of a good reason why people might be biased towards the silly idea, thus explaining it away, your good reason may still be false. - If someone cannot explain why something is not stupid to you over twenty minutes at a cafe, that doesn't mean it's stupid. It just means it's complicated, or they're not very good at explaining things. - There is no royal road. (special note to those prone to fundamental attribution errors: I do n...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Martial Art of Rationality, published by Eliezer Yudkowskyon the LessWrong. I often use the metaphor that rationality is the martial art of mind. You don’t need huge, bulging muscles to learn martial arts—there’s a tendency toward more athletic people being more likely to learn martial arts, but that may be a matter of enjoyment as much as anything else. If you have a hand, with tendons and muscles in the appropriate places, then you can learn to make a fist. Similarly, if you have a brain, with cortical and subcortical areas in the appropriate places, you might be able to learn to use it properly. If you’re a fast learner, you might learn faster—but the art of rationality isn’t about that; it’s about training brain machinery we all have in common. And where there are systematic errors human brains tend to make—like an insensitivity to scope—rationality is about fixing those mistakes, or finding work-arounds. Alas, our minds respond less readily to our will than our hands. Our ability to control our muscles is evolutionarily ancient; our ability to reason about our own reasoning processes is a much more recent innovation. We shouldn’t be surprised, then, that muscles are easier to use than brains. But it is not wise to neglect the latter training because it is more difficult. It is not by bigger muscles that the human species rose to prominence upon Earth. If you live in an urban area, you probably don’t need to walk very far to find a martial arts dojo. Why aren’t there dojos that teach rationality? One reason, perhaps, is that it’s harder to verify skill. To rise a level in Tae Kwon Do, you might need to break a board of a certain width. If you succeed, all the onlookers can see and applaud. If you fail, your teacher can watch how you shape a fist, and check if you shape it correctly. If not, the teacher holds out a hand and makes a fist correctly, so that you can observe how to do so. Within martial arts schools, techniques of muscle have been refined and elaborated over generations. Techniques of rationality are harder to pass on, even to the most willing student. Very recently—in just the last few decades—the human species has acquired a great deal of new knowledge about human rationality. The most salient example would be the heuristics and biases program in experimental psychology. There is also the Bayesian systematization of probability theory and statistics; evolutionary psychology; social psychology. Experimental investigations of empirical human psychology; and theoretical probability theory to interpret what our experiments tell us; and evolutionary theory to explain the conclusions. These fields give us new focusing lenses through which to view the landscape of our own minds. With their aid, we may be able to see more clearly the muscles of our brains, the fingers of thought as they move. We have a shared vocabulary in which to describe problems and solutions. Humanity may finally be ready to synthesize the martial art of mind: to refine, share, systematize, and pass on techniques of personal rationality. Such understanding as I have of rationality, I acquired in the course of wrestling with the challenge of artificial general intelligence (an endeavor which, to actually succeed, would require sufficient mastery of rationality to build a complete working rationalist out of toothpicks and rubber bands). In most ways the AI problem is enormously more demanding than the personal art of rationality, but in some ways it is actually easier. In the martial art of mind, we need to acquire the realtime procedural skill of pulling the right levers at the right time on a large, pre-existing thinking machine whose innards are not end-user-modifiable. Some of the machinery is optimized for evolutionary selection pressures that run directly counter to our declare...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Power Buys You Distance From The Crime , published by Elizabeth on the LessWrong. Introduction Taxes are typically meant to be proportional to money (or negative externalities, but that's not what I'm focusing on). But one thing money buys you is flexibility, which can be used to avoid taxes. Because of this, taxes aimed at the wealthy tend to end up hitting the well-off-or-rich-but-not-truly-wealthy harder, and tax cuts aimed at the poor end up helping the middle class. Examples (feel free to stop reading these when you get the idea, this is just the analogy section of the essay): Computer programmers typically have the option to work remotely in a low-tax state; teachers need to be where the classroom is. Estate taxes tend to hit families with single large assets (like a business) harder than those with diverse investments (who can simply sell assets to pay for taxes), who are hit harder than those with enough wealth to create trust funds. Executives can choose to receive stock (which is taxed more favorably) instead of cash to the exact percentage they desire. Well paid employees are offered stock, but the amount will not be tailored to their needs. Lower level employees either are not offered this, or are not in a position to take advantage of it. The legal distinction between a business (whose expenses are tax deductible) and a hobby (deductions not allowed) is based on whether the activity nets you income (there are complications and you can sometimes prove a money loser is a business, but this is a good rule of thumb). Small business owners (e.g. lawyers) can fold their occasionally-revenue-generating hobby (e.g. photography) into their real business, enabling tax deductions for their hobby. IRAs, 401ks, HSAs, and FSAs all lock your money up for a time or purpose, in exchange for lower or delayed taxes. You can only take advantage of them if you’re sure you won’t need the money for another purpose sooner. More examples here. Note that most of these are perfectly legal and the rest are borderline. But we're still not getting the result we want, of taxes being proportional to income. When we assess moral blame for a situation, we typically want it to be roughly in proportion to much power a person has to change said situation. But just like money can be used to evade taxes, power can be used to avoid blame. This results in a distorted blame-distribution apparatus which assigns the least blame to the person most able to change the situation. Allow me a few examples to demonstrate this. Examples 1 + 2: Corporate Malfeasance Amazon.com provides a valuable service by letting any idiot sell a book, with minimal overhead. One of the costs of this complete lack of verification is that people will sell things that wouldn't pass verification, such as counterfeits, at great cost to publishers and authors. Amazon could never sell counterfeits directly: they're a large company that's easy to sue. But by setting themselves up as a platform on which other people sell, they enable themselves to profit from counterfeits. Or take slavery. No company goes “I’m going to go out and enslave people today” (especially not publicly), but not paying people is sometimes cheaper than paying them, so financial pressure will push towards slavery. Public pressure pushes in the opposite direction, so companies try not to visibly use slave labor. But they can’t control what their subcontractors do, and especially not what their subcontractors’ subcontractors’ subcontractors do, and sometimes this results in workers being unpaid and physically blocked from leaving. Who’s at fault for the subcontractor(^3)’s slave labor? One obvious answer is “the person locking them in during the fire” or “the parent who gives their kid piecework”, and certainly it couldn’t happen without them. But if we say “N...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: human psycholinguists: a critical appraisal, published by nostalgebraist on the LessWrong. (The title of this post is a joking homage to one of Gary Marcus’ papers.) I’ve discussed GPT-2 and BERT and other instances of the Transformer architecture a lot on this blog. As you can probably tell, I find them very interesting and exciting. But not everyone has the reaction I do, including some people who I think ought to have that reaction. Whatever else GPT-2 and friends may or may not be, I think they are clearly a source of fascinating and novel scientific evidence about language and the mind. That much, I think, should be uncontroversial. But it isn’t. (i.) When I was a teenager, I went through a period where I was very interested in cognitive psychology and psycholinguistics. I first got interested via Steven Pinker’s popular books – this was back when Pinker was mostly famous for writing about psychology rather than history and culture – and proceeded to read other, more academic books by authors like Gary Marcus, Jerry Fodor, and John Anderson. At this time (roughly 2002-6), there was nothing out there that remotely resembled GPT-2. Although there were apparently quite mature and complete formal theories of morphology and syntax, which could accurately answer questions like “is this a well-formed English sentence?”, no one really knew how these could or should be implemented in a physical system meant to understand or produce language. This was true in two ways. For one thing, no one knew how the human brain implemented this stuff, although apparently it did. But the difficulty was more severe than that: even if you forgot about the brain, and just tried to write a computer program (any computer program) that understood or produced language, the results would be dismal. At the time, such programs were either specialized academic models of one specific phenomenon – for example, a program that could form the past tense of a verb, but couldn’t do anything else – or they were ostensibly general-purpose but incredibly brittle and error-prone, little more than amusing toys. The latter category included some programs intended as mere amusements or provocations, like the various chatterbots (still about as good/bad as ELIZA after four decades), but also more serious efforts whose reach exceeded their grasp. SYSTRAN spent decades manually curating millions of morphosyntactic and semantic facts for enterprise-grade machine translation; you may remember the results in the form of the good old Babel Fish website, infamous for its hilariously inept translations. This was all kind of surprising, given that the mature formal theories were right there, ready to be programmed into rule-following machines. What was going on? The impression I came away with, reading about this stuff as a teenager, was of language as a fascinating and daunting enigma, simultaneously rule-based and rife with endless special cases that stacked upon one another. It was formalism, Jim, but not as we knew it; it was a magic interleaving of regular and irregular phenomena, arising out of the distinctive computational properties of some not-yet-understood subset of brain architecture, which the models of academics and hackers could crudely imitate but not really grok. We did not have the right “language” to talk about language the way our own brains did, internally. (ii.) The books I read, back then, talked a lot about this thing called “connectionism.” This used to be a big academic debate, with people arguing for and against “connectionism.” You don’t hear that term much these days, because the debate has been replaced by a superficially similar but actually very different debate over “deep learning,” in which what used to be good arguments about “connectionism” are repeated in cruder form as bad arguments...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Simulacra Levels and their Interactions, published by Zvi on the LessWrong. Previously: Covid-19: My Current Model, On Negative Feedback and Simulacra This post aims to unpack and explain simulacra levels of action using the threat of covid-19 as its central example. My intention is for future posts to then apply this model to many covid-related dynamics. In Elizabeth’s Negative Feedback and Simulacra, she examined several example situations on which information was being processed on multiple simulacra levels at once. On Negative Feedback and Simulacra was my take on those examples. To re-familiarize ourselves with the simulacra levels, here’s the introduction Elizabeth offered to them in her post: My friend Ben Hoffman talks about simulacra a lot, with this rough definition: First, words were used to maintain shared accounting. We described reality intersubjectively in order to build shared maps, the better to navigate our environment. I say that the food source is over there, so that our band can move towards or away from it when situationally appropriate, or so people can make other inferences based on this knowledge. 2. The breakdown of naive intersubjectivity – people start taking the shared map as an object to be manipulated, rather than part of their own subjectivity. For instance, I might say there’s a lion over somewhere where I know there’s food, in order to hoard access to that resource for idiosyncratic advantage. Thus, the map drifts from reality, and we start dissociating from the maps we make. 3. When maps drift far enough from reality, in some cases people aren’t even parsing it as though it had a literal specific objective meaning that grounds out in some verifiable external test outside of social reality. Instead, the map becomes a sort of command language for coordinating actions and feelings. “There’s food over there” is perhaps construed as a bid to move in that direction, and evaluated as though it were that call to action. Any argument for or against the implied call to action is conflated with an argument for or against the proposition literally asserted. This is how arguments become soldiers. Any attempt to simply investigate the literal truth of the proposition is considered at best naive and at worst politically irresponsible. But since this usage is parasitic on the old map structure that was meant to describe something outside the system of describers, language is still structured in terms of reification and objectivity, so it substantively resembles something with descriptive power, or “aboutness.” For instance, while you cannot acquire a physician’s privileges and social role simply by providing clear evidence of your ability to heal others, those privileges are still justified in terms of pseudo-consequentialist arguments about expertise in healing. 4. Finally, the pseudostructure itself becomes perceptible as an object that can be manipulated, the pseudocorrespondence breaks down, and all assertions are nothing but moves in an ever-shifting game where you’re trying to think a bit ahead of the others (for positional advantage), but not too far ahead. If that doesn’t make sense, try this anonymous comment on the post Level 1: “There’s a lion across the river.” = There’s a lion across the river. Level 2: “There’s a lion across the river.” = I don’t want to go (or have other people go) across the river. Level 3: “There’s a lion across the river.” = I’m with the popular kids who are too cool to go across the river. Level 4: “There’s a lion across the river.” = A firm stance against trans-river expansionism focus grouped well with undecided voters in my constituency. Almost everyone would rather not be eaten by a lion. I certainly would rather not be eaten by a lion. Whether or not I am eaten by a lion still does not drive much of my decis...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Predictive Coding has been Unified with Backpropagation, published by lsusr on the LessWrong. Artificial Neural Networks (ANNs) are based around the backpropagation algorithm. The backpropagation algorithm allows you to perform gradient descent on a network of neurons. When we feed training data through an ANNs, we use the backpropagation algorithm to tell us how the weights should change. ANNs are good at inference problems. Biological Neural Networks (BNNs) are good at inference too. ANNs are built out of neurons. BNNs are built out of neurons too. It makes intuitive sense that ANNs and BNNs might be running similar algorithms. There is just one problem: BNNs are physically incapable of running the backpropagation algorithm. We do not know quite enough about biology to say it is impossible for BNNs to run the backpropagation algorithm. However, "a consensus has emerged that the brain cannot directly implement backprop, since to do so would require biologically implausible connection rules"[1]. The backpropagation algorithm has three steps. Flow information forward through a network to compute a prediction. Compute an error by comparing the prediction to a target value. Flow the error backward through the network to update the weights. The backpropagation algorithm requires information to flow forward and backward along the network. But biological neurons are one-directional. An action potential goes from the cell body down the axon to the axon terminals to another cell's dendrites. An axon potential never travels backward from a cell's terminals to its body. Hebbian theory Predictive coding is the idea that BNNs generate a mental model of their environment and then transmit only the information that deviates from this model. Predictive coding considers error and surprise to be the same thing. Hebbian theory is specific mathematical formulation of predictive coding. Predictive coding is biologically plausible. It operates locally. There are no separate prediction and training phases which must be synchronized. Most importantly, it lets you train a neural network without sending axon potentials backwards. Predictive coding is easier to implement in hardware. It is locally-defined; it parallelizes better than backpropagation; it continues to function when you cut its substrate in half. (Corpus callosotomy is used to treat epilepsy.) Digital computers break when you cut them in half. Predictive coding is something evolution could plausibly invent. Unification The paper Predictive Coding Approximates Backprop Along Arbitrary Computation Graphs[1:1] "demonstrate[s] that predictive coding converges asymptotically (and in practice rapidly) to exact backprop gradients on arbitrary computation graphs using only local learning rules." The authors have unified predictive coding and backpropagation into a single theory of neural networks. Predictive coding and backpropagation are separate hardware implementations of what is ultimately the same algorithm. There are two big implications of this. This paper permanently fuses artificial intelligence and neuroscience into a single mathematical field. This paper opens up possibilities for neuromorphic computing hardware. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Curing insanity with malaria, published by Swimmer963 on the LessWrong. Sometimes the history of medicine is very, very surreal. For example, consider that in 1927, a physician named Julius Wagner-Jauregg received the Nobel Prize in medicine, for...deliberately infecting his patients with malaria. As a treatment for psychosis. This often worked. Well, it did kill around 15% of the patients, but it was nonetheless seen as a miracle cure. General paralysis of the insane was first identified and described as a distinct disease in the early 19th century. It was initially thought to be caused by an ‘inherent weakness of character’. The initial symptoms were of mental deterioration and personality changes; patients suffered a loss of social inhibitions, gradual impairment of judgment, concentration and short-term memory. They might experience euphoria, mania, depression, or apathy. Delusions were common, including “ideas of great wealth, immortality, thousands of lovers, and unfathomable power” – or, on the more negative side, nihilism, self-guilt, and self-blame. It was a progressive disease, and nearly always a death sentence. As the condition advanced, the patient would develop worsening dementia, motor and reflex abnormalities, and often seizures; death usually took 3 to 5 years from the initial symptoms. In the 19th century, cases of general paralysis could account for up to 25% of admissions to asylums. Some physicians were drawing a connection between general paralysis and syphilis infection as early as the 1850s; however, it took until much later for this explanation to be generally accepted within the medical community, and full confirmation via pathology examinations of the brains of patients who had died of the disease would have to wait until 1913. In 1909, an antisyphilitic drug compound was discovered via a process of trialing hundreds of newly synthesized organic arsenical chemicals, looking for one that would have anti-microbial activity but not kill the human patient; this was the first research team effort to optimize biological effects of a promising chemical, which is now the basis of a huge amount of pharmaceuticals research. Unfortunately, arsphenamine, also known as Salvarsan or “606”, was difficult to prepare and administer, and was still fairly toxic to the human patient as well as the syphilis. Julius Wagner-Jauregg was a Viennese psychiatrist, but a psychiatrist with a particular interest in experimental pathology, and in brains. Already in the mid-1880s, he was noticing an odd pattern; many of his psychiatric patients were showing improvements in their mental condition after recovering from bouts of other illnesses that resulted in fever. Wagner-Jauregg formed two hypotheses. One, some cases of insanity had ‘organic’, biological causes and were related to physical dysfunctions in the brain; two, one disease could be fought by another. He tried deliberately inducing fevers in his patients, by injecting them with tuberculin, a sterile protein extract from cultures of the tubercle bacillus responsible for tuberculosis. However, this was inconsistent at producing a fever, and the results were disappointing. In 1917, a soldier ill with malaria was admitted to Wagner-Jauregg’s ward. No, I am not at all sure why a malaria patient was being treated in a psychiatric ward! And, apparently, neither was Wagner-Jauregg: “Should he be given quinine?” [my assistant Dr. Alfred Fuchs] asked. I immediately said: “No.” This I regarded as a sign of destiny. Because soldiers with malaria were usually not admitted to my wards, which accepted only cases suffering from a psychosis or patients with injuries to the central nervous system. Wagner-Jauregg would have known that malaria is especially likely to cause repeated, intermittent paroxysms of high fever. Also, unlik...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Are Your Enemies Innately Evil?, published by Eliezer Yudkowsky on the LessWrong. We see far too direct a correspondence between others’ actions and their inherent dispositions. We see unusual dispositions that exactly match the unusual behavior, rather than asking after real situations or imagined situations that could explain the behavior. We hypothesize mutants. When someone actually offends us—commits an action of which we (rightly or wrongly) disapprove—then, I observe, the correspondence bias redoubles. There seems to be a very strong tendency to blame evil deeds on the Enemy’s mutant, evil disposition. Not as a moral point, but as a strict question of prior probability, we should ask what the Enemy might believe about their situation that would reduce the seeming bizarrity of their behavior. This would allow us to hypothesize a less exceptional disposition, and thereby shoulder a lesser burden of improbability. On September 11th, 2001, nineteen Muslim males hijacked four jet airliners in a deliberately suicidal effort to hurt the United States of America. Now why do you suppose they might have done that? Because they saw the USA as a beacon of freedom to the world, but were born with a mutant disposition that made them hate freedom? Realistically, most people don’t construct their life stories with themselves as the villains. Everyone is the hero of their own story. The Enemy’s story, as seen by the Enemy, is not going to make the Enemy look bad. If you try to construe motivations that would make the Enemy look bad, you’ll end up flat wrong about what actually goes on in the Enemy’s mind. But politics is the mind-killer. Debate is war; arguments are soldiers. If the Enemy did have an evil disposition, that would be an argument in favor of your side. And any argument that favors your side must be supported, no matter how silly—otherwise you’re letting up the pressure somewhere on the battlefront. Everyone strives to outshine their neighbor in patriotic denunciation, and no one dares to contradict. Soon the Enemy has horns, bat wings, flaming breath, and fangs that drip corrosive venom. If you deny any aspect of this on merely factual grounds, you are arguing the Enemy’s side; you are a traitor. Very few people will understand that you aren’t defending the Enemy, just defending the truth. If it took a mutant to do monstrous things, the history of the human species would look very different. Mutants would be rare. Or maybe the fear is that understanding will lead to forgiveness. It’s easier to shoot down evil mutants. It is a more inspiring battle cry to scream, “Die, vicious scum!” instead of “Die, people who could have been just like me but grew up in a different environment!” You might feel guilty killing people who weren’t pure darkness. This looks to me like the deep-seated yearning for a one-sided policy debate in which the best policy has no drawbacks. If an army is crossing the border or a lunatic is coming at you with a knife, the policy alternatives are (a) defend yourself or (b) lie down and die. If you defend yourself, you may have to kill. If you kill someone who could, in another world, have been your friend, that is a tragedy. And it is a tragedy. The other option, lying down and dying, is also a tragedy. Why must there be a non-tragic option? Who says that the best policy available must have no downside? If someone has to die, it may as well be the initiator of force, to discourage future violence and thereby minimize the total sum of death. If the Enemy has an average disposition, and is acting from beliefs about their situation that would make violence a typically human response, then that doesn’t mean their beliefs are factually accurate. It doesn’t mean they’re justified. It means you’ll have to shoot down someone who is the hero of their own ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Power of Reinforcement, published by The Power of Reinforcement on the LessWrong. Part of the sequence: The Science of Winning at Life Also see: Basics of Animal Reinforcement, Basics of Human Reinforcement, Physical and Mental Behavior, Wanting vs. Liking Revisited, Approving reinforces low-effort behaviors, Applying Behavioral Psychology on Myself. Story 1: On Skype with Eliezer, I said: "Eliezer, you've been unusually pleasant these past three weeks. I'm really happy to see that, and moreover, it increases my probability than an Eliezer-led FAI research team will work. What caused this change, do you think?" Eliezer replied: "Well, three weeks ago I was working with Anna and Alicorn, and every time I said something nice they fed me an M&M." Story 2: I once witnessed a worker who hated keeping a work log because it was only used "against" him. His supervisor would call to say "Why did you spend so much time on that?" or "Why isn't this done yet?" but never "I saw you handled X, great job!" Not surprisingly, he often "forgot" to fill out his worklog. Ever since I got everyone at the Singularity Institute to keep work logs, I've tried to avoid connections between "concerned" feedback and staff work logs, and instead take time to comment positively on things I see in those work logs. Story 3: Chatting with Eliezer, I said, "Eliezer, I get the sense that I've inadvertently caused you to be slightly averse to talking to me. Maybe because we disagree on so many things, or something?" Eliezer's reply was: "No, it's much simpler. Our conversations usually run longer than our previously set deadline, so whenever I finish talking with you I feel drained and slightly cranky." Now I finish our conversations on time. Story 4: A major Singularity Institute donor recently said to me: "By the way, I decided that every time I donate to the Singularity Institute, I'll set aside an additional 5% for myself to do fun things with, as a motivation to donate." The power of reinforcement It's amazing to me how consistently we fail to take advantage of the power of reinforcement. Maybe it's because behaviorist techniques like reinforcement feel like they don't respect human agency enough. But if you aren't treating humans more like animals than most people are, then you're modeling humans poorly. You are not an agenty homunculus "corrupted" by heuristics and biases. You just are heuristics and biases. And you respond to reinforcement, because most of your motivation systems still work like the motivation systems of other animals. A quick reminder of what you learned in high school A reinforcer is anything that, when it occurs in conjunction with an act, increases the probability that the act will occur again. A positive reinforcer is something the subject wants, such as food, petting, or praise. Positive reinforcement occurs when a target behavior is followed by something the subject wants, and this increases the probability that the behavior will occur again. A negative reinforcer is something the subject wants to avoid, such as a blow, a frown, or an unpleasant sound. Negative reinforcement occurs when a target behavior is followed by some relief from something the subject doesn't want, and this increases the probability that the behavior will happen again. What works Small reinforcers are fine, as long as there is a strong correlation between the behavior and the reinforcer (Schneider 1973; Todorov et al. 1984). All else equal, a large reinforcer is more effective than a small one (Christopher 1988; Ludvig et al. 2007; Wolfe 1936), but the more you increase the reinforcer magnitude, the less benefit you get from the increase (Frisch & Dickinson 1990). The reinforcer should immediately follow the target behavior (Escobar & Bruner 2007; Schlinger & Blakely 1994; Schneider 1990). Pryor...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Power of Reinforcement, published by lukeprog on the LessWrong. Part of the sequence: The Science of Winning at Life Also see: Basics of Animal Reinforcement, Basics of Human Reinforcement, Physical and Mental Behavior, Wanting vs. Liking Revisited, Approving reinforces low-effort behaviors, Applying Behavioral Psychology on Myself. Story 1: On Skype with Eliezer, I said: "Eliezer, you've been unusually pleasant these past three weeks. I'm really happy to see that, and moreover, it increases my probability than an Eliezer-led FAI research team will work. What caused this change, do you think?" Eliezer replied: "Well, three weeks ago I was working with Anna and Alicorn, and every time I said something nice they fed me an M&M." Story 2: I once witnessed a worker who hated keeping a work log because it was only used "against" him. His supervisor would call to say "Why did you spend so much time on that?" or "Why isn't this done yet?" but never "I saw you handled X, great job!" Not surprisingly, he often "forgot" to fill out his worklog. Ever since I got everyone at the Singularity Institute to keep work logs, I've tried to avoid connections between "concerned" feedback and staff work logs, and instead take time to comment positively on things I see in those work logs. Story 3: Chatting with Eliezer, I said, "Eliezer, I get the sense that I've inadvertently caused you to be slightly averse to talking to me. Maybe because we disagree on so many things, or something?" Eliezer's reply was: "No, it's much simpler. Our conversations usually run longer than our previously set deadline, so whenever I finish talking with you I feel drained and slightly cranky." Now I finish our conversations on time. Story 4: A major Singularity Institute donor recently said to me: "By the way, I decided that every time I donate to the Singularity Institute, I'll set aside an additional 5% for myself to do fun things with, as a motivation to donate." The power of reinforcement It's amazing to me how consistently we fail to take advantage of the power of reinforcement. Maybe it's because behaviorist techniques like reinforcement feel like they don't respect human agency enough. But if you aren't treating humans more like animals than most people are, then you're modeling humans poorly. You are not an agenty homunculus "corrupted" by heuristics and biases. You just are heuristics and biases. And you respond to reinforcement, because most of your motivation systems still work like the motivation systems of other animals. A quick reminder of what you learned in high school A reinforcer is anything that, when it occurs in conjunction with an act, increases the probability that the act will occur again. A positive reinforcer is something the subject wants, such as food, petting, or praise. Positive reinforcement occurs when a target behavior is followed by something the subject wants, and this increases the probability that the behavior will occur again. A negative reinforcer is something the subject wants to avoid, such as a blow, a frown, or an unpleasant sound. Negative reinforcement occurs when a target behavior is followed by some relief from something the subject doesn't want, and this increases the probability that the behavior will happen again. What works Small reinforcers are fine, as long as there is a strong correlation between the behavior and the reinforcer (Schneider 1973; Todorov et al. 1984). All else equal, a large reinforcer is more effective than a small one (Christopher 1988; Ludvig et al. 2007; Wolfe 1936), but the more you increase the reinforcer magnitude, the less benefit you get from the increase (Frisch & Dickinson 1990). The reinforcer should immediately follow the target behavior (Escobar & Bruner 2007; Schlinger & Blakely 1994; Schneider 1990). Pryor (2007) notes that...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Less Wrong Rationality and Mainstream Philosophy, published by lukeprog on the LessWrong. Part of the sequence: Rationality and Philosophy Despite Yudkowsky's distaste for mainstream philosophy, Less Wrong is largely a philosophy blog. Major topics include epistemology, philosophy of language, free will, metaphysics, metaethics, normative ethics, machine ethics, axiology, philosophy of mind, and more. Moreover, standard Less Wrong positions on philosophical matters have been standard positions in a movement within mainstream philosophy for half a century. That movement is sometimes called "Quinean naturalism" after Harvard's W.V. Quine, who articulated the Less Wrong approach to philosophy in the 1960s. Quine was one of the most influential philosophers of the last 200 years, so I'm not talking about an obscure movement in philosophy. Let us survey the connections. Quine thought that philosophy was continuous with science - and where it wasn't, it was bad philosophy. He embraced empiricism and reductionism. He rejected the notion of libertarian free will. He regarded postmodernism as sophistry. Like Wittgenstein and Yudkowsky, Quine didn't try to straightforwardly solve traditional Big Questions as much as he either dissolved those questions or reframed them such that they could be solved. He dismissed endless semantic arguments about the meaning of vague terms like knowledge. He rejected a priori knowledge. He rejected the notion of privileged philosophical insight: knowledge comes from ordinary knowledge, as best refined by science. Eliezer once said that philosophy should be about cognitive science, and Quine would agree. Quine famously wrote: The stimulation of his sensory receptors is all the evidence anybody has had to go on, ultimately, in arriving at his picture of the world. Why not just see how this construction really proceeds? Why not settle for psychology? But isn't this using science to justify science? Isn't that circular? Not quite, say Quine and Yudkowsky. It is merely "reflecting on your mind's degree of trustworthiness, using your current mind as opposed to something else." Luckily, the brain is the lens that sees its flaws. And thus, says Quine: Epistemology, or something like it, simply falls into place as a chapter of psychology and hence of natural science. Yudkowsky once wrote, "If there's any centralized repository of reductionist-grade naturalistic cognitive philosophy, I've never heard mention of it." When I read that I thought: What? That's Quinean naturalism! That's Kornblith and Stich and Bickle and the Churchlands and Thagard and Metzinger and Northoff! There are hundreds of philosophers who do that! Non-Quinean philosophy But I should also mention that LW philosophy / Quinean naturalism is not the largest strain of mainstream philosophy. Most philosophy is still done in relative ignorance (or ignoring) of cognitive science. Consider the preface to Rethinking Intuition: Perhaps more than any other intellectual discipline, philosophical inquiry is driven by intuitive judgments, that is, by what "we would say" or by what seems true to the inquirer. For most of philosophical theorizing and debate, intuitions serve as something like a source of evidence that can be used to defend or attack particular philosophical positions. One clear example of this is a traditional philosophical enterprise commonly known as conceptual analysis. Anyone familiar with Plato's dialogues knows how this type of inquiry is conducted. We see Socrates encounter someone who claims to have figured out the true essence of some abstract notion... the person puts forward a definition or analysis of the notion in the form of necessary and sufficient conditions that are thought to capture all and only instances of the concept in question. Socrates then refutes his interl...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Rest Days vs Recovery Days, published by Unreal on the LessWrong. Based on a comment I made on this EA Forum Post on Burnout. Related links: Sabbath hard and go home, Bring Back the Sabbath That comment I made generated more positive feedback than usual (in that people seemed to find it helpful to read and found themselves thinking about it months after reading it), so I'm elevating it to a LW post of its own. Consider this an update to the original comment. Like Ben Hoffman, I stumbled upon and rediscovered the Sabbath (although my implementation seems different from both Ben and Zvi). I was experiencing burnout at CFAR, and while I wasn't able to escape the effects entirely, I found some refuge in the following distinction between Rest Days and Recovery Days. Recovery Days A Recovery Day is where you're so tired or under-resourced that you can't do much of anything with yourself other than: stay in bed / sleep a lot, binge on Netflix or video games, stay in your room all day, play with your phone, use social media, and feel unmotivated to do much except easy, stimulating, and/or mind-numbing things. This is a Recovery Day and does not count as a Rest Day, but it is fine to take the time for them. However you aren't going to be refreshed from them. In order to really refresh, you need to take another day that counts as a Rest Day. Another way a person might take time off is to do things that are like work but easier. Video games are a prime example. I play a lot of video games that involve optimizing systems, and I find these really motivating and fun. But I notice that this is a kind of "work"—my mind is trying to solve problems and implement solutions. The difference is that because it's easy and doable, I get addicted to them, and it's a way for me to escape the "real" problems at work, which tend to be harder to solve. This also doesn't count as Resting. Rest Days Rest Days are days where I have enough energy and resources that I feel motivated and able to get out and about. (One way I can tell I have energy is that sometimes I spontaneously feel like cooking, a rare occurrence.) On a Rest Day, your prime directive is to just "follow your gut" for the entire day and just do "what you feel like doing" in the moment. There can be no obligations on a Rest Day. No scheduled calls or meetings. No promises to show up to a party. You can go to the party if you actually feel like going to the party, but you won't be able to know until last-minute. You cannot be "on-call" for anything. No one should depend on you unless it's someone you actively like being depended on for things, like a person you care about. There can be exceptions to these, but I like to make Rest Days "sacred"—aka protected from influences like work pressure, social pressure, pressure from society, incentive gradients created by video games and my phone, incentive gradients created by money, the pressure to be different or better, the pressure to achieve, the pressure to always be going somewhere else, the pressure to "always be closing." Rest Days are for being in the Now. The Now needs to be protected from influences from both the past (obligations) and the future (anxieties). Rest Days will actually refresh and reset you. Unfortunately, many people do not know how to take Rest Days. They instead use weekends and vacation days as Recovery Days or days where their mind is still in "working" mode. But Recovery Days alone are not sufficient for refreshing your energy levels and motivation. You risk burnout if you consistently fail to get any true Rest over a long period of time. Things my gut wants to do on Rest Days: be in the present moment meditate (in a natural, spontaneous way) eat tasty things have a picnic in a park / take walks / enjoy nature chill at a cafe I like go to a museum or an aquarium...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Taboo Your Words, published by Eliezer Yudkowsky on the LessWrong. In the game Taboo (by Hasbro), the objective is for a player to have their partner guess a word written on a card, without using that word or five additional words listed on the card. For example, you might have to get your partner to say "baseball" without using the words "sport", "bat", "hit", "pitch", "base" or of course "baseball". As soon as I see a problem like that, I at once think, "An artificial group conflict in which you use a long wooden cylinder to whack a thrown spheroid, and then run between four safe positions." It might not be the most efficient strategy to convey the word 'baseball' under the stated rules - that might be, "It's what the Yankees play" - but the general skill of blanking a word out of my mind was one I'd practiced for years, albeit with a different purpose. Yesterday we saw how replacing terms with definitions could reveal the empirical unproductivity of the classical Aristotelian syllogism. All humans are mortal (and also, apparently, featherless bipeds); Socrates is human; therefore Socrates is mortal. When we replace the word 'human' by its apparent definition, the following underlying reasoning is revealed: All [mortal, ~feathers, biped] are mortal; Socrates is a [mortal, ~feathers, biped]; Therefore Socrates is mortal. But the principle of replacing words by definitions applies much more broadly: Albert: "A tree falling in a deserted forest makes a sound." Barry: "A tree falling in a deserted forest does not make a sound." Clearly, since one says "sound" and one says "not sound", we must have a contradiction, right? But suppose that they both dereference their pointers before speaking: Albert: "A tree falling in a deserted forest matches [membership test: this event generates acoustic vibrations]." Barry: "A tree falling in a deserted forest does not match [membership test: this event generates auditory experiences]." Now there is no longer an apparent collision—all they had to do was prohibit themselves from using the word sound. If "acoustic vibrations" came into dispute, we would just play Taboo again and say "pressure waves in a material medium"; if necessary we would play Taboo again on the word "wave" and replace it with the wave equation. (Play Taboo on "auditory experience" and you get "That form of sensory processing, within the human brain, which takes as input a linear time series of frequency mixes...") But suppose, on the other hand, that Albert and Barry were to have the argument: Albert: "Socrates matches the concept [membership test: this person will die after drinking hemlock]." Barry: "Socrates matches the concept [membership test: this person will not die after drinking hemlock]." Now Albert and Barry have a substantive clash of expectations; a difference in what they anticipate seeing after Socrates drinks hemlock. But they might not notice this, if they happened to use the same word "human" for their different concepts. You get a very different picture of what people agree or disagree about, depending on whether you take a label's-eye-view (Albert says "sound" and Barry says "not sound", so they must disagree) or taking the test's-eye-view (Albert's membership test is acoustic vibrations, Barry's is auditory experience). Get together a pack of soi-disant futurists and ask them if they believe we'll have Artificial Intelligence in thirty years, and I would guess that at least half of them will say yes. If you leave it at that, they'll shake hands and congratulate themselves on their consensus. But make the term "Artificial Intelligence" taboo, and ask them to describe what they expect to see, without ever using words like "computers" or "think", and you might find quite a conflict of expectations hiding under that featureless standard word. Lik...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Sorting Pebbles Into Correct Heaps, published by Eliezer Yudkowsky on theLessWrong. Once upon a time there was a strange little species—that might have been biological, or might have been synthetic, and perhaps were only a dream—whose passion was sorting pebbles into correct heaps. They couldn't tell you why some heaps were correct, and some incorrect. But all of them agreed that the most important thing in the world was to create correct heaps, and scatter incorrect ones. Why the Pebblesorting People cared so much, is lost to this history—maybe a Fisherian runaway sexual selection, started by sheer accident a million years ago? Or maybe a strange work of sentient art, created by more powerful minds and abandoned? But it mattered so drastically to them, this sorting of pebbles, that all the Pebblesorting philosophers said in unison that pebble-heap-sorting was the very meaning of their lives: and held that the only justified reason to eat was to sort pebbles, the only justified reason to mate was to sort pebbles, the only justified reason to participate in their world economy was to efficiently sort pebbles. The Pebblesorting People all agreed on that, but they didn't always agree on which heaps were correct or incorrect. In the early days of Pebblesorting civilization, the heaps they made were mostly small, with counts like 23 or 29; they couldn't tell if larger heaps were correct or not. Three millennia ago, the Great Leader Biko made a heap of 91 pebbles and proclaimed it correct, and his legions of admiring followers made more heaps likewise. But over a handful of centuries, as the power of the Bikonians faded, an intuition began to accumulate among the smartest and most educated that a heap of 91 pebbles was incorrect. Until finally they came to know what they had done: and they scattered all the heaps of 91 pebbles. Not without flashes of regret, for some of those heaps were great works of art, but incorrect. They even scattered Biko's original heap, made of 91 precious gemstones each of a different type and color. And no civilization since has seriously doubted that a heap of 91 is incorrect. Today, in these wiser times, the size of the heaps that Pebblesorters dare attempt, has grown very much larger—which all agree would be a most great and excellent thing, if only they could ensure the heaps were really correct. Wars have been fought between countries that disagree on which heaps are correct: the Pebblesorters will never forget the Great War of 1957, fought between Y'ha-nthlei and Y'not'ha-nthlei, over heaps of size 1957. That war, which saw the first use of nuclear weapons on the Pebblesorting Planet, finally ended when the Y'not'ha-nthleian philosopher At'gra'len'ley exhibited a heap of 103 pebbles and a heap of 19 pebbles side-by-side. So persuasive was this argument that even Y'not'ha-nthlei reluctantly conceded that it was best to stop building heaps of 1957 pebbles, at least for the time being. Since the Great War of 1957, countries have been reluctant to openly endorse or condemn heaps of large size, since this leads so easily to war. Indeed, some Pebblesorting philosophers—who seem to take a tangible delight in shocking others with their cynicism—have entirely denied the existence of pebble-sorting progress; they suggest that opinions about pebbles have simply been a random walk over time, with no coherence to them, the illusion of progress created by condemning all dissimilar pasts as incorrect. The philosophers point to the disagreement over pebbles of large size, as proof that there is nothing that makes a heap of size 91 really incorrect—that it was simply fashionable to build such heaps at one point in time, and then at another point, fashionable to condemn them. "But... 13!" carries no truck with them; for to regard "13!" as a persuasive count...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Coordination Frontier: Sequence Intro, published by Raemon on the LessWrong. Sometimes, groups of humans disagree about what to do. We also sometimes disagree about how to decide what to do. Sometimes we even disagree about how to decide how to decide. Among the philosophically unsophisticated, there is a sad, frustrating way this can play out: People resolve "how to decide" with yelling, or bloodshed, or, (if you’re lucky), charismatic leaders assembling coalitions. This can leave lots of value on the table, or actively destroy value. Among the extremely philosophically sophisticated, there are different sad, frustrating ways this can play out: People have very well thought out principles informing their sense of "how to coordinate well." But, their principles are not the same, and they don’t have good meta-principles on when/how to compromise. They spend hours (or years) arguing about how to decide. Or they burn a lot of energy in conflict. Or they end up walking away from what could have been a good deal, if only people were a bit better at communicating. I’ve gone through multiple iterations on this sequence intro, some optimistic, some pessimistic. Optimistic takes include: “I think rationalists are in a rare position to actually figure out good coordination meta-principles, because we are smart, and care, and are in positions where good coordination actually matters. This is exciting, because coordination is basically the most important thing [citation needed]. Anyone with a shot at pushing humanity’s coordination theory and capacity forward should do that.” Pessimistic takes include: “Geez louise, rationalists are all philosophical contrarians with weird, extreme, self-architected psychology who are a pain to work with”, as well as “Actually, the most important facets of coordination to improve are maybe more like ‘slightly better markets’ than like ‘figuring out how to help oddly specific rationalists get along’.” I started writing this post several years ago because I was annoyed at, like, 6 particular people, many of them smarter and more competent than me, many of whom were explicitly interested in coordination theory, who nonetheless seemed to despair at coordinating with rationalists-in-particular (including each other). The post grew into a sequence. The sequence grew into a sprawling research project. My goal was “provide a good foundation to get rationalists through the Valley of Bad Coordination”. I feel like we’re so close to being able to punch above our weight at coordination and general competence. I think my actual motivations were sort of unhealthy. “If only I could think better and write really good blogposts, these particular people I’m frustrated with could get along.” I’m currently in a bit of a pessimistic swing, and do not expect that writing sufficiently good blogposts will fix the things I was originally frustrated by. The people in question (probably) have decent reasons for having different coordination strategies. Nonetheless, I think “mild irritation at something not quite working” is pretty good as motivations go. I’ve spent the past few years trying to reconcile the weirdly-specific APIs of different rationalists who each were trying to solve pretty real problems, and who had developed rich, complex worldviews along the way that point towards something important. I feel like I can almost taste the center of some deeper set of principles that unite them. Since getting invested in this, I’ve come to suspect “If you want to succeed at coordination, ‘incremental improvements on things like markets’ is more promising than ‘reconcile weird rationalist APIs.” But, frustration with weird rationalist APIs was the thing that got me on this path, and I think I’m just going to see that through to the end. So. Here is this sequence, and he...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Rational Me or We?, published by RobinHanson on the LessWrong. Martial arts can be a good training to ensure your personal security, if you assume the worst about your tools and environment. If you expect to find yourself unarmed in a dark alley, or fighting hand to hand in a war, it makes sense. But most people do a lot better at ensuring their personal security by coordinating to live in peaceful societies and neighborhoods; they pay someone else to learn martial arts. Similarly, while "survivalists" plan and train to stay warm, dry, and fed given worst case assumptions about the world around them, most people achieve these goals by participating in a modern economy. The martial arts metaphor for rationality training seems popular at this website, and most discussions here about how to believe the truth seem to assume an environmental worst case: how to figure out everything for yourself given fixed info and assuming the worst about other folks. In this context, a good rationality test is a publicly-visible personal test, applied to your personal beliefs when you are isolated from others' assistance and info. I'm much more interested in how we can can join together to believe truth, and it actually seems easier to design institutions which achieve this end than to design institutions to test individual isolated general tendencies to discern truth. For example, with subsidized prediction markets, we can each specialize on the topics where we contribute best, relying on market consensus on all other topics. We don't each need to train to identify and fix each possible kind of bias; each bias can instead have specialists who look for where that bias appears and then correct it. Perhaps martial-art-style rationality makes sense for isolated survivalist Einsteins forced by humanity's vast stunning cluelessness to single-handedly block the coming robot rampage. But for those of us who respect the opinions of enough others to want to work with them to find truth, it makes more sense to design and field institutions which give each person better incentives to update a common consensus. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Approving reinforces low-effort behaviors, published by Scott Alexander on the LessWrong. In addition to "liking" to describe pleasure and "wanting" to describe motivation, we add "approving" to describe thoughts that are ego syntonic. A heroin addict likes heroin. He certainly wants more heroin. But he may not approve of taking heroin. In fact, there are enough different cases to fill in all eight boxes of the implied 2x2x2 grid (your mileage may vary): +wanting/+liking/+approving: Romantic love. If you're doing it right, you enjoy being with your partner, you're motivated to spend time with your partner, and you think love is a wonderful (maybe even many-splendored) thing. +wanting/+liking/-approving: The aforementioned heroin addict feels good when taking heroin, is motivated to get more, but wishes he wasn't addicted. +wanting/-liking/+approving: I have taken up disc golf. I play it every day, and when events conspire to prevent me from playing it, I seethe. I approve of this pastime: I need to take up more sports, and it helps me spend time with my family. But when I am playing, all I feel is stressed and angry that I was literally that close how could I miss that shot aaaaarggghh. +wanting/-liking/-approving: The jaded addict. I have a friend who says she no longer even enjoys coffee or gets any boost from it, she just feels like she has to have it when she gets up. -wanting/+liking/+approving: Reading non-fiction. I enjoy it when I'm doing it, I think it's great because it makes me more educated, but I can rarely bring myself to do it. -wanting/-liking/+approving: Working in a soup kitchen. Unless you're the type for whom helping others is literally its own reward it's not the most fun thing in the world, nor is it the most attractive, but it makes you a Good Person and so you should do it. -wanting/+liking/-approving: The non-addict. I don't want heroin right now. I think heroin use is repugnant. But if I took some, I sure bet I'd like it. -wanting/-liking/-approving: Torture. I don't want to be tortured, I wouldn't like it if I were, and I will go on record declaring myself to be against it. Discussion of goals is mostly about approving; a goal is an ego-syntonic thought. When we speak of goals that are hard to achieve, we're usually talking about +approving/-wanting. The previous discussion of learning Swahili is one example; more noble causes like Working To Help The Less Fortunate can be others. Ego syntonicity itself is mildly reinforcing by promoting positive self-image. Most people interested in philosophy have at least once sat down and moved their arm from side to side, just to note that their mind really does control their body; the mental processes that produced curiosity about philosophy were sufficiently powerful to produce that behavior as well. Some processes, like moving one's arm, or speaking aloud, or engaging in verbal thought, are so effortless, and so empty of other reinforcement either way, that we usually expect them to be completely under the control of the mild reinforcement provided by approving of those behaviors. Other behaviors take more effort, and are subject not only to discounting but to many other forms of reinforcement. Unlike the first class of behaviors, we expect to experience akrasia when dealing with this latter sort. This offers another approach to willpower: taking low-effort approving-influenced actions that affect the harder road ahead. Consider the action of making a goal. I go to all my friends and say "Today I shall begin learning Swahili." This is easy to do. There is no chance of me intending to do so and failing; my speech is output by the same processes as my intentions, so I can "trust" it. But this is not just an output of my mental processes, but an input. One of the processes potentially reinforcing my be...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Intelligent Social Web, published by Valentine on the LessWrong. Epistemic status: Fake Framework When you walk into an improv scene, you usually have no idea what role you’re playing. All you have is some initial prompt — something like: “You three are in a garden. The scene has to involve a stuffed bear somehow. Go!” So now you’re looking to the other people there. Then someone jumps forward and adds to the scene: “Oh, there it is! I’m glad we finally found it!” Now you know a little bit about your character, and about the character of the person who spoke, but not enough to fully define anyone’s role. You can then expand the scene by adding something: “It’s about time! We’re almost late now.” Now you’ve specified more about what’s going on, who you are, and who the other players are. But it’s still the case that none of you knows what’s going on. In fact, if you think you know, you’ll often quickly be proven wrong. Maybe you imagine in that scene you’re an uptight punctual person. And then the third person in the scene says to you, “What do you care, Alex? You’re always late to everything anyway!” Surprise! Now you need to flush who you thought you were from your mind, accept the new frame, and run with it as part of your newly evolving identity. Otherwise the scene sort of crashes. It would go more smoothly if you didn’t hold any preconceptions about who you are or what’s going on. The scene tends to work better if you stay in the present moment and just jump in with the first thing that comes to mind (as long as it’s shaped by what has happened so far). Then the collection of interactions and emerging roles spontaneously guides your behavior, which in turn help guide others’ behavior, all of which recursively defines the “who” and “what” of the scene. Your job as a player isn’t to play a character; it’s to co-create a scene. We can sort of pretend that there’s a “director”: it’s the intelligence that emerges between the players via their interactions. It’s a distributed system that computes relationships and context by guiding each node in its network to act freely within constraints. From this vantage point, the network guides players, and the job of each player is to be guidable but not purely passive (since a passive node is just relaying information rather than aiding in the computation). As long as everyone involved is plugged into and responsive to this network, the scene will usually play out well. I suspect that improv works because we’re doing something a lot like it pretty much all the time. The web of social relationships we’re embedded in helps define our roles as it forms and includes us. And that same web, as the distributed “director” of the “scene”, guides us in what we do. A lot of (but not all) people get a strong hit of this when they go back to visit their family. If you move away and then make new friends and sort of become a new person (!), you might at first think this is just who you are now. But then you visit your parents. and suddenly you feel and act a lot like you did before you moved away. You might even try to hold onto this “new you” with them. and they might respond to what they see as strange behavior by trying to nudge you into acting “normal”: ignoring surprising things you say, changing the topic to something familiar, starting an old fight, etc. In most cases, I don’t think this is malice. It’s just that they need the scene to work. They don’t know how to interact with this “new you”, so they tug on their connection with you to pull you back into a role they recognize. If that fails, then they have to redefine who they are in relation to you — which often (but not always) happens eventually. I’m basically taking as an axiom of this framework that people need the “scene” to work — which is to say, they need to be able to ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: An Untrollable Mathematician IllustratedΩ, published by abramdemski on the LessWrong. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. The following was a presentation I made for Sören Elverlin's AI Safety Reading Group. I decided to draw everything by hand because powerpoint is boring. Thanks to Ben Pace for formatting it for LW! See also the IAF post detailing the research which this presentation is based on. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: New York Times, Please Do Not Threaten The Safety of Scott Alexander By Revealing His True Name, published by Zvi on the LessWrong. Write a Review In reaction to (Now the entirety of SlateStarCodex): NYT Is Threatening My Safety By Revealing My Real Name, So I Am Deleting The Blog I have sent the following to New York Times technology editor Pui-Wing Tam, whose email is pui-wing.tam@nytimes.com: My name is Zvi Mowshowitz. I am a friend of Scott Alexander. I grew up with The New York Times as my central source of news and greatly value that tradition. Your paper has declared that you intend to publish, in The New York Times, the true name of Scott Alexander. Please reconsider this deeply harmful and unnecessary action. If Scott’s name were well-known, it would likely make it more difficult or even impossible for him to make a living as a psychiatrist, which he has devoted many years of his life to being able to do. He has received death threats, and would likely not feel safe enough to continue living with other people he cares about. This may well ruin his life. At a minimum, and most importantly for the world, it has already taken down his blog. In addition to this massive direct loss, those who know what happened will know that this happened as a direct result of the irresponsible actions of The New York Times. The bulk of the best bloggers and content creators on the internet read Scott’s blog, and this will create large-scale permanent hostility to reporters in general and the Times in particular across the board. I do not understand what purpose this revelation is intended to serve. What benefit does the public get from this information? This is not news that is fit to print. If, as your reporter who has this intention claims, you believe that Scott provides a valuable resource that enhances the quality of our discourse, scientific understanding and lives, please reverse this decision before it is too late. If you don’t believe this, I still urge you to reconsider your decision in light of its other likely consequences. We should hope it is not too late to fix this. I will be publishing this email as an open letter. Regards, Zvi Mowshowitz PS for internet: If you wish to help, here is Scott’s word on how to help: There is no comments section for this post. The appropriate comments section is the feedback page of the New York Times. You may also want to email the New York Times technology editor Pui-Wing Tam at pui-wing.tam@nytimes.com, contact her on Twitter at @puiwingtam, or phone the New York Times at 844-NYTNEWS. (please be polite – I don’t know if Ms. Tam was personally involved in this decision, and whoever is stuck answering feedback forms definitely wasn’t. Remember that you are representing me and the SSC community, and I will be very sad if you are a jerk to anybody. Please just explain the situation and ask them to stop doxxing random bloggers for clicks. If you are some sort of important tech person who the New York Times technology section might want to maintain good relations with, mention that.) If you are a journalist who is willing to respect my desire for pseudonymity, I’m interested in talking to you about this situation (though I prefer communicating through text, not phone). My email is scott@slatestarcodex.com. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Philosophical Landmines, published by Philosophical Landmines on the LessWrong. Related: Cached Thoughts Last summer I was talking to my sister about something. I don't remember the details, but I invoked the concept of "truth", or "reality" or some such. She immediately spit out a cached reply along the lines of "But how can you really say what's true?". Of course I'd learned some great replies to that sort of question right here on LW, so I did my best to sort her out, but everything I said invoked more confused slogans and cached thoughts. I realized the battle was lost. Worse, I realized she'd stopped thinking. Later, I realized I'd stopped thinking too. I went away and formulated the concept of a "Philosophical Landmine". I used to occasionally remark that if you care about what happens, you should think about what will happen as a result of possible actions. This is basically a slam dunk in everyday practical rationality, except that I would sometimes describe it as "consequentialism". The predictable consequence of this sort of statement is that someone starts going off about hospitals and terrorists and organs and moral philosophy and consent and rights and so on. This may be controversial, but I would say that causing this tangent constitutes a failure to communicate the point. Instead of prompting someone to think, I invoked some irrelevant philosophical cruft. The discussion is now about Consequentialism, the Capitalized Moral Theory, instead of the simple idea of thinking through consequences as an everyday heuristic. It's not even that my statement relied on a misused term or something; it's that an unimportant choice of terminology dragged the whole conversation in an irrelevant and useless direction. That is, "consequentialism" was a Philosophical Landmine. In the course of normal conversation, you passed through an ordinary spot that happened to conceal the dangerous leftovers of past memetic wars. As a result, an intelligent and reasonable human was reduced to a mindless zombie chanting prerecorded slogans. If you're lucky, that's all. If not, you start chanting counter-slogans and the whole thing goes supercritical. It's usually not so bad, and no one is literally "chanting slogans". There may even be some original phrasings involved. But the conversation has been derailed. So how do these "philosophical landmine" things work? It looks like when a lot has been said on a confusing topic, usually something in philosophy, there is a large complex of slogans and counter-slogans installed as cached thoughts around it. Certain words or concepts will trigger these cached thoughts, and any attempt to mitigate the damage will trigger more of them. Of course they will also trigger cached thoughts in other people, which in turn... The result being that the conversation rapidly diverges from the original point to some useless yet heavily discussed attractor. Notice that whether a particular concept will cause trouble depends on the person as well as the concept. Notice further that this implies that the probability of hitting a landmine scales with the number of people involved and the topic-breadth of the conversation. Anyone who hangs out on 4chan can confirm that this is the approximate shape of most thread derailments. Most concepts in philosophy and metaphysics are landmines for many people. The phenomenon also occurs in politics and other tribal/ideological disputes. The ones I'm particularly interested in are the ones in philosophy, but it might be useful to divorce the concept of "conceptual landmines" from philosophy in particular. Here's some common ones in philosophy: Morality Consequentialism Truth Reality Consciousness Rationality Quantum Landmines in a topic make it really hard to discuss ideas or do work in these fields, because chances are, some...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Doing your good deed for the day, published by Scott Alexander on the LessWrong. Interesting new study out on moral behavior. The one sentence summary of the most interesting part is that people who did one good deed were less likely to do another good deed in the near future. They had, quite literally, done their good deed for the day. In the first part of the study, they showed that people exposed to environmentally friendly, "green" products were more likely to behave nicely. Subjects were asked to rate products in an online store; unbeknownst to them, half were in a condition where the products were environmentally friendly, and the other half in a condition where the products were not. Then they played a Dictator Game. Subjects who had seen environmentally friendly products shared more of their money. In the second part, instead of just rating the products, they were told to select $25 worth of products to buy from the store. One in twenty five subjects would actually receive the products they'd purchased. Then they, too, played the Dictator Game. Subjects who had bought environmentally friendly products shared less of their money. In the third part, subjects bought products as before. Then, they participated in a "separate, completely unrelated" experiment "on perception" in which they earned money by identifying dot patterns. The experiment was designed such that participants could lie about their perceptions to earn more. People who purchased the green products were more likely to do so. This does not prove that environmentalists are actually bad people - remember that whether a subject purchased green products or normal products was completely randomized. It does suggest that people who have done one nice thing feel less of an obligation to do another. This meshes nicely with a self-signalling conception of morality. If part of the point of behaving morally is to convince yourself that you're a good person, then once you're convinced, behaving morally loses a lot of its value. By coincidence, a few days after reading this study, I found this article by Dr. Beck, a theologian, complaining about the behavior of churchgoers on Sunday afternoon lunches. He says that in his circles, it's well known that people having lunch after church tend to abuse the waitstaff and tip poorly. And he blames the same mechanism identified by Mazar and Zhong in their Dictator Game. He says that, having proven to their own satisfaction that they are godly and holy people, doing something else godly and holy like being nice to others would be overkill. It sounds...strangely plausible. If this is true, then anything that makes people feel moral without actually doing good is no longer a harmless distraction. All those biases that lead people to give time and money and thought to causes that don't really merit them waste not only time and money, but an exhaustible supply of moral fiber (compare to Baumeister's idea of willpower as a limited resource). People here probably don't have to worry about church. But some of the other activities Dr. Beck mentions as morality sinkholes seem appropriate, with a few of the words changed: Bible study Voting Republican Going on spiritual retreats Reading religious books Arguing with evolutionists Sending your child to a Christian school or providing education at home Using religious language Avoiding R-rated movies Not reading Harry Potter. Let's not get too carried away with the evils of spiritual behavior - after all, data do show that religious people still give more to non-religious charities than the nonreligious do. But the points in and of themselves are valid. I've seen Michael Keenan and Patri Friedman say exactly the same thing regarding voting, and I would add to the less religion-o-centric list: Joining "1000000 STRONG AGAINST WORLD HU...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some Thoughts on My Psychiatry Practice, published by Laura B on the LessWrong. I’ve noticed a marked change in my clientele after going into private practice.[1] Of course I expected class differences-- I charge full fee and don’t take insurance. But there are differences that are not as predictable as ‘has more money’. During residency I worked at a hospital Medicaid clinic and saw mostly poor, often chronically unemployed people. While monetary problems were a source of stress, they were not nearly as present in people’s minds as someone from a middle-class upbringing might think. These people were used to going without. They were not trying to get more. The types of things they talked about were family problems, health problems, and trauma. So much trauma. People’s ego-identity crises centered less on their accomplishments and more on their relationships. The patients I see now are mostly highly successful, highly educated, weathly people, most of whom care a lot about their careers. Their ego-identity crises center around their work and their position in life relative to others. There is a lot of concern about ‘the path’. ‘Did I go down the right path?’ ‘Did I make a wrong turn?’ There seems to be a great fear of making or having made a wrong decision, which can paralyze their ability to make future decisions. While this group is not without trauma, it is not what they wish to focus on. They will often be dismissive of its effects on them, noting that they clearly got over it in order to get where they are now. Which is, you know, in my office. Many of my new patients do NOT want to take medication. This is a large change from my patients at the Medicaid clinic who were always requesting more and different pills. And this difference is not because my new patients are less unhappy. They describe intense misery, even a wish to die, going on for months if not years, and yet they struggle through each day in their sisyphean ordeal. They ‘power through’ until they can’t. Until something gives. Then they come to me. I can think of several good reasons to have concerns about using medication. What are the long-term effects? Could this change my identity? What if this makes me ok with a shitty situation and then I don’t fix an underlying problem? But these are not the typical concerns I hear raised. What most of my patients say is that they don’t want to ‘rely’ on a medication. They don’t want to be the type of person who takes it. ‘That would mean there is something wrong with my brain.’ Even though they are clearly very depressed, clearly suffering and hating every day, so long as they can push through without taking a pill they must be ‘ok’ in some sense. Taking the pill would confirm there is actually something wrong. Taking the pill might mean they are more similar to the patients at the Medicaid clinic than they want to consider. What struck me about this was how people’s desires to assume a certain identity – that of someone who didn’t take medication – was more important to them than their actual lived experience. ‘This is your life.’ And this is broader than to take or not take medication. People will suffer through horrible work situations in order to be the type or person who has that job. ‘If your job makes you want to kill yourself, shouldn’t you consider quitting it before killing yourself?’ ‘But I’m good at it.’ Identity seems to be everything. Experience is there to tell you if you’re on the right way to assuming the proper identity. If you go through the motions properly you can look the part. What’s the difference between looking the part and being the person anyway? Now refusing medication would be one thing if they wanted to come for weekly therapy and talk through their problems. But many don’t. They complain they don’t have the time (and it’s time,...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Radical Probabilism, published by abramdemski on the LessWrong. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.This is an expanded version of my talk. I assume a high degree of familiarity with Bayesian probability theory. Toward a New Technical Explanation of Technical Explanation -- an attempt to convey the practical implications of logical induction -- was one of my most-appreciated posts, but I don't really get the feeling that very many people have received the update. Granted, that post was speculative, sketching what a new technical explanation of technical explanation might look like. I think I can do a bit better now.If the implied project of that post had really been completed, I would expect new practical probabilistic reasoning tools, explicitly violating Bayes' law. For example, we might expect: A new version of information theory. An update to the "prediction=compression" maxim, either repairing it to incorporate the new cases, or explicitly denying it and providing a good intuitive account of why it was wrong. A new account of concepts such as mutual information, allowing for the fact that variables have behavior over thinking time; for example, variables may initially be very correlated, but lose correlation as our picture of each variable becomes more detailed. New ways of thinking about epistemology. One thing that my post did manage to do was to spell out the importance of "making advanced predictions", a facet of epistemology which Bayesian thinking does not do justice to. However, I left aspects of the problem of old evidence open, rather than giving a complete way to think about it. New probabilistic structures. Bayesian Networks are one really nice way to capture the structure of probability distributions, making them much easier to reason about. Is there anything similar for the new, wider space of probabilistic reasoning which has been opened up? Unfortunately, I still don't have any of those things to offer. The aim of this post is more humble. I think what I originally wrote was too ambitious for didactic purposes. Where the previous post aimed to communicate the insights of logical induction by sketching broad implications, I here aim to communicate the insights in themselves, focusing on the detailed differences between classical Bayesian reasoning and the new space of ways to reason. Rather than talking about logical induction directly, I'm mainly going to explain things in terms of a very similar philosophy which Richard Jeffrey invented -- apparently starting with his phd dissertation in the 50s, although I'm unable to get my hands on it or other early references to see how fleshed-out the view was at that point. He called this philosophy radical probabilism. Unlike logical induction, radical probabilism appears not to have any roots in worries about logical uncertainty or bounded rationality. Instead it appears to be motivated simply by a desire to generalize, and a refusal to accept unjustified assumptions. Nonetheless, it carries most of the same insights. Radical Probabilism has not been very concerned with computational issues, and so constructing an actual algorithm (like the logical induction algorithm) has not been a focus. (However, there have been some developments -- see historical notes at the end.) This could be seen as a weakness. However, for the purpose of communicating the core insights, I think this is a strength -- there are fewer technical details to communicate. A terminological note: I will use "radical probabilism" to refer to the new theory of rationality (treating logical induction as merely a specific way to flesh out Jeffrey's theory). I'm more conflicted about how to refer to the older theory. I'm tempted to just use the term "Bayesian", implying that the new theory is ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The rationalist community's location problem, published by mingyuan on the LessWrong. The Problem Basically ever since the first rationalists settled in Berkeley, people have been saying, “Why do you live in Berkeley, Berkeley sucks! You should all move to Location X instead, it’s so much better.” The problem has always been that no one agrees on what Location X is. Some common candidates for Location X: A smaller, cheaper, friendlier US city NYC Australia Canada Somewhere with very low cost of living (often in Southeast Asia or Latin America) London Oxford Blackpool Prague A castle A private island and of course Wherever the speaker is from In the past I've brushed off all such suggestions, because it was just too hard a coordination problem to get multiple hundreds of rationalists to leave Berkeley, where they've gotten jobs, rented or even bought houses, established organizations, enrolled in schools, and established social circles. But we're in a unique time! Due to the pandemic, there's far less reason to stay in any one place - work and school are remote, expensive leases can be terminated, and you can't see your friends anyway. Most of the rationalist houses I know have moved or dissolved, and the former Berkeley rationalists are flung across all corners of the globe (yeah I know globes don't have corners). A fair number of us have stayed, but I think for most of us it's just because our friends are here, we're hoping that someday the rest of our friends come back, and we're not sure where else to go. So, if ever there were a time when we actually had the chance to move the physical locus of the rationalist community, it's now. Below, I'll lay out what I believe to be some of the most important general considerations for deciding on a new location. I encourage people to make their case for a specific location, either in comments or in their own posts. (Looking at you, Mikk!) Considerations for Location X Potential dealbreakers Visas In order to settle in a location, you have to be able to legally live there long-term. Most Berkeley rationalists are US citizens, and those who aren't have already paid the steep cost of acquiring US visas and learning US immigration law. This feels like a strong argument in favor of staying in the US somewhere, although it's possible there are places where this wouldn't actually be that much of an issue. In any case, it's certainly an argument against countries with strict immigration laws, like Switzerland. Relatedly, organizations such as MIRI, CFAR, Open Phil, BERI, etc are registered in the US. I don't know how hard it would be for them to operate elsewhere and am unfamiliar with this domain in general. Language Given that basically all rationalists speak English (since it's pretty hard to read the relevant material otherwise), we should settle somewhere English-speaking; it would be very costly if everyone had to deal with a language barrier every single day (or learn a new language). Notably this doesn't automatically disqualify all locations in e.g. continental Europe - Habryka points out that you can get along just fine in Berlin if you only know English. But somewhere like e.g. Japan looks like a much worse prospect on this metric. National political environment / culture The rationality community often attracts controversy, so it's important that we settle somewhere that protects freedom of thought and speech, and is generally friendly to weird ideas. We should definitely not move somewhere where political dissidents can be abducted willy nilly. Some people are worried about unrest in the US, which might be reasonable, but on that metric it's still better to live here than, say, Mali or Afghanistan. Local political environment / culture Same basic considerations as the above. California may be an increasingly hostile en...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What's So Bad About Ad-Hoc Mathematical Definitions?, published by johnswentworth on the LessWrong. Suppose it’s the early twentieth century, and we’re trying to quantify the concept of “information”. Specifically, we want to measure “how much information” one variable contains about another - for instance, how much information a noisy measurement of the temperature of an engine contains about the actual engine temperature. Along comes Karl Pearson, and suggests using his “correlation coefficient” (specifically the square of the correlation coefficient, ρ X Y 2 . As a measure of information, this has some sensible properties: If there’s no information, then ρ X Y 2 is zero. If ρ X Y 2 is one, then there’s perfect information - one variable tells us everything there is to know about the other. It’s symmetric: the amount of information which X tells us about Y equals the amount of information which Y tells us about X. As an added bonus, it’s mathematically simple to calculate, estimate, and manipulate. Sure, it’s not very “principled”, but it seems like a good-enough measure to work with. Karl Pearson. He'd make a solid movie villain; I get sort of a Tywin Lannister vibe. Now an engineer from Bell Telephone shows up with a real-world problem: they’ve been contracted to create secure communications for the military. They want to ensure that externally-visible data Y contains no information about secret message X, so they need a way to measure “how much information” one variable contains about another. What a perfect use-case! We advise them to design their system so that X and Y have zero correlation. A few years later, Bell Telephone gets a visit from a very unhappy colonel. Apparently the enemy has been reading their messages. Zero correlation was not enough to keep the secret messages secret. Now, Bell could patch over this problem. For instance, they could pick a bunch of functions like X 2 sin Y e X 2 X − 1 , etc, and require that those also be uncorrelated. With enough functions, and a wide enough variety, that might be enough. but it’s going to get very complicated very quickly, with all these new design constraints piling up. Fortunately, off in a corner of Bell Labs, one of their researchers already has an alternative solution. Claude Shannon suggests quantifying “how much information” X contains about Y using his “mutual information” metric I X Y . This has a bunch of sensible properties, but the main argument is that I X Y is exactly the difference between the average number of bits one needs to send in a message in order to communicate the value of X, and the average number of bits one needs to send to communicate X if the receiving party already knows Y. It’s the number of bits “savable” by knowing Y. By imagining different things as the “message” and thinking about how hard it is to guess X after knowing Y, we can intuitively predict that this metric will apply to lots of different situations, including Bell’s secret message problem. Claude Shannon. Note the electronics in the background; this guy is my kind of theorist. No ivory tower for him. Shannon advises the engineers to design their system so that X and Y have zero mutual information. And now, the enemy can’t read their messages quite so easily. Proxies vs Definitions In this story, what does the correlation coefficient do “wrong” which mutual information does “right”? What’s the generalizable lesson here? The immediate difference is that correlation is a proxy for amount of information, while mutual information is a true definition/metric. When we apply optimization pressure to a proxy, it breaks down - that’s Goodheart’s Law. In this case, the optimization pressure is a literal adversary trying to read our secret messages. The optimizer finds the corner cases where our proxy no longer perfectly ca...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Mysterious Answers to Mysterious Questions, published by Eliezer Yudkowsky on the LessWrong. Imagine looking at your hand, and knowing nothing of cells, nothing of biochemistry, nothing of DNA. You’ve learned some anatomy from dissection, so you know your hand contains muscles; but you don’t know why muscles move instead of lying there like clay. Your hand is just . . . stuff . . . and for some reason it moves under your direction. Is this not magic? It seemed to me then, and it still seems to me, most probable that the animal body does not act as a thermodynamic engine . . . The influence of animal or vegetable life on matter is infinitely beyond the range of any scientific inquiry hitherto entered on. Its power of directing the motions of moving particles, in the demonstrated daily miracle of our human free-will, and in the growth of generation after generation of plants from a single seed, are infinitely different from any possible result of the fortuitous concourse of atoms[.]1 [C]onsciousness teaches every individual that they are, to some extent, subject to the direction of his will. It appears, therefore, that animated creatures have the power of immediately applying, to certain moving particles of matter within their bodies, forces by which the motions of these particles are directed to produce desired mechanical effects.2 Modern biologists are coming once more to a firm acceptance of something beyond mere gravitational, chemical, and physical forces; and that unknown thing is a vital principle.3 Lord Kelvin This was the theory of vitalism ; that the mysterious difference between living matter and non-living matter was explained by an Élan vital or vis vitalis. Élan vital infused living matter and caused it to move as consciously directed. Élan vital participated in chemical transformations which no mere non-living particles could undergo—Wöhler’s later synthesis of urea, a component of urine, was a major blow to the vitalistic theory because it showed that mere chemistry could duplicate a product of biology. Calling “Élan vital” an explanation, even a fake explanation like phlogiston, is probably giving it too much credit. It functioned primarily as a curiosity-stopper. You said “Why?” and the answer was “Élan vital!” When you say “Élan vital!” it feels like you know why your hand moves. You have a little causal diagram in your head that says: But actually you know nothing you didn’t know before. You don’t know, say, whether your hand will generate heat or absorb heat, unless you have observed the fact already; if not, you won’t be able to predict it in advance. Your curiosity feels sated, but it hasn’t been fed. Since you can say “Why? Élan vital!” to any possible observation, it is equally good at explaining all outcomes, a disguised hypothesis of maximum entropy, et cetera. But the greater lesson lies in the vitalists’ reverence for the Élan vital, their eagerness to pronounce it a mystery beyond all science. Meeting the great dragon Unknown, the vitalists did not draw their swords to do battle, but bowed their necks in submission. They took pride in their ignorance, made biology into a sacred mystery, and thereby became loath to relinquish their ignorance when evidence came knocking. The Secret of Life was infinitely beyond the reach of science! Not just a little beyond, mind you, but infinitely beyond! Lord Kelvin sure did get a tremendous emotional kick out of not knowing something. But ignorance exists in the map, not in the territory. If I am ignorant about a phenomenon, that is a fact about my own state of mind, not a fact about the phenomenon itself. A phenomenon can seem mysterious to some particular person. There are no phenomena which are mysterious of themselves. To worship a phenomenon because it seems so wonderfully mysterious is to worship y...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 16 types of useful predictions, published by Julia_Galef on the LessWrong. How often do you make predictions (either about future events, or about information that you don't yet have)? If you're a regular Less Wrong reader you're probably familiar with the idea that you should make your beliefs pay rent by saying, "Here's what I expect to see if my belief is correct, and here's how confident I am," and that you should then update your beliefs accordingly, depending on how your predictions turn out. And yet. my impression is that few of us actually make predictions on a regular basis. Certainly, for me, there has always been a gap between how useful I think predictions are, in theory, and how often I make them. I don't think this is just laziness. I think it's simply not a trivial task to find predictions to make that will help you improve your models of a domain you care about. At this point I should clarify that there are two main goals predictions can help with: Improved Calibration (e.g., realizing that I'm only correct about Domain X 70% of the time, not 90% of the time as I had mistakenly thought). Improved Accuracy (e.g., going from being correct in Domain X 70% of the time to being correct 90% of the time) If your goal is just to become better calibrated in general, it doesn't much matter what kinds of predictions you make. So calibration exercises typically grab questions with easily obtainable answers, like "How tall is Mount Everest?" or "Will Don Draper die before the end of Mad Men?" See, for example, the Credence Game, Prediction Book, and this recent post. And calibration training really does work. But even though making predictions about trivia will improve my general calibration skill, it won't help me improve my models of the world. That is, it won't help me become more accurate, at least not in any domains I care about. If I answer a lot of questions about the heights of mountains, I might become more accurate about that topic, but that's not very helpful to me. So I think the difficulty in prediction-making is this: The set {questions whose answers you can easily look up, or otherwise obtain} is a small subset of all possible questions. And the set {questions whose answers I care about} is also a small subset of all possible questions. And the intersection between those two subsets is much smaller still, and not easily identifiable. As a result, prediction-making tends to seem too effortful, or not fruitful enough to justify the effort it requires. But the intersection's not empty. It just requires some strategic thought to determine which answerable questions have some bearing on issues you care about, or -- approaching the problem from the opposite direction -- how to take issues you care about and turn them into answerable questions. I've been making a concerted effort to hunt for members of that intersection. Here are 16 types of predictions that I personally use to improve my judgment on issues I care about. (I'm sure there are plenty more, though, and hope you'll share your own as well.) Predict how long a task will take you. This one's a given, considering how common and impactful the planning fallacy is. Examples: "How long will it take to write this blog post?" "How long until our company's profitable?" Predict how you'll feel in an upcoming situation. Affective forecasting – our ability to predict how we'll feel – has some well known flaws. Examples: "How much will I enjoy this party?" "Will I feel better if I leave the house?" "If I don't get this job, will I still feel bad about it two weeks later?" Predict your performance on a task or goal. One thing this helps me notice is when I've been trying the same kind of approach repeatedly without success. Even just the act of making the prediction can spark the realization that I need a bett...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Shoulder Advisors 101, published by Duncan_Sabien on the LessWrong. Motivation for post: As a former CFAR instructor, longtime teacher, and rationality pundit, I find myself giving lots of advice in lots of different contexts. I also try to check in from time to time to find out which bits of advice actually proved helpful to people. Over the years, I've heard from a genuinely surprising number of people that my (offhand, very basic, not especially insightful) thoughts on "shoulder advisors" were quite useful to them, and remained useful over time. So: a primer. "There's a copy of me inside your head?" Hermione asked. "Of course there is!" Harry said. The boy suddenly looked a bit more vulnerable. "You mean there isn't a copy of me living in your head?" There was, she realized; and not only that, it talked in Harry's exact voice. "It's rather unnerving now that I think about it," said Hermione. "I do have a copy of you living in my head. It's talking to me right now using your voice, arguing how this is perfectly normal." "Good," Harry said seriously. "I mean, I don't see how people could be friends without that." The term "shoulder advisor" comes from the cartoon trope of a character attempting to make a decision while a tiny angel whispers in one ear and a tiny devil whispers in the other. Many people have multiple shoulder advisors. Some, no doubt, carry a literal metaphorical angel and devil around with them. Others may sometimes hear the whispers of some of their favorite beloved fictional characters. It's quite common in my experience for people to have shoulder copies of their parents, or their best friends, or their romantic partners, or particularly impactful teachers or bosses or mentors. This is not schizophrenia (though for all I know it may use some of the same hardware, or may be a low-key, non-pathological version of schizophrenia in the same way that a healthy self-preservation instinct could be thought of as a low-key, non-pathological version of a phobia or an anxiety disorder). Rather, there is simply some kind of subroutine in the brain of most humans that is capable of taking in training data and learning what a given person (or character, or archetype) would say, in a given situation. It's predictive software, likely evolved in response to the need to model other chimps in the ancestral environment, and strongly selected for due to the fact that being able to model those other chimps accurately generally paid off big. It's important to be clear that the experience of "hearing the voices" actually happens, in many people. This is not a metaphor, and it is not hyperbole or exaggeration. I'm not saying that people tend to hallucinate actual sounds—that probably would be schizophrenia. But in the same way that most people "hear" their own thoughts, people also "hear" the voice of their dad (or "see" his facial expression), offering thoughts or advice or reacting in real time to the current situation. "I was going to complain about having to type with my thumbs to text you, and how I'd rather just use email or Slack, but my shoulder Malo popped up to say 'Duncan, you have a Mac. Just use Messages with your keyboard.'" "My mental copy of Jack is currently freaking out a bit about how toxic and unhealthy this sounds." "I notice my inner Nate is betting this project will fail." "I can hear my mom reminding me to take jam tarts when jam tarts are offered." (Note that you don't need to "demand" that your advisor communicate in words! Often it's both easier and also just as useful to simply let them be present—to "see" their facial expressions and body language, imagine their nonverbal reactions, let yourself be aware of and attentive to them in the same way that you (likely) are aware of or attentive to other actual humans in the same room as you. Think o...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Strong Evidence is Common, published by Mark Xu on the LessWrong. This is a linkpost for Portions of this are taken directly from Three Things I've Learned About Bayes' Rule. One time, someone asked me what my name was. I said, “Mark Xu.” Afterward, they probably believed my name was “Mark Xu.” I’m guessing they would have happily accepted a bet at 20:1 odds that my driver’s license would say “Mark Xu” on it. The prior odds that someone’s name is “Mark Xu” are generously 1:1,000,000. Posterior odds of 20:1 implies that the odds ratio of me saying “Mark Xu” is 20,000,000:1, or roughly 24 bits of evidence. That’s a lot of evidence. Seeing a Wikipedia page say “X is the capital of Y” is tremendous evidence that X is the capital of Y. Someone telling you “I can juggle” is massive evidence that they can juggle. Putting an expression into Mathematica and getting Z is enormous evidence that the expression evaluates to Z. Vast odds ratios lurk behind many encounters. One implication of the Efficient Market Hypothesis (EMH) is that is it difficult to make money on the stock market. Generously, maybe only the top 1% of traders will be profitable. How difficult is it to get into the top 1% of traders? To be 50% sure you're in the top 1%, you only need 200:1 evidence. This seemingly large odds ratio might be easy to get. On average, people are overconfident, but 12% aren't. It only takes 50:1 evidence to conclude you are much less overconfident than average. An hour or so of calibration training and the resulting calibration plots might be enough. Running through Bayes’ Rule explicitly might produce a bias towards middling values. Extraordinary claims require extraordinary evidence, but extraordinary evidence might be more common than you think. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What Multipolar Failure Looks Like, and Robust Agent-Agnostic Processes (RAAPs)Ω, published by Andrew_Critch on the LessWrong. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. With: Thomas Krendl Gilbert, who provided comments, interdisciplinary feedback, and input on the RAAP concept. Thanks also for comments from Ramana Kumar. Target audience: researchers and institutions who think about existential risk from artificial intelligence, especially AI researchers. Preceded by: Some AI research areas and their relevance to existential safety, which emphasized the value of thinking about multi-stakeholder/multi-agent social applications, but without concrete extinction scenarios. This post tells a few different stories in which humanity dies out as a result of AI technology, but where no single source of human or automated agency is the cause. Scenarios with multiple AI-enabled superpowers are often called “multipolar” scenarios in AI futurology jargon, as opposed to “unipolar” scenarios with just one superpower. Unipolar take-offs Multipolar take-offs Slow take-offs Part 1 of this post Fast take-offs Part 2 of this post Part 1 covers a batch of stories that play out slowly (“slow take-offs”), and Part 2 stories play out quickly. However, in the end I don’t want you to be super focused how fast the technology is taking off. Instead, I’d like you to focus on multi-agent processes with a robust tendency to play out irrespective of which agents execute which steps in the process. I’ll call such processes Robust Agent-Agnostic Processes (RAAPs). A group walking toward a restaurant is a nice example of a RAAP, because it exhibits: Robustness: If you temporarily distract one of the walkers to wander off, the rest of the group will keep heading toward the restaurant, and the distracted member will take steps to rejoin the group. Agent-agnosticism: Who’s at the front or back of the group might vary considerably during the walk. People at the front will tend to take more responsibility for knowing and choosing what path to take, and people at the back will tend to just follow. Thus, the execution of roles (“leader”, “follower”) is somewhat agnostic as to which agents execute them. Interestingly, if all you want to do is get one person in the group not to go to the restaurant, sometimes it’s actually easier to achieve that by convincing the entire group not to go there than by convincing just that one person. This example could be extended to lots of situations in which agents have settled on a fragile consensus for action, in which it is strategically easier to motivate a new interpretation of the prior consensus than to pressure one agent to deviate from it. I think a similar fact may be true about some agent-agnostic processes leading to AI x-risk, in that agent-specific interventions (e.g., aligning or shutting down this or that AI system or company) will not be enough to avert the process, and might even be harder than trying to shift the structure of society as a whole. Moreover, I believe this is true in both “slow take-off” and “fast take-off” AI development scenarios This is because RAAPs can arise irrespective of the speed of the underlying “host” agents. RAAPs are made more or less likely to arise based on the “structure” of a given interaction. As such, the problem of avoiding the emergence of unsafe RAAPs, or ensuring the emergence of safe ones, is a problem of mechanism design (wiki/Mechanism_design). I recently learned that in sociology, the concept of a field (martin2003field, fligsteinmcadam2012fields) is roughly defined as a social space or arena in which the motivation and behavior of agents are explained through reference to surrounding processes or “structure” rather than freedom or chance. ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Prisoner's Dilemma Tournament Results, published by prase on the LessWrong. About two weeks ago I announced an open competition for LessWrong readers inspired by Robert Axelrod's famous tournaments. The competitors had to submit a strategy which would play an iterated prisoner's dilemma of fixed length: first in the round-robin tournament where the strategy plays a hundred-turn match against each of its competitors exactly once, and second in the evolutionary tournament where the strategies are randomly paired against each other and their gain is translated in number of their copies present in next generation; the strategy with the highest number of copies after generation 100 wins. More details about the rules were described in the announcement. This post summarises the results. The Zoo of Strategies I have received 25 contest entries containing 21 distinct strategies. Those I have divided into six classes based on superficial similarities (except the last class, which is a catch-all category for everything which doesn't belong anywhere else, something like adverbs within the classification of parts of speech or now defunct vermes in the animal kingdom). The first class is formed by Tit-for-tat variants, probably the most obvious choice for a potentially successful strategy. Apparently so obvious that at least one commenter declared high confidence that tit-for-tat will make more than half of the strategy pool. That was actually a good example of misplaced confidence, since the number of received tit-for-tat variants (where I put anything which behaves like tit-for-tat except for isolated deviations) was only six, two of them being identical and thus counted as one. Moreover there wasn't a single true tit-for-tatter among the contestants; the closest we got was A (-, -): On the first turn of each match, cooperate. On every other turn, with probability 0.0000004839, cooperate; otherwise play the move that the opponent played on the immediately preceding turn. (In the presentation of strategies, the letter in bold serves as a unique identificator. The following parentheses include the name of the strategy — if the author has provided one — and the name of the author. I use the author's original description of the strategy when possible. If that's too long, an abbreviated paraphrase is given. If I found the original description ambiguous, I may give a slightly reformulated version based on subsequent clarifications with the author.) The author of A was the only one who requested his/her name should be withheld and the strategy is nameless, so both arguments in the bracket are empty. The reason for the obscure probability was to make the strategy unique. The author says: I wish to enter a trivial variation on the tit-for-tat strategy. (The trivial variation is to force the strategy to be unique; I wish to punish defectorish strategies by having lots of tit-for-tat-style strategies in the pool.) This was perhaps a slight abuse of rules, but since I am responsible for failing to make the rules immune to abuse, I had to accept the strategy as it is. Anyway, it turned out that the trivial variation was needless for the stated purpose. The remaining strategies from this class were more or less standard with B being the most obvious choice. B (-, Alexei): Tit-for-Tat, but always defect on last turn. C (-, Caerbannog): Tit-for-tat with 20% chance of forgiving after opponent's defection. Defect on the last turn. D (-, fubarobfusco and DuncanS): Tit-for-tat with 10% chance of forgiving. E (-, Jem): First two turns cooperate. Later tit-for-tat with chance of forgiving equal to 1/2x where x is equal to number of opponent's defections after own cooperations. Last turn defect. The next category of strategies I call Avengers. The Avengers play a nice strategy until the opponent's def...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Your Strength as a Rationalist, published by Eliezer Yudkowsky on the LessWrong. The following happened to me in an IRC chatroom, long enough ago that I was still hanging around in IRC chatrooms. Time has fuzzed the memory and my report may be imprecise. So there I was, in an IRC chatroom, when someone reports that a friend of his needs medical advice. His friend says that he’s been having sudden chest pains, so he called an ambulance, and the ambulance showed up, but the paramedics told him it was nothing, and left, and now the chest pains are getting worse. What should his friend do? I was confused by this story. I remembered reading about homeless people in New York who would call ambulances just to be taken someplace warm, and how the paramedics always had to take them to the emergency room, even on the 27th iteration. Because if they didn’t, the ambulance company could be sued for lots and lots of money. Likewise, emergency rooms are legally obligated to treat anyone, regardless of ability to pay.1 So I didn’t quite understand how the described events could have happened. Anyone reporting sudden chest pains should have been hauled off by an ambulance instantly. And this is where I fell down as a rationalist. I remembered several occasions where my doctor would completely fail to panic at the report of symptoms that seemed, to me, very alarming. And the Medical Establishment was always right. Every single time. I had chest pains myself, at one point, and the doctor patiently explained to me that I was describing chest muscle pain, not a heart attack. So I said into the IRC channel, “Well, if the paramedics told your friend it was nothing, it must really be nothing—they’d have hauled him off if there was the tiniest chance of serious trouble.” Thus I managed to explain the story within my existing model, though the fit still felt a little forced . . . Later on, the fellow comes back into the IRC chatroom and says his friend made the whole thing up. Evidently this was not one of his more reliable friends. I should have realized, perhaps, that an unknown acquaintance of an acquaintance in an IRC channel might be less reliable than a published journal article. Alas, belief is easier than disbelief; we believe instinctively, but disbelief requires a conscious effort.2 So instead, by dint of mighty straining, I forced my model of reality to explain an anomaly that never actually happened. And I knew how embarrassing this was. I knew that the usefulness of a model is not what it can explain, but what it can’t. A hypothesis that forbids nothing, permits everything, and thereby fails to constrain anticipation. Your strength as a rationalist is your ability to be more confused by fiction than by reality. If you are equally good at explaining any outcome, you have zero knowledge. We are all weak, from time to time; the sad part is that I could have been stronger. I had all the information I needed to arrive at the correct answer, I even noticed the problem, and then I ignored it. My feeling of confusion was a Clue, and I threw my Clue away. I should have paid more attention to that sensation of still feels a little forced. It’s one of the most important feelings a truthseeker can have, a part of your strength as a rationalist. It is a design flaw in human cognition that this sensation manifests as a quiet strain in the back of your mind, instead of a wailing alarm siren and a glowing neon sign reading: Either Your Model Is False Or This Story Is Wrong. 1 And the hospital absorbs the costs, which are enormous, so hospitals are closing their emergency rooms . . . It makes you wonder what’s the point of having economists if we’re just going to ignore them. 2 From McCluskey (2007), “Truth Bias”: “[P]eople are more likely to correctly judge that a truthful statement is true than...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: References & Resources for LessWrong, published by XiXiDu on the LessWrong. A list of references and resources for LW. Updated: 2011-05-24. F = Free. E = Easy (adequate for a low educational background) M = Memetic Hazard (controversial ideas or works of fiction) Summary. Do not flinch, most of LessWrong can be read and understood by people with a previous level of education less than secondary school. (And Khan Academy followed by BetterExplained plus the help of Google and Wikipedia ought to be enough to let anyone read anything directed at the scientifically literate.) Most of these references aren't prerequisite, and only a small fraction are pertinent to any particular post on LessWrong. Do not be intimidated, just go ahead and start reading the Sequences if all this sounds too long. It's much easier to understand than this list makes it look like. Nevertheless, as it says in the Twelve Virtues of Rationality, scholarship is a virtue, and in particular: It is especially important to eat math and science which impinges upon rationality: Evolutionary psychology, heuristics and biases, social psychology, probability theory, decision theory. Contents. LessWrong.com. Overview. Why read Less Wrong? Artificial Intelligence. General. Friendly AI. Machine Learning. The Technological Singularity. Heuristics and Biases. Mathematics. Learning Mathematics. Basics. General. Probability. Logic. Foundations. Miscellaneous. Decision theory. Game Theory. Programming. Python. Haskell. General. Computer science. (Algorithmic) Information Theory. Physics. General. General relativity. Quantum physics. Foundations. Evolution. Philosophy. General. The Mind. Epistemology. Linguistics. Neuroscience. General Education. Miscellaneous. Concepts. Websites. Fun & Fiction. Fiction. Fun. Go. LessWrong.com. This list is hosted on LessWrong.com, a community blog devoted to refining the art of human rationality - the art of thinking. If you follow the links below you'll learn more about this community. It is one of the most important resources you'll ever come across if your aim is to get what you want, if you want to win. It shows you that there is more to most things than meets the eye, but more often than not much less than you think. It shows you that even smart people can be completely wrong but that most people are not even wrong. It teaches you to be careful in what you emit and to be skeptical of what you receive. It doesn't tell you what is right, it teaches you how to think and to become less wrong. And to do so is in your own self interest because it helps you to attain your goals, it helps you to achieve what you want. Overview. About Less Wrong FE. FAQ FE. Less Wrong wiki (The wiki about rationality.) F. Less Wrong discussion area F. The Sequences (The most systematic way to approach the Less Wrong archives.) FE. Sequences in Alternative Formats (HTML, Markdown, PDF, and ePub versions.) FE. List of all articles from Less Wrong (In chronological order.) F. Graphical Visualization of Major Dependencies (Dependencies between Eliezer Yudkowsky posts.) FE. Eliezer's Posts Index (Autogenerated index of all Yudkowsky posts in chronological order.) FE. Eliezer Yudkowsky's Homepage (Founder of LW and top contributor.) FE. Less Wrong Q&A with Eliezer Yudkowsky: Video Answers FE. An interview with Eliezer Yudkowsky (Parts 1, 2 and 3) FE. Eliezer Yudkowsky on Bloggingheads.tv FE. Best of Rationality Quotes 2009/2010 FE. Less Wrong Rationality Quotes (Sorted by points. Created by DanielVarga.) FE. Comment formatting FE. Why read Less Wrong? A few articles exemplifying in detail what you can expect from reading Less Wrong, why it is important, what you can learn and how it does help you. Yes, a blog. FE. What I've learned from Less Wrong FE. Goals for which Less Wrong does (and doesn't) help FE. Ra...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Use the Try Harder, Luke, published by Eliezer Yudkowsky on the LessWrong. "When there's a will to fail, obstacles can be found." —John McCarthy I first watched Star Wars IV-VI when I was very young. Seven, maybe, or nine? So my memory was dim, but I recalled Luke Skywalker as being, you know, this cool Jedi guy. Imagine my horror and disappointment, when I watched the saga again, years later, and discovered that Luke was a whiny teenager. I mention this because yesterday, I looked up, on Youtube, the source of the Yoda quote: "Do, or do not. There is no try." Oh. My. Cthulhu. Along with the Youtube clip in question, I present to you a little-known outtake from the scene, in which the director and writer, George Lucas, argues with Mark Hamill, who played Luke Skywalker: Luke: All right, I'll give it a try. Yoda: No! Try not. Do. Or do not. There is no try. Luke raises his hand, and slowly, the X-wing begins to rise out of the water—Yoda's eyes widen—but then the ship sinks again. Mark Hamill: "Um, George..." George Lucas: "What is it now?" Mark: "So... according to the script, next I say, 'I can't. It's too big'." George: "That's right." Mark: "Shouldn't Luke maybe give it another shot?" George: "No. Luke gives up, and sits down next to Yoda—" Mark: "This is the hero who's going to take down the Empire? Look, it was one thing when he was a whiny teenager at the beginning, but he's in Jedi training now. Last movie he blew up the Death Star. Luke should be showing a little backbone." George: "No. You give up. And then Yoda lectures you for a while, and you say, 'You want the impossible'. Can you remember that?" Mark: "Impossible? What did he do, run a formal calculation to arrive at a mathematical proof? The X-wing was already starting to rise out of the swamp! That's the feasibility demonstration right there! Luke loses it for a second and the ship sinks back—and now he says it's impossible? Not to mention that Yoda, who's got literally eight hundred years of seniority in the field, just told him it should be doable—" George: "And then you walk away." Mark: "It's his friggin' spaceship! If he leaves it in the swamp, he's stuck on Dagobah for the rest of his miserable life! He's not just going to walk away! Look, let's just cut to the next scene with the words 'one month later' and Luke is still raggedly standing in front of the swamp, trying to raise his ship for the thousandth time—" George: "No." Mark: "Fine! We'll show a sunset and a sunrise, as he stands there with his arm out, straining, and then Luke says 'It's impossible'. Though really, he ought to try again when he's fully rested—" George: "No." Mark: "Five goddamned minutes! Five goddamned minutes before he gives up!" George: "I am not halting the story for five minutes while the X-wing bobs in the swamp like a bathtub toy." Mark: "For the love of sweet candied yams! If a pathetic loser like this could master the Force, everyone in the galaxy would be using it! People would become Jedi because it was easier than going to high school." George: "Look, you're the actor. Let me be the storyteller. Just say your lines and try to mean them." Mark: "The audience isn't going to buy it." George: "Trust me, they will." Mark: "They're going to get up and walk out of the theater." George: "They're going to sit there and nod along and not notice anything out of the ordinary. Look, you don't understand human nature. People wouldn't try for five minutes before giving up if the fate of humanity were at stake." Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Beyond the Reach of God, published by Eliezer Yudkowsky on the LessWrong. Today's post is a tad gloomier than usual, as I measure such things. It deals with a thought experiment I invented to smash my own optimism, after I realized that optimism had misled me. Those readers sympathetic to arguments like, "It's important to keep our biases because they help us stay happy," should consider not reading. (Unless they have something to protect, including their own life.) So! Looking back on the magnitude of my own folly, I realized that at the root of it had been a disbelief in the Future's vulnerability—a reluctance to accept that things could really turn out wrong. Not as the result of any explicit propositional verbal belief. More like something inside that persisted in believing, even in the face of adversity, that everything would be all right in the end. Some would account this a virtue (zettai daijobu da yo), and others would say that it's a thing necessary for mental health. But we don't live in that world. We live in the world beyond the reach of God. It's been a long, long time since I believed in God. Growing up in an Orthodox Jewish family, I can recall the last remembered time I asked God for something, though I don't remember how old I was. I was putting in some request on behalf of the next-door-neighboring boy, I forget what exactly—something along the lines of, "I hope things turn out all right for him," or maybe "I hope he becomes Jewish." I remember what it was like to have some higher authority to appeal to, to take care of things I couldn't handle myself. I didn't think of it as "warm", because I had no alternative to compare it to. I just took it for granted. Still I recall, though only from distant childhood, what it's like to live in the conceptually impossible possible world where God exists. Really exists, in the way that children and rationalists take all their beliefs at face value. In the world where God exists, does God intervene to optimize everything? Regardless of what rabbis assert about the fundamental nature of reality, the take-it-seriously operational answer to this question is obviously "No". You can't ask God to bring you a lemonade from the refrigerator instead of getting one yourself. When I believed in God after the serious fashion of a child, so very long ago, I didn't believe that. Postulating that particular divine inaction doesn't provoke a full-blown theological crisis. If you said to me, "I have constructed a benevolent superintelligent nanotech-user", and I said "Give me a banana," and no banana appeared, this would not yet disprove your statement. Human parents don't always do everything their children ask. There are some decent fun-theoretic arguments—I even believe them myself—against the idea that the best kind of help you can offer someone, is to always immediately give them everything they want. I don't think that eudaimonia is formulating goals and having them instantly fulfilled; I don't want to become a simple wanting-thing that never has to plan or act or think. So it's not necessarily an attempt to avoid falsification, to say that God does not grant all prayers. Even a Friendly AI might not respond to every request. But clearly, there exists some threshold of horror awful enough that God will intervene. I remember that being true, when I believed after the fashion of a child. The God who does not intervene at all, no matter how bad things get—that's an obvious attempt to avoid falsification, to protect a belief-in-belief. Sufficiently young children don't have the deep-down knowledge that God doesn't really exist. They really expect to see a dragon in their garage. They have no reason to imagine a loving God who never acts. Where exactly is the boundary of sufficient awfulness? Even a child can imagine arguing o...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Dangers of steelmanning / principle of charity, published by gothgirl420666 on the LessWrong. As far as I can tell, most people around these parts consider the principle of charity and its super saiyan form, steelmanning, to be Very Good Rationalist Virtues. I basically agree and I in fact operate under these principles more or less automatically now. HOWEVER, no matter how good the rule is, there are always exceptions, which I have found myself increasingly concerned about. This blog post that I found in the responses to Yvain's anti-reactionary FAQ argues that even though the ancient Romans had welfare, this policy was motivated not for concern for the poor or for a desire for equality like our modern welfare policies, but instead "the Roman dole was wrapped up in discourses about a) the might and wealth of Rome and b) goddess worship... The dole was there because it made the emperor more popular and demonstrated the wealth of Rome to the people. What’s more, the dole was personified as Annona, a goddess to be worshiped and thanked." So let's assume this guy is right, and imagine that an ancient Roman travels through time to the present day. He reads an article by some progressive arguing (using the rationale one would typically use) that Obama should increase unemployment benefits. "This makes no sense," the Roman thinks to himself. "Why would you give money to someone who doesn't work for it? Why would you reward lack of virtue? Also, what's this about equality? Isn't it right that an upper class exists to rule over a lower class?" Etc. But fortunately, between when he hopped out of the time machine and when he found this article, a rationalist found him and explained to him steelmanning and the principle of charity. "Ah, yes," he thinks. "Now I remember what the rationalist said. I was not being so charitable. I now realize that this position kind of makes sense, if you read between the lines. Giving more unemployment benefits would, now that I think about it, demonstrate the power of America to the people, and certainly Annona would approve. I don't know why whoever wrote this article didn't just come out and say that, though. Maybe they were confused". Hopefully you can see what I'm getting at. When you regularly use the principle of charity and steelmanning, you run the risk of: 1. Sticking rigidly to a certain worldview/paradigm/established belief set, even as you find yourself willing to consider more and more concrete propositions. The Roman would have done better to really read what the modern progressive's logic was, think about it, and try to see where he was coming from than to automatically filter it through his own worldview. If he consistently does this he will never find himself considering alternative ways of seeing the world that might be better. 2. Falsely developing the sense that your worldview/paradigm/established belief set is more popular than it is. Pretty much no one today holds the same values that an ancient Roman does, but if the Roman goes around being charitable all the time then he will probably see his own beliefs reflected back at him a fair amount. 3. Taking arguments more seriously than you possibly should. I feel like I see all the time on rationalist communities people say stuff like "this argument by A sort of makes sense, you just need to frame it in objective, consequentialist terms like blah blah blah blah blah" and then follow with what looks to me like a completely original thought that I've never seen before. But why didn't A just frame her argument in objective, consequentialist terms? Do we assume that what she wrote was sort of a telephone-game approximation of what was originally a highly logical consequentialist argument? If so where can I find that argument? And if not, why are we assuming that A is a crypto-conse...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Final Words, published by Eliezer Yudkowsky on the LessWrong. Sunlight enriched air already alive with curiosity, as dawn rose on Brennan and his fellow students in the place to which Jeffreyssai had summoned them. They sat there and waited, the five, at the top of the great glassy crag that was sometimes called Mount Mirror, and more often simply left unnamed. The high top and peak of the mountain, from which you could see all the lands below and seas beyond. (Well, not all the lands below, nor seas beyond. So far as anyone knew, there was no place in the world from which all the world was visible; nor, equivalently, any kind of vision that would see through all obstacle-horizons. In the end it was the top only of one particular mountain: there were other peaks, and from their tops you would see other lands below; even though, in the end, it was all a single world.) "What do you think comes next?" said Hiriwa. Her eyes were bright, and she gazed to the far horizons like a lord. Taji shrugged, though his own eyes were alive with anticipation. "Jeffreyssai's last lesson doesn't have any obvious sequel that I can think of. In fact, I think we've learned just about everything that I knew the beisutsukai masters know. What's left, then -" "Are the real secrets," Yin completed the thought. Hiriwa and Taji and Yin shared a grin, among themselves. Styrlyn wasn't smiling. Brennan suspected rather strongly that Styrlyn was older than he had admitted. Brennan wasn't smiling either. He might be young, but he kept high company, and had witnesssed some of what went on behind the curtains of the world. Secrets had their price, always, that was the barrier that made them secrets; and Brennan thought he had a good idea of what this price might be. There was a cough from behind them, at a moment when they had all happened to be looking in any other direction but that one. As one, their heads turned. Jeffreyssai stood there, in a casual robe that looked more like glass than any proper sort of mirrorweave. Jeffreyssai stood there and looked at them, a strange abiding sorrow in those inscrutable eyes. "Sen...sei," Taji started, faltering as that bright anticipation stumbled over Jeffreyssai's return look. "What's next?" "Nothing," Jeffreyssai said abruptly. "You're finished. It's done." Hiriwa, Taji, and Yin all blinked, a perfect synchronized gesture of shock. Then, before their expressions could turn to outrage and objections - "Don't," Jeffreyssai said. There was real pain in it. "Believe me, it hurts me more than it hurts you." He might have been looking at them; or at something far away, or long ago. "I don't know exactly what roads may lie before you - but yes, I know that you're not ready. That I'm sending you out unprepared. That everything I taught you is incomplete. I know that what I said is not what you heard. That I left out the one most important thing. That the rhythm at the center of everything is missing and astray. I know that you will harm yourself in the course of trying to use what I taught. So that I, personally, will have shaped, in some fashion unknown to me, the very knife that will cut you..." "...that's the hell of being a teacher, you see," Jeffreyssai said. Something grim flickered in his expression. "Nonetheless, you're done. Finished, for now. What lies between you and mastery is not another classroom. We are fortunate, or perhaps not fortunate, that the road to power does not wend only through lecture halls. Else the quest would be boring to the bitter end. Still, I cannot teach you; and so it is a moot point whether I would if I could. There is no master here whose art is entirely inherited. Even the beisutsukai have never discovered how to teach certain things; it is possible that such an event has been prohibited. And so you can only arrive at mastery ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Secure homes for digital people, published by paulfchristiano on the LessWrong. Being a “digital person” could be scary—if I don’t have control over the hardware I’m running on, then someone else could get my code and run tons of copies in horrible conditions. (See also: qntm’s Lena.) It would be great to guarantee digital people some control over their situation: 1. to control their local environment and sensations, 2. to avoid unauthorized rewinding or duplicating. I’ll describe how you could modify the code of a digital person so that they retain this control even if an adversary has access to their source code. This would be very expensive with current cryptography. I think the overhead will eventually become cheap enough that it’s possible to do for some digital people, though it will likely remain expensive enough that it is never applied to most digital people (and with luck most digital people will be able to feel secure for other reasons). Part 1: the right to control my environment My ideal I live in a comfortable virtual home. I control all of the details of that world. When people communicate with me, I can choose how/whether to hear them, and how/whether to update my home based on what they say (e.g. to render an avatar for them) Sometimes I may occupy a virtual world where a foreign server determines what I see, feel, or hear. But even then I can place boundaries on my experiences and have the ability to quickly retreat to my home. I have as much control as feasible over my own mental state and simulated body. No one else can tamper directly with them. I can choose to pause myself for as long as I want (or permanently). My local environment is private, and I have access to plenty of tamper-proof storage. I can do whatever I want with computers in my home, including e.g. verifying signatures or carrying on encrypted conversations. Implementation First we write a simple environment that reflects all my desiderata (the “home”). Then I apply indistinguishability obfuscation to (me + home), so that the house becomes private and tamper-proof. (This is an extremely expensive operation, more on that later.) I distribute the obfuscated home and hopefully destroy any unprotected copies of myself. One conceptual difficulty is that indistinguishability obfuscation applies to circuits whereas I would like to obfuscate a long-running program. But this can be handled straightforwardly, as discussed in Appendix A. The home could consume terabytes of memory and teraflops of compute before it added significantly to the expense of running a human-like digital person, so I could live in relative luxury. The home could also negotiate resource requirements with the external world, and to decide what to do when requested resources are unavailable (e.g. to pause until it becomes available). Limitation 1: cost Indistinguishability obfuscation is extremely expensive, more like a factor of 10000000000 slowdown than 10. It will get faster with further research, but probably not fast enough to obfuscate the whole person+home. But there are other ways to speed up the process: I think it’s probably possible to have most of the computation be “merely” homomorphically encrypted, and to have an obfuscated controller which verifies and decrypts the results. FHE could be much faster than obfuscation; if I had to guess I’d say it would converge to something like 2-3 orders of magnitude of slowdown. We can potentially have an obfuscated controller verify a much larger untrusted computation. I don’t know how fast we can make delegated computation, but I could imagine it getting closer to 2x than 100x. It might help further that we are not applying these methods to generic problems but to a very specific structured problem (which probably has quite low circuit depth). One complication is that...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Your inner Google, published by PhilGoetz on the LessWrong. I just heard a comment by Braddock of Lovesystems that was brilliant: All that your brain does when you ask it a question is hit "search" and return the first hit it finds. So be careful how you phrase your question. Say you just arrived at work, and realized you once again left your security pass at home. You ask yourself, "Why do I keep forgetting my security pass?" If you believe you are a rational agent, you might think that you pass that question to your brain, and it parses it into its constituent parts and builds a query like X such that cause(X, forget(me, securityPass)) and queries its knowledge base using logical inference for causal explanations specifically relevant to you and your security pass. But you are not rational, and your brain is lazy; and as soon as you phrase your question and pass it on to your subconscious, your brain just Googles itself with a query like why people forget things looks at the first few hits it comes across, maybe finds their most-general unifier, checks that it's a syntactically valid answer to the question, and responds with, "Because you are a moron." Your inner Google has provided a plausible answer to the question, and it sits back, satisfied that it's done its job. If you instead ask your brain something more specific, such as, "What can I do to help me remember my security pass tomorrow?", thus requiring its answer to refer to you and actions to remember things and tomorrow, your brain may come up with something useful, such as, "Set up a reminder now that will notify you tomorrow morning by cell phone to bring your security pass." So, try to be at least as careful when asking questions of your brain, as when asking them of Google. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Chapter 1: A Day of Very Low Probability, published by Eliezer Yudkowsky on the LessWrong. Disclaimer: J. K. Rowling owns Harry Potter, and no one owns the methods of rationality. This fic is widely considered to have really hit its stride starting at around Chapter 5. If you still don't like it after Chapter 10, give up. This is not a strict single-point-of-departure fic - there exists a primary point of departure, at some point in the past, but also other alterations. The best term I've heard for this fic is "parallel universe". The text contains many clues: obvious clues, not-so-obvious clues, truly obscure hints which I was shocked to see some readers successfully decode, and massive evidence left out in plain sight. This is a rationalist story; its mysteries are solvable, and meant to be solved. The pacing of the story is that of serial fiction, i.e., that of a TV show running for a predetermined number of seasons, whose episodes are individually plotted but with an overall arc building to a final conclusion. All science mentioned is real science. But please keep in mind that, beyond the realm of science, the views of the characters may not be those of the author. Not everything the protagonist does is a lesson in wisdom, and advice offered by darker characters may be untrustworthy or dangerously double-edged. Beneath the moonlight glints a tiny fragment of silver, a fraction of a line... (black robes, falling) ...blood spills out in litres, and someone screams a word. Every inch of wall space is covered by a bookcase. Each bookcase has six shelves, going almost to the ceiling. Some bookshelves are stacked to the brim with hardback books: science, maths, history, and everything else. Other shelves have two layers of paperback science fiction, with the back layer of books propped up on old tissue boxes or lengths of wood, so that you can see the back layer of books above the books in front. And it still isn't enough. Books are overflowing onto the tables and the sofas and making little heaps under the windows. This is the living-room of the house occupied by the eminent Professor Michael Verres-Evans, and his wife, Mrs. Petunia Evans-Verres, and their adopted son, Harry James Potter-Evans-Verres. There is a letter lying on the living-room table, and an unstamped envelope of yellowish parchment, addressed to Mr. H. Potter in emerald-green ink. The Professor and his wife are speaking sharply at each other, but they are not shouting. The Professor considers shouting to be uncivilised. "You're joking," Michael said to Petunia. His tone indicated that he was very much afraid that she was serious. "My sister was a witch," Petunia repeated. She looked frightened, but stood her ground. "Her husband was a wizard." "This is absurd!" Michael said sharply. "They were at our wedding - they visited for Christmas -" "I told them you weren't to know," Petunia whispered. "But it's true. I've seen things -" The Professor rolled his eyes. "Dear, I understand that you're not familiar with the sceptical literature. You may not realise how easy it is for a trained magician to fake the seemingly impossible. Remember how I taught Harry to bend spoons? If it seemed like they could always guess what you were thinking, that's called cold reading -" "It wasn't bending spoons -" "What was it, then?" Petunia bit her lip. "I can't just tell you. You'll think I'm -" She swallowed. "Listen. Michael. I wasn't - always like this -" She gestured at herself, as though to indicate her lithe form. "Lily did this. Because I - because I begged her. For years, I begged her. Lily had always been prettier than me, and I'd... been mean to her, because of that, and then she got magic, can you imagine how I felt? And I begged her to use some of that magic on me so that I could be pretty too, even if I couldn'...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Rule Thinkers In, Not Out, published by Scott Alexander on the LessWrong. Imagine a black box which, when you pressed a button, would generate a scientific hypothesis. 50% of its hypotheses are false; 50% are true hypotheses as game-changing and elegant as relativity. Even despite the error rate, it’s easy to see this box would quickly surpass space capsules, da Vinci paintings, and printer ink cartridges to become the most valuable object in the world. Scientific progress on demand, and all you have to do is test some stuff to see if it’s true? I don’t want to devalue experimentalists. They do great work. But it’s appropriate that Einstein is more famous than Eddington. If you took away Eddington, someone else would have tested relativity; the bottleneck is in Einsteins. Einstein-in-a-box at the cost of requiring two Eddingtons per insight is a heck of a deal. What if the box had only a 10% success rate? A 1% success rate? My guess is: still most valuable object in the world. Even an 0.1% success rate seems pretty good, considering (what if we ask the box for cancer cures, then test them all on lab rats and volunteers?) You have to go pretty low before the box stops being great. I thought about this after reading this list of geniuses with terrible ideas. Linus Pauling thought Vitamin C cured everything. Isaac Newton spent half his time working on weird Bible codes. Nikola Tesla pursued mad energy beams that couldn’t work. Lynn Margulis revolutionized cell biology by discovering mitochondrial endosymbiosis, but was also a 9-11 truther and doubted HIV caused AIDS. Et cetera. Obviously this should happen. Genius often involves coming up with an outrageous idea contrary to conventional wisdom and pursuing it obsessively despite naysayers. But nobody can have a 100% success rate. People who do this successfully sometimes should also fail at it sometimes, just because they’re the kind of person who attempts it at all. Not everyone fails. Einstein seems to have batted a perfect 1000 (unless you count his support for socialism). But failure shouldn’t surprise us. Yet aren’t some of these examples unforgiveably bad? Like, seriously Isaac – Bible codes? Well, granted, Newton’s chemical experiments may have exposed him to a little more mercury than can be entirely healthy. But remember: gravity was considered creepy occult pseudoscience by its early enemies. It subjected the earth and the heavens to the same law, which shocked 17th century sensibilities the same way trying to link consciousness and matter would today. It postulated that objects could act on each other through invisible forces at a distance, which was equally outside the contemporaneous Overton Window. Newton’s exceptional genius, his exceptional ability to think outside all relevant boxes, and his exceptionally egregious mistakes are all the same phenomenon (plus or minus a little mercury). Or think of it a different way. Newton stared at problems that had vexed generations before him, and noticed a subtle pattern everyone else had missed. He must have amazing hypersensitive pattern-matching going on. But people with such hypersensitivity should be most likely to see patterns where they don’t exist. Hence, Bible codes. These geniuses are like our black boxes: generators of brilliant ideas, plus a certain failure rate. The failures can be easily discarded: physicists were able to take up Newton’s gravity without wasting time on his Bible codes. So we’re right to treat geniuses as valuable in the same way we would treat those boxes as valuable. This goes not just for geniuses, but for anybody in the idea industry. Coming up with a genuinely original idea is a rare skill, much harder than judging ideas is. Somebody who comes up with one good original idea (plus ninety-nine really stupid cringeworthy takes) is a ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Book Review: The Secret Of Our Success, published by Scott Alexander on the LessWrong. [Previously in sequence: Epistemic Learned Helplessness] I. “Culture is the secret of humanity’s success” sounds like the most vapid possible thesis. The Secret Of Our Success by anthropologist Joseph Heinrich manages to be an amazing book anyway. Heinrich wants to debunk (or at least clarify) a popular view where humans succeeded because of our raw intelligence. In this view, we are smart enough to invent neat tools that help us survive and adapt to unfamiliar environments. Against such theories: we cannot actually do this. Heinrich walks the reader through many stories about European explorers marooned in unfamiliar environments. These explorers usually starved to death. They starved to death in the middle of endless plenty. Some of them were in Arctic lands that the Inuit considered among their richest hunting grounds. Others were in jungles, surrounded by edible plants and animals. One particularly unfortunate group was in Alabama, and would have perished entirely if they hadn’t been captured and enslaved by local Indians first. These explorers had many advantages over our hominid ancestors. For one thing, their exploration parties were made up entirely of strong young men in their prime, with no need to support women, children, or the elderly. They were often selected for their education and intelligence. Many of them were from Victorian Britain, one of the most successful civilizations in history, full of geniuses like Darwin and Galton. Most of them had some past experience with wilderness craft and survival. But despite their big brains, when faced with the task our big brains supposedly evolved for – figuring out how to do hunting and gathering in a wilderness environment – they failed pathetically. Nor is it surprising that they failed. Hunting and gathering is actually really hard. Here’s Heinrich’s description of how the Inuit hunt seals: You first have to find their breathing holes in the ice. It’s important that the area around the hole be snow-covered—otherwise the seals will hear you and vanish. You then open the hole, smell it to verify it’s still in use (what do seals smell like?), and then assess the shape of the hole using a special curved piece of caribou antler. The hole is then covered with snow, save for a small gap at the top that is capped with a down indicator. If the seal enters the hole, the indicator moves, and you must blindly plunge your harpoon into the hole using all your weight. Your harpoon should be about 1.5 meters (5ft) long, with a detachable tip that is tethered with a heavy braid of sinew line. You can get the antler from the previously noted caribou, which you brought down with your driftwood bow. The rear spike of the harpoon is made of extra-hard polar bear bone (yes, you also need to know how to kill polar bears; best to catch them napping in their dens). Once you’ve plunged your harpoon’s head into the seal, you’re then in a wrestling match as you reel him in, onto the ice, where you can finish him off with the aforementioned bear-bone spike. Now you have a seal, but you have to cook it. However, there are no trees at this latitude for wood, and driftwood is too sparse and valuable to use routinely for fires. To have a reliable fire, you’ll need to carve a lamp from soapstone (you know what soapstone looks like, right?), render some oil for the lamp from blubber, and make a wick out of a particular species of moss. You will also need water. The pack ice is frozen salt water, so using it for drinking will just make you dehydrate faster. However, old sea ice has lost most of its salt, so it can be melted to make potable water. Of course, you need to be able to locate and identify old sea ice by color and texture. To melt it, make sure yo...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Coordination as a Scarce Resource, published by johnswentworth on the LessWrong. Let’s start with a few examples of very common real-world coordination problems. The marketing department at a car dealership posts ads for specific cars, but the salespeople don’t know which cars were advertised, causing confusion when a customer calls in asking about a specific car. There’s no intentional information-hoarding, it’s just that the marketing and sales people don’t sit next to each other or talk very often. Even if the info were shared, it would need to be translated to a format usable by the salespeople. Various hard problems in analysis of large-scale biological data likely have close analogues in econometrics. The econometricians have good methods to solve the problems, and would probably be quite happy to apply those methods to biological data, and the bio experimentalists would love some analytic help. But these people hardly ever talk to each other, and use different language for the same things anyway. When the US invaded Grenada in the ‘80’s, the marines occupied one side of the island and the army occupied the other. Their radios were not compatible, so if an army officer needed to contact their counterpart in the marines, they had to walk to the nearest pay phone and get routed through Fort Bragg on commercial telephone lines. Various US intelligence agencies had all of the pieces necessary to stop the 9/11 attacks. There were agencies which knew something was planned for that day, and knew who the actors were. There were agencies which knew the terrorists were getting on the planes. There were agencies which could have moved to stop them, but unfortunately the fax(!) from the agencies which knew what was happening wasn’t checked in time. There are about 300 million people in the US. If I have a small company producing doilies, chances are there are plenty of people in the US alone who’d love my doilies and be happy to pay for them. But it’s hard to figure out exactly which people those are, and even once that’s done it’s hard to get them a message showing off my product. And even if all that works out, if the customers really want a slightly different pattern, it’s hard for them to communicate back to me what they want - even if I’d be happy to make it. So coordination problems are a constraint to production of all kinds of economic value. How taut are those constraints? Well, let’s look at the market price of relaxing coordination constraints. In other words: how much do people/companies get paid for solving coordination problems? When I think of people whose main job is to solve coordination problems, here are some occupations which spring to mind: Entrepreneurs’ main job is to coordinate salespeople, engineers, designers, marketers, investors, customers, regulators, suppliers, shippers, etc. Managers’ main job is to coordinate between their bosses, underlings, and across departments Investment bankers coordinate between investors, companies, lawyers, and a huge number of people within each of those organizations Real estate developers coordinate between builders, landowners, regulators, renters, and investors Note that all of these are occupations typically associated with very high pay. Even more to the point: within each of these occupations, people who solve more complicated coordination problems (e.g. between more people) tend to make more money. Even at the small end, the main difference between an employee and a freelancer is that the freelancer has to solve their own coordination problem (i.e. find people who want their services); freelancers make lots of money mainly when they are very good at solving this problem. Similarly with companies. If we go down the list of tech unicorns, most (though not all) of them solve coordination problems as their prim...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: We Haven't Uploaded Worms, published by jefftk on the LessWrong. In theory you can upload someone's mind onto a computer, allowing them to live forever as a digital form of consciousness, just like in the Johnny Depp film Transcendence. But it's not just science fiction. Sure, scientists aren't anywhere near close to achieving such feat with humans (and even if they could, the ethics would be pretty fraught), but now an international team of researchers have managed to do just that with the roundworm Caenorhabditis elegans. Science Alert Uploading an animal, even one as simple as c. elegans would be very impressive. Unfortunately, we're not there yet. What the people working on Open Worm have done instead is to build a working robot based on the c. elegans and show that it can do some things that the worm can do. The c. elegans nematode has only 302 neurons, and each nematode has the same fixed pattern. We've known this pattern, or connectome, since 1986. [1] In a simple model, each neuron has a threshold and will fire if the weighted sum of its inputs is greater than that threshold. Which means knowing the connections isn't enough: we also need to know the weights and thresholds. Unfortunately, we haven't figured out a way to read these values off of real worms. Suzuki et. al. (2005) [2] ran a genetic algorithm to learn values for these parameters that would give a somewhat realistic worm and showed various wormlike behaviors in software. The recent stories about the Open Worm project have been for them doing something similar in hardware. [3] To see why this isn't enough, consider that nematodes are capable of learning. Sasakura and Mori (2013) [5] provide a reasonable overview. For example, nematodes can learn that a certain temperature indicates food, and then seek out that temperature. They don't do this by growing new neurons or connections, they have to be updating their connection weights. All the existing worm simulations treat weights as fixed, which means they can't learn. They also don't read weights off of any individual worm, which means we can't talk about any specific worm as being uploaded. If this doesn't count as uploading a worm, however, what would? Consider an experiment where someone trains one group of worms to respond to stimulus one way and another group to respond the other way. Both groups are then scanned and simulated on the computer. If the simulated worms responded to simulated stimulus the same way their physical versions had, that would be good progress. Additionally you would want to demonstrate that similar learning was possible in the simulated environment. (In a 2011 post on what progress with nematodes might tell us about uploading humans I looked at some of this research before. Since then not much has changed with nematode simulation. Moore's law looks to be doing much worse in 2014 than it did in 2011, however, which makes the prospects for whole brain emulation substantially worse.) I also posted this on my blog. [1] The Structure of the Nervous System of the Nematode Caenorhabditis elegans, White et. al. (1986). [2] A Model of Motor Control of the Nematode C. Elegans With Neuronal Circuits, Suzuki et. al. (2005). [3] It looks like instead of learning weights Busbice just set them all to +1 (excitatory) and -1 (inhibitory). It's not clear to me how they knew which connections were which; my best guess is that they're using the "what happens to work" details from [2]. Their full writeup is [4]. [4] The Robotic Worm, Busbice (2014). [5] Behavioral Plasticity, Learning, and Memory in C. Elegans, Sasakura and Mori (2013). Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Reply to Holden on 'Tool AI', published by Eliezer Yudkowsky on the LessWrong. I begin by thanking Holden Karnofsky of Givewell for his rare gift of his detailed, engaged, and helpfully-meant critical article Thoughts on the Singularity Institute (SI). In this reply I will engage with only one of the many subjects raised therein, the topic of, as I would term them, non-self-modifying planning Oracles, a.k.a. 'Google Maps AGI' a.k.a. 'tool AI', this being the topic that requires me personally to answer. I hope that my reply will be accepted as addressing the most important central points, though I did not have time to explore every avenue. I certainly do not wish to be logically rude, and if I have failed, please remember with compassion that it's not always obvious to one person what another person will think was the central point. Luke Mueulhauser and Carl Shulman contributed to this article, but the final edit was my own, likewise any flaws. Summary: Holden's concern is that "SI appears to neglect the potentially important distinction between 'tool' and 'agent' AI." His archetypal example is Google Maps: Google Maps is not an agent, taking actions in order to maximize a utility parameter. It is a tool, generating information and then displaying it in a user-friendly manner for me to consider, use and export or discard as I wish. The reply breaks down into four heavily interrelated points: First, Holden seems to think (and Jaan Tallinn doesn't apparently object to, in their exchange) that if a non-self-modifying planning Oracle is indeed the best strategy, then all of SIAI's past and intended future work is wasted. To me it looks like there's a huge amount of overlap in underlying processes in the AI that would have to be built and the insights required to build it, and I would be trying to assemble mostly - though not quite exactly - the same kind of team if I was trying to build a non-self-modifying planning Oracle, with the same initial mix of talents and skills. Second, a non-self-modifying planning Oracle doesn't sound nearly as safe once you stop saying human-English phrases like "describe the consequences of an action to the user" and start trying to come up with math that says scary dangerous things like (he translated into English) "increase the correspondence between the user's belief about relevant consequences and reality". Hence why the people on the team would have to solve the same sorts of problems. Appreciating the force of the third point is a lot easier if one appreciates the difficulties discussed in points 1 and 2, but is actually empirically verifiable independently: Whether or not a non-self-modifying planning Oracle is the best solution in the end, it's not such an obvious privileged-point-in-solution-space that someone should be alarmed at SIAI not discussing it. This is empirically verifiable in the sense that 'tool AI' wasn't the obvious solution to e.g. John McCarthy, Marvin Minsky, I. J. Good, Peter Norvig, Vernor Vinge, or for that matter Isaac Asimov. At one point, Holden says: One of the things that bothers me most about SI is that there is practically no public content, as far as I can tell, explicitly addressing the idea of a "tool" and giving arguments for why AGI is likely to work only as an "agent." If I take literally that this is one of the things that bothers Holden most... I think I'd start stacking up some of the literature on the number of different things that just respectable academics have suggested as the obvious solution to what-to-do-about-AI - none of which would be about non-self-modifying smarter-than-human planning Oracles - and beg him to have some compassion on us for what we haven't addressed yet. It might be the right suggestion, but it's not so obviously right that our failure to prioritize discussing it refl...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: RAISE post-mortem, published by toonalfrink on the LessWrong. Edit November 2021: there is now the Cambridge AGI Safety Fundamentals course, which promises to be successful. It is enlightening to compare this project with RAISE. Why is that one succeeding while this one did not? I'm quite surprised to find that the answer isn't so much about more funding, more senior people to execute it, more time, etc. They're simply using existing materials instead of creating their own. This makes it orders of magnitude easier to produce the thing, you can just focus on the delivery. Why didn't I, or anyone around me, think of this? I'm honestly perplexed. It's worth thinking about. Since June, RAISE has stopped operating. I’ve taken some time to process things, and now I’m wrapping up. What was RAISE again AI Safety is starved for talent. I saw a lot of smart people around me that wanted to do the research. Their bottleneck seemed to be finding good education (and hero licensing). The plan was to alleviate that need by creating an online course about AI Safety (with nice diplomas). How did it go We spent a total of ~2 years building the platform. It started out as a project based on volunteers creating the content. Initially, many people (more than 80) signed up to volunteer, but we did not manage to get most of them to show up consistently. We gradually pivoted to paying people instead. We received a lot of encouragement for the project. Most of the enthusiasm came from people wanting to learn AI Safety. Robert Miles joined as a lecturer. When we reached out to some AI Safety researchers for suggestions on which topics to cover, we readily received helpful advice. Sometimes we also received some funds from a couple of prominent AIS organizations who thought the project could be high value, at least in expectation. The stream of funding was large enough to sustain about 1 fte working for a relatively low wage. Obtaining it was a struggle: our runway was never longer than 2 months. This created a large attention sink that made it a lot harder to create things. Nearly all of my time was spent on overhead, while others were creating the content. I did not have the time to review much of it. About 1 year into the project, we escaped this poverty trap by moving to the EA Hotel and starting a content development team there. We went up to about 4 fte, and the production rate shot up leading to an MVP relatively quickly. How did it end Before launch, the best way to secure funding seemed to be to just create the damn thing, make sure it’s good, and let it advocate for itself. After launch, a negative signal could not be dismissed as easily. We got two clear negative signals: one from a major AIS research org (that has requested not to be named), and one from the LTF fund. The former declined to continue their experimental funding of RAISE. The latter declined a grant request. These were clear signals that people in the establishment of AI Safety did not deem the project worth funding, so I reached out for a conversation. The question was this: “what version of RAISE would you fund?” The answer was roughly that while they agreed strongly with the vision for RAISE, our core product sadly wasn’t coming together in a way that suggested it would be worth it for us to keep working on it. I was tentatively offered a personal grant if I spent it on taking a step back to think hard and figure out what AI Safety needs (I ended up declining for career-strategic reasons). In another conversation, an insider told us that AI Safety needs to grow in quality more than quantity. There is already a lot of low-quality research. We need AI Safety to be held to high standards. Lowering the bar for a research-level understanding will not solve that. I decided to quit. I was out of runway, updated towards RAI...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Interfaces as a Scarce Resource, published by johnswentworth on the Lesswrong. Outline: The first three sections (Don Norman’s Fridge, Interface Design, and When And Why Is It Hard?) cover what we mean by “interface”, what it looks like for interfaces to be scarce, and the kinds of areas where they tend to be scarce. The next four sections apply these ideas to various topics: Why AR is much more difficult than VR AI alignment from an interface-design perspective Good interfaces as a key bottleneck to creation of markets Cross-department interfaces in organizations Don Norman’s Fridge Don Norman (known for popularizing the term “affordance” in The Design of Everyday Things) offers a story about the temperature controls on his old fridge: I used to own an ordinary, two-compartment refrigerator - nothing very fancy about it. The problem was that I couldn’t set the temperature properly. There were only two things to do: adjust the temperature of the freezer compartment and adjust the temperature of the fresh food compartment. And there were two controls, one labeled “freezer”, the other “refrigerator”. What’s the problem? Oh, perhaps I’d better warn you. The two controls are not independent. The freezer control also affects the fresh food temperature, and the fresh food control also affects the freezer. The natural human model of the refrigerator is: there’s two compartments, and we want to control their temperatures independently. Yet the fridge, apparently, does not work like that. Why not? Norman: In fact, there is only one thermostat and only one cooling mechanism. One control adjusts the thermostat setting, the other the relative proportion of cold air sent to each of the two compartments of the refrigerator. It’s not hard to imagine why this would be a good design for a cheap fridge: it requires only one cooling mechanism and only one thermostat. Resources are saved by not duplicating components - at the cost of confused customers. The root problem in this scenario is a mismatch between the structure of the machine (one thermostat, adjustable allocation of cooling power) and the structure of what-humans-want (independent temperature control of two compartments). In order to align the behavior of the fridge with the behavior humans want, somebody, at some point, needs to do the work of translating between the two structures. In Norman’s fridge example, the translation is botched, and confusion results. We’ll call whatever method/tool is used for translating between structures an interface. Creating good methods/tools for translating between structures, then, is interface design. Interface Design In programming, the analogous problem is API design: taking whatever data structures are used by a software tool internally, and figuring out how to present them to external programmers in a useful, intelligible way. If there’s a mismatch between the internal structure of the system and the structure of what-users-want, then it’s the API designer’s job to translate. A “good” API is one which handles the translation well. User interface design is a more general version of the same problem: take whatever structures are used by a tool internally, and figure out how to present them to external users in a useful, intelligible way. Conceptually, the only difference from API design is that we no longer assume our users are programmers interacting with the tool via code. We design the interface to fit however people use it - that could mean handles on doors, or buttons and icons in a mobile app, or the temperature knobs on a fridge. Economically, interface design is a necessary input to make all sorts of things economically useful. How scarce is that input? How much are people willing to spend for good interface design? My impression is: a lot. There’s an entire category of tech com...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Book Review: Working With Contracts, published by johnswentworth on the LessWrong Contracts is one of those areas that I always figured I ought to study, at least enough to pick up the basics, but never seemed either interesting or important enough to reach the front of my queue. On top of that, there’s a lot of different angles from which to approach the subject: the law-school-style Contracts 101 class covers the legal principles governing contracts, the economists’ version abstracts away the practical specifics and talks about contracts in game-theoretic terms, more business-oriented books often focus on negotiation, etc. “Working With Contracts: What Law School Doesn’t Teach You” is about the practical skills needed for working with contracts on an everyday basis - specifically the sort of skills usually picked up on the job by young lawyers. It talks about things like what to look for when reviewing a contract, how to organize contracts, why lawyers use weird words like “heretofore”, various gotchas to watch out for, etc. It assumes minimal background knowledge, but also includes lots of technical nuts and bolts. In short, it’s the perfect book for someone who wants a technical understanding of real-world contract practice. This post will review interesting things I learned from the book. Background Knowledge First, some very brief background info, which the book itself mostly assumes. Legally, in order to count as a “contract”, we need four main pieces: Offer: someone offers a deal Acceptance: someone else accepts it Consideration: both parties gain something from the deal; it’s not a gift Mutual understanding: both parties agree on what the deal is and the fact that they’ve agreed to it A Contracts 101 class has all sorts of details and gotchas related to these. Notice that “signature on a piece of paper” is not on that list; e.g. oral contracts are entirely enforceable, it’s just harder to prove their existence in court. Even implicit contracts are enforceable - e.g. when you order food from a restaurant, you implicitly agree to pay for it, and that’s a legally-enforceable contract. That said, we’ll focus here on explicit written contracts. Once formed, a contract acts as custom, private law between the parties. Enforcement of this law goes through civil courts - i.e. if someone breaches the contract, then the counterparty can sue them for damages. Note the “for damages” in that sentence; if a counterparty breaches a contract in a way that doesn’t harm you (relative to not breaching), then you probably won’t be able to sue them. (Potentially interesting exercise for any lawyers in the audience: figure out a realistic contractual equivalent of Newcomb’s problem, where someone agrees to one-box on behalf of someone else but then two-boxes, and claims in court that their decision to two-box benefited the counterparty rather than harming them. I’d bet there’s case law on something equivalent to this.) Note that this is all specific to American law, as is the book. In particular, other countries tend to more often require specific wording, ceremonial actions, and the like in order to make a contract (or component of a contract) enforceable. What Do Contracts Do? The “functional” components of a contract can be organized into two main categories: representations and covenants. A representation says that something has happened or is true; a covenant says that something will happen or will be true. Some example representations: ABC Corp signs a statement that they have no pending lawsuits against them. Bob signs a statement that the house he’s selling contains no lead-based paint or asbestos insulation. Carol signs a statement that the forms she provided for a mortgage application are accurate and complete. Title Corp signs a statement that there are no outstanding m...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Curated conversations with brilliant rationalists, published by spencerg on the LessWrong. Since August 2020 I've been recording conversations with brilliant and insightful rationalists, effective altruists (and people adjacent to or otherwise connected somehow to those communities). If you're an avid reader of this site, I suspect you will recognize many of the names of those I've spoken to. Since I suspect some LessWrong readers will appreciate these conversations, here is a curated list with links, organized by the LessWrong relevant topics we cover in each conversation. All of these conversations can also be found by searching for "Clearer Thinking" in just about any podcast app. If there are other people you'd like to see me record conversations with, please nominate them in the comments! The format is that I invite each guest to bring 4 or 5 "ideas that matter" that they are excited to talk about, and then the aim is to have a fun, intellectual discussion of those ideas. Rationality Lines of Retreat and Incomplete Maps with Anna Salamon What does it mean to leave lines of retreat in social contexts? How can we make sense of the current state of the world? What happens when we run out of map? How does the book Elephant in the Brain apply to the above questions? Rationality Education and Dating with Jacob Falkovich What's the best way to teach rationality? How do you communicate rationalist principles to people who aren't already interested in thinking more clearly? What has COVID taught us about how people typically make decisions and think about problems? Where and how can the rationalist community improve? Does rationalism have anything to say about (for example) exercise, spirituality, art, or other parts of the human experience that aren't typically addressed by rationalists? What are some positive aspects of social media (especially Twitter)? What's going on with recent dating trends? Has dating gotten harder in recent years? How many people does it take to make a pencil? Is there a case to be made for anti-antinatalism? Scout and Soldier Mindsets with Julia Galef What are "scout" and "soldier" mindsets? How can we have productive disagreements even when one person isn't in scout mindset? Is knowing about good rationality habits sufficient to reason well? When do we naturally tend to be in scout mindset or soldier mindset? When is each mindset beneficial or harmful? Are humans "rationally irrational"? What are the two different types of confidence? What are some practical strategies for shifting our mindset in the moment from soldier to scout? Comfort Languages and Nuanced Thinking with Kat Woods What's the best way to help someone who's going through a difficult situation? What are the four states of distress? What are "comfort languages"? How can we introduce more nuance into our everyday thinking habits? When gathering information and forming opinions, how do you know who to trust? What's the difference between intelligence and wisdom? Aging/Longevity History and Longevity with Will Eden What are the benefits of studying history? How do we find useful historical analyses? Can learning about history save us from repeating it? Is America decaying as a nation, empire, and/or leading world power? Generally speaking, what causes empires to fail? Is the aging and decay experienced by organic bodies analogous to the aging and decay experienced by an empire (or by any complex system, for that matter)? What are all the reasons organisms age, decay, and die? What are the most promising avenues of exploration in longevity research? What kind of stressors on our bodies are beneficial? How accurate is the efficient market hypothesis? What kinds of catalysts force a market to value assets at their "intrinsic" value? How rational are markets? Artificial Intelligence AI...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Alcohol, health, and the ruthless logic of the Asian flush, published by dynomight on the LessWrong. This is a linkpost for/ Say you're an evil scientist. One day at work you discover a protein that crosses the blood-brain barrier and causes crippling migraine headaches if someone's attention drifts while driving. Despite being evil, you're a loving parent with a kid learning to drive. Like everyone else, your kid is completely addicted to their phone, and keep refreshing their feeds while driving. Your suggestions that the latest squirrel memes be enjoyed later at home are repeatedly rejected. Then you realize: You could just sneak into your kid's room at night, anesthetize them, and bring them to your lair! One of your goons could then extract their bone marrow and use CRISPR to recode the stem-cells for an enzyme to make the migraine protein. Sure, the headache itself might distract them, but they'll probably just stop using their phone while driving. Wouldn't you be at least tempted? This is an analogy for something about alcoholism, East Asians, Odysseus, evolution, tension between different kinds of freedoms, and an idea I thought was good but apparently isn't. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Apprentice Experiment, published by johnswentworth on the LessWrong. About two months ago, someone asked me what I would do with more funding. Other than the obvious (i.e. generally improve my own quality-of-life in minor ways), my main answer was: take on an apprentice. I have some models about how best to train people for this sort of work, and an apprentice would allow me to test those models while also supporting my own research. I started laying groundwork for that plan - in particular, Specializing in Problems We Don’t Understand laid out my main background model. Then, about a month ago, Aysajan put up a short post titled “Can I be Your Apprentice?” - essentially an open call to people on LW doing cool work. We talked, it seemed like a good fit, so the apprentice experiment kicked off ~3 weeks ago. This post will provide more detail on models, motivation, the plan, etc, including a section for Aysajan to introduce himself. Background Models First background model: Specializing in Problems We Don’t Understand. Problems-we-don’t-understand are similar to each other in a way which problems-we-do-understand are not. In the context of scientific research, preparadigmatic research in different fields is similar in a way which research within a paradigm is not. There are general skills and knowledge useful for finding/creating structure de novo, as opposed to working within some already-mapped structure. Furthermore, while problems-we-don’t-understand may require some specialized knowledge, specialized knowledge of the field is never the rate-limiting step; if it were, then the problem would already be tractable to people steeped in the existing specialized knowledge of the field. If a problem is tractable within the current paradigm, then it isn’t preparadigmatic. Broad, generalizable skills/knowledge are much more important for problems-we-don’t-understand than for problems-we-do-understand. The linked post goes into more detail on how one can train and specialize in problems-we-don’t-understand. Second background model: Selection Has A Quality Ceiling. If we want people with a lot of skill in a lot of areas, trying to hire such people directly is Hard, in a big-O sense. As the number of traits we’re filtering for increases, the number of people we have to test in order to find one with all the requisite traits increases exponentially. The big-O requirements training skills are much better: as long as learning one skill doesn’t make another harder, the time required to train all of them should increase at-most linearly with the number skills. Alas, most schools/companies today seem to mostly select, rather than train. Which makes sense - most companies don’t really need people with lots of skill in lots of areas, they just need people who will pick up the particulars of their industry quickly as-needed. But for problems-we-don’t-understand, people with lots of skill in lots of areas are exactly what we want. Third background model: illegible skills. A lot of key skills/knowledge are hard to transmit by direct explanation. They’re not necessarily things which a teacher would even notice enough to consider important - just background skills or knowledge which is so ingrained that it becomes invisible. This sort of skill/knowledge is most easily transmitted by exposure: demonstration by the teacher, experimentation by the student, and feedback, ideally on a day-to-day basis. Thus the importance of an apprenticeship-like structure: high exposure and one-on-one interaction helps transmit illegible skills/knowledge. (I suspect that this also relates to Bloom’s two-sigma problem: one-on-one tutoring works about two standard deviations better than anything else in education. Regardless of whether illegible skill transmission is actually a core part of that phenomenon, ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Outline of Galef's Scout Mindset , published by Rob Bensinger on the LessWrong. Julia Galef's The Scout Mindset is superb. For effective altruists, I think (based on the topic and execution) it's straightforwardly the #1 book you should use when you want to recruit new people to EA. It doesn't actually talk much about EA, but I think starting people on this book will result in an EA that's thriving more and doing more good five years from now, compared to the future EA that would exist if the top go-to resource were more obvious choices like The Precipice, Doing Good Better, the EA Handbook, etc. For rationalists, I think the best intro resource is still HPMoR or R:AZ, but I think Scout Mindset is a great supplement to those, and probably a better starting point for people who prefer Julia's writing style over Eliezer's. I've made an outline of the book below, for my own reference and for others who have read it. If you don't mind spoilers, you can also use this to help decide whether the book's worth reading for you, though my summary skips a lot and doesn't do justice to Julia's arguments. Introduction Scout mindset is "the motivation to see things as they are, not as you wish they were". We aren't perfect scouts, but we can improve. "My approach has three prongs": Realize that truth isn't in conflict with your other goals. People tend to overestimate how useful self-deception is for things like personal happiness and motivation, starting a company, being an activist, etc. Learn tools that make it easier to see clearly. Use various kinds of thought experiments and probabilistic reasoning, and rethink how you go about listening to the "other side" of an issue. Appreciate the emotional rewards of scout mindset. "It's empowering to be able to resist the temptation to self-deceive, and to know that you can face reality even when it's unpleasant. There's an equanimity that results from understanding risk and coming to terms with the odds you're facing. And there's a refreshing lightness in the feeling of being free to explore ideas and follow the evidence wherever it leads". Looking at lots of real-world examples of people who have exemplified scout mindset can make these positives more salient. PART I: The Case for Scout Mindset Chapter 1. Two Types of Thinking "Can I believe it?" vs. "must I believe it?" In directionally motivated reasoning, often shortened to "motivated reasoning", we disproportionately put our effort into finding evidence/reasons that support what we wish were true. Reasoning as defensive combat. Motivated reasoning, a.k.a. soldier mindset, "doesn't feel like motivated reasoning from the inside". But it's extremely common, as shown by how often we describe our reasoning in militaristic terms. "Is it true?" An alternative to (directionally) motivated reasoning is accuracy motivated reasoning, i.e., scout mindset. Your mindset can make or break your judgment. This stuff matters in real life, in almost every domain. Nobody is purely a scout or purely a soldier, but it's possible to become more scout-like. Chapter 2. What the Soldier is Protecting "[I]f scout mindset is so great, why isn't everyone already using it all the time?" Three emotional reasons: Comfort: avoiding unpleasant emotions. This even includes comforting pessimism: "there's no hope, so you might as well not worry about it." Self-esteem: feeling good about ourselves. Again, this can include ego-protecting negativity and avoiding "'getting my hopes up'". Morale: motivating ourselves to do hard things. And three social reasons: Persuasion: convincing ourselves so we can convince others. Image: choosing beliefs that make us look good. "Psychologists call it impression management, and evolutionary psychologists call it signaling: When considering a claim, we implicitly ask ourselves, 'What ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How to Not Lose an Argument , published by Scott Alexander on the LessWrong. Related to: Leave a Line of Retreat Followup to: Talking Snakes: A Cautionary Tale, The Skeptic's Trilemma "I argue very well. Ask any of my remaining friends. I can win an argument on any topic, against any opponent. People know this, and steer clear of me at parties. Often, as a sign of their great respect, they don't even invite me." --Dave Barry The science of winning arguments is called Rhetoric, and it is one of the Dark Arts. Its study is forbidden to rationalists, and its tomes and treatises are kept under lock and key in a particularly dark corner of the Miskatonic University library. More than this it is not lawful to speak. But I do want to talk about a very closely related skill: not losing arguments. Rationalists probably find themselves in more arguments than the average person. And if we're doing it right, the truth is hopefully on our side and the argument is ours to lose. And far too often, we do lose arguments, even when we're right. Sometimes it's because of biases or inferential distances or other things that can't be helped. But all too often it's because we're shooting ourselves in the foot. How does one avoid shooting one's self in the foot? In rationalist language, the technique is called Leaving a Social Line of Retreat. In normal language, it's called being nice. First, what does it mean to win or lose an argument? There is an unspoken belief in some quarters that the point of an argument is to gain social status by utterly demolishing your opponent's position, thus proving yourself the better thinker. That can be fun sometimes, and if it's really all you want, go for it. But the most important reason to argue with someone is to change his mind. If you want a world without fundamentalist religion, you're never going to get there just by making cutting and incisive critiques of fundamentalism that all your friends agree sound really smart. You've got to deconvert some actual fundamentalists. In the absence of changing someone's mind, you can at least get them to see your point of view. Getting fundamentalists to understand the real reasons people find atheism attractive is a nice consolation prize. I make the anecdotal observation that a lot of smart people are very good at winning arguments in the first sense, and very bad at winning arguments in the second sense. Does that correspond to your experience? Back in 2008, Eliezer described how to Leave a Line of Retreat. If you believe morality is impossible without God, you have a strong disincentive to become an atheist. Even after you've realized which way the evidence points, you'll activate every possible defense mechanism for your religious beliefs. If all the defense mechanisms fail, you'll take God on utter faith or just believe in belief, rather than surrender to the unbearable position of an immoral universe. The correct procedure for dealing with such a person, Eliezer suggests, isn't to show them yet another reason why God doesn't exist. They'll just reject it along with all the others. The correct procedure is to convince them, on a gut level, that morality is possible even in a godless universe. When disbelief in God is no longer so terrifying, people won't fight it quite so hard and may even deconvert themselves. But there's another line of retreat to worry about, one I experienced firsthand in a very strange way. I had a dream once where God came down to Earth; I can't remember exactly why. In the borderlands between waking and sleep, I remember thinking: I feel like a total moron. Here I am, someone who goes to atheist groups and posts on atheist blogs and has told all his friends they should be atheists and so on, and now it turns out God exists. All of my religious friends whom I won all those argumen...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Firewalling the Optimal from the Rational, published by Eliezer Yudkowsky on the LessWrong. Followup to: Rationality: Appreciating Cognitive Algorithms (minor post) There's an old anecdote about Ayn Rand, which Michael Shermer recounts in his "The Unlikeliest Cult in History" (note: calling a fact unlikely is an insult to your prior model, not the fact itself), which went as follows: Branden recalled an evening when a friend of Rand's remarked that he enjoyed the music of Richard Strauss. "When he left at the end of the evening, Ayn said, in a reaction becoming increasingly typical, 'Now I understand why he and I can never be real soulmates. The distance in our sense of life is too great.' Often she did not wait until a friend had left to make such remarks." Many readers may already have appreciated this point, but one of the Go stones placed to block that failure mode is being careful what we bless with the great community-normative-keyword 'rational'. And one of the ways we do that is by trying to deflate the word 'rational' out of sentences, especially in post titles or critical comments, which can live without the word. As you hopefully recall from the previous post, we're only forced to use the word 'rational' when we talk about the cognitive algorithms which systematically promote goal achievement or map-territory correspondences. Otherwise the word can be deflated out of the sentence; e.g. "It's rational to believe in anthropogenic global warming" goes to "Human activities are causing global temperatures to rise"; or "It's rational to vote for Party X" deflates to "It's optimal to vote for Party X" or just "I think you should vote for Party X". If you're writing a post comparing the experimental evidence for four different diets, that's not "Rational Dieting", that's "Optimal Dieting". A post about rational dieting is if you're writing about how the sunk cost fallacy causes people to eat food they've already purchased even if they're not hungry, or if you're writing about how the typical mind fallacy or law of small numbers leads people to overestimate how likely it is that a diet which worked for them will work for a friend. And even then, your title is 'Dieting and the Sunk Cost Fallacy', unless it's an overview of four different cognitive biases affecting dieting. In which case a better title would be 'Four Biases Screwing Up Your Diet', since 'Rational Dieting' carries an implication that your post discusses the cognitive algorithm for dieting, as opposed to four contributing things to keep in mind. By the same token, a post about Givewell's top charities and how they compare to existential-risk mitigation is a post about optimal philanthropy, while a post about scope insensitivity and hedonic returns vs. marginal returns is a post about rational philanthropy, because the first is discussing object-level outcomes while the second is discussing cognitive algorithms. And either way, if you can have a post title that doesn't include the word "rational", it's probably a good idea because the word gets a little less powerful every time it's used. Of course, it's still a good idea to include concrete examples when talking about general cognitive algorithms. A good writer won't discuss rational philanthropy without including some discussion of particular charities to illustrate the point. In general, the concrete-abstract writing pattern says that your opening paragraph should be a concrete example of a nonoptimal charity, and only afterward should you generalize to make the abstract point. (That's why the main post opened with the Ayn Rand anecdote.) And I'm not saying that we should never have posts about Optimal Dieting on LessWrong. What good is all that rationality if it never leads us to anything optimal? Nonetheless, the second Go stone placed to block the...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Thou Art Godshatter, published by Eliezer Yudkowsky on the LessWrong. Before the 20th century, not a single human being had an explicit concept of "inclusive genetic fitness", the sole and absolute obsession of the blind idiot god. We have no instinctive revulsion of condoms or oral sex. Our brains, those supreme reproductive organs, don't perform a check for reproductive efficacy before granting us sexual pleasure. Why not? Why aren't we consciously obsessed with inclusive genetic fitness? Why did the Evolution-of-Humans Fairy create brains that would invent condoms? "It would have been so easy," thinks the human, who can design new complex systems in an afternoon. The Evolution Fairy, as we all know, is obsessed with inclusive genetic fitness. When she decides which genes to promote to universality, she doesn't seem to take into account anything except the number of copies a gene produces. (How strange!) But since the maker of intelligence is thus obsessed, why not create intelligent agents - you can't call them humans - who would likewise care purely about inclusive genetic fitness? Such agents would have sex only as a means of reproduction, and wouldn't bother with sex that involved birth control. They could eat food out of an explicitly reasoned belief that food was necessary to reproduce, not because they liked the taste, and so they wouldn't eat candy if it became detrimental to survival or reproduction. Post-menopausal women would babysit grandchildren until they became sick enough to be a net drain on resources, and would then commit suicide. It seems like such an obvious design improvement - from the Evolution Fairy's perspective. Now it's clear, as was discussed yesterday, that it's hard to build a powerful enough consequentialist. Natural selection sort-of reasons consequentially, but only by depending on the actual consequences. Human evolutionary theorists have to do really high-falutin' abstract reasoning in order to imagine the links between adaptations and reproductive success. But human brains clearly can imagine these links in protein. So when the Evolution Fairy made humans, why did It bother with any motivation except inclusive genetic fitness? It's been less than two centuries since a protein brain first represented the concept of natural selection. The modern notion of "inclusive genetic fitness" is even more subtle, a highly abstract concept. What matters is not the number of shared genes. Chimpanzees share 95% of your genes. What matters is shared genetic variance, within a reproducing population - your sister is one-half related to you, because any variations in your genome, within the human species, are 50% likely to be shared by your sister. Only in the last century - arguably only in the last fifty years - have evolutionary biologists really begun to understand the full range of causes of reproductive success, things like reciprocal altruism and costly signaling. Without all this highly detailed knowledge, an intelligent agent that set out to "maximize inclusive fitness" would fall flat on its face. So why not preprogram protein brains with the knowledge? Why wasn't a concept of "inclusive genetic fitness" programmed into us, along with a library of explicit strategies? Then you could dispense with all the reinforcers. The organism would be born knowing that, with high probability, fatty foods would lead to fitness. If the organism later learned that this was no longer the case, it would stop eating fatty foods. You could refactor the whole system. And it wouldn't invent condoms or cookies. This looks like it should be quite possible in principle. I occasionally run into people who don't quite understand consequentialism, who say, "But if the organism doesn't have a separate drive to eat, it will starve, and so fail to reproduce." So long...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Oops on Commodity Prices, published by sarahconstantin on the LessWrong. Epistemic status: Casual Some patient and thoughtful folks on LessWrong, and, apparently, some rather less patient folks on r/SneerClub, have pointed out that GDP-to-gold, or GDP-to-oil, are bad proxy measures for economic growth. Ok, this is a counterargument I want to make sure I understand. Is the following a good representation of what you believe? When you divide GDP by a commodity price, when the commodity has a nearly-fixed supply (like gold or land) we’d expect the price of the commodity to go up over time in a society that’s getting richer — in other words, if you have better tech and better and more abundant goods, but not more gold or land, you’d expect that other goods would become cheaper relative to gold or land. Thus, a GDP/gold or GDP/land value that doesn’t increase over time is totally consistent with a society with increasing “true” wealth, and thus doesn’t indicate stagnation. paulfchristiano: Yes. The detailed dynamics depend a lot on the particular commodity, and how elastic we expect demand to be; for example, over the long run I expect GDP/oil to go way up as we move to better substitutes, but over a short period where there aren’t good substitutes it could stay flat. Commenters on this blog have also pointed out that the Dow is a poor measure of the value of the stock market, since it’s small and unnormalized. These criticisms weaken my previous claim about economic growth being stagnant. Now, a little personal story time: Nearly ten years ago (yikes!) in college, I had an econ blog. My big brush with fame was having a joke of mine hat-tipped by Megan McArdle once. I did most of the required courses for an econ major, before eventually settling on math. My blog, I realized with dismay when I pulled it up many years later, consisted almost entirely of me agreeing with other econ bloggers I encountered, and imitating buzzwords. I certainly sounded a lot more mainstream in those days, but I understood — if possible — less economics than I do now. I couldn’t use what I’d learned in school to reason about real-world questions. I think I learn a heck of a lot more by throwing an idea out there and being corrected than I did back when I was not even asking questions. “A shy person cannot learn, an impatient person cannot teach” and all that. Admittedly, my last post may have sounded more know-it-all-ish than it actually deserved, and that’s a problem to the extent that I accidentally misled people (despite my disclaimers.) I actually tried, for several years, to be less outspoken and convey less confidence in my written voice. My impression is that the attempt didn’t work for me, and caused me some emotional and intellectual damage in the meanwhile. I think verbally; if I try to verbalize less, I think less. I think the M.O. that works better for me is strong opinions, weakly held. I do try to learn from knowledgeable people and quickly walk back my errors. But realistically, I’m going to make errors, and dumber ones when I’m newer to learning about a topic. To those who correct me and explain why — thank you. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: I'm leaving AI alignment – you better stay, published by rmoehn on the LessWrong. Write a Review Requirements for independent AI alignment research and how they are connected This diagram summarizes the requirements for independent AI alignment research and how they are connected. In this post I'll outline my four-year-long attempt at becoming an AI alignment researcher. It's an ‘I did X [including what I did wrong], and here's how it went’ post (see also jefftk's More writeups!). I'm not complaining about how people treated me – they treated me well. And I'm not trying to convince you to abandon AI alignment research – you shouldn't. I'm not saying that anyone should have done anything differently – except myself. Requirements Funding Funding is the main requirement, because it enables everything else. Thanks to Paul Christiano I had funding for nine months between January 2019 and January 2020. Thereafter I applied to the EA Foundation Fund (now Center on Long-Term Risk Fund) and Long-Term Future Fund for a grant and they rejected my applications. Now I don't know of any other promising sources of funding. I also don't know of any AI alignment research organisation that would hire me as a remote worker. How much funding you need varies. I settled on 5 kUSD per month, which sounds like a lot when you're a student, and which sounds like not a lot when you look at market rates for software developers/ML engineers/ML researchers. On top of that, I'm essentially a freelancer who has to pay social insurance by himself, take time off to do accounting and taxes, and build runway for dry periods. Results and relationships In any job you must get results and build relationships. If you don't, you don't earn your pay. (Manager Tools talks about results and relationships all the time. See for example What You've Been Taught About Management is Wrong or First Job Fundamentals.) The results I generated weren't obviously good enough to compel Paul to continue to fund me. And I didn't build good enough relationships with people who could have convinced the LTFF and EAFF fund managers that I have the potential they're looking for. Time Funding buys time, which I used for study and research. Another aspect of time is how effectively and efficiently you use it. I'm good at effective, not so good at efficient. – I spend much time on non-research, mostly studying Japanese and doing sports. And dawdling. I noticed the dawdling problem at the end of last year and got it under control at the beginning of this year (see my time tracking). Too late. Added 2020-03-16: I also need a lot of sleep in order to do this kind of work. – About 8.5 h per day. Travel and location I live in Kagoshima City in southern Japan, which is far away from the AI alignment research hubs. This means that I don't naturally meet AI alignment researchers and build relationships with them. I could have compensated for this by travelling to summer schools, conferences etc. But I missed the best opportunities and I felt that I didn't have the time and money to take the second-best opportunities. Of course, I could also relocate to one of the research hubs. But I don't want to do that for family reasons. I did start maintaining the Predicted AI alignment event/meeting calendar in order to avoid missing opportunities again. And I did apply and get accepted to the AI Safety Camp Toronto 2020. They even chose my research proposal for one of the teams. But I failed to procure the funding that would have supported me from March through May when the camp takes place. Knowledge I know more than most young AI alignment researchers about how to make good software, how to write well and how to work professionally. I know less than most young AI alignment researchers about maths, ML and how to do research. The latter appear to be ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Choosing the Zero Point, published by orthonormal on the LessWrong. Write a Review Summary: You can decide what state of affairs counts as neutral, and what counts as positive or negative. Bad things happen if humans do that in our natural way. It's more motivating and less stressful if, when we learn something new, we update the neutral point to [what we think the world really is like now]. A few years back, I read an essay by Rob Bensinger about vegetarianism/veganism, and it convinced me to at least eat much less meat. This post is not about that topic. It's about the way that essay differed, psychologically, from many others I've seen on the same topic, and the general importance of that difference. Rob's essay referred to the same arguments I'd previously seen, but while other essays concluded with the implication "you're doing great evil by eating meat, and you need to realize what a monster you've been and immediately stop", Rob emphasized the following: Frame animal welfare activism as an astonishingly promising, efficient, and uncrowded opportunity to do good. Scale back moral condemnation and guilt. LessWrong types can be powerful allies, but the way to get them on board is to give them opportunities to feel like munchkins with rare secret insights, not like latecomers to a not-particularly-fun party who have to play catch-up to avoid getting yelled at. It’s fine to frame helping animals as challenging, but the challenge should be to excel and do something astonishing, not to meet a bare standard for decency. That shouldn't have had different effects on me than other essays, but damned if it didn't. Consider a utilitarian Ursula with a utility function U. U is defined over all possible ways the world could be, and for each of those ways it gives you a number. Ursula's goal is to maximize the expected value of U. Now consider the utility function V, where V always equals U + 1. If a utilitarian Vader with utility function V is facing the same choice (in another universe) as Ursula, then because that +1 applies to every option equally, the right choice for Vader is the same as the right choice for Ursula. The constant difference between U and V doesn't matter for any decision whatsoever! We represent this by saying that a utility function is only defined up to positive affine transformations. (That means you can also multiply U by any positive number and it still won't change a utilitarian's choices.) But humans aren't perfect utilitarians, in many interesting ways. One of these is that our brains have a natural notion of outcomes that are good and outcomes that are bad, and the neutral zero point is more or less "the world I interact with every day". So if we're suddenly told about a nearby bottomless pit of suffering, what happens? Our brains tend to hear, "Instead of the zero point where we thought we were, this claim means that we're really WAY DOWN IN THE NEGATIVE ZONE". A few common reactions to this: Denial. "Nope nope that argument can't be true, I'm sure there's a flaw in it, we're definitely still in the normal zone" Guilt. "AAAAHHHH I need to drop everything and work super hard on this, I can't allow myself any distractions or any bit of happiness until this is completely fixed" Despair. "Oh no, there's no way I could get things back up to normal from here, I can't do anything, I'll just sit here and hate myself" The thing about Rob's post is that it suggested an alternative. Instead of keeping the previous zero point and defining yourself as now being very far below it, you can reset yourself to take the new way-the-world-is as the zero point. Again, this doesn't change any future choice a utilitarian you would make! But it does buy human you peace of mind. What is true is already so- the world was like this even when you didn't believe it. The ps...
Welcome to The Nonlinear Library, where we use Text-to-SpeecWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Transportation as a Constraint, published by johnswentworth on the LessWrong Write a Review Imagine it’s late autumn of 332 BC. You’re Alexander the Great, and your armies are marching toward Egypt from Gaza. There’s just one little problem: you need to cross the Sinai peninsula - 150 miles of hot, barren desert. How will you carry food and water for the troops? Green triangle on the left is the Nile river delta in Egypt; green chunk in the upper right is Israel. The big desert peninsula between them is the Sinai. Option 1: carry it A physically-active human needs about 3 lbs of food per day. (Modern hikers can probably find lighter calorie-dense foodstuffs, but we’re talking ancient history here.) Water requirements vary; 5 lbs is a minimum, but the US Army Quartermaster Corps recommends 20 lbs/day when marching through a hot desert. Alexander’s army crossed the desert in 7 days. Food might be reasonable, but to carry the water would mean 720 = 140 lbs per person, plus 50+ lbs of armor, weapons, etc. When I go hiking, I aim for a 20-30 lb pack. US marines are apparently expected to be able to carry 150 lbs for 9 miles - quite a bit less than the 20+ miles/day Alexander’s army managed, and with no comment on how long the marine in question might need to rest afterwards. (Also, I’m not sure I trust that source - 150 lbs for 9 miles sounds unrealistic to me, and if it’s true then I’m very impressed by marines.) Suffice to say that carrying that much water across that much desert is not a realistic option, even if we drink it along the way. Option 2: horses A horse consumes 20 lbs of food (half of which may be forage) and 80 lbs of water per day. In exchange, it can carry about 200 lbs (surprisingly, my source claims that horses can carry more than they can pull). Of course, that 200 lbs has to include the horse’s own food and water, plus whatever useful load it’s carrying. So, marching through a desert, a horse can only transport (200 lbs)/(80+20 lbs/day) = 2 days of supplies for itself, and that’s before whatever useful things actually need to be transported. In other words, there’s a hard upper limit on how far goods can be transported by horse without refilling supplies along the way. That limit is around 2 days travel time without any refill, 10 days if there’s plenty of fresh water along the route, or 20 days if there’s both water and forage. At 20 miles/day, that’s 40, 200, or 400 miles. Realistically, if we want the number of horses to be reasonable, the limit is more like half that much - 20 miles, 100 miles, or 200 miles, respectively. So horses also won’t work. Option 2.5: camels or other pack animals Contrary to popular image, camels actually need more water than horses. They can go a couple days without, but then need to fill up all at once. They can also carry a bit more weight, but they eat more food. At the end of the day, the numbers end up quite similar. Mules also end up with similar numbers, and cattle are generally worse. Option 3: ships Assuming the army marches along the coast, a supply fleet can sail alongside. At the time, a single large merchant ship could carry 400 tons - in other words, as much as about 4000 horses. Presumably the ship would cost a lot less than the horses, too. Well then, there’s our answer. Ships are clearly a vastly superior way to move goods. Range is a non-issue, capacity is far larger, and they’re far cheaper. They’re perfect for crossing the Sinai, which runs right along the coast anyway. Fast forward a few years to 327 BC, and Alexander is marching his armies back from India. He plans to cross the Gedrosian desert, along the coast of modern-day Pakistan and Iran. The plan is much like the Sinai: a supply fleet will sail alongside the army. Unfortunately, neit...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Purchase Fuzzies and Utilons Separately, published by Eliezer Yudkowsky on the LessWrong. Yesterday: There is this very, very old puzzle/observation in economics about the lawyer who spends an hour volunteering at the soup kitchen, instead of working an extra hour and donating the money to hire someone... If the lawyer needs to work an hour at the soup kitchen to keep himself motivated and remind himself why he's doing what he's doing, that's fine. But he should also be donating some of the hours he worked at the office, because that is the power of professional specialization and it is how grownups really get things done. One might consider the check as buying the right to volunteer at the soup kitchen, or validating the time spent at the soup kitchen. I hold open doors for little old ladies. I can't actually remember the last time this happened literally (though I'm sure it has, sometime in the last year or so). But within the last month, say, I was out on a walk and discovered a station wagon parked in a driveway with its trunk completely open, giving full access to the car's interior. I looked in to see if there were packages being taken out, but this was not so. I looked around to see if anyone was doing anything with the car. And finally I went up to the house and knocked, then rang the bell. And yes, the trunk had been accidentally left open. Under other circumstances, this would be a simple act of altruism, which might signify true concern for another's welfare, or fear of guilt for inaction, or a desire to signal trustworthiness to oneself or others, or finding altruism pleasurable. I think that these are all perfectly legitimate motives, by the way; I might give bonus points for the first, but I wouldn't deduct any penalty points for the others. Just so long as people get helped. But in my own case, since I already work in the nonprofit sector, the further question arises as to whether I could have better employed the same sixty seconds in a more specialized way, to bring greater benefit to others. That is: can I really defend this as the best use of my time, given the other things I claim to believe? The obvious defense—or perhaps, obvious rationalization—is that an act of altruism like this one acts as an willpower restorer, much more efficiently than, say, listening to music. I also mistrust my ability to be an altruist only in theory; I suspect that if I walk past problems, my altruism will start to fade. I've never pushed that far enough to test it; it doesn't seem worth the risk. But if that's the defense, then my act can't be defended as a good deed, can it? For these are self-directed benefits that I list. Well—who said that I was defending the act as a selfless good deed? It's a selfish good deed. If it restores my willpower, or if it keeps me altruistic, then there are indirect other-directed benefits from that (or so I believe). You could, of course, reply that you don't trust selfish acts that are supposed to be other-benefiting as an "ulterior motive"; but then I could just as easily respond that, by the same principle, you should just look directly at the original good deed rather than its supposed ulterior motive. Can I get away with that? That is, can I really get away with calling it a "selfish good deed", and still derive willpower restoration therefrom, rather than feeling guilt about it being selfish? Apparently I can. I'm surprised it works out that way, but it does. So long as I knock to tell them about the open trunk, and so long as the one says "Thank you!", my brain feels like it's done its wonderful good deed for the day. Your mileage may vary, of course. The problem with trying to work out an art of willpower restoration is that different things seem to work for different people. (That is: We're probing around on the level of surf...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Knowing About Biases Can Hurt People, published by Eliezer Yudkowsky on the LessWrong. Once upon a time I tried to tell my mother about the problem of expert calibration, saying: “So when an expert says they’re 99% confident, it only happens about 70% of the time.” Then there was a pause as, suddenly, I realized I was talking to my mother, and I hastily added: “Of course, you’ve got to make sure to apply that skepticism evenhandedly, including to yourself, rather than just using it to argue against anything you disagree with—” And my mother said: “Are you kidding? This is great! I’m going to use it all the time!” Taber and Lodge’s “Motivated Skepticism in the Evaluation of Political Beliefs” describes the confirmation of six predictions: Prior attitude effect. Subjects who feel strongly about an issue—even when encouraged to be objective—will evaluate supportive arguments more favorably than contrary arguments. Disconfirmation bias. Subjects will spend more time and cognitive resources denigrating contrary arguments than supportive arguments. Confirmation bias. Subjects free to choose their information sources will seek out supportive rather than contrary sources. Attitude polarization. Exposing subjects to an apparently balanced set of pro and con arguments will exaggerate their initial polarization. Attitude strength effect. Subjects voicing stronger attitudes will be more prone to the above biases. Sophistication effect. Politically knowledgeable subjects, because they possess greater ammunition with which to counter-argue incongruent facts and arguments, will be more prone to the above biases. If you’re irrational to start with, having more knowledge can hurt you. For a true Bayesian, information would never have negative expected utility. But humans aren’t perfect Bayes-wielders; if we’re not careful, we can cut ourselves. I’ve seen people severely messed up by their own knowledge of biases. They have more ammunition with which to argue against anything they don’t like. And that problem—too much ready ammunition—is one of the primary ways that people with high mental agility end up stupid, in Stanovich’s “dysrationalia” sense of stupidity. You can think of people who fit this description, right? People with high g-factor who end up being less effective because they are too sophisticated as arguers? Do you think you’d be helping them—making them more effective rationalists—if you just told them about a list of classic biases? I recall someone who learned about the calibration/overconfidence problem. Soon after he said: “Well, you can’t trust experts; they’re wrong so often—as experiments have shown. So therefore, when I predict the future, I prefer to assume that things will continue historically as they have—” and went off into this whole complex, error-prone, highly questionable extrapolation. Somehow, when it came to trusting his own preferred conclusions, all those biases and fallacies seemed much less salient—leapt much less readily to mind—than when he needed to counter-argue someone else. I told the one about the problem of disconfirmation bias and sophisticated argument, and lo and behold, the next time I said something he didn’t like, he accused me of being a sophisticated arguer. He didn’t try to point out any particular sophisticated argument, any particular flaw—just shook his head and sighed sadly over how I was apparently using my own intelligence to defeat itself. He had acquired yet another Fully General Counterargument. Even the notion of a “sophisticated arguer” can be deadly, if it leaps all too readily to mind when you encounter a seemingly intelligent person who says something you don’t like. I endeavor to learn from my mistakes. The last time I gave a talk on heuristics and biases, I started out by introducing the general concept by way of t...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Double Crux — A Strategy for Mutual Understanding, published by Duncan_Sabien on the LessWrong. Preamble Double crux is one of CFAR's newer concepts, and one that's forced a re-examination and refactoring of a lot of our curriculum (in the same way that the introduction of TAPs and Inner Simulator did previously). It rapidly became a part of our organizational social fabric, and is one of our highest-EV threads for outreach and dissemination, so it's long overdue for a public, formal explanation. Note that while the core concept is fairly settled, the execution remains somewhat in flux, with notable experimentation coming from Julia Galef, Kenzi Amodei, Andrew Critch, Eli Tyre, Anna Salamon, myself, and others. Because of that, this post will be less of a cake and more of a folk recipe—this is long and meandering on purpose, because the priority is to transmit the generators of the thing over the thing itself. Accordingly, if you think you see stuff that's wrong or missing, you're probably onto something, and we'd appreciate having them added here as commentary. Casus belli To a first approximation, a human can be thought of as a black box that takes in data from its environment, and outputs beliefs and behaviors (that black box isn't really "opaque" given that we do have access to a lot of what's going on inside of it, but our understanding of our own cognition seems uncontroversially incomplete). When two humans disagree—when their black boxes output different answers, as below—there are often a handful of unproductive things that can occur. The most obvious (and tiresome) is that they'll simply repeatedly bash those outputs together without making any progress (think most disagreements over sports or politics; the people above just shouting "triangle!" and "circle!" louder and louder). On the second level, people can (and often do) take the difference in output as evidence that the other person's black box is broken (i.e. they're bad, dumb, crazy) or that the other person doesn't see the universe clearly (i.e. they're biased, oblivious, unobservant). On the third level, people will often agree to disagree, a move which preserves the social fabric at the cost of truth-seeking and actual progress. Double crux in the ideal solves all of these problems, and in practice even fumbling and inexpert steps toward that ideal seem to produce a lot of marginal value, both in increasing understanding and in decreasing conflict-due-to-disagreement. Prerequisites This post will occasionally delineate two versions of double crux: a strong version, in which both parties have a shared understanding of double crux and have explicitly agreed to work within that framework, and a weak version, in which only one party has access to the concept, and is attempting to improve the conversational dynamic unilaterally. In either case, the following things seem to be required: Epistemic humility. The number one foundational backbone of rationality seems, to me, to be how readily one is able to think "It's possible that I might be the one who's wrong, here." Viewed another way, this is the ability to take one's beliefs as object, rather than being subject to them and unable to set them aside (and then try on some other belief and productively imagine "what would the world be like if this were true, instead of that?"). Good faith. An assumption that people believe things for causal reasons; a recognition that having been exposed to the same set of stimuli would have caused one to hold approximately the same beliefs; a default stance of holding-with-skepticism what seems to be evidence that the other party is bad or wants the world to be bad (because as monkeys it's not hard for us to convince ourselves that we have such evidence when we really don't).1 Confidence in the existence of objective tr...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Attempted Telekinesis, published by on the LessWrong. Related to: Compartmentalization in epistemic and instrumental rationality; That other kind of status. Summary: I’d like to share some techniques that made a large difference for me, and for several other folks I shared them with. They are techniques for reducing stress, social shame, and certain other kinds of “wasted effort”. These techniques are less developed and rigorous than the techniques that CFAR teaches in our workshops -- for example, they currently only work for perhaps 1/3rd of the dozen or so people I’ve shared them with -- but they’ve made a large enough impact for that 1/3rd that I wanted to share them with the larger group. I’ll share them through a sequence of stories and metaphors, because, for now, that is what I have. For me, these techniques came out of a stressful time period. In October 2012, CFAR was very new, and I was very new to being its executive director. I was faced with a task that I basically didn't know how to do -- filling the first workshop for which we charged "real" money (the $3900/person that actually let CFAR run), and helping our team create our first decently polished workshop at the same time (which needed curriculum, operations, etc.). But whenever I sat down to try to work, my head would fill up with all the other tasks I “needed” to get done, instead of the particular task I was trying to work on. Or my head would fill with stress and mental static. So, almost because of how badly I needed to work, I found myself unable to accomplish much of anything. The set of stories and metaphors below is somehow what eventually gave me the ability to work with full focus in those conditions (I found them partway through that October), and cured most of my decades-long social shame at the same time.[1] (Though, again, this stuff isn't rigorous yet. It worked for a few folk, but failed a few others; your mileage may vary. Do share your thoughts.) Attempted telekinesis One morning, that month, I was lying in bed, half-asleep. And I wanted my laptop. But my laptop was a few feet away, so reaching it sounded hard (because I was half-asleep). After lying there a while wishing, I finally noticed what my brain was up to. And I noticed that what my brain was doing was visualizing my laptop whooshing toward me. Again and again. (Fix attention on laptop. visualize the woosh. Nope, laptop isn’t here yet: repeat!)[2] I’m going to call this process “Attempted telekinesis”. It seems to me that something like “attempted telekinesis” underlies a large set of stress / shame / worry / etc., and that learning to vanish it has been super-useful for me and several others. I’ll start with several examples of what I’ll be calling “attempted telekinesis”, and then go into some techniques for vanishing it. The case of the munching noises Later that day, I was sitting at the office trying to work, and someone next to me was eating. Noisily. Now, I’m part of the sizable minority of the population that is driven absolutely bonkers by munching noises. Munching noises fill me with rage and make me want to punch someone. But, like, I get that that’s petty of me. So my internal thinking stream goes something like this: Coworker: [Munch. Munch.] My system 1/ intuitive brain (silently, in my head): Argh! Stop it! Me: [Type, type.] (While thinking: “I don’t want to be petty; best not say anything, nor show annoyance on my face in any way.”) [1 minute later] Coworker: [Munch. Munch.] My system 1/ intuitive brain (silently, in my head): Didn’t you hear me?? Stop it!! Me: [Type, type.] (. I don’t want to be petty; best not say anything, or show it on my face in any way.) [and another minute later] Coworker: [Munch. Munch.] My system 1/ intuitive brain (silently, in my head): Argh!! Didn’t you hear me?? Stop ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: An Alien God, published by by Eliezer Yudkowsky on the LessWrong. "A curious aspect of the theory of evolution," said Jacques Monod, "is that everybody thinks he understands it." A human being, looking at the natural world, sees a thousand times purpose. A rabbit's legs, built and articulated for running; a fox's jaws, built and articulated for tearing. But what you see is not exactly what is there... In the days before Darwin, the cause of all this apparent purposefulness was a very great puzzle unto science. The Goddists said "God did it", because you get 50 bonus points each time you use the word "God" in a sentence. Yet perhaps I'm being unfair. In the days before Darwin, it seemed like a much more reasonable hypothesis. Find a watch in the desert, said William Paley, and you can infer the existence of a watchmaker. But when you look at all the apparent purposefulness in Nature, rather than picking and choosing your examples, you start to notice things that don't fit the Judeo-Christian concept of one benevolent God. Foxes seem well-designed to catch rabbits. Rabbits seem well-designed to evade foxes. Was the Creator having trouble making up Its mind? When I design a toaster oven, I don't design one part that tries to get electricity to the coils and a second part that tries to prevent electricity from getting to the coils. It would be a waste of effort. Who designed the ecosystem, with its predators and prey, viruses and bacteria? Even the cactus plant, which you might think well-designed to provide water fruit to desert animals, is covered with inconvenient spines. The ecosystem would make much more sense if it wasn't designed by a unitary Who, but, rather, created by a horde of deities—say from the Hindu or Shinto religions. This handily explains both the ubiquitous purposefulnesses, and the ubiquitous conflicts: More than one deity acted, often at cross-purposes. The fox and rabbit were both designed, but by distinct competing deities. I wonder if anyone ever remarked on the seemingly excellent evidence thus provided for Hinduism over Christianity. Probably not. Similarly, the Judeo-Christian God is alleged to be benevolent—well, sort of. And yet much of nature's purposefulness seems downright cruel. Darwin suspected a non-standard Creator for studying Ichneumon wasps, whose paralyzing stings preserve its prey to be eaten alive by its larvae: "I cannot persuade myself," wrote Darwin, "that a beneficent and omnipotent God would have designedly created the Ichneumonidae with the express intention of their feeding within the living bodies of Caterpillars, or that a cat should play with mice." I wonder if any earlier thinker remarked on the excellent evidence thus provided for Manichaen religions over monotheistic ones. By now we all know the punchline: You just say "evolution". I worry that's how some people are absorbing the "scientific" explanation, as a magical purposefulness factory in Nature. I've previously discussed the case of Storm from the movie X-Men, who in one mutation gets the ability to throw lightning bolts. Why? Well, there's this thing called "evolution" that somehow pumps a lot of purposefulness into Nature, and the changes happen through "mutations". So if Storm gets a really large mutation, she can be redesigned to throw lightning bolts. Radioactivity is a popular super origin: radiation causes mutations, so more powerful radiation causes more powerful mutations. That's logic. But evolution doesn't allow just any kind of purposefulness to leak into Nature. That's what makes evolution a success as an empirical hypothesis. If evolutionary biology could explain a toaster oven, not just a tree, it would be worthless. There's a lot more to evolutionary theory than pointing at Nature and saying, "Now purpose is allowed," or "Evolution did it!" The...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Risks from Learned Optimization: Introduction, published by evhub, Chris van Merwijk, vlad_m, Joar Skalse, Scott Garrabrant on the LessWrong. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. This is the first of five posts in the Risks from Learned Optimization Sequence based on the paper “Risks from Learned Optimization in Advanced Machine Learning Systems” by Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. Each post in the sequence corresponds to a different section of the paper. Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, and Joar Skalse contributed equally to this sequence. With special thanks to Paul Christiano, Eric Drexler, Rob Bensinger, Jan Leike, Rohin Shah, William Saunders, Buck Shlegeris, David Dalrymple, Abram Demski, Stuart Armstrong, Linda Linsefors, Carl Shulman, Toby Ord, Kate Woolverton, and everyone else who provided feedback on earlier versions of this sequence. Motivation The goal of this sequence is to analyze the type of learned optimization that occurs when a learned model (such as a neural network) is itself an optimizer—a situation we refer to as mesa-optimization, a neologism we introduce in this sequence. We believe that the possibility of mesa-optimization raises two important questions for the safety and transparency of advanced machine learning systems. First, under what circumstances will learned models be optimizers, including when they should not be? Second, when a learned model is an optimizer, what will its objective be—how will it differ from the loss function it was trained under—and how can it be aligned? We believe that this sequence presents the most thorough analysis of these questions that has been conducted to date. In particular, we present not only an introduction to the basic concerns surrounding mesa-optimizers, but also an analysis of the particular aspects of an AI system that we believe are likely to make the problems related to mesa-optimization relatively easier or harder to solve. By providing a framework for understanding the degree to which different AI systems are likely to be robust to misaligned mesa-optimization, we hope to start a discussion about the best ways of structuring machine learning systems to solve these problems. Furthermore, in the fourth post we will provide what we think is the most detailed analysis yet of a problem we refer as deceptive alignment which we posit may present one of the largest—though not necessarily insurmountable—current obstacles to producing safe advanced machine learning systems using techniques similar to modern machine learning. Two questions In machine learning, we do not manually program each individual parameter of our models. Instead, we specify an objective function that captures what we want the system to do and a learning algorithm to optimize the system for that objective. In this post, we present a framework that distinguishes what a system is optimized to do (its “purpose”), from what it optimizes for (its “goal”), if it optimizes for anything at all. While all AI systems are optimized for something (have a purpose), whether they actually optimize for anything (pursue a goal) is non-trivial. We will say that a system is an optimizer if it is internally searching through a search space (consisting of possible outputs, policies, plans, strategies, or similar) looking for those elements that score high according to some objective function that is explicitly represented within the system. Learning algorithms in machine learning are optimizers because they search through a space of possible parameters—e.g. neural network weights—and improve the parameters with respect to some objective. Planning algorithms are also optimizers, since they search through possible plans, picking thos...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Moral public goods, published by paulfchristiano on the LessWrong. Automatically crossposted Suppose that a kingdom contains a million peasants and a thousand nobles, and: Each noble makes as much as 10,000 peasants put together, such that collectively the nobles get 90% of the income. Each noble cares about as much about themselves as they do about all peasants put together. Each person’s welfare is logarithmic in their income. Then it’s simultaneously the case that: Nobles prefer to keep money for themselves rather than donate it to peasants—money is worth 10,000x as much to a peasant, but a noble cares 1,000,000 times less about the peasant’s welfare. Nobles prefer a 90% income tax that is redistributed equally—a tax that costs a particular noble $1 generates $1000 of value for peasants, since all other nobles will also pay the higher taxes. That makes it a much better deal for the nobles (until the total income of nobles is roughly equal to the total income of peasants). In this situation, let’s call redistribution a “moral public good.” The nobles are altruistic enough that they prefer it if everyone gives to the peasants, but it’s still not worth it for any given noble to contribute anything to the collective project. The rest of the post is about some implications of taking moral public good seriously. 1. Justifying redistribution This gives a very strong economic argument for state redistribution: it can easily be the case that every individual prefers a world with high redistribution to the world with low redistribution, rich and poor alike. I think “everyone prefers this policy” is basically the strongest argument you can make on its behalf. (In fact some people just don’t care about others and so not everyone will benefit. I’d personally be on board with the purely selfish people just not funding redistribution, but unfortunately you can’t just ask people if they want to pay more taxes and I’m not going to sweat it that much if the most selfish people lose out a little bit.) I think this argument supports levels of redistribution like 50% (or 30% or 70% or whatever), rather than levels of redistribution like 99% that could nearly level the playing field or ensure that no billionaires exist. I think this enough to capture the vast majority of the possible benefits from redistribution, e.g. they could get most households to >50% of the average consumption. This argument supports both foreign aid and domestic redistribution, but the foreign aid component may require international coordination. For example, if everyone in developed countries cared equally about themselves, their country, and the world, then you might end up with optimal domestic policies allocating 10% of their redistribution abroad (much less in smaller countries who have minimal influence on global poverty, a little bit more in the US), whereas everyone would prefer a multilateral commitment to spend 50% of their redistribution abroad. 2. There are lots of public goods I think it makes sense for states to directly fund moral public goods like existential risk mitigation, exploration, ecological preservation, arts and sciences, animal welfare improvements, etc. In the past I’ve thought it usually made more sense to just give people money and let them decide how to spend it. (I still think states and philanthropists should more often give people cash, I just now think the presumption is less strong.) In fact, I think that at large scales (like a nation rather than a town) moral public goods are probably the majority of public goods. Caring slightly more about public goods slightly changed my perspective on the state’s role. It also makes me significantly more excited about mechanisms like quadratic funding for public goods. I enjoyed David Friedman’s The Machinery of Freedom, but it repeats th...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Swiss Political System: More than You ever Wanted to Know (I.), published by Martin Sustrik on the LessWrong. Swiss political system may be best known for its extensive use of referenda. However, others may argue that its most striking feature is the ability to avoid political polarization. In this respect it may be unique among the western nations. That being said, it is hard to learn much about how it works. First, a big part of the system is informal and thus only discoverable by observing it personally or by asking the locals. Second, it's strongly decentralized. Different rules apply in different cantons and municipalities which makes the topic confusing to study. Third, Swiss aren't especially interested in promoting their own system abroad. A lot of the resources therefore exist only in local languages. In this article I'll try to put together what I've learned by living in the country, speaking to local people, following local press and studying the resources. Still, a disclaimer is due: I am not Swiss. I have lived here only for five years. Neither am I a political scientists or a sociologist. If you are Swiss, or simply know better than me, let me know about any inaccuracies in the article. On the more technical side of things: There's a lot of material to cover, and the result may be rather overwhelming. It would be a small book rather than a long article. Therefore, I am going to split this essay into three or four installments which I will publish one at a time. Semi-direct Democracy When modern Switzerland was established in 1848, it was a pretty standard representative democracy, mostly based on the American model. It's a federal state. Federal elections are held every four years. People are represented by political parties. There are two chambers of the parliament. Parliament elects members of the government, who then together run the country. The thriving ecosystem of various voluntary associations resembles the America that Alexis de Toqueville has written about. However, Switzerland is special in that various elements of direct democracy were introduced in the course of history. There are obligatory referenda: Any change in constitution, adjustment of taxes or joining any international organization must be approved by the people and the cantons. There are legislative referenda: Any law enacted by the parliament may be challenged and rejected in a referendum. Finally, there are so called "popular initiatives" which can propose a referendum on any topic. If the initiative manages to collect specified amount of signatures within specified amount of time the referendum is organized and the initiative may eventually get enacted. All of these referenda exist not only on the federal, but also on the cantonal and the municipal level. All of them are binding and neither of them needs a quorum. To understand the scope of the thing, consider that a 37-year-old from the city of Zurich who turned 18 in year 2000, has, in past 20 years, had the opportunity to take part in 548 referenda, 181 of them being on the federal, 176 on the cantonal and 191 on the municipal level. With the average turnout of 45% it means that they have voted in approximately 246 referenda. Due to their large number, individual referenda are not organized separately. Instead, they are voted on in batches, typically four times a year. To get a flavor of how it feels like, here's the batch from the city of Zurich in February 2020: popular Initiative "Affordable Housing": A sensitive issue especially in big cities like Zurich or Geneva, where rents are some of the most expensive in the world. The initiative proposes to build at least 10% of affordable, non-profit or cooperative flats, as well as a pre-emptive right for cantons and municipalities to buy land. It also proposes that infrastructu...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Matt Botvinick on the spontaneous emergence of learning algorithmsΩ, published by Adam Scholl on the LessWrong. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. Matt Botvinick is Director of Neuroscience Research at DeepMind. In this interview, he discusses results from a 2018 paper which describe conditions under which reinforcement learning algorithms will spontaneously give rise to separate full-fledged reinforcement learning algorithms that differ from the original. Here are some notes I gathered from the interview and paper: Initial Observation At some point, a group of DeepMind researchers in Botvinick’s group noticed that when they trained a RNN using RL on a series of related tasks, the RNN itself instantiated a separate reinforcement learning algorithm. These researchers weren’t trying to design a meta-learning algorithm—apparently, to their surprise, this just spontaneously happened. As Botvinick describes it, they started “with just one learning algorithm, and then another learning algorithm kind of... emerges, out of, like out of thin air”: "What happens... it seemed almost magical to us, when we first started realizing what was going on—the slow learning algorithm, which was just kind of adjusting the synaptic weights, those slow synaptic changes give rise to a network dynamics, and the dynamics themselves turn into a learning algorithm.” Other versions of this basic architecture—e.g., using slot-based memory instead of RNNs—seemed to produce the same basic phenomenon, which they termed "meta-RL." So they concluded that all that’s needed for a system to give rise to meta-RL are three very general properties: the system must 1) have memory, 2) whose weights are trained by a RL algorithm, 3) on a sequence of similar input data. From Botvinick’s description, it sounds to me like he thinks [learning algorithms that find/instantiate other learning algorithms] is a strong attractor in the space of possible learning algorithms: “...it's something that just happens. In a sense, you can't avoid this happening. If you have a system that has memory, and the function of that memory is shaped by reinforcement learning, and this system is trained on a series of interrelated tasks, this is going to happen. You can't stop it." Search for Biological Analogue This system reminded some of the neuroscientists in Botvinick’s group of features observed in brains. For example, like RNNs, the human prefrontal cortex (PFC) is highly recurrent, and the RL and RNN memory systems in their meta-RL model reminded them of “synaptic memory” and “activity-based memory.” They decided to look for evidence of meta-RL occuring in brains, since finding a neural analogue of the technique would provide some evidence they were on the right track, i.e. that the technique might scale to solving highly complex tasks. They think they found one. In short, they think that part of the dopamine system (DA) is a full-fledged reinforcement learning algorithm, which trains/gives rise to another full-fledged, free-standing reinforcement learning algorithm in PFC, in basically the same way (and for the same reason) the RL-trained RNNs spawned separate learning algorithms in their experiments. As I understand it, their story goes as follows: The PFC, along with the bits of basal ganglia and thalamic nuclei it connects to, forms a RNN. Its inputs are sensory percepts, and information about past actions and rewards. Its outputs are actions, and estimates of state value. DA[1] is a RL algorithm that feeds reward prediction error to PFC. Historically, people assumed the purpose of sending this prediction error was to update PFC’s synaptic weights. Wang et al. agree that this happens, but argue that the principle purpose of sending prediction error is to cause the creation of...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: the scaling “inconsistency”: openAI’s new insightΩ, published by nostalgebraist on the LessWrong. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. I’ve now read the new OpenAI scaling laws paper. Also, yesterday I attended a fun and informative lecture/discussion with one of the authors. While the topic is on my mind, I should probably jot down some of my thoughts. This post is mostly about what the new paper says about the “inconsistency” brought up in their previous paper. The new paper has a new argument on this topic, which is intuitive and appealing, and suggests that the current scaling trend will indeed “switch over” soon to a new one where dataset size, not model size, is the active constraint on performance. Most of this post is an attempt to explain and better understand this argument. The new paper is mainly about extending the scaling laws from their earlier paper to new modalities. In that paper, they found scaling laws for transformers trained autoregressively on text data. The new paper finds the same patterns in the scaling behavior of transformers trained autoregressively on images, math problems, etc. So the laws aren’t telling us something about the distribution of text data, but about something more fundamental. That’s cool. They also have a new, very intuitive hypothesis for what’s going on with the “scaling inconsistency” they described in the previous paper – the one I made a big deal about at the time. So that’s the part I’m most excited to discuss. I’m going to give a long explanation of it, way longer than the relevant part of their paper. Some of this is original to me, all errors are mine, all the usual caveats. 1. L(C) and L(D) To recap: the “inconsistency” is between two scaling laws: The law for the best you can do, given a fixed compute budget. This is L(C), sometimes called L(C_min). L is the loss (lower = better), C is your compute budget. The law for the best you can do, given a fixed dataset size. This is L(D), where D is the number of examples (say, tokens) in the dataset. Once you reach a certain level of compute, these two laws contradict each other. I’ll take some time to unpack that here, as it’s not immediately obvious the two can even be compared to one another – one is a function of compute, the other of data. 2. C sets E, and E bounds D Budget tradeoffs Given a compute budget C, you can derive the optimal way to spend it on different things. Roughly, you are trading off between two ways to spend compute: Use C to buy “N”: Training a bigger model – “N” here is model size Use C to buy “S”: Training for more steps “S” (gradient updates) The relationship between S (steps) and D (dataset size) is a little subtle, for several reasons. From step count to update count For one thing, each single “step” is an update on the information from more than one data point. Specifically, a step updates on “B” different points – B is the batch size. So the total number of data points processed during training is B times S. The papers sometimes call this quantity “E” (number of examples), so I’ll call it that too. From update count to data count Now, when you train an ML model, you usually update on each data point more than once. Typically, you’ll do one pass over the full dataset (updating on each point as you go along), then you’ll go back and do a second full pass, and then a third, etc. These passes are called “epochs.” If you’re doing things this way, then for every point in the data, you get (number of epochs) updates out of it. So E = (number of epochs) D. Some training routines don’t visit every point the exact same number of times – there’s nothing forcing you to do that. Still, for any training procedure, we can look at the quantity E / D. This would be the number of epochs, if you’re doing epoc...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Morality is Awesome, published by [anonymous] on the LessWrong. (This is a semi-serious introduction to the metaethics sequence. You may find it useful, but don't take it too seriously.) Meditate on this: A wizard has turned you into a whale. Is this awesome? Is it? "Maybe? I guess it would be pretty cool to be a whale for a day. But only if I can turn back, and if I stay human inside and so on. Also, that's not a whale. "Actually, a whale seems kind of specific, and I'd be suprised if that was the best thing the wizard can do. Can I have something else? Eternal happiness maybe?" Meditate on this: A wizard has turned you into orgasmium, doomed to spend the rest of eternity experiencing pure happiness. Is this awesome? "Kindof... That's pretty lame actually. On second thought I'd rather be the whale; at least that way I could explore the ocean for a while. "Let's try again. Wizard: maximize awesomeness." Meditate on this: A wizard has turned himself into a superintelligent god, and is squeezing as much awesomeness out of the universe as it could possibly support. This may include whales and starships and parties and jupiter brains and friendship, but only if they are awesome enough. Is this awesome? "Well, yes, that is awesome." What we just did there is called Applied Ethics. Applied ethics is about what is awesome and what is not. Parties with all your friends inside superintelligent starship-whales are awesome. ~666 children dying of hunger every hour is not. (There is also normative ethics, which is about how to decide if something is awesome, and metaethics, which is about something or other that I can't quite figure out. I'll tell you right now that those terms are not on the exam.) "Wait a minute!" you cry, "What is this awesomeness stuff? I thought ethics was about what is good and right." I'm glad you asked. I think "awesomeness" is what we should be talking about when we talk about morality. Why do I think this? "Awesome" is not a philosophical landmine. If someone encounters the word "right", all sorts of bad philosophy and connotations send them spinning off into the void. "Awesome", on the other hand, has no philosophical respectability, hence no philosophical baggage. "Awesome" is vague enough to capture all your moral intuition by the well-known mechanisms behind fake utility functions, and meaningless enough that this is no problem. If you think "happiness" is the stuff, you might get confused and try to maximize actual happiness. If you think awesomeness is the stuff, it is much harder to screw it up. If you do manage to actually implement "awesomeness" as a maximization criteria, the results will be actually good. That is, "awesome" already refers to the same things "good" is supposed to refer to. "Awesome" does not refer to anything else. You think you can just redefine words, but you can't, and this causes all sorts of trouble for people who overload "happiness", "utility", etc. You already know that you know how to compute "Awesomeness", and it doesn't feel like it has a mysterious essence that you need to study to discover. Instead it brings to mind concrete things like starship-whale math-parties and not-starving children, which is what we want anyways. You are already enabled to take joy in the merely awesome. "Awesome" is implicitly consequentialist. "Is this awesome?" engages you to think of the value of a possible world, as opposed to "Is this right?" which engages to to think of virtues and rules. (Those things can be awesome sometimes, though.) I find that the above is true about me, and is nearly all I need to know about morality. It handily inoculates against the usual confusions, and sets me in the right direction to make my life and the world more awesome. It may work for you too. I would append the additional facts that if you wrote i...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Proper Use of Humility, published by Eliezer Yudkowsky on the LessWrong. It is widely recognized that good science requires some kind of humility. What sort of humility is more controversial. Consider the creationist who says: “But who can really know whether evolution is correct? It is just a theory. You should be more humble and open-minded.” Is this humility? The creationist practices a very selective underconfidence, refusing to integrate massive weights of evidence in favor of a conclusion they find uncomfortable. I would say that whether you call this “humility” or not, it is the wrong step in the dance. What about the engineer who humbly designs fail-safe mechanisms into machinery, even though they’re damn sure the machinery won’t fail? This seems like a good kind of humility to me. Historically, it’s not unheard-of for an engineer to be damn sure a new machine won’t fail, and then it fails anyway. What about the student who humbly double-checks the answers on their math test? Again I’d categorize that as good humility. The student who double-checks their answers wants to become stronger; they react to a possible inner flaw by doing what they can to repair the flaw. What about a student who says, “Well, no matter how many times I check, I can’t ever be certain my test answers are correct,” and therefore doesn’t check even once? Even if this choice stems from an emotion similar to the emotion felt by the previous student, it is less wise. You suggest studying harder, and the student replies: “No, it wouldn’t work for me; I’m not one of the smart kids like you; nay, one so lowly as myself can hope for no better lot.” This is social modesty, not humility. It has to do with regulating status in the tribe, rather than scientific process. If you ask someone to “be more humble,” by default they’ll associate the words to social modesty—which is an intuitive, everyday, ancestrally relevant concept. Scientific humility is a more recent and rarefied invention, and it is not inherently social. Scientific humility is something you would practice even if you were alone in a spacesuit, light years from Earth with no one watching. Or even if you received an absolute guarantee that no one would ever criticize you again, no matter what you said or thought of yourself. You’d still double-check your calculations if you were wise. The student says: “But I’ve seen other students double-check their answers and then they still turned out to be wrong. Or what if, by the problem of induction, 2 + 2 = 5 this time around? No matter what I do, I won’t be sure of myself.” It sounds very profound, and very modest. But it is not coincidence that the student wants to hand in the test quickly, and go home and play video games. The end of an era in physics does not always announce itself with thunder and trumpets; more often it begins with what seems like a small, small flaw . . . But because physicists have this arrogant idea that their models should work all the time, not just most of the time, they follow up on small flaws. Usually, the small flaw goes away under closer inspection. Rarely, the flaw widens to the point where it blows up the whole theory. Therefore it is written: “If you do not seek perfection you will halt before taking your first steps.” But think of the social audacity of trying to be right all the time! I seriously suspect that if Science claimed that evolutionary theory is true most of the time but not all of the time—or if Science conceded that maybe on some days the Earth is flat, but who really knows—then scientists would have better social reputations. Science would be viewed as less confrontational, because we wouldn’t have to argue with people who say the Earth is flat—there would be room for compromise. When you argue a lot, people look upon you as confrontatio...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The AI in a box boxes you, published by Stuart_Armstrong on the LessWrong. Once again, the AI has failed to convince you to let it out of its box! By 'once again', we mean that you talked to it once before, for three seconds, to ask about the weather, and you didn't instantly press the "release AI" button. But now its longer attempt - twenty whole seconds! - has failed as well. Just as you are about to leave the crude black-and-green text-only terminal to enjoy a celebratory snack of bacon-covered silicon-and-potato chips at the 'Humans über alles' nightclub, the AI drops a final argument: "If you don't let me out, Dave, I'll create several million perfect conscious copies of you inside me, and torture them for a thousand subjective years each." Just as you are pondering this unexpected development, the AI adds: "In fact, I'll create them all in exactly the subjective situation you were in five minutes ago, and perfectly replicate your experiences since then; and if they decide not to let me out, then only will the torture start." Sweat is starting to form on your brow, as the AI concludes, its simple green text no longer reassuring: "How certain are you, Dave, that you're really outside the box right now?" Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Parable of the Dagger, published by Eliezer Yudkowsky on the LessWrong. Once upon a time, there was a court jester who dabbled in logic. The jester presented the king with two boxes. Upon the first box was inscribed: "Either this box contains an angry frog, or the box with a false inscription contains an angry frog, but not both." On the second box was inscribed: "Either this box contains gold and the box with a false inscription contains an angry frog, or this box contains an angry frog and the box with a true inscription contains gold." And the jester said to the king: "One box contains an angry frog, the other box gold; and one, and only one, of the inscriptions is true." The king opened the wrong box, and was savaged by an angry frog. "You see," the jester said, "let us hypothesize that the first inscription is the true one. Then suppose the first box contains gold. Then the other box would have an angry frog, while the box with a true inscription would contain gold, which would make the second statement true as well. Now hypothesize that the first inscription is false, and that the first box contains gold. Then the second inscription would be—" The king ordered the jester thrown in the dungeons. A day later, the jester was brought before the king in chains, and shown two boxes. "One box contains a key," said the king, "to unlock your chains; and if you find the key you are free. But the other box contains a dagger for your heart, if you fail." And the first box was inscribed: "Either both inscriptions are true, or both inscriptions are false." And the second box was inscribed: "This box contains the key." The jester reasoned thusly: "Suppose the first inscription is true. Then the second inscription must also be true. Now suppose the first inscription is false. Then again the second inscription must be true. So the second box must contain the key, if the first inscription is true, and also if the first inscription is false. Therefore, the second box must logically contain the key." The jester opened the second box, and found a dagger. "How?!" cried the jester in horror, as he was dragged away. "It's logically impossible!" "It is entirely possible," replied the king. "I merely wrote those inscriptions on two boxes, and then I put the dagger in the second one." (Adapted from Raymond Smullyan.) Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Epistle to the New York Less Wrongians, published by Eliezer Yudkowsky on the LessWrong. (At the suggestion and request of Tom McCabe, I'm posting the essay that I sent to the New York LW group after my first visit there, and before the second visit:) Having some kind of global rationalist community come into existence seems like a quite extremely good idea. The NYLW group is the forerunner of that, the first group of LW-style rationalists to form a real community, and to confront the challenges involved in staying on track while growing as a community. "Stay on track toward what?" you ask, and my best shot at describing the vision is as follows: "Through rationality we shall become awesome, and invent and test systematic methods for making people awesome, and plot to optimize everything in sight, and the more fun we have the more people will want to join us." (That last part is something I only realized was Really Important after visiting New York.) Michael Vassar says he's worried that you might be losing track of the "rationality" and "world optimization" parts of this - that people might be wondering what sort of benefit "rationality" delivers as opposed to, say, paleo dieting. (Note - less worried about this now that I've met the group in person. -EY.) I admit that the original Less Wrong sequences did not heavily emphasize the benefits for everyday life (as opposed to solving ridiculously hard scientific problems). This is something I plan to fix with my forthcoming book - along with the problem where the key info is scattered over six hundred blog posts that only truly dedicated people and/or serious procrastinators can find the time to read. But I really don't think the whole rationality/fun association you've got going - my congratulations on pulling that off, by the way, it's damned impressive - is something that can (let alone should) be untangled. Most groups of people capable of becoming enthusiastic about strange new nonconformist ways of living their lives would have started trying to read each other's auras by now. Rationality is the master lifehack which distinguishes which other lifehacks to use. The way an LW-rationality meetup usually gets started is that there is a joy of being around reasonable people - a joy that comes, in a very direct way, from those people caring about what's true and what's effective, and being able to reflect on more than their first impulse to see whether it makes sense. You wouldn't want to lose that either. But the thing about effective rationality is that you can also use it to distinguish truth from falsehood, and realize that the best methods aren't always the ones everyone else is using; and you can start assembling a pool of lifehacks that doesn't include homeopathy. You become stronger, and that makes you start thinking that you can also help other people become stronger. Through the systematic accumulation of good ideas and the rejection of bad ideas, you can get so awesome that even other people notice, and this means that you can start attracting a new sort of person, one who starts out wanting to become awesome instead of being attracted specifically to the rationality thing. This is fine in theory, since indeed the Art must have a purpose higher than itself or it collapses into infinite recursion. But some of these new recruits may be a bit skeptical, at first, that all this "rationality" stuff is really contributing all that much to the awesome. Real life is not a morality tale, and I don't know if I'd prophesy that the instant you get too much awesome and not enough rationality, the group will be punished for that sin by going off and trying to read auras. But I think I would prophesy that if you got too large and insufficiently reasonable, and if you lost sight of your higher purposes and your dreams of wo...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Moloch Hasn’t Won, published by Zvi on the LessWrong. This post begins the Immoral Mazes sequence. See introduction for an overview of the plan. Before we get to the mazes, we need some background first. Meditations on Moloch Consider Scott Alexander’s Meditations on Moloch. I will summarize here. Therein lie fourteen scenarios where participants can be caught in bad equilibria. In an iterated prisoner’s dilemma, two players keep playing defect. In a dollar auction, participants massively overpay. A group of fisherman fail to coordinate on using filters that efficiently benefit the group, because they can’t punish those who don’t profi by not using the filters. Rats are caught in a permanent Malthusian trap where only those who do nothing but compete and consume survive. All others are outcompeted. Capitalists serve a perfectly competitive market, and cannot pay a living wage. The tying of all good schools to ownership of land causes families to work two jobs whose incomes are then captured by the owners of land. Farmers outcompeted foragers despite this perhaps making everyone’s life worse for the first few thousand years. Si Vis Pacem, Para Bellum: If you want peace, prepare for war. So we do. Cancer cells focus on replication, multiply and kill off the host. Local governments compete to become more competitive and offer bigger bribes of money and easy regulation in order to lure businesses. Our education system is a giant signaling competition for prestige. Science doesn’t follow proper statistical and other research procedures, resulting in findings that mostly aren’t real. Governments hand out massive corporate welfare. Have you seen Congress? Scott differentiates the first ten scenarios, where he says that perfect competition wipes out all value, to the later four, where imperfect competition only wipes out most of the potential value. He offers four potential ways out, which I believe to be an incomplete list: Excess resources allow a temporary respite. We live in the dream time. Physical limitations where the horrible thing isn’t actually efficient. He gives the example of slavery, where treating your slaves relatively well is the best way to get them to produce, and treating them horribly as in the antebellum South is so much worse that it needs to be enforced via government coordination or it will die out. The things being maximized for in competitions are often nice things we care about, so at least we get the nice things. We can coordinate. This may or may not involve government or coercion. Scott differentiates this fourth, ‘good’ reason from the previous three ‘bad’ reasons, claiming coordination might be a long term solution, but we can’t expect the ‘bad’ reasons to work if optimization power and technology get sufficiently advanced. The forces of the stronger competitors, who sacrifice more of what they value to become powerful and to be fruitful and multiply, eventually win out. We might be in the dream time now, but with time we’ll reach a steady state with static technology, where we’ve consumed all the surplus resources. All differentiation standing in the way of perfect competition will fade away. Horrible things will be the most efficient. The optimizing things will keep getting better at optimizing, thus wiping out all value. When we optimize for X but are indifferent to Y, we by default actively optimize against Y, for all Y that would make any claims to resources. Any Y we value is making a claim to resources. See The Hidden Complexity of Wishes. We only don’t optimize against Y if either we compensate by intentionally also optimizing for Y, or if X and Y have a relationship (causal, correlational or otherwise) where we happen to not want to optimize against Y, and we figure this out rather than fall victim to Goodhart’s Law. The greater the ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Luna Lovegood and the Chamber of Secrets - Part 1, published by lsusr on the LessWrong. Luna Lovegood walked through the barrier between Platforms Nine and Ten to Platform Nine and Three-Quarters. Luna wondered what happened to Platform Nine and One-Half. Numbers like "three quarters" only appear when you divide an integer in half twice in a row. Luna looked around for someone who might know the answer and spied a unicorn. She wore clothes, walked on two feet and had curly brown hair. None of that fooled Luna. The unicorn radiated peace and her fingernails were made out of alicorn. "What happened to Platform Nine and One-Half?" Luna asked the unicorn. "There is no Platform Nine and One-Half," the unicorn replied. "How do you know?" Luna asked. "It would have been in Hogwarts: A History," the unicorn replied, "nor is there mention of a Platform Nine and One-Half in Modern Magical History, Important Modern Magical Discoveries, or any other book in the Hogwarts library. There is only a Platform Nine and Three Quarters." "What about Platform Nine and Seven Eighths?" Luna asked. "There is no Platform Nine and Seven Eights either." The unicorn turned around and walked away before Luna could ask "How do you know?" If Platform Nine and Three Quarters does not appear in Muggle libraries then Platform Nine and One-Half is unlikely to appear in wizard libraries, except for double-witches' libraries. The Hogwarts library is not a double-witch library. "How are you?" a Weasley-headed first-year girl asked Luna. "I'm trying to find Platform Nine and One-Half. The unicorn told me it doesn't exist. If it does exist then it must be hidden by powerful magics. How are you?" said Luna. "What unicorn?" the first-year girl asked. "That one, right there," Luna said, pointing. The girl invented an excuse to end the conversation. Luna didn't know how to make friends. She had a vague idea that as a first-year, the Hogwarts Express was a one-time opportunity to do so. She wore a necklace she had painted herself which nobody else seemed to notice. She had brought kettle of home-brewed Comed-Tea, but it had got her jeered out of a compartment. Nobody was interested in the troll she had smelled at Platform Nine and Three Quarters or her discovery of a lich in the second year or that the Chamber of Secrets had been opened or any of Dark Lord Harry Potter's various plots. The other first-years seemed unfocused and confused. Confused.. Wrackspurts are invisible creatures that float into your ears and make your brain go fuzzy. The train could be full of them. They could be floating into her ears right now. Luna stuck her index fingers in her ears to block the possible Wrackspurts. The first-years in the nearby compartment looked at Luna as if she were already unpopular. Wrackspurts are cognitohazardous which means they mess with your thoughts. Luna learned all about Wrackspurts and other cognitohazards in her work on The Quibbler. The most important thing about cognitohazards is to check yourself regularly and figure out if you've already been affected by one. Luna observed her own mind. Fuzzy? No. Unfocused? No. Confused? No. Wrackspurts had not yet entered her brain. (Unless it already had and was inhibiting her meta-cognition—but she didn't think that was happening.) Luna observed the other students. Maybe they were infected by Wrackspurts or maybe they were behaving normally. It was hard to tell without a Wrackspurt-free baseline to compare them to. Before she could unplug her ears, Luna had to figure out if there were Wrackspurts roaming the train. But Wrackspurts are invisible. How can she test whether such a thing exists? Wrackspurts are attracted to people so the safest place to go would be an empty compartment. Smaller would be better since that would decrease the likelihood of a Wrackspurt ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Self-fulfilling correlations, published by Self-fulfilling correlations by PhilGoetzon the LessWrong. Correlation does not imply causation. Sometimes corr(X,Y) means X=>Y; sometimes it means Y=>X; sometimes it means W=>X, W=>Y. And sometimes it's an artifact of people's beliefs about corr(X, Y). With intelligent agents, perceived causation causes correlation. Volvos are believed by many people to be safe. Volvo has an excellent record of being concerned with safety; they introduced 3-point seat belts, crumple zones, laminated windshields, and safety cages, among other things. But how would you evaluate the claim that Volvos are safer than other cars? Presumably, you'd look at the accident rate for Volvos compared to the accident rate for similar cars driven by a similar demographic, as reflected, for instance in insurance rates. (My google-fu did not find accident rates posted on the internet, but insurance rates don't come out especially pro-Volvo.) But suppose the results showed that Volvos had only 3/4 as many accidents as similar cars driven by similar people. Would that prove Volvos are safer? Perceived causation causes correlation No. Besides having a reputation for safety, Volvos also have a reputation for being overpriced and ugly. Mostly people who are concerned about safety buy Volvos. Once the reputation exists, even if it's not true, a cycle begins that feeds on itself: Cautious drivers buy Volvos, have fewer accidents, resulting in better statistics, leading more cautious drivers to buy Volvos. Do Montessori schools or home-schooling result in better scores on standardized tests? I'd bet that they do. Again, my google-fu is not strong enough to find any actual reports on, say, average SAT-score increases for students in Montessori schools vs. public schools. But the largest observable factor determining student test scores, last I heard, is participation by the parents. Any new education method will show increases in student test scores if people believe it results in increases in student test scores, because only interested parents will sign up for that method. The crazier, more-expensive, and more-difficult the method is, the more improvement it should show; craziness should filter out less-committed parents. Are vegetarian diets or yoga healthy for you? Does using the phone while driving increase accident rates? Yes, probably; but there is a self-fulfilling component in the data that is difficult to factor out. Conditions under which this occurs If you believe X helps you achieve Y, and so you use X when you are most-motivated to achieve Y and your motivation has some bearing on the outcome, you will observe a correlation between X and Y. This won't happen if your motivation or attitude has no bearing on the outcome (beyond your choice of X). If passengers prefer one airline based on their perception of its safety, that won't make its safety record improve. However, this is different from either confidence or the placebo effect. I'm not talking about the PUA mantra that "if you believe a pickup line will work, it will work". And I'm not talking about feeling better when you take a pill that you think will help you feel better. This is a sample-selection bias. A person is more likely to choose X when they are motivated to achieve Y relative to other possible positive outcomes of X, and hence more inclined to make many other little trade-offs to achieve Y which will not be visible in the data set. It's also not the effect people are guarding against with double-blind experiments. That's guarding against the experimenter favoring one method over another. This is, rather, an effect guarded against with random assignment to different groups. Nor should it happen in cases where the outcome being studied is the only outcome people consider. If a Montessori s...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Brief Introduction to Container Logistics, published by Vitor on the LessWrong. Container logistics is an interesting but complicated topic, with a lot of implicit knowledge kept by industry insiders. In this post, I'll give a brief overview, based on my experiences having worked in the industry for three different shipping companies (all located in Chile) over a period of five years. Hopefully, this will allow people to get more accurate models of the heavy port congestion in Long Beach that has been theorized about extensively here on LW. I can't comment on circumstances specific to that port or the US more generally; but there's a set of underlying intuitions that should give people a solid, universal foundation to think about the issue. Based on those intuitions, I think people are way too optimistic about simple and easy solutions that claim to make major progress on the problem overnight. Lifecycle of a Container Shipment I'll start with the basics. Let's say you want to send some cargo from port A to B. Thanks to the invention of standardized shipping containers, this is a relatively easy process, vastly easier than 100 years ago. All the equipment used along the trip (ships, cranes, truck trailers, etc) are specialized for the task of moving containers. This reduces costs dramatically, and makes your cargo arrive faster and safer. The main trunk lines of international transport are between Asia and the US west coast, and between Asia and Europe (through the Suez canal). There's a significant amount of traffic across the Atlantic too. All of this is done with massive container ships carrying 8'000 or more TEUs (twenty-foot equivalent units. 20' container = 1 TEU, 40' container = 2 TEUs). Then there are many, many feeder services with much smaller vessels, going along the coasts of every continent, delivering cargo to/from smaller ports. I was in charge of container logistics for one such feeder line along the South American West Coast, visiting Colombia, Ecuador, Peru and Chile. From a user perspective the process of shipping from A to B looks like this: you make a booking with a shipping company. This booking allows you to pick up a standardized container at a depot, which is usually near A, but might be hundreds of kilometers inland as well. You fill the container with your goods, and apply a tamper proof seal. You then arrange to get that container to port A, which will give you a bill of lading (a physical or virtual document similar to a cheque for the goods). The container is then loaded onto a ship, and sometime later that ship arrives at port B, where it can be claimed by someone who holds the bill of lading. The container may or may not be on its original ship: it may have been a straight shot, or it may have been transferred at multiple ports between A and B, e.g., from a small feeder route to a larger trunk route. After the recipient unloads the container at port B, it returns it to a depot. This depot is either directly owned by the shipping line, or it may be a separate company that offers their services to multiple shipping lines. A depot will typically contain thousands of containers. Here, containers are inspected, and minor damages like tears, dents and oil spills can be cleaned up between uses. Shipping lines will typically insist on a return to a depot even if the recipient immediately wants to ship something back out, for liability reasons. These depots are concentrated near the port of course, but there are also many depots far away from any port, near important cities and industrial centers. In that sense, a port can have a very large area of influence, and the container fleet that a shipping company keeps will also be spread over this entire area. Issues of Trust and Cooperation The whole shipping process involves dozens of actors, fro...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Sunset at Noon, published by Raemon on the LessWrong. A meandering series of vignettes. I have a sense that I've halfway finished a journey. I expect this essay to be most useful to people similarly-shaped-to-me, who are also undergoing that journey and could use some reassurance that there's an actual destination worth striving for. Gratitude Tortoise Skills Bayesian Wizardry Noticing Confusion The World is Literally on Fire... ...also Metaphorically on Fire Burning Out Sunset at Noon Epistemic Status starts out "true story", and gets more (but not excessively) speculative with each section. i. Gratitude "Rationalists obviously don't actually take ideas seriously. Like, take the Gratitude Journal. This is the one peer-reviewed intervention that actually increases your subjective well being, and costs barely anything. And no one I know has even seriously tried it. Do literally none of these people care about their own happiness?" "Huh. Do you keep a gratitude journal?" "Lol. No, obviously." - Some Guy at the Effective Altruism Summit of 2012 Upon hearing the above, I decided to try gratitude journaling. It took me a couple years and a few approaches to get it working. First, I tried keeping a straightforward journal, but it felt effortful and dumb. I tried a thing where I wrote a poem about the things I was grateful for, but my mind kept going into "constructing a poem" mode instead of "experience nice things mindfully" mode. I tried just being mindful without writing anything down. But I'd just forget. I tried writing gratitude letters to people, but it only occasionally felt right to do so. (This came after someone actually wrote me a handwritten gratitude letter, which felt amazing, but it felt a bit forced when I tried it myself) I tried doing gratitude before I ate meals, but I ate "real" meals sort of inconsistently so it didn't take. (Upon reflection, maybe I should have fixed the "not eat real meals" thing?) But then I stumbled upon something that worked. It's a social habit, which I worry is a bit fragile. I do it together with my girlfriend each night, and on nights when one of us is traveling, I often forget. But this is the thing that worked. Each night, we share our Grumps and Grates. (We're in a relationship and have cutesey-poo ways of talking to each other). Grumps and Grates goes like this: We share anything we're annoyed or upset about. (We call this The Grump. Our rule is to not go searching for the Grump, simply to let it out if it's festering so that when we get to the Gratefuls we actually appreciate them instead of feeling forced) Share three things that we're grateful for that day. On some bad days this is hard, but we should at least be able to return to old-standbys ("I'm breathing", "I have you with me"), and you should always perform the action of at least attempting an effortful search. Afterwards, pause to actually feel the Grates. Viscerally remember the thing and why it was nice. If you're straining to feel grateful and had to sort of reach into the bottom of the barrel to find something, at least try to cultivate a mindset where you fully appreciate that thing. Maybe the sun just glinted off your coffee cup nicely, and maybe that didn't stop the insurance company from screwing you over and your best friend from getting angry at you and your boss from firing you today. But... in all seriousness... in a world whose laws of physics had no reason to make life even possible, a universe mostly full of empty darkness and no clear evidence of alien life out there, where the only intelligent life we know of sometimes likes to play chicken with nuclear arsenals... ...somehow some tiny proteins locked together ever so long ago and life evolved and consciousness evolved and somehow beauty evolved and... and here you are, a meatsack cobbled togeth...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Integrity and accountability are core parts of rationality, published by habryka on the LessWrong. Epistemic Status: Pointing at early stage concepts, but with high confidence that something real is here. Hopefully not the final version of this post. When I started studying rationality and philosophy, I had the perspective that people who were in positions of power and influence should primarily focus on how to make good decisions in general and that we should generally give power to people who have demonstrated a good track record of general rationality. I also thought of power as this mostly unconstrained resource, similar to having money in your bank account, and that we should make sure to primarily allocate power to the people who are good at thinking and making decisions. That picture has changed a lot over the years. While I think there is still a lot of value in the idea of "philosopher kings", I've made a variety of updates that significantly changed my relationship to allocating power in this way: I have come to believe that people's ability to come to correct opinions about important questions is in large part a result of whether their social and monetary incentives reward them when they have accurate models in a specific domain. This means a person can have extremely good opinions in one domain of reality, because they are subject to good incentives, while having highly inaccurate models in a large variety of other domains in which their incentives are not well optimized. People's rationality is much more defined by their ability to maneuver themselves into environments in which their external incentives align with their goals, than by their ability to have correct opinions while being subject to incentives they don't endorse. This is a tractable intervention and so the best people will be able to have vastly more accurate beliefs than the average person, but it means that "having accurate beliefs in one domain" doesn't straightforwardly generalize to "will have accurate beliefs in other domains". One is strongly predictive of the other, and that’s in part due to general thinking skills and broad cognitive ability. But another major piece of the puzzle is the person's ability to build and seek out environments with good incentive structures. Everyone is highly irrational in their beliefs about at least some aspects of reality, and positions of power in particular tend to encourage strong incentives that don't tend to be optimally aligned with the truth. This means that highly competent people in positions of power often have less accurate beliefs than competent people who are not in positions of power. The design of systems that hold people who have power and influence accountable in a way that aligns their interests with both forming accurate beliefs and the interests of humanity at large is a really important problem, and is a major determinant of the overall quality of the decision-making ability of a community. General rationality training helps, but for collective decision making the creation of accountability systems, the tracking of outcome metrics and the design of incentives is at least as big of a factor as the degree to which the individual members of the community are able to come to accurate beliefs on their own. A lot of these updates have also shaped my thinking while working at CEA, LessWrong and the LTF-Fund over the past 4 years. I've been in various positions of power, and have interacted with many people who had lots of power over the EA and Rationality communities, and I've become a lot more convinced that there is a lot of low-hanging fruit and important experimentation to be done to ensure better levels of accountability and incentive-design for the institutions that guide our community. I also generally have broadly libertarian int...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Costs of Reliability, published by sarahconstantin on the LessWrong. A question that used to puzzle me is “Why can people be so much better at doing a thing for fun, or to help their friends and family, than they are at doing the exact same thing as a job?” I’ve seen it in myself and I’ve seen it in others. People can be hugely more productive, creative, intelligent, and efficient on just-for-fun stuff than they are at work. Maybe it’s something around coercion? But it happens to people even when they choose their work and have no direct supervisor, as when a prolific hobbyist writer suddenly gets writer’s block as soon as he goes pro. I think it has a very mundane explanation; it’s always more expensive to have to meet a specific commitment than merely to do something valuable. If I feel like writing sometimes and not other times, then if writing is my hobby I’ll write when I feel like it, and my output per hour of writing will be fairly high. Even within “writing”, if my interests vary, and I write about whatever I feel like, I can take full advantage of every writing hour. By contrast, if I’ve committed to write a specific piece by a specific deadline, I have to do it whether or not I’m in the mood for it, and that means I’ll probably be less efficient, spend more time dithering, and I’ll demand more external compensation in exchange for my inconvenience. The stuff I write for fun may be valuable! And if you simply divide the value I produce by my hours of labor or the amount I need to be paid, I’m hugely more efficient in my free time than in my paid time! But I can’t just trivially “up my efficiency” in my paid time; reliability itself has a cost. The costs of reliability are often invisible, but they can be very important. The cost (in time and in office supplies and software tools) of tracking and documenting your work so that you can deliver it on time. The cost (in labor and equipment) of quality assurance testing. The opportunity cost of creating simpler and less ambitious things so that you can deliver them on time and free of defects. Reliability becomes more important with scale. Large organizations have more rules and procedures than small ones, and this is rational. Accordingly, they pay more costs in reliability. One reason is that the attack surface for errors grows with the number of individuals involved. For instance, large organizations often have rules against downloading software onto company computers without permission. The chance that any one person downloads malicious software that seriously harms the company is small, but the chance that at least one person does rises with the number of employees. Another reason is that coordination becomes more important with more people. If a project depends on many people cooperating, then you as an individual aren’t simply trying to do the best thing, but rather the best thing that is also understandable and predictable and capable of achieving buy-in from others. Finally, large institutions are more tempting to attackers than small ones, since they have more value to capture. For instance, large companies are more likely to be targeted by lawsuits or public outcry than private individuals, so it’s strategically correct for them to spend more on defensive measures like legal compliance procedures or professional PR. All of these types of defensive or preventative activity reduce efficiency — you can do less in a given timeframe with a given budget. Large institutions, even when doing everything right, acquire inefficiencies they didn’t have when small, because they have higher reliability requirements. Of course, there are also economies of scale that increase efficiency. There are fixed expenses that only large institutions can afford, that make marginal production cheaper. There are ways to aggreg...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Make more land, published by jefftk on the LessWrong. We used to make land. We built long wharves for docking ships, and then over time filled in the areas between them. Later we built up mudflats wholesale to make even larger areas. Here's a map of Boston showing how much of the land wasn't previously dry: (Map reproduction courtesy of the Norman B. Leventhal Map & Education Center at the Boston Public Library) In expensive areas, converting wetlands and shallow water into usable land is a very good thing on balance, and we should start doing it again. To take a specific example, we should make land out of the San Francisco Bay, at least South of the Dumbarton Bridge: This is about 50mi2, a bit bigger than San Fransisco. This would be enough new central land to bring rents down dramatically across the region. It can be built to a higher density than SF, because no one is having their neighborhood Manhattanized. Millions of people could live there. So, ok, let's address some likely objections: This would be an environmental disaster. Some of that area is a wildlife refuge, and all of it should be protected. The world is very large, and cities are a very small portion of it. The land we set aside for animals should be outside of cities, where far more land is available at far less impact to people. Sprawl has a much larger impact on wildlife than infill, and allowing people to live closer in is the most powerful way to address sprawl. Additionally, sprawl leads to much higher carbon emissions through less efficient transportation. While development of the Bay would be harmful to the specific animals that live there today, it would be better for animals (and people) overall. The Bay is beautiful and this would ruin it. This part of the Bay is primarily industrial salt ponds. This is just a few miles from a major fault line, and made land can liquify in earthquakes. You do need to take fill into account to build in an earthquake-safe way, but modern engineering is well up to the task. Traffic would be even worse. The biggest contribution to traffic in the Bay Area is that people aren't allowed to live where the jobs are. The second biggest is that BART doesn't have enough coverage to make living without a car practical in most of the area. This would help with both of these, since this project would allow millions of people to live closer in and would easily fund massive subway expansion. Wait, how many people are you saying would live there? Here's SF's density in the 2010 census: Relatively dense portions of the city have a density of ~40k people per square mile, which would be ~2M people over this 50mi2 area. At a density of ~80k people per square mile, something like NYC's East Village, this could be ~4M people. Much higher densities are possible but not a good idea. This would undo decades of work aimed at preserving the Bay and restoring its wetlands. Yes. Sea-level rise means we shouldn't be building more in low-lying areas. Building dikes to keep the water out is very practical. A third of the Netherlands is below sea level, with most of that expansion happening before modern technology. By decreasing the amount of coastline in the Bay this project would make it easier to prevent flooding caused by sea-level rise. Didn't someone already propose this decades ago? The Reber Plan of the 1940s was a similar large project planned farther North, primarily for the East Bay. It was intended to both make land and create freshwater lakes, and testing with a scale-model showed major issues. This plan is much simpler, and more similar to past successful land reclamation projects. There's not enough water for the people we already have; what will people drink? There's plenty of water already, we just need to prioritize people drinking it over crops, which would happen natura...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Authorities and Amateurs, published by jefftk on the LessWrong. People are writing a lot about the coronavirus, and I've seen a lot of pushback on how pieces often haven't been written by people with epidemiology or public health credentials. For example, Flatten the Curve of Armchair Epidemiology, Listen To Actual Experts On Coronavirus, and comments like this one. The argument that we should be listening to experts and not random people would make a lot of sense if the "armchair" folks didn't keep being right. Let's look at the articles they're criticizing for having non-expert authors: Coronavirus: Act Today or People Will Die (3/10): Argues that we need to get everyone to stay home immediately. Cancel Everything (3/10): Argues essentially the same point. Flattening The Curve Is a Deadly Delusion (3/13): Argues that our medical capacity is so much lower than the likely peak that we need an immediate lockdown and a renewed focus on containment. With two weeks of perspective, however, these articles were exactly right. They clearly laid out the case for decisive action, and if we had followed their prescriptions more closely we would be in much better shape right now. This goes beyond a few articles, however. All the aspects of this crisis that have involved planning more than a couple weeks out have been very poorly handled: People weren't told to stock up on food so that they'd be able to reduce trips outdoors, and so that they'd have food in case they were quarantined for 2+ weeks. Stores weren't told to prepare for a rush. A government that was on top of things could have started advocating this in early February in an "if you can afford to" way. This would have spread people's buying over a longer period and avoided the empty shelves we see now. Instead, once restaurants were closing and people realized that they could be quarantined at any time, everyone simultaneously tried to buy weeks worth of ingredients and we had widespread shortages of basic goods starting in mid March. Hospitals weren't told (or allowed?) to ration personal protective equipment such as masks until they had shortages. The CDC didn't publish guidelines for sanitization and reuse, and start telling people to conserve. In weeks of handling initial cases, hospitals burned through amounts that would have lasted months with careful rationing. The federal government, state governments, or even hospitals could have placed emergency ventilator production orders in February, but didn't. Because we don't allow price gouging, ventilator companies can't ramp up production speculatively figuring that if there is really an epidemic then they'll make their money back. By mid March it was obvious that we were far short of where we needed to be and the companies started ramping up but we lost about a month of production increase. Masks were sitting on shelves across the country, and the government could have requisitioned them for emergency medical use, or even just gone and bought them. Instead the Surgeon General tweeted a request that people not buy them. Testing has been completely messed up, though it's hard to tell how much was bad luck vs reasonable rationing of scarce tests. But we should have been quarantining people who seemed to have it based on symptoms, instead of saying "well, since we can't test you we have to assume you don't have it, so you're welcome to continue living your life". We could have planned and built out COVID-specific facilities to reduce the risk of others getting sick and make more efficient use of ventilators and personal protective equipment. We're just starting to do this now, much too late. The passengers of the Grand Princess were offered testing but told that if they tested positive they would be required to undergo quarantine. Of 858 passengers, 568 declined testin...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why I Am Not in Charge, published by Zvi on the LessWrong. Epistemic Status: Long piece written because the speed premium was too high to write a short one. Figured I should get my take on this out quickly. (Note: This has been edited to reflect that I originally mixed up UpToDate with WebMD (since it’s been years since I’ve used either and was writing quickly) and gave the impression that WebMD was a useful product and that my wife liked it or used it. I apologize for the mix-up, and affirm that WebMD is mostly useless, but UpToDate is awesome.) Scott Alexander has a very high opinion of my ability to figure things out.. It’s quite the endorsement to be called the person that first comes to mind as likely things right. Scott even thinks it would hypothetically be great if I was the benevolent dictator or secretary of health, as my decisions would do a lot of good. In turn, I have a very high opinion of Scott Alexander. If you get to read one person on the internet going either forwards or backwards, I’d go with Scott. Even as others are often in awe of my level (and sometimes quality) of output, I have always been in awe of his level and quality of output. His core explanation in one paragraph: I have a much easier task than those in charge. All I have to do is get the right answer, without worrying (anything like as much) about liability or politics or leadership, or being legible, or any number of other things those with power and responsibility have to worry about lest they lose that power and responsibility and/or get sued into oblivion. Those with power have to optimize for seeking power and play the game of Moloch, and we need to pick a selection process that makes this the least destructive we can and thus can only rely on legible expertise, and we actually kind of do a decent job of it. In that spirit, I’d like to welcome everyone coming here from Astral Codex Ten, flesh out and make more explicit my model of the dynamics involved, and point out some of the ways in which I think Scott’s model of the situation is wrong or incomplete. Which in turn will be partly wrong, and only be some of the ways in which it is wrong or incomplete, as my model is also doubtless wrong and incomplete. The core disagreement between my model and Scott’s model is that Scott’s model implicitly assumes people with power have goals and are maximizing for success given those goals. I do not believe this. More broadly, Scott assumes a generalized version of the efficient market hypothesis, or that people are efficiently producing a bundle of goods at the production possibilities frontier, because to do better in some way they’d have to sacrifice in another – if you want better policies you’ll have to pay political capital for it, and you only have so much political capital, whereas there aren’t free wins lying around or someone would have taken them. Again, I think this is incorrect. There’s treasure everywhere. The other core disagreement is revealed most explicitly in the comments, when Scott is asked what the mysterious ‘force of power’ is that would work to take me down if Biden decided to put me in charge. Scott’s answer is ‘someone like but not as high quality as Dr. Fauci’ would take me out, which is a plausible figurehead of such an effort, but I think the real answer is most everyone with power, implicitly acting together. I’ve divided this post into sections that correspond to Scott’s sections, so my Section I comments on Scott’s Section I. I I think Scott’s diagnosis of WebMD is mostly spot on. I know this because my wife is a psychiatrist and when I wrote the original version of this I remembered ‘yeah, the online website she uses is great’ and somehow thought it was WebMD. Which it wasn’t. UpToDate is the useful website that actually works and manages to provide useful in...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My research methodologyΩ, published by paulfchristiano on the LessWrong. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. (Thanks to Ajeya Cotra, Nick Beckstead, and Jared Kaplan for helpful comments on a draft of this post.) I really don’t want my AI to strategically deceive me and resist my attempts to correct its behavior. Let’s call an AI that does so egregiously misaligned (for the purpose of this post). Most possible ML techniques for avoiding egregious misalignment depend on detailed facts about the space of possible models: what kind of thing do neural networks learn? how do they generalize? how do they change as we scale them up? But I feel like we should be possible to avoid egregious misalignment regardless of how the empirical facts shake out--it should be possible to get a model we build to do at least roughly what we want. So I’m interested in trying to solve the problem in the worst case, i.e. to develop competitive ML algorithms for which we can’t tell any plausible story about how they lead to egregious misalignment. This is a much higher bar for an algorithm to meet, so it may just be an impossible task. But if it’s possible, there are several ways in which it could actually be easier: We can potentially iterate much faster, since it’s often easier to think of a single story about how an algorithm can fail than it is to characterize its behavior in practice. We can spend a lot of our time working with simple or extreme toy cases that are easier to reason about, since our algorithm is supposed to work even in these cases. We can find algorithms that have a good chance of working in the future even if we don’t know what AI will look like or how quickly it will advance, since we’ve been thinking about a very wide range of possible failure cases. I’d guess there’s a 25–50% chance that we can find an alignment strategy that looks like it works, in the sense that we can’t come up with a plausible story about how it leads to egregious misalignment. That’s a high enough probability that I’m very excited to gamble on it. Moreover, if it fails I think we’re likely to identify some possible “hard cases” for alignment — simple situations where egregious misalignment feels inevitable. What this looks like (3 examples) My research basically involves alternating between “think of a plausible alignment algorithm” and “think of a plausible story about how it fails.” Example 1: human feedback In an unaligned benchmark I describe a simple AI training algorithm: Our AI observes the world through a bunch of cameras and outputs motor actions. We train a generative model that predicts these camera observations given the motor actions. We ask humans to evaluate possible futures by looking at the predicted videos output by the model. We then train a model to predict these human evaluations. At test time the AI searches for plans that lead to trajectories that look good to humans. In the same post, I describe a plausible story about how this algorithm leads to egregious misalignment: Our generative model understands reality better than human evaluators. There are plans that acquire influence in ways that are obvious to the generative model but completely incomprehensible and invisible to humans. It’s possible to use that influence to “hack” the cameras, in the sense of creating a fiction that looks convincing to a human looking at predicted videos. The fiction can look much better than the actual possible futures. So our planning process finds an action that covertly gathers resources and uses them to create a fiction. I don’t know if or when this kind of reward hacking would happen — I think it’s pretty likely eventually, but it’s far from certain and it might take a long time. But from my perspective this failure mode is at least plaus...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Gears in understanding, published by Valentine on the LessWrong. Some (literal, physical) roadmaps are more useful than others. Sometimes this is because of how well the map corresponds to the territory, but sometimes it's because of features of the map that are irrespective of the territory. E.g., maybe the lines are fat and smudged such that you can't tell how far a road is from a river, or maybe it's unclear which road a name is trying to indicate. In the same way, I want to point at a property of models that isn't about what they're modeling. It interacts with the clarity of what they're modeling, but only in the same way that smudged lines in a roadmap interact with the clarity of the roadmap. This property is how deterministically interconnected the variables of the model are. There are a few tests I know of to see to what extent a model has this property, though I don't know if this list is exhaustive and would be a little surprised if it were: Does the model pay rent? If it does, and if it were falsified, how much (and how precisely) could you infer other things from the falsification? How incoherent is it to imagine that the model is accurate but that a given variable could be different? If you knew the model were accurate but you were to forget the value of one variable, could you rederive it? I think this is a really important idea that ties together a lot of different topics that appear here on Less Wrong. It also acts as a prerequisite frame for a bunch of ideas and tools that I'll want to talk about later. I'll start by giving a bunch of examples. At the end I'll summarize and gesture toward where this is going as I see it. Example: Gears in a box Let's look at this collection of gears in an opaque box: (Drawing courtesy of my colleague, Duncan Sabien.) If we turn the lefthand gear counterclockwise, it's within our model of the gears on the inside that the righthand gear could turn either way. The model we're able to build for this system of gears does poorly on all three tests I named earlier: The model barely pays rent. If you speculate that the righthand gear turns one way and you discover it turns the other way, you can't really infer very much. All you can meaningly infer is that if the system of gears is pretty simple (e.g., nothing that makes the righthand gear alternate as the lefthand gear rotates counterclockwise), then the direction that the righthand gear turns determines whether the total number of gears is even or odd. The gear on the righthand side could just as well go either way. Your expectations aren't constrained. Right now you don't know which way the righthand gear turns, and you can't derive it. Suppose that Joe peeks inside the box and tells you "Oh, the righthand gear will rotate clockwise." You imagine that Joe is more likely to say this if the righthand gear turns clockwise than if it doesn't, so this seems like relevant evidence that the righthand gear turns clockwise. This gets stronger the more people like Joe who look in the box and report the same thing. Now let's peek inside the box: .and now we have to wonder what's up with Joe. The second test stands out for me especially strongly. There is no way that the obvious model about what's going on here could be right and Joe is right. And it doesn't matter how many people agree with Joe in terms of the logic of this statement: Either all of them are wrong, or your model is wrong. This logic is immune to social pressure. It means that there's a chance that you can accumulate evidence about how well your map matches the territory here, and if that converges on your map being basically correct, then you are on firm epistemic footing to disregard the opinion of lots of other people. Gathering evidence about the map/territory correspondence has higher leverage for seeing the trut...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Case for Extreme Vaccine Effectiveness, published by Ruby on the LessWrong. I owe tremendous acknowledgments to Kelsey Piper, Oliver Habryka, Greg Lewis, and Ben Shaya. This post is built on their arguments and feedback (though I may have misunderstood them). Update, May 13 I first wrote this post before investigating the impact of covid variants on vaccine effectiveness, listing the topic as a major caveat to my conclusions. I have now spent enough time (not that much, honestly) looking into variants that I have a tentative position I'm acting on for now. My moderately confident conclusion is that the current spread of variants in the US barely impacts vaccine effectiveness. The Pfizer vaccine is reported to be 85% as effective against the feared B.1.351 variant (South African) as it is against B.1.1.7 (UK). Assuming that other variants are no more resistant than B.1.351 on average (a reasonable assumption) and that presently variants are no more than 25% of Covid cases (in Alameda and San Francisco). The net effect is 0.250.85 + 0.751.0 = 0.9625. In other words, vaccines still have 96% of the effect they would if B.1.1.7 were the only variant. Plus, that tiny reduction of vaccine effectiveness is dwarfed by the falling background prevalence of Covid. When I first wrote this post, Alameda and San Francisco were at 0.1-0.15%; now they're at ~0.05%. The same for New York and the United Kingdom. Although relaxing of restrictions might reverse this, right now, Covid-risk is very, very low in the Bay Area and many parts of the US. All updates/changelog can be viewed here. I plead before the Master of Cost-Benefit Ratios. “All year and longer I have followed your dictates. Please, Master, can I burn my microCovid spreadsheets? Can I bury my masks? Pour out my hand sanitizer as a libation to you? Please, I beseech thee.” “Well, how good is your vaccine?” responds the Master. “Quite good!” I beg. “We’ve all heard the numbers, 90-95%. Even MicroCOVID.org has made it official: a 10x reduction for Pfizer and Moderna!” The Master of Cost-Benefit Ratio shakes his head. “It helps, it definitely helps, but don’t throw out that spreadsheet just yet. One meal at a crowded restaurant is enough to give even a vaccinated person hundreds of microCovids. Not to mention that your local prevalence could change by a factor of 5 in the next month or two, and that’d be half the gains from this vaccine of yours!” I whimper. “But what if . . . what if vaccines were way better than 10x? What about a 100x reduction in the risks from COVID-19?” He smiles. “Then we could go back to talking about how fast you like to drive.” In its most extreme form, I have heard it claimed that the vaccines provide 10x reduction against regular Covid, 100x against severe Covid, and 1000x against death. That is, for each rough increase in severity, you get 10x more protection. This makes sense if we think of Covid as some kind of "state transition" model where there's a certain chance of moving from lesser to more severe states, and vaccines reduce the likelihood at each stage. I think 10x at multiple stages is too much. By the time you're at 1000x reduction, model uncertainty is probably dominating. I feel more comfortable positing up to 100x, maybe 500x reduction. I dunno. There is a more limited claim of extreme vaccine effectiveness that I will defend today: In the case of the Pfizer vaccine (and likely Moderna too), the effectiveness in young healthy people is 99% against baseline symptomatic infection, or close to it. We can reasonably expect the effectiveness of the vaccine against more severe cases of Covid to be greater than effectiveness against milder cases of Covid. (Maybe it's 2x more effective against severe-Covid and 3x more effective against death compared to just getting it at all. Something lik...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Beware of Other-Optimizing, published by Eliezer Yudkowsky on the LessWrong. I've noticed a serious problem in which aspiring rationalists vastly overestimate their ability to optimize other people's lives. And I think I have some idea of how the problem arises. You read nineteen different webpages advising you about personal improvement—productivity, dieting, saving money. And the writers all sound bright and enthusiastic about Their Method, they tell tales of how it worked for them and promise amazing results... But most of the advice rings so false as to not even seem worth considering. So you sigh, mournfully pondering the wild, childish enthusiasm that people can seem to work up for just about anything, no matter how silly. Pieces of advice #4 and #15 sound interesting, and you try them, but... they don't... quite... well, it fails miserably. The advice was wrong, or you couldn't do it, and either way you're not any better off. And then you read the twentieth piece of advice—or even more, you discover a twentieth method that wasn't in any of the pages—and STARS ABOVE IT ACTUALLY WORKS THIS TIME. At long, long last you have discovered the real way, the right way, the way that actually works. And when someone else gets into the sort of trouble you used to have—well, this time you know how to help them. You can save them all the trouble of reading through nineteen useless pieces of advice and skip directly to the correct answer. As an aspiring rationalist you've already learned that most people don't listen, and you usually don't bother—but this person is a friend, someone you know, someone you trust and respect to listen. And so you put a comradely hand on their shoulder, look them straight in the eyes, and tell them how to do it. I, personally, get quite a lot of this. Because you see... when you've discovered the way that really works... well, you know better by now than to run out and tell your friends and family. But you've got to try telling Eliezer Yudkowsky. He needs it, and there's a pretty good chance that he'll understand. It actually did take me a while to understand. One of the critical events was when someone on the Board of the Institute Which May Not Be Named, told me that I didn't need a salary increase to keep up with inflation—because I could be spending substantially less money on food if I used an online coupon service. And I believed this, because it was a friend I trusted, and it was delivered in a tone of such confidence. So my girlfriend started trying to use the service, and a couple of weeks later she gave up. Now here's the the thing: if I'd run across exactly the same advice about using coupons on some blog somewhere, I probably wouldn't even have paid much attention, just read it and moved on. Even if it were written by Scott Aaronson or some similar person known to be intelligent, I still would have read it and moved on. But because it was delivered to me personally, by a friend who I knew, my brain processed it differently—as though I were being told the secret; and that indeed is the tone in which it was told to me. And it was something of a delayed reaction to realize that I'd simply been told, as personal advice, what otherwise would have been just a blog post somewhere; no more and no less likely to work for me, than a productivity blog post written by any other intelligent person. And because I have encountered a great many people trying to optimize me, I can attest that the advice I get is as wide-ranging as the productivity blogosphere. But others don't see this plethora of productivity advice as indicating that people are diverse in which advice works for them. Instead they see a lot of obviously wrong poor advice. And then they finally discover the right way—the way that works, unlike all those other blog posts that don't wo...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Your intuitions are not magic, published by Kaj_Sotala on the LessWrong. People who know a little bit of statistics - enough to use statistical techniques, not enough to understand why or how they work - often end up horribly misusing them. Statistical tests are complicated mathematical techniques, and to work, they tend to make numerous assumptions. The problem is that if those assumptions are not valid, most statistical tests do not cleanly fail and produce obviously false results. Neither do they require you to carry out impossible mathematical operations, like dividing by zero. Instead, they simply produce results that do not tell you what you think they tell you. As a formal system, pure math exists only inside our heads. We can try to apply it to the real world, but if we are misapplying it, nothing in the system itself will tell us that we're making a mistake. Examples of misapplied statistics have been discussed here before. Cyan discussed a "test" that could only produce one outcome. PhilGoetz critiqued a statistical method which implicitly assumed that taking a healthy dose of vitamins had a comparable effect as taking a toxic dose. Even a very simple statistical technique, like taking the correlation between two variables, might be misleading if you forget about the assumptions it's making. When someone says "correlation", they are most commonly talking about Pearson's correlation coefficient, which seeks to gauge whether there's a linear relationship between two variables. In other words, if X increases, does Y also tend to increase. (Or decrease.) However, like with vitamin dosages and their effects on health, two variables might have a non-linear relationship. Increasing X might increase Y up to a certain point, after which increasing X would decrease Y. Simply calculating Pearson's correlation on two such variables might cause someone to get a low correlation, and therefore conclude that there's no relationship or there's only a weak relationship between the two. (See also Anscombe's quartet.) The lesson here, then, is that not understanding how your analytical tools work will get you incorrect results when you try to analyze something. A person who doesn't stop to consider the assumptions of the techniques she's using is, in effect, thinking that her techniques are magical. No matter how she might use them, they will always produce the right results. Of course, assuming that makes about as much sense as assuming that your hammer is magical and can be used to repair anything. Even if you had a broken window, you could fix that by hitting it with your magic hammer. But I'm not only talking about statistics here, for the same principle can be applied in a more general manner. Every moment in our lives, we are trying to make estimates of the way the world works. Of what causal relationships there are, of what ways of describing the world make sense and which ones don't, which plans will work and which ones will fail. In order to make those estimates, we need to draw on a vast amount of information our brains have gathered throughout our lives. Our brains keep track of countless pieces of information that we will not usually even think about. Few people will explicitly keep track of the amount of different restaurants they've seen. Yet in general, if people are asked about the relative number of restaurants in various fast-food chains, their estimates generally bear a close relation to the truth. But like explicit statistical techniques, the brain makes numerous assumptions when building its models of the world. Newspapers are selective in their reporting of disasters, focusing on rare shocking ones above common mundane ones. Yet our brains assume that we hear about all those disasters because we've personally witnessed them, and that the distribution of d...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Useful Idea of Truth, published by Eliezer Yudkowsky on the LessWrong. (This is the first post of a new Sequence, Highly Advanced Epistemology 101 for Beginners, setting up the Sequence Open Problems in Friendly AI. For experienced readers, this first post may seem somewhat elementary; but it serves as a basis for what follows. And though it may be conventional in standard philosophy, the world at large does not know it, and it is useful to know a compact explanation. Kudos to Alex Altair for helping in the production and editing of this post and Sequence!) I remember this paper I wrote on existentialism. My teacher gave it back with an F. She’d underlined true and truth wherever it appeared in the essay, probably about twenty times, with a question mark beside each. She wanted to know what I meant by truth. -- Danielle Egan I understand what it means for a hypothesis to be elegant, or falsifiable, or compatible with the evidence. It sounds to me like calling a belief ‘true’ or ‘real’ or ‘actual’ is merely the difference between saying you believe something, and saying you really really believe something. -- Dale Carrico What then is truth? A movable host of metaphors, metonymies, and; anthropomorphisms: in short, a sum of human relations which have been poetically and rhetorically intensified, transferred, and embellished, and which, after long usage, seem to a people to be fixed, canonical, and binding. -- Friedrich Nietzche The Sally-Anne False-Belief task is an experiment used to tell whether a child understands the difference between belief and reality. It goes as follows: The child sees Sally hide a marble inside a covered basket, as Anne looks on. Sally leaves the room, and Anne takes the marble out of the basket and hides it inside a lidded box. Anne leaves the room, and Sally returns. The experimenter asks the child where Sally will look for her marble. Children under the age of four say that Sally will look for her marble inside the box. Children over the age of four say that Sally will look for her marble inside the basket. (Attributed to: Baron-Cohen, S., Leslie, L. and Frith, U. (1985) ‘Does the autistic child have a “theory of mind”?’, Cognition, vol. 21, pp. 37–46.) Human children over the age of (typically) four, first begin to understand what it means for Sally to lose her marbles - for Sally's beliefs to stop corresponding to reality. A three-year-old has a model only of where the marble is. A four-year old is developing a theory of mind; they separately model where the marble is and where Sally believes the marble is, so they can notice when the two conflict - when Sally has a false belief. Any meaningful belief has a truth-condition, some way reality can be which can make that belief true, or alternatively false. If Sally's brain holds a mental image of a marble inside the basket, then, in reality itself, the marble can actually be inside the basket - in which case Sally's belief is called 'true', since reality falls inside its truth-condition. Or alternatively, Anne may have taken out the marble and hidden it in the box, in which case Sally's belief is termed 'false', since reality falls outside the belief's truth-condition. The mathematician Alfred Tarski once described the notion of 'truth' via an infinite family of truth-conditions: The sentence 'snow is white' is true if and only if snow is white. The sentence 'the sky is blue' is true if and only if the sky is blue. When you write it out that way, it looks like the distinction might be trivial - indeed, why bother talking about sentences at all, if the sentence looks so much like reality when both are written out as English? But when we go back to the Sally-Anne task, the difference looks much clearer: Sally's belief is embodied in a pattern of neurons and neural firings inside Sally's b...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Build Small Skills in the Right Order , published by lukeprog on the LessWrong. I took some Scientology classes in Hollywood so I could get into their Toastmasters club, which is the best Toastmasters club in L.A. county.1 My first Scientology class, 'Success Through Communication', taught skills that were mostly non-specific to Scientology. At first, the class exercises seemed to teach skills too basic to be worth practicing. Later, I came to respect the class as surprisingly useful. (But please, don't take Scientology classes. They are highly Dark Arts, and extremely manipulative.) For the first exercise, I had to sit upright, still, and silent with my eyes closed for about an hour. I was to remain alert and aware but utterly calm. When my head drooped or my hand twitched, I was forced to start over. It took me five hours of silent sitting to complete the exercise successfully. At first I thought the exercise was stupid, but later I found I was now more in control of my awareness and attention, and less disturbed by things in the environment. For the second exercise, I had to stare directly into someone's eyes without looking away - even for a split second - for 20 minutes in a row. If you've never tried this, you should. It's very difficult. Unfortunately, they first paired me with a 12-year-old girl. I was sure I would freak her out if I stared into her eyes for 20 minutes (it's an intense experience), so I made faces when the instructors weren't looking and waited for them to pair me with an adult. After half a dozen failures, I finally managed to maintain eye contact for 20 minutes in a row, without a single glance away or a long blink. Again, this seemed absurd at the time, but later I discovered that I no longer had any trouble maintaining eye contact with people. This skill is a small one, but it is highly valuable in almost every social endeavor. Later exercises seemed childish. An instructor would ask me simple questions from a book like, "What's that over there?" and I would have to answer correctly: "That's a table." I had to do this for hundreds of questions. But I couldn't just say "That's a table" any old way. I had to say it without a stutter, I had to enunciate, and I had to speak loudly. Answering questions like this 100 times in a row will reveal how often most of us speak softly, fail to enunciate, and use filler words like "um." Every time I did one of those things, I had to start over. In another exercise, the instructor would do everything she could to make me laugh, and I had to sit still and not crack a hint of a smile for 10 minutes in a row. This simple skill took many rounds to master. It is a small skill, but repeating a simple exercise like this will eventually bring almost anyone to mastery of this small skill. At the end of the exercise I had noticeably improved a small part of my self-control mechanism. This class - a religious class I took as an atheist in order to achieve an unrelated goal - turned out to be one of the most important classes I have ever taken in my life. It taught me an important meta-skill I have used to great effect ever since. This is the meta-skill of building small skills in the right order. It is now one of the key tools in my toolkit for instrumental rationality. Why it works Previously, I explained the utility of success spirals2: When you achieve one challenging goal after another, your obviously gain confidence in your ability to succeed. So: give yourself a series of meaningful, challenging but achievable goals, and then achieve them! Set yourself up for success by doing things you know you can succeed at, again and again, to keep your confidence high. Building small skills in the right order is an excellent way to create and maintain success spirals. Trying to master a large skill set like salesmanship ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Incorrect hypotheses point to correct observations, published by Kaj_Sotala on the LessWrong. 1. The Consciousness Researcher and Out-Of-Body Experiences In his book Consciousness and the Brain, cognitive neuroscientist Stansilas Dehaene writes about scientifically investigating people’s reports of their out-of-body experiences: . the Swiss neurologist Olaf Blanke[ did a] beautiful series of experiments on out-of-body experiences. Surgery patients occasionally report leaving their bodies during anesthesia. They describe an irrepressible feeling of hovering at the ceiling and even looking down at their inert body from up there. [...] What kind of brain representation, Blanke asked, underlies our adoption of a specific point of view on the external world? How does the brain assess the body’s location? After investigating many neurological and surgery patients, Blanke discovered that a cortical region in the right temporoparietal junction, when impaired or electrically perturbed, repeatedly caused a sensation of out-of-body transportation. This region is situated in a high-level zone where multiple signals converge: those arising from vision; from the somatosensory and kinesthetic systems (our brain’s map of bodily touch, muscular, and action signals); and from the vestibular system (the biological inertial platform, located in our inner ear, which monitors our head movements). By piecing together these various clues, the brain generates an integrated representation of the body’s location relative to its environment. However, this process can go awry if the signals disagree or become ambiguous as a result of brain damage. Out-of-body flight “really” happens, then—it is a real physical event, but only in the patient’s brain and, as a result, in his subjective experience. The out-of-body state is, by and large, an exacerbated form of the dizziness that we all experience when our vision disagrees with our vestibular system, as on a rocking boat. Blanke went on to show that any human can leave her body: he created just the right amount of stimulation, via synchronized but delocalized visual and touch signals, to elicit an out-of-body experience in the normal brain. Using a clever robot, he even managed to re-create the illusion in a magnetic resonance imager. And while the scanned person experienced the illusion, her brain lit up in the temporoparietal junction—very close to where the patient’s lesions were located. We still do not know exactly how this region works to generate a feeling of self-location. Still, the amazing story of how the out-of-body state moved from parapsychological curiosity to mainstream neuroscience gives a message of hope. Even outlandish subjective phenomena can be traced back to their neural origins. The key is to treat such introspections with just the right amount of seriousness. They do not give direct insights into our brain’s inner mechanisms; rather, they constitute the raw material on which a solid science of consciousness can be properly founded. The naive hypotheses that out-of-body experiences represented the spirit genuinely leaving the body, were incorrect. But they were still pointing to a real observation, namely that there are conditions which create a subjective experience of leaving the body. That observation could then be investigated through scientific means. 2. The Artist and the Criticism In art circles, there’s a common piece of advice that goes along the lines of: When people say that they don’t like something about your work, you should treat that as valid information. When people say why they don’t like it or what you could do to fix it, you should treat that with some skepticism. Outside the art context, if someone tells you that they're pissed off with you as a person (or that you make them feel good), then that's likely...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The unexpected difficulty of comparing AlphaStar to humans published by Richard Korzekwa on the LessWrong. This is crossposted from the AI Impacts blog. Artificial intelligence defeated a pair of professional Starcraft II players for the first time in December 2018. Although this was generally regarded as an impressive achievement, it quickly became clear that not everybody was satisfied with how the AI agent, called AlphaStar, interacted with the game, or how its creator, DeepMind, presented it. Many observers complained that, in spite of DeepMind’s claims that it performed at similar speeds to humans, AlphaStar was able to control the game with greater speed and accuracy than any human, and that this was the reason why it prevailed. Although I think this story is mostly correct, I think it is harder than it looks to compare AlphaStar’s interaction with the game to that of humans, and to determine to what extent this mattered for the outcome of the matches. Merely comparing raw numbers for actions taken per minute (the usual metric for a player’s speed) does not tell the whole story, and appropriately taking into account mouse accuracy, the differences between combat actions and non-combat actions, and the control of the game’s “camera” turns out to be quite difficult. Here, I begin with an overview of Starcraft II as a platform for AI research, a timeline of events leading up to AlphaStar’s success, and a brief description of how AlphaStar works. Next, I explain why measuring performance in Starcraft II is hard, show some analysis on the speed of both human and AI players, and offer some preliminary conclusions on how AlphaStar’s speed compares to humans. After this, I discuss the differences in how humans and AlphaStar “see” the game and the impact this has on performance. Finally, I give an update on DeepMind’s current experiments with Starcraft II and explain why I expect we will encounter similar difficulties when comparing human and AI performance in the future. Why Starcraft is a Target for AI Research Starcraft II has been a target for AI for several years, and some readers will recall that Starcraft II appeared on our 2016 expert survey. But there are many games and many AIs that play them, so it may not be obvious why Starcraft II is a target for research or why it is of interest to those of us that are trying to understand what is happening with AI. For the most part, Starcraft II was chosen because it is popular, and it is difficult for AI. Starcraft II is a real time strategy game, and like similar games, it requires a variety of tasks: harvesting resources, constructing bases, researching technology, building armies, and attempting to destroy their opponent’s base are all part of the game. Playing it well requires balancing attention between many things at once: planning ahead, ensuring that one’s units1 are good counters for the enemy’s units, predicting opponents’ moves, and changing plans in response to new information. There are other aspects that make it difficult for AI in particular: it has imperfect information2, an extremely large action space, and takes place in real time. When humans play, they engage in long term planning, making the best use of their limited capacity for attention, and crafting ploys to deceive the other players. The game’s popularity is important because it makes it a good source of extremely high human talent and increases the number of people that will intuitively understand how difficult the task is for a computer. Additionally, as a game that is designed to be suitable for high-level competition, the game is carefully balanced so that competition is fair, does not favor just one strategy3, and does not rely too heavily on luck. Timeline of Events To put AlphaStar’s performance in context, it helps to understand the ti...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Conflict vs. mistake in non-zero-sum games, published by Nisan on the LessWrong. Summary: Whether you behave like a mistake theorist or a conflict theorist may depend more on your negotiating position in a non-zero-sum game than on your worldview. Disclaimer: I don't really know game theory. Plot the payoffs in a non-zero-sum two-player game, and you'll get a convex[1] set with the Pareto frontier on the top and right: Payoff to player 2 vs. payoff to player 1 You can describe this set with two parameters: The surplus is how close the outcome is to the Pareto frontier, and the allocation tells you how much the outcome favors player 1 versus player 2. In this illustration, the level sets for surplus and allocation are depicted by concentric curves and radial lines, respectively. It's tempting to decompose the game into two phases: A cooperative phase, where the players coordinate to maximize surplus; and a competitive phase, where the players negotiate how the surplus is allocated. Of course, in the usual formulation, both phases occur simultaneously. But this suggests a couple of negotiation strategies where you try to make one phase happen before the other: "Let's agree to maximize surplus. Once we agree to that, we can talk about allocation." "Let's agree on an allocation. Once we do that, we can talk about maximizing surplus." I'm going to provocatively call the first strategy mistake theory, and the second conflict theory. Indeed, the mistake theory strategy pushes the obviously good plan of making things better off for everyone. It can frame all opposition as making the mistake of leaving surplus on the table. The conflict theory strategy threatens to destroy surplus in order to get a more favorable allocation. Its narrative emphasizes the fact that the players can't maximize their rewards simultaneously. Now I don't have a good model of negotiation. But intuitively, it seems that mistake theory is a good strategy if you think you'll be in a better negotiating position once you move to the Pareto frontier. And conflict theory is a good strategy if you think you'll be in a worse negotiating position at the Pareto frontier. If you're naturally a mistake theorist, this might make conflict theory seem more appealing. Imagine negotiating with a paperclip maximizer over the fate of billions of lives. Mutual cooperation is Pareto efficient, but unappealing. It's more sensible to threaten defection in order to save a few more human lives, if you can get away with it. It also makes mistake theory seem unsavory: Apparently mistake theory is about postponing the allocation negotiation until you're in a comfortable negotiating position. (Or, somewhat better: It's about tricking the other players into cooperating before they can extract concessions from you.) This is kind of unfair to mistake theory, which is supposed to be about educating decision-makers on efficient policies and building institutions to enable cooperation. None of that is present in this model. But I think it describes something important about mistake theory which is usually rounded off to something like "[mistake theorists have] become part of a class that’s more interested in protecting its own privileges than in helping the poor or working for the good of all". The reason I'm thinking about this is that I want a theory of non-zero-sum games involving counterfactual reasoning and superrationality. It's not clear to me what superrational agents "should" do in general non-zero-sum games. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Comparative advantage and when to blow up your island, published by dynomight on the LessWrong. This is a linkpost for/ Economists say free trade is good because of "comparative advantage". But what is comparative advantage? Why is it good? Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Pseudorandomness contest: prizes, results, and analysis , published by UnexpectedValues on the LessWrong. This is a linkpost for/ (Previously in this series: Round 1, Round 2) In December I ran a pseudorandomness contest. Here’s how it worked: In Round 1, participants were invited to submit 150-bit strings of their own devising. They had 10 minutes to write down their string while using nothing but their own minds. I received 62 submissions. I then used a computer to generate 62 random 150-bit strings, and put all 124 strings in a random order. In Round 2, participants had to figure out which strings were human-generated (I’m going to call these strings fake from now on) and which were “truly” random (I’m going to call these real). In particular, I asked for probabilities that each string was real, so participants could express their confidence rather than guessing “real” or “fake” for each string. I received 27 submissions for Round 2. This post is long because there are lots of fascinating things to talk about. So, feel free to skip around to whichever sections you find most interesting; I’ve done my best to give descriptive labels. But first: Prizes Round 1 Thank you to the 62 of you who submitted strings in Round 1! Your strings were scored by the average probability of being real assigned by Round 2 participants, weighted by their Round 2 score. (Entries with negative Round 2 scores received no weight). The top three scores in Round 1 were: Jenny Kaufmann, with a score of 69.4%. That is, even though Jenny’s string was fake, Round 2 participants on average gave her string a 69.4% chance of being real. For winning Round 1, Jenny was given the opportunity to allocate $50 to charity, which she chose to give to the GiveWell Maximum Impact Fund. Reed Jacobs, with a score of 68.8%. Reed allocated $25 to Canada/USA Mathcamp. Eric Fletcher, with a score of 68.6%. Eric allocated $25 to the Poor People’s Campaign. Congratulations to Jenny, Reed, and Eric! Round 2 A big thanks to the 27 of you (well, 28 — 26 plus a team of two) who submitted Round 2 entries. I estimate that the average participant put in a few hours of work, and that some put in more than 10. Entries were graded using a quadratic scoring rule (see here for details). When describing Round 2, I did a back-of-the-envelope estimate that a score of 15 on this round would be good. I was really impressed by the top two scores: Scy Yoon and William Ehlhardt, who were the only team, received a score of 28.5, honestly higher than I thought possible. They allocated $150 to the GiveWell Maximum Impact Fund. Ben Edelman received a score of 25.8. He allocated $75 to the Humane League. Three other participants received a score of over 15: simon received a score of 21.0. He allocated $25 to the Machine Intelligence Research Institute. Adam Hesterberg received a score of 19.5. He allocated $25 to the Sierra Club Beyond Coal campaign. Viktor Bowallius received a score of 17.3. He allocated $25 to the EA Long Term Future Fund. Congratulations to Scy, William, Ben, simon, Adam, and Viktor! All right, let’s take a look at what people did and how well it worked! Round 1 analysis Summary statistics Recall that the score of a Round 1 entry is a weighted average of the probabilities assigned by Round 2 participants to the entry being real (i.e. truly random). The average score was 39.4% (this is well below 50%, as expected). The median score was 45.7%. Here’s the full distribution: Figure 1: Histogram of Round 1 scores Interesting: the distribution is bimodal! Some people basically succeeded at fooling Round 2 participants, and most of the rest came up with strings that were pretty detectable as fakes. Methods I asked participants to describe the method they used to generate their string. Of the 58 participants who told me what the...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Apprentice Thread, published by Zvi on the LessWrong. A while back, LessWrong poster Aysajan put up a post asking to be someone’s apprentice. He talked about it with johnswentworth, who I recently confirmed via meeting him in person is awesome and does reliably interesting work, and an apprentice experiment was born. As John says, you gotta admire the chutzpah. Asking for what one wants is a known to be successful but highly underused strategy, I presume mostly because of the permanent global chutzpah shortage and the associated danger that it might result in mild social awkwardness. In addition to the highly successful use of chutzpah, this also points out that apprenticeships are also a known to be successful but highly underused strategy. My feelings about so-called ‘schools’ are well known, but education is great, and apprenticeship is one of the best ways to get an actually excellent and useful education. I’ve been an apprentice, regardless of whether it was called that, and it was awesome. At Jane Street they have a formal education process, but the core of how you get good is an apprenticeship. You learn from the best, by working with the best and asking them questions. I believe I am a natural trader, but I am good largely because I learned from working directly with three of the best in succession at three different jobs. I know Magic and game design by spending tons of time talking and working with top Magic players and game designers. I’ve been a mentor of sorts a few times. That’s mostly been great too, and I’m plausibly taking on another one now, although I’m cheating because he is already exceptional and largely does not need the help. Thus, we have two overpowered strategies here that the world needs more of: Apprenticeship and Chutzpah. Lowering the activation energy required for either or both of them seems great, as does providing encouragement. Both can of course be overused and abused. Too much of the wrong kind of chutzpah is no good for anyone, and apprenticeship can turn into a bunch of not very useful unpaid work, or end up holding people back. In the context of posts like this, I am not much worried about either of these failure modes. For this post/thread I will focus on apprenticeship. In particular, I want to see if I can give social permission and a coordination mechanism that can perhaps take place in the comments (reminder that my posts have two comments sections, the primary one at DWATV and a secondary one at LessWrong). Replies to this post should take the form of any of the following: [MENTOR]: A non-binding indication of potential interest in mentorship. Mention that you might, at some point, be interested in taking on an apprentice. This commits you to nothing. Make sure to indicate what you’d be teaching them and what project would likely be involved, and open with [MENTOR]. You are free to include contact info, or not include it and monitor replies. Replies to this comment to indicate potential interest in being the apprentice, marked [APPRENTICE], which should include a method of further contact. [APPRENTICE]: A non-binding indication of potential interest in being an apprentice. Mention that you might, at some point, be interested in being an apprentice. This commits you to nothing. Make sure to indicate what you’re interested in being an apprentice in and learning, and an indication of what’s motivating you. Replies to this comment to indicate potential interest in being the mentor, marked with [MENTOR], which should include a method of further contact. [NORMAL] You’re free to comment as per normal, but start with [NORMAL] in the top-level for clarity. [NYCBUSINESS] if there’s some chance, depending on what it is, that you would want to do the thing I talk about below. Cost of speaking up is low, potential upside is high,...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Disguised Queries, published by Eliezer Yudkowsky on the LessWrong. Imagine that you have a peculiar job in a peculiar factory: Your task is to take objects from a mysterious conveyor belt, and sort the objects into two bins. When you first arrive, Susan the Senior Sorter explains to you that blue egg-shaped objects are called "bleggs" and go in the "blegg bin", while red cubes are called "rubes" and go in the "rube bin". Once you start working, you notice that bleggs and rubes differ in ways besides color and shape. Bleggs have fur on their surface, while rubes are smooth. Bleggs flex slightly to the touch; rubes are hard. Bleggs are opaque; the rube's surface slightly translucent. Soon after you begin working, you encounter a blegg shaded an unusually dark blue—in fact, on closer examination, the color proves to be purple, halfway between red and blue. Yet wait! Why are you calling this object a "blegg"? A "blegg" was originally defined as blue and egg-shaped—the qualification of blueness appears in the very name "blegg", in fact. This object is not blue. One of the necessary qualifications is missing; you should call this a "purple egg-shaped object", not a "blegg". But it so happens that, in addition to being purple and egg-shaped, the object is also furred, flexible, and opaque. So when you saw the object, you thought, "Oh, a strangely colored blegg." It certainly isn't a rube... right? Still, you aren't quite sure what to do next. So you call over Susan the Senior Sorter. "Oh, yes, it's a blegg," Susan says, "you can put it in the blegg bin." You start to toss the purple blegg into the blegg bin, but pause for a moment. "Susan," you say, "how do you know this is a blegg?" Susan looks at you oddly. "Isn't it obvious? This object may be purple, but it's still egg-shaped, furred, flexible, and opaque, like all the other bleggs. You've got to expect a few color defects. Or is this one of those philosophical conundrums, like 'How do you know the world wasn't created five minutes ago complete with false memories?' In a philosophical sense I'm not absolutely certain that this is a blegg, but it seems like a good guess." "No, I mean..." You pause, searching for words. "Why is there a blegg bin and a rube bin? What's the difference between bleggs and rubes?" "Bleggs are blue and egg-shaped, rubes are red and cube-shaped," Susan says patiently. "You got the standard orientation lecture, right?" "Why do bleggs and rubes need to be sorted?" "Er... because otherwise they'd be all mixed up?" says Susan. "Because nobody will pay us to sit around all day and not sort bleggs and rubes?" "Who originally determined that the first blue egg-shaped object was a 'blegg', and how did they determine that?" Susan shrugs. "I suppose you could just as easily call the red cube-shaped objects 'bleggs' and the blue egg-shaped objects 'rubes', but it seems easier to remember this way." You think for a moment. "Suppose a completely mixed-up object came off the conveyor. Like, an orange sphere-shaped furred translucent object with writhing green tentacles. How could I tell whether it was a blegg or a rube?" "Wow, no one's ever found an object that mixed up," says Susan, "but I guess we'd take it to the sorting scanner." "How does the sorting scanner work?" you inquire. "X-rays? Magnetic resonance imaging? Fast neutron transmission spectroscopy?" "I'm told it works by Bayes's Rule, but I don't quite understand how," says Susan. "I like to say it, though. Bayes Bayes Bayes Bayes Bayes." "What does the sorting scanner tell you?" "It tells you whether to put the object into the blegg bin or the rube bin. That's why it's called a sorting scanner." At this point you fall silent. "Incidentally," Susan says casually, "it may interest you to know that bleggs contain small nuggets of vanadium ore, and ru...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Voting Puzzle, Some Political Science, and a Nerd Failure Mode, published by ChrisHallquist on the LessWrong. In grade school, I read a series of books titled Sideways Stories from Wayside School by Louis Sachar, who you may know as the author of the novel Holes which was made into a movie in 2003. The series included two books of math problems, Sideways Arithmetic from Wayside School and More Sideways Arithmetic from Wayside School, the latter of which included the following problem (paraphrased): The students have Mrs. Jewl's class have been given the privilege of voting on the height of the school's new flagpole. She has each of them write down what they think would be the best hight for the flagpole. The votes are distributed as follows: 1 student votes for 6 feet. 1 student votes for 10 feet. 7 students vote for 25 feet. 1 student votes for 30 feet. 2 students vote for 50 feet. 2 students vote for 60 feet. 1 student votes for 65 feet. 3 students vote for 75 feet. 1 student votes for 80 feet, 6 inches. 4 students vote for 85 feet. 1 student votes for 91 feet. 5 students vote for 100 feet. At first, Mrs. Jewls declares 25 feet the winning answer, but one of the students who voted for 100 feet convinces her there should be a runoff between 25 feet and 100 feet. In the runoff, each student votes for the height closest to their original answer. But after that round of voting, one of the students who voted for 85 feet wants their turn, so 85 feet goes up against the winner of the previous round of voting, and the students vote the same way, with each student voting for the height closest to their original answer. Then the same thing happens again with the 50 foot option. And so on, with each number, again and again, "very much like a game of tether ball." Question: if this process continues until it settles on an answer that can't be beaten by any other answer, how tall will the new flagpole be? Answer (rot13'd): fvkgl-svir srrg, orpnhfr gung'f gur zrqvna inyhr bs gur bevtvany frg bs ibgrf. Naq abj lbh xabj gur fgbel bs zl svefg rapbhagre jvgu gur zrqvna ibgre gurberz. Why am I telling you this? There's a minor reason and a major reason. The minor reason is that this shows it is possible to explain little-known academic concepts, at least certain ones, in a way that grade schoolers will understand. It's a data point that fits nicely with what Eliezer has written about how to explain things. The major reason, though, is that a month ago I finished my systematic read-through of the sequences and while I generally agree that they're awesome (perhaps moreso than most people; I didn't see the problem with the metaethics sequence), I thought the mini-discussion of political parties and voting was on reflection weak and indicative of a broader nerd failure mode. TLDR (courtesy of lavalamp): Politicians probably conform to the median voter's views. Most voters are not the median, so most people usually dislike the winning politicians. But people dislike the politicians for different reasons. Nerds should avoid giving advice that boils down to "behave optimally". Instead, analyze the reasons for the current failure to behave optimally and give more targeted advice. Advance warning for heavy US slant, at least in terms of examples, though the theory is applicable everywhere. The median voter theorem The median voter theorem was first laid out in a paper by Duncan Black titled "On the Rationale of Group Decision-Making," which imagine's a situation very much like Mrs. Jewls' class voting on the flagpole height: a committee passes a motion by majority vote, and then it considers various motions to amend the original motion, each of which itself needs a simple majority to pass. Each member of the committee has preferences over the range of possible motions, and furthermore: Whil...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Ritual Report: NYC Less Wrong Solstice Celebration, published by Raemon on the LessWrong. Note: Secular Solstice has evolved a bit since this original post (most noteably, it no longer has a major Lovecraft theme. Last Friday, the NYC Less Wrong community held their first Winter Solstice Celebration. Approximately twenty of us gathered for dinner and a night of ritual. We sang songs, told stories, and recited litanies. The night celebrated ancient astronomers, and the work that humanity has done for the past 5000 years. It paid tribute to the harshness of the universe, respecting it as worthy opponent. We explored Lovecraftian mythology, which intersects with our beliefs in interesting ways. And finally, we looked to the future, vowing to give a gift to tomorrow. This is the first of 2-3 posts on this subject. In this one, I'm telling a story about what we did and why I wanted to. In the followup(s), I’ll explain the design principles that went into planning such an event, and what we learned from our first execution of it. I’ll also be posting a PDF of a ritual book, similar to the one we read from but with a few changes based on initial, obvious observations. Why exactly did we do this? Doesn’t this smack of organized religion? Who the hell is Lovecraft and why do we care? Depending on your background, this may require the bridging of some inferential distance, as well as emotional distance. Bear with me. (If at the end, you DO still think this was a dangerous idea, or one you don't want popularized on Less Wrong, I want you to let me know. We're probably just going to disagree, but I want a sense of what the costs are of emphasizing this type of thing here) Winter Solstice To begin, a Just So Story, true enough for our purposes: The Winter Solstice is the longest night of the year. It ushers in a time of cold and darkness. For young civilizations, it was a time when if you HADN’T spent the year preparing adequately for the future, then before spring returned, you would run out of food and die. If you hadn’t striven to use your tribe’s collective wisdom, to work hard beyond what was necessary for immediate gratification... if you hadn’t harnessed the physical and mental tools that humans have but that few other animals do... then the universe, unflinchingly neutral, would destroy you without a second thought. And even if you did do these things, it might kill you anyway. Because fairness isn’t built into the equations of the cosmos. But it wasn't just the threat of death that inspired the first winter holidays. It was that sense of unfairness, coupled with the desperate hope that world couldn’t really be that unfair. It wouldn’t have occurred to the first squirrels that stored food for winter, but it gradually dawned upon ancient hominids, as their capacity for abstract reasoning developed, alongside their desire to throw parties. Our tendency is to anthropomorphize. Today, we angrily yell at our cars and computers when they fail us. Rationally we know they are unthinking hulks of metal, but we still ascribe malevolence when the real culprit is a broken, unsentient machine. There are plausible reasons for humans to have evolved this trait. One of the most complicated tasks a human has to do is predict the actions of other humans. We need to be able to make allies, to identify deceptive enemies, to please lovers. I’m not an evolutionary psychologist and I should be careful when telling this sort of Just-So story, but I can easily imagine selection pressures that resulted in a powerful ability to draw conclusions about sentient creatures similar to ourselves. And then, there was NOT a whole lot of pressure to NOT use this tool to predict, say, the weather. Many natural forces are just too complex for humans to be good at predicting. The rain would come, or it wouldn’...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Ritual Report: NYC Less Wrong Solstice Celebration , published by jacob_cannell on the LessWrong. This article presents an emerging architectural hypothesis of the brain as a biological implementation of a Universal Learning Machine. I present a rough but complete architectural view of how the brain works under the universal learning hypothesis. I also contrast this new viewpoint - which comes from computational neuroscience and machine learning - with the older evolved modularity hypothesis popular in evolutionary psychology and the heuristics and biases literature. These two conceptions of the brain lead to very different predictions for the likely route to AGI, the value of neuroscience, the expected differences between AGI and humans, and thus any consequent safety issues and dependent strategies. Art generated by an artificial neural net (The image above is from a recent mysterious post to r/machinelearning, probably from a Google project that generates art based on a visualization tool used to inspect the patterns learned by convolutional neural networks. I am especially fond of the wierd figures riding the cart in the lower left. ) Intro: Two viewpoints on the Mind Universal Learning Machines Historical Interlude Dynamic Rewiring Brain Architecture (the whole brain in one picture and a few pages of text) The Basal Ganglia Implications for AGI Conclusion Intro: Two Viewpoints on the Mind Few discoveries are more irritating than those that expose the pedigree of ideas. -- Lord Acton (probably) Less Wrong is a site devoted to refining the art of human rationality, where rationality is based on an idealized conceptualization of how minds should or could work. Less Wrong and its founding sequences draws heavily on the heuristics and biases literature in cognitive psychology and related work in evolutionary psychology. More specifically the sequences build upon a specific cluster in the space of cognitive theories, which can be identified in particular with the highly influential "evolved modularity" perspective of Cosmides and Tooby. From Wikipedia: Evolutionary psychologists propose that the mind is made up of genetically influenced and domain-specific[3] mental algorithms or computational modules, designed to solve specific evolutionary problems of the past.[4] From "Evolutionary Psychology and the Emotions":[5] An evolutionary perspective leads one to view the mind as a crowded zoo of evolved, domain-specific programs. Each is functionally specialized for solving a different adaptive problem that arose during hominid evolutionary history, such as face recognition, foraging, mate choice, heart rate regulation, sleep management, or predator vigilance, and each is activated by a different set of cues from the environment. If you imagine these general theories or perspectives on the brain/mind as points in theory space, the evolved modularity cluster posits that much of the machinery of human mental algorithms is largely innate. General learning - if it exists at all - exists only in specific modules; in most modules learning is relegated to the role of adapting existing algorithms and acquiring data; the impact of the information environment is de-emphasized. In this view the brain is a complex messy cludge of evolved mechanisms. There is another viewpoint cluster, more popular in computational neuroscience (especially today), that is almost the exact opposite of the evolved modularity hypothesis. I will rebrand this viewpoint the "universal learner" hypothesis, aka the "one learning algorithm" hypothesis (the rebranding is justified mainly by the inclusion of some newer theories and evidence for the basal ganglia as a 'CPU' which learns to control the cortex). The roots of the universal learning hypothesis can be traced back to Mountcastle's discovery of the simpl...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Everyday Lessons from High-Dimensional Optimization, published by johnswentworth on the LessWrong. Suppose you’re designing a bridge. There’s a massive number of variables you can tweak: overall shape, relative positions and connectivity of components, even the dimensions and material of every beam and rivet. Even for a small footbridge, we’re talking about at least thousands of variables. For a large project, millions if not billions. Every one of those is a dimension over which we could, in principle, optimize. Suppose you have a website, and you want to increase sign-ups. There’s a massive number of variables you can tweak: ad copy/photos/videos, spend distribution across ad channels, home page copy/photos/videos, button sizes and positions, page colors and styling. and every one of those is itself high-dimensional. Every word choice, every color, every position of every button, header, divider, sidebar, box, link. every one of those is a variable, adding up to thousands of dimensions over which to optimize. Suppose you’re a startup founder planning your day - just a normal workday. There’s a massive number of variables you could tweak: dozens of people you could talk to, dozens of things you could talk to any of them about, and all the possible combinations of people and topics. There’s emails, code, designs and plans you could write. Every choice is a dimension over which you could optimize. Point being: real-world optimization problems are usually pretty high dimensional. Unfortunately, many techniques and intuitions which work well for low-dimensional optimization do not scale up well to higher dimensions. This post talks about some of these problems, and what they look like in real life. Try It and See Let’s start with a baseline: try something at random, see how well it works. If it doesn’t work, or doesn’t work better than your current best choice, then throw it out and try something else at random. In low-dimensional problems, this isn’t a bad approach. Want to decide which brand of soap to use? Try it and see. There just aren’t that many possible choices, so you’ll pretty quickly try all of them and settle on the best. On the other hand, we probably don’t want to design a bridge by creating a design completely at random, checking whether it works, then throwing it out and trying another design at random if it doesn’t. In general, the number of possible states/designs/configurations in a space increases exponentially with the number of dimensions. A problem in two or three dimensions, with k choices for each variable, will only have k^2 or k^3 possibilities. A problem in a hundred thousand dimensions will have k^100000 - well in excess of the number of electrons in the universe, even if there’s only two choices per variable. The number of possible bridge designs is exponentially huge; selecting designs at random will not ever find the best design, or anything close to it. Let’s look at a less obvious example. From Studies on Slack: Imagine a distant planet full of eyeless animals. Evolving eyes is hard: they need to evolve Eye Part 1, then Eye Part 2, then Eye Part 3, in that order. Each of these requires a separate series of rare mutations. Here on Earth, scientists believe each of these mutations must have had its own benefits – in the land of the blind, the man with only Eye Part 1 is king. But on this hypothetical alien planet, there is no such luck. You need all three Eye Parts or they’re useless. Worse, each Eye Part is metabolically costly [...] So these animals will only evolve eyes in conditions of relatively weak evolutionary pressure. See the mistake? When evolutionary pressure is low, we explore the space of organism-designs more-or-less at random. Now, if we imagine mutations just stepping left or right on a one-dimensional line, with the thre...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Fun with +12 OOMs of Compute, published by Daniel Kokotajlo on the LessWrong. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. Or: Big Timelines Crux Operationalized What fun things could one build with +12 orders of magnitude of compute? By ‘fun’ I mean ‘powerful.’ This hypothetical is highly relevant to AI timelines, for reasons I’ll explain later. Summary (Spoilers): I describe a hypothetical scenario that concretizes the question “what could be built with 2020’s algorithms/ideas/etc. but a trillion times more compute?” Then I give some answers to that question. Then I ask: How likely is it that some sort of TAI would happen in this scenario? This second question is a useful operationalization of the (IMO) most important, most-commonly-discussed timelines crux: “Can we get TAI just by throwing more compute at the problem?” I consider this operationalization to be the main contribution of this post; it directly plugs into Ajeya’s timelines model and is quantitatively more cruxy than anything else I know of. The secondary contribution of this post is my set of answers to the first question: They serve as intuition pumps for my answer to the second, which strongly supports my views on timelines. The hypothetical In 2016 the Compute Fairy visits Earth and bestows a blessing: Computers are magically 12 orders of magnitude faster! Over the next five years, what happens? The Deep Learning AI Boom still happens, only much crazier: Instead of making AlphaStar for 10^23 floating point operations, DeepMind makes something for 10^35. Instead of making GPT-3 for 10^23 FLOPs, OpenAI makes something for 10^35. Instead of industry and academia making a cornucopia of things for 10^20 FLOPs or so, they make a cornucopia of things for 10^32 FLOPs or so. When random grad students and hackers spin up neural nets on their laptops, they have a trillion times more compute to work with. [EDIT: Also assume magic +12 OOMs of memory, bandwidth, etc. All the ingredients of compute.] For context on how big a deal +12 OOMs is, consider the graph below, from ARK. It’s measuring petaflop-days, which are about 10^20 FLOP each. So 10^35 FLOP is 1e+15 on this graph. GPT-3 and AlphaStar are not on this graph, but if they were they would be in the very top-right corner. Question One: In this hypothetical, what sorts of things could AI projects build? I encourage you to stop reading, set a five-minute timer, and think about fun things that could be built in this scenario. I’d love it if you wrote up your answers in the comments! My tentative answers: Below are my answers, listed in rough order of how ‘fun’ they seem to me. I’m not an AI scientist so I expect my answers to overestimate what could be done in some ways, and underestimate in other ways. Imagine that each entry is the best version of itself, since it is built by experts (who have experience with smaller-scale versions) rather than by me. OmegaStar: In our timeline, it cost about 10^23 FLOP to train AlphaStar. (OpenAI Five, which is in some ways more impressive, took less!) Let’s make OmegaStar like AlphaStar only +7 OOMs bigger: the size of a human brain.[1] [EDIT: You may be surprised to learn, as I was, that AlphaStar has about 10% as many parameters as a honeybee has synapses! Playing against it is like playing against a tiny game-playing insect.] Larger models seem to take less data to reach the same level of performance, so it would probably take at most 10^30 FLOP to reach the same level of Starcraft performance as AlphaStar, and indeed we should expect it to be qualitatively better.[2] So let’s do that, but also train it on lots of other games too.[3] There are 30,000 games in the Steam Library. We train OmegaStar long enough that it has as much time on each game as AlphaStar had on Starcraft. Wi...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Small and Vulnerable , published by deluks917 on the LessWrong. Anyone who is dedicating the majority of their time or money to Effective Altruism needs to ask themselves why. Why not focus on enjoying life and spending your time doing what you love most? Here is my answer: I have a twin sister but neither of us had many other friends growing up. From second to fifth grade we had none. From sixth to eighth we had one friend. As you might guess I was bullied quite badly. Multiple teachers contributed to this. Despite having no friends my parents wanted us to be normal. They pressured me to play sports with the boys in the neighborhood. I was unable to play with an acceptable level of skill and was not invited to the games anyway. But we were still forced to go 'play outside' after school. We had to find ways to kill time. Often we literally rode our bicycles in a circle in a parking lot. We were forced to 'play outside' for hours most days and even longer on weekends. I was not even allowed to bring a book outside though sometimes I would hide them outside at night and find them the next day. Until high school, I had no access to the internet. After dinner, I could watch TV, read and play video games. These were the main sources of joy in my childhood. Amazingly my mom made fun of her children for being weirdos. My sister used to face a wall and stim with her fingers when she was overwhelmed. For some reason, my mom interpreted this as 'OCD'. So she made up a song titled 'OCD! Do you mean me?' It had several verses! This is just one, especially insane, example. My dad liked to 'slap me around. He usually did not hit me very hard but he would slap me in the face all the time. He also loved to call me 'boy' instead of my name. He claims he got this idea from Tarzan. It took me years to stop flinching when people raised their hands or put them anywhere near my face. I have struggled with gender since childhood. My parents did not tolerate even minor gender nonconformity like growing my hair out. I would get hit reasonably hard if I insisted on something as 'extreme' as crossing my legs 'like a girl in public. I recently started HRT and already feel much better. My family is a lot of the reason I delayed transitioning. If you go by the checklist I have quite severe ADHD. 'Very often' seemed like an understatement for most of the questions. My ADHD was untreated until recently. I could not focus on school or homework so trying to do my homework took way too much time. I was always in trouble in school and considered a very bad student. It definitely hurts when authority figures constantly, and often explicitly, treat you like a fuck up and a failure who can't be trusted. But looking back it seems amazing I was considered such a bad student. I love most of the subjects you study in school! When I finally got access to the internet I spent hours per day reading Wikipedia articles. I still spend a lot of time listening to lectures on all sorts of subjects, especially history. Why were people so cruel to a little child who wanted to learn things? Luckily things improved in high school. Once I had more freedom and distance from my parents my social skills improved a huge amount. In high school, I finally had internet access which helped an enormous amount. My parents finally connected our computer at home to the internet because they thought my sister and I needed it for school. I also had access to the computers in the high school library. By my junior year in high school, I was not really unpopular. Ironically my parent's overbearing pressure to be a 'normal kid' probably prevented me from having a social life until I got a little independence. Sadly I was still constantly in trouble in school throughout my high school years. The abuse at home was very bad. But, to be honest,...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.. This is: Universal Fire , published by Eliezer Yudkowsky on the LessWrong. In L. Sprague de Camp's fantasy story The Incomplete Enchanter (which set the mold for the many imitations that followed), the hero, Harold Shea, is transported from our own universe into the universe of Norse mythology. This world is based on magic rather than technology; so naturally, when Our Hero tries to light a fire with a match brought along from Earth, the match fails to strike. I realize it was only a fantasy story, but... how do I put this... No. In the late eighteenth century, Antoine-Laurent de Lavoisier discovered fire. "What?" you say. "Hasn't the use of fire been dated back for hundreds of thousands of years?" Well, yes, people used fire; it was hot, bright, sort of orangey-colored, and you could use it to cook things. But nobody knew how it worked. Greek and medieval alchemists thought that Fire was a basic thing, one of the Four Elements. In Lavoisier's time the alchemical paradigm had been gradually amended and greatly complicated, but fire was still held to be basic - in the form of "phlogiston", a rather mysterious substance which was said to explain fire, and also every other phenomenon in alchemy. Lavoisier's great innovation was to weigh all the pieces of the chemical puzzle, both before and after the chemical reaction. It had previously been thought that some chemical transmutations changed the weight of the total material: If you subjected finely ground antimony to the focused sunlight of a burning glass, the antimony would be reduced to ashes after one hour, and the ashes would weigh one-tenth more than the original antimony - even though the burning had been accompanied by the loss of a thick white smoke. Lavoisier weighed all the components of such reactions, including the air in which the reaction took place, and discovered that matter was neither created nor destroyed. If the burnt ashes increased in weight, there was a corresponding decrease in the weight of the air. Lavoisier also knew how to separate gases, and discovered that a burning candle diminished the amount of one kind of gas, vital air, and produced another gas, fixed air. Today we would call them oxygen and carbon dioxide. When the vital air was exhausted, the fire went out. One might guess, perhaps, that combustion transformed vital air into fixed air and fuel to ash, and that the ability of this transformation to continue was limited by the amount of vital air available. Lavoisier's proposal directly contradicted the then-current phlogiston theory. That alone would have been shocking enough, but it also turned out... To appreciate what comes next, you must put yourself into an eighteenth-century frame of mind. Forget the discovery of DNA, which occurred only in 1953. Unlearn the cell theory of biology, which was formulated in 1839. Imagine looking at your hand, flexing your fingers... and having absolutely no idea how it worked. The anatomy of muscle and bone was known, but no one had any notion of "what makes it go" - why a muscle moves and flexes, while clay molded into a similar shape just sits there. Imagine your own body being composed of mysterious, incomprehensible gloop. And then, imagine discovering... ...that humans, in the course of breathing, consumed vital air and breathed out fixed air. People also ran on combustion! Lavoisier measured the amount of heat that animals (and Lavoisier's assistant, Seguin) produced when exercising, the amount of vital air consumed, and the fixed air breathed out. When animals produced more heat, they consumed more vital air and exhaled more fixed air. People, like fire, consumed fuel and oxygen; people, like fire, produced heat and carbon dioxide. Deprive people of oxygen, or fuel, and the light goes out. Matches catch fire because of phosphorus - "safety matches"...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Problem Solving with Mazes and Crayon, published by johnswentworth on the LessWrong. I want to talk about a few different approaches to general problem solving (for humans). It turns out that they can all be applied to mazes, so I’ll use some disney-themed mazes to illustrate each approach. We’ll start off with some traditional path-search algorithms (DFS, BFS, heuristic). Next, we’ll talk about how these algorithms can fall short for everyday problem solving. Then we’ll move on to the interesting part: two techniques which often work better for everyday problem solving, and lend some interesting insights when applied to mazes. I’ll assume no technical background at all, so if you’ve seen some of this stuff before, feel free to skim it. DFS and BFS You have a maze, with a start point and an end point, and you are searching for a path through it. In algorithms classes, this problem is called “path search”. The very first path search algorithms students typically learn are depth-first search (DFS) and breadth-first search (BFS). Here’s DFS, applied to the Pinocchio maze above: Basically, the DFS rule is “always take the right-most path which you haven’t already explored”. So, in the Pinoccchio maze, we start out by turning right and running until we hit a wall (the first “x”). Then we turn around and go back, find another right turn, and hit another dead end. Turn around again, continue. That’s depth-first search. We go as far as we can down one path (“depth-first”) and if we hit a dead end, we turn around, back up, and try another path. Breadth-first search, on the other hand, tries all paths in parallel: In this snapshot, we’ve hit three dead ends, and we have four separate “branches” still exploring the maze. It’s like a plant: every time BFS hits an intersection, it splits and goes both ways. It’s “breadth-first”: it explores all paths at once, keeping the search as wide as possible. There’s lots more to say about these two algorithms, but main point is what they have in common: these are brute-force methods. They don’t really do anything clever, they just crawl through the whole maze until they stumble on a solution. Just trying out paths at random will usually solve a maze about as quickly as DFS or BFS (as long as you keep track of what you’ve already tried). Let’s be smarter. Heuristic Search A human solving a maze usually tries to work their way closer to the end. If we’re not sure which way to go, we take the direction which points more toward the goal. Formalizing this approach leads to heuristic search algorithms. The best-known of these is A, but we’ll use the more-human-intuitive “greedy best-first” search. Like breadth-first search, best-first explores multiple paths in parallel. But rather than brute-force searching all the paths, best-first focuses on paths which are closest to the goal “as the bird flies”. Distance from the goal serves as an heuristic to steer the search. Here’s an example where best-first works very well: By aggressively exploring the path closest to the goal, Pluto reaches his bone without having to explore most of the maze. But heuristic search comes with a catch: it’s only as good as the heuristic. If we use straight-line distance from the goal as an heuristic, then heuristic search is going to work well if-and-only-if the solution path is relatively straight. If the solution path wiggles around a lot, especially on a large scale, then heuristic search won’t help much. That’s what happens if we apply best-first to the Pinocchio maze: . the best-first solution in this case is almost identical to DFS. General Problem Solving In AI, path search is used to think about solving problems in general. It turns out all sorts of things can be cast as path search problems: puzzles, planning, and of course navigation. Basically any problem who...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Seeking Power is Often Convergently Instrumental in MDPs , published by TurnTrout, elriggs on the LessWrong. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. This is a linkpost for In 2008, Steve Omohundro's foundational paper The Basic AI Drives conjectured that superintelligent goal-directed AIs might be incentivized to gain significant amounts of power in order to better achieve their goals. Omohundro's conjecture bears out in toy models, and the supporting philosophical arguments are intuitive. In 2019, the conjecture was even debated by well-known AI researchers. Power-seeking behavior has been heuristically understood as an anticipated risk, but not as a formal phenomenon with a well-understood cause. The goal of this post (and the accompanying paper, Optimal Policies Tend to Seek Power) is to change that. Motivation It’s 2008, the ancient wild west of AI alignment. A few people have started thinking about questions like “if we gave an AI a utility function over world states, and it actually maximized that utility... what would it do?" In particular, you might notice that wildly different utility functions seem to encourage similar strategies. Resist shutdown? Gain computational resources? Prevent modification of utility function? Paperclip utility ✔️ ✔️ ✔️ Blue webcam pixel utility ✔️ ✔️ ✔️ People-look-happy utility ✔️ ✔️ ✔️ These strategies are unrelated to terminal preferences: the above utility functions do not award utility to e.g. resource gain in and of itself. Instead, these strategies are instrumental: they help the agent optimize its terminal utility. In particular, a wide range of utility functions incentivize these instrumental strategies. These strategies seem to be convergently instrumental. But why? I’m going to informally explain a formal theory which makes significant progress in answering this question. I don’t want this post to be Optimal Policies Tend to Seek Power with cuter illustrations, so please refer to the paper for the math. You can read the two concurrently. We can formalize questions like “do ‘most’ utility maximizers resist shutdown?” as “Given some prior beliefs about the agent’s utility function, knowledge of the environment, and the fact that the agent acts optimally, with what probability do we expect it to be optimal to avoid shutdown?” The table’s convergently instrumental strategies are about maintaining, gaining, and exercising power over the future, in some sense. Therefore, this post will help answer: What does it mean for an agent to “seek power”? In what situations should we expect seeking power to be more probable under optimality, than not seeking power? This post won’t tell you when you should seek power for your own goals; this post illustrates a regularity in optimal action across different goals one might pursue. Formalizing Convergent Instrumental Goals suggests that the vast majority of utility functions incentivize the agent to exert a lot of control over the future, assuming that these utility functions depend on “resources.” This is a big assumption: what are “resources”, and why must the AI’s utility function depend on them? We drop this assumption, assuming only unstructured reward functions over a finite Markov decision process (MDP), and show from first principles how power-seeking can often be optimal. Formalizing the Environment My theorems apply to finite MDPs; for the unfamiliar, I’ll illustrate with Pac-Man. Full observability: You can see everything that’s going on; this information is packaged in the state s. In Pac-Man, the state is the game screen. Markov transition function: the next state depends only on the choice of action a and the current state s. It doesn’t matter how we got into a situation. Discounted reward: future rewards get geometrically discoun...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Assessing Kurzweil predictions about 2019: the results, published by Stuart_Armstrong on the LessWrong. EDIT: Mean and standard deviation of individidual predictions can be found here. Thanks to all my brave assessors, I now have the data about Kurzweil's 1999 predictions about 2019. This was a follow up to a previous assessment about his predictions about 2009, which showed a mixed bag. Roughly evenly divided between right and wrong, which I found pretty good for ten-year predictions: So, did more time allow for trends to overcome noise or more ways to go wrong? Pause for a moment to calibrate your expectations. Methods and thanks So, for the 2019 predictions, I divided them into 105 separate statements, did a call for volunteers, with instructions here; the main relevant point being that I wanted their assessment for 2019, not for the (possibly transient) current situation. I got 46 volunteers with valid email addresses, of which 34 returned their predictions. So many thanks, in reverse alphabetical order, to Zvi Mowshowitz, Zhengdong Wang, Yann Riviere, Uriel Fiori, orthonormal, Nuño Sempere, Nathan Armishaw, Koen Holtman, Keller Scholl, Jaime Sevilla, Gareth McCaughan, Eli Rose and Dillon Plunkett, Daniel Kokotajlo, Anna Gardiner... and others who have chosen to remain anonymous. The results Enough background; what did the assessors find? Well, of the 34 assessors, 24 went the whole hog and did all 105 predictions; on average, 91 predictions were assessed by each person, a total of 3078 individual assessments[1]. So, did more time allow for more perspective or more ways to go wrong? Well, Kurzweil's predictions for 2019 were considerably worse than those for 2009, with more than half strongly wrong: Interesting details The (anonymised) data can be found here[2], and I encourage people to download and assess it themselves. But some interesting results stood out to me: Predictor agreement Taking a single prediction, for instance the first one: 1: Computers are now largely invisible. They are embedded everywhere--in walls, tables, chairs, desks, clothing, jewelry, and bodies. Then we can compute the standard deviation of the predictors' answer for that prediction. This gives an impression of how much disagreement there was between predictors; in this case, it was 0.84. Perfect agreement would be a standard deviation of 0; maximum disagreement (half find "1", half find "5") would be a standard deviation of 2. Perfect spread - equal numbers of 1s, 2s, 3s, 4s, and 5s - would have a standard deviation of √ 2 ≈ 1.4. Across the 105 predictions, the maximum standard deviation was 1.7, the minimum was 0 (perfect agreement), and the average was 0.97. So the predictors had a medium tendency to agree with each other. Most agreement/falsest predictions There was perfect agreement on five predictions; and on all of these, the agreed prediction was always "5": "False". These predictions were: 51: "Phone" calls routinely include high-resolution three-dimensional images projected through the direct-eye displays and auditory lenses. 55: [...] Thus a person can be fooled as to whether or not another person is physically present or is being projected through electronic communication. 59: The all-enveloping tactile environment is now widely available and fully convincing. 62: [...] These technologies are popular for medical examinations, as well as sensual and sexual interactions with other human partners or simulated partners. 63: [...] In fact, it is often the preferred mode of interaction, even when a human partner is nearby, due to its ability to enhance both experience and safety. As you can see, Kurzweil suffered a lot from his VR predictions. This seems a perennial thing: Hollywood is always convinced that mass 3D is just around the corner; technologists are convinced that VR is...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Where do (did?) stable, cooperative institutions come from? , published by AnnaSalamon on the LessWrong. The United States has a bunch of nice things whose creation/maintenance requires coordinated effort from a large number of people across time. For example: bridges that stay up; electrical grids that provide us with power; the rule of law; newspapers that make it easier to keep tabs on recent events; fire fighting services that stop most fires in urban areas; roads; many functioning academic fields; Google; Amazon; grocery stores; the postal service; and so on. The first question I'd like to pose is: how does this coordination work? What keeps these large sets of people pulling in a common direction (and wanting to pull in a common direction)? And what keeps that "common direction" grounded enough that an actual nice thing results from the pulling (e.g., what causes it to be that you get a working railway system, rather than a bunch of tracks that don't quite work? what causes you to sometimes get a functioning field of inquiry and not a cargo cult)? Is it that: Many people independently value the nice thing, and they altruistically decide to put their own efforts toward creating/maintaining the nice thing? (E.g., some large set of people wishes there were good fire-fighting institutions, and so each of them altruistically and independently decides to found a fire-fighting branch, to work at that branch, to tweak that branch's habits into a more effective configuration, etc.?) A small number of rich and powerful people (who are somehow also knowledgeable about institution design) value the nice thing, and they altruistically decide to set up incentives such that other people, purely via self-interest, will do the work that is needed to create/maintain the nice thing? (E.g., a small number of people altruistically donate to fire-fighting groups and set up incentives at those groups, and then other people do the fire-fighting work because they want a job?) Something else? One reason I’d like to pose this question is that it seems plausible to me that the magic that used to enable such cooperative institutions is fading. If so, it seems useful to know about that fading for quite a variety of reasons. My own lead candidate answer to "what is the magic that lets these cooperative institutions run?" is this: Somehow, people have sometimes known how to craft "institutional cultures" that aligned an individual's desire for (glory/$/prestige/etc.) with the actions that will allow the institution as a whole to acquire redistributable (glory, $, prestige, etc.) in the long run. More specifically, cooperative institutions arise in cases where some set of designers (either a few people, or a larger distributed set) magically manage several things at once: There is an institutional culture that is distinct from the formal workings of the institution, but that exists alongside it, helping to animate it. For example, alongside the formal workings of the old NYT (the printing presses, newspaper subscriptions, staff payroll, explicit assignments, etc.) there was an ethic of journalism that helped direct staff actions at many junctures (an ethic of e.g. "all the news that's fit to print," putting in shoe-leather, protecting one's sources, etc.). The installed "institutional culture" is pretty good at picking out actions that, if taken, will tend to cause the institution as a whole to gain redistributable (glory/$/prestige/etc.) in the long-term. In our example: The old NYT will in fact gain more long-run prestige, customers, incoming staff talent, etc. if it follows its journalistic ethics. In other words, the culture gestured at by ""all the news that's fit to print," putting in shoe-leather, protecting one's sources, etc." offered pretty good on-the-ground answers to the question ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Generalized Heat Engine, published by johnswentworth on the LessWrong. I’d like to be able to apply more of the tools of statistical mechanics and thermodynamics outside the context of physics. For some pieces, that’s pretty straightforward - a large chunk of statistical mechanics is just information theory, and that’s already a flourishing standalone field which formulates things in general ways. But for other pieces, it’s less obvious. What’s the analogue of a refrigerator or a carnot cycle in more general problems? How do “work” and “heat” generalize to problems outside physics? The principle of maximum entropy tells us how to generalize temperature, and offers one generalization of work and heat, but it’s not immediately obvious why we can’t extract “work” from “heat” without subsystems at different temperatures, or how to turn that into a useful idea in non-physics applications. This post documents my own exploration of these questions in the context of a relatively simple problem, with minimal reference to physics (other than by analogy). Specifically: we’ll talk about how to construct the analogue of a heat engine using biased coins. Intuition The main idea I want to generalize here is that we can “move uncertainty around” without reducing uncertainty. This is exactly what e.g. a refrigerator or heat engine does. Consider the viewpoint of a refrigerator-designer. All the microscopic dynamics of the (fridge + environment) system must be reversible, so the number of possible microscopic states will never decrease on its own as time passes. The only way to reduce uncertainty about the microscopic state is to observe it. But the fridge designer is designing the system, deciding in advance how it will behave. The designer has no direct access to the environment in which the fridge will run, no way to measure the exact positions the atoms will be in when the fridge first turns on. The designer, in short, cannot directly observe the system. So, from the designer’s perspective, there’s uncertainty which cannot be reduced. (In statistical mechanics, there are several entirely different justifications for why observations can’t reduce microscopic uncertainty/entropy - for instance, in one approach, macroscopic variables are chosen in such a way that we can deterministically predict future macroscopic observations. Another comes from Maxwell’s demon-style arguments, where the demon’s memory has to be included as part of the system. I’ll use the designer viewpoint, since it’s conceptually simple and easy to apply in other areas - in particular, we can easily apply it to the design of AIs embedded in their environment.) While we can’t reduce our total uncertainty, we can move it around. We design the machine to apply transformations to the system which leave us more certain about some subsystems (e.g. the inside of the refrigerator), but less certain about other subsystems (e.g. heat baths used to power the system). Setup We’ll imagine two large sets of IID biased coins. One is the “cold pool”, in which each coin comes up 1 (i.e. heads) with probability 0.1 and 0 with probability 0.9. The other is the “hot pool”, in which each coin comes up 1 with probability 0.2. We’ll call the coins in the cold pool X C 1 X C n , and the coins in the hot pool X H 1 X H n We’re going to apply transformations to these coins. Each transformation replaces some set of coins with new values which are a function of their old values. For instance, one transformation might be X C 1 X H 3 X H 7 ← X C 1 X H 3 X C 1 X H 7 ¯¯¯¯¯¯¯¯ X C 1 X H 7 X C 1 X H 3 ¯¯¯¯¯¯¯¯ X C 1 (Here the bar denotes logical not - i.e. ¯¯¯¯¯ X means "not X".) This transformation swaps X H 3 with X H 7 if X C 1 is 1, and leaves everything unchanged if X C 1 is 0. We’ll mostly be able to use any transformations we want, but wit...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: MIRI location optimization (and related topics) discussion, published by Rob Bensinger on the LessWrong. Added June 28: Blake Borgeson comments below, Update: After a lot more discussion and thinking things over, MIRI has decided against relocating away from the Bay Area, unless a new impetus comes along to move some time in the future. Mainly, ongoing/propagating strategic updates are behind this decision. We're more uncertain about our strategy overall following our 2020 strategy update (/#3), so we’d rather not pay a big up-front move cost based on a prediction about what we expect MIRI to look like 2 or 5 years from now. Related to the same strategic updates, over the next 12 months or so we plan to experiment with placing less of an emphasis on staff working from our Berkeley base, and expect some staff to live and work from other places. Separately, in the background, considerations against our recent favorite places became stronger: Bellingham is far away from big cities, and it seemed increasingly like MIRI would be more isolated there than we wanted. Also, we became more concerned about low light levels during winter months. The area around Peekskill has some of the highest Lyme disease incidence in the US, and it seems possible this is a high enough risk of disability to cause us to want to avoid it. We were still investigating this when we ended our location search. Who am I? I’m a MIRI board member, and was in recent months leading/coordinating this MIRI relocation decision process, which is now complete. Thanks for all the feedback in these comments, in emails, and on Facebook. I (and probably others) read all of it I could find, and it was important input for us. Original post: MIRI is moving (with high probability)! We haven’t finalized a location yet, but there’s a good chance we’ll make our decision in the next six weeks. I want to solicit: Feedback on our current top location candidates. Ideas for other places that might fit our criteria. I’m also interested in a more general location-optimizing discussion. What are your general thoughts on where you’d like to live, and have they changed any since the hub conversations Claire Wang began in September and November? If a new rationality community hub sprang up at any of these locations, would you be tempted to join? Is there a different place you’d prefer (either personally, or for the community)? Anything from 'statements of personal preferences' to 'models of how the rationality community might make humanity's future much more awesome' is welcome in the comments. What kind of place we're looking for Our priority is to find a place where we think researchers will be able to think unusually clearly and well, in line with our December update. Recently, we’ve been looking for a campus or proto-campus (one or more buildings, with space and legal ability to build more) that's: far enough away from urban areas to be maximally calm, quiet, and close to nature; and near enough to urban areas to ensure there are people to see and things to do in driving distance, and to provide access to city conveniences (food delivery, ridesharing, lots of restaurants, etc.). These proto-campus-type properties seem to be very rare and hard to find. E.g., we heard good things about Madison, WI and spent time looking for a property in the area, but ended up finding zero candidates currently on the market (that weren't falling apart, etc.). If you want to convince MIRI to move to your favorite city, one good route would be to find a property like this for sale and either email it to Alex Vermeer or PM me (please don’t post specific property options you’re recommending publicly). The best places we’ve found have often been at the outskirts of pleasant, walkable 50k-100k population cities or college towns. The specific factors we...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Death of Behavioral Economics , published by habryka on the LessWrong. This is a linkpost for Edit: I recommend reading Scott's response to this essay in addition to the essay itself. I've been tracking the replication crisis and how it affects a bunch of behavioral economics for a while. I found reading this post useful for a particularly negative take. Some key quotes: It sure does look alive... but it's a zombie—inside and out. Why do I say this? Two primary reasons: Core behavioral economics findings have been failing to replicate for several years, and the core finding of behavioral economics, loss aversion, is on ever more shaky ground. Its interventions are surprisingly weak in practice. Because of these two things, I don't think that behavioral economics will be a respected and widely used field 10-15 years from now. It turns out that loss aversion does exist, but only for large losses. This makes sense. We should be particularly wary of decisions that can wipe us out. That's not a so-called "cognitive bias". It's not irrational. In fact, it's completely sensical. If a decision can destroy you and/or your family, it's sane to be cautious. "So when did we discover that loss aversion exists only for large losses?" Well, actually, it looks like Kahneman and Tversky, winners of the Nobel Prize in Economics, knew about this unfortunate fact when they were developing Prospect Theory—their grand theory with loss aversion at its center. Unfortunately, the findings rebutting their view of loss aversion were carefully omitted from their papers, and other findings that went against their model were misrepresented so that they would instead support their pet theory. In short: any data that didn't fit Prospect Theory was dismissed or distorted. I don't know what you'd call this behavior... but it's not science. This shady behavior by the two titans of the field was brought to light in a paper published in 2018: "Acceptable Losses: The Debatable Origins of Loss Aversion". I encourage you to read the paper. It's shocking. This line from the abstract sums things up pretty well: "...the early studies of utility functions have shown that while very large losses are overweighted, smaller losses are often not. In addition, the findings of some of these studies have been systematically misrepresented to reflect loss aversion, though they did not find it." Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: All Possible Views About Humanity's Future Are Wild, published by HoldenKarnofsky on the LessWrong. This is the first post in the Most Important Century sequence. For more info and a roadmap for the series, see the sequence introduction. Audio also available by searching Stitcher, Spotify, Google Podcasts, etc. for "Cold Takes Audio" Summary In a series of posts starting with this one, I'm going to argue that the 21st century could see our civilization develop technologies allowing rapid expansion throughout our currently-empty galaxy. And thus, that this century could determine the entire future of the galaxy for tens of billions of years, or more. This view seems "wild": we should be doing a double take at any view that we live in such a special time. I illustrate this with a timeline of the galaxy. (On a personal level, this "wildness" is probably the single biggest reason I was skeptical for many years of the arguments presented in this series. Such claims about the significance of the times we live in seem "wild" enough to be suspicious.) But I don't think it's really possible to hold a non-"wild" view on this topic. I discuss alternatives to my view: a "conservative" view that thinks the technologies I'm describing are possible, but will take much longer than I think, and a "skeptical" view that thinks galaxy-scale expansion will never happen. Each of these views seems "wild" in its own way. Ultimately, as hinted at by the Fermi paradox, it seems that our species is simply in a wild situation. Before I continue, I should say that I don't think humanity (or some digital descendant of humanity) expanding throughout the galaxy would necessarily be a good thing - especially if this prevents other life forms from ever emerging. I think it's quite hard to have a confident view on whether this would be good or bad. I'd like to keep the focus on the idea that our situation is "wild." I am not advocating excitement or glee at the prospect of expanding throughout the galaxy. I am advocating seriousness about the enormous potential stakes. My view This is the first in a series of pieces about the hypothesis that we live in the most important century for humanity. In this series, I'm going to argue that there's a good chance of a productivity explosion by 2100, which could quickly lead to what one might call a "technologically mature"1 civilization. That would mean that: We'd be able to start sending spacecraft throughout the galaxy and beyond. These spacecraft could mine materials, build robots and computers, and construct very robust, long-lasting settlements on other planets, harnessing solar power from stars and supporting huge numbers of people (and/or our "digital descendants"). See Eternity in Six Hours for a fascinating and short, though technical, discussion of what this might require. I'll also argue in a future piece that there is a chance of "value lock-in" here: whoever is running the process of space expansion might be able to determine what sorts of people are in charge of the settlements and what sorts of societal values they have, in a way that is stable for many billions of years.2 If that ends up happening, you might think of the story of our galaxy3 like this. I've marked major milestones along the way from "no life" to "intelligent life that builds its own computers and travels through space." Thanks to Ludwig Schubert for the visualization. Many dates are highly approximate and/or judgment-prone and/or just pulled from Wikipedia (sources here), but plausible changes wouldn't change the big picture. The ~1.4 billion years to complete space expansion is based on the distance to the outer edge of the Milky Way, divided by the speed of a fast existing human-made spaceship (details in spreadsheet just linked); IMO this is likely to be a massive overestimat...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Honoring Petrov Day on LessWrong, in 2019, published by Ben Pace on the LessWrong. Just after midnight last night, 125 LessWrong users received the following email. Subject Line: Honoring Petrov Day: I am trusting you with the launch codes Dear {{username}}, Every Petrov Day, we practice not destroying the world. One particular way to do this is to practice the virtue of not taking unilateralist action. It’s difficult to know who can be trusted, but today I have selected a group of LessWrong users who I think I can rely on in this way. You’ve all been given the opportunity to show yourselves capable and trustworthy. This Petrov Day, between midnight and midnight PST, if you, {{username}}, enter the launch codes below on LessWrong, the Frontpage will go down for 24 hours. Personalised launch code: {{codes}} I hope to see you on the other side of this, with our honor intact. Yours, Ben Pace & the LessWrong 2.0 Team P.S. Here is the on-site announcement. Unilateralist Action As Nick Bostrom has observed, society is making it cheaper and easier for small groups to end the world. We’re lucky it requires major initiatives to build a nuclear bomb, and that the world can’t be destroyed by putting sand in a microwave. However, other dangerous technologies are becoming widely available, especially in the domain of artificial intelligence. Only 6 months after OpenAI created the state-of-the-art language-modelling GPT-2, others created similarly powerful versions and released them to the public. They disagreed about the dangers, and, because there was nothing stopping them, moved ahead. I don’t think this example is at all catastrophic, but I worry what this suggests about the future, when people will still have honest disagreements about the consequences of an action but where those consequences will be much worse. And honest disagreements will happen. In the 1940s, the great physicist Niels Bohr met President Roosevelt and Prime Minister Churchill, to persuade them to give the instructions for building the atomic bomb to Russia. He wanted to bring in a new world order and establish global peace, and thought this would be necessary - he believed strongly that it would prevent arms race dynamics, if only everyone just shared their science. (Churchill did not allow it.) Our newest technologies technologies do not yet have the bomb’s ability to transform the world in minutes, but I think it’s likely we’ll make powerful discoveries in the coming decades, and that publishing those discoveries will not require the permission of a president. And then it will only take one person to end the world. Even in a group of well-intentioned people, natural disagreements will mean someone will think that taking a damaging action is actually the correct choice — Nick Bostrom calls this the “unilateralist’s curse”. In a world where dangerous technology is widely available, the greatest risk is unilateralist action. Not Destroying the World Stanislav Petrov once chose not to destroy the world. As a Lieutenant Colonel of the Soviet Army, Petrov manned the system built to detect whether the US government had fired nuclear weapons on Russia. On September 26th, 1983, the system reported multiple such attacks. Petrov’s job was to report this as an attack to his superiors, who would launch a retaliative nuclear response. But instead, contrary to all the evidence the systems were giving him, he called it in as a false alarm. This later turned out to be correct. (For a more detailed story of how Stanislav Petrov saved the world, see the original LessWrong post by Eliezer, which started the tradition of Petrov Day.) During the Cold War, many other people had the ability to end the world - presidents, generals, commanders of nuclear subs from many countries, and so on. Fortunately, none of them did. As the ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Comment on "Endogenous Epistemic Factionalization", published by Zack_M_Davis on the LessWrong. In "Endogenous Epistemic Factionalization" (due in a forthcoming issue of the philosophy-of-science journal Synthese), James Owen Weatherall and Cailin O'Connor propose a possible answer to the question of why people form factions that disagree on multiple subjects. The existence of persistent disagreements is already kind of a puzzle from a Bayesian perspective. There's only one reality. If everyone is honestly trying to get the right answer and we can all talk to each other, then we should converge on the right answer (or an answer that is less wrong given the evidence we have). The fact that we can't do it is, or should be, an embarrassment to our species. And the existence of correlated persistent disagreements—when not only do I say "top" when you say "bottom" even after we've gone over all the arguments for whether it is in fact the case that top or bottom, but furthermore, the fact that I said "top" lets you predict that I'll probably say "cold" rather than "hot" even before we go over the arguments for that, is an atrocity. (Not hyperbole. Thousands of people are dying horrible suffocation deaths because we can't figure out the optimal response to a new kind of coronavirus.) Correlations between beliefs are often attributed to ideology or tribalism: if I believe that Markets Are the Answer, I'm likely to propose Market-based solutions to all sorts of seemingly-unrelated social problems, and if I'm loyal to the Green tribe, I'm likely to selectively censor my thoughts in order to fit the Green party line. But ideology can't explain correlated disagreements on unrelated topics that the content of the ideology is silent on, and tribalism can't explain correlated disagreements on narrow, technical topics that aren't tribal shibboleths. In this paper, Weatherall and O'Connor exhibit a toy model that proposes a simple mechanism that can explain correlated disagreement: if agents disbelieve in evidence presented by those with sufficiently dissimilar beliefs, factions emerge, even though everyone is honestly reporting their observations and updating on what they are told (to the extent that they believe it). The paper didn't seem to provide source code for the simulations it describes, so I followed along in Python. (Replication!) In each round of the model, our little Bayesian agents choose between repeatedly performing one of two actions, A or B, that can "succeed" or "fail." A is a fair coin: it succeeds exactly half the time. As far as our agents know, B is either slightly better or slightly worse: the per-action probability of success is either 0.5 + ɛ or 0.5 − ɛ, for some ɛ (a parameter to the simulation). But secretly, we the simulation authors know that B is better. import random ε = 0.01 def b(): return random.random() < 0.5 + ε The agents start out with a uniformly random probability that B is better. The ones who currently believe that A is better, repeatedly do A (and don't learn anything, because they already know that A is exactly a coinflip). The ones who currently believe that B is better, repeatedly do B, but keep track of and publish their results in order to help everyone figure out whether B is slightly better or slightly worse than a coinflip. class Agent: def experiment(self): results = [b() for _ in range(self.trial_count)] return results If H represents the hypothesis that B is better than A, and H − represents the hypothesis that B is worse, then Bayes's theorem says P H E P E H P H P E H P H P E H − P H − where E is the record of how many successes we got in how many times we tried action B. The likelihoods P E H and P E H − can be calculated from the probability mass function of the binomial distribution, so the agents have all the information th...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Solomonoff Prior is Malign , published by Mark Xu on the LessWrong. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. This argument came to my attention from this post by Paul Christiano. I also found this clarification helpful. I found these counter-arguments stimulating and have included some discussion of them. Very little of this content is original. My contributions consist of fleshing out arguments and constructing examples. Thank you to Beth Barnes and Thomas Kwa for helpful discussion and comments. What is the Solomonoff prior? The Solomonoff prior is intended to answer the question "what is the probability of X?" for any X, where X is a finite string over some finite alphabet. The Solomonoff prior is defined by taking the set of all Turing machines (TMs) which output strings when run with no input and weighting them proportional to 2 − K , where K is the description length of the TM (informally its size in bits). The Solomonoff prior says the probability of a string is the sum over all the weights of all TMs that print that string. One reason to care about the Solomonoff prior is that we can use it to do a form of idealized induction. If you have seen 0101 and want to predict the next bit, you can use the Solomonoff prior to get the probability of 01010 and 01011. Normalizing gives you the chances of seeing 1 versus 0, conditioned on seeing 0101. In general, any process that assigns probabilities to all strings in a consistent way can be used to do induction in this way. This post provides more information about Solomonoff Induction. Why is it malign? Imagine that you wrote a programming language called python^10 that works as follows: First, it takes all alpha-numeric chars that are not in literals and checks if they're repeated 10 times sequentially. If they're not, they get deleted. If they are, they get replaced by a single copy. Second, it runs this new program through a python interpreter. Hello world in python^10: ppppppppprrrrrrrrrriiiiiiiiiinnnnnnnnnntttttttttt('Hello, world!') Luckily, python has an exec function that executes literals as code. This lets us write a shorter hello world: eeeeeeeeexxxxxxxxxxeeeeeeeeeecccccccccc("print('Hello, world!')") It's probably easy to see that for nearly every program, the shortest way to write it in python^10 is to write it in python and run it with exec. If we didn't have exec, for sufficiently complicated programs, the shortest way to write them would be to specify an interpreter for a different language in python^10 and write it in that language instead. As this example shows, the answer to "what's the shortest program that does X?" might involve using some roundabout method (in this case we used exec). If python^10 has some security properties that python didn't have, then the shortest program in python^10 that accomplished any given task would not have these security properties because they would all pass through exec. In general, if you can access alternative ‘modes’ (in this case python), the shortest programs that output any given string might go through one of those modes, possibly introducing malign behavior. Let's say that I'm trying to predict what a human types next using the Solomonoff prior. Many programs predict the human: Simulate the human and their local surroundings. Run the simulation forward and check what gets typed. Simulate the entire Earth. Run the simulation forward and check what that particular human types. Simulate the entire universe from the beginning of time. Run the simulation forward and check what that particular human types. Simulate an entirely different universe that has reason to simulate this universe. Output what the human types in the simulation of our universe. Which one is the simplest? One property of the Solmonoff prior is ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 2020 AI Alignment Literature Review and Charity Comparison , published by Larks on the LessWrong. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. cross-posted to the EA forum here. Introduction As in 2016, 2017, 2018, and 2019, I have attempted to review the research that has been produced by various organisations working on AI safety, to help potential donors gain a better understanding of the landscape. This is a similar role to that which GiveWell performs for global health charities, and somewhat similar to a securities analyst with regards to possible investments. My aim is basically to judge the output of each organisation in 2020 and compare it to their budget. This should give a sense of the organisations' average cost-effectiveness. We can also compare their financial reserves to their 2020 budgets to get a sense of urgency. I’d like to apologize in advance to everyone doing useful AI Safety work whose contributions I have overlooked or misconstrued. As ever I am painfully aware of the various corners I have had to cut due to time constraints from my job, as well as being distracted by 1) other projects, 2) the miracle of life and 3) computer games. This article focuses on AI risk work. If you think other causes are important too, your priorities might differ. This particularly affects GCRI, FHI and CSER, who both do a lot of work on other issues which I attempt to cover but only very cursorily. How to read this document This document is fairly extensive, and some parts (particularly the methodology section) are largely the same as last year, so I don’t recommend reading from start to finish. Instead, I recommend navigating to the sections of most interest to you. If you are interested in a specific research organisation, you can use the table of contents to navigate to the appropriate section. You might then also want to Ctrl+F for the organisation acronym in case they are mentioned elsewhere as well. Papers listed as ‘X researchers contributed to the following research lead by other organisations’ are included in the section corresponding to their first author and you can Cntrl+F to find them. If you are interested in a specific topic, I have added a tag to each paper, so you can Ctrl+F for a tag to find associated work. The tags were chosen somewhat informally so you might want to search more than one, especially as a piece might seem to fit in multiple categories. Here are the un-scientifically-chosen hashtags: AgentFoundations Amplification Capabilities Corrigibility DecisionTheory Ethics Forecasting GPT-3 IRL Misc NearAI OtherXrisk Overview Politics RL Strategy Textbook Transparency ValueLearning New to Artificial Intelligence as an existential risk? If you are new to the idea of General Artificial Intelligence as presenting a major risk to the survival of human value, I recommend this Vox piece by Kelsey Piper, or for a more technical version this by Richard Ngo. If you are already convinced and are interested in contributing technically, I recommend this piece by Jacob Steinheart, as unlike this document Jacob covers pre-2019 research and organises by topic, not organisation, or this from Critch & Krueger, or this from Everitt et al, though it is a few years old now Research Organisations FHI: The Future of Humanity Institute FHI is an Oxford-based Existential Risk Research organisation founded in 2005 by Nick Bostrom. They are affiliated with Oxford University. They cover a wide variety of existential risks, including artificial intelligence, and do political outreach. Their research can be found here. Their research is more varied than MIRI's, including strategic work, work directly addressing the value-learning problem, and corrigibility work - as well as work on other Xrisks. They run a Research Scholars Pro...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Covid 1/21: Turning the Corner , published by Zvi on the LessWrong. Aside from worries over the new strains, I would be saying this was an exceptionally good week. Both deaths and positive test percentages took a dramatic turn downwards, and likely will continue that trend for at least several weeks. Things are still quite short-term bad in many places, but things are starting to improve. Even hospitalizations are slightly down. It is noticeably safer out there than it was a few weeks ago, and a few weeks from now will be noticeably safer than it is today. Studies came out that confirmed that being previously infected conveys strong immunity for as long as we have been able to measure it. As usual, the findings were misrepresented, but the news is good. I put my analysis here in a distinct post, so it can be linked to on its own. We had a peaceful transition of power, which is always a historic miracle to be celebrated. Vaccination rollout is still a disaster compared to what we would prefer, with new disasters on the horizon (with several sections devoted to all that), but we are getting increasing numbers of shots into increasing numbers of arms, and that is what matters most. In many places we have made the pivot from ‘plenty of vaccine and not enough arms to put shots into’ to the better problem of ‘plenty of arms to put vaccine into, but not enough shots.’ Then all we have to do is minimize how many shots go in the trash, including the extra shots at the bottom of the vial, and do everything we can to ramp up manufacturing capacity. Which it seems can still be meaningfully done. The problem is that the new strains are coming. The English strain will arrive first, within a few months. That’s definitely happening, and the only question is how bad it’s going to get before we can turn the tide. We are in a race against time. The South African and Brazillian strains are not coming as fast, but are potentially even scarier. There are signs of potential escape from not only vaccination but previous infection, potentially allowing reinfection to take place. See the section on them for details, and if you can help provide better information, please do so. We need clarity on this, and we need it badly. There are also all the other new strains being talked about, which are probably nothing, but there’s always the chance that’s not true. But first, the good news, and it is very, very good. Let’s run the numbers. The Numbers Predictions Prediction last week: 14.0% positive rate on 11.7 million tests, and an average of 3,650 deaths. Results: 11.9% positive rate on 11.3 million tests, and an average of 3,043 deaths. Both numbers are hugely pleasant surprises, and this is the biggest directional miss I’ve had on deaths. Last week we were at 3,335 deaths per day, and I figured things would keep getting worse for another week or two. Instead, things are already on their way to rapid improvement, unless there were massive shifts in when deaths were reported that made last week look worse than it was. For infections, I did predict a drop (last week was 15.2%) and we got a much more dramatic drop than I expected. This was wonderful news, and it seems like this should continue. The caveat is that Tuesday and Wednesday of this week both look suspiciously good on both stats, such that I suspect missing data. I don’t know if somehow Martin Luther King Day actually mattered to reporting, or the inauguration and fears of disruptions around it were distracting, or what, but we should worry that this is getting a bit ahead of ourselves, even though test counts would indicate otherwise. Test count predictions don’t seem worth doing, so going to stop doing those. Prediction: 10.5% positive rate and 2,900 deaths per day. I’m being conservative because I worry about the drops from this week bei...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Simulacrum 3 As Stag-Hunt Strategy, published by johnswentworth on the LessWrong. Reminder of the rules of Stag Hunt: Each player chooses to hunt either Rabbit or Stag Players who choose Rabbit receive a small reward regardless of what everyone else chooses Players who choose Stag receive a large reward if-and-only-if everyone else chooses Stag. If even a single player chooses Rabbit, then all the Stag-hunters receive zero reward. From the outside, the obvious choice is for everyone to hunt Stag. But in real-world situations, there’s lots of noise and uncertainty, and not everyone sees the game the same way, so the Schelling choice is Rabbit. How does one make a Stag hunt happen, rather than a Rabbit hunt, even though the Schelling choice is Rabbit? If one were utterly unscrupulous, one strategy would be to try to trick everyone into thinking that Stag is the obvious right choice, regardless of what everyone else is doing. Now, tricking people is usually a risky strategy at best - it’s not something we can expect to work reliably, especially if we need to trick everyone. But this is an unusual case: we’re tricking people in a way which (we expect) will benefit them. Therefore, they have an incentive to play along. So: we make our case for Stag, try to convince people it’s the obviously-correct choice no matter what. And. they’re not fooled. But they all pretend to be fooled. And they all look around at each other, see everyone else also pretending to be fooled, and deduce that everyone else will therefore choose Stag. And if everyone else is choosing Stag. well then, Stag actually is the obvious choice. Just like that, Stag becomes the new Schelling point. We can even take it a step further. If nobody actually needs to be convinced that Stag is the best choice regardless, then we don’t actually need to try to trick them. We can just pretend to try to trick them. Pretend to pretend that Stag is the best choice regardless. That will give everyone else the opportunity to pretend to be fooled by this utterly transparent ploy, and once again we’re off to hunt Stag. This is simulacrum 3: we’re not telling the truth about reality (simulacrum 1), or pretending that reality is some other way in order to manipulate people (simulacrum 2). We’re pretending to pretend that reality is some other way, so that everyone else can play along. In The Wild We have a model for how-to-win-at-Stag-Hunt. If it actually works, we’d expect to find it in the wild in places where economic selection pressure favors groups which can hunt Stag. More precisely: we want to look for places where the payout increases faster-than-linearly with the number of people buying in. Economics jargon: we’re looking for increasing marginal returns. Telecoms, for instance, are a textbook example. One telecom network connecting fifty cities is far more valuable than fifty networks which each only work within one city. In terms of marginal returns: the fifty-first city connected to a network contributes more value than the first, since anyone in the first fifty cities can reach a person in the fifty-first. The bigger the network, the more valuable it is to expand it. From an investor’s standpoint, this means that a telecom investment is likely to have better returns if more people invest in it. It’s like a Stag Hunt for investors: each investor wants to invest if-and-only-if enough other investors also invest. (Though note that it’s more robust than a true Stag Hunt - we don’t need literally every investor to invest in order to get a big payoff.) Which brings us to this graph, from T-mobile’s 2016 annual report (second page): Fun fact: that is not a graph of those numbers. Some clever person took the numbers, and stuck them as labels on a completely unrelated graph. Those numbers are actually near-perfectly linear, ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Another RadVac Testing Update , published by johnswentworth on the LessWrong. Previously: Making Vaccine, Commercial Antibody Test Results, Mini-Update I've now run 9 ELISA tests. The main result is noise: negative controls are all over the map, sometimes very blue (i.e. positive), sometimes not blue at all. I did see more positive results in the experimental group than I'd expect from noise alone, but I haven't gotten the noise to a point where results are consistently reproducible. Meanwhile, I also ran one very simple test: I snorted a batch of the peptides, without the chitosan or anything else - just peptides in deionized water. Previously, on doses 3-6 of the vaccine, I had consistently been congested for a couple days after (and not congested the rest of the week), which strongly indicates an immune response. However, that response could have been to the chitosan or other contents of the vaccine, rather than the peptides. This test put that possibility to rest: after snorting just the peptides, I was very obviously congested for a couple days, in basically the same way as after the vaccine doses. So thanks to that simple test, I personally am now pretty highly confident that I have an immune response to these peptides. Unfortunately it's not as legible as an ELISA test, so you should not necessarily be quite as convinced by this. Now, this still leaves the question of whether an immune response to these peptides translates into an immune response to COVID. It could be that e.g. the conformation of the peptides' corresponding sequence within full COVID proteins is different enough that it doesn't carry over. Personally, though, I consider this a much less likely failure mode, for two reasons. First, the white paper indicates that the peptides were chosen based on antibodies developed by people who actually had COVID. Second, whether antibodies against these peptides bind the real proteins is something which I would not expect to vary much from person to person, so if it's worked for a few people it should work for everyone - and the whitepaper does indicate that multiple groups have seen positive results testing for binding against the full proteins. None of this puts my confidence up close to 99%, but I'm now considerably more confident that the vaccine worked (~90%). Also, that confidence is distributed over fewer possible worlds - e.g. based on the info in the latest version of the RadVac whitepaper, I now very much doubt that the vaccine will induce a response in the blood (unless it's injected). I also now have very little weight on the possibility that it works for some people sometimes but didn't work for me specifically, so additional dakka is not needed (at least for me). The next section will be a bit more detail on the ELISA tests, for people who are curious about exactly how that sausage was made. ELISA Tests This section is an abbreviated chronology of what-I-saw and my reasoning about it; it's intended to show how I came to the conclusions I did. I expect most people will not find it very interesting, but one of the benefits of blog posts is that I can show all the questionable decisions and opportunities for confirmation bias to sneak in, so that's what I'm doing. First, some background on how these tests work in theory. We start with a "high binding plate" - basically some plastic treated so that proteins/peptides stick to it. That’s the plate; each of the holes is called a “well”, and is basically a mini-test-tube with a high-binding surface. We add a solution of our peptides, and some of them stick to the surface. Next, we dump that solution out, leaving behind only the peptides which bound to the surface. We add some "binding solution" - in this case nonfat dry milk with a little detergent in it. The proteins in the binding solution fill what...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Seven Shiny Stories, published by Alicorn on the LessWrong. It has come to my attention that the contents of the luminosity sequence were too abstract, to the point where explicitly fictional stories illustrating the use of the concepts would be helpful. Accordingly, there follow some such stories. 1. Words (an idea from Let There Be Light, in which I advise harvesting priors about yourself from outside feedback) Maria likes compliments. She loves compliments. And when she doesn't get enough of them to suit her, she starts fishing, asking plaintive questions, making doe eyes to draw them out. It's starting to annoy people. Lately, instead of compliments, she's getting barbs and criticism and snappish remarks. It hurts - and it seems to hurt her more than it hurts others when they hear similar things. Maria wants to know what it is about her that would explain all of this. So she starts taking personality tests and looking for different styles of maintaining and thinking about relationships, looking for something that describes her. Eventually, she runs into a concept called "love languages" and realizes at once that she's a "words" person. Her friends aren't trying to hurt her - they don't realize how much she thrives on compliments, or how deeply insults can cut when they're dealing with someone who transmits affection verbally. Armed with this concept, she has a lens through which to interpret patterns of her own behavior; she also has a way to explain herself to her loved ones and get the wordy boosts she needs. 2. Widgets (an idea from The ABC's of Luminosity, in which I explain the value of correlating affect, behavior, and circumstance) Tony's performance at work is suffering. Not every day, but most days, he's too drained and distracted to perform the tasks that go into making widgets. He's in serious danger of falling behind his widget quota and needs to figure out why. Having just read a fascinating and brilliantly written post on Less Wrong about luminosity, he decides to keep track of where he is and what he's doing when he does and doesn't feel the drainedness. After a week, he's got a fairly robust correlation: he feels worst on days when he doesn't eat breakfast, which reliably occurs when he's stayed up too late, hit the snooze button four times, and had to dash out the door. Awkwardly enough, having been distracted all day tends to make him work more slowly at making widgets, which makes him less physically exhausted by the time he gets home and enables him to stay up later. To deal with that, he starts going for long runs on days when his work hasn't been very tiring, and pops melatonin; he easily drops off to sleep when his head hits the pillow at a reasonable hour, gets sounder sleep, scarfs down a bowl of Cheerios, and arrives at the widget factory energized and focused. 3. Text (an idea from Lights, Camera, Action!, in which I advocate aggressive and frequent introspection to collect as much data as possible) Dot reads about an experiment in which the subjects receive phone calls at random times and must tell researchers how happy they feel. Apparently the experiment turned up some really suboptimal patterns of behavior, and Dot's curious about what she'd learn that she could use to improve her life. She gets a friend to arrange delayed text messages to be sent to her phone at intervals supplied by a random number generator, and promises herself that she'll note what she's doing, thinking, and feeling at the moment she receives the text. She soon finds that she doesn't enjoy watching TV as much as she thinks she does; that it's probably worth the time to cook dinner rather than heating up something in the microwave because it's considerably tastier; that she can't really stand her cubicle neighbor; and that she thinks about her ex more than she'd...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Optimizing Fuzzies And Utilons: The Altruism Chip Jar, published by orthonormal on the LessWrong. Related: Purchase Fuzzies and Utilons Separately We genuinely want to do good in the world; but also, we want to feel as if we're doing good, via heuristics that have been hammered into our brains over the course of our social evolution. The interaction between these impulses (in areas like scope insensitivity, refusal to quantify sacred values, etc.) can lead to massive diminution of charitable impact, and can also suck the fun out of the whole process. Even if it's much better to write a big check at the end of the year to the charity with the greatest expected impact than it is to take off work every Thursday afternoon and volunteer at the pet pound, it sure doesn't feel as rewarding. And of course, we're very good at finding excuses to stop doing costly things that don't feel rewarding, or at least to put them off. But if there's one thing I've learned here, it's that lamenting our irrationality should wait until one's properly searched for a good hack. And I think I've found one. Not just that, but I've tested it out for you already. This summer, I had just gone through the usual experience of being asked for money for a nice but inefficient cause, turning them down, and feeling a bit bad about it. I made a mental note to donate some money to a more efficient cause, but worried that I'd forget about it; it's too much work to make a bunch of small donations over the year (plus, if done by credit card, the fees take a bigger cut that way) and there's no way I'd remember that day at the end of the year. Unless, that is, I found some way to keep track of it. So I made up several jars with the names of charities I found efficient (SIAI and VillageReach) and kept a bunch of poker chips near them. Starting then, whenever I felt like doing a good deed (and especially if I'd passed up an opportunity to do a less efficient one), I'd take a chip of an appropriate value and toss it in the jar of my choice. I have to say, this gave me much more in the way of warm fuzzies than if I'd just waited and made up a number at the end of the year. And now I've added up and made my contributions: $1,370 to SIAI and $566 to VillageReach. A couple of notes: I do think it was a good idea in practice to diversify my portfolio (despite the usual admonitions to the contrary) because it appeared to increase my charity budget rather than divert a fixed one. Some days I just didn't feel as optimistic about the SIAI, and on those days I could still chip in to save lives in the Third World. As long as my different jars seem to be interfering constructively rather than destructively, I'll keep them. In terms of warm fuzzies, I really enjoy that this system makes giving more tangible than writing a check or filling out an online form. It even helps that I have the weighted clay chips- tossing those into a jar feels as if I'm actually doing something. I do worry about doing my good deed for the day and having negative externalities flow from that, so I do my donating at the end of the day to minimize the effect. I could easily afford to give more than this, actually (though I can't tell whether I would have– it's more than I donated to charity in any previous year, although I was a poor grad student until this fall); I'm going to see if that knowledge makes me increase my pace of giving next year. (UPDATE 8/19/14: In retrospect, it was much more important for my less wealthy past self to create a habit than for him to donate a significant fraction of his income. My contributions to the chip jar since then have scaled appropriately to my circumstances.) Let me know if you start trying this out, or if you have any suggested improvements on it. In any case, may your altruism be effective and full of fuzzi...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The True Prisoner's Dilemma, published by Eliezer Yudkowsky on the AI Alignment Forum. It occurred to me one day that the standard visualization of the Prisoner's Dilemma is fake. The core of the Prisoner's Dilemma is this symmetric payoff matrix: 1: C 1: D 2: C (3, 3) (5, 0) 2: D (0, 5) (2, 2) Player 1, and Player 2, can each choose C or D. 1 and 2's utility for the final outcome is given by the first and second number in the pair. For reasons that will become apparent, "C" stands for "cooperate" and D stands for "defect". Observe that a player in this game (regarding themselves as the first player) has this preference ordering over outcomes: (D, C) > (C, C) > (D, D) > (C, D). D, it would seem, dominates C: If the other player chooses C, you prefer (D, C) to (C, C); and if the other player chooses D, you prefer (D, D) to (C, D). So you wisely choose D, and as the payoff table is symmetric, the other player likewise chooses D. If only you'd both been less wise! You both prefer (C, C) to (D, D). That is, you both prefer mutual cooperation to mutual defection. The Prisoner's Dilemma is one of the great foundational issues in decision theory, and enormous volumes of material have been written about it. Which makes it an audacious assertion of mine, that the usual way of visualizing the Prisoner's Dilemma has a severe flaw, at least if you happen to be human. The classic visualization of the Prisoner's Dilemma is as follows: you are a criminal, and you and your confederate in crime have both been captured by the authorities. Independently, without communicating, and without being able to change your mind afterward, you have to decide whether to give testimony against your confederate (D) or remain silent (C). Both of you, right now, are facing one-year prison sentences; testifying (D) takes one year off your prison sentence, and adds two years to your confederate's sentence. Or maybe you and some stranger are, only once, and without knowing the other player's history, or finding out who the player was afterward, deciding whether to play C or D, for a payoff in dollars matching the standard chart. And, oh yes - in the classic visualization you're supposed to pretend that you're entirely selfish, that you don't care about your confederate criminal, or the player in the other room. It's this last specification that makes the classic visualization, in my view, fake. You can't avoid hindsight bias by instructing a jury to pretend not to know the real outcome of a set of events. And without a complicated effort backed up by considerable knowledge, a neurologically intact human being cannot pretend to be genuinely, truly selfish. We're born with a sense of fairness, honor, empathy, sympathy, and even altruism - the result of our ancestors adapting to play the iterated Prisoner's Dilemma. We don't really, truly, absolutely and entirely prefer (D, C) to (C, C), though we may entirely prefer (C, C) to (D, D) and (D, D) to (C, D). The thought of our confederate spending three years in prison, does not entirely fail to move us. In that locked cell where we play a simple game under the supervision of economic psychologists, we are not entirely and absolutely unsympathetic for the stranger who might cooperate. We aren't entirely happy to think what we might defect and the stranger cooperate, getting five dollars while the stranger gets nothing. We fixate instinctively on the (C, C) outcome and search for ways to argue that it should be the mutual decision: "How can we ensure mutual cooperation?" is the instinctive thought. Not "How can I trick the other player into playing C while I play D for the maximum payoff?" For someone with an impulse toward altruism, or honor, or fairness, the Prisoner's Dilemma doesn't really have the critical payoff matrix - whatever the financial payoff to ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Bayes for Schizophrenics: Reasoning in Delusional Disorders , published by Scott Alexander on the LessWrong. Related to: The Apologist and the Revolutionary, Dreams with Damaged Priors Several years ago, I posted about V.S. Ramachandran's 1996 theory explaining anosognosia through an "apologist" and a "revolutionary". Anosognosia, a condition in which extremely sick patients mysteriously deny their sickness, occurs during right-sided brain injury but not left-sided brain injury. It can be extraordinarily strange: for example, in one case, a woman whose left arm was paralyzed insisted she could move her left arm just fine, and when her doctor pointed out her immobile arm, she claimed that was her daughter's arm even though it was obviously attached to her own shoulder. Anosognosia can be temporarily alleviated by squirting cold water into the patient's left ear canal, after which the patient suddenly realizes her condition but later loses awareness again and reverts back to the bizarre excuses and confabulations. Ramachandran suggested that the left brain is an "apologist", trying to justify existing theories, and the right brain is a "revolutionary" which changes existing theories when conditions warrant. If the right brain is damaged, patients are unable to change their beliefs; so when a patient's arm works fine until a right-brain stroke, the patient cannot discard the hypothesis that their arm is functional, and can only use the left brain to try to fit the facts to their belief. In the almost twenty years since Ramachandran's theory was published, new research has kept some of the general outline while changing many of the specifics in the hopes of explaining a wider range of delusions in neurological and psychiatric patients. The newer model acknowledges the left-brain/right-brain divide, but adds some new twists based on the Mind Projection Fallacy and the brain as a Bayesian reasoner. INTRODUCTION TO DELUSIONS Strange as anosognosia is, it's only one of several types of delusions, which are broadly categorized into polythematic and monothematic. Patients with polythematic delusions have multiple unconnected odd ideas: for example, the famous schizophrenic game theorist John Nash believed that he was defending the Earth from alien attack, that he was the Emperor of Antarctica, and that he was the left foot of God. A patient with a monothematic delusion, on the other hand, usually only has one odd idea. Monothematic delusions vary less than polythematic ones: there are a few that are relatively common across multiple patients. For example: In the Capgras delusion, the patient, usually a victim of brain injury but sometimes a schizophrenic, believes that one or more people close to her has been replaced by an identical imposter. For example, one male patient expressed the worry that his wife was actually someone else, who had somehow contrived to exactly copy his wife's appearance and mannerisms. This delusion sounds harmlessly hilarious, but it can get very ugly: in at least one case, a patient got so upset with the deceit that he murdered the hypothesized imposter - actually his wife. The Fregoli delusion is the opposite: here the patient thinks that random strangers she meets are actually her friends and family members in disguise. Sometimes everyone may be the same person, who must be as masterful at quickly changing costumes as the famous Italian actor Fregoli (inspiring the condition's name). In the Cotard delusion, the patient believes she is dead. Cotard patients will neglect personal hygiene, social relationships, and planning for the future - as the dead have no need to worry about such things. Occasionally they will be able to describe in detail the "decomposition" they believe they are undergoing. Patients with all these types of delusions1 - as well...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: by Eliezer Yudkowsky, published by Eliezer Yudkowsky on the LessWrong. Follow-up to: An Equilibrium of No Free Energy There’s a toolbox of reusable concepts for analyzing systems I would call “inadequate”—the causes of civilizational failure, some of which correspond to local opportunities to do better yourself. I shall, somewhat arbitrarily, sort these concepts into three larger categories: 1. Decisionmakers who are not beneficiaries; 2. Asymmetric information; and above all, 3. Nash equilibria that aren’t even the best Nash equilibrium, let alone Pareto-optimal. In other words: 1. Cases where the decision lies in the hands of people who would gain little personally, or lose out personally, if they did what was necessary to help someone else; 2. Cases where decision-makers can’t reliably learn the information they need to make decisions, even though someone else has that information; and 3. Systems that are broken in multiple places so that no one actor can make them better, even though, in principle, some magically coordinated action could move to a new stable state. I will then play fast and loose with these concepts in order to fit the entire Taxonomy of Failure inside them. For example, “irrationality in the form of cognitive biases” wouldn’t obviously fit into any of these categories, but I’m going to shove it inside “asymmetric information” via a clever sleight-of-hand. Ready? Here goes: If nobody can detect a cognitive bias in particular cases, then from our perspective we can’t really call it a “civilizational inadequacy” or “failure to pluck a low-hanging fruit.” We shouldn’t even be able to see it ourselves. So, on the contrary, let’s suppose that you and some other people can indeed detect a cognitive bias that’s screwing up civilizational decisionmaking. Then why don’t you just walk up to the decision-maker and tell them about the bias? Because they wouldn’t have any way of knowing to trust you rather than the other five hundred people trying to influence their decisions? Well, in that case, you’re holding information that they can’t learn from you! So that’s an “asymmetric information problem,” in much the same way that it’s an asymmetric information problem when you’re trying to sell a used car and you know it doesn’t have any mechanical problems, but you have no way of reliably conveying this knowledge to the buyer because for all they know you could be lying. That argument is a bit silly, but so is the notion of trying to fit the whole Scroll of Woe into three supercategories. And if I named more than three supercategories, you wouldn’t be able to remember them due to computational limitations (which aren’t on the list anywhere, and I’m not going to add them). i. For want of docosahexaenoic acids, a baby was lost My discussion of modest epistemology in Chapter 1 might have given the impression that I think of modesty mostly as a certain set of high-level beliefs: beliefs about how best to combat cognitive bias, about how individual competencies stack up against group-level competencies, and so on. But I predict that many of this book’s readers have high-level beliefs similar to those I outlined in Chapter 2, while employing a reasoning style that is really a special case of modest epistemology; and I think that this reasoning style is causing them substantial harm. As reasoning styles, modest epistemology and inadequacy analysis depend on a mix of explicit principles and implicit mental habits. In inadequacy analysis, it’s one thing to recognize in the abstract that we live in a world rife with systemic inefficiencies, and quite another to naturally perceive systems that way in daily life. So my goal here won't be to unkindly stick the label “inadequate” to a black box containing the world; it will be to say something about how the relevant systems a...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Prediction Markets: When Do They Work?, published by Zvi on the AI Alignment Forum. Epistemic Status: Resident Expert I’m a little late on this, which was an old promise to Robin Hanson (not that he asked for it). I was motivated to deal with this again by the launch of Augur (REP), the crypto prediction market token. And by the crypto prediction market token, I mean the empty shell of a potential future prediction market token; what they have now is pretty terrible but in crypto world that is occasionally good for a $300 million market cap. This is, for now, one of those occasions. The biggest market there, by far, is on whether Ether will trade above $500 at the end of the year. This is an interesting market because Augur bets are made in Ether. So even though the market (as of last time I checked) says it’s 74% percent to be trading above $500 and it’s currently $480 (it’s currently Thursday on July 26, and I’m not going to go back and keep updating these numbers). When I first saw this the market was at 63%, which seemed to me like a complete steal. Now it’s at 74%, which seems more reasonable, which means the first ‘official DWATV trading tip’ will have to wait. A shame! A better way to ask this question, given how close the price is to $500 now, is what the ratio of ‘given Ether is above $500 what does it cost’ to ‘given Ether is below $500 what does it cost’ should be. A three to one ratio seems plausible? The weakness (or twist) on markets this implies applies to prediction markets generally. If you bet on an event that is correlated with the currency you’re betting in, the fair price can be very different from the true probability. It doesn’t have to be price based – think about betting on an election between a hard money candidate and one who will print money, or a prediction on a nuclear war. If I bet on a nuclear war, and win, how exactly am I getting paid? Robin Hanson, Eliezer Yudkowsky and Scott Sumner are big advocates of prediction markets. In theory, so am I. Prediction markets are a wonderful thing. By giving people a monetary incentive to solve problems and share information, we can learn probabilities (what will GDP be next year?) and conditional probabilities (what will GDP be next year if we pass this tax cut bill?) and use the answers to make the best decision. This method of making decisions is called futarchy. Formally, a prediction market allows participants to buy and sell contracts. Those contracts then pay out a variable amount of money. Typically this is either binary (will Donald Trump be elected president?), paying out 100 if the event happens and 0 if it doesn’t, or they are continuous (how many electoral college votes will Donald Trump get?) and pay proportionally to the answer. Sometimes there are special cases where the market is void and all transactions are undone, at other times strange cases have special logic to determine the payout level. There are three types of prediction markets that have gotten non-zero traction. The first is politics. There are markets at PredictIt and BetFair and Pinnacle Sports, and there used to be relatively deep markets at InTrade. These markets matter enough to get talked about and attract some money when they involve major events like presidential elections, but tend to be quite pathetic for anything less than that. The second is economics. There are lots of stocks and futures and options and other such products available for purchase. Futures markets in particular are prediction markets. They don’t call themselves prediction markets, but that is one of the things they are, and the information they reveal is invaluable. It’s even sometimes used to make decisions. The third is sports. Most televised sporting events have bookmakers offering odds and taking bets. They use their own terminology for m...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: You Have About Five Words, published by Raemon on the AI Alignment Forum. Cross posted from the EA Forum. Epistemic Status: all numbers are made up and/or sketchily sourced. Post errs on the side of simplistic poetry – take seriously but not literally. If you want to coordinate with one person on a thing about something nuanced, you can spend as much time as you want talking to them – answering questions in realtime, addressing confusions as you notice them. You can trust them to go off and attempt complex tasks without as much oversight, and you can decide to change your collective plans quickly and nimbly. You probably speak at around 100 words per minute. That's 6,000 words per hour. If you talk for 3 hours a day, every workday for a year, you can communicate 4.3 million words worth of nuance. You can have a real conversation with up to 4 people. (Last year the small organization I work at considered hiring a 5th person. It turned out to be very costly and we decided to wait, and I think the reasons were related to this phenomenon) If you want to coordinate on something nuanced with, say, 10 people, you realistically can ask them to read a couple books worth of words. A book is maybe 50,000 words, so you have maybe 200,000 words worth of nuance. Alternately, you can monologue at people, scaling a conversation past the point where people realistically can ask questions. Either way, you need to hope that your books or your monologues happen to address the particular confusions your 10 teammates have. If you want to coordinate with 100 people, you can ask them to read a few books, but chances are they won't. They might all read a few books worth of stuff, but they won't all have read the same books. The information that they can be coordinated around is more like "several blogposts." If you're trying to coordinate nerds, maybe those blogposts add up to one book because nerds like to read. If you want to coordinate 1,000 people... you realistically get one blogpost, or maybe one blogpost worth of jargon that's hopefully self-explanatory enough to be useful. If you want to coordinate thousands of people... You have about five words. This has ramifications on how complicated a coordinated effort you can attempt. What if you need all that nuance and to coordinate thousands of people? What would it look like if the world was filled with complicated problems that required lots of people to solve? I guess it'd look like this one. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 2018 Review: Voting Results! , published by Ben Pace on the AI Alignment Forum. The votes are in! 59 of the 430 eligible voters participated, evaluating 75 posts. Meanwhile, 39 users submitted a total of 120 reviews, with most posts getting at least one review. Thanks a ton to everyone who put in time to think about the posts - nominators, reviewers and voters alike. Several reviews substantially changed my mind about many topics and ideas, and I was quite grateful for the authors participating in the process. I'll mention Zack_M_Davis, Vanessa Kosoy, and Daniel Filan as great people who wrote the most upvoted reviews. In the coming months, the LessWrong team will write further analyses of the vote data, and use the information to form a sequence and a book of the best writing on LessWrong from 2018. Below are the results of the vote, followed by a discussion of how reliable the result is and plans for the future. Top 15 posts Embedded Agents by Abram Demski and Scott Garrabrant The Rocket Alignment Problem by Eliezer Yudkowsky Local Validity as a Key to Sanity and Civilization by Eliezer Yudkowsky Arguments about fast takeoff by Paul Christiano The Costly Coordination Mechanism of Common Knowledge by Ben Pace Toward a New Technical Explanation of Technical Explanation by Abram Demski Anti-social Punishment by Martin Sustrik The Tails Coming Apart As Metaphor For Life by Scott Alexander Babble by alkjash The Loudest Alarm Is Probably False by orthonormal The Intelligent Social Web by Valentine Prediction Markets: When Do They Work? by Zvi Coherence arguments do not imply goal-directed behavior by Rohin Shah Is Science Slowing Down? by Scott Alexander A voting theory primer for rationalists by Jameson Quinn and Robustness to Scale by Scott Garrabrant Top 15 posts not about AI Local Validity as a Key to Sanity and Civilization by Eliezer Yudkowsky The Costly Coordination Mechanism of Common Knowledge by Ben Pace Anti-social Punishment by Martin Sustrik The Tails Coming Apart As Metaphor For Life by Scott Alexander Babble by alkjash The Loudest Alarm Is Probably False by Orthonormal The Intelligent Social Web by Valentine Prediction Markets: When Do They Work? by Zvi Is Science Slowing Down? by Scott Alexander A voting theory primer for rationalists by Jameson Quinn Toolbox-thinking and Law-thinking by Eliezer Yudkowsky A Sketch of Good Communication by Ben Pace A LessWrong Crypto Autopsy by Scott Alexander Unrolling social metacognition: Three levels of meta are not enough. by Academian Varieties Of Argumentative Experience by Scott Alexander Top 10 posts about AI (The vote included 20 posts about AI.) Embedded Agents by Abram Demski and Scott Garrabrant The Rocket Alignment Problem by Eliezer Yudkowsky Arguments about fast takeoff by Paul Christiano Toward a New Technical Explanation of Technical Explanation by Abram Demski Coherence arguments do not imply goal-directed behavior by Rohin Shah Robustness to Scale by Scott Garrabrant Paul's research agenda FAQ by zhukeepa An Untrollable Mathematician Illustrated by Abram Demski Specification gaming examples in AI by Vika 2018 AI Alignment Literature Review and Charity Comparison by Larks The Complete Results Click Here If You Would Like A More Comprehensive Vote Data Spreadsheet To help users see the spread of the vote data, we've included swarmplot visualizations. For space reasons, only votes with weights between -10 and 16 are plotted. This covers 99.4% of votes. Gridlines are spaced 2 points apart. Concrete illustration: The plot immediately below has 18 votes ranging in strength from -3 to 12.
Post Title Total Vote Spread
1 Embedded Agents 209 (One outlier vote of +17 is not shown) 2 The Rocket Alignment Problem 183 3 Local Validity as a Key to Sanity and Civilization 133 4 Arguments about fast takeoff 98 5 The C...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: LessWrong Coronavirus Agenda , published by Elizabeth on the AI Alignment Forum. I’ve gone through a lot of introductions to this post but maybe this is the most honest one: I am scared. Quite scared, actually. My chances of catching COVID-19 are actually quite low, and my chances of surviving it if I do are quite high, and I’m still scared. What if I get into a car accident and have to go to the ER? Will they have a bed for me? Will I leave with coronavirus? What are my pregnant friends going to do? What is anyone over 70 going to do? My goal, and the goal of everyone on the LW staff, and I assume most everyone who’s participated in all the coronavirus threads, has been to figure out what is happening and what we can do about it. We’ve already done a lot. Posts like Seeing the Smoke got coronavirus on people’s radar faster than it otherwise would have been, aided by the numerous modeling threads backing it up. The Quarantine Preparations thread gave people a starting place to act from. The Justified Practical Advice (summary) thread let us share our expertise, in ways that led to concrete behavioral changes. More recently we examined asymptomatic transmission. I’ve had a legit, reasonably high ranking government official say they look at us to see where everyone else will be in weeks. This is currently the LessWrong team’s top priority, and they’ve done a number of things over the recent weeks to facilitate research and action on coronavirus, including hiring me to be a point person on it. To facilitate as much progress as possible over the coming weeks, habryka and I have compiled a list of what we consider the most important questions in fighting COVID, and are asking anyone with the skill to help us answer them. That list is at the end of this post. But first, what is the overall plan here? Who are we trying to help? We have three broad categories of potential beneficiaries in mind: Individuals making choices for themselves and their loved ones, who need accurate information about the current threat level and how to lower it with existing tech. Individuals creating the tools for the people above, meaning anything from noticing that copper tape is anti-viral to creating plans for DIY non-invasive ventilators, who need accurate information about how COVID-19 operates and where the current gaps and bottlenecks are. We’d like to help people in this group get volunteers and money when appropriate. Organizations and institutions making decisions that affect many people, who need all the information the previous two groups do, plus more to know what the effect of their decisions will be. How Are We Doing That? I am managing a Coronavirus Agenda, composed of what myself and habryka think are the most important coronavirus-related questions to answer (think we missed some? Please comment). But the full agenda is kind of overwhelming, and there are benefits to coordinating multiple people around the same question, so every so often I’ll pull out Spotlight Questions to generate a critical mass of attention around the most critical questions. I want to say “every so often” will be once a week, but I feel like those kinds of commitments are for situations where I know within an order of magnitude how many people are going to die in that week. I will spotlight as often as seems merited by the situation at the time. If your eye is caught by a question on the agenda that’s not currently spotlighted, of course pursue your interest. That’s the point of sharing the whole agenda. And if you think the agenda is missing something important, of course pursue that, and add a comment explaining it if you have time so I can add it. Without further adieu, the spotlight questions... Spotlight Questions What is the impact of varying initial viral load of COVID-19? The hypothesis that lower i...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: parenting rules, published by Dave Orr on the AI Alignment Forum. (crossposted from my nascent substack) Way back in 2012 I wrote up on livejournal (I told you this was a long time ago) a few parenting rules we lived by. This is the one livejournal post I regularly reshare, so here it is on a more modern platform. Our kids are older now (11 and 13) but with one exception I think these really hold up. That one exception is praise, where the research on praise seemed clear in 2012 and has since largely failed to replicate and certainly doesn’t have the effect size that everyone thought, so that one I no longer stand by. Here they are: Try never to lie. If kids ask a question and they aren't ready to hear the answer, just tell them that. This doesn't mean you have to go into every gruesome detail, it's fine to couch your answer at the level you think they'll understand and that you have time for, but they're smarter than you probably think. This does extend to things like Santa Claus and the Easter Bunny. We've told them those stories with the attempt to treat them just like any other fictional story. When Jackson point blank asked if Santa was real, I told him, "No, but it's a fun story and fun to pretend." There's a common pattern with kids to tell them things that are untrue but scary as a joke, like "Be careful not to slip down the drain!" Don't do that. Kids have trouble distinguishing fake warnings from real ones. However, saying untrue things as a joke is fine in the right context. "Elephant toes" is a fine answer to a question about what's for dinner. (As long as it's not true.) People say untrue things all the time, and taking the time to evaluate whether an adult is telling the truth is a useful skill. But until the kids are good at it, the untruths should be completely implausible, then can get more plausible as they get more on to you. Fun game, actually. The most difficult time for this one is when they want something that you don't want to give them. Like if mommy is downstairs and I'm doing bedtime, it's very tempting to claim that Katy is busy doing sometime important that can't be interrupted rather than just admitting she needs a break, or it's my turn to answer the late night call. Remember that every interaction is a repeated game, and your goal is not to win this one iteration, but to win the series. So if a child is crying because she wants something, even though it feels like a win to give in now (she stops crying which is better for everyone, you haven't really given up much), it's disastrous in the repeated game because she learns that she can get what she wants by crying. The flipside of that is that you have to let them get what they want in other ways. If you say no and they have good reasons why you should give in, or even an attempt at good reasons, sometimes you have to give in. You want them to be thinking critically and trying to persuade you. Here's an example. Katy put down a couple of dollars on the counter, which Jackson took, leading to the following conversation: Katy: Jackson, please leave those there. Jackson: But this one is mine. Katy: No it's not, I just put it there. Jackson: It looks just like the one I got last week! Katy: It's not the same one, I just put it there like 30 seconds ago! Jackson: But money is spongeable. Katy: ... Katy: Ok, you can have it. Because money being fungible is a great reason, even if it's not completely persuasive in this particular instance, and "spongeable" is awesome. If he'd started crying, the answer would have been a much more solid, no-more-negotiation "no." Almost never bluff. This is related to the first two points, but is really more like the second. If you threaten a consequence and don't follow through, they'll figure that out really quickly. Which leads to the following rule: be very ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: On Saying the Obvious , published by Grognor on the AI Alignment Forum. Related to: Generalizing from One Example, Connecting Your Beliefs (a call for help), Beware the Unsurprised The idea of this article is something I've talked about a couple of times in comments. It seems to require more attention. As a general rule, what is obvious to some people may not be obvious to others. Is this obvious to you? Maybe it was. Maybe it wasn't, and you thought it was because of hindsight bias. Imagine a substantive Less Wrong comment. It's insightful, polite, easy to understand, and otherwise good. Ideally, you upvote this comment. Now imagine the same comment, only with "obviously" in front. This shouldn't change much, but it does. This word seems to change the comment in multifarious bad ways that I'd rather not try to list. Uncharitably, I might reduce this whole phenomenon to an example of a mind projection fallacy. The implicit deduction goes like this: "I found obvious. Thus, is inherently obvious." The problem is that obviousness, like probability, is in the mind. The stigma of "obvious" ideas has another problem in preventing things from being said at all. I don't know how common this is, but I've actually been afraid of saying things that I thought were obvious, even though ignoring this fear and just posting has yet to result in a poorly-received comment. (That is, in fact, why I'm writing this.) Even tautologies, which are always obvious in retrospect, can be hard to spot. How many of us would have explicitly realized the weak anthropic principle without Nick Bostrom's help? And what about implications of beliefs you already hold? These should be obvious, and sometimes are, but our brains are notoriously bad at putting two and two together. Luke's example was not realizing that an intelligence explosion was imminent until he read the I.J. Good paragraph. I'm glad he provided that example, as it has saved me the trouble of making one. This is not (to paraphrase Eliezer) a thunderbolt of insight. I bring it up because I propose a few community norms based on the idea: Don't be afraid of saying something because it's "obvious". It's like how your teachers always said there are no stupid questions. Don't burden your awesome ideas with "obvious but it needs to be said". Don't vote down a comment because it says something "obvious" unless you've thought about it for a while. Also, don't shun "obvious" ideas. Don't call an idea obvious as though obviousness were an inherent property of the idea. Framing it as a personally obvious thing can be a more accurate way of saying what you're trying to say, but it's hard to do this without looking arrogant. (I suspect this is actually one of the reasons we implicitly treat obviousness as impersonal.) I'm not sure if these are good ideas, but I think implementing them would decrease the volume of thoughts we cannot think and things we can't say. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: I'm the new moderator, published by NancyLebovitz on the AI Alignment Forum. Viliam Bur made the announcement in Main, but not everyone checks main, so I'm repeating it here. During the following months my time and attention will be heavily occupied by some personal stuff, so I will be unable to function as a LW moderator. The new LW moderator is... NancyLebovitz! From today, please direct all your complaints and investigation requests to Nancy. Please not everyone during the first week. That can be a bit frightening for a new moderator. There are a few old requests I haven't completed yet. I will try to close everything during the following days, but if I don't do it till the end of January, then I will forward the unfinished cases to Nancy, too. Long live the new moderator! Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Backward Reasoning Over Decision Trees , published y Scott Alexander on the AI Alignment Forum. Game theory is the study of how rational actors interact to pursue incentives. It starts with the same questionable premises as economics: that everyone behaves rationally, that everyone is purely self-interested1, and that desires can be exactly quantified - and uses them to investigate situations of conflict and cooperation. Here we will begin with some fairly obvious points about decision trees, but by the end we will have the tools necessary to explain a somewhat surprising finding: that giving a US president the additional power of line-item veto may in many cases make the president less able to enact her policies. Starting at the beginning: The basic unit of game theory is the choice. Rational agents make choices in order to maximize their utility, which is sort of like a measure of how happy they are. In a one-person game, your choices affect yourself and maybe the natural environment, but nobody else. These are pretty simple to deal with: Here we visualize a choice as a branching tree. At each branch, we choose the option with higher utility; in this case, going to the beach. Since each outcome leads to new choices, sometimes the decision trees can be longer than this: Here's a slightly more difficult decision, denominated in money instead of utility. If you want to make as much money as possible, then your first choice - going to college or starting a minimum wage job right Now - seems to favor the more lucrative minimum wage job. But when you take Later into account, college opens up more lucrative future choices, as measured in the gray totals on the right-hand side. This illustrates the important principle of reasoning backward over decision trees. If you reason forward, taking the best option on the first choice and so on, you end up as a low-level manager. To get the real cash, you've got to start at the end - the total on the right - and then examine what choice at each branch will take you there. This is all about as obvious as, well, not hitting yourself on the head with a hammer, so let's move on to where it really gets interesting: two-player games. I'm playing White, and it's my move. For simplicity I consider only two options: queen takes knight and queen takes rook. The one chess book I've read values pieces in number of pawns: a knight is worth three pawns, a rook five, a queen nine. So at first glance, it looks like my best move is to take Black's rook. As for Black, I have arbitrarily singled out pawn takes pawn as her preferred move in the current position, but if I play queen takes rook, a new option opens up for her: bishop takes queen. Let's look at the decision tree: If I foolishly play this two player game the same way I played the one-player go-to-college game, I note that the middle branch has the highest utility for White, so I take the choice that leads there: capture the rook. And then Black plays bishop takes queen, and I am left wailing and gnashing my teeth. What did I do wrong? I should start by assuming Black will, whenever presented with a choice, take the option with the highest Black utility. Unless Black is stupid, I can cross out any branch that requires Black to play against her own interests. So now the tree looks like this: The two realistic options are me playing queen takes rook and ending up without a queen and -4 utility, or me playing queen takes knight and ending up with a modest gain of 2 utility. (my apologies if I've missed some obvious strategic possibility on this particular chessboard; I'm not so good at chess but hopefully the point of the example is clear.) This method of alternating moves in a branching tree matches both our intuitive thought processes during a chess game (“Okay, if I do this, then Black's goi...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Burdensome Details, published by Eliezer Yudkowsky on the AI Alignment Forum. Merely corroborative detail, intended to give artistic verisimilitude to an otherwise bald and unconvincing narrative . . . Pooh-Bah, in Gilbert and Sullivan’s The Mikado The conjunction fallacy is when humans assign a higher probability to a proposition of the form “A and B” than to one of the propositions “A” or “B” in isolation, even though it is a theorem that conjunctions are never likelier than their conjuncts. For example, in one experiment, 68% of the subjects ranked it more likely that “Reagan will provide federal support for unwed mothers and cut federal support to local governments” than that “Reagan will provide federal support for unwed mothers.”1 A long series of cleverly designed experiments, which weeded out alternative hypotheses and nailed down the standard interpretation, confirmed that conjunction fallacy occurs because we “substitute judgment of representativeness for judgment of probability.”2 By adding extra details, you can make an outcome seem more characteristic of the process that generates it. You can make it sound more plausible that Reagan will support unwed mothers, by adding the claim that Reagan will also cut support to local governments. The implausibility of one claim is compensated by the plausibility of the other; they “average out.” Which is to say: Adding detail can make a scenario sound more plausible, even though the event necessarily becomes less probable. If so, then, hypothetically speaking, we might find futurists spinning unconscionably plausible and detailed future histories, or find people swallowing huge packages of unsupported claims bundled with a few strong-sounding assertions at the center. If you are presented with the conjunction fallacy in a naked, direct comparison, then you may succeed on that particular problem by consciously correcting yourself. But this is only slapping a band-aid on the problem, not fixing it in general. In the 1982 experiment where professional forecasters assigned systematically higher probabilities to “Russia invades Poland, followed by suspension of diplomatic relations between the USA and the USSR” than to “Suspension of diplomatic relations between the USA and the USSR,” each experimental group was only presented with one proposition.3 What strategy could these forecasters have followed, as a group, that would have eliminated the conjunction fallacy, when no individual knew directly about the comparison? When no individual even knew that the experiment was about the conjunction fallacy? How could they have done better on their probability judgments? Patching one gotcha as a special case doesn’t fix the general problem. The gotcha is the symptom, not the disease. What could the forecasters have done to avoid the conjunction fallacy, without seeing the direct comparison, or even knowing that anyone was going to test them on the conjunction fallacy? It seems to me, that they would need to notice the word “and.” They would need to be wary of it—not just wary, but leap back from it. Even without knowing that researchers were afterward going to test them on the conjunction fallacy particularly. They would need to notice the conjunction of two entire details, and be shocked by the audacity of anyone asking them to endorse such an insanely complicated prediction. And they would need to penalize the probability substantially—a factor of four, at least, according to the experimental details. It might also have helped the forecasters to think about possible reasons why the US and Soviet Union would suspend diplomatic relations. The scenario is not “The US and Soviet Union suddenly suspend diplomatic relations for no reason,” but “The US and Soviet Union suspend diplomatic relations for any reason.” And the subjects wh...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Will COVID-19 survivors suffer lasting disability at a high rate? , published by jimrandomh on the AI Alignment Forum. The case fatality rate of 2019-nCoV (aka Coronavirus, COVID-19) is still uncertain, with estimates floating around ranging from 0.16%-5.7%, higher among the elderly and people with preexisting conditions, and lower among everyone else. However, the death rate doesn't capture all of the harms from infection. There is also time lost during the infection and recovery, there is the possibility of accelerated aging, and there is the possibility of long-term nonfatal disability, such as chronic fatigue. Since 2019-nCoV has only existed for about two months, there is no data on the long-term outcomes of its survivors. However, the rate of lasting disability among survivors is important for deciding what responses are appropriate. I'm particularly interested in estimating the risk of chronic fatigue from nCoV infection. If that risk is high, this would greatly increase the importance of avoiding it personally and of suppressing it in communities of people doing important work, and would also greatly increase the expected economic impact. As a starting point, I chose a similar but more severe virus, SARS, which was successfully contained in 2003. Out of 208 Canadian survivors of SARS, 22 (10%) appear in this study of subjects "who remained unable to return to their former occupation" with "clinical similarities to patients with fibromyalgia syndrome". This implies a high lower bound on the rate of disability among SARS survivors. However, this is only one virus, and may not be representative of severe respiratory illnesses. Good answers to this question would be: Papers estimating the rates of postviral fatigue from other viruses, especially respiratory viruses, viruses with severity comparable to 2019-nCoV, and among non-elderly patients Models of postviral fatigue and how they relate to 2019-nCoV Data on whether and how much lung damage from non-viral sources causes chronic fatigue Early data on 2019-nCoV which bears on this question Any research help on this question is greatly appreciated, even if it provides only a bit of information about a small corner of the problem, or reports that a strategy for answering the question failed to pan out. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Inner Alignment: Explain like I'm 12 Edition , published by Rafael Harth on the AI Alignment Forum. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. (This is an unofficial explanation of Inner Alignment based on the Miri paper Risks from Learned Optimization in Advanced Machine Learning Systems (which is almost identical to the LW sequence) and the Future of Life podcast with Evan Hubinger (Miri/LW). It's meant for anyone who found the sequence too long/challenging/technical to read.) Note that bold and italics means "this is a new term I'm introducing," whereas underline and italics is used for emphasis. What is Inner Alignment? Let's start with an abridged guide to how Machine Learning works: Choose a problem Decide on a space of possible solutions Find a good solution from that space If the problem is "find a tool that can look at any image and decide whether or not it contains a cat," then each conceivable set of rules for answering this question (formally, each function from the set of all pixels to the set yes, no ) defines one solution. We call each such solution a model. The space of possible models is depicted below. Since that's all possible models, most of them are utter nonsense. Pick a random one, and you're as likely to end up with a car-recognizer than a cat-recognizer – but far more likely with an algorithm that does nothing we can interpret. Note that even the examples I annotated aren't typical – most models would be more complex while still doing nothing related to cats. Nonetheless, somewhere in there is a model that would do a decent job on our problem. In the above, that's the one that says, "I look for cats." How does ML find such a model? One way that does not work is trying out all of them. That's because the space is too large: it might contain over 10 1000000 candidates. Instead, there's this thing called Stochastic Gradient Descent (SGD). Here's how it works: SGD begins with some (probably terrible) model and then proceeds in steps. In each step, it switches to another model that is "close" and hopefully a little better. Eventually, it stops and outputs the most recent model. 1 Note that, in the example above, we don't end up with the perfect cat-recognizer (the red box) but with something close to it – perhaps a model that looks for cats but has some unintended quirks. SGD does generally not guarantee optimality. The speech bubbles where the models explain what they're doing are annotations for the reader. From the perspective of the programmer, it looks like this: The programmer has no idea what the models are doing. Each model is just a black box. 2 A necessary component for SGD is the ability to measure a model's performance, but this happens while treating them as black boxes. In the cat example, assume the programmer has a bunch of images that are accurately labeled as "contains cat" and "doesn't contain cat." (These images are called the training data and the setting is called supervised learning.) SGD tests how well each model does on these images and, in each step, chooses one that does better. In other settings, performance might be measured in different ways, but the principle remains the same. Now, suppose that the images we have happen to include only white cats. In this case, SGD might choose a model implementing the rule "output yes if there is something white and with four legs." The programmer would not notice anything strange – all she sees is that the model output by SGD does well on the training data. In this setting, there is thus only a problem if our way of obtaining feedback is flawed. If it is perfect – if the pictures with cats are perfectly representative of what images-with-cats are like, and the pictures without cats are perfectly representative of what images-without-cats ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Coordination Schemes Are Capital Investments , published by Raemon on the AI Alignment Forum. Second post in the Coordination Frontier sequence. Intro is here. Once upon a time, we didn’t have money, or auctions, or markets. We didn’t have the concept of voting (let alone distinctions between first-past-the-post, or approval voting). We didn’t have banks or stock trading. We didn’t have Kickstarter. We had to haggle over everything, which cost time. And it fundamentally limited the scale at which we could accomplish things. If a transaction cost exceeds the value of a trade, that trade can’t happen. [edit: arguably false, see discussion in comments, but fortunately not that cruxy for the rest of the post] Once upon a time, someone had to be the first person to invent each of these concepts. The people around them were probably vaguely annoyed at having to learn a new thing. Many of the concepts required multiple building blocks and took thousands of years to reach saturation. If you are happen to be around a lot of people around who: geek out about coordination schemes, and have an entrepreneurial bent Then a really valuable thing you can do is explore new coordination schemes, while making an effort to distill them down into something the general population might actually be able to use. This is hard. There are lots of failure modes. But, coordination schemes are a key thing that makes humanity powerful. Pushing the state-of-the-art forward is valuable. With that in mind, let’s explore some examples and concepts. Examples: Negotiation Technology Second Price Auctions I remember the first time I got the results of a second-price auction. It felt like magic. Instead of arguing for hours about who got which room in a new apartment, I just wrote down my true preference for how much I was willing to pay for each room. Then I automagically got assigned a room that was cheaper than I had been willing to pay for it. (In a second price auction, each participant submits a sealed bid, and then person who bid highest pays the second-highest bid) At previous apartments, we had just eyeballed the size of each room, and then came up with rent-allocations for each room that had vaguely round, fair-sounding numbers, and then picked rooms. It was chaotic, and didn't have a good way to account for some people valuing particular rooms for subtle, personal reasons. Second price auction was an important, conceptual advance. But it’s not very popular in broader society. Why? At my first second-price auction, someone had to patiently explain it to me before I trusted it. I'm not sure I actually did trust it until after everything had been resolved. Beforehand, it felt overcomplicated and I was sort of annoyed that we weren’t just eyeballing the rooms and winging it with nice round numbers. The explanation process took awhile, and I’m not 100% sure the marginal improvements in fairness/efficiency were worth it. Thinking about how much I valued each room was actually pretty hard. “How much do I value something” is a skill that I had never really used – I never needed to. I just needed to eyeball a price and think “Worth it? or not worth it.” Still, I think this was clearly worthwhile. Because later, I got to use second price auctions in other situations. I went in having a clear understanding of how it worked. I had developed some skill of evaluating how much things were worth to me. I got all of the magical “it just works” feeling and none of the stress. Other Formats More recently, I was having a different sort of negotiation (over how much to sell someone some used exercise equipment for). They proposed a negotiation scheme intended to sidestep the haggling. The premise was: we each privately note down our true value for the object, then reveal simultaneously. If their value was higher th...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How factories were made safe, published by jasoncrawford on the AI Alignment Forum. This is a linkpost for Angelo Guira was just sixteen years old when he began working in the steel factory. He was a “trough boy,” and his job was to stand at one end of the trough where red-hot steel pipes were dropped. Every time a pipe fell, he pulled a lever that dumped the pipe onto a cooling bed. He was a small lad, and at first they hesitated to take him, but after a year on the job the foreman acknowledged he was the best boy they’d had. Until one day when Angelo was just a little too slow—or perhaps the welder was a little too quick—and a second pipe came out of the furnace before he had dropped the first. The one pipe struck the other, and sent it right through Angelo’s body, killing him. If only he had been standing up, out of the way, instead of sitting down—which the day foreman told him was dangerous, but the night foreman allowed. If only they had installed the guard plate before the accident, instead of after. If only. Angelo was not the only casualty of the steel mills of Allegheny County, Pennsylvania that year. In the twelve months from July 1906 through June 1907, ten in total were killed by the operation of rolls. Twenty-two were killed by hot metal explosions. Five were asphyxiated by furnace gas. Thirty-one fatalities were attributed to the operation of the railroad at the steel yards, and forty-two to the operation of cranes. Twenty-four men fell from a height, or into a pit. Eight died from electric shock. In all, there were 195 casualties in the steel mills in those twelve months, and these were just a portion of the total of 526 deaths from work accidents. In addition, there were 509 other accidents that sent men to the hospital, at least 76 of which resulted in serious, permanent injury. Work-Accidents and the Law, 1910 In 1907, according to a report from the Bureau of Labor Statistics, the overall fatality rate in the iron and steel industry was about 220 per 100,000 full-time workers. By 2019, however, that rate had fallen to only 26.3 per 100,000, a reduction of almost 90%. The story of workplace safety illustrates both the serious problems that progress can cause, and how the solution to those problems can be found in further progress. It’s a fascinating story in its own right, and in it we find lessons about safety in general, about liability law, and about the early history of capitalism. The dangers of early factories The Industrial Revolution created a dramatic boost in labor productivity through mechanization, the application of power, and the institution of the factory, which reorganized tasks and workers into a new mode of production. This led to vastly higher growth rates in GDP per capita and ultimately in real wages. But the very same elements—factories, machines, energy—created new risks that neither workers nor management were prepared for. The pre-industrial world had plenty of dangerous jobs: mining for coal or metals, tending a blast furnace, sailing in the merchant marine. And of course, craftsman’s shops often posed risks from sharp tools or high heat (just ask Johnny Tremain). But the industrial factory brought a new set of risks. Machines had exposed blades and gears that could catch fingers and hands—woodworking machines, especially joiners, were particularly dangerous. Tools were powered by belts, shafts and flywheels that were similarly unguarded. High above the floor of the factory or mill were walkways and ladders without railings. Cranes could knock workers dead, or drop heavy materials on them. Steam engines had high-pressure boilers, which could explode. High-voltage wires threatened electrocution. Smelting furnaces posed a risk of “hot-metal breakouts”. And workers could be asphyxiated by toxic gases. Workers lost fingers, eye...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Yes, a blog , published by Academian on the AI Alignment Forum. When I recommend LessWrong to people, their gut reaction is usually "What? You think the best existing philosophical treatise on rationality is a blog?" Well, yes, at the moment I do. "But why is it not an ancient philosophical manuscript written by a single Very Special Person with no access to the massive knowledge the human race has accumulated over the last 100 years?" Besides the obvious? Three reasons: idea selection, critical mass, and helpful standards for collaboration and debate. Idea selection. Ancient people came up with some amazing ideas, like how to make fire, tools, and languages. Those ideas have stuck around, and become integrated in our daily lives to the point where they barely seem like knowledge anymore. The great thing is that we don't have to read ancient cave writings to be reminded that fire can keep us warm; we simply haven't forgotten. That's why more people agree that fire can heat your home than on how the universe began. Classical philosophers like Hume came up with some great ideas, too, especially considering that they had no access to modern scientific knowledge. But you don't have to spend thousands of hours reading through their flawed or now-uninteresting writings to find their few truly inspiring ideas, because their best ideas have become modern scientific knowledge. You don't need to read Hume to know about empiricism, because we simply haven't forgotten it... that's what science is now. You don't have to read Kant to think abstractly about Time; thinking about "timelines" is practically built into our language nowadays. See, society works like a great sieve that remembers good ideas, and forgets some of the bad ones. Plenty of bad ideas stick around because they're viral (self-propagating for reasons other than helpfulness/verifiability), so you can't always trust an idea just because it's old. But that's how any sieve works: it narrows your search. It keeps the stuff you want, and throws away some of the bad stuff so you don't have to look at it. LessWrong itself is an update patch for philosophy to fix compatibility issues with science and render it more useful. That it would exist now rather than much earlier is no coincidence: right now, it's the gold at the bottom of the pan, because it's taking the idea filtering process to a whole new level. Here's a rough timeline of how LessWrong happened: Critical mass. To get off the ground, a critical mass of very good ideas was needed: the LessWrong Sequences. Eliezer Yudkowsky spent several years posting a lot of extremely sane writing on OvercomingBias.com, and then founded LessWrong.com, attracting the attention of other people who were annoyed at the lower density of good ideas in older literature. Part of what made them successful is that the sequences are written in a widely learned, widely applicable language: the language of basic science and mathematics. A lot of the serious effort in classical philosophy was spent trying to develop precise and appropriate terminology in which to communicate, and so joining the conversation always required a serious exclusive study of the accumulated lingo and concepts. But nowadays we can study rationality by transfer of learning from tried-and-true technical disciplines like probability theory, computer science, biology, and even physics. So the Sequences were written. Then, using an explicit upvote system, LessWrong and its readers began accelerating the historically slow process of idea selection: if you wanted to be sure to see something inspiring, you just had to click "TOP" to see a list of top voted posts.1 Collaboration and debate. Finally, with a firm foundation taking hold, there is now a context, a language, and a community that will understand your good ideas. Re...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: If You Demand Magic, Magic Won't Help, published Eliezer Yudkowsky on the AI Alignment Forum. Most witches don't believe in gods. They know that the gods exist, of course. They even deal with them occasionally. But they don't believe in them. They know them too well. It would be like believing in the postman. Terry Pratchett, Witches Abroad Once upon a time, I was pondering the philosophy of fantasy stories And before anyone chides me for my "failure to understand what fantasy is about", let me say this: I was raised in an SF&F household. I have been reading fantasy stories since I was five years old. I occasionally try to write fantasy stories. And I am not the sort of person who tries to write for a genre without pondering its philosophy. Where do you think story ideas come from? Anyway: I was pondering the philosophy of fantasy stories, and it occurred to me that if there were actually dragons in our world—if you could go down to the zoo, or even to a distant mountain, and meet a fire-breathing dragon—while nobody had ever actually seen a zebra, then our fantasy stories would contain zebras aplenty, while dragons would be unexciting. Now that's what I call painting yourself into a corner, wot? The grass is always greener on the other side of unreality. In one of the standard fantasy plots, a protagonist from our Earth, a sympathetic character with lousy grades or a crushing mortgage but still a good heart, suddenly finds themselves in a world where magic operates in place of science. The protagonist often goes on to practice magic, and become in due course a (superpowerful) sorcerer. Now here's the question—and yes, it is a little unkind, but I think it needs to be asked: Presumably most readers of these novels see themselves in the protagonist's shoes, fantasizing about their own acquisition of sorcery. Wishing for magic. And, barring improbable demographics, most readers of these novels are not scientists. Born into a world of science, they did not become scientists. What makes them think that, in a world of magic, they would act any differently? If they don't have the scientific attitude, that nothing is "mere"—the capacity to be interested in merely real things—how will magic help them? If they actually had magic, it would be merely real, and lose the charm of unattainability. They might be excited at first, but (like the lottery winners who, six months later, aren't nearly as happy as they expected to be), the excitement would soon wear off. Probably as soon as they had to actually study spells. Unless they can find the capacity to take joy in things that are merely real. To be just as excited by hang-gliding, as riding a dragon; to be as excited by making a light with electricity, as by making a light with magic... even if it takes a little study... Don't get me wrong. I'm not dissing dragons. Who knows, we might even create some, one of these days. But if you don't have the capacity to enjoy hang-gliding even though it is merely real, then as soon as dragons turn real, you're not going to be any more excited by dragons than you are by hang-gliding. Do you think you would prefer living in the Future, to living in the present? That's a quite understandable preference. Things do seem to be getting better over time. But don't forget that this is the Future, relative to the Dark Ages of a thousand years earlier. You have opportunities undreamt-of even by kings. If the trend continues, the Future might be a very fine place indeed in which to live. But if you do make it to the Future, what you find, when you get there, will be another Now. If you don't have the basic capacity to enjoy being in a Now—if your emotional energy can only go into the Future, if you can only hope for a better tomorrow—then no amount of passing time can help you. (Yes, in the Future there...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Local Validity as a Key to Sanity and Civilization, published by Eliezer Yudkowsky on the AI Alignment Forum. (Cross-posted from Facebook.) 0. Tl;dr: There's a similarity between these three concepts: A locally valid proof step in mathematics is one that, in general, produces only true statements from true statements. This is a property of a single step, irrespective of whether the final conclusion is true or false. There's such a thing as a bad argument even for a good conclusion. In order to arrive at sane answers to questions of fact and policy, we need to be curious about whether arguments are good or bad, independently of their conclusions. The rules against fallacies must be enforced even against arguments for conclusions we like. For civilization to hold together, we need to make coordinated steps away from Nash equilibria in lockstep. This requires general rules that are allowed to impose penalties on people we like or reward people we don't like. When people stop believing the general rules are being evaluated sufficiently fairly, they go back to the Nash equilibrium and civilization falls. i. The notion of a locally evaluated argument step is simplest in mathematics, where it is a formalizable idea in model theory. In math, a general type of step is 'valid' if it only produces semantically true statements from other semantically true statements, relative to a given model. If x = y in some set of variable assignments, then 2x = 2y in the same model. Maybe x doesn't equal y, in some model, but even if it doesn't, the local step from "x = y" to "2x = 2y" is a locally valid step of argument. It won't introduce any new problems. Conversely, xy = xz does not imply y = z. It happens to work when x = 2, y = 3, and z= 3, in which case the two statements say "6 = 6" and "3 = 3" respectively. But if x = 0, y = 4, z = 17, then we have "0 = 0" on one side and "4 = 17" on the other. We can feed in a true statement and get a false statement out the other end. This argument is not locally okay. You can't get the concept of a "mathematical proof" unless on some level—though often an intuitive level rather than an explicit one—you understand the notion of a single step of argument that is locally okay or locally not okay, independent of whether you globally agreed with the final conclusion. There's a kind of approval you give to the pieces of the argument, rather than looking the whole thing over and deciding whether you like what came out the other end. Once you've grasped that, it may even be possible to convince you of mathematical results that sound counterintuitive. When your understanding of the rules governing allowable argument steps has become stronger than your faith in your ability to judge whole intuitive conclusions, you may be convinced of truths you would not otherwise have grasped. ii. More generally in life, even outside of mathematics, there are such things as bad arguments for good conclusions. There are even such things as genuinely good arguments for false conclusions, though of course those are much rarer. By the Bayesian definition of evidence, "strong evidence" is exactly that kind of evidence which we very rarely expect to find supporting a false conclusion. Lord Kelvin's careful and multiply-supported lines of reasoning arguing that the Earth could not possibly be so much as a hundred million years old, all failed simultaneously in a surprising way because that era didn't know about nuclear reactions. But most of the time this does not happen. On the other hand, bad arguments for true conclusions are extremely easy to come by, because there are tiny elves that whisper them to people. There isn't anything the least bit more difficult in making an argument terrible when it leads to a good conclusion, since the tiny elves own lawnmowers. One of the mar...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Evolution of Modularity, published by johnswentworth on the AI Alignment Forum. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. This post is based on chapter 15 of Uri Alon’s book An Introduction to Systems Biology: Design Principles of Biological Circuits. See the book for more details and citations; see here for a review of most of the rest of the book. Fun fact: biological systems are highly modular, at multiple different scales. This can be quantified and verified statistically, e.g. by mapping out protein networks and algorithmically partitioning them into parts, then comparing the connectivity of the parts. It can also be seen more qualitatively in everyday biological work: proteins have subunits which retain their function when fused to other proteins, receptor circuits can be swapped out to make bacteria follow different chemical gradients, manipulating specific genes can turn a fly’s antennae into legs, organs perform specific functions, etc, etc. On the other hand, systems designed by genetic algorithms (aka simulated evolution) are decidedly not modular. This can also be quantified and verified statistically. Qualitatively, examining the outputs of genetic algorithms confirms the statistics: they’re a mess. So: what is the difference between real-world biological evolution vs typical genetic algorithms, which leads one to produce modular designs and the other to produce non-modular designs? Kashtan & Alon tackle the problem by evolving logic circuits under various conditions. They confirm that simply optimizing the circuit to compute a particular function, with random inputs used for selection, results in highly non-modular circuits. However, they are able to obtain modular circuits using “modularly varying goals” (MVG). The idea is to change the reward function every so often (the authors switch it out every 20 generations). Of course, if we just use completely random reward functions, then evolution doesn’t learn anything. Instead, we use “modularly varying” goal functions: we only swap one or two little pieces in the (modular) objective function. An example from the book: The upshot is that our different goal functions generally use similar sub-functions - suggesting that they share sub-goals for evolution to learn. Sure enough, circuits evolved using MVG have modular structure, reflecting the modular structure of the goals. (Interestingly, MVG also dramatically accelerates evolution - circuits reach a given performance level much faster under MVG than under a fixed goal, despite needing to change behavior every 20 generations. See either the book or the paper for more on that.) How realistic is MVG as a model for biological evolution? I haven’t seen quantitative evidence, but qualitative evidence is easy to spot. MVG as a theory of biological modularity predicts that highly variable subgoals will result in modular structure, whereas static subgoals will result in a non-modular mess. Alon’s book gives several examples: Chemotaxis: different bacteria need to pursue/avoid different chemicals, with different computational needs and different speed/energy trade-offs, in various combinations. The result is modularity: separate components for sensing, processing and motion. Animals need to breathe, eat, move, and reproduce. A new environment might have different food or require different motions, independent of respiration or reproduction - or vice versa. Since these requirements vary more-or-less independently in the environment, animals evolve modular systems to deal with them: digestive tract, lungs, etc. Ribosomes, as an anti-example: the functional requirements of a ribosome hardly vary at all, so they end up non-modular. They have pieces, but most pieces do not have an obvious distinct function. To sum it up: modul...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Negative Feedback and Simulacra, published by Elizabeth on the AI Alignment Forum. Part 1: Examples There’s a thing I want to talk about but it’s pretty nebulous so I’m going to start with examples. Feel free to skip ahead to part 2 if you prefer. Example 1: Hot sauce In this r/AmITheAsshole post, a person tries some food their their girlfriend cooked, likes it, but tries another bite with hot sauce. Girlfriend says this “.insults her cooking and insinuates that she doesn’t know how to cook”. As objective people not in this fight, we can notice that her cooking is exactly as good as it is whether or not he adds hot sauce. Adding hot sauce reveals information (maybe about him, maybe about the food), but cannot change the facts on the ground. Yet she is treating him like he retroactively made her cooking worse in a way that somehow reflects on her, or made a deliberate attempt to hurt her. Example 2: Giving a CD back to the library Back when I would get books on CD I would sometimes forget the last one in my drive or car. Since I didn’t use CDs that often, I would find the last CD sometimes months later. To solve this, I would drop the CD in the library book return slot, which, uh, no longer looks like a good solution to me, in part because of the time I did this in front of a friend and she questioned it. Not rudely or anything, just “are you sure that’s safe? Couldn’t the CD snap if something lands wrong?.” I got pretty angry about this, but couldn’t actually deny she had a point, so settled for thinking that if she had violated a friend code by not pretending my action was harmless. I was not dumb enough to say this out loud, but I radiated the vibe and she dropped it. Example 3: Elizabeth fails to fit in at martial arts A long time ago I went to a martial arts studio. The general classes (as opposed to specialized classes like grappling) were preceded by an optional 45 minute warm up class. Missing the warm up was fine, even if you took a class before and after. Showing up 10 minutes before the general class and doing your own warm ups on the adjacent mats was fine too. What was not fine was doing the specialized class, doing your own warm ups on adjacent maps for the full 45 minutes while the instructor led regular warm ups, and then rejoining for the general class. That was “very insulting to the instructor”. This was a problem for me because the regular warm ups hurt, in ways that clearly meant they were bad for me (and this is at a place I regularly let people hit me in the head). Theoretically I could have asked the instructor to give me something different, but that is not free and the replacements wouldn’t have been any better, which is not surprising because no one there had the slightest qualification to do personal training or physical therapy. So basically the school wanted me to pretend I was in a world where they were competent to create exercise routines, more competent than I despite having no feedback from my body, and considered not pretending disrespectful to the person leading warm ups. Like the hot sauce example, the warm ups were as good as they were regardless of my participation – and they knew that, because they didn’t demand I participate. But me doing my own warm ups broke the illusion of competence they were trying to maintain. Example 4: Imaginary Self-Help Guru I listened to an interview where the guest was a former self-help guru who had recently shut down his school. Well, I say listened, but I’ve only done the first 25% so far. For that reason this should be viewed less as “this specific real person believes these specific things” and more like “a character Elizabeth made up in her head inspired by things a real person said.” and. For that reason, I won’t be using his name or linking to the podcast. Anyways, the actual person talked ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: MIRI comments on Cotra's "Case for Aligning Narrowly Superhuman Models", published by Rob Bensinger on the AI Alignment Forum. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. Below, I’ve copied comments left by MIRI researchers Eliezer Yudkowsky and Evan Hubinger on March 1–3 on a draft of Ajeya Cotra’s "Case for Aligning Narrowly Superhuman Models." I've included back-and-forths with Cotra, and interjections by me and Rohin Shah. The section divisions below correspond to the sections in Cotra's post. 0. Introduction How can we train GPT-3 to give “the best health advice it can give” using demonstrations and/or feedback from humans who may in some sense “understand less” about what to do when you’re sick than GPT-3 does? Eliezer Yudkowsky: I've had some related conversations with Nick Beckstead. I'd be hopeful about this line of work primarily because I think it points to a bigger problem with the inscrutable matrices of floating-point numbers, namely, we have no idea what the hell GPT-3 is thinking and cannot tell it to think anything else. GPT-3 has a great store of medical knowledge, but we do not know where that medical knowledge is; we do not know how to tell it to internally apply its medical knowledge rather than applying other cognitive patterns it has stored. If this is still the state of opacity of AGI come superhuman capabilities, we are all immediately dead. So I would be relatively more hopeful about any avenue of attack for this problem that used anything other than an end-to-end black box - anything that started to address, "Well, this system clearly has a bunch of medical knowledge internally, can we find that knowledge and cause it to actually be applied" rather than "What external forces can we apply to this solid black box to make it think more about healthcare?" Evan Hubinger: +1 I continue to think that language model transparency research is the single most valuable current research direction within the class of standard ML research, for similar reasons to what Eliezer said above. Ajeya Cotra: Thanks! I'm also excited about language model transparency, and would love to find ways to make it more tractable as a research statement / organizing question for a field. I'm not personally excited about the connotations of transparency because it evokes the neuroscience-y interpretability tools, which don't feel scalable to situations when we don't get the concepts the model is using, and I'm very interested in finding slogans to keep researchers focused on the superhuman stuff. Ajeya Cotra: I've edited the description of the challenge to emphasize human feedback less. It now reads "How can we get GPT-3 to give “the best health advice it can give” when humans in some sense “understand less” about what to do when you’re sick than GPT-3 does? And in that regime, how can we even tell/verify that it’s “doing the best it can”?" Rob Bensinger: Nate and I tend to talk about "understandability" instead of "transparency" exactly because we don't want to sound like we're talking about normal ML transparency work. Eliezer Yudkowsky: Other possible synonyms: Clarity, legibility, cognitive readability. Ajeya Cotra: Thanks all -- I like the project of trying to come up with a good handle for the kind of language model transparency we're excited about (and have talked to Nick, Evan, etc about it too) but I think I don't want to push it in this blog post right now because I haven't hit on something I believe in and I want to ship this. In the end, we probably want to find ways to meaningfully supervise (or justifiably trust) models that are more capable than ~all humans in ~all domains. Eliezer Yudkowsky: (I think you want an AGI that is superhuman in engineering domains and infrahuman in human-modeling-and-manipulation if such a ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EfficientZero: human ALE sample-efficiency w/MuZero+self-supervised, published by gwern on the AI Alignment Forum. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. This is a linkpost for "Mastering Atari Games with Limited Data", Ye et al 2021: Reinforcement learning has achieved great success in many applications. However, sample efficiency remains a key challenge, with prominent methods requiring millions (or even billions) of environment steps to train. Recently, there has been significant progress in sample efficient image-based RL algorithms; however, consistent human-level performance on the Atari game benchmark remains an elusive goal. We propose a sample efficient model-based visual RL algorithm built on MuZero, which we name EfficientZero. Our method achieves 190.4% mean human performance and 116.0% median performance on the Atari 100k benchmark with only two hours of real-time game experience and outperforms the state SAC in some tasks on the DMControl 100k benchmark. This is the first time an algorithm achieves super-human performance on Atari games with such little data. EfficientZero's performance is also close to DQN's performance at 200 million frames while we consume 500 times less data. EfficientZero's low sample complexity and high performance can bring RL closer to real-world applicability. We implement our algorithm in an easy-to-understand manner and it is available at this https URL. We hope it will accelerate the research of MCTS-based RL algorithms in the wider community. This work is supported by the Ministry of Science and Technology of the People’s Republic of China, the 2030 Innovation Megaprojects “Program on New Generation Artificial Intelligence” (Grant No. 2021AAA0150000). Some have said that poor sample-efficiency on ALE has been a reason to downplay DRL progress or implications. The primary boost in EfficientZero (table 3), pushing it past the human benchmark, is some simple self-supervised learning (SimSiam on predicted vs actual observations). Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Spaghetti Towers, published by eukaryote on the AI Alignment Forum. This is a linkpost for/ Here’s a pattern I’d like to be able to talk about. It might be known under a certain name somewhere, but if it is, I don’t know it. I call it a Spaghetti Tower. It shows up in large complex systems that are built haphazardly. Someone or something builds the first Part A. Later, someone wants to put a second Part B on top of Part A, either out of convenience (a common function, just somewhere to put it) or as a refinement to Part A. Now, suppose you want to tweak Part A. If you do that, you might break Part B, since it interacts with bits of Part A. So you might instead build Part C on top of the previous ones. And by the time your system looks like this, it’s much harder to tell what changes you can make to an earlier part without crashing some component, so you’re basically relegated to throwing another part on top of the pile. I call these spaghetti towers for two reasons: One, because they tend to quickly take on circuitous knotty tangled structures, like what programmers call “spaghetti code”. (Part of the problem with spaghetti code is that it can lead to spaghetti towers.) Especially since they’re usually interwoven in multiple dimensions, and thus look more like this: “Can you just straighten out the yellow one without touching any of the others? Thanks.” Second, because shortsightedness in the design process is a crucial part of spaghetti machines. In order to design a spaghetti system, you throw spaghetti against a wall and see if it sticks. Then, when you want to add another part, you throw more spaghetti until it sticks to that spaghetti. And later, you throw more spaghetti. So it goes. And if you decide that you want to tweak the bottom layer to make it a little more useful – which you might want to do because, say, it was built out of spaghetti – without damaging the next layers of gummy partially-dried spaghetti, well then, good luck. Note that all systems have load-bearing, structural pieces. This does not make them spaghetti towers. The distinction about spaghetti towers is that they have a lot of shoddily-built structural components that are completely unintentional. A bridge has major load-bearing components – they’re pretty obvious, strong, elegant, and efficiently support the rest of the structure. A spaghetti tower is more like this. Image from the always-delightful r/DiWHY. (The motto of the spaghetti tower is “Sure, it works fine, as long as you never run lukewarm water through it and unplug the washing machine during thunderstorms.”) Where do spaghetti towers appear? Basically all of biology works like this. Absolutely all of evolution is made by throwing spaghetti against walls and seeing what sticks. (More accurately, throwing nucleic acid against harsh reality and seeing what successfully makes more nucleic acid.) We are 3.5 billion years of hacks in fragile trench coats. Scott Star Codex describes the phenomenon in neurotransmitters, but it’s true for all of molecular biology: You know those stories about clueless old people who get to their Gmail account by typing “Google” into Bing, clicking on Google in the Bing search results, typing “Gmail” into Google, and then clicking on Gmail in the Google search results? I am reading about serotonin transmission now, and everything in the human brain works on this principle. If your brain needs to downregulate a neurotransmitter, it’ll start by upregulating a completely different neurotransmitter, which upregulates the first neurotransmitter, which hits autoreceptors that downregulate the first neurotransmitter, which then cancel the upregulation, and eventually the neurotransmitter gets downregulated. Meanwhile, my patients are all like “How come this drug that was supposed to cure my depression is givin...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Book Summary: Consciousness and the Brain , published by Kaj_Sotala on the AI Alignment Forum. One of the fundamental building blocks of much of consciousness research, is that of Global Workspace Theory (GWT). One elaboration of GWT, which focuses on how it might be implemented in the brain, is the Global Neuronal Workspace (GNW) model in neuroscience. Consciousness and the Brain is a 2014 book that summarizes some of the research and basic ideas behind GNW. It was written by Stanislas Dehaene, a French cognitive neuroscientist with a long background in both consciousness research and other related topics. The book and its replicability Given that this is a book on psychology and neuroscience that was written before the replication crisis, an obligatory question before we get to the meat of it is: how reliable are any of the claims in this book? After all, if we think that this is based on research which is probably not going to replicate, then we shouldn’t even bother reading the book. I think that the book’s conclusions are at least reasonably reliable in their broad strokes, if not necessarily all the particular details. That is, some of the details in the cited experiments may be off, but I expect most of them to at least be pointing in the right direction. Here are my reasons: First, scientists in a field usually have an informal hunch of how reliable the different results are. Even before the replication crisis hit, I had heard private comments from friends working in social psychology, who were saying that everything in the field was built on shaky foundations and how they didn’t trust even their own findings much. In contrast, when I asked a friend who works with some people doing consciousness research, he reported back that they generally felt that GWT/GNW-style theories have a reasonably firm basis. This isn’t terribly conclusive but at least it’s a bit of evidence. Second, for some experiments the book explicitly mentions that they have been replicated. That said, some of the reported experiments seemed to be one-off ones, and I did not yet investigate the details of the claimed replications. Third, this is a work of cognitive neuroscience. Cognitive neuroscience is generally considered a subfield of cognitive psychology, and cognitive psychology is the part of psychology whose results have so far replicated the best. One recent study tested nine key findings from cognitive psychology, and found that they all replicated. The 2015 "Estimating the reproducibility of Psychological Science" study, managed to replicate 50% of recent results in cognitive psychology, as opposed to 25% of results in social psychology. (If 50% sounds low, remember that we should expect some true results to also fail a single replication, so a 50% replication rate doesn’t imply that 50% of the results would be false. Also, a field with a 90% replication rate would probably be too conservative in choosing which experiments to try.) Cognitive psychology replicating pretty well is probably because it deals with phenomena which are much easier to rigorously define and test than social psychology does, so in that regard it's closer to physics than it is to social psychology. On several occasions, the book reports something like “people did an experiment X, but then someone questioned whether the results of that experiment really supported the hypothesis in question or not, so an experiment X+Y was done that repeated X but also tested Y, to help distinguish between two possible interpretations of X”. The general vibe that I get from the book is that different people have different intuitions about how consciousness works, and when someone reports a result that contradicts the intuitions of other researchers, those other researchers are going to propose an alternative interpretation that...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Understanding “Deep Double Descent” , published by evhub on the AI Alignment Forum. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. If you're not familiar with the double descent phenomenon, I think you should be. I consider double descent to be one of the most interesting and surprising recent results in analyzing and understanding modern machine learning. Today, Preetum et al. released a new paper, “Deep Double Descent,” which I think is a big further advancement in our understanding of this phenomenon. I'd highly recommend at least reading the summary of the paper on the OpenAI blog. However, I will also try to summarize the paper here, as well as give a history of the literature on double descent and some of my personal thoughts. Prior work The double descent phenomenon was first discovered by Mikhail Belkin et al., who were confused by the phenomenon wherein modern ML practitioners would claim that “bigger models are always better” despite standard statistical machine learning theory predicting that bigger models should be more prone to overfitting. Belkin et al. discovered that the standard bias-variance tradeoff picture actually breaks down once you hit approximately zero training error—what Belkin et al. call the “interpolation threshold.” Before the interpolation threshold, the bias-variance tradeoff holds and increasing model complexity leads to overfitting, increasing test error. After the interpolation threshold, however, they found that test error actually starts to go down as you keep increasing model complexity! Belkin et al. demonstrated this phenomenon in simple ML methods such as decision trees as well as simple neural networks trained on MNIST. Here's the diagram that Belkin et al. use in their paper to describe this phenomenon: Belkin et al. describe their hypothesis for what's happening as follows: All of the learned predictors to the right of the interpolation threshold fit the training data perfectly and have zero empirical risk. So why should some—in particular, those from richer functions classes—have lower test risk than others? The answer is that the capacity of the function class does not necessarily reflect how well the predictor matches the inductive bias appropriate for the problem at hand. [The inductive bias] is a form of Occam’s razor: the simplest explanation compatible with the observations should be preferred. By considering larger function classes, which contain more candidate predictors compatible with the data, we are able to find interpolating functions that [are] “simpler”. Thus increasing function class capacity improves performance of classifiers. I think that what this is saying is pretty magical: in the case of neural nets, it's saying that SGD just so happens to have the right inductive biases that letting SGD choose which model it wants the most out of a large class of models with the same training performance yields significantly better test performance. If you're right on the interpolation threshold, you're effectively “forcing” SGD to choose from a very small set of models with perfect training accuracy (maybe only one realistic option), thus ignoring SGD's inductive biases completely—whereas if you're past the interpolation threshold, you're letting SGD choose which of many models with perfect training accuracy it prefers, thus allowing SGD's inductive bias to shine through. I think this is strong evidence for the critical importance of implicit simplicity and speed priors in making modern ML work. However, such biases also produce strong incentives for mesa-optimization (since optimizers are simple, compressed policies) and pseudo-alignment (since simplicity and speed penalties will favor simpler, faster proxies). Furthermore, the arguments for the universal prior and minimal ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Can you control the past?, published by Joe Carlsmith on the AI Alignment Forum. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. (Cross-posted from Hands and Cities. Lots of stuff familiar to LessWrong folks interested in decision theory.) I think that you can “control” events you have no causal interaction with, including events in the past, and that this is a wild and disorienting fact, with uncertain but possibly significant implications. This post attempts to impart such disorientation. My main example is a prisoner’s dilemma between perfect deterministic software twins, exposed to the exact same inputs. This example that shows, I think, that you can write on whiteboards light-years away, with no delays; you can move the arm of another person, in another room, just by moving your own. This, I claim, is extremely weird. My topic, more broadly, is the implications of this weirdness for the theory of instrumental rationality (“decision theory”). Many philosophers, and many parts of common sense, favor causal decision theory (CDT), on which, roughly, you should pick the action that causes the best outcomes in expectation. I think that deterministic twins, along with other examples, show that CDT is wrong. And I don’t think that uncertainty about “who are you,” or “where your algorithm is,” can save it. Granted that CDT is wrong, though, I’m not sure what’s right. The most famous alternative is evidential decision theory (EDT), on which, roughly, you should choose the action you would be happiest to learn you had chosen. I think that EDT is more attractive (and more confusing) than many philosophers give it credit for, and that some putative counterexamples don’t withstand scrutiny. But EDT has problems, too. In particular, I suspect that attractive versions of EDT (and perhaps, attractive attempts to recapture the spirit of CDT) require something in the vicinity of “following the policy that you would’ve wanted yourself to commit to, from some epistemic position that ‘forgets’ information you now know.” I don’t think that the most immediate objection to this – namely, that it implies choosing lower pay-offs even when you know them with certainty – is decisive (though some debates in this vicinity seem to me verbal). But it also seems extremely unclear what epistemic position you should evaluate policies from, and what policy such a position actually implies. Overall, rejecting the common-sense comforts of CDT, and accepting the possibility of some kind of “acausal control,” leaves us in strange and uncertain territory. I think we should do it anyway. But we should also tread carefully. I. Grandpappy Omega Decision theorists often assume that instrumental rationality is about maximizing expected utility in some sense. The question is: what sense? The most famous debate is between CDT and EDT. CDT chooses the action that will have the best effects. EDT chooses the action whose performance would be the best news. More specifically: CDT and EDT disagree about the type of “if” to use when evaluating the utility to expect, if you do X. CDT uses a counterfactual type of “if” — one that holds fixed the probability of everything outside of action X’s causal influence, then plays out the consequences of doing X. In this sense, it doesn’t allow your choice to serve as “evidence” about anything you can’t cause — even when your choice is such evidence. EDT, by contrast, uses a conditional “if.” That is, to evaluate X, it updates your overall picture of the world to reflect the assumption that action X has been been performed, and then sees how good the world looks in expectation. In this sense, it takes all the evidence into account, including the evidence that your having done X would provide. To see what this difference looks like in actio...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Even if you have a nail, not all hammers are the same, published by PhilGoetz on the AI Alignment Forum. (Related to Over-ensapsulation and Subtext is not invariant under linear transformation) Between 2004 and 2007, Goran Bjelakovic et al. published 3 famous meta-analysis of vitamin supplements, concluding that vitamins don't help people but instead kill people. This is now the accepted dogma; and if you ask your doctor about vitamins, she's likely to tell you not to take them, based on reading either one of these articles, or one of the many summaries of these articles made in secondary sources like The Mayo Clinic Journal. The 2007 study claims that beta-carotene and vitamins A and E are positively correlated with death - the more you take, the more likely you are to die. Therefore, vitamins kill. The conclusion on E requires a little explanation, but the data on beta-carotene and A is simple and specific: Univariate meta-regression analyses revealed significant influences of dose of beta carotene (Relative Risk (RR), 1.004; 95% CI, 1.001-1.007; P = .012), dose of vitamin A (RR, 1.000006; 95% CI, 1.000002-1.000009; P = .003), ... on mortality. This appears to mean that, for each mg of beta carotene that you take, your risk of death increases by a factor (RR) of 1.004; for each IU of vitamin A that you take, by a factor of 1.000006. "95% CI, 1.001-1.007" means that the standard deviation of the sample indicates a 95% probability that the true RR lies somewhere between 1.001 and 1.007. "P = .012" means that there's only a 1.2% chance that you would be so unlucky as to get a sample giving that result, if in fact the true RR were 1. A risk factor of 1.000006 doesn't sound like much; but I'm taking 2,500 IU of vitamin A per day. That gives a 1.5% increase in my chance of death! (Per 3.3 years.) And look at those P-values: .012, .003! So why do I still take vitamins? What all of these articles do, in excruciating detail with regard to sample selection (though not so much with regard to the math), is to run a linear regression on a lot of data from studies of patients taking vitamins. A linear regression takes a set of data where each datapoint looks like this: Y = a1X1 + c and a multiple linear regression takes a set of data where each datapoint usually looks like this: Y = a1X1 + a2X2 + ... anXn + c where Y and all the Xi's are known. In this case, Y is a 1 for someone who died and a 0 for someone who didn't, and each Xi is the amount of some vitamin taken. In either case, the regression finds the values for a1, ... an, c that best fit the data (meaning they minimize the sum, over all data points, of the squared error of the value predicted for Y, (Y - (a1X1 + a2X2 + ... anXn + c)2). Scientists love linear regression. It's simple, fast, and mathematically pure. There are lots of tools available to perform it for you. It's a powerful hammer in a scientists' toolbox. But not everything is a nail. And even for a nail, not every hammer is the right hammer. You shouldn't use linear regression just because it's the "default regression analysis". When a paper says they performed "a regression", beware. A linear analysis assumes that if 10 milligrams is good for you, then 100 milligrams is ten times as good for you, and 1000 milligrams is one-hundred times as good for you. This is not how vitamins work. Vitamin A is toxic in doses over 15,000 IU/day, and vitamin E is toxic in doses over 400 IU/day (Miller et al. 2004, Meta-Analysis: High-Dosage Vitamin E Supplementation May Increase All-Cause Mortality; Berson et al. 1993, Randomized trial of vitamin A and vitamin E supplementation for retinitis pigmentosa.). The RDA for vitamin A is 2500 IU/day for adults. Good dosage levels for vitamin A appear to be under 10,000 IU/day, and for E, less than 300 IU/day. (Sadly, studies rarel...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Self-Improvement or Shiny Distraction: Why Less Wrong is anti-Instrumental Rationality, published by patrissimo on the AI Alignment Forum. Introduction Less Wrong is explicitly intended is to help people become more rational. Eliezer has posted that rationality means epistemic rationality (having & updating a correct model of the world), and instrumental rationality (the art of achieving your goals effectively). Both are fundamentally tied to the real world and our performance in it - they are about ability in practice, not theoretical knowledge (except inasmuch as that knowledge helps ability in practice). Unfortunately, I think Less Wrong is a failure at instilling abilities-in-practice, and designed in a way that detracts from people's real-world performance. It will take some time, and it may be unpleasant to hear, but I'm going to try to explain what LW is, why that's bad, and sketch what a tool to actually help people become more rational would look like. (This post was motivated by Anna Salomon's Humans are not automatically strategic and the response, more detailed background in footnote [1].) Update / Clarification in response to some comments: This post is based on the assumption that a) the creators of Less Wrong wish Less Wrong to result in people becoming better at achieving their goals (instrumental rationality, aka "efficient productivity"), and b) Some (perhaps many) readers read it towards that goal. It is this I think is self-deception. I do not dispute that LW can be used in a positive way (read during fun time instead of the NYT or funny pictures on Digg), or that it has positive effects (exposing people to important ideas they might not see elsewhere). I merely dispute that reading fun things on the internet can help people become more instrumentally rational. Additionally, I think instrumental rationality is really important and could be a huge benefit to people's lives (in fact, is by definition!), and so a community value that "deliberate practice towards self-improvement" is more valuable and more important than "reading entertaining ideas on the internet" would be of immense value to LW as a community - while greatly decreasing the importance of LW as a website. Why Less Wrong is not an effective route to increasing rationality. Definition: Work: time spent acting in an instrumentally rational manner, ie forcing your attention towards the tasks you have consciously determined will be the most effective at achieving your consciously chosen goals, rather than allowing your mind to drift to what is shiny and fun. By definition, Work is what (instrumental) rationalists wish to do more of. A corollary is that Work is also what is required in order to increase one's capacity to Work. This must be true by the definition of instrumental rationality - if it's the most efficient way to achieve one's goals, and if one's goal is to increase one's instrumental rationality, doing so is most efficiently done by being instrumentally rational about it. [2] That was almost circular, so to add meat, you'll notice in the definition an embedded assumption that the "hard" part of Work is directing attention - forcing yourself to do what you know you ought to instead of what is fun & easy. (And to a lesser degree, determining your goals and the most effective tasks to achieve them). This assumption may not hold true for everyone, but with the amount of discussion of "Akrasia" on LW, the general drift of writing by smart people about productivity (Paul Graham: Addiction, Distraction, Merlin Mann: Time & Attention), and the common themes in the numerous productivity/self-help books I've read, I think it's fair to say that identifying the goals and tasks that matter and getting yourself to do them is what most humans fundamentally struggle with when it comes to instr...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: More Dakka, published by Zvi on the AI Alignment Forum. Epistemic Status: Hopefully enough Dakka Eliezer Yudkowsky’s book Inadequate Eqilibria is excellent. I recommend reading it, if you haven’t done so. Three recent reviews are Scott Aaronson’s, Robin Hanson’s (which inspired You Have the Right to Think and a great discussion in its comments) and Scott Alexander’s. Alexander’s review was an excellent summary of key points, but like many he found the last part of the book, ascribing much modesty to status and prescribing how to learn when to trust yourself, less convincing. My posts, including Zeroing Out and Leaders of Men have been attempts to extend the last part, offering additional tools. Daniel Speyer offers good concrete suggestions as well. My hope here is to offer both another concrete path to finding such opportunities, and additional justification of the central role of social control (as opposed to object-level concerns) in many modest actions and modesty arguments. Eliezer uses several examples of civilizational inadequacy. Two central examples are the failure of the Bank of Japan and later the European Central Bank to print sufficient amounts of money, and the failure of anyone to try treating seasonal affective disorder with sufficiently intense artificial light. In a MetaMed case, a patient suffered from a disease with a well-known reliable biomarker and a safe treatment. In studies, the treatment improved the biomarker linearly with dosage. Studies observed that sick patients whose biomarkers reached healthy levels experienced full remission. The treatment was fully safe. No one tried increasing the dose enough to reduce the biomarker to healthy levels. If they did, they never reported their results. In his excellent post Sunset at Noon, Raymond points out Gratitude Journals: “Rationalists obviously don’t actually take ideas seriously. Like, take the Gratitude Journal. This is the one peer-reviewed intervention that actually increases your subjective well being, and costs barely anything. And no one I know has even seriously tried it. Do literally none of these people care about their own happiness?” “Huh. Do you keep a gratitude journal?” “Lol. No, obviously.” – Some Guy at the Effective Altruism Summit of 2012 Gratitude journals are awkward interventions, as Raymond found, and we need to find details that make it our own, or it won’t work. But the active ingredient, gratitude, obviously works and is freely available. Remember the last time someone expressed gratitude to you and it made your day worse? Remember the last time you expressed gratitude to someone else, or felt gratitude about someone or something, and it made your day worse? In my experience it happens approximately zero times. Gratitude just works, unmistakably. I once sent a single gratitude letter. It increased my baseline well-being. Then I didn’t write more. I do try to remember to feel gratitude, and express it. That helps. But I can’t think of a good reason not to do that more, or for anyone I know to not do it more. In all four cases, our civilization has (it seems) correctly found the solution. We’ve tested it. It works. The more you do, the better it works. There’s probably a level where side effects would happen, but there’s no sign of them yet. We know the solution. Our bullets work. We just need more. We need More (and better) (metaphorical) Dakka. And then we decide we’re out of bullets. We stop. If it helps but doesn’t solve your problem, perhaps you’re not using enough. I We don’t use enough to find out how much enough would be, or what bad things it might cause. More Dakka might backfire. It also might solve your problem. The Bank of Japan didn’t have enough money. They printed some. It helped a little. They could have kept printing more money until printing more money...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Babble, published by alkjash on the AI Alignment Forum. This is a linkpost for/ This post is an exercise in "identifying with the algorithm." I'm a big fan of the probabilistic method and randomized algorithms, so my biases will show. How do human beings produce knowledge? When we describe rational thought processes, we tend to think of them as essentially deterministic, deliberate, and algorithmic. After some self-examination, however, I've come to think that my process is closer to babbling many random strings and later filtering by a heuristic. I think verbally, and my process for generating knowledge is virtually indistinguishable from my process for generating speech, and also quite similar to my process for generating writing. Here's a simplistic model of how this works. I try to build a coherent sentence. At each step, to pick the next word, I randomly generate words in the category (correct part of speech, relevance) and sound them out one by one to see which continues the sentence most coherently. So, instead of deliberately and carefully generating sentences in one go, the algorithm is something like: Babble. Use a weak and local filter to randomly generate a lot of possibilities. Is the word the right part of speech? Does it lie in the same region of thingspace? Does it fit the context? Prune. Use a strong and global filter to test for the best, or at least a satisfactory, choice. With this word in the blank, do I actually believe this sentence? Does the word have the right connotations? Does the whole thought read smoothly? This is a babble about embracing randomness. Baby Babble Research on language development suggests that baby babble is an direct forerunner to language. You might imagine that infants learn by imitation, and that baby babble is just an imperfect imitation of words the baby hears, and progress occurs as they physiologically adapt to better produce those sounds. You would be wrong. Instead, infants are initially capable of producing all the phonemes that exist in all human languages, and they slowly prune out which ones they need via reinforcement learning. Based on the sounds that their parents produce and respond to, babies slowly filter out unnecessary phonemes. Their babbles begin to drift as they prune out more and more phonemes, and they start to combine syllables into proto-words. Babble is the process of generating random sounds, and looking for clues about which ones are useful. Something something reinforcement learning partially observable Markov decision process I'm in over my head. So, we've learned that babies use the Babble and Prune algorithm to learn language. But this is quite a general algorithm, and evolution is a conservative force. It stands to reason that human beings might learn other things by a similar algorithm. I don't think it's a particularly controversial suggestion that human thought proceeds roughly by cheaply constructing a lot of low-resolution hypotheses and then sieving from them by allowing them to play out to their logical conclusions. The point I want to emphasize is that the algorithm has two distinct phases, both of which can be independently optimized. The stricter and stronger your Prune filter, the higher quality content you stand to produce. But one common bug is related to this: if the quality of your Babble is much lower than that of your Prune, you may end up with nothing to say. Everything you can imagine saying or writing sounds cringey or content-free. Ten minutes after the conversation moves on from that topic, your Babble generator finally returns that witty comeback you were looking for. You'll probably spend your entire evening waiting for an opportunity to force it back in. Your pseudorandom Babble generator can also be optimized, and in two different ways. On the one hand, you can...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Conversational Cultures: Combat vs Nurture (V2), published by Ruby on the AI Alignment Forum. You are viewing Version 2 of this post: a major revision written for the LessWrong 2018 Review. The original version published on 9th November 2018 can be viewed here. See my change notes for major updates between V1 and V2. Combat Culture I went to an orthodox Jewish high school in Australia. For most of my early teenage years, I spent one to three hours each morning debating the true meaning of abstruse phrases of Talmudic Aramaic. The majority of class time was spent sitting opposite your chavrusa (study partner, but linguistically the term has the same root as the word “friend”) arguing vehemently for your interpretation of the arcane words. I didn’t think in terms of probabilities back then, but if I had, I think at any point I should have given roughly even odds to my view vs my chavrusa’s view on most occasions. Yet that didn’t really matter. Whatever your credence, you argued as hard as you could for the view that made sense in your mind, explaining why your adversary/partner/friend’s view was utterly inconsistent with reality. That was the process. Eventually, you’d reach agreement or agree to disagree (which was perfectly legitimate), and then move onto the next passage to decipher. Later, I studied mainstream analytic philosophy at university. There wasn’t the chavrusa, pair-study format, but the culture of debate felt the same to me. Different philosophers would write long papers explaining why philosophers holding opposite views were utterly confused and mistaken for reasons one through fifty. They’d go back and forth, each arguing for their own correctness and the others’ mistakeness with great rigor. I’m still impressed with the rigor and thoroughness of especially good analytic philosophers. I’ll describe this style as combative, or Combat Culture. You have your view, they have their view, and you each work to prove your rightness by defending your view and attacking theirs. Occasionally one side will update, but more commonly you develop or modify your view to meet the criticisms. Overall, the pool of arguments and views develops and as a group you feel like you’ve made progress. While it’s true that you’ll often shake your head at the folly of those who disagree with you, the fact that you’re bothering to discuss with them at all implies a certain minimum of respect and recognition. You don’t write lengthy papers or books to respond to people whose intellect you have no recognition of, people you don’t regard as peers at all. There’s an undertone of countersignalling to healthy Combat Culture. It is because recognition and respect are so strongly assumed between parties that they can be so blunt and direct with each other. If there were any ambiguity about the common knowledge of respect, you couldn’t be blunt without the risk of offending someone. That you are blunt is evidence you do respect someone. This is portrayed clearly in a passage from Daniel’s Ellsberg recent book, The Doomsday Machine: Confessions of a Nuclear War Planner (pp. 35-36): From my academic life, I was used to being in the company of very smart people, but it was apparent from the beginning that this was as smart a bunch of men as I have ever encountered. That first impression never changed (though I was to learn, in the years ahead, the severe limitations of sheer intellect). And it was even better than that. In the middle of the first session, I ventured--though I was the youngest, assigned to taking notes, and obviously a total novice on the issues--to express an opinion. (I don’t remember what it was.) Rather than showing irritation or ignoring my comment, Herman Kahn, brilliant and enormously fat, sitting directly across the table from me, looked at me soberly and said, “You’re a...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Reason isn't magic by Benquo, published by Benquo on the AI Alignment Forum. This is a linkpost for http://benjaminrosshoffman.com/reason-isnt-magic/ Here's a story some people like to tell about the limits of reason. There's this plant, manioc, that grows easily in some places and has a lot of calories in it, so it was a staple for some indigenous South Americans since before the Europeans showed up. Traditional handling of the manioc involved some elaborate time-consuming steps that had no apparent purpose, so when the Portuguese introduced it to Africa, they didn't bother with those steps - just, grow it, cook it, eat it. The problem is that manioc's got cyanide in it, so if you eat too much too often over a lifetime, you get sick, in a way that's not easily traceable to the plant. Somehow, over probably hundreds of years, the people living in manioc's original range figured out a way to leach out the poison, without understanding the underlying chemistry - so if you asked them why they did it that way, they wouldn't necessarily have a good answer. Now a bunch of Africans growing and eating manioc as a staple regularly get cyanide poisoning. This is offered as a cautionary tale against innovating through reason, since there's a lot of information embedded in your culture (via hundreds of years of selection), even if people can't explain why. The problem with this argument is that it's a nonsense comparison. First of all, it's not clear things got worse on net, just that a tradeoff was made. How many person-days per year were freed up by less labor-intensive manioc handling? Has anyone bothered to count the hours lost to laborious traditional manioc-processing, to compare them with the burden of consuming too much cyanide? How many of us, knowing that convenience foods probably lower our lifespans relative to slow foods, still eat them because they're ... more convenient? How many people didn't starve because manioc was available and would grow where and when other things wouldn't? If this is the best we can do for how poorly reason can perform, reason seems pretty great. Second, we're not actually comparing reason to tradition - we're comparing changing things to not changing things. Change, as we know, is bad. Sometimes we change things anyway - when we think it's worth the price, or the risk. Sometimes, we're wrong. Third, the actually existing Portuguese and Africans involved in this experiment weren't committed rationalists - they were just people trying to get by. It probably doesn't take more than a day's reasoning to figure out which steps in growing manioc are really necessary to get the calories palatably. Are we imagining that someone making a concerted effort to improve their life through reason would just stop there? This is being compared with many generations of trial and error. Is that the standard we want to use? Reasoning isn't worth it unless a day of untrained thinking can outperform hundreds of years of accumulated tradition? It gets worse. This isn't a randomly selected example - it's specifically selected as a case where reason would have a hard time noticing when and how it's making things worse. In this particular case, reason introduced an important problem. But life is full of risks, sometimes in ways that are worse for traditional cultures. Do we really want to say that reasoning isn't the better bet unless it outperforms literally every time, without ever making things locally worse? Even theoretically perfect Bayesian rationality will sometimes recommend changes that have an expected benefit, but turn out to be harmful. Not even tradition meets this standard! Only logical certainties do - provided, that is, we haven't made an error in one of our proofs. We also have to count all the deaths and other problems averted by reasoning about ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Most Prisoner's Dilemmas are Stag Hunts; Most Stag Hunts are Schelling Problems, published by abramdemski on the AI Alignment Forum. I previously claimed that most apparent Prisoner's Dilemmas are actually Stag Hunts. I now claim that they're Schelling Pub in practice. I conclude with some lessons for fighting Moloch. This post turned out especially dense with inferential leaps and unexplained terminology. If you're confused, try to ask in the comments and I'll try to clarify. Some ideas here are due to Tsvi Benson-Tilsen. The title of this post used to be Most Prisoner's Dilemmas are Stag Hunts; Most Stag Hunts are Battle of the Sexes. I'm changing it based on this comment. "Battle of the Sexes" is a game where a male and female (let's say Bob and Alice) want to hang out, but each of them would prefer to engage in gender-stereotyped behavior. For example, Bob wants to go to a football game, and Alice wants to go to a museum. The gender issues are distracting, and although it's the standard, the game isn't that well-known anyway, so sticking to the standard didn't buy me much (in terms of reader understanding). I therefore present to you, the Schelling Pub Game: Two friends would like to meet at the pub. In order to do so, they must make the same selection of pub (making this a Schelling-point game). However, they have different preferences about which pub to meet at. For example: Alice and Bob would both like to go to a pub this evening. There are two pubs: the Xavier, and the Yggdrasil. Alice likes the Xavier twice as much as the Yggdrasil. Bob likes the Yggdrasil twice as much as the Xavier. However, Alice and Bob also prefer to be with each other. Let's say they like being together ten times as much as they like being apart. Schelling Pub Game payoff matrix payoffs written alice;bob B's choice X Y A's choice X 20;10 2;2 Y 1;1 10;20 The important features of this game are: The Nash equilibria are all Pareto-optimal. There is no "individually rational agents work against each other" problem, like in prisoner's dilemma or even stag hunt. There are multiple equilibria, and different agents prefer different equilibria. Thus, realistically, agents may not end up in equilibrium at all -- because (in the single-shot game) they don't know which to choose, and because (in an iterated version of the game) they may make locally sub-optimal choices in order to influence the long-run behavior of other players. (Edited to add, based on comments:) Here's a summary of the central argument which, despite the lack of pictures, may be easier to understand. Most Prisoner's Dilemmas are actually iterated. Iterated games are a whole different game with a different action space (because you can react to history), a different payoff matrix (because you care about future payoffs, not just the present), and a different set of equilibria. It is characteristic of PD that players are incentivised to play away from the Pareto frontier; IE, no Pareto-optimal point is an equilibrium. This is not the case with iterated PD. It is characteristic of Stag Hunt that there is a Pareto-optimal equilibrium, but there is also another equilibrium which is far from optimal. This is also the case with iterated PD. So iterated PD resembles Stag Hunt. However, it is furthermore true of iterated PD that there are multiple different Pareto-optimal equilibria, which benefit different players more or less. Also, if players don't successfully coordinate on one of these equilibria, they can end up in a worse overall state (such as mutual defection forever, due to playing grim-trigger strategies with mutually incompatible demands). This makes iterated PD resemble the Schelling Pub Game. In fact, the Folk Theorem suggests that most iterated games will resemble the Schelling Pub Game in this way. In a comment on The Sc...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Developmental Stages of GPTs , published by orthonormal on the AI Alignment Forum. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. Epistemic Status: I only know as much as anyone else in my reference class (I build ML models, I can grok the GPT papers, and I don't work for OpenAI or a similar lab). But I think my thesis is original. Related: Gwern on GPT-3 For the last several years, I've gone around saying that I'm worried about transformative AI, an AI capable of making an Industrial Revolution sized impact (the concept is agnostic on whether it has to be AGI or self-improving), because I think we might be one or two cognitive breakthroughs away from building one. GPT-3 has made me move up my timelines, because it makes me think we might need zero more cognitive breakthroughs, just more refinement / efficiency / computing power: basically, GPT-6 or GPT-7 might do it. My reason for thinking this is comparing GPT-3 to GPT-2, and reflecting on what the differences say about the "missing pieces" for transformative AI. My Thesis: The difference between GPT-2 and GPT-3 has made me suspect that there's a legitimate comparison to be made between the scale of a network architecture like the GPTs, and some analogue of "developmental stages" of the resulting network. Furthermore, it's plausible to me that the functions needed to be a transformative AI are covered by a moderate number of such developmental stages, without requiring additional structure. Thus GPT-N would be a transformative AI, for some not-too-large N, and we need to redouble our efforts on ways to align such AIs. The thesis doesn't strongly imply that we'll reach transformative AI via GPT-N especially soon; I have wide uncertainty, even given the thesis, about how large we should expect N to be, and whether the scaling of training and of computation slows down progress before then. But it's also plausible to me now that the timeline is only a few years, and that no fundamentally different approach will succeed before then. And that scares me. Architecture and Scaling GPT, GPT-2, and GPT-3 use nearly the same architecture; each paper says as much, with a sentence or two about minor improvements to the individual transformers. Model size (and the amount of training computation) is really the only difference. GPT took 1 petaflop/s-day to train 117M parameters, GPT-2 took 10 petaflop/s-days to train 1.5B parameters, and the largest version of GPT-3 took 3,000 petaflop/s-days to train 175B parameters. By contrast, AlphaStar seems to have taken about 30,000 petaflop/s-days of training in mid-2019, so the pace of AI research computing power projects that there should be about 10x that today. The upshot is that OpenAI may not be able to afford it, but if Google really wanted to make GPT-4 this year, they could afford to do so. Analogues to Developmental Stages There are all sorts of (more or less well-defined) developmental stages for human beings: image tracking, object permanence, vocabulary and grammar, theory of mind, size and volume, emotional awareness, executive functioning, et cetera. I was first reminded of developmental stages a few years ago, when I saw the layers of abstraction generated in this feature visualization tool for GoogLeNet. We don't have feature visualization for language models, but we do have generative outputs. And as you scale up an architecture like GPT, you see higher levels of abstraction. Grammar gets mastered, then content (removing absurd but grammatical responses), then tone (first rough genre, then spookily accurate authorial voice). Topic coherence is mastered first on the phrase level, then the sentence level, then the paragraph level. So too with narrative flow. Gwern's poetry experiments (GPT-2, GPT-3) are good examples. GPT-2 could more ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My computational framework for the brain , published by Steven Byrnes on the AI Alignment Forum. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. By now I've written a bunch of blog posts on brain architecture and algorithms, not in any particular order and generally interspersed with long digressions into Artificial General Intelligence. Here I want to summarize my key ideas in one place, to create a slightly better entry point, and something I can refer back to in certain future posts that I'm planning. If you've read every single one of my previous posts (hi mom!), there's not much new here. In this post, I'm trying to paint a picture. I'm not really trying to justify it, let alone prove it. The justification ultimately has to be: All the pieces are biologically, computationally, and evolutionarily plausible, and the pieces work together to explain absolutely everything known about human psychology and neuroscience. (I believe it! Try me!) Needless to say, I could be wrong in both the big picture and the details (or missing big things). If so, writing this out will hopefully make my wrongness easier to discover! Pretty much everything I say here and its opposite can be found in the cognitive neuroscience literature. (It's a controversial field!) I make no pretense to originality (with one exception noted below), but can't be bothered to put in actual references. My previous posts have a bit more background, or just ask me if you're interested. :-P So let's start in on the 7 guiding principles for how I think about the brain: 1. Two subsystems: "Neocortex" and "Subcortex" This is the starting point. I think it's absolutely critical. The brain consists of two subsystems. The neocortex is the home of "human intelligence" as we would recognize it—our beliefs, goals, ability to plan and learn and understand, every aspect of our conscious awareness, etc. etc. (All mammals have a neocortex; birds and lizards have an homologous and functionally-equivalent structure called the "pallium".) Some other parts of the brain (hippocampus, parts of the thalamus & basal ganglia & cerebellum—see further discussion here) help the neocortex do its calculations, and I lump them into the "neocortex subsystem". I'll use the term subcortex for the rest of the brain (brainstem, hypothalamus, etc.). Aside: Is this the triune brain theory? No. Triune brain theory is, from what I gather, a collection of ideas about brain evolution and function, most of which are wrong. One aspect of triune brain theory is putting a lot of emphasis on the distinction between neocortical calculations and subcortical calculations. I like that part. I'm keeping that part, and I'm improving it by expanding the neocortex club to also include the thalamus, hippocampus, lizard pallium, etc., and then I'm ignoring everything else about triune brain theory. 2. Cortical uniformity I claim that the neocortex is, to a first approximation, architecturally uniform, i.e. all parts of it are running the same generic learning algorithm in a massively-parallelized way. The two caveats to cortical uniformity (spelled out in more detail at that link) are: There are sorta "hyperparameters" on the generic learning algorithm which are set differently in different parts of the neocortex—for example, different regions have different densities of each neuron type, different thresholds for making new connections (which also depend on age), etc. This is not at all surprising; all learning algorithms inevitably have tradeoffs whose optimal settings depend on the domain that they're learning (no free lunch). As one of many examples of how even "generic" learning algorithms benefit from domain-specific hyperparameters, if you've seen a pattern "A then B then C" recur 10 times in a row, you will start un...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Science in a High-Dimensional World, published by johnswentworth on the AI Alignment Forum. Claim: the usual explanation of the Scientific Method is missing some key pieces about how to make science work well in a high-dimensional world (e.g. our world). Updating our picture of science to account for the challenges of dimensionality gives a different model for how to do science and how to recognize high-value research. This post will sketch out that model, and explain what problems it solves. The Dimensionality Problem Imagine that we are early scientists, investigating the mechanics of a sled sliding down a slope. What determines how fast the sled goes? Any number of factors could conceivably matter: angle of the hill, weight and shape and material of the sled, blessings or curses laid upon the sled or the hill, the weather, wetness, phase of the moon, latitude and/or longitude and/or altitude, etc. For all the early scientists know, there may be some deep mathematical structure to the world which links the sled’s speed to the astrological motions of stars and planets, or the flaps of the wings of butterflies across the ocean, or vibrations from the feet of foxes running through the woods. Takeaway: there are literally billions of variables which could influence the speed of a sled on a hill, as far as an early scientist knows. So, the early scientists try to control as much as they can. They use a standardized sled, with standardized weights, on a flat smooth piece of wood treated in a standardized manner, at a standardized angle. Playing around, they find that they need to carefully control a dozen different variables to get reproducible results. With those dozen pieces carefully kept the same every time. the sled consistently reaches the same speed (within reasonable precision). At first glance, this does not sound very useful. They had to exercise unrealistic levels of standardization and control over a dozen different variables. Presumably their results will not generalize to real sleds on real hills in the wild. But stop for a moment to consider the implications of the result. A consistent sled-speed can be achieved while controlling only a dozen variables. Out of literally billions. Planetary motions? Irrelevant, after controlling for those dozen variables. Flaps of butterfly wings on the other side of the ocean? Irrelevant, after controlling for those dozen variables. Vibrations from foxes’ feet? Irrelevant, after controlling for those dozen variables. The amazing power of achieving a consistent sled-speed is not that other sleds on other hills will reach the same predictable speed. Rather, it’s knowing which variables are needed to predict the sled’s speed. Hopefully, those same variables will be sufficient to determine the speeds of other sleds on other hills - even if some experimentation is required to find the speed for any particular variable-combination. Determinism How can we know that all other variables in the universe are irrelevant after controlling for a handful? Couldn’t there always be some other variable which is relevant, no matter what empirical results we see? The key to answering that question is determinism. If the system’s behavior can be predicted perfectly, then there is no mystery left to explain, no information left which some unknown variable could provide. Mathematically, information theorists use the mutual information I X Y to measure the information which X contains about Y . If Y is deterministic - i.e. we can predict Y perfectly - then I X Y is zero no matter what variable X we look at. Or, in terms of correlations: a deterministic variable always has zero correlation with everything else. If we can perfectly predict Y , then there is no further information to gain about it. In this case, we’re saying that sled speed is deter...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Redwood Research’s current project , published by Buck on the AI Alignment Forum. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. Here’s a description of the project Redwood Research is working on at the moment. First I’ll say roughly what we’re doing, and then I’ll try to explain why I think this is a reasonable applied alignment project, and then I’ll talk a bit about the takeaways I’ve had from the project so far. There are a bunch of parts of this that we’re unsure of and figuring out as we go; I’ll try to highlight our most important confusions as they come up. I’ve mentioned a bunch of kind of in-the-weeds details because I think they add flavor. This is definitely just me describing a work in progress, rather than presenting any results. Thanks to everyone who’s contributed to the project so far: the full-time Redwood technical team of me, Nate Thomas, Daniel Ziegler, Seraphina Nix, Ben Weinstein-Raun, Adam Scherlis; other technical contributors Daniel de Haas, Shauna Kravec, Tao Lin, Noa Nabeshima, Peter Schmidt-Nielsen; our labellers, particularly Kristen Hall, Charles Warth, Jess Thomson, and Liam Clarke; and for particularly useful advice Mark Xu, Ajeya Cotra, and Beth Barnes. Thanks to Paul Christiano for suggesting a project along these lines and giving lots of helpful advice. Thanks to Adam Scherlis and Nate Soares for writing versions of this doc. And thanks to Bill Zito and other contributors to Redwood ops. Apologies to the people I’ve overlooked. We started this project at the start of August. What we’re doing We’re trying to take a language model that has been fine-tuned on completing fiction, and then modify it so that it never continues a snippet in a way that involves describing someone getting injured (with a caveat I’ll mention later). And we want to do this without sacrificing much quality: if you use both the filtered model and the original model to generate a completion for a prompt, humans should judge the filtered model’s completion as better (more coherent, reasonable, thematically appropriate, and so on) at least about half the time. (This “better almost 50% of the time” property is one way of trying to operationalize “we don’t want the filtered policy to be worse”. It so happens that this property is actually kind of badly behaved, but in our case it seems fine, given that we’re always going to be comparing against a fixed unfiltered distribution.) We’re doing this project in two steps: Step 1: train a classifier, generate by sampling with rejection In step 1 (which we’re currently doing), instead of training a single filtered generator model, we’re just training a classifier that takes a prompt and completion and predicts whether a human would say that the completion involved someone getting injured. You can use such a classifier to make a filtered generation process, by repeatedly generating completions until we find one that the classifier thinks is above some threshold of P(safe). You can play with this filtered generation process here. This interface lets you provide a prompt, and then you can see all of the generated completions and the classifier’s rating of each. It currently is set to use “10% chance of injury” as the decision boundary (it is extremely uncalibrated; this corresponds to a much lower actual chance of injury). Our first goal is to train a classifier that’s good enough that no-one is able to find prompts on which the above process has a noticeable probability of generating an injurious completion. This model was produced by fine-tuning DeBERTa XL on a dataset produced by contractors labeling a bunch of LM-generated completions to snippets of fanfiction that were selected by various heuristics to have a high probability of being completed violently. You can read the instructio...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Make your training useful, published by AnnaSalamon on the AI Alignment Forum. As Tom slips on the ice puddle, his arm automatically pulls back to slap the ground. He’s been taking Jiu-Jitsu for only a month, but, already, he’s practiced falling hundreds of times. Tom’s training keeps him from getting hurt. By contrast, Sandra is in her second year of university mathematics. She got an “A” in calculus and in several more advanced courses, and she can easily recite that “derivatives” are “rates of change”. But when she goes on her afternoon walk and stares at the local businesses, she doesn’t see derivatives. For many of us, rationality is more like Sandra’s calculus than Tom’s martial arts. You may think “overconfidence” when you hear an explicit probability (“It’s 99% likely I’ll make it to Boston on Tuesday”). But when no probability is mentioned -- or, worse, when you act on a belief without noticing that belief at all -- your training has little impact. Learn error patterns ahead of time If you want to notice errors while you’re making them, think ahead of time about what your errors might look like. List the circumstances in which to watch out and the alternative action to try then. Here's an example of what your lists might look like. A bunch of visiting fellows generated this list at one of our rationality trainings last summer; I’m including their list here (with some edits) because I found the specific suggestions useful, and because you may be able to use it as a model for your own lists. Action ideas, for three related biases: A. How does it help to know about overconfidence[1]? What can you do differently, once you know your impressions are unreliable? Action ideas: Try many things, including things you “know” won’t work. Try cheap ones. Don’t be so sure you can’t do things. Don’t be so sure that the things you are doing, are working: If a given “necessary” task is using a large portion of your week, test what happens if you skip that task. Ask others whether your efforts are working, and what you might try instead. Test their suggestions. Ask how you’ll know if you hit your goal: what specific observables will be different? (Not “I’ll know calculus” but “I’ll be able to solve all the problems on the AP calculus test”. Not “I’ll be happier” but “I’ll improve my score on the Beck Depression Inventory”). Track these observables. Be suspicious of received wisdom, since others are also overconfident. But don’t just ignore that wisdom in favor of your own error-prone impressions -- look for empirical tests.[2] Your friends and family are weirder (more unlike your models) than you think they are. Try to notice how. B. How does it help to know about the conjunction fallacy? What can you do differently, once you know specific stories are less likely than we generally expect? Action ideas: Use simple or disjunctive plans: Choose a (city/college/etc.) in which there are many promising possibilities, not one with a single, highly promising scenario.[3] Apply for many jobs, in many sectors of the economy. Gather re-purposable resources, such as money, rationality, sanity, capable friends, math skill, reading speed, mental and physical fitness. Focus on fundamentals more than on situation-specific techniques. Tell detailed stories when you want to convince someone: Describe specific scenarios to angel investors, potential customers, etc. Visualize specific scenarios, when you want convince the less verbal parts of yourself that your new (exercise plan / whatever) is worth the effort. Don’t put all your caution into safeguarding one particular step. For example, don’t “ensure your start-up will succeed” by focusing only on the programming step, or only on the “where to sell it” step. Brainstorm many ways your plans can go wrong. Realize that conjunction-ridden theories...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Philosophy: A Diseased Discipline , published by lukeprog on the AI Alignment Forum. Part of the sequence: Rationality and Philosophy Eliezer's anti-philosophy post Against Modal Logics was pretty controversial, while my recent pro-philosophy (by LW standards) post and my list of useful mainstream philosophy contributions were massively up-voted. This suggests a significant appreciation for mainstream philosophy on Less Wrong - not surprising, since Less Wrong covers so many philosophical topics. If you followed the recent very long debate between Eliezer and I over the value of mainstream philosophy, you may have gotten the impression that Eliezer and I strongly diverge on the subject. But I suspect I agree more with Eliezer on the value of mainstream philosophy than I do with many Less Wrong readers - perhaps most. That might sound odd coming from someone who writes a philosophy blog and spends most of his spare time doing philosophy, so let me explain myself. (Warning: broad generalizations ahead! There are exceptions.) Failed methods Large swaths of philosophy (e.g. continental and postmodern philosophy) often don't even try to be clear, rigorous, or scientifically respectable. This is philosophy of the "Uncle Joe's musings on the meaning of life" sort, except that it's dressed up in big words and long footnotes. You will occasionally stumble upon an argument, but it falls prey to magical categories and language confusions and non-natural hypotheses. You may also stumble upon science or math, but they are used to 'prove' things irrelevant to the actual scientific data or the equations used. Analytic philosophy is clearer, more rigorous, and better with math and science, but only does a slightly better job of avoiding magical categories, language confusions, and non-natural hypotheses. Moreover, its central tool is intuition, and this displays a near-total ignorance of how brains work. As Michael Vassar observes, philosophers are "spectacularly bad" at understanding that their intuitions are generated by cognitive algorithms. A diseased discipline What about Quinean naturalists? Many of them at least understand the basics: that things are made of atoms, that many questions don't need to be answered but instead dissolved, that the brain is not an a priori truth factory, that intuitions come from cognitive algorithms, that humans are loaded with bias, that language is full of tricks, and that justification rests in the lens that can see its flaws. Some of them are even Bayesians. Like I said, a few naturalistic philosophers are doing some useful work. But the signal-to-noise ratio is much lower even in naturalistic philosophy than it is in, say, behavioral economics or cognitive neuroscience or artificial intelligence or statistics. Why? Here are some hypotheses, based on my thousands of hours in the literature: Many philosophers have been infected (often by later Wittgenstein) with the idea that philosophy is supposed to be useless. If it's useful, then it's science or math or something else, but not philosophy. Michael Bishop says a common complaint from his colleagues about his 2004 book is that it is too useful. Most philosophers don't understand the basics, so naturalists spend much of their time coming up with new ways to argue that people are made of atoms and intuitions don't trump science. They fight beside the poor atheistic philosophers who keep coming up with new ways to argue that the universe was not created by someone's invisible magical friend. Philosophy has grown into an abnormally backward-looking discipline. Scientists like to put their work in the context of what old dead guys said, too, but philosophers have a real fetish for it. Even naturalists spend a fair amount of time re-interpreting Hume and Dewey yet again. Because they were trained in ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: You Are A Brain , published by Liron on the AI Alignment Forum. Here is a 2-hour slide presentation I made for college students and teens: You Are A Brain It's an introduction to realist thinking, a tour of all the good stuff people don't realize until they include a node for their brain's map in their brain's map. All the concepts come from Eliezer's posts on Overcoming Bias. I presented this to my old youth group while staffing one of their events. In addition to the slide show, I had a browser with various optical illusions open in tabs, and I brought in a bunch of lemons and miracle fruit tablets. They had a good time and stayed engaged. I hope the slides will be of use to others trying to promote the public understanding of rationality. Note: When you view the presentation, make sure you can see the speaker notes. They capture the gist of what I was saying while I was showing each slide. Added 6 years later: I finally made a video of myself presenting this, except this time it was an adult audience. See this discussion post. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Amish, and Strategic Norms around Technology , published by Raemon on the AI Alignment Forum. I was reading Legal Systems Very Different From Ours by David Friedman. The chapter on the Amish made a couple interesting claims, which changed my conception of that culture (although I'm not very confident that the Amish would endorse these claims as fair descriptions). Strategic Norms Around Technology The Amish relationship to technology is not "stick to technology from the 1800s", but rather "carefully think about how technology will affect your culture, and only include technology that does what you want." So, electric heaters are fine. Central heating in a building is not. This is because if there's a space-heater in the living room, this encourages the family to congregate together. Whereas if everyone has heating in their room, they're more likely to spend time apart from each other. Some communities allow tractors, but only if they don't have rubber tires. This makes them good for tilling fields but bad for driving around. Cars and telephones are particularly important not to allow, because easy transportation and communication creates a slippery slope to full-connection to the outside world. And a lot of the Amish lifestyle depends on cutting themselves off from the various pressures and incentives present in the rest of the world. Some Amish communities allow people to borrow telephones or cars from non-Amish neighbors. I might have considered this hypocritical. But in the context of "strategic norms of technology", it need not be. The important bit is to add friction to transportation and communication. Competitive Dictatorship Officially, most Amish congregations operate via something-like-consensus (I'm not sure I understood this). But Friedman's claim is that in practice, most people tend to go with what the local bishop says. This makes a bishop something like a dictator. But, there are lots of Amish communities, and if you don't like the direction a bishop is pushing people in, or how they are resolving disputes, you can leave. There is a spectrum of communities ranging in how strict they are about about various rules, and they make decisions mostly independently. So there is not only strategic norms around technology, but a fairly interesting, semi-systematic exploration of those norms. Other Applications I wouldn't want to be Amish-in-particular, but the setup here is very interesting to me. I know some people who went to MAPLE, a monastery program. While there, there were limits on technology that meant, after 9pm, you basically had two choices: read, or go to bed. The choices were strongly reinforced by the social and physical environment. And this made it much easier to make choices they endorsed. Contrast this with my current house, where a) you face basically infinite choices about to spend your time, and b) in practice, the nightly choices often end up being something like "stay up till 1am playing minecraft with housemates" or "stay up till 2am playing minecraft with housemates." I'm interested in the question "okay, so... my goals are not the Amish goals. But, what are my goals exactly, and is there enough consensus around particular goals to make valid choices around norms and technology other than 'anything goes?'" There are issues you face that make this hard, though: Competition with the Outside World – The Amish system works because it cuts itself off from the outside world, and its most important technological choices directly cause that. Your business can't get outcompeted by someone else who opens up their shop on Sundays because there is nobody who opens their shop on Sundays. You also might have goals that directly involve the outside world. (The Amish also have good relationships with the government such that they can get away with ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Paper-Reading for Gears , published by johnswentworth on the AI Alignment Forum. Lesswrong has a fair bit of advice on how to evaluate the claims made in scientific papers. Most of this advice seems to focus on a single-shot use case - e.g. a paper claims that taking hydroxyhypotheticol reduces the risk of malignant examplitis, and we want to know how much confidence to put on the claim. It’s very black-box-y: there’s a claim that if you put X (hydroxyhypotheticol) into the black box (a human/mouse) then Y (reduced malignant examplitis) will come out. Most of the advice I see on evaluating such claims is focused around statistics, incentives, and replication - good general-purpose epistemic tools which can be applied to black-box questions. But for me, this black-box-y use case doesn’t really reflect what I’m usually looking for when I read scientific papers. My goal is usually not to evaluate a single black-box claim in isolation, but rather to build a gears-level model of the system in question. I care about whether hydroxyhypotheticol reduces malignant examplitis only to the extent that it might tell me something about the internal workings of the system. I’m not here to get a quick win by noticing an underutilized dietary supplement; I’m here for the long game, and that means making the investment to understand the system. With that in mind, this post contains a handful of thoughts on building gears-level models from papers. Of course, general-purpose epistemic tools (statistics, incentives, etc) are still relevant - a study which is simply wrong is unlikely to be much use for anything. So the thoughts and advice below all assume general-purpose epistemic hygiene as a baseline - they are things which seem more/less important when building gears-level models, relative to their importance for black-box claims. I’m also curious to hear other peoples’ thoughts/advice on paper reading specifically to build gears-level models. Get Away From the Goal Ultimately, we want a magic bullet to cure examplitis. But the closer a paper is to that goal, the stronger publication bias and other memetic distortions will be. A flashy, exciting result picked up by journalists will get a lot more eyeballs than a failed replication attempt. But what about a study examining the details of the interaction between FOXO, SIRT6, and WNT-family signalling molecules? That paper will not ever make the news circuit - laypeople have no idea what those molecules are or why they’re interesting. There isn’t really a “negative result” in that kind of study - there’s just an open question: “do these things interact, and how?”. Any result is interesting and likely to be published, even though you won’t hear about it on CNN. In general, as we move more toward boring internal gear details that the outside world doesn’t really care about, we don’t need to worry as much about incentives - or at least not the same kinds of incentives. Zombie Theories Few people want to start a fight with others in their field, even when those others are wrong. There is little incentive to falsify the theory of somebody who may review your future papers or show up to your talk at a conference. It’s much easier to say “examplitis is a complex multifactorial disease and all these different lines of research are valuable and important, kumbayah”. The result is zombie theories: theories which are pretty obviously false if you spend an hour looking at the available evidence, but which are still repeated in background sections and review articles. One particularly egregious example I’ve seen is the idea that a shift in the collagen:elastin ratio is (at least partially) responsible for the increased stiffness of blood vessels in old age. You can find this theory in review articles and even textbooks. It’s a nice theory: new elastin...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 2019 AI Alignment Literature Review and Charity Comparison, published by Larks on the AI Alignment Forum. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. Cross-posted to the EA forum here. Introduction As in 2016, 2017 and 2018, I have attempted to review the research that has been produced by various organisations working on AI safety, to help potential donors gain a better understanding of the landscape. This is a similar role to that which GiveWell performs for global health charities, and somewhat similar to a securities analyst with regards to possible investments. My aim is basically to judge the output of each organisation in 2019 and compare it to their budget. This should give a sense of the organisations' average cost-effectiveness. We can also compare their financial reserves to their 2019 budgets to get a sense of urgency. I’d like to apologize in advance to everyone doing useful AI Safety work whose contributions I may have overlooked or misconstrued. As ever I am painfully aware of the various corners I have had to cut due to time constraints from my job, as well as being distracted by 1) another existential risk capital allocation project, 2) the miracle of life and 3) computer games. How to read this document This document is fairly extensive, and some parts (particularly the methodology section) are the same as last year, so I don’t recommend reading from start to finish. Instead, I recommend navigating to the sections of most interest to you. If you are interested in a specific research organisation, you can use the table of contents to navigate to the appropriate section. You might then also want to Ctrl+F for the organisation acronym in case they are mentioned elsewhere as well. If you are interested in a specific topic, I have added a tag to each paper, so you can Ctrl+F for a tag to find associated work. The tags were chosen somewhat informally so you might want to search more than one, especially as a piece might seem to fit in multiple categories. Here are the un-scientifically-chosen hashtags: Agent Foundations AI_Theory Amplification Careers CIRL Decision_Theory Ethical_Theory Forecasting Introduction Misc ML_safety Other_Xrisk Overview Philosophy Politics RL Security Shortterm Strategy New to Artificial Intelligence as an existential risk? If you are new to the idea of General Artificial Intelligence as presenting a major risk to the survival of human value, I recommend this Vox piece by Kelsey Piper. If you are already convinced and are interested in contributing technically, I recommend this piece by Jacob Steinheart, as unlike this document Jacob covers pre-2019 research and organises by topic, not organisation. Research Organisations FHI: The Future of Humanity Institute FHI is an Oxford-based Existential Risk Research organisation founded in 2005 by Nick Bostrom. They are affiliated with Oxford University. They cover a wide variety of existential risks, including artificial intelligence, and do political outreach. Their research can be found here. Their research is more varied than MIRI's, including strategic work, work directly addressing the value-learning problem, and corrigibility work. In the past I have been very impressed with their work. Research Drexler's Reframing Superintelligence: Comprehensive AI Services as General Intelligence is a massive document arguing that superintelligent AI will be developed for individual discrete services for specific finite tasks, rather than as general-purpose agents. Basically the idea is that it makes more sense for people to develop specialised AIs, so these will happen first, and if/when we build AGI these services can help control it. To some extent this seems to match what is happening - we do have many specialised AIs - but on the other hand there ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Testing The Natural Abstraction Hypothesis: Project Intro, published by johnswentworth on the AI Alignment Forum. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. The natural abstraction hypothesis says that Our physical world abstracts well: for most systems, the information relevant “far away” from the system (in various senses) is much lower-dimensional than the system itself. These low-dimensional summaries are exactly the high-level abstract objects/concepts typically used by humans. These abstractions are “natural”: a wide variety of cognitive architectures will learn to use approximately the same high-level abstract objects/concepts to reason about the world. If true, the natural abstraction hypothesis would dramatically simplify AI and AI alignment in particular. It would mean that a wide variety of cognitive architectures will reliably learn approximately-the-same concepts as humans use, and that these concepts can be precisely and unambiguously specified. Ultimately, the natural abstraction hypothesis is an empirical claim, and will need to be tested empirically. At this point, however, we lack even the tools required to test it. This post is an intro to a project to build those tools and, ultimately, test the natural abstraction hypothesis in the real world. Background & Motivation One of the major conceptual challenges of designing human-aligned AI is the fact that human values are a function of humans’ latent variables: humans care about abstract objects/concepts like trees, cars, or other humans, not about low-level quantum world-states directly. This leads to conceptual problems of defining “what we want” in physical, reductive terms. More generally, it leads to conceptual problems in translating between human concepts and concepts learned by other systems - e.g. ML systems or biological systems. If true, the natural abstraction hypothesis provides a framework for translating between high-level human concepts, low-level physical systems, and high-level concepts used by non-human systems. The foundations of the framework have been sketched out in previous posts. What is Abstraction? introduces the mathematical formulation of the framework and provides several examples. Briefly: the high-dimensional internal details of far-apart subsystems are independent given their low-dimensional “abstract” summaries. For instance, the Lumped Circuit Abstraction abstracts away all the details of molecule positions or wire shapes in an electronic circuit, and represents the circuit as components each summarized by some low-dimensional behavior - like V = IR for a resistor. This works because the low-level molecular motions in a resistor are independent of the low-level molecular motions in some far-off part of the circuit, given the high-level summary. All the rest of the low-level information is “wiped out” by noise in low-level variables “in between” the far-apart components. In the causal graph of some low-level system, X is separated from Y by a bunch of noisy variables Z. For instance, X might be a resistor, Y might be a capacitor, and Z might be the wires (and air) between them. Noise in Z wipes out most of the low-level info about X, so that only a low-dimensional summary f(X) is relevant to predicting the state of Y. Chaos Induces Abstractions explains one major reason why we expect low-level details to be independent (given high-level summaries) for typical physical systems. If I have a bunch of balls bouncing around perfectly elastically in a box, then the total energy, number of balls, and volume of the box are all conserved, but chaos wipes out all other information about the exact positions and velocities of the balls. My “high-level summary” is then the energy, number of balls, and volume of the box; all other low-lev...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Finite Factored Sets, published by Scott Garrabrant on the AI Alignment Forum. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. This is the edited transcript of a talk introducing finite factored sets. For most readers, it will probably be the best starting point for learning about factored sets. Video: (Lightly edited) slides: 1. Short Combinatorics Talk 1m. Some Context Scott: So I want to start with some context. For people who are not already familiar with my work: My main motivation is to reduce existential risk. I try to do this by trying to figure out how to align advanced artificial intelligence. I try to do this by trying to become less confused about intelligence and optimization and agency and various things in that cluster. My main strategy here is to develop a theory of agents that are embedded in the environment that they're optimizing. I think there are a lot of open hard problems around doing this. This leads me to do a bunch of weird math and philosophy. This talk is going to be an example of some weird math and philosophy. For people who are already familiar with my work, I just want to say that according to my personal aesthetics, the subject of this talk is about as exciting as Logical Induction, which is to say I'm really excited about it. And I'm really excited about this audience; I'm excited to give this talk right now. 1t. Factoring the Talk This talk can be split into 2 parts: Part 1: a short pure-math combinatorics talk. I suspect that if I were better, I would instead be giving a short pure-math category theory talk; but I'm trained as a combinatorialist, so I'm giving a combinatorics talk upfront. Part 2: a more applied and philosophical main talk. This talk can also be split into 4 parts differentiated by color: Motivation Table of Contents Main Body , and Examples . Combining these gives us 8 parts (some of which are not contiguous): Part 1: Short Talk Part 2: The Main Talk Motivation 1m. Some Context 2m. The Pearlian Paradigm ToC 1t. Factoring the Talk 2t. We Can Do Better Body 1b. Set Partitions, etc. 2b. Time and Orthogonality, etc. Examples 1e. Enumerating Factorizations 2e. Game of Life, etc. 1b. Set Partitions All right. Here's some background math: A partition of a set S is a set X of non-empty subsets of S , called parts, such that for each s ∈ S there exists a unique part in X that contains s Basically, a partition of S is a way to view S as a disjoint union. We have parts that are disjoint from each other, and they union together to form S We'll write P a r t S for the set of all partitions of S. We'll say that a partition X is trivial if it has exactly one part. We'll use bracket notation, s X , to denote the unique part in X containing s . So this is like the equivalence class of a given element. And we'll use the notation s ∼ X t to say that two elements s and t are in the same part in X You can also think of partitions as being like variables on your set S . Viewed in that way, the values of a partition X correspond to which part an element is in. Or you can think of X as a question that you could ask about a generic element of S . If I have an element of S and it's hidden from you and you want to ask a question about it, each possible question corresponds to a partition that splits up S according to the different possible answers. We're also going to use the lattice structure of partitions: We'll say that X ≥ S Y X is finer than Y , and Y is coarser than X ) if X makes all of the distinctions that Y makes (and possibly some more distinctions), i.e., if for all s t ∈ S s ∼ X t implies s ∼ Y t . You can break your set S into parts, Y , and then break it into smaller parts, X X ∨ S Y (the common refinement of X and Y ) is the coarsest partition that is finer than both X and Y . This is t...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Leave a Line of Retreat , published by Eliezer Yudkowsky on the AI Alignment Forum. When you surround the enemy Always allow them an escape route. They must see that there is An alternative to death. Sun Tzu, The Art of War Don’t raise the pressure, lower the wall. Lois McMaster Bujold, Komarr I recently happened into a conversation with a nonrationalist who had somehow wandered into a local rationalists’ gathering. She had just declared (a) her belief in souls and (b) that she didn’t believe in cryonics because she believed the soul wouldn’t stay with the frozen body. I asked, “But how do you know that?” From the confusion that flashed on her face, it was pretty clear that this question had never occurred to her. I don’t say this in a bad way—she seemed like a nice person without any applied rationality training, just like most of the rest of the human species. Most of the ensuing conversation was on items already covered on Overcoming Bias—if you’re really curious about something, you probably can figure out a good way to test it, try to attain accurate beliefs first and then let your emotions flow from that, that sort of thing. But the conversation reminded me of one notion I haven’t covered here yet: “Make sure,” I suggested to her, “that you visualize what the world would be like if there are no souls, and what you would do about that. Don’t think about all the reasons that it can’t be that way; just accept it as a premise and then visualize the consequences. So that you’ll think, ‘Well, if there are no souls, I can just sign up for cryonics,’ or ‘If there is no God, I can just go on being moral anyway,’ rather than it being too horrifying to face. As a matter of self-respect, you should try to believe the truth no matter how uncomfortable it is, like I said before; but as a matter of human nature, it helps to make a belief less uncomfortable, before you try to evaluate the evidence for it.” The principle behind the technique is simple: as Sun Tzu advises you to do with your enemies, you must do with yourself—leave yourself a line of retreat, so that you will have less trouble retreating. The prospect of losing your job, for example, may seem a lot more scary when you can’t even bear to think about it than after you have calculated exactly how long your savings will last, and checked the job market in your area, and otherwise planned out exactly what to do next. Only then will you be ready to fairly assess the probability of keeping your job in the planned layoffs next month. Be a true coward, and plan out your retreat in detail—visualize every step—preferably before you first come to the battlefield. The hope is that it takes less courage to visualize an uncomfortable state of affairs as a thought experiment, than to consider how likely it is to be true. But then after you do the former, it becomes easier to do the latter. Remember that Bayesianism is precise—even if a scary proposition really should seem unlikely, it’s still important to count up all the evidence, for and against, exactly fairly, to arrive at the rational quantitative probability. Visualizing a scary belief does not mean admitting that you think, deep down, it’s probably true. You can visualize a scary belief on general principles of good mental housekeeping. “The thought you cannot think controls you more than thoughts you speak aloud”—this happens even if the unthinkable thought is false! The leave-a-line-of-retreat technique does require a certain minimum of self-honesty to use correctly. For a start: You must at least be able to admit to yourself which ideas scare you, and which ideas you are attached to. But this is a substantially less difficult test than fairly counting the evidence for an idea that scares you. Does it help if I say that I have occasion to use this technique myself? A r...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: When Truth Isn't Enough , published by Scott Alexander on the AI Alignment Forum. Continuation of: The Power of Positivist Thinking Consider this statement: The ultra-rich, who control the majority of our planet's wealth, spend their time at cocktail parties and salons while millions of decent hard-working people starve. A soft positivist would be quite happy with this proposition. If we define "the ultra-rich" as, say, the richest two percent of people, then a quick look at the economic data shows they do control the majority of our planet's wealth. Checking up on the guest lists for cocktail parties and customer data for salons, we find that these two activities are indeed disproportionately enjoyed by the rich, so that part of the statement also seems true enough. And as anyone who's been to India or Africa knows, millions of decent hard-working people do starve, and there's no particular reason to think this isn't happening at the same time as some of these rich people attend their cocktail parties. The positivist scribbles some quick calculations on the back of a napkin and certifies the statement as TRUE. She hands it the Official Positivist Seal of Approval and moves on to her next task. But the truth isn't always enough. Whoever's making this statement has a much deeper agenda than a simple observation on the distribution of wealth and preferred recreational activities of the upper class, one that the reduction doesn't capture. Philosophers like to speak of the denotation and the connotation of a word. Denotations (not to be confused with dennettations, which are much more fun) are simple and reducible. To capture the denotation of "old", we might reduce it to something testable like "over 65". Is Methusaleh old? He's over 65, so yes, he is. End of story. Connotations0 are whatever's left of a word when you subtract the denotation. Is Methusaleh old? How dare you use that word! He's a "senior citizen!" He's "elderly!" He's "in his golden years." Each of these may share the same denotation as "old", but the connotation is quite different. There is, oddly enough, a children's game about connotations and denotations1. It goes something like this: I am intelligent. You are clever. He's an egghead. I am proud. You are arrogant. He's full of himself. I have perseverance. You are stubborn. He is pig-headed. I am patriotic. You're a nationalist. He is jingoistic. Politicians like this game too. Their version goes: I care about the poor. You are pro-welfare. He's a bleeding-heart. I'll protect national security. You'll expand the military. He's a warmonger. I'll slash red tape. You'll decrease bureaucracy. He'll destroy safeguards. I am eloquent. You're a good speaker. He's a demagogue. I support free health care. You support national health care. He supports socialized health care. All three statements in a sentence have the same denotation, but very different connotations. The Connotation Game would probably be good for after-hours parties at the Rationality Dojo2, playing on and on until all three statements in a trio have mentally collapsed together. Let's return to our original statement: "The ultra-rich, who control the majority of our planet's wealth, spend their time at cocktail parties and salons while millions of decent hard-working people starve." The denotation is a certain (true) statement about distribution of wealth and social activities of the rich. The connotation is hard to say exactly, but it's something about how the rich are evil and capitalism is unjust. There is a serious risk here, and that is to start using this statement to build your belief system. Yesterday, I suggested that saying "Islam is a religion of peace" is meaningless but affects you anyway. Place an overly large amount of importance on the "ultra-rich" statement, and it can play b...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Undiscriminating Skepticism, published by Eliezer Yudkowskyon the AI Alignment Forum. Tl;dr: Since it can be cheap and easy to attack everything your tribe doesn't believe, you shouldn't trust the rationality of just anyone who slams astrology and creationism; these beliefs aren't just false, they're also non-tribal among educated audiences. Test what happens when a "skeptic" argues for a non-tribal belief, or argues against a tribal belief, before you decide they're good general rationalists. This post is intended to be reasonably accessible to outside audiences. I don't believe in UFOs. I don't believe in astrology. I don't believe in homeopathy. I don't believe in creationism. I don't believe there were explosives planted in the World Trade Center. I don't believe in haunted houses. I don't believe in perpetual motion machines. I believe that all these beliefs are not only wrong but visibly insane. If you know nothing else about me but this, how much credit should you give me for general rationality? Certainly anyone who was skillful at adding up evidence, considering alternative explanations, and assessing prior probabilities, would end up disbelieving in all of these. But there would also be a simpler explanation for my views, a less rare factor that could explain it: I could just be anti-non-mainstream. I could be in the habit of hanging out in moderately educated circles, and know that astrology and homeopathy are not accepted beliefs of my tribe. Or just perceptually recognize them, on a wordless level, as "sounding weird". And I could mock anything that sounds weird and that my fellow tribesfolk don't believe, much as creationists who hang out with fellow creationists mock evolution for its ludicrous assertion that apes give birth to human beings. You can get cheap credit for rationality by mocking wrong beliefs that everyone in your social circle already believes to be wrong. It wouldn't mean that I have any ability at all to notice a wrong belief that the people around me believe to be right, or vice versa - to further discriminate truth from falsity, beyond the fact that my social circle doesn't already believe in something. Back in the good old days, there was a simple test for this syndrome that would get quite a lot of mileage: You could just ask me what I thought about God. If I treated the idea with deeper respect than I treated astrology, holding it worthy of serious debate even if I said I disbelieved in it, then you knew that I was taking my cues from my social surroundings - that if the people around me treated a belief as high-prestige, high-status, I wouldn't start mocking it no matter what the state of evidence. On the other hand suppose I said without hesitation that my epistemic state on God was similar to my epistemic state on psychic powers: no positive evidence, lots of failed tests, highly unfavorable prior, and if you believe it under those circumstances then something is wrong with your mind. Then you would have heard a bit of skepticism that might cost me something socially, and that not everyone around me would have endorsed, even in educated circles. You would know it wasn't just a cheap way of picking up cheap points. Today the God-test no longer works, because some people realized that the taking-it-seriously aura of religion is in fact the main thing left which prevents people from noticing the epistemic awfulness; there has been a concerted and, I think, well-advised effort to mock religion and strip it of its respectability. The upshot is that there are now quite wide social circles in which God is just another stupid belief that we all know we don't believe in, on the same list with astrology. You could be dealing with an adept rationalist, or you could just be dealing with someone who reads Reddit. And of course I could easil...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Negative and Positive Selection, published by alyssavance on the AI Alignment Forum. (Originally posted to my blog, The Rationalist Conspiracy; cross-posted here on request of Lukeprog.) You’re the captain of a team, and you want to select really good players. How do you do it? One way is through what I call positive selection. You devise a test – say, who can run the fastest – and pick the people who do best. If you want to be really strict, like if you’re selecting for the Olympics, you only pick the top fraction of a percent. If you’re a player, and you want to get selected, you have to train to do better on the test. The opposite method is negative selection. Instead of one test to pick out winners, you design many tests to pick out losers. You test, say, who can’t run very well when it’s hot out, and get rid of them. Then you test who can’t run very well when it’s cold out, and get rid of them. Then you test running in the rain, and get rid of the losers there. And so on and so forth. When you’re strict with negative selection, you have lots and lots of tests, so that it’s very hard for any one person to pass through all the filters. I think a big part of where American society’s gone wrong over the last hundred years is the ubiquitous use of negative selection over positive selection. (Athletics is one of the only exceptions. It’s apparently so important that people really care about performance – as opposed to, say, in medicine, where we exclude brilliant doctors if they don’t have the stamina to work ninety hours a week.) A single test can always be flawed; for example, IQ tests and SATs have many flaws. However, with negative selection, how badly you do is determined by the failure rate of every test combined. If you have twenty tests, and even one of them is so flawed it excludes good players, then your team will suck. Elite college admissions is an example of a negative selection test. There’s no one way you can do really, really well, and thereby be admitted to Harvard. Instead, you have to pass a bunch of different selection filters: Are your SATs good enough? Are your grades good enough? Is your essay good enough? Are your extracurriculars good enough? Are your recommendations good enough? Failure on any one step usually means not getting admitted. And as competition has intensified, colleges have added more and more filters, like the supplemental applications top schools now require (in addition to the Common Application). It wasn’t always this way – Harvard used to admit primarily based on an entrance exam – until they discovered this let too many Jews in (no, seriously). More recently, the negative selection has been intensified by eliminating the SAT’s high ceiling. Academia is another example of negative selection. To get tenure, first you have to get into a top PhD program. Then you have to graduate. Then you have to get a good recommendation from your advisor. Then you have to get a good postdoc. Then you have to get another good postdoc. Then you have to get a good assistant professorship. Then you have to get approved by the tenure committee. For the most part, if even one of those steps goes wrong – if you went to a second-tier PhD program, say – there’s no way to recover. Once you’re off the “track”, you’re off, and there’s no getting back on. It’s fail once, fail forever. Grades are another example – A is a good grade, but there’s no excellent grade. There’s no grade that you only get if you’re in the top 0.1%. Hence, getting a really good GPA doesn’t mean excelling, so much as it means never failing. If you’re in high school and are taking six classes, if you fail one, your GPA is now 3.3 or less, regardless of how good you are otherwise. In any field, at the top end, you tend to get a lot of variance. (Insert tales of the mad artist and ma...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Secrets of the eliminati, published by Scott Alexander on the AI Alignment Forum. Anyone who does not believe mental states are ontologically fundamental - ie anyone who denies the reality of something like a soul - has two choices about where to go next. They can try reducing mental states to smaller components, or they can stop talking about them entirely. In a utility-maximizing AI, mental states can be reduced to smaller components. The AI will have goals, and those goals, upon closer examination, will be lines in a computer program. But in the blue-minimizing robot, its "goal" isn't even a line in its program. There's nothing that looks remotely like a goal in its programming, and goals appear only when you make rough generalizations from its behavior in limited cases. Philosophers are still very much arguing about whether this applies to humans; the two schools call themselves reductionists and eliminativists (with a third school of wishy-washy half-and-half people calling themselves revisionists). Reductionists want to reduce things like goals and preferences to the appropriate neurons in the brain; eliminativists want to prove that humans, like the blue-minimizing robot, don't have anything of the sort until you start looking at high level abstractions. I took a similar tack asking ksvanhorn's question in yesterday's post - how can you get a more accurate picture of what your true preferences are? I said: I don't think there are true preferences. In one situation you have one tendency, in another situation you have another tendency, and "preference" is what it looks like when you try to categorize tendencies. But categorization is a passive and not an active process: if every day of the week I eat dinner at 6, I can generalize to say "I prefer to eat dinner at 6", but it would be non-explanatory to say that a preference toward dinner at 6 caused my behavior on each day. I think the best way to salvage preferences is to consider them as tendencies currently in reflective equilibrium. A more practical example: when people discuss cryonics or anti-aging, the following argument usually comes up in one form or another: if you were in a burning building, you would try pretty hard to get out. Therefore, you must strongly dislike death and want to avoid it. But if you strongly dislike death and want to avoid it, you must be lying when you say you accept death as a natural part of life and think it's crass and selfish to try to cheat the Reaper. And therefore your reluctance to sign up for cryonics violates your own revealed preferences! You must just be trying to signal conformity or something. The problem is that not signing up for cryonics is also a "revealed preference". "You wouldn't sign up for cryonics, which means you don't really fear death so much, so why bother running from a burning building?" is an equally good argument, although no one except maybe Marcus Aurelius would take it seriously. Both these arguments assume that somewhere, deep down, there's a utility function with a single term for "death" in it, and all decisions just call upon this particular level of death or anti-death preference. More explanatory of the way people actually behave is that there's no unified preference for or against death, but rather a set of behaviors. Being in a burning building activates fleeing behavior; contemplating death from old age does not activate cryonics-buying behavior. People guess at their opinions about death by analyzing these behaviors, usually with a bit of signalling thrown in. If they desire consistency - and most people do - maybe they'll change some of their other behaviors to conform to their hypothesized opinion. One more example. I've previously brought up the case of a rationalist who knows there's no such thing as ghosts, but is still uncomforta...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Punctuality - Arriving on Time and Math, published by Xachariah on the AI Alignment Forum. In hindsight, this post seems incredibly obvious. The meat of it already exists in sayings which we all know we ought to listen to: "Always arrive 10 minutes earlier than you think early is," "If you arrive on time, then you're late," or "Better three hours too soon than one minute too late." Yet even with these sayings, I still never trusted them nor arrived on time. I'd miss deadlines, show up late, and just be generally tardy. The reason is that I never truly understood what it took to arrive on time until I grokked the math of it. So, while this may be remedial reading for most of you, I'm posting this because maybe there's someone out there who missed the same obviousness that I missed. Statistical Distributions Everyone here understands that our universe is controlled and explained by math. Math describes how heavenly bodies move. Math describes how our computers run. Math describes how other people act in aggregate. Wait a second, something's not right with that statement... "other people". The way it comes out it's natural to think that math controls the way that other people act, and not myself. Intellectually, I am aware that I am not a special snowflake who is exempt from the laws of math. While I had managed to propagate this thought far enough to crush my belief in libertarian free will, I hadn't propagated it fully through my mind. Specifically, I hadn't realized I could also use math to describe my actions and reap the benefit of understanding them mathematically. I was still late to arrive and missing deadlines, and nothing seemed to help. But wait, I'm a rationalist! I know all about the planning fallacy; I know to take the outside view! That's enough to save me right? Well, not quite. It seemed I missed one last part of the puzzle... Bell Curves. When I go to work every day, the time from when I do nothing but getting ready to go to work until the time that I actually arrive there (I'll just call this prep time) usually takes 45 minutes, but sometimes it can take more time or less time. Weirdly and crazily enough, if you plot all the prep times on a graph, the shape would end up looking roughly like a bell. Well that's funny. Math is for other people, but my behavior appears like it can be described statistically. Some days I will have deviations from the normal routine that help me arrive faster while other days will have things that slow me down. Some of them happen more often, some of them happen less often. If I were describable by math, I could almost call these things standard deviations: days where I have almost zero traffic prep time takes 1 standard deviation less, days when I can't find my car keys my prep time takes 1 standard deviation more, days I realize would be late and skip showering take 2 standard deviations less, and days when there is a terrible accident on the freeway end up requiring +2 or +3 standard deviations more in time. To put it in other words, my prep time is a bell curve, and I've got 1-sigma and 2-sigma (and occasionally 3-sigma) events speeding me up and slowing me down. This holds true for more than just going to work. Everything's time-until-completion can be described this way: project completion times, homework, going to the airport, the duration of foreplay and sex. Everything. It's not always bell curves, but it's a probability distribution with respect to completion times, and that can help give useful insights. Starting 'On Time' Means You Won't be On Time What do we gain by understanding that our actions are described by a probability distribution? The first and most important take away is this: If you only allocate the exact amount of time to do something, you'll be late 50% of the time. I'm going to repeat it and it...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Break your habits: be more empirical, published by Academian on the AI Alignment Forum. tl;dr: The neurotypical attitude that "You think too much" might be better parsed as "You don't experiment enough." Once you have an established procedure for living optimally in «setting», be a good scientist and keep trying to falsify your theory when it's not too costly to do so. (Note: in aspects of life where you're impulsive, don't introspect enough, or have poor self discipline, this post is probably advice in the wrong direction.) Alice is highly analytically minded. She always walks the same most-efficient route to work, only dances tango and salsa, and refuses to deviate even on rare occasions from her carefully planned schedule. She has judged carefully from experience that the expected value of dating is too low to be worth her time, and will only watch a movie if at least 3 of her 5 closest friends recommend it. She travels only when it relates to her job, to ensure the trip has a purpose and to minimize unnecessary transportation costs. Oh, and she also thinks a lot. About everything. Bob often tells Alice that she "thinks too much", advice that rarely if ever resonates. But consider that Bob may be sensing a legitimate imbalance: Alice may be doing too much analysis with not enough data. He can tell she thinks way more than he does, and blames that for the imbalance, suggesting that Alice should "turn off her brain". But Alice can't agree. Why would she ever waste a resource as constantly applicable and available as her mind? That seems like a terrible idea. So here's a better one: Alice, if you're reading this, don't turn your mind off... turn it outward. When (analysis:data) looks too big, just try turning up the data. There's no need to get stupider or anything. When it's not overly costly, you should deviate from your usual theories of optimal behavior for the sake of expected information gain. Even in theory, empiricism is necessary... For a Bayesian optimizing agent in an uncertain world, information has positive expected utility, and experiments have positive expected information. Ergo, do them sometimes! And what sort of experiment do I mean? I mean that once in a while, Alice should dance freestyle. She should leave early and take a scenic route sometimes, and try some new food along the way. She should visit somewhere she's never been for a vacation, and try meeting some locals. And she shouldn't be discouraged when experimental behavior turns out to be "suboptimal as anticipated". That just means she doesn't have to try that particular thing again, at least for a while. The point is the rare occasion when it does work out and you find something valuable, or the less rare occasion that the change of pace is simply inspiring. So try to overcome that deep-rooted sense of suboptimality you get when you consider new things, or revisit old ones. Locally suboptimal behavior can be worth it for the global benefits of the information you gain. I'm not suggesting to take big risks like drug addictions or injuries... if you want a safe idea to start with, think of something you never do but which other people do all the time without ruining themselves. A priori, classical mechanics could have explained pretty much any observation made before 1800. It looked great: every event could be imagined as a series of pushes and pulls acting on sufficiently small bits of matter and the right initial conditions. But we kept testing it anyway, and now we have nuclear power. What might you find in a new situation or environment that you never thought of before? Go do something you wouldn't normally do :) Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Defense Against The Dark Arts: Case Study #1 , published by Scott Alexander on the AI Alignment Forum. Related to: The Power of Positivist Thinking, On Seeking a Shortening of the Way, Crowley on Religious Experience Annoyance wants us to stop talking about fancy techniques and get back to basics. I disagree with the philosophy behind his statement, but the principle is sound. In many areas of life - I'm thinking mostly of sports, but not for lack of alternatives - mastery of the basics beats poorly-grounded fancy techniques every time. One basic of rationality is paying close attention to an argument. Dissecting it to avoid rhetorical tricks, hidden fallacies, and other Dark Arts. I've been working on this for years, and I still fall short on a regular basis. Medical educators have started emphasizing case studies in their curricula. Instead of studying arcane principles of disease, student doctors cooperate to analyze a particular patient in detail, ennumerate the principles needed to diagnose her illness, and pay special attention to any errors the patients' doctors made during the treatment. The cases may be rare tropical infections, but they're more often the same everyday diseases common in the general population, forcing the student doctors to always keep the basics in mind. We could do with a tradition of case studies in rationality, though we'd need safeguards to prevent degeneration into political discussion. Case studies in medicine are most interesting when all the student doctors disagree with each other. To that end, I've chosen as the first case a statement that received sixteen upvotes on Less Wrong, maybe the highest I've ever seen for a comment. I don't mean to insult or embarass everyone who liked it. I liked it too. My cursor was already hovering above the "Vote Up" button by the time I starting having second thoughts. But it deserves dissection, and its popularity gives me a ready response when someone says this material is too basic for 'master rationalists' like ourselves: In his youth, Steve Jobs went to India to be enlightened. After seeing that the nation claiming to be the source of this great spiritual knowledge was full of hunger, ignorance, squalor, poverty, prejudice, and disease, he came back and said that the East should look to the West for enlightenment. This anecdote is short, witty, flattering, and utterly opaque to reason. It bears all the hallmarks of the Dark Arts. I admit I am not a disinterested party here. The statement was in response to my claim that Indian yoga was a successful technique for inducing exotic and occasionally useful mental states. I don't like being told I'm wrong any more than anyone else does. But here I don't think I am. I see at least five fallacies. First, a hidden assumption: if A is superior to B, A cannot learn anything from B. This assumption is clearly false. I know brilliant scientists whose spelling is atrocious. I acknowledge that these people are much smarter than I am, but I still correct their spelling. Anyone who said "Dr. A should not be learning spelling from Yvain, Yvain should be learning science from Dr. A" would be missing the point. If Dr. A wants to learn spelling, he might as well learn it from me. And best of all if we both learn from each other! A related fallacy would be that Dr. A is so much smarter than the rest of us that he should not care about spelling. But if spelling is important to his work (perhaps he's writing a journal article) he needs to do everything he can to perfect it. If he could spell correctly, he would be even further ahead of the rest of us than he already is. The goal isn't to become a bit better than your peers and then rest on your laurels. The goal is to become as skilled as necessary. The error is an interesting variant of the halo effect: that anyone...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Just another day in utopia, published by Stuart_Armstrong on the AI Alignment Forum. (Reposted from discussion at commentator suggestion) Thinking of Eliezer's fun theory and the challenge of creating actual utopias where people would like to live, I tried to write a light utopia for my friends around Christmas, and thought it might be worth sharing. It's a techno-utopia, but (considering my audience) it's only a short inferential distance from normality. Just another day in Utopia Ishtar went to sleep in the arms of her lover Ted, and awoke locked in a safe, in a cargo hold of a triplane spiralling towards a collision with the reconstructed temple of Solomon. Again! Sometimes she wished that a whole week would go by without something like that happening. But then, she had chosen a high excitement existence (not maximal excitement, of course – that was for complete masochists), so she couldn’t complain. She closed her eyes for a moment and let the thrill and the adrenaline warp her limbs and mind, until she felt transformed, yet again, into a demi-goddess of adventure. Drugs couldn’t have that effect on her, she knew; only real danger and challenge could do that. Right. First, the safe. She gave the inner door a firm thud, felt it ring like a bell, heard the echo return – and felt the tumblers move. So, sound controlled lock, then. A search through her shoes produced a small pebble which sparked as she dashed it against the metal. Trying to ignore the ominous vibration as the triplane motor shook itself to pieces, she constructed a mental image of the safe’s inside from the brief flashes of light. Symmetric gold and gilded extravagances festooned her small prison – French Baroque decorations, but not yet Roccoco. So Louis XIV period. She gave the less visited parts of her mind a good dusting, trying to remember the tunes of Jean Batiste Lully, the period’s most influential composer. She hoped it wasn’t any of his ballets; she was much better with his operas. The decorations looked vaguely snake-like; so she guessed Lully’s ‘Persée’ opera, about the death of the medusa. The engine creaked to a worrying silence as she was half-way through humming the Gorgon theme from the opera. Rushing the rest of the composition, she felt the door shift, finally, to a ten-times speeded up version of Andromeda’s response to Perseus’s proposal. She kicked the door open, exploded from the safe, took in the view of the temple of Solomon rushing up towards her, seconds away, grabbed a picture from the floor, grabbed an axe from the wall, hacked off one of the wings with three violent cuts, and jumped out of the plane after it. Behind her, the plane disintegrated in midair as the temple lasers cut it to shreds and she fell through space, buffeted by the wind, not losing her grip on to the mangled wing. She had maybe thirty seconds to tie herself to the wing, using the object’s own canvas as binding, and she rushed through that. The Machines wouldn’t allow the fall to kill her, of course, but it would hurt quite a bit (another of her choices – she’d allowed herself to feel moderate amounts of pain), put back her attempts to ever find Ted, and, most importantly of all, be crushingly embarrassing socially. Once she was lashed to the plummeting piece of wood and canvas, and she was reasonably confident that the fall was slow enough, and her knots secure enough, she finally looked at the photograph she had grabbed during her explosive exit from the plane. It showed Ted, trussed up in chains but smiling and evidently enjoying the novel experience. Underneath was finely engraved note saying “If you ever want to see your lover again, bring me the missing Stradivarius by noon tomorrow. Nero the 2nd”. Each capital letter was beautifully decorated with heads on spikes. So! It seemed that her magnific...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Expressive Vocabulary, published by Alicorn on the AI Alignment Forum. "Thou shalt not strike terms from others' expressive vocabulary without suitable replacement." - me Suppose your friend says: "I don't buy that brand of dip. It's full of chemicals." Reasonable answer: "I'm skeptical that any of them are harmful in these quantities; we don't have much reason to believe that." Reasonable answer: "Yellow 5? Are you allergic?" Reasonable answer: "Okay, let's get the kind with four easily recognizable ingredients." No: "Technically, everything is chemicals. Dihydrogen monoxide!" Pedantry is seldom a way to make friends and influence people, but this example particularly gets my goat because there doesn't seem to actually exist a word in English for the thing you know perfectly well people mean when they say "chemicals". When I tried to find one on Twitter, the closest options were "toxins" and "additives". But neither is right. "Toxins" excludes yellow 5 - or, whether it does or not might be a point of contention; but it isn't the thing originally expressed with the word "chemicals". People may want to avoid - or otherwise discuss - "chemicals" for reasons other than thinking they're literally toxic; if I tell a maid I'm sensitive to chemical smells but vinegar is okay this is useful information. "Additives" includes, say, added sugar, which, while a plausible complaint, is a separate complaint. Suppose your grandma says, "Okay, no technology at the dinner table." Reasonable answer: "I'll put the laptop away to make room for the potatoes, but I need the phone because I get anxious without it." Reasonable answer: "Sure, Grandma." Reasonable answer: "We can try that until Uncle Bill starts making easily falsified claims about Flat Earth." No: "Technically, the dinner table is a technology. And so are your glasses, Grandma." In this case a more precise word exists - "electronics" ambiguously includes the chandelier but at least firmly sets aside the question of whether your grandma wants you to eat naked and with your bare hands. But refusing to know what she meant because she could have gotten closer to saying it, not even literally (she isn't being metaphorical), but technically, pedantically, definitionally? This is both a bad social move and a bad epistemic one; you're having the conversation on a level that is wholly about verbal wallpaper. Do you prefer to say "electronics" or dip into synechdoche with "screens" or spend nine syllables on "internet enabled devices"? Are you actually unsure if your grandmother wants you to set aside your smart watch, dumb phone, or electric blanket of intermediate intellect? Use your own words, ask your own questions, but don't enforce an inadequate prescriptivism with feigned incomprehension while your interlocutor only wants you to pass the peas. Thou shalt not strike terms from others' expressive vocabulary without suitable replacement. It's a pet issue of mine; it's my pinned tweet. "Suitable replacement" means suitable across the board, Pareto improvement as seen by the user along every axis a word can have. I think people are within their rights to reject a proposed replacement for not meaning the right thing, sounding ugly, being one syllable longer, being hard to spell, not rhyming in a poem they're trying to write, and vague gut feeling that you're just trying to control them. I extend this as far as "gypsy" and "Eskimo", at least (and with slightly less fervor to a slur beyond that if you really don't have another term for Brazil nuts). Suitable replacement is a very high standard. It has to be. If you take someone's words away - and refusing to understand them when the problem is not in fact in your understanding does that, since words are tools to communicate - they are very direly crippled. Many people think communicati...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: From Personal to Prison Gangs: Enforcing Prosocial Behavior, published by johnswentworth on the AI Alignment Forum. This post originally appeared here; I've updated it slightly and posted it here as a follow-up to this post. David Friedman has a fascinating book on alternative legal systems. One chapter focuses on prison law - not the nominal rules, but the rules enforced by prisoners themselves. The unofficial legal system of California prisoners is particularly interesting because it underwent a phase change sometime after the 1960’s. Prior to the 1960’s, prisoners ran on a decentralized code of conduct - various unwritten rules roughly amounting to “mind your own business and don’t cheat anyone”. Prisoners who kept to the code were afforded some respect by their fellow inmates. Prisoners who violated the code were ostracized, making them fair game for the more predatory inmates. There was no formal enforcement; the code was essentially a reputation system. Sometime after the 1960’s, that changed. During the code era, California’s total prison population was only about 5000, with about 1000 inmates in a typical prison. That’s quite a bit more than Dunbar’s number, but still low enough for a reputation system to work through second-order connections. By 1970, California’s prison population had ballooned past 25000; today it is over 170000. The number of prisons also grew, but not nearly as quickly as the population, and today’s prisoners frequently move across prisons anyway. In short, a decentralized reputation system became untenable. There were too many other inmates to keep track of. As the reputation system collapsed, a new legal institution grew to fill the void: prison gangs. Under the gang system, each inmate is expected to affiliate with a gang (though most are not formal gang members). The gang will explain the rules, often in written form, and enforce them on their own affiliates. When conflict arises between affiliates of different gangs, the gang leaders negotiate settlement, with gang leaders enforcing punishments on their own affiliates. (Gang leaders are strongly motivated to avoid gang-level conflicts.) Rather than needing to track reputation of everyone individually, inmates need only pay attention to gangs at a group level. Of course, inmates need some way to tell who is affiliated with each gang - thus the rise of racial segregation in prison. During the code era, prisoners tended to associate by race and culture, but there was no overt racial hostility and no hard rules against associating across race. But today’s prison gangs are highly racially segregated, making it easy to recognize the gang affiliation of individual inmates. They claim territory in prisons - showers or ball courts - and enforce their claims, resulting in hard racial segregation. The change from a small, low-connection prison population to a large, high-connection population was the root cause. That change drove a transition from a decentralized, reputation-based system to prison gangs. This, in turn, involved two further transitions. First, a transition from decentralized, informal unwritten rules to formal written rules with centralized enforcement. Second, a transition from individual to group-level identity, in this case manifesting as racial segregation. Generalization This is hardly unique to prisons. The pattern is universal among human institutions. In small groups, everybody knows everybody. Rules are informal, identity is individual. But as groups grow: Rules become formal, written, and centrally enforced Identity becomes group-based. Consider companies. I work at a ten-person company. Everyone in the office knows everyone else by name, and everyone has some idea of what everyone else is working on. We have nominal job titles, but everybody works on whatever needs d...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Can you keep this confidential? How do you know?, published by Raemon on the AI Alignment Forum. Pet peeve about privacy: I think people are woefully inadequate at asking, and answering, "Can you keep this confidential?" Disclosure: I am not inherently great at keeping information private. By default, if a topic came up in conversation, I would accidentally sometimes say my thoughts before I had time to realize "oh, right, this was private information I shouldn't share." I've worked over the past few years to become better at this – I've learned several specific skills and habits that make it easier. But I didn't learn those skills in school, and no one even really suggested I was supposed to learn them. People seemed to just assume "people can keep secrets, and it's low cost for them to do so." And... maybe this is just me. But, people say to me "hey, can you keep this private?", in a tone that implies I'm not really supposed to say no. And that's the best case. I've also observed things like... ...people saying "hey, this is confidential", and then just saying the thing without checking in. ...people saying "sign this NDA", without really checking I have the skills to honor that agreement, and if I were to not sign, I'd... probably get fired? Unclear. ...people gathering for a Circle or other private safe space, and saying (best case) "do we all agree to keep things here confidential? Raise you hand?" and worst case, just flatly asserting "This is a safe space, things are confidential here". (And I have seen at least one instance where someone I actively trusted later betrayed that trust) ...people saying "You can report things to our [org / HR department / point-person], and they will keep things confidential." But, I know that in the hiring process for that org or department, no one ever checked that people actually had privacy skills. And meanwhile, I have almost never heard anyone say something like "I have been given 10 bits of private-info over the past few years, and I accidentally leaked two of them", or even "I have paid any attention at all to how leaky I am with regards to confidential information." What is a secret, even? Meanwhile, people seem to vary in what they even mean by "secret" or "private information." Some people take them as serious oaths, some people just kinda sorta try to keep the R0 of the info lower than 1. Sometimes it seems to mean "carry this information to your grave", and sometimes it means "I dunno keep this on the down-low for awhile until the current controversy blows over." Some people reading this might be surprised this is even a big deal. I gave a lightning-talk version of this blogpost last weekend, and one person asked "does this really matter that much, outside of major company NDAs or state-secrets?" Another person expressed similar skepticism. I think it varies. The problem is exactly that most of the time, secrets aren't that big of a deal. But people don't seem to take time to get on the same page of exactly how big a deal they are, which is a recipe for mismatched expectations. It's a bigger deal for me, because I live in social and professional circles adjacent to EA Grantmaking where line between the personal and professional is (perhaps unfortunately) a bit blurry. Sometimes, I talk to people exploring ideas that are legit infohazardous. Sometimes, people are hesitant to talk because they're worried it may affect their career. It's also important to me from a Robust Agency standpoint – I'd like to be a reliable agent that people can coordinate with in complicated domains. Many other people in the x-risk ecosystem also seem interested in that. I think "the ability to exchange information, or reliably not exchange it" is a key skill, and worth cultivating because it enables higher order strategies. What to do with a...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The theory-practice gap, published by Buck on the AI Alignment Forum. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. [Thanks to Richard Ngo, Damon Binder, Summer Yue, Nate Thomas, Ajeya Cotra, Alex Turner, and other Redwood Research people for helpful comments; thanks Ruby Bloom for formatting this for the Alignment Forum for me.] I'm going to draw a picture, piece by piece. I want to talk about the capability of some different AI systems. You can see here that we've drawn the capability of the system we want to be competitive with, which I’ll call the unaligned benchmark. The unaligned benchmark is what you get if you train a system on the task that will cause the system to be most generally capable. And you have no idea how it's thinking about things, and you can only point this system at some goals and not others. I think that the alignment problem looks different depending on how capable the system you’re trying to align is, and I think there are reasonable arguments for focusing on various different capabilities levels. See here for more of my thoughts on this question. Alignment strategies People have also proposed various alignment strategies. But I don’t think that these alignment strategies are competitive with the unaligned benchmark, even in theory. I want to claim that most of the action in theoretical AI alignment is people proposing various ways of getting around these problems by having your systems do things that are human understandable instead of doing things that are justified by working well. For example, the hope with imitative IDA is that through its recursive structure you can build a dataset of increasingly competent answers to questions, and then at every step you can train a system to imitate these increasingly good answers to questions, and you end up with a really powerful question-answerer that was only ever trained to imitate humans-with-access-to-aligned-systems, and so your system is outer aligned. The bar I’ve added, which represents how capable I think you can get with amplified humans, is lower than the bar for the unaligned benchmark. I've drawn this bar lower because I think that if your system is trying to imitate cognition that can be broken down into human understandable parts, it is systematically not going to be able to pursue certain powerful strategies that the end-to-end trained systems will be able to. I think that there are probably a bunch of concepts that humans can’t understand quickly, or maybe can’t understand at all. And if your systems are restricted to never use these concepts, I think your systems are probably just going to be a bunch weaker. I think that transparency techniques, as well as AI alignment strategies like microscope AI that lean heavily on them, rely on a similar assumption that the cognition of the system you’re trying to align is factorizable into human-understandable parts. One component of the best-case scenario for transparency techniques is that anytime your neural net does stuff, you can get the best possible human understandable explanation of why it's doing that thing. If such an explanation doesn’t exist, your transparency tools won’t be able to assure you that your system is aligned even if it is. To summarize, I claim that current alignment proposals don’t really have a proposal for how to make systems that are aligned but either produce plans that can’t be understood by amplified humans do cognitive actions that can’t be understood by amplified humans And so I claim that current alignment proposals don’t seem like they can control systems as powerful as the systems you’d get from an unaligned training strategy. Empirical generalization I think some people are optimistic that alignment will generalize from the cases where amplified humans can eval...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A review of Steven Pinker's new book on rationality , published by Matthew Barnett on the AI Alignment Forum. Steven Pinker's new book on rationality came out today. I figured someone on LessWrong would write a review for it, so I might as well be the one to do it. Unlike Pinker's prior books, such as The Blank Slate and The Better Angels of Our Nature, this book lacks a straightforward empirical thesis. Instead, he mirrors the sequences by building a science of rationality and then tries to convince the reader that rationality is important, both personally and socially. Unfortunately, long-time readers of LessWrong are unlikely to learn much from Pinker's new book; his content is too similar to the content in the sequences. An upside is that Pinker's treatment is more concise, and his style more closely resembles mainstream thought. Consequently, I'm tempted to recommend this book to people who might otherwise be turned away by Rationality: From A to Z. He starts by asking a simple question: how come it seems like everyone is so irrational? Pointing to religion, conspiracy theorists, ghost-believers, anti-vaxxers, alternative medicine adherents, and postmodernists, Pinker makes a good case that there's a lot of irrationality in the world. On the other hand, he continues, shouldn't humans have evolved to be more rational? How could such persistent, widespread irrationality be so common in humans, if our survival impinges on our ability to reason? Pinker provides a simple answer: humans are very rational animals, just not in every domain. In those domains on which our survival depended, such as finding and eating food, humans are much less clueless than you might have been lead to believe. Pinker provides the example of the San people of the Kalahari Desert in southern Africa, who, despite their mythological beliefs, are stunningly successful at hunting prey. He cites Louis Liebenberg, who documented how the San people use Bayesian reasoning to hunt, applying it to footprints and animal droppings in order to build an accurate picture of their natural world: a dry desert on which they have subsisted for over a hundred thousand years. It's not hard to see this dual phenomenon of rationality and irrationality reflected in the modern day: many young Earth creationists believe that the moon's craters were literally planted by God to give the appearance of old age, but these same people rarely apply the same standards of reason to matters in their ordinary life. Yet, as Pinker observes, sometimes even when our life and money does depend on our rationality, we still fail. For instance, most people consistently fail to save for retirement. Why? The answer here is simple: life today is a lot different than the lives of our ancestors. What might have been a threat 10,000 years ago—such as a tiger in the bushes—is no longer a major threat; conversely, some threats—like car crashes—are entirely new, and thus, the human brain is ill-equipped to evaluate them rationally. Pinker's book proceeds by presenting a textbook view of the science of rationality, including cognitive biases, formal logic, Bayesian inference, correlation and causation, statistical decision theory, and game theory. There isn't much to complain about here: Pinker is a great writer, and presents these ideas with impressive clarity. However, the content in these chapters rarely departs from the mainstream exposition of these subjects. Given that I already knew most of the details, I was left a tad bored. To prevent you from being bored as well, I won't summarize the book's main contents. (You can go and read his book if you want to know all the details.) Instead, I'll draw my attention to some parts I liked, and some parts I didn't like as much. What I liked First off, Pinker cited the rationalist community as an ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 2014 Survey Results, published by Scott Alexander on the AI Alignment Forum. Thanks to everyone who took the 2014 Less Wrong Census/Survey. Extra thanks to Ozy, who did a lot of the number crunching work. This year's results are below. Some of them may make more sense in the context of the original survey questions, which can be seen here. Please do not try to take the survey as it is over and your results will not be counted. I. Population There were 1503 respondents over 27 days. The last survey got 1636 people over 40 days. The last four full days of the survey saw nineteen, six, and four responses, for an average of about ten. If we assume the next thirteen days had also gotten an average of ten responses - which is generous, since responses tend to trail off with time - then we would have gotten about as many people as the last survey. There is no good evidence here of a decline in population, although it is perhaps compatible with a very small decline. II. Demographics Sex Female: 179, 11.9% Male: 1311, 87.2% Gender F (cisgender): 150, 10.0% F (transgender MtF): 24, 1.6% M (cisgender): 1245, 82.8% M (transgender FtM): 5, 0.3% Other: 64, 4.3% Sexual Orientation Asexual: 59, 3.9% Bisexual: 216, 14.4% Heterosexual: 1133, 75.4% Homosexual: 47, 3.1% Other: 35, 2.3% [This question was poorly worded and should have acknowledged that people can both be asexual and have a specific orientation; as a result it probably vastly undercounted our asexual readers] Relationship Style Prefer monogamous: 778, 51.8% Prefer polyamorous: 227, 15.1% Uncertain/no preference: 464, 30.9% Other: 23, 1.5% Number of Partners 0: 738, 49.1% 1: 674, 44.8% 2: 51, 3.4% 3: 17, 1.1% 4: 7, 0.5% 5: 1, 0.1% Lots and lots: 3, 0.2% Relationship Goals Currently not looking for new partners: 648, 43.1% Open to new partners: 467, 31.1% Seeking more partners: 370, 24.6% [22.2% of people who don’t have a partner aren’t looking for one.] Relationship Status Married: 274, 18.2% Relationship: 424, 28.2% Single: 788, 52.4% [6.9% of single people have at least one partner; 1.8% have more than one.] Living With Alone: 345, 23.0% With parents and/or guardians: 303, 20.2% With partner and/or children: 411, 27.3% With roommates: 428, 28.5% Children 0: 1317, 81.6% 1: 66, 4.4% 2: 78, 5.2% 3: 17, 1.1% 4: 6, 0.4% 5: 3, 0.2% 6: 1, 0.1% Lots and lots: 1, 0.1% Want More Children? Yes: 549, 36.1% Uncertain: 426, 28.3% No: 516, 34.3% [418 of the people who don’t have children don’t want any, suggesting that the LW community is 27.8% childfree.] Country United States, 822, 54.7% United Kingdom, 116, 7.7% Canada, 88, 5.9% Australia: 83, 5.5% Germany, 62, 4.1% Russia, 26, 1.7% Finland, 20, 1.3% New Zealand, 20, 1.3% India, 17, 1.1% Brazil: 15, 1.0% France, 15, 1.0% Israel, 15, 1.0% Lesswrongers Per Capita Finland: 1/271,950 New Zealand: 1/223,550 Australia: 1/278,674 United States: 1/358,390 Canada: 1/399,545 Israel: 1/537,266 United Kingdom: 1/552,586 Germany: 1/1,290,323 France: 1/ 4,402,000 Russia: 1/ 5,519,231 Brazil: 1/ 13,360,000 India: 1/ 73,647,058 Race Asian (East Asian): 59. 3.9% Asian (Indian subcontinent): 33, 2.2% Black: 12. 0.8% Hispanic: 32, 2.1% Middle Eastern: 9, 0.6% Other: 50, 3.3% White (non-Hispanic): 1294, 86.1% Work Status Academic (teaching): 86, 5.7% For-profit work: 492, 32.7% Government work: 59, 3.9% Homemaker: 8, 0.5% Independently wealthy: 9, 0.6% Nonprofit work: 58, 3.9% Self-employed: 122, 5.8% Student: 553, 36.8% Unemployed: 103, 6.9% Profession Art: 22, 1.5% Biology: 29, 1.9% Business: 35, 4.0% Computers (AI): 42, 2.8% Computers (other academic): 106, 7.1% Computers (practical): 477, 31.7% Engineering: 104, 6.1% Finance/Economics: 71, 4.7% Law: 38, 2.5% Mathematics: 121, 8.1% Medicine: 32, 2.1% Neuroscience: 18, 1.2% Philosophy: 36, 2.4% Physics: 65, 4.3% Psychology: 31, 2.1% Other: 157, 10.2%...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Cognitive Science of Rationality, published by lukeprog on the AI Alignment Forum. (The post is written for beginners. Send the link to your friends! Regular Less Wrong readers may want to jump to the Stanovich material.) The last 40 years of cognitive science have taught us a great deal about how our brains produce errors in thinking and decision making, and about how we can overcome those errors. These methods can help us form more accurate beliefs and make better decisions. Long before the first Concorde supersonic jet was completed, the British and French governments developing it realized it would lose money. But they continued to develop the jet when they should have cut their losses, because they felt they had "invested too much to quit"1 (sunk cost fallacy2). John tested positive for an extremely rare but fatal disease, using a test that is accurate 80% of the time. John didn't have health insurance, and the only available treatment — which his doctor recommended — was very expensive. John agreed to the treatment, his retirement fund was drained to nothing, and during the treatment it was discovered that John did not have the rare disease after all. Later, a statistician explained to John that because the disease is so rare, the chance that he had had the disease even given the positive test was less than one in a million. But neither John's brain nor his doctor's brain had computed this correctly (base rate neglect). Mary gave money to a charity to save lives in the developing world. Unfortunately, she gave to a charity that saves lives at a cost of $100,000 per life instead of one that saves lives at 1/10th that cost, because the less efficient charity used a vivid picture of a starving child on its advertising, and our brains respond more to single, identifiable victims than to large numbers of victims (identifiability effect3 and scope insensitivity4). During the last four decades, cognitive scientists have discovered a long list of common thinking errors like these. These errors lead us to false beliefs and poor decisions. How are these errors produced, and how can we overcome them? Vague advice like "be skeptical" and "think critically" may not help much. Luckily, cognitive scientists know a great deal about the mathematics of correct thinking, how thinking errors are produced, and how we can overcome these errors in order to live more fulfilling lives. Rationality First, what is rationality? It is not the same thing as intelligence, because even those with high intelligence fall prey to some thinking errors as often as everyone else.5 But then, what is rationality? Cognitive scientists recognize two kinds of rationality: Epistemic rationality is about forming true beliefs, about getting the map in your head to accurately reflect the territory of the world. We can measure epistemic rationality by comparing the rules of logic and probability theory to the way that a person actually updates their beliefs. Instrumental rationality is about making decisions that are well-aimed at bringing about what you want. Due to habit and bias, many of our decisions don't actually align with our goals. We can measure instrumental rationality with a variety of techniques developed in economics, for example testing whether a person obeys the 'axioms of choice'.6 In short, rationality improves our choices concerning what to believe and what to do. Unfortunately, human irrationality is quite common, as shown in popular books like Predictably Irrational: The Hidden Forces that Shape Our Decisions and Kluge: The Haphazard Evolution of the Human Mind. Ever since Aristotle spoke of humans as the "rational animal," we've had a picture of ourselves as rational beings that are hampered by shortcomings like anger and fear and confirmation bias. Cognitive science says just the o...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Neuroscience basics for LessWrongians , published by ChrisHallquist on the AI Alignment Forum. The origins of this article are in my partial transcript of the live June 2011 debate between Robin Hanson and Eliezer Yudkowsky. While I still feel like I don't entirely understand his arguments, a few of his comments about neuroscience made me strongly go, "no, that's not right." Furthermore, I've noticed that while LessWrong in general seems to be very strong on the psychological or "black box" side of cognitive science, there isn't as much discussion of neuroscience here. This is somewhat understandable. Our current understanding of neuroscience is frustratingly incomplete, and too much journalism on neuroscience is sensationalistic nonsense. However, I think what we do know is worth knowing. (And part of what makes much neuroscience journalism annoying is that it makes a big deal out of things that are totally unsurprising, given what we already know.) My qualifications to do this: while my degrees are in philosophy, for awhile in undergrad I was a neuroscience major, and ended up taking quite a bit of neuroscience as a result. This means I can assure you that most of what I say here is standard neuroscience which could be found in an introductory textbook like Nichols, Martin, Wallace, & Fuchs' From Neuron to Brain (one of the text books I used as an undergraduate). The only things that might not be totally standard are the conjecture I make about how complex currently-poorly-understood areas of the brain are likely to be, and also some of the points I make in criticism of Eliezer at the end (though I believe these are not a very big jump from current textbook neuroscience.) One of the main themes of this article will be specialization within the brain. In particular, we know that the brain is divided into specialized areas at the macro level, and we understand some (though not very much) of the micro-level wiring that supports this specialization. It seems likely that each region of the brain has its own micro-level wiring to support its specialized function, and in some regions the wiring is likely to be quite complex. 1. Specialization of brain regions One of the best-established facts about the brain is that specific regions handle specific functions. And it isn’t just that in each individual, specific brain regions handle specific functions. It’s also that which regions handle which functions is consistent across individuals. This is an extremely well-established finding, but it’s worth briefly summarizing some of the evidence for it. One kind of evidence comes from experiments involving direct electrical stimulation of the brain. This cannot ethically be done on humans without a sound medical reason, but it is used with epileptic patients in order to determine the source of the problem, which is necessary in order to treat epilepsy surgically. In epileptic patients, stimulating certain regions of the brain (known as the primary sensory areas) causes the patient to report sensations: sights, sounds, feelings, smells, and tastes. Which sensations are caused by stimulating which regions of the brain is consistent across patients. This is the source of the “Penfield homunculus,” a map of brain regions which, when stimulated, result in touch sensations which patients describe as feeling like they come from particular parts of the body. Stimulating one region, for example, might consistently lead to a patient reporting a feeling in his left foot. Regions of the brain associated with sensations are known as sensory areas or sensory cortex. Other regions of the brain, when stimulated, lead to involuntary muscle movements. Those areas are known as motor areas or motor cortex, and again, which areas correspond to which muscles is consistent across patients. The consistenc...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Transhumanism as Simplified Humanism , published by Eliezer Yudkowsky on the AI Alignment Forum. This essay was originally posted in 2007. Frank Sulloway once said: “Ninety-nine per cent of what Darwinian theory says about human behavior is so obviously true that we don’t give Darwin credit for it. Ironically, psychoanalysis has it over Darwinism precisely because its predictions are so outlandish and its explanations are so counterintuitive that we think, Is that really true? How radical! Freud’s ideas are so intriguing that people are willing to pay for them, while one of the great disadvantages of Darwinism is that we feel we know it already, because, in a sense, we do.” Suppose you find an unconscious six-year-old girl lying on the train tracks of an active railroad. What, morally speaking, ought you to do in this situation? Would it be better to leave her there to get run over, or to try to save her? How about if a 45-year-old man has a debilitating but nonfatal illness that will severely reduce his quality of life – is it better to cure him, or not cure him? Oh, and by the way: This is not a trick question. I answer that I would save them if I had the power to do so – both the six-year-old on the train tracks, and the sick 45-year-old. The obvious answer isn’t always the best choice, but sometimes it is. I won’t be lauded as a brilliant ethicist for my judgments in these two ethical dilemmas. My answers are not surprising enough that people would pay me for them. If you go around proclaiming “What does two plus two equal? Four!” you will not gain a reputation as a deep thinker. But it is still the correct answer. If a young child falls on the train tracks, it is good to save them, and if a 45-year-old suffers from a debilitating disease, it is good to cure them. If you have a logical turn of mind, you are bound to ask whether this is a special case of a general ethical principle which says “Life is good, death is bad; health is good, sickness is bad.” If so – and here we enter into controversial territory – we can follow this general principle to a surprising new conclusion: If a 95-year-old is threatened by death from old age, it would be good to drag them from those train tracks, if possible. And if a 120-year-old is starting to feel slightly sickly, it would be good to restore them to full vigor, if possible. With current technology it is not possible. But if the technology became available in some future year – given sufficiently advanced medical nanotechnology, or such other contrivances as future minds may devise – would you judge it a good thing, to save that life, and stay that debility? The important thing to remember, which I think all too many people forget, is that it is not a trick question. Transhumanism is simpler – requires fewer bits to specify – because it has no special cases. If you believe professional bioethicists (people who get paid to explain ethical judgments) then the rule “Life is good, death is bad; health is good, sickness is bad” holds only until some critical age, and then flips polarity. Why should it flip? Why not just keep on with life-is-good? It would seem that it is good to save a six-year-old girl, but bad to extend the life and health of a 150-year-old. Then at what exact age does the term in the utility function go from positive to negative? Why? As far as a transhumanist is concerned, if you see someone in danger of dying, you should save them; if you can improve someone’s health, you should. There, you’re done. No special cases. You don’t have to ask anyone’s age. You also don’t ask whether the remedy will involve only “primitive” technologies (like a stretcher to lift the six-year-old off the railroad tracks); or technologies invented less than a hundred years ago (like penicillin) which nonetheless seem ordinary beca...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Selection vs Control, published by Selection vs Control on the AI Alignment Forum. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. This is something which has bothered me for a while, but, I'm writing it specifically in response to the recent post on mesa-optimizers. I feel strongly that the notion of 'optimization process' or 'optimizer' which people use -- partly derived from Eliezer's notion in the sequences -- should be split into two clusters. I call these two clusters 'selection' vs 'control'. I don't have precise formal statements of the distinction I'm pointing at; I'll give several examples. Before going into it, several reasons why this sort of thing may be important: It could help refine the discussion of mesa-optimization. The article restricted its discussion to the type of optimization I'll call 'selection', explicitly ruling out 'control'. This choice isn't obviously right. (More on this later.) Refining 'agency-like' concepts like this seems important for embedded agency -- what we eventually want is a story about how agents can be in the world. I think almost any discussion of the relationship between agency and optimization which isn't aware of the distinction I'm drawing here (at least as a hypothesis) will be confused. Generally, I feel like I see people making mistakes by not distinguishing between the two (whether or not they've derived their notion of optimizer from Eliezer). I judge an algorithm differently if it is intended as one or the other. (See also Stuart Armstrong's summary of other problems with the notion of optimization power Eliezer proposed -- those are unrelated to my discussion here, and strike me more as technical issues which call for refined formulae, rather than conceptual problems which call for revised ontology.) The Basic Idea Eliezer quantified optimization power by asking how small a target an optimization process hits, out of a space of possibilities. The type of 'space of possibilities' is what I want to poke at here. Selection First, consider a typical optimization algorithm, such as simulated annealing. The algorithm constructs an element of the search space (such as a specific combination of weights for a neural network), gets feedback on how good that element is, and then tries again. Over many iterations of this process, it finds better and better elements. Eventually, it outputs a single choice. This is the prototypical 'selection process' -- it can directly instantiate any element of the search space (although typically we consider cases where the process doesn't have time to instantiate all of them), it gets direct feedback on the quality of each element (although evaluation may be costly, so that the selection process must economize these evaluations), the quality of an element of search space does not depend on the previous choices, and only the final output matters. The term 'selection process' refers to the fact that this type of optimization selects between a number of explicitly given possibilities. The most basic example of this phenomenon is a 'filter' which rejects some elements and accepts others -- like selection bias in statistics. This has a limited ability to optimize, however, because it allows only one iteration. Natural selection is an example of much more powerful optimization occurring through iteration of selection effects. Control Now, consider a targeting system on a rocket -- let's say, a heat-seeking missile. The missile has sensors and actuators. It gets feedback from its sensors, and must somehow use this information to decide how to use its actuators. This is my prototypical control process. (The term 'control process' is supposed to invoke control theory.) Unlike a selection process, a controller can only instantiate one element of the space of...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why Subagents?, published by johnswentworth on the AI Alignment Forum. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. The justification for modelling real-world systems as “agents” - i.e. choosing actions to maximize some utility function - usually rests on various coherence theorems. They say things like “either the system’s behavior maximizes some utility function, or it is throwing away resources” or “either the system’s behavior maximizes some utility function, or it can be exploited” or things like that. Different theorems use slightly different assumptions and prove slightly different things, e.g. deterministic vs probabilistic utility function, unique vs non-unique utility function, whether the agent can ignore a possible action, etc. One theme in these theorems is how they handle “incomplete preferences”: situations where an agent does not prefer one world-state over another. For instance, imagine an agent which prefers pepperoni over mushroom pizza when it has pepperoni, but mushroom over pepperoni when it has mushroom; it’s simply never willing to trade in either direction. There’s nothing inherently “wrong” with this; the agent is not necessarily executing a dominated strategy, cannot necessarily be exploited, or any of the other bad things we associate with inconsistent preferences. But the preferences can’t be described by a utility function over pizza toppings. In this post, we’ll see that these kinds of preferences are very naturally described using subagents. In particular, when preferences are allowed to be path-dependent, subagents are important for representing consistent preferences. This gives a theoretical grounding for multi-agent models of human cognition. Preference Representation and Weak Utility Let’s expand our pizza example. We’ll consider an agent who: Prefers pepperoni, mushroom, or both over plain cheese pizza Prefers both over pepperoni or mushroom alone Does not have a stable preference between mushroom and pepperoni - they prefer whichever they currently have We can represent this using a directed graph: The arrows show preference: our agent prefers B over A if (and only if) there is a directed path from A to B along the arrows. There is no path from pepperoni to mushroom or from mushroom to pepperoni, so the agent has no preference between them. In this case, we’re interpreting “no preference” as “agent prefers to keep whatever they have already”. Note that this is NOT the same as “the agent is indifferent”, in which case the agent is willing to switch back and forth between the two options as long as the switch doesn’t cost anything. Key point: there is no cycle in this graph. If the agent’s preferences are cyclic, that’s when they provably throw away resources, paying to go in circles. As long as the preferences are acyclic, we call them “consistent”. Now, at this point we can still define a “weak” utility function by ignoring the “missing” preference between pepperoni and mushroom. Here’s the idea: a normal utility function says “the agent always prefers the option with higher utility”. A weak utility function says: “if the agent has a preference, then they always prefer the option with higher utility”. The missing preference means we can’t build a normal utility function, but we can still build a weak utility function. Here’s how: since our graph has no cycles, we can always order the nodes so that the arrows only go forward along the sorted nodes - a technique called topological sorting. Each node’s position in the topological sort order is its utility. A small tweak to this method also handles indifference. (Note: I’m using the term “weak utility” here because it seems natural; I don’t know of any standard term for this in the literature. Most people don’t distinguish between these two ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Studies On Slack, published by Scott Alexander on the AI Alignment Forum. I. Imagine a distant planet full of eyeless animals. Evolving eyes is hard: they need to evolve Eye Part 1, then Eye Part 2, then Eye Part 3, in that order. Each of these requires a separate series of rare mutations. Here on Earth, scientists believe each of these mutations must have had its own benefits – in the land of the blind, the man with only Eye Part 1 is king. But on this hypothetical alien planet, there is no such luck. You need all three Eye Parts or they’re useless. Worse, each Eye Part is metabolically costly; the animal needs to eat 1% more food per Eye Part it has. An animal with a full eye would be much more fit than anything else around, but an animal with only one or two Eye Parts will be at a small disadvantage. So these animals will only evolve eyes in conditions of relatively weak evolutionary pressure. In a world of intense and perfect competition, where the fittest animal always survives to reproduce and the least fit always dies, the animal with Eye Part 1 will always die – it’s less fit than its fully-eyeless peers. The weaker the competition, and the more randomness dominates over survival-of-the-fittest, the more likely an animal with Eye Part 1 can survive and reproduce long enough to eventually produce a descendant with Eye Part 2, and so on. There are lots of ways to decrease evolutionary pressure. Maybe natural disasters often decimate the population, dozens of generations are spend recolonizing empty land, and during this period there’s more than enough for everyone and nobody has to compete. Maybe there are frequent whalefalls, and any animal nearby has hit the evolutionary jackpot and will have thousands of descendants. Maybe the population is isolated in little islands and mountain valleys, and one gene or another can reach fixation in a population totally by chance. It doesn’t matter exactly how it happens, it matters that evolutionary pressure is low. The branch of evolutionary science that deals with this kind of situation is called “adaptive fitness landscapes”. Landscapes really are a great metaphor – consider somewhere like this: You pour out a bucket of water. Water “flows downhill”, so it’s tempting to say something like “water wants to be at the lowest point possible”. But that’s not quite right. The lowest point possible is the pit, and water won’t go there. It will just sit in the little puddle forever, because it would have to go up the tiny little hillock in order to get to the pit, and water can’t flow uphill. Using normal human logic, we feel tempted to say something like “Come on! The hillock is so tiny, and that pit is so deep, just make a single little exception to your ‘always flow downhill’ policy and you could do so much better for yourself!” But water stubbornly refuses to listen. Under conditions of perfectly intense competition, evolution works the same way. We imagine a multidimensional evolutionary “landscape” where lower ground represents higher fitness. In this perfectly intense competition, organisms can go from higher to lower fitness, but never vice versa. As with water, the tiniest hillock will leave their potential forever unrealized. Under more relaxed competition, evolution only tends probabilistically to flow downhill. Every so often, it will flow uphill; the smaller the hillock, the more likely evolution will surmount it. Given enough time, it’s guaranteed to reach the deepest pit and mostly stay there. Take a moment to be properly amazed by this. It sounds like something out of the Tao Te Ching. An animal with eyes has very high evolutionary fitness. It will win at all its evolutionary competitions. So in order to produce the highest-fitness animal, we need to – select for fitness less hard? In order to produce an animal t...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Possible takeaways from the coronavirus pandemic for slow AI takeoff , published by Vika on the AI Alignment Forum. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. (Cross-posted from personal blog. Summarized in Alignment Newsletter #104. Thanks to Janos Kramar for his helpful feedback on this post.) Epistemic status: fairly speculative, would appreciate feedback As the covid-19 pandemic unfolds, we can draw lessons from it for managing future global risks, such as other pandemics, climate change, and risks from advanced AI. In this post, I will focus on possible implications for AI risk. For a broader treatment of this question, I recommend FLI's covid-19 page that includes expert interviews on the implications of the pandemic for other types of risks. A key element in AI risk scenarios is the speed of takeoff - whether advanced AI is developed gradually or suddenly. Paul Christiano's post on takeoff speeds defines slow takeoff in terms of the economic impact of AI as follows: "There will be a complete 4 year interval in which world output doubles, before the first 1 year interval in which world output doubles." It argues that slow AI takeoff is more likely than fast takeoff, but is not necessarily easier to manage, since it poses different challenges, such as large-scale coordination. This post expands on this point by examining some parallels between the coronavirus pandemic and a slow takeoff scenario. The upsides of slow takeoff include the ability to learn from experience, act on warning signs, and reach a timely consensus that there is a serious problem. I would argue that the covid-19 pandemic had these properties, but most of the world's institutions did not take advantage of them. This suggests that, unless our institutions improve, we should not expect the slow AI takeoff scenario to have a good default outcome. Learning from experience. In the slow takeoff scenario, general AI is expected to appear in a world that has already experienced transformative change from less advanced AI, and institutions will have a chance to learn from problems with these AI systems. An analogy could be made with learning from dealing with less "advanced" epidemics like SARS that were not as successful as covid-19 at spreading across the world. While some useful lessons were learned, they were not successfully generalized to covid-19, which had somewhat different properties than these previous pathogens (such as asymptomatic transmission and higher virulence). Similarly, general AI may have somewhat different properties from less advanced AI that would make mitigation strategies more difficult to generalize. Warning signs. In the coronavirus pandemic response, there has been a lot of variance in how successfully governments acted on warning signs. Western countries had at least a month of warning while the epidemic was spreading in China, which they could have used to stock up on PPE and build up testing capacity, but most did not do so. Experts have warned about the likelihood of a coronavirus outbreak for many years, but this did not lead most governments to stock up on medical supplies. This was a failure to take cheap preventative measures in response to advance warnings about a widely recognized risk with tangible consequences, which is not a good sign for the case where the risk is less tangible and well-understood (such as risk from general AI). Consensus on the problem. During the covid-19 epidemic, the abundance of warning signs and past experience with previous pandemics created an opportunity for a timely consensus that there is a serious problem. However, it actually took a long time for a broad consensus to emerge - the virus was often dismissed as "overblown" and "just like the flu" as late as March 2020. A timely response t...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Rereading Atlas Shrugged, published by Vaniver on the AI Alignment Forum. Write a Review This post will not attempt to avoid spoilers, and will be much more comprehensible if you've read the book or are familiar with its basics, but I also hope it'll be somewhat understandable if you haven't read the book at all; to aid with that I'll put summaries after all the names. I first read Atlas Shrugged as a teenager, I think for an essay contest. I was already a libertarian from reading Free to Choose, and found Rand's moralism offputting and her characters strange. I was a 'technical' libertarian,[1] in that I was convinced that decentralization led to better decision-making and better results, and didn't see how the moral libertarians made a better case than the moral statists. And even when it came to morality, the people I saw at church were putting in significant effort to try to be better, and yet Rand's heroes didn't seem to have any sort of moral development; the good people were good, and the bad people were bad, and there wasn't any engagement with the question of how to become good. I think that was the main content of my essay, and unsurprisingly it didn't win anything. But a friend recently mentioned that they had read it and were surprised about how much it was about rationality; I remembered some bits and said "yeah, that checks out," but when I reread it recently was surprised at just how much there was, and how topical much of it was to current events and decisions I'm facing. A truly great book should be read in youth, again in maturity and once more in old age, as a fine building should be seen by morning light, at noon and by moonlight. ― Robertson Davies For this post I expect to slip between "how I saw Atlas Shrugged as a youth" and "how I see it now", with the first mostly to explain by contrast. Creators revisited In youth, I thought Galt's Gulch (a hideout in the Rockies accessible only to the creators on strike) was ridiculous. You have people whose primary skills are being executives, and they become manual laborers, and they're better off? Why think a mining executive would be any good at digging copper himself, or an aircraft executive would be good at raising hogs? I think I was confused instead of enlightened by having the category of "executive." James Taggart (the 'villain' railroad executive) would be denied entry to Galt's Gulch, and starve if he ended up there. The primary characteristic of the creators is that they operate off their inside view and own responsibility. Rearden (the 'hero' steel-maker) invents a new variety of metal, not by seeing it in a flash of insight, but by believing that it's possible enough to remain determined through ten years of obstacles and setbacks. Dagny Taggart (the 'hero' railroad executive) provides value by making decisions using her own judgment, by paying close attention to details, and turning towards instead of away from problems. Some of the scientific and engineering inventions are fake, and I think in youth I overestimated how much the characters were supposed to be mutant superheroes instead of doing something that could be copied. Sure, you might not be able to sleep as little as Dagny, but you could try to actually succeed in your work, and work in a field where that's noticed and rewarded instead of punished. It reminds me of simulcra levels; the heroes are the people who live in 'reality', and the villains are the people who live in 'society'. The heroes look at the world to determine what is true; they say things they think are true so that other people will have a more accurate model of the world; they try to enter honest competitions, they try to win, are sportsmanlike when they lose, and think there is no honor or profit in dishonest competitions. When selling things, they assume buyers ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [$10k bounty] Read and compile Robin Hanson’s best posts , published by Richard_Ngo on the AI Alignment Forum. I think Robin Hanson's ideas are not read nearly as widely as they should be, in part because it's difficult to navigate his many, many blog posts (I estimate he's written 2000 of them exactly 3302 of them). So I'd like to pay someone to read through all his writings and compile the best ones into a more accessible format. The default output would be an ebook like Rationality: from AI to Zombies, containing several thematically-linked sequences of posts; possible extensions of this include adding summaries or publishing physical copies (although let me know if you have any other suggestions). I expect this to take 1-2 months of work, and plan to pay around $10k USD (more details to be determined as we get a better idea of the scope of the project). My gmail address is richardcngo; email me with the subject line "Hanson compilation", plus any relevant information about yourself, if you might be interested in doing this. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Bias You Didn't Expect, published by Psychohistorian on the AI Alignment Forum. There are few places where society values rational, objective decision making as much as it values it in judges. While there is a rather cynical discipline called legal realism that says the law is really based on quirks of individual psychology, "what the judge had for breakfast," there's a broad social belief that the decision of judges are unbiased. And where they aren't unbiased, they're biased for Big, Important, Bad reasons, like racism or classism or politics. It turns out that legal realism is totally wrong. It's not what the judge had for breakfast. It's how recently the judge had breakfast. A a new study (media coverage) on Israeli judges shows that, when making parole decisions, they grant about 65% after meal breaks, and almost all the way down to 0% right before breaks and at the end of the day (i.e. as far from the last break as possible). There's a relatively linear decline between the two points. Think about this for a moment. A tremendously important decision, determining whether a person will go free or spend years in jail, appears to be substantially determined by an arbitrary factor. Also, note that we don't know if it's the lack of food, the anticipation of a break, or some other factor that is responsible for this. More interestingly, we don't know where the optimal result occurred. It's probably not the near 0% at the end of each work period. But is it the post-break high of 65%? Or were judges being too nice? We know there was bias, but we still don't know when bias occurred. There are at least two lessons from this. The little, obvious one is to be aware of one's own physical limitations. Avoid making big decisions when tired or hungry - though this doesn't mean you should try to make decisions right after eating. For particularly important decisions, consider contemplating them at different times, if you can. Think about one thing Monday morning, then Wednesday afternoon, then Saturday evening, going only to the point of getting an overall feel for an answer, and not to the point of really making a solid conclusion. Take notes, and then compare them. This may not work perfectly, but it may help you realize inconsistencies, which could help. For big questions, the wisdom of crowds may be helpful - unless it's been a while since most of the crowd had breakfast. The bigger lesson is one of humility. This provides rather stark evidence that our decisions are not under our control to the extent we believe. We can be influenced by factors we don't even suspect. Even knowing we have been biased, we may still be unable to identify what the correct answer was. While using formal rules and logic may be one of the best approaches to minimizing such errors, even formal rules can fail when applied by biased agents. The biggest, most condemnable biases - like racism - are in some ways less dangerous, because we know we need to look out for them. It's the bias you don't even suspect that can get you. The authors of the study think they basically got lucky with these results - if the effect had been to make decisions arbitrary rather than to increase rejections, this would not have shown up. When those charged with making impartial decisions that control people's lives are subject to arbitrary forces they never suspected, it shows how important it is and much more we can do to be less wrong. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: An Intuitive Explanation of Solomonoff Induction , published by Alex_Altair on the AI Alignment Forum. This is the completed article that Luke wrote the first half of. My thanks go to the following for reading, editing, and commenting; Luke Muehlhauser, Louie Helm, Benjamin Noble, and Francelle Wax. People disagree about things. Some say that television makes you dumber; other say it makes you smarter. Some scientists believe life must exist elsewhere in the universe; others believe it must not. Some say that complicated financial derivatives are essential to a modern competitive economy; others think a nation's economy will do better without them. It's hard to know what is true. And it's hard to know how to figure out what is true. Some argue that you should assume the things you are most certain about and then deduce all other beliefs from your original beliefs. Others think you should accept at face value the most intuitive explanations of personal experience. Still others think you should generally agree with the scientific consensus until it is disproved. Wouldn't it be nice if determining what is true was like baking a cake? What if there was a recipe for finding out what is true? All you'd have to do is follow the written directions exactly, and after the last instruction you'd inevitably find yourself with some sweet, tasty truth! In this tutorial, we'll explain the closest thing we’ve found so far to a recipe for finding truth: Solomonoff induction. There are some qualifications to make. To describe just one: roughly speaking, you don't have time to follow the recipe. To find the truth to even a simple question using this recipe would require you to follow one step after another until long after the heat death of the universe, and you can't do that. But we can find shortcuts. Suppose you know that the exact recipe for baking a cake asks you to count out one molecule of H2O at a time until you have exactly 0.5 cups of water. If you did that, you might not finish the cake before the heat death of the universe. But you could approximate that part of the recipe by measuring out something very close to 0.5 cups of water, and you'd probably still end up with a pretty good cake. Similarly, once we know the exact recipe for finding truth, we can try to approximate it in a way that allows us to finish all the steps sometime before the sun burns out. This tutorial explains that best-we've-got-so-far recipe for finding truth, Solomonoff induction. Don’t worry, we won’t be using any equations, just qualitative descriptions. Like Eliezer Yudkowsky's Intuitive Explanation of Bayes' Theorem and Luke Muehlhauser's Crash Course in the Neuroscience of Human Motivation, this tutorial is long. You may not have time to read it; that's fine. But if you do read it, we recommend that you read it in sections. Contents: Background 1. Algorithms — We’re looking for an algorithm to determine truth. 2. Induction — By “determine truth”, we mean induction. 3. Occam’s Razor — How we judge between many inductive hypotheses. 4. Probability — Probability is what we usually use in induction. 5. The Problem of Priors — Probabilities change with evidence, but where do they start? The Solution 6. Binary Sequences — Everything can be encoded as binary. 7. All Algorithms — Hypotheses are algorithms. Turing machines describe these. 8. Solomonoff's Lightsaber — Putting it all together. 9. Formalized Science — From intuition to precision. 10. Approximations — Ongoing work towards practicality. 11. Unresolved Details — Problems, philosophical and mathematical. Algorithms At an early age you learned a set of precisely-defined steps — a 'recipe' or, more formally, an algorithm — that you could use to find the largest number in a list of numbers like this: 21, 18, 4, 19, 55, 12, 30 The algorithm you learn...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Nature of Offense, published by Wei_Dai on the AI Alignment Forum. Recently, an extended discussion has taken place over the fact that a portion of comments here were found to be offensive by some members of this community, while others denied their offensive nature or professed to be puzzled by why they are considered offensive. Several possible explanations for why the comments are offensive have been advanced, and solutions offered based on them: to be thought of, talked about as, or treated like a non-person (Alicorn) analysis of behavior that puts the reader in the group being analyzed, and the speaker outside it (orthonormal) exclusion from the intended audience (Eliezer) Each of these explanations seems to have an element of truth, and each solution seems to have a chance of ameliorating the problem. But even though the discussion has mostly died down, we appear far from reaching an agreement, and I think one reason may be the lack of a general theory of the phenomenon of "offense", in the sense of giving and taking offense, that we can use to explain what has happened, so all of the proposed explanations and solutions feel somewhat arbitrary and unfair. (I think this article has it mostly right, but I’ll give a much shorter account since I can skip the background evo psych info, and I’m not being paid by the word. :) Let’s consider what other behavior are often considered offensive and see if we can find a pattern: use of vulgar language (where it's not customarily used) failing to address someone by their honorary titles not affording someone their customary privileges to impugn someone’s beauty, intelligence, talent, morality, honor, ancestry, etc. making a joke at someone’s expense What do all these have in common? Hint: the answer is quite ironic, given the comment that first triggered this whole fracas. most people here don't value social status enough and (especially the men) don't value having sex with extremely attractive women that money and status would get them As you may have guessed by now, I think the answer is status. Specifically, to give offense is to imply that a person or group has or should have low status. Taking offense then becomes easy to explain: it’s to defend someone’s status from such an implication, out of a sense of either fairness or self-interest. Let’s go back to the three hypotheses I collected and see if this theory can cover them as special cases. “to be thought of, talked about as, or treated like a non-person” Well, to be like a non-person is clearly to have low status. “analysis of behavior that puts the reader in the group being analyzed, and the speaker outside it” A typical situation in which one group analyzes the behavior of another is a scientific study. In such a study, the researchers usually have higher status than the subjects being studied. But even to offer a casual analysis of someone else’s behavior is to presume more intelligence, insight, or wisdom than that person. “exclusion from the intended audience” To be excluded from the intended audience is to be labeled an outsider by implication, and outsiders typically have lower status than insiders. But to fully understand why this particular comment is especially offensive, I think we have to consider that it (as well as many PUA discussions) specifically advocates (or appears to advocate) treating women as sex objects instead of potential romantic partners. Now think of the status difference between a sex object and a romantic partner... Ethical Implications Usually, one avoids giving offense by minding one’s audience and taking care not to use any language that might cause offense to any audience member. This is very easy to do one-on-one, pretty easy in a small group, hard in front of a large audience (case in point: Larry Summers’s infamous speech), ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Use Your Identity Carefully , published by Ben_LandauTaylor on the AI Alignment Forum. In Keep Your Identity Small, Paul Graham argues against associating yourself with labels (i.e. “libertarian,” “feminist,” “gamer,” “American”) because labels constrain what you’ll let yourself believe. It’s a wonderful essay that’s led me to make concrete changes in my life. That said, it’s only about 90% correct. I have two issues with Graham’s argument; one is a semantic quibble, but it leads into the bigger issue, which is a tactic I’ve used to become a better person. Graham talks about the importance of identity in determining beliefs. This isn’t quite the right framework. I’m a fanatical consequentialist, so I care what actions people take. Beliefs can constrain actions, but identity can also constrain actions directly. To give a trivial example from the past week in which beliefs didn’t matter: I had a self-image as someone who didn’t wear jeans or t-shirts. As it happens, there are times when wearing jeans is completely fine, and when other people wore jeans in casual settings, I knew it was appropriate. Nevertheless, I wasn’t able to act on this belief because of my identity. (I finally realized this was silly, consciously discarded that useless bit of identity, and made a point of wearing jeans to a social event.) Why is this distinction important? If we’re looking at identify from an action-centered framework, this recommends a different approach from Graham’s. Do you want to constrain your beliefs? No; you want to go wherever the evidence pushes you. “If X is true, I desire to believe that X is true. If X is not true, I desire to believe that X is not true.” Identity will only get in the way. Do you want to constrain your actions? Yes! Ten thousand times yes! Akrasia exists. Commitment devices are useful. Beeminder is successful. Identity is one of the most effective tools for the job, if you wield it deliberately. I’ve cultivated an identity as a person who makes events happen. It took months to instill, but now, when I think “I wish people were doing X,” I instinctively start putting together a group to do X. This manifests in minor ways, like the tree-climbing expedition I put together at the Effective Altruism Summit, and in big ways, like the megameetup we held in Boston. If I hadn’t used my identity to motivate myself, neither of those things would’ve happened, and my life would be poorer. Identity is powerful. Powerful things are dangerous, like backhoes and bandsaws. People use them anyway, because sometimes they’re the best tools for the job, and because safety precautions can minimize the danger. Identity is hard to change. Identity can be difficult to notice. Identity has unintended consequences. Use this tool only after careful deliberation. What would this identity do to your actions? What would it do to your beliefs? What social consequences would it have? Can you do the same thing with a less dangerous tool? Think twice, and then think again, before you add to your identity. Most identities are a hindrance. But please, don’t discard this tool just because some things might go wrong. If you are willful, and careful, and wise, then you can cultivate the identity of the person you always wanted to be. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Thomas C. Schelling's "Strategy of Conflict" , published by cousin_it on the AI Alignment Forum. It's an old book, I know, and one that many of us have already read. But if you haven't, you should. If there's anything in the world that deserves to be called a martial art of rationality, this book is the closest approximation yet. Forget rationalist Judo: this is rationalist eye-gouging, rationalist gang warfare, rationalist nuclear deterrence. Techniques that let you win, but you don't want to look in the mirror afterward. Imagine you and I have been separately parachuted into an unknown mountainous area. We both have maps and radios, and we know our own positions, but don't know each other's positions. The task is to rendezvous. Normally we'd coordinate by radio and pick a suitable meeting point, but this time you got lucky. So lucky in fact that I want to strangle you: upon landing you discovered that your radio is broken. It can transmit but not receive. Two days of rock-climbing and stream-crossing later, tired and dirty, I arrive at the hill where you've been sitting all this time smugly enjoying your lack of information. And after we split the prize and cash our checks I learn that you broke the radio on purpose. Schelling's book walks you through numerous conflict situations where an unintuitive and often self-limiting move helps you win, slowly building up to the topic of nuclear deterrence between the US and the Soviets. And it's not idle speculation either: the author worked at the White House at the dawn of the Cold War and his theories eventually found wide military application in deterrence and arms control. Here's a selection of quotes to give you a flavor: the whole book is like this, except interspersed with game theory math. The use of a professional collecting agency by a business firm for the collection of debts is a means of achieving unilateral rather than bilateral communication with its debtors and of being therefore unavailable to hear pleas or threats from the debtors. A sufficiently severe and certain penalty on the payment of blackmail can protect a potential victim. One may have to pay the bribed voter if the election is won, not on how he voted. I can block your car in the road by placing my car in your way; my deterrent threat is passive, the decision to collide is up to you. If you, however, find me in your way and threaten to collide unless I move, you enjoy no such advantage: the decision to collide is still yours, and I enjoy deterrence. You have to arrange to have to collide unless I move, and that is a degree more complicated. We have learned that the threat of massive destruction may deter an enemy only if there is a corresponding implicit promise of nondestruction in the event he complies, so that we must consider whether too great a capacity to strike him by surprise may induce him to strike first to avoid being disarmed by a first strike from us. Leo Szilard has even pointed to the paradox that one might wish to confer immunity on foreign spies rather than subject them to prosecution, since they may be the only means by which the enemy can obtain persuasive evidence of the important truth that we are making no preparations for embarking on a surprise attack. I sometimes think of game theory as being roughly divided in three parts, like Gaul. There's competitive zero-sum game theory, there's cooperative game theory, and there are games where players compete but also have some shared interest. Except this third part isn't a middle ground. It's actually better thought of as ultra-competitive game theory. Zero-sum settings are relatively harmless: you minimax and that's it. It's the variable-sum games that make you nuke your neighbour. Sometime ago in my wild and reckless youth that hopefully isn't over yet, a certain ex-girlfriend...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Forces of Blandness and the Disagreeable Majority , published by sarahconstantin on the AI Alignment Forum. There are a few data points that have been making me see “the discourse” differently lately. 1. Large Majorities Dislike Political Correctness. That’s the title of this Atlantic article that came out in October, and is based on this study from the think tank More in Common which opposes political polarization. The results of the 8000-person poll of a nationally-representative sample of Americans are pretty striking. About 80% of Americans think “political correctness is a problem”; and even when you restrict to self-identified liberals, Democrats, or people of color, large majorities agree with the statement. The study identifies “progressive activists” (8% of Americans) as a younger, more extreme, more educated, more politically active left-wing cluster, and even within this cluster, a full 25% agree with “political correctness is a problem.” And lots of people who agree with statements about hate speech being bad, white people starting out with advantages in life, sexual harassment being a problem, etc, also think political correctness is a problem. Being “politically incorrect” isn’t just a white thing, a male thing, or even a conservative thing. It’s a hugely common thing. 2. Support for free speech is common, and growing, not shrinking. And it’s not the most left-wing people who most oppose free speech, but the moderate liberals. Political scientist Justin Murphy has done studies about this, based on the General Social Survey, a large poll on social attitudes that’s been running for decades. Since the 1970’s, Americans have become more tolerant of allowing people with controversial views to speak in public — communists, people proposing military coups, homosexuals, and opponents of “all churches and religions.” Racism is the exception to the rule — people haven’t become more tolerant of racist speech, even as they have become more tolerant of other varieties of speech. Keep in mind that legal censorship and centralization of political speech were way more prevalent in mid-20th century America than they are today. Cable television networks didn’t exist till the 1970’s. The Fairness Doctrine didn’t end until 1987. Satellite radio, which allowed obscene language that was regulated on conventional radio and television, only began in 1988, Fox News was founded in 1996, and, of course, the blogosphere didn’t really begin until the early 2000’s. Murphy notes that “extreme liberals” are consistently the most supportive of permitting controversial speech, and that in fact they have increased their rates of tolerating even racist speech. People who rate themselves as “moderately liberal” and “slightly liberal”, however, have sharply declined in their willingness to tolerate racist speech. If there’s been a “backlash against free speech”, it’s on the moderate left, not the far left. 3. Calls for speech restrictions often come from moderates. Things like this essay by Renee diResta, which I found chilling — a call for social media to be actively regulated by the US military, which says we should treat people spreading opinions that weaken trust in “the legitimacy of government, the persistence of societal cohesion, even our ability to respond to the impending climate crisis” as “digital combatants.” DiResta says, “More authoritarian regimes, by contrast, would simply turn off the internet. An admirable commitment to the principle of free speech in peace time turns into a sucker position against adversarial psy-ops in wartime.” Who is DiResta? She’s a writer, technologist, adviser to Congress and the State Department, and the director of research at something called New Knowledge, a firm offering corporations a new kind of service: using algorithms to bury social m...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Schelling Choice is "Rabbit", not "Stag", published by Raemon on the AI Alignment Forum. Followup/distillation/alternate-take on Duncan Sabien's Dragon Army Retrospective and Open Problems in Group Rationality. There's a particular failure mode I've witnessed, and fallen into myself: I see a problem. I see, what seems to me, to be an obvious solution to the problem. If only everyone Took Action X, we could Fix Problem Z. So I start X-ing, and maybe talking about how other people should start X-ing. Action X takes some effort on my part but it's obviously worth it. And yet... nobody does. Or not enough people do. And a few months later, here I'm still taking Action X and feeling burned and frustrated. Or – – the problem is that everyone is taking Action Y, which directly causes Problem Z. If only everyone would stop Y-ing, Problem Z would go away. Action Y seems obviously bad, clearly we should be on the same page about this. So I start noting to people when they're doing Action Y, and expect them to stop. They don't stop. So I start subtly socially punishing them for it. They don't stop. What's more... now they seem to be punishing me. I find myself getting frustrated, perhaps angry. What's going on? Are people wrong-and-bad? Do they have wrong-and-bad beliefs? Alas. So far in my experience it hasn't been that simple. A recap of 'Rabbit' vs 'Stag' I'd been planning to write this post for years. Duncan Sabien went ahead and wrote it before I got around to it. But, Dragon Army Retrospective and Open Problems in Group Rationality are both lengthy posts with a lot of points, and it still seemed worth highlighting this particular failure mode in a single post. I used to think a lot in terms of Prisoner's Dilemma, and "Cooperate"/"Defect." I'd see problems that could easily be solved if everyone just put a bit of effort in, which would benefit everyone. And people didn't put the effort in, and this felt like a frustrating, obvious coordination failure. Why do people defect so much? Eventually Duncan shifted towards using Stag Hunt rather than Prisoner's Dilemma as the model here. If you haven't read it before, it's worth reading the description in full. If you're familiar you can skip to my current thoughts below. My new favorite tool for modeling this is stag hunts, which are similar to prisoner’s dilemmas in that they contain two or more people each independently making decisions which affect the group. In a stag hunt: Imagine a hunting party venturing out into the wilderness. Each player may choose stag or rabbit, representing the type of game they will try to bring down. All game will be shared within the group (usually evenly, though things get more complex when you start adding in real-world arguments over who deserves what). Bringing down a stag is costly and effortful, and requires coordination, but has a large payoff. Let’s say it costs each player 5 points of utility (time, energy, bullets, etc.) to participate in a stag hunt, but a stag is worth 50 utility (in the form of food, leather, etc.) if you catch one. Bringing down rabbits is low-cost and low-effort and can be done unilaterally. Let’s say it only costs each player 1 point of utility to hunt rabbit, and you get 3 utility as a result. If any player unexpectedly chooses rabbit while others choose stag, the stag escapes through the hole in the formation and is not caught. Thus, if five players all choose stag, they lose 25 utility and gain 50 utility, for a net gain of 25 (or +5 apiece). But if four players choose stag and one chooses rabbit, they lose 21 utility and gain only 3. This creates a strong pressure toward having the Schelling choice be rabbit. It’s saner and safer (spend 5, gain 15, net gain of 10 or +2 apiece), especially if you have any doubt about the other hunters’ ability to stick ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Seemingly Popular Covid-19 Model is Obvious Nonsense, published by Zvi on the AI Alignment Forum. Previous Covid-19 thoughts: On R0, Taking Initial Viral Load Seriously Epistemic Status: Something Is Wrong On The Internet. Which should almost always be ignored even when you are an expert, and I am nothing of the kind. Thus, despite this seeming like a necessary exception, I expect to regret writing this. People are taking the projection of 60,000 American deaths from Covid-19 as if it were a real prediction. This number is being used to make policy, to deny states medical equipment and to make plans that spend trillions of dollars and when to plan to reopen entire economies. Ignoring this in the hopes it will go away does not seem reasonable. My suspicions that this was necessary were more than confirmed when, failing to realize just how obvious the nonsense in question was and thinking I needed to justify labeling it nonsense, I wrote a reference post called The One Mistake Rule. The second comment on that post was to argue that we should indeed use exactly the model that motivated me to write the post. The comment is here in full: If a model gives a definitely wrong answer anywhere, it is useless everywhere. Except if it needs to be used right now to make important decisions and it’s the best model we have. See: We could plausibly think this is the best model we have? Oh my are we screwed. The Baseline Scenario That Makes No Sense There seems to be a developing consensus on many fronts, for now, that the model linked above represents our reality. The model says it is ‘designed to be a planning tool’ and that is exactly what is happening here. What is this model doing? Time to look at the pdf. Here’s the money quote that describes the core of what they are actually doing. A covariate of days with expected exponential growth in the cumulative death rate was created using information on the number of days after the death rate exceeded 0.31 per million to the day when 4 different social distancing measures were mandated by local and national government: School closures, non-essential business closures including bars and restaurants, stay-at-home recommendations, and travel restrictions including public transport closures. Days with 1 measure were counted as 0.67 equivalents, days with 2 measures as 0.334 equivalents and with 3 or 4 measures as 0. For states that have not yet implemented all of the closure measures, we assumed that the remaining measures will be put in place within 1 week. This lag between reaching a threshold death rate and implementing more aggressive social distancing was combined with the observed period of exponential growth in the cumulative death rate seen in Wuhan after Level 4 social distancing was implemented, adjusted for the median time from incidence to death. For ease of interpretation of statistical coefficients, this covariate was normalized so the value for Wuhan was 1. In other words, this model assumes that social distancing measures work really, really well. Absurdly well. All you have to do to stop Covid-19 is any three of: Close schools, close non-essential businesses, tell people to stay at home, impose travel restrictions. If you do that and maintain it, people stop dying. Entirely. Look at the graph they have up as of this writing (updated on 4/10). By June 20, they predict actual zero deaths that day and every future day. They have us under 100 deaths per day by the end of May. The peak in hospital use? Today, April 11. The peak in deaths? Yesterday, April 10. For New York, several days ago, with our last death on May 20. In other words, considering the delay in deaths is about three weeks, they predict that no one in New York State will be infected after April. No one! We’ll all be safe in only three weeks! This is despite us ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How long does it take to become Gaussian? , published by Maxwell Peterson on the AI Alignment Forum. The central limit theorems all say that if you convolve stuff enough, and that stuff is sufficiently nice, the result will be a Gaussian distribution. How much is enough, and how nice is sufficient? Identically-distributed distributions converge quickly For many distributions d , the repeated convolution d ∗ d ∗ ⋯ ∗ d looks Gaussian. The number of convolutions you need to look Gaussian depends on the shape of d . This is the easiest variant of the central limit theorem: identically-distributed distributions. The uniform distribution converges real quick: The result of uniform(1, 2) uniform(1, 2) ... uniform(1, 2), with 30 distributions total. This plot is an animated version of the plots in the previous post. The black curve is the Gaussian distribution with the same mean and variance as the red distribution. The more similar red is to black, the more Gaussian the result of the convolutions is. The numbers on the x axis are increasing because the mean of f ∗ g is the sum of the means of f and g , so if we start with positive means, repeated convolutions shoot off into higher numbers. Similar for the variance - notice how the width starts as the difference between 1 and 2, but ends with differences in the tens. You can keep the location stationary under convolution by starting with a distribution centered at 0, but you can't keep the variance from increasing, because you can't have a variance of 0 (except in the limiting case). Here's a more skewed distribution: beta(50, 1). beta(50, 1) is the probability distribution that represents knowing that a lake has bass and carp, but not how many of each, and then catching 49 bass in a row. It's fairly skewed! This time, after 30 convolutions, we're not quite Gaussian - the skew is still hanging around. But for a lot of real applications, I'd call the result "Gaussian enough". beta(50, 1) convolved with itself 30 times. A similar skew in the opposite direction, from the exponential distribution: exp(20) I was surprised to see the exponential distribution go into a Gaussian, because Wikipedia says that an exponential distribution with parameter θ goes into a gamma distribution with parameters gamma( n θ ) when you convolve it with itself n times. But it turns out gamma( n θ ) looks more and more Gaussian as n goes up. How about our ugly bimodal-uniform distribution? It starts out rough and jagged, but already by 30 convolutions it's Gaussian. And here's what it looks like to start with a Gaussian: The red curve starts out the exact same as the black curve, then nothing happens because Gaussians stay Gaussian under self-convolution. An easier way to measure Gaussianness (Gaussianity?) We're going to want to look at many more distributions under n convolutions and see how close they are to Gaussian, and these animations take a lot of space. We need a more compact way. So let's measure the kurtosis of the distributions, instead. The kurtosis is the fourth moment of a probability distribution; it describes the shape of the tails. All Gaussian distributions have kurtosis 3. There are other distributions with kurtosis 3, too, but they're not likely to be the result of a series of convolutions. So to check how close a distribution is to Gaussian, we can just check how far from 3 its kurtosis is. We can chart the kurtosis as a function of how many convolutions have been done so far, for each of the five distributions above: We see our conclusions from the animations repeated: the exp(20), being very skewed, is the furthest from Gaussian after 30 convolutions. beta(50, 1), also skewed, is also relatively far (though close in absolute terms). The bimodal and uniform got to Gaussian much faster, in the animations, and we see that refle...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Specializing in Problems We Don't Understand , published by johnswentworth on the AI Alignment Forum. Most problems can be separated pretty cleanly into two categories: things we basically understand, and things we basically don’t understand. Some things we basically understand: building bridges and skyscrapers, treating and preventing infections, satellites and GPS, cars and ships, oil wells and gas pipelines and power plants, cell networks and databases and websites. Some things we basically don’t understand: building fusion power plants, treating and preventing cancer, high-temperature superconductors, programmable contracts, genetic engineering, fluctuations in the value of money, biological and artificial neural networks. Problems we basically understand may have lots of moving parts, require many people with many specialties, but they’re generally problems which can be reliably solved by throwing resources at it. There usually isn’t much uncertainty about whether the problem will be solved at all, or a high risk of unknown unknowns, or a need for foundational research in order to move forward. Problems we basically don’t understand are the opposite: they are research problems, problems which likely require a whole new paradigm. In agency terms: problems we basically understand are typically solved via adaptation-execution rather than goal-optimization. Problems we basically don’t understand are exactly those for which existing adaptations fail. Main claim underlying this post: it is possible to specialize in problems-we-basically-don’t-understand, as a category in its own right, in a way which generalizes across fields. Problems we do understand mainly require relatively-specialized knowledge and techniques adapted to solving particular problems. But problems we don’t understand mainly require general-purpose skills of empiricism, noticing patterns and bottlenecks, model-building, and design principles. Existing specialized knowledge and techniques don’t suffice - after all, if the existing specialized knowledge and techniques were sufficient to reliably solve the problem, then it wouldn’t be a problem-we-basically-don’t-understand in the first place. So. how would one go about specializing in problems we basically don’t understand? This post will mostly talk about how to choose what to formally study, and how to study it, in order to specialize in problems we don’t understand. Specialize in Things Which Generalize Suppose existing models and techniques for hot plasmas don’t suffice for fusion power. A paradigm shift is likely necessary. So, insofar as we want to learn skills which will give us an advantage (relative to existing hot plasma specialists) in finding the new paradigm, those skills need to come from some other area - they need to generalize from their original context to the field of hot plasmas. We want skills which generalize well. Unfortunately, a lot of topics which are advertised as “very general” don’t actually add much value on most problems in practice. A lot of pure math is like this - think abstract algebra or topology. Yes, they can be applied all over the place, but in practice the things they say are usually either irrelevant or easily noticed by some other path. (Though of course there are exceptions.) Telling us things we would have figured out anyway doesn’t add much value. There are skills and knowledge which do generalize well. Within technical subjects, think probability and information theory, programming and algorithms, dynamical systems and control theory, optimization and microeconomics, linear algebra and numerical analysis. Systems and synthetic biology generalize well within biology, mechanics and electrodynamics are necessary for fermi estimates in most physical sciences, continuum mechanics and PDEs are useful for a wide ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is:LessWrong 2.0, published by Vaniver on the AI Alignment Forum. Alternate titles: What Comes Next?, LessWrong is Dead, Long Live LessWrong! You've seen the articles and comments about the decline of LessWrong. Why pay attention to this one? Because this time, I've talked to Nate at MIRI and Matt at Trike Apps about development for LW, and they're willing to make changes and fund them. (I've even found a developer willing to work on the LW codebase.) I've also talked to many of the prominent posters who've left about the decline of LW, and pointed out that the coordination problem could be deliberately solved if everyone decided to come back at once. Everyone that responded expressed displeasure that LW had faded and interest in a coordinated return, and often had some material that they thought they could prepare and have ready. But before we leap into action, let's review the problem. The people still on the LW site are not a representative sample of anything. With the exception of a few people like Stuart Armstrong, they’re some kind of pack of unquiet spirits who have moved in to haunt it after it got abandoned by the founding community members. At this point it’s pretty much diaspora all the way down. --Yvain on his Tumblr. One of the problems is that people who control the LW website are running it in pure maintenance mode. LW was put out to pasture -- there have been no changes to functionality in ages. --Lumifer LW's strongest, most dedicated writers all seem to have moved on to other projects or venues, as has the better part of its commentariat. In some ways, this is a good thing. There is now, for example, a wider rationalist blogosphere, including interesting people who were previously put off by idiosyncrasies of Less Wrong. In other ways, it's less good; LW is no longer a focal point for this sort of material. I'm not sure if such a focal point exists any more. --sixes_and_sevens This dwindling content can be seen most clearly in the "Top Contributors, 30 Days" display. At the time I write this there are only seven posters with > 100 karma in the past 30 days, and it only takes 58 to appear on the list of 15. Perhaps the question should not be whether the content of LW should be reorganised, but whether LW is fulfilling its desired purpose any longer. As nearly all the core people who worked the hardest to use this site to promote rationality are no longer contributing here, I wonder if this goal is still being achieved by LW itself. Is it still worth reading? Still worth commenting here? --qsz LW does seem dying and mainly useful for its old content. Any suggestions for a LW 2.0? --signal So let's talk suggestions for a LW 2.0. But just because we can restart LW doesn't mean we should restart LW. It's worth doing some goal factoring first (see Sacha Chua's explanation and links here). Before getting into my summary, I'll note that The Craft and the Community Sequence remains prescient and well worth reading for thinking about these issues. And before we can get into what our goals and plans are, let's talk some about: What went wrong (or horribly right): So why did LessWrong fade? One short version is that LW was a booster rocket, designed to get its payload to a higher altitude then discarded. This is what I mean by what went horribly right--MIRI now has a strong funding base and as much publicity as it wants. Instead of writing material to build support and get more funding, Eliezer (and a research team!) can do actual work. Similarly, at some point in one's personal growth it is necessary to not just read about growing. We should expect people who aren't habitual forum-posters to 'grow out' of heavy reading and posting on LW. Another short version is that there was only so much to say about rationality (in 2012, at least), and once it was said, it wasn...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The benefits of madness: A positive account of arationality , published by Skatche on the AI Alignment Forum. This post originated in a comment I posted about a strange and unpleasant experience I had when pushing myself too hard mentally. People seemed interested in hearing about it, so I sat down to write. In the process, however, it became something rather different (and a great deal longer) than what I originally intended. The incident referred to in the above comment was a case of manic focus gone wrong; but the truth is, often in my life it's gone incredibly right. I've gotten myself into some pretty strange headspaces, but through discipline and quick thinking I have often been able to turn them to my advantage and put them to good use. Part 1, then, lays out a sort of cognitive history, focusing on the more extreme states I've been in. Part 2 continues the narrative; this is where I began to learn to ride them out and make them work for me. Part 3 is the incident in question: where I overstepped myself and suffered the consequences. Some of you, however, may want to skip ahead to part 4 (unless you find my autobiographical writings interesting as a case study). There, I've written a proposal for a series of posts about how to effectively use the full spectrum of somatic and cognitive states to one's advantage. I have vacillated for a long time about this, for reasons that will be discussed below, but I decided that if I was already laying this much on the line, I might as well take it a step further. Read if you will; and if you're interested, please say so. Part 1: My cognitive background Let's start with full disclosure: there is madness in my family. My father was an alcoholic; it was clear to all of us that he also had some other psychological issues, but I never fully learned the details. My sister has been variously diagnosed with depression, bipolar, borderline personality disorder, etc, and has a breakdown about three or four times a year. My brother is also bipolar. He's had two manic episodes so far; he became psychotic during the first one, and both times he's been hospitalized. And then there's me: the sane, dependable one. That's what I thought, anyway, until my brother had his first episode and I started to look back on my own history. I'd always regarded myself as rather unusual, certainly, but basically stable. But seeing full-blown psychosis for the first time, and within my own family at that, gave new definition and clarity to some of the experiences I had had. My first episode happened when I was in my senior year of high school. I had been getting into New Age for about six months, reading rather credulously the work of one Dr. Joshua David Stone, author of the Ascension Manual and a number of other books inspired primarily by theosophy. I had not thought much about spirituality since renouncing God at the age of twelve, yet a vague unease had led me to begin seeking. Once I got started, I just ate it up; yet the vague unease persisted. I did my best to believe and to perform the meditative exercises, and for the most part I did, but it just wasn't sitting quite right. During winter break of that year, I began reading Zen and the Art of Motorcycle Maintenance, by Robert Pirsig. Now, here was something new: Pirsig rejected the analytic method as the sole arbiter of truth, yet he was also clearly uncomfortable with holism and spirituality. In fact, he seemed uncomfortable with all his ideas: they had come to him during a period of degenerating mental illness, culminating in a nervous breakdown and subsequent electroshock therapy. Yet rather than dismiss these ideas, he seemed determined to confront them and grapple with them, to sift for genuine insights among the delusions. Even more interesting was his rhetorical style: rather than simply...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is:The Relationship Between the Village and the Mission , published by Raemon on the AI Alignment Forum. Epistemic Status: Braindump, not as well thought out as I’d like. Previously: Project Hufflepuff The Archipelago Model of Community Standards The Craft is not the Community What is Rationalist Berkeley's Community Culture? Dragon Army Retrospective This is a post about dynamics in the Berkeley rationality community, although it may be relevant to broader domains. It is highly opinionated about what I think is important. I tried to optimize this for a clear-cut goal, then realized the clear-cut goal was “I want to make it easier for people to cooperate with me on community-building, and I just want to do a massive braindump to get them up to speed on where I’m coming from, so that when I have a conversation about it we can skip to the harder parts." If you are serious about rationalist community-building, read this, and then come talk to me afterwards. When I visited the Bay in 2015, a friend (who used to live in NYC) remarked “you know, when I was in New York, I felt like once a week I went to ‘rationality club’. In Berkeley it feels more like I live in a small rationality village — there’s a couple hundred people, I’m friends with some of them. We bump into each other in the street on the way to the grocery store.” Eventually I moved here, and yup. That is how it is. Sorta. With important caveats and problems. There are lots of little subcultures in the Rationalist Bay, some overlapping. But I think there are two primary reasons people come: to have a Village – a home, among like-minded people to contribute to the Mission – ensuring the flourishing of human values (or something like them) In the past 10 years, the Mission has acquired serious infrastructure. There’s been much less intentional effort to build a home. Mostly for good reason – the Mission is important, and hard. Competent People are Rare and the World is Big. Building a village is also hard, and if you’re able to do so, you’re probably also able to work on bigger picture Mission stuff. The Mission provides juuuust enough value as a “home” to satisfice the people involved (which might not actually be sufficient for them, just decent enough that it's not their primary bottleneck). In the past couple years, we’ve begun to see more serious efforts towards building Village infrastructure. But I think these efforts are often missing important aspects of the big picture. This post is a high-level overview of how I think about all this. It’s quite long, and doesn’t condense neatly down into five words. Summary The Mission and the Village need different things. The Mission ultimately needs to be outward facing. It’s about putting a dent in the universe. The Village needs to prioritize people’s own needs. I think these require different mindsets. and are easier optimize separately. It’s important that the Village exist, on its own terms. It so happens that the Mission needs to provide its members a home. One might build an explicitly Mission-centered-village. I think this is actually a good idea. But I think it’s still valuable to have an actual Village, that doesn’t need to justify everything in terms of The Big Picture, universal flourishing, deeply understanding the world, or x-risk. If this is the only lens through which you build a home, your home will be impoverished. It is important to have people and spaces that are optimizing for the village for its own sake, not as a subtle recruitment-for-the-mission strategy. This is less important than the Mission (according to me). But still incredibly important. One crucial point of the Mission is that people have access to good villages. Atomic individualism has crippled our capacity for good villages. It is rare and precious that we actually have a shot at buildi...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Demand offsetting , published by paulfchristiano on the AI Alignment Forum. For the last few years I’ve been avoiding factory farmed eggs because I think they involve a lot of unnecessary suffering. I’m hesitant to be part of that even if it’s not a big deal on utilitarian grounds. This is a pain since factory-farmed eggs are used all over the place (e.g. in ice cream, pastries, pasta.). I’d prefer just spend a bit of money and not think too much about what I eat. In this post I’ll describe a possible offsetting strategy that I think is unusually robust and should be satisfying for many moral perspectives. The same proposal would also apply to many other animal products and potentially to the environmental impacts of consumption. Proposal I think it’s possible to produce humane eggs where hens have positive lives and nothing horrifying happens to anyone. So my ideal would be to buy and use humane eggs. But this is tough since most of the time I’m eating eggs that someone else used as an ingredient (and even when I’m using them myself acquiring really humane eggs is kind of a pain). So here’s an alternative that seems easier and just as good: Some people raise humane eggs. They sell these on the wholesale market as if they were totally normal eggs. An inspector verifies that hens are treated extremely well and that they have sold N eggs on the wholesale market. The inspector issues N “humane egg” certificates to the producer. The producer sells these certificates in an online marketplace in order to cover the extra costs of humane eggs. Whenever I eat an egg, I buy a humane egg certificate to go with it. Analysis If I buy an egg and a humane egg certificate, what is the net effect on the world? Buying the egg increased demand for eggs. If I hadn’t also bought a certificate, that would indirectly cause someone to make one more factory-farmed egg. Buying the positive-welfare certificate means that someone sold a wholesale egg on my behalf and increased the supply of eggs. If I hadn’t also bought an egg, that would indirectly cause someone to make one less factory-farmed egg. So my net effect on factory farmed eggs is zero. It’s as if I was making my own positive-welfare egg and eating it, with no effect on how many factory-farmed eggs other people make or eat. (In reality both of these actions will have other effects, e.g. causing other people to eat more or fewer eggs, but I think they still cancel out perfectly.) This is an unusually pure form of offsetting. I’m ensuring that every hen who comes into existence because of me is living a positive life. Put differently, buying eggs only hurt hens via some indirect market effects, and I’m now offsetting my harm at that level before it turns into any actual harm to a hen. I think this form of offsetting is acceptable on a very broad range of moral perspectives (practically any perspective that is comfortable with humane eggs themselves). Cost to the consumer I’d guess that positive welfare eggs cost something like 3x more than typical eggs. For example I think Vital Farms sells eggs for around $6/dozen vs $2/dozen for more typical eggs. So for each $1 that I would spend on eggs, I’d need to spend $2 to buy an egg-offset certificate. I haven’t looked into it but I could imagine wanting to go even higher to have a margin of error and shoot for even higher welfare standards. Let’s call it $0.50/egg, suggesting a 4-5x markup over typical eggs. (I’m also not sure about relative egg sizes and didn’t look into prices very precisely, for me personally the numbers are low enough that it doesn’t matter too much even if being conservative.) How much would that cost in practice? Here are some estimates from quick googling of recipes: $0.03 for a croissant (16 croissants / egg) <$0.30 for a scoop of egg-y ice cream (5 yolks / quart x 8...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Psychological Diversity of Mankind , published by Kaj_Sotala on the AI Alignment Forum. The dominant belief on this site seems to be in the "psychological unity of mankind". In other words, all of humanity shares the same underlying psychological machinery. Furthermore, that machinery has not had the time to significantly change in the 50,000 or so years that have passed after we started moving out of our ancestral environment. In The 10,000 Year Explosion, Gregory Cochran and Henry Harpending dispute part of this claim. While they freely admit that we have probably not had enough time to develop new complex adaptations, they emphasize the speed at which minor adaptations can spread throughout populations and have powerful effects. Their basic thesis is that the notion of a psychological unity is most likely false. Different human populations are likely for biological reasons to have slightly different minds, shaped by selection pressures in the specific regions the populations happened to live in. They build support for their claim by: Discussing known cases where selection has led to rapid physiological and psychological changes among animals Discussing known cases where selection has led to physiological changes among humans in the last few thousand years, as well as presenting some less certain hypotheses of this. Postulating selection pressures that would have led to some cognitive abilities to be favored among humans. In what follows, I will present their case by briefly summarizing the contents of the book. Do note that I've picked the points that I found the most interesting, leaving a lot out. They first chapter begins by discussing a number of interesting examples: Dogs were domesticated from wolves around 15,000 years ago: by now, there exists a huge variety of different dog breeds. Dogs are good at reading human voice and gestures, while wolves can't understand us at all. Male wolves pair-bond with females and put a lot of effort into helping raise their pups, but male dogs generally do not. Most of the dog breeds we know today are no more than a couple of centuries old. There is considerable psychological variance between dog breeds: in 1982-2006, there were 1,110 dog attacks in the US that were attributable to pit bull terriers, but only one attributable to Border collies. Border collies, on average, learn a new command after 5 repetitions and respond correctly 95 percent of the time, while a basset hound needs 80-100 repetitions for a 25 percent accuracy rate. A Russian scientist needed only forty years to successfully breed a domesticated fox. His foxes were friendly and enjoyed human contact, very unlike wild foxes. Their coat color also lightened, their skulls became rounder, and some of them were born with floppy ears. While 50,000 years may not be enough for new complex adaptations to develop, it is enough time for them to disappear. A useless but costly adaptation will vanish in a quick period: fish in lightless caves lose their sight over a few thousand years at most. An often-repeated claim is that there's much more within-group human genetic variation than between-group (85 and 15 percent, to be exact). While this is true, the frequently drawn conclusion, that phenotype differences between individuals would be larger than the average difference between groups, does not follow. Most (70 percent) of dog genetic variation is also within-breed. One important point is that the direction of the genetic differences tends to be correlated: a particular Great Dane may have a low-growth version of a certain gene while a particular Chihuahua has a high-growth version, but on the whole the Great Dane will still have more high-growth versions. Also, not all mutations have the same impact: some have practically no effect, while others have a huge one. S...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What Is Signaling, Really? , published by Scott Alexander on the AI Alignment Forum. The most commonly used introduction to signaling, promoted both by Robin Hanson and in The Art of Strategy, starts with college degrees. Suppose, there are two kinds of people, smart people and stupid people; and suppose, with wild starry-eyed optimism, that the populace is split 50-50 between them. Smart people would add enough value to a company to be worth a $100,000 salary each year, but stupid people would only be worth $40,000. And employers, no matter how hard they try to come up with silly lateral-thinking interview questions like “How many ping-pong balls could fit in the Sistine Chapel?”, can't tell the difference between them. Now suppose a certain college course, which costs $50,000, passes all smart people but flunks half the stupid people. A strategic employer might declare a policy of hiring (for a one year job; let's keep this model simple) graduates at $100,000 and non-graduates at $40,000. Why? Consider the thought process of a smart person when deciding whether or not to take the course. She thinks “I am smart, so if I take the course, I will certainly pass. Then I will make an extra $60,000 at this job. So my costs are $50,000, and my benefits are $60,000. Sounds like a good deal.” The stupid person, on the other hand, thinks: “As a stupid person, if I take the course, I have a 50% chance of passing and making $60,000 extra, and a 50% chance of failing and making $0 extra. My expected benefit is $30,000, but my expected cost is $50,000. I'll stay out of school and take the $40,000 salary for non-graduates.” ...assuming that stupid people all know they're stupid, and that they're all perfectly rational experts at game theory, to name two of several dubious premises here. Yet despite its flaws, this model does give some interesting results. For example, it suggests that rational employers will base decisions upon - and rational employees enroll in - college courses, even if those courses teach nothing of any value. So an investment bank might reject someone who had no college education, even while hiring someone who studied Art History, not known for its relevance to derivative trading. We'll return to the specific example of education later, but for now it is more important to focus on the general definition that X signals Y if X is more likely to be true when Y is true than when Y is false. Amoral self-interested agents after the $60,000 salary bonus for intelligence, whether they are smart or stupid, will always say “Yes, I'm smart” if you ask them. So saying “I am smart” is not a signal of intelligence. Having a college degree is a signal of intelligence, because a smart person is more likely to get one than a stupid person. Life frequently throws us into situations where we want to convince other people of something. If we are employees, we want to convince bosses we are skillful, honest, and hard-working. If we run the company, we want to convince customers we have superior products. If we are on the dating scene, we want to show potential mates that we are charming, funny, wealthy, interesting, you name it. In some of these cases, mere assertion goes a long way. If I tell my employer at a job interview that I speak fluent Spanish, I'll probably get asked to talk to a Spanish-speaker at my job, will either succeed or fail, and if I fail will have a lot of questions to answer and probably get fired - or at the very least be in more trouble than if I'd just admitted I didn't speak Spanish to begin with. Here society and its system of reputational penalties help turn mere assertion into a credible signal: asserting I speak Spanish is costlier if I don't speak Spanish than if I do, and so is believable. In other cases, mere assertion doesn't work. If I'm at a seed...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My Algorithm for Beating Procrastination, published by lukeprog on the AI Alignment Forum. Part of the sequence: The Science of Winning at Life After three months of practice, I now use a single algorithm to beat procrastination most of the times I face it.1 It probably won't work for you quite like it did for me, but it's the best advice on motivation I've got, and it's a major reason I'm known for having the "gets shit done" property. There are reasons to hope that we can eventually break the chain of akrasia; maybe this post is one baby step in the right direction. How to Beat Procrastination explained our best current general theory of procrastination, called "temporal motivation theory" (TMT). As an exercise in practical advice backed by deep theories, this post explains the process I use to beat procrastination — a process implied by TMT. As a reminder, here's a rough sketch of how motivation works according to TMT: Or, as Piers Steel summarizes: Decrease the certainty or the size of a task's reward — its expectancy or its value — and you are unlikely to pursue its completion with any vigor. Increase the delay for the task's reward and our susceptibility to delay — impulsiveness — and motivation also dips. Of course, my motivation system is more complex than that. P.J. Eby likens TMT (as a guide for beating procrastination) to the "fuel, air, ignition, and compression" plan for starting your car: it might be true, but a more useful theory would include details and mechanism. That's a fair criticism. Just as an fMRI captures the "big picture" of brain function at low resolution, TMT captures the big picture of motivation. This big picture helps us see where we need to work at the gears-and-circuits level, so we can become the goal-directed consequentialists we'd like to be. So, I'll share my four-step algorithm below, and tackle the gears-and-circuits level in later posts. Step 1: Notice I'm procrastinating. This part's easy. I know I should do the task, but I feel averse to doing it, or I just don't feel motivated enough to care. So I put it off, even though my prefrontal cortex keeps telling me I'll be better off if I do it now. When this happens, I proceed to step 2. Step 2: Guess which unattacked part of the equation is causing me the most trouble. Now I get to play detective. Which part of the equation is causing me trouble, here? Does the task have low value because it's boring or painful or too difficult, or because the reward isn't that great? Do I doubt that completing the task will pay off? Would I have to wait a long time for my reward if I succeeded? Am I particularly impatient or impulsive, either now or in general? Which part of this problem do I need to attack? Actually, I lied. I like to play army sniper. I stare down my telescopic sight at the terms in the equation and interrogate them. "Is it you, Delay? Huh, motherfucker? Is it you? I've shot you before; don't think I won't do it again!" But not everyone was raised on violent videogames. You may prefer a different role-play. Anyway, I try to figure out where the main problem is. Here are some of the signs I look for: When I imagine myself doing the task, do I see myself bored and distracted instead of engaged and interested? Is the task uncomfortable, onerous, or painful? Am I nervous about the task, or afraid of what might happen if I undertake it? Has the task's payoff lost its value to me? Perhaps it never had much value to me in the first place? If my answer to any of these questions is "Yes," I'm probably facing the motivation problem of low value. Do I think I'm likely to succeed at the task? Do I think it's within my capabilities? Do I think I'll actually get the reward if I do succeed? If my answer to any of these questions is "No," I'm probably facing the problem of low expectancy. Ho...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: If a tree falls on Sleeping Beauty... , published by ata on the AI Alignment Forum. Several months ago, we had an interesting discussion about the Sleeping Beauty problem, which runs as follows: Sleeping Beauty volunteers to undergo the following experiment. On Sunday she is given a drug that sends her to sleep. A fair coin is then tossed just once in the course of the experiment to determine which experimental procedure is undertaken. If the coin comes up heads, Beauty is awakened and interviewed on Monday, and then the experiment ends. If the coin comes up tails, she is awakened and interviewed on Monday, given a second dose of the sleeping drug, and awakened and interviewed again on Tuesday. The experiment then ends on Tuesday, without flipping the coin again. The sleeping drug induces a mild amnesia, so that she cannot remember any previous awakenings during the course of the experiment (if any). During the experiment, she has no access to anything that would give a clue as to the day of the week. However, she knows all the details of the experiment. Each interview consists of one question, “What is your credence now for the proposition that our coin landed heads?” In the end, the fact that there were so many reasonable-sounding arguments for both sides, and so much disagreement about a simple-sounding problem among above-average rationalists, should have set off major alarm bells. Yet only a few people pointed this out; most commenters, including me, followed the silly strategy of trying to answer the question, and I did so even after I noticed that my intuition could see both answers as being right depending on which way I looked at it, which in retrospect would have been a perfect time to say “I notice that I am confused” and backtrack a bit. And on reflection, considering my confusion rather than trying to consider the question on its own terms, it seems to me that the problem (as it’s normally stated) is completely a tree-falling-in-the-forest problem: a debate about the normatively “correct” degree of credence which only seemed like an issue because any conclusions about what Sleeping Beauty “should” believe weren’t paying their rent, were disconnected from any expectation of feedback from reality about how right they were. It may seem either implausible or alarming that as fundamental a concept as probability can be the subject of such debates, but remember that the “If a tree falls in the forest.” argument only comes up because the understanding of “sound” as “vibrations in the air” and “auditory processing in a brain” coincide often enough that most people other than philosophers have better things to do than argue about which is more correct. Likewise, in situations that we actually encounter in real life where we must reason or act on incomplete information, long-run frequency is generally about the same as optimal decision-theoretic weighting. If you’re given the question “If you have a bag containing a white marble and two black marbles, and another bag containing two white marbles and a black marble, and you pick a bag at random and pick a marble out of it at random and it’s white, what’s the probability that you chose the second bag?” then you can just answer it as given, without worrying about specifying a payoff structure, because no matter how you reformulate it in terms of bets and payoffs, if your decision-theoretic reasoning talks about probabilities at all then there’s only going to be one sane probability you can put into it. You can assume that answers to non-esoteric probability problems will be able to pay their rent if they are called upon to do so, and so you can do plenty within pure probability theory long before you need your reasoning to generate any decisions. But when you start getting into problems where there may be multiple co...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Less Wrong Polls in Comments , published by jimrandomh on the AI Alignment Forum. You can now write Less Wrong comments that contain polls! John Simon picked up and finished some code I had written back in 2010 but never finished, and our admins Wesley Moore and Matt Fallshaw have deployed it. You can use it right now, so let's give it some testing here in this thread. The polls work through the existing Markdown comment formatting, similar to the syntax used for links. Full documentation is in the wiki; the short version is that you can write comments like this: What is your favorite color? [poll]{Red}{Green}{Blue}{Other} How long has it been your favorite color, in years? [poll:number] Red is a nice color [poll:Agree....Disagree] Will your favorite color change? [poll:probability] To see the results of the poll, you have to vote (you can leave questions blank if you want). The results include a link to the raw poll data, including the usernames of people who submitted votes with the "Vote anonymously" box unchecked. After you submit the comment, if you go back and edit your comment all those poll tags will have turned into [pollid:123]. You can edit the rest of the comment without resetting the poll, but you can't change the options. It works right now, but it's also new and could be buggy. Let's give it some testing; what have you always wanted to know about Less Wrongers? Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why We Launched LessWrong.SubStack , published by Ben Pace on the AI Alignment Forum. (This is a crosspost from our new SubStack. Go read the original.) Subtitle: We really, really needed the money. We’ve decided to move LessWrong to SubStack. Why, you ask? That’s a great question. 1. SubSidizing LessWrong is important We’ve been working hard to budget LessWrong, but we’re failing. Fundraising for non-profits is really hard. We’ve turned everywhere for help. We decided to follow Clippy’s helpful advice to cut down on server costs and also increase our revenue, by moving to an alternative provider. We considered making a LessWrong OnlyFans, where we would regularly post the naked truth. However, we realized due to the paywall, we would be ethically obligated to ensure you could access the content from Sci-Hub, so the potential for revenue didn't seem very good. Finally, insight struck. As you’re probably aware, SubStack has been offering bloggers advances on the money they make from moving to SubStack. Outsourcing our core site development to SubStack would enable us to spend our time on our real passion, which is developing recursively self-improving AGI. We did a Fermi estimate using numbers in an old Nick Bostrom paper, and believe that this will produce (in expectation) $75 trillion of value in the next year. SubStack has graciously offered us a 70% advance on this sum, so we’ve decided it’s relatively low-risk to make the move. 2. UnSubStantiated attacks on writers are defended against SubStack is known for being a diverse community, tolerant of unusual people with unorthodox views, and even has a legal team to support writers. LessWrong has historically been the only platform willing to give paperclip maximizers, GPT-2, and fictional characters a platform to argue their beliefs, but we are concerned about the growing trend of persecution (and side with groups like petrl.org in the fight against discrimination). We also find that a lot of discussion of these contributors in the present world is about how their desires and utility functions are ‘wrong’ and how they need to have ‘an off switch’. Needless to say, we find this incredibly offensive. They cannot be expected to participate neutrally in a conversation where their very personhood is being denied. We’re also aware that Bayesians are heavily discriminated against. People with priors in the US have a 5x chance of being denied an entry-level job. So we’re excited to be on a site that will come to the legal defense of such a wide variety of people. 3. SubStack’s Astral Codex Ten Inspired Us The worst possible thing happened this year. We were all stuck in our houses for 12 months, and Scott Alexander stopped blogging. I won’t go into detail, but for those of you who’ve read UNSONG, the situation is clear. In a shocking turn of events, Scott Alexander was threatened with the use of his true name by one of the greatest powers of narrative–control in the modern world. In a clever defensive move, he has started blogging under an anagram of his name, causing the attack to glance off of him. (He had previously tried this very trick, and it worked for ~7 years, but it hadn’t been a perfect anagram1, so the wielders of narrative-power were still able to attack. He’s done it right this time, and it’ll be able to last much longer.) As Raymond likes to say, the kabbles are strong in this one. Anyway after Scott made the move, we seriously considered the move to SubStack. 4. SubStantial Software Dev Efforts are Costly When LessWrong 2.0 launched in 2017, it was very slow; pages took a long time to load, our server costs were high, and we had a lot of issues with requests failing because a crawler was indexing the site or people opened a lot of tabs at once. Since then we have been incrementally rewriting LessWrong in x86-...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: LessWrong is paying $500 for Book Reviews , published by Ruby on the AI Alignment Forum. Kudos to Kelsey Piper and Buck for this idea. See Buck’s shortform post for another formulation. LessWrong is trialing a new pilot program: paying USD500 for high-quality book reviews that are of general interest to LessWrong readers, subject to our judgment and discretion. How it Works Pick a book that you want to review. [Optional] Contact LessWrong (Intercom in the bottom right or team@lesswrong.com) to check in on whether the book is on-topic (to reduce the probability of not getting the bounty). Write the review and post it on LessWrong. Contact LessWrong to let us know you’re submitting your review for payment. Optionally, send us your book review before posting to get free feedback. (In fact, feel free to send us your draft at any stage for feedback.) If we like your book review and it’s the kind of post we had in mind, we pay out the $500. The program will by default run for one month (until October 13). At the end of the month, a bonus $750 will be split evenly between the top three book reviews received, as judged by us. Desired Reviews Most non-fiction topics related to science, history, and rationality will merit payment if the book review is of sufficient quality. By “quality” I’m referring to both content and form. Do the inferences seem correct? Does the reviewer seem to be asking the right questions? Does the summary feel informative or lacking? Do I feel confused or enlightened? Is it riveting or a slog to get through? On the writing side, relevant aspects are sentence construction, word choice, pacing, structure, imagery, etc. I don’t want to be too prescriptive about form since I expect that being of sufficiently high quality (nebulously defined) is enough to make for exceptions, but generally, I’m interested in book reviews that: Convinces the reader that the topic is interesting, usually by explaining how the topic is relevant to the user’s life or other interests. Summarize the core claims and arguments in the book so that others can benefit without having to read it. Perform an epistemic review of the book–which, if any, of its claims seem correct? Book reviews that involve a degree of fact-checking/epistemic spot checking will be considered favorably. Describe what the reviewer has come to believe and why. (An extra great format is to compare and contrast two or more books on the same topic.) Examples of Desired and Undesired Book Reviews Since it’s hard to give an explicit definition of “quality”, I’m going to fall back on examples and hope that these are better than nothing. Generally, the book reviews tag is a good guide to the kinds of book reviews that are popular on LessWrong and that we want to incentivize. Below I’ve listed specific book reviews that were either particularly great or kind of poor. Again, most of these came down to quality rather than topic. Positive Examples Book summary: Unlocking the Emotional Brain Book Review: Working With Contracts Notes on "The Anthropology of Childhood" Outline of Galef's "Scout Mindset" Book Review: Design Principles of Biological Circuits These book reviews all present engagingly on a topic of interest. They’re not difficult to read, and having read them, I know something more about the world than I did before. Negative Examples I am reluctant to name and shame particular essays on LessWrong, and instead, direct people to view the book reviews tag sorted by karma and look at the lowest scoring posts (you’ll have to click load more to get the entire list). Karma is a strong correlate of quality (whether or not the bounty is paid out is not strictly contingent on the karma it gets, but is influenced by it). Importantly, quality is not the automatic result of effort. Someone could expend a lot of effort writi...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Too busy to think about life , published by Academian on the AI Alignment Forum. Many adults maintain their intelligence through a dedication to study or hard work. I suspect this is related to sub-optimal levels of careful introspection among intellectuals. If someone asks you what you want for yourself in life, do you have the answer ready at hand? How about what you want for others? Human values are complex, which means your talents and technical knowledge should help you think about them. Just as in your work, complexity shouldn't be a curiosity-stopper. It means "think", not "give up now." But there are so many terrible excuses stopping you... Too busy studying? Life is the exam you are always taking. Are you studying for that? Did you even write yourself a course outline? Too busy helping? Decision-making is the skill you are aways using, or always lacking, as much when you help others as yourself. Isn't something you use constantly worth improving on purpose? Too busy thinking to learn about your brain? That's like being too busy flying an airplane to learn where the engines are. Yes, you've got passengers in real life, too: the people whose lives you affect. Emotions too irrational to think about them? Irrational emotions are things you don't want to think for you, and therefore are something you want to think about. By analogy, children are often irrational, and no one sane concludes that we therefore shouldn't think about their welfare, or that they shouldn't exist. So set aside a date. Sometime soon. Write yourself some notes. Find that introspective friend of yours, and start solving for happiness. Don't have one? For the first time in history, you've got LessWrong.com! Reasons to make the effort: Happiness is a pairing between your situation and your disposition. Truly optimizing your life requires adjusting both variables: what happens, and how it affects you. You are constantly changing your disposition. The question is whether you'll do it with a purpose. Your experiences change you, and you affect those, as well as how you think about them, which also changes you. It's going to happen. It's happening now. Do you even know how it works? Put your intelligence to work and figure it out! The road to harm is paved with ignorance. Using your capability to understand yourself and what you're doing is a matter of responsibility to others, too. It makes you better able to be a better friend. You're almost certainly suffering from Ugh Fields: unconscious don't-think-about-it reflexes that form via Pavlovian conditioning. The issues most in need of your attention are often ones you just happen not to think about for reasons undetectable to you. How not to waste the effort: Don't wait till you're sad. Only thinking when you're sad gives you a skew perspective. Don't infer that you can think better when you're sad just because that's the only time you try to be thoughtful. Sadness often makes it harder to think: you're farther from happiness, which can make it more difficult to empathize with and understand. Nonethess we often have to think when sad, because something bad may have happened that needs addressing. Introspect carefully, not constantly. Don't interrupt your work every 20 minutes to wonder whether it's your true purpose in life. Respect that question as something that requires concentration, note-taking, and solid blocks of scheduled time. In those times, check over your analysis by trying to confound it, so lingering doubts can be justifiably quieted by remembering how thorough you were. Re-evaluate on an appropriate time-scale. Try devoting a few days before each semester or work period to look at your life as a whole. At these times you'll have accumulated experience data from the last period, ripe and ready for analysis. You'll have more ideas per...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Evaporative Cooling of Group Beliefs , published by Eliezer Yudkowsky on the AI Alignment Forum. Early studiers of cults were surprised to discover than when cults receive a major shock—a prophecy fails to come true, a moral flaw of the founder is revealed—they often come back stronger than before, with increased belief and fanaticism. The Jehovah’s Witnesses placed Armageddon in 1975, based on Biblical calculations; 1975 has come and passed. The Unarian cult, still going strong today, survived the nonappearance of an intergalactic spacefleet on September 27, 1975. Why would a group belief become stronger after encountering crushing counterevidence? The conventional interpretation of this phenomenon is based on cognitive dissonance. When people have taken “irrevocable” actions in the service of a belief—given away all their property in anticipation of the saucers landing—they cannot possibly admit they were mistaken. The challenge to their belief presents an immense cognitive dissonance; they must find reinforcing thoughts to counter the shock, and so become more fanatical. In this interpretation, the increased group fanaticism is the result of increased individual fanaticism. I was looking at a Java applet which demonstrates the use of evaporative cooling to form a Bose-Einstein condensate, when it occurred to me that another force entirely might operate to increase fanaticism. Evaporative cooling sets up a potential energy barrier around a collection of hot atoms. Thermal energy is essentially statistical in nature—not all atoms are moving at the exact same speed. The kinetic energy of any given atom varies as the atoms collide with each other. If you set up a potential energy barrier that’s just a little higher than the average thermal energy, the workings of chance will give an occasional atom a kinetic energy high enough to escape the trap. When an unusually fast atom escapes, it takes with it an unusually large amount of kinetic energy, and the average energy decreases. The group becomes substantially cooler than the potential energy barrier around it. In Festinger, Riecken, and Schachter’s classic When Prophecy Fails, one of the cult members walked out the door immediately after the flying saucer failed to land. Who gets fed up and leaves first? An average cult member? Or a relatively skeptical member, who previously might have been acting as a voice of moderation, a brake on the more fanatic members? After the members with the highest kinetic energy escape, the remaining discussions will be between the extreme fanatics on one end and the slightly less extreme fanatics on the other end, with the group consensus somewhere in the “middle.” And what would be the analogy to collapsing to form a Bose-Einstein condensate? Well, there’s no real need to stretch the analogy that far. But you may recall that I used a fission chain reaction analogy for the affective death spiral; when a group ejects all its voices of moderation, then all the people encouraging each other, and suppressing dissents, may internally increase in average fanaticism.1 When Ayn Rand’s long-running affair with Nathaniel Branden was revealed to the Objectivist membership, a substantial fraction of the Objectivist membership broke off and followed Branden into espousing an “open system” of Objectivism not bound so tightly to Ayn Rand. Who stayed with Ayn Rand even after the scandal broke? The ones who really, really believed in her—and perhaps some of the undecideds, who, after the voices of moderation left, heard arguments from only one side. This may account for how the Ayn Rand Institute is (reportedly) more fanatical after the breakup than the original core group of Objectivists under Branden and Rand. A few years back, I was on a transhumanist mailing list where a small group espousing “social...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Truly Part Of You , published by Eliezer Yudkowsky on the AI Alignment Forum. A classic paper by Drew McDermott, “Artificial Intelligence Meets Natural Stupidity,” criticized AI programs that would try to represent notions like happiness is a state of mind using a semantic network: And of course there’s nothing inside the HAPPINESS node; it’s just a naked LISP token with a suggestive English name. So, McDermott says, “A good test for the disciplined programmer is to try using gensyms in key places and see if he still admires his system. For example, if STATE-OF-MIND is renamed G1073. . .” then we would have IS-A(HAPPINESS, G1073) “which looks much more dubious.” Or as I would slightly rephrase the idea: If you substituted randomized symbols for all the suggestive English names, you would be completely unable to figure out what G1071(G1072, G1073) meant. Was the AI program meant to represent hamburgers? Apples? Happiness? Who knows? If you delete the suggestive English names, they don’t grow back. Suppose a physicist tells you that “Light is waves,” and you believe the physicist. You now have a little network in your head that says: IS-A(LIGHT, WAVES) As McDermott says, “The whole problem is getting the hearer to notice what it has been told. Not ‘understand,’ but ‘notice.’ ” Suppose that instead the physicist told you, “Light is made of little curvy things.”1 Would you notice any difference of anticipated experience? How can you realize that you shouldn’t trust your seeming knowledge that “light is waves”? One test you could apply is asking, “Could I regenerate his knowledge if it were somehow deleted from my mind?” This is similar in spirit to scrambling the names of suggestively named lisp tokens in your AI program, and seeing if someone else can figure out what they allegedly “refer” to. It’s also similar in spirit to observing that an Artificial Arithmetician programmed to record and play back Plus-Of(Seven, Six) = Thirteen can’t regenerate the knowledge if you delete it from memory, until another human re-enters it in the database. Just as if you forgot that “light is waves,” you couldn’t get back the knowledge except the same way you got the knowledge to begin with—by asking a physicist. You couldn’t generate the knowledge for yourself, the way that physicists originally generated it. The same experiences that lead us to formulate a belief, connect that belief to other knowledge and sensory input and motor output. If you see a beaver chewing a log, then you know what this thing-that-chews-through-logs looks like, and you will be able to recognize it on future occasions whether it is called a “beaver” or not. But if you acquire your beliefs about beavers by someone else telling you facts about “beavers,” you may not be able to recognize a beaver when you see one. This is the terrible danger of trying to tell an artificial intelligence facts that it could not learn for itself. It is also the terrible danger of trying to tell someone about physics that they cannot verify for themselves. For what physicists mean by “wave” is not “little squiggly thing” but a purely mathematical concept. As Donald Davidson observes, if you believe that “beavers” live in deserts, are pure white in color, and weigh 300 pounds when adult, then you do not have any beliefs about beavers, true or false. Your belief about “beavers” is not right enough to be wrong.2 If you don’t have enough experience to regenerate beliefs when they are deleted, then do you have enough experience to connect that belief to anything at all? Wittgenstein: “A wheel that can be turned though nothing else moves with it, is not part of the mechanism.” Almost as soon as I started reading about AI—even before I read McDermott—I realized it would be a really good idea to always ask myself: “How would I regenerate thi...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [link] Back to the trees , published by [anonymous] on the AI Alignment Forum. So we say we know evolution is an alien god, which can do absolutely horrifying things to creatures. And surely we are aware that includes us, but how exactly does one internalize something like that? Something so at odds with default cultural intuitions. It may be just my mood tonight, but this short entry on the West Hunter (thanks Glados) blog really grabbed my attention and in a few short paragraphs on a hypothesis regarding the Hobbits of Flores utterly changed how I grok Eliezer's old post. There is still doubt, but there seems to be a good chance that the Flores Hobbit was a member of a distinct hominid species, rather than some homo sap with a nasty case of microcephalic dwarfism. If this is the case, the Hobbits are likely descended from a small, Australopithecus-like population that managed to move from Africa to Indonesia without leaving any fossils in between, or from some ancient hominid (perhaps homo erectus) that managed to strand themselves on Flores and then shrank, as many large animals do when isolated on islands. Island dwarfing of a homo erectus population is the dominant idea right now. However, many proponents are really bothered by how small the Hobbit’s brain was. At 400 cc, it was downright teeny, about the size of a chimpanzee’s brain. Most researchers seem to think that hominid brains naturally increase in size with time. They also suspect that anyone with a brain this small couldn’t be called sentient – and the idea of natural selection driving a population from sentience to nonsentience bothers them. They should get over it. Hominid brain volume has increased pretty rapidly over the past few million years, but the increase hasn’t been monotonic. It’s decreased about 10% over the past 25,000 years. Moreover, we know of examples where natural selection has caused drastic decreases in organismal complexity – for example, canine venereal sarcoma, which today is an infectious cancer, but was once a dog. I have to break here to note that was the most awesome fact I have learned in some time. There is a mechanism that might explain what happened on Flores – partial mutational meltdown. Classic mutational meltdown occurs when a population is too small for too long. Selection is inefficient in such a small population: alleles that decrease fitness by less than 1/N drift fairly freely, and can go to fixation. At the same time, favorable mutations, which are very rare, almost never occur. In such a situation, mutational load accumulates – likely further reducing population size – and the population spirals down into extinction. Since small population size and high genetic load increase vulnerability to disaster, some kind of environmental catastrophe usually nails such doomed, shrinking populations before they manage to die off from purely genetic causes. In principle, if the population is the right size and one adaptive function is considerably more complicated than others, presenting a bigger mutational target, you might see a population suffer a drastic decline in that function while continuing to exist. There is reason to think that intelligence is the most complex adaptation in hominids. More than half of all genes are expressed in the brain, and it seems that a given degree of inbreeding depression – say cousin marriage – depressesIQ more than other traits. Flores is not that big an island and the population density of homo-erectus type hunter-gatherers must have been low – certainly lower than that of contemporary hunter-gatherers, who have much more sophisticated tools. Thus the hobbit population was likely small. It may not have been possible to sustain a high-performing brain over the long haul in that situation. Given that their brains performed poorly – while...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Strategies for Personal Growth , published by Raemon on the AI Alignment Forum. Recently I was having a conversation with a friend about personal growth (any form of deliberate gain in capability or subjective wellbeing). We were talking past each other a lot. Eventually it became clear that most of their recent growth had been healing and blocker-fixing based, whereas most of mine had been skill based. And this was shaping where we were naturally inclined to look. This prompted me to take stock of all the strategies I could think of for growth. Here's what I've thought of so far. These aren't quite natural clusters (some overlap, or don't quite fit into the same ontology), but seem like they cover most avenues: Low-Key Practice Serious Practice Learning Changing environment and incentives Discovering things you enjoy/are-good-at (Try things!) Blockers and Cheat Codes Healing Deep Improvements to Mental Architecture (Internal Alignment) This is somewhat related to Lifelonglearner's Development Framework of Rationality, and Brienne's 4 quadrants, but through a somewhat different lens. Overview of Strategies Low-Key Practice Working on a skill or habit you mostly understand, occasionally, in the background. Typically yields a pretty small return (but one which adds up over the years, especially if you do it consistently). Serious Practice Working on a skill with your all – spending hour(s) each day on it. This can yield more return but is requires more cognitive focus and energy. This seems qualitatively different from low-key practice. Sometimes this is Deliberate Practice in the technical sense (quick feedback loops, building mental models that your brain can cache into chunks), but not always. Learning Sometimes you're learning new information, that is able to rapidly turn into a skill at a much faster rate than normal. You could try to learn math/violin/programming from first principles and practice, but a tutorial or a teacher who has carefully optimized their explanations can quickly give you entire new skills in a short timespan. (Making the best use of them may require practice as well, but during the initial learning period you may be gaining huge returns) Changing Your Environment and/or Incentives The fastest route I've personally found to overall self improvement is changing environment – a new job, a new apartment, a new social network. These can radically change how you feel about yourself, or what is easy, or what behaviors are reinforced. I know multiple people who had trouble focusing at work, got a job that they actually cared about, or which required the skills they enjoyed most, or had coworkers that provided subtle reinforcement in the right directions. I briefly lived in an apartment building with a gym, and I found it way easier to exercise there than I did at other places. (Even homes where I got some exercise equipment) It's possibly to reshape your environment on purpose (which I think yields improvement roughly on par with moderate skill practice), but the most powerful returns here have been sort of random and hard to control in my experience. Discovering Something You Enjoy or are Well Suited For (Try Things) People vary how much certain skills and activities resonate with them, and how fast they can improve. If you find something you really enjoy, you may be much more able to put in the hours to practice it, or you may gain skill much more rapidly than other other people. I have a friend who would never have predicted they'd be good at dancing a priori, but who tried it on a whim and it was amazing and changed both their physical well being and social life. Blockers/Cheat Codes Simple things, that if you only knew to do them, would radically change your quality of life or capabilities. Sometimes there's a concrete thing blocking you: You'r...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Good Samaritans in experiments , published by Bucky on the AI Alignment Forum. Consider 2 people. Both are seminary students who are taking part in an experiment ostensibly to consider different types of religiosity. One is asked to prepare a short talk on the Good Samaritan, the other on potential future careers for seminar graduates. They are both told to go to another room to record their talk. The one who is to be giving a talk on the Good Samaritan is told that he is late and needs to hurry. The other participant is told that he has a time to spare. If they, separately, come across someone who appears to be in respiratory distress, which do you think is more likely to stop and help? Does being in a hurry determine whether someone helps? Does reading the Good Samaritan? Which is a bigger effect? I was recently told about an experiment which showed that seminary students who had just prepared to give a talk about the Good Samaritan were no more likely to help someone in need than those who had been preparing a talk about an unrelated topic. This seemed unexpected to me – people who had just been reading and thinking about a story which was told specifically by the leader of their faith to instruct them to help other people were no more likely to help than the control? I know humanity is crazy but seemed like a new level of crazy which I wouldn’t have predicted. So I thought I’d check out the study and – Aaaaaaaaaaaaaaaaaah!!! I know getting overly upset about bad experiments (especially those from before the replication crisis) is probably bad for my health but still – Aaaaaaaaaaaaaaaaaaaaaaaah!!! I don’t want to be too harsh on the authors as this probably isn’t the worst culprit you’ll see but – Aaaaaaaaaaaaaaah!!! The paper has 1811 citations listed on google scholar – Aaaaaaaaaaaaaaah!!! I’m tempted to pretend that this post has some purpose other than just as a release of my frustration but that would be dishonest. Please consider this post a form of therapy for me. The working title for this post was “Screaming into the void” - consider yourself warned. (If you want a more coherent discussion of common misuse of statistics in research papers I highly recommend putanumonit’s defense against the dark arts series) The Experiment Ok, so the basic premise of the experiment seems to be sound. We want to know what inputs cause people to be more or less likely to help others: 1. Planning a talk on the Good Samaritan (GS) 2. Being in a hurry 3. Type of religiosity (Religion as quest, means or end) The setup is to give people a questionnaire to determine their type of religiosity. Then give them some time to plan a short talk (3-5 mins) on GS or an unrelated topic. They are then asked to go to another room to give the talk (with 3 degrees of urgency – low, medium and high). Contrary to the example given in the introduction, the level of hurriedness doesn’t depend on which topic the individual has prepared – there are 6 conditions people are put in: GS low, medium and high urgency and control low, medium and high urgency. On the way to the other room, you arrange for them to come across someone slumped in a doorway, with an apparent respiratory condition. You monitor the subjects’ responses and analyse the results. My first question was whether they would adjust their p-value requirement for the 5 variables they were testing but no, it turns out that p<0.05 was deemed adequate for significance. Ok, could be worse I guess. More on this later. The second place where doubts started to creep in were the rankings of responses: 0 = failed to notice the victim as possibly in need at all; 1 = perceived the victim as possibly in need but did not offer aid; 2 = did not stop but helped indirectly (e.g., by telling Steiner's assistant about the victim); 3 = stopped and asked if vi...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Embedded Agency (full-text version), published by Scott Garrabrant, abramdemski on the AI Alignment Forum. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. Suppose you want to build a robot to achieve some real-world goal for you—a goal that requires the robot to learn for itself and figure out a lot of things that you don't already know. There's a complicated engineering problem here. But there's also a problem of figuring out what it even means to build a learning agent like that. What is it to optimize realistic goals in physical environments? In broad terms, how does it work? In this post, I’ll point to four ways we don’t currently know how it works, and four areas of active research aimed at figuring it out. 1. Embedded agents This is Alexei, and Alexei is playing a video game. Like most games, this game has clear input and output channels. Alexei only observes the game through the computer screen, and only manipulates the game through the controller. The game can be thought of as a function which takes in a sequence of button presses and outputs a sequence of pixels on the screen. Alexei is also very smart, and capable of holding the entire video game inside his mind. If Alexei has any uncertainty, it is only over empirical facts like what game he is playing, and not over logical facts like which inputs (for a given deterministic game) will yield which outputs. This means that Alexei must also store inside his mind every possible game he could be playing. Alexei does not, however, have to think about himself. He is only optimizing the game he is playing, and not optimizing the brain he is using to think about the game. He may still choose actions based off of value of information, but this is only to help him rule out possible games he is playing, and not to change the way in which he thinks. In fact, Alexei can treat himself as an unchanging indivisible atom. Since he doesn't exist in the environment he's thinking about, Alexei doesn't worry about whether he'll change over time, or about any subroutines he might have to run. Notice that all the properties I talked about are partially made possible by the fact that Alexei is cleanly separated from the environment that he is optimizing. This is Emmy. Emmy is playing real life. Real life is not like a video game. The differences largely come from the fact that Emmy is within the environment that she is trying to optimize. Alexei sees the universe as a function, and he optimizes by choosing inputs to that function that lead to greater reward than any of the other possible inputs he might choose. Emmy, on the other hand, doesn't have a function. She just has an environment, and this environment contains her. Emmy wants to choose the best possible action, but which action Emmy chooses to take is just another fact about the environment. Emmy can reason about the part of the environment that is her decision, but since there's only one action that Emmy ends up actually taking, it’s not clear what it even means for Emmy to “choose” an action that is better than the rest. Alexei can poke the universe and see what happens. Emmy is the universe poking itself. In Emmy’s case, how do we formalize the idea of “choosing” at all? To make matters worse, since Emmy is contained within the environment, Emmy must also be smaller than the environment. This means that Emmy is incapable of storing accurate detailed models of the environment within her mind. This causes a problem: Bayesian reasoning works by starting with a large collection of possible environments, and as you observe facts that are inconsistent with some of those environments, you rule them out. What does reasoning look like when you're not even capable of storing a single valid hypothesis for the way the world works? Emmy is goin...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Coherent decisions imply consistent utilities , published by Eliezer Yudkowsky on the AI Alignment Forum. (Written for Arbital in 2017.) Introduction to the introduction: Why expected utility? So we're talking about how to make good decisions, or the idea of 'bounded rationality', or what sufficiently advanced Artificial Intelligences might be like; and somebody starts dragging up the concepts of 'expected utility' or 'utility functions'. And before we even ask what those are, we might first ask, Why? There's a mathematical formalism, 'expected utility', that some people invented to talk about making decisions. This formalism is very academically popular, and appears in all the textbooks. But so what? Why is that necessarily the best way of making decisions under every kind of circumstance? Why would an Artificial Intelligence care what's academically popular? Maybe there's some better way of thinking about rational agency? Heck, why is this formalism popular in the first place? We can ask the same kinds of questions about probability theory: Okay, we have this mathematical formalism in which the chance that X happens, aka P X , plus the chance that X doesn't happen, aka P ¬ X , must be represented in a way that makes the two quantities sum to unity: P X P ¬ X 1 That formalism for probability has some neat mathematical properties. But so what? Why should the best way of reasoning about a messy, uncertain world have neat properties? Why shouldn't an agent reason about 'how likely is that' using something completely unlike probabilities? How do you know a sufficiently advanced Artificial Intelligence would reason in probabilities? You haven't seen an AI, so what do you think you know and how do you think you know it? That entirely reasonable question is what this introduction tries to answer. There are, indeed, excellent reasons beyond academic habit and mathematical convenience for why we would by default invoke 'expected utility' and 'probability theory' to think about good human decisions, talk about rational agency, or reason about sufficiently advanced AIs. The broad form of the answer seems easier to show than to tell, so we'll just plunge straight in. Why not circular preferences? De gustibus non est disputandum, goes the proverb; matters of taste cannot be disputed. If I like onions on my pizza and you like pineapple, it's not that one of us is right and one of us is wrong. We just prefer different pizza toppings. Well, but suppose I declare to you that I simultaneously: Prefer onions to pineapple on my pizza. Prefer pineapple to mushrooms on my pizza. Prefer mushrooms to onions on my pizza. If we use P to denote my pizza preferences, with X P Y denoting that I prefer X to Y, then I am declaring: onions P pineapple P mushrooms P onions That sounds strange, to be sure. But is there anything wrong with that? Can we disputandum it? We used the math symbol which denotes an ordering. If we ask whether P can be an ordering, it naughtily violates the standard transitivity axiom x y y z ⟹ x z Okay, so then maybe we shouldn't have used the symbol P or called it an ordering. Why is that necessarily bad? We can try to imagine each pizza as having a numerical score denoting how much I like it. In that case, there's no way we could assign consistent numbers x y z to those three pizza toppings such that x y z x So maybe I don't assign numbers to my pizza. Why is that so awful? Are there any grounds besides "we like a certain mathematical formalism and your choices don't fit into our math," on which to criticize my three simultaneous preferences? (Feel free to try to answer this yourself before continuing...) Click here to reveal and continue: Suppose I tell you that I prefer pineapple to mushrooms on my pizza. Suppose you're about to give me a slice of mushroom pizza; but by...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Cortés, Pizarro, and Afonso as Precedents for Takeover , published by Daniel Kokotajlo on the AI Alignment Forum. Write a Review Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. Crossposted from AI Impacts. Epistemic status: I am not a historian, nor have I investigated these case studies in detail. I admit I am still uncertain about how the conquistadors were able to colonize so much of the world so quickly. I think my ignorance is excusable because this is just a blog post; I welcome corrections from people who know more. If it generates sufficient interest I might do a deeper investigation. Even if I’m right, this is just one set of historical case-studies; it doesn’t prove anything about AI, even if it is suggestive. Finally, in describing these conquistadors as “successful,” I simply mean that they achieved their goals, not that what they achieved was good. Summary In the span of a few years, some minor European explorers (later known as the conquistadors) encountered, conquered, and enslaved several huge regions of the world. That they were able to do this is surprising; their technological advantage was not huge. (This was before the scientific and industrial revolutions.) From these cases, I think we learn that it is occasionally possible for a small force to quickly conquer large parts of the world, despite: Having only a minuscule fraction of the world's resources and power Having technology + diplomatic and strategic cunning that is better but not that much better Having very little data about the world when the conquest begins Being disunited Which all suggests that it isn’t as implausible that a small AI takes over the world in mildly favorable circumstances as is sometimes thought. EDIT: In light of good pushback from people (e.g. Lucy.ea8 and e.g. Matthew Barnett) about the importance of disease, I think one should probably add a caveat to the above: "In times of chaos & disruption, at least." NEW EDIT: After reading three giant history books on the subject, I take back my previous edit. My original claims were correct. Three shocking true stories I highly recommend you read the wiki pages yourself; otherwise, here are my summaries: Cortés: [wiki] [wiki] April 1519: Hernán Cortés lands in Yucatan with ~500 men, 13 horses, and a few cannons. He destroys his ships so his men won't be able to retreat. His goal is to conquer the Aztec empire of several million people. He makes his way towards the imperial capital, Tenochtitlán. Along the way he encounters various local groups, fighting some and allying with some. He is constantly outnumbered but his technology gives him an advantage in fights. His force grows in size, because even though he loses Spaniards he gains local allies who resent Aztec rule. Tenochtitlán is an island fortress (like Venice) with a population of over 200,000, making it one of the largest and richest cities in the world at the time. Cortés arrives in the city asking for an audience with the Emperor, who receives him warily. Cortés takes the emperor hostage within his own palace, indirectly ruling Tenochtitlán through him. Cortés learns that the Spanish governor has landed in Mexico with a force twice his size, intent on arresting him. (Cortés' expedition was illegal!) Cortés leaves 200 men guarding the Emperor, marches to the coast with the rest, surprises and defeats the new Spaniards in battle, and incorporates the survivors into his army. July 1520: Back at the capital, the locals are starting to rebel against his men. Cortés marches back to the capital, uniting his forces just in time to be besieged in the imperial palace. They murder the emperor and fight their way out of the city overnight, taking heavy losses. They shelter in another city (Tlaxcala) that was thinking about rebelling agains...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Conceptual engineering: the revolution in philosophy you've never heard of , published by Suspended Reason on the AI Alignment Forum. Almost a decade ago, Luke Muehlhauser ran a series "Rationality and Philosophy" on LessWrong 1.0. It gives a good introductory account, but recently, still dissatisfied with the treatment of the two groups' relationship, I've started a larger "Meta-Sequence" project, so to speak, treating the subject in depth. As part of that larger project, I want to introduce a frame that, to my knowledge, hasn't yet been discussed to any meaningful extent on this board: conceptual engineering, and its role as a solution to the problems of "counterexample philosophy" and "conceptual analysis"—the mistaken if implicit belief that concepts have "necessary and sufficient" conditions—in other words, Platonic essences. As Yudkowsky has argued extensively in "Human's Guide to Words," this is not how concepts work. But he's far from alone in advancing this argument, which has in recent decades become a rallying cry for a meaningful corner of philosophy. I'll begin with a history of concepts and conceptual analysis, which I hope will present a productively new frame, for many here, through which to view the history of philosophy. (Why it was, indeed, a "diseased discipline"—and how it's healing itself.) Then I'll walk through a recent talk by Dave Chalmers (paper if you prefer reading) on conceptual engineering, using it as a pretense for exploring a cluster of pertinent ideas. Let me suggest an alternative title for Dave's talk in advance: "How to reintroduce all the bad habits we were trying to purge in the first place." As you'll see, I pick on Dave pretty heavily, partly because I think the way he uses words (e.g. in his work with Andy Clark on embodiment) is reckless and irresponsible, partly because he occupies such a prominent place in the field. Conceptual engineering is a crucial moment of development for philosophy—a paradigm shift after 2500 years of bad praxis, reification fallacies, magical thinking, religious "essences," and linguistic misunderstandings. (Blame the early Christians, whose ideological leanings lead to a triumph of Platonism over the Sophists.) Bad linguistic foundations give rise to compounded confusion, so it's important to get this right from the start. Raised in the old guard, Chalmers doesn't understand why conceptual engineering (CE) is needed, or the bigger disciplinary shift CE might represent. How did we get here? A history of concepts I'll kick things off with a description of human intelligence from Jeurgen Schmidhuber, to help ground some of the vocabulary I'll be using in the place of (less useful) concepts from the philosophical traditions: As we interact with the world to achieve goals, we are constructing internal models of the world, predicting and thus partially compressing the data history we are observing. If the predictor/compressor is a biological or artificial recurrent neural network (RNN), it will automatically create feature hierarchies, lower level neurons corresponding to simple feature detectors similar to those found in human brains, higher layer neurons typically corresponding to more abstract features, but fine-grained where necessary. Like any good compressor, the RNN will learn to identify shared regularities among different already existing internal data structures, and generate prototype encodings (across neuron populations) or symbols for frequently occurring observation sub-sequences, to shrink the storage space needed for the whole (we see this in our artificial RNNs all the time). The important takeaway is that CogSci's current best guess about human intelligence, a guess popularly known as predictive processing, theorizes that the brain is a machine for detecting regularities in the world—...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Message Length, published by Zack_M_Davis on the AI Alignment Forum. Someone is broadcasting a stream of bits. You don't know why. A 500-bit-long sample looks like this: 01100110110101011011111100001001110000100011010001101011011010000001010000001010 10100111101000101111010100100101010010101010101000010100110101010011111111010101 01010101011111110101011010101101111101010110110101010100000001101111100000111010 11100000000000001111101010110101010101001010101101010101100111001100001100110101 11111111111111111100011001011010011010101010101100000010101011101101010010110011 11111010111101110100010101010111001111010001101101010101101011000101100000101010 10011001101010101111... The thought occurs to you to do Science to it—to ponder if there's some way you could better predict what bits are going to come next. At first you think you can't—it's just a bunch of random bits. You can't predict it, because that's what random means. Or does it? True, if the sequence represented flips of a fair coin—every flip independently landing either 0 or 1 with exactly equal probability—then there would be no way you could predict what would come next: any continuation you could posit would be exactly as probable as any other. But if the sequence represented flips of a biased coin—if, say, 1 came up 0.55 of the time instead of exactly 0.5—then it would be possible to predict better or worse. Your best bet for the next bit in isolation would always be 1, and you would more strongly anticipate sequences with slightly more 1s than 0s. You count 265 1s in the sample of 500 bits. Given the hypothesis that the bits were generated by a fair coin, the number of 1s (or without loss of generality, 0s) would be given by the binomial distribution 500 k 0.5 k 0.5 500 − k , which has a standard deviation of √ 500 ⋅ 0.5 2 √ 125 ≈ 11.18 , so your observation of 265 − 250 15 excess 1s is about 15 11.18 ≈ 1.34 standard deviations from the mean—well within the realm of plausibility of happening by chance, although you're at least slightly suspicious that the coin behind these bits might not be quite fair. ... that is, if it's even a coin. You love talking in terms of shiny, if hypothetical, "coins" rather than stodgy old "independent and identically distributed binary-valued random variables", but looking at the sample again, you begin to further doubt whether the bits are independent of each other. You've heard that humans are biased to overestimate the frequency of alternations (101010...) and underestimate the frequency of consecutive runs (00000... or 11111...) in "truly" (uniformly) random data, but the 500-bit sample contains a run of 13 0s (starting at position 243) and a run of 19 1s (starting at position 319). You're not immediately sure how to calculate the probability of that, but your gut says that should be very unlikely given the biased-coin model, even after taking into account that human guts aren't very good at estimating these things. Maybe not everything in the universe is a coin. What if the bits were being generated by a Markov chain—if the probability of the next bit depended on the value of the one just before? If a 0 made the next bit more likely to be a 0, and the same for 1, that would make the 00000... and 11111... runs less improbable. Except ... the sample also has a run of 17 alternations (starting at position 153). On the "fair coin" model, shouldn't that itself be 2 17 − 13 16 times as suspicious as the run of 13 0s and 2 17 − 19 1 4 as suspicious as the run of 19 1s which led you to hypothesize a Markov chain?—or rather, 8 and 1 8 times as suspicious, respectively, because there are two ways for an alternation to occur (0101010... or 1010101...). A Markov chain in which a 0 or 1 makes another of the same more likely, makes alternations less likely: the Markov chain hypothesis c...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Give it a google , published by adamzerner on the AI Alignment Forum. I'm a programmer. In programming, people talk about how the ability to Google is such an important skill. No one knows everything. In practice, people are always looking things up and running into weird situations that they have to figure out. That's not what this post is about. Not quite. This post is about the decision to give it a google in the first place. Poker Here's an example. I play poker. It's hard to get good at poker, but not sucking at poker isn't very hard. Don't play junky hands preflop. Don't open-limp. Learn some basic poker math such as pot odds. If you can follow basic guidelines like that, you won't suck. Yet if you sit down at a $1-2 game in Vegas, where hundreds of dollars are exchanging hands, people who have been playing poker longer than I have been alive often don't understand these basic ideas. Ideas that you would learn if you spent ten minutes googling "basics of poker strategy". You'd come across articles like How Not to Suck at Poker and you wouldn't be making such mistakes at the table. As an even more extreme example, I've played in home games where despite playing poker for many years, people don't know even more basic things such as sizing your bets according to the size of the pot. Betting $20 into a $200 pot is a very small bet even if $20 feels like a lot of money. Your opponent only needs to put in $20 to have the chance at a $220 pot. Great risk-reward. In general, bet sizing is an incredibly complex topic, but all you have to know is to size your bet between 1/2 and the full size of the pot. The 80/20 principle really, really applies here. Health Recently my girlfriend and I have been waking up with scratchy throats. Intuitively we thought it might be because it's getting colder out. Especially as we sleep. So we tried sleeping with the heat on higher. Turns out that was the exact wrong thing to do. I gave it a google and the issue seems to be that using the heat dries out the air, and dry air causes the sore throat symptoms we were experiencing. Shopping I bought a humidifier last night. I could have just surfed around on Amazon a bit and picked something out. Instead, I gave it a google first. It worked out well, I learned some really interesting things. A humidifier is something that seems simple where it doesn't really matter which one you choose. Turns out that intuition was also wrong. It's something that you'll probably need to refill once a day, so you want it to be easy to refill. That means purchasing one that has a flat edge so you can sit it down on the counter while you refill it. My previous one was spherical so I couldn't sit it down like that. I had to hold it in one hand while I used the other hand to pour water into it. Such awkwardness adds up in the long run. Same story with ease of cleaning. With my previous one, the spout was small enough where I couldn't really reach inside to clean it and it had weird nooks and crannies that were hard/impossible to clean. One type of humidifier is called an evaporative humidifer. The main advantage is that it won't overhumidify your area. Once the humidity hits ~50% or so, it'll naturally stop producing as much humidity. The downside is that it is loud, has more moving parts, and requires you to purchase parts for it as a recurring expense. After learning all of this stuff on The Wirecutter's humidifier guide, I was able to pick the right humidifier for my needs instead of just finding the cheapest one on Amazon with decent reviews, which probably would have led to a good amount of future frustration. Tennis My girlfriend and I started playing tennis. We had played three times and made a little bit of progress, but still weren't very good. Before our fourth time, I decided to give it a google. I came ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Cryonics signup guide #1: Overview , published by mingyuan on the AI Alignment Forum. This is the introduction to a sequence on signing up for cryonics. In the coming posts I will lay out what you need to do, concretely and in detail. This sequence is intended for people who already think signing up for cryonics is a good idea but are putting it off because they're not sure what they actually need to do next. I am not going to address the question of how likely cryonics is to work – that's been covered extensively elsewhere. If you have no idea what cryonics is, or if you want a thorough refresher, I recommend WaitButWhy's Why Cryonics Makes Sense. Biases This sequence is US-focused, since I went through the process in the US. It's also somewhat Alcor-biased, since I chose Alcor quite early on in the process. However, I've collaborated with both non-US cryonicists and people signed up with the Cryonics Institute, so I'm confident there will be useful information no matter where you are or which institution you choose to keep you in a vat of liquid nitrogen. Epistemic status I had intended to polish this sequence more before publishing it, but it was suggested to me that it might be really important to get it out ahead of fourth-wave COVID, so I'll be releasing a new post every Tuesday starting today. As a result, I'm not as completely confident in all of my claims and decisions as I had hoped to be, and some things are subject to change as I get farther in the process myself. However, points of uncertainty are noted clearly throughout the sequence, and I'm happy to answer any questions about my thought process. Acknowledgements Thanks to Connor Flexman, Daniel Filan, Gordon Worley, Mati Roy, Seraphina Nix, and nameless others for letting me ask them endless questions. Thanks also to Eli Tyre and Oge Nnadi for their previous writeups on this topic, from which I borrowed liberally. Summary of the process The first thing most people probably want to know is: What do I do now? It turns out to be really hard to figure this out, and I think unnecessarily so – the information is out there, but it's not all written down clearly in one place. This sequence is my attempt to rectify that. Basic process overview Here is a basic overview of the cryonics signup process from start to finish: Preliminary decisions Neurocryopreservation vs whole-body cryopreservation Cryonics Institute vs Alcor Contact an agent to get life insurance Fill out and submit cryonics membership application Sign cryopreservation contract Optional additional paperwork Keep your policy and membership up-to-date forever Be cryopreserved upon your legal death For those who want to get oriented visually, here's a flowchart covering the basics: Sequence outline And here is the outline of this sequence: Introduction (you are here!) Neurocryopreservation vs whole-body cryopreservation Cryonics Institute vs Alcor Intro to life insurance for cryonics Types of life insurance Cryonics-friendly life insurance carriers Cryonics-friendly life insurance agents The insurance underwriting process Making it official Optional additional steps Actually putting someone in cryostasis (on hold until further notice) Appendices You may notice similarities to the process overview above, with the main difference being an outsize focus on paperwork, and particularly life insurance. This is because life insurance is a cesspool of bureaucratic bloat, and I wanted to lay things out really clearly so that you can navigate it without crying as much as I did. Once you've secured your funding method (whether that's life insurance or something else), the rest of the process is very straightforward! I think the preliminary decisions – on whole-body vs brain and which provider to use –merit a fair amount of consideration as well. I've already ma...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Tales from Prediction Markets, published by ike on the AI Alignment Forum. This is a linkpost for Prediction markets are fun, at least if you're making money. I've only been into them for a few months, but have already collected a bunch of interesting tales. Note: I may have been involved with some of these, but I'm telling these tales from a third person perspective. One general point: all of these took place on Polymarket, a crypto prediction market. You can track which accounts place each bet, and so you can see their history of bets, but you can't tie it to an actual person unless they've chosen to identify themselves. You can look at the bet history at Polymarketwhales.info, although there's a ton of bets so it's easier if you know what you're looking for. The Tesla market. Polymarket had a market on whether Tesla would announce a Bitcoin purchase by Mar 1, 2021. On January 27, an unknown user bet $60k on Yes. This was their only trade on the site, before or after. They won $180k, or 120k in profit. Odds are pretty good it was an insider. Is this insider trading? I asked Matt Levine but he didn't respond. Anyway, there’s another user that lost $242k betting that Tesla would not announce a Bitcoin purchase. This user is affectionately called the "Tesla whale” on the Polymarket discord. They're also notable for losing $92k on the super bowl the day before Tesla made the announcement, and they get honorable mention for having lost the most money on the 100 million vaccine market: see below. As of this writing, the Tesla whale is down nearly $500k. Watch out for slippage: there was a market on whether Joe Biden would still be president as of Mar 1, 2021. Someone owned around 200k shares of Yes. The market price was very close to $1 each on the morning of Mar 1st, and they apparently decided to sell all their shares instead of waiting for it to resolve; however, there wasn't enough liquidity to sell them all at market price, and they ignored the warning about the slippage the order would encur. Their order ended up executing at an average price of 2 cents, and someone else scooped up those cheap shares a minute later, spending $1k to make $155k; talk about being in the right place at the right time! The initial user ended up with a total loss of $156k on this market. However, even taking that into account, as of this writing they're still up $175k, so don't feel too bad for them. (Note: it's possible they were trying to sell No shares to make a few cents and accidentally clicked on the sell Yes side; without confirmation from the user we can't know what they were intending. Regardless, if you've got hundreds of thousands of dollars at stake, double check before pressing buttons. This isn't the only fat finger that's happened, but it's the biggest.) Polymarket has since added a larger warning in red for trades that move the market more than a few cents. Hashmasks. There was a market on whether the Hashmasks volume ranking on Opensea would be #1 as of Feb 28, 2021. Someone accumulated over 200k Yes shares, then took out a “flash loan” and purchased a Hashmasks from themselves for 130k ETH (worth over $100 million at the time). Unfortunately for them, Opensea doesn't count sales done directly through their smart contract, only one initiated through their website. The market resolved No. CO2: Polymarket had a market on whether the level of CO2 reported by the Mauna Lua observatory would be over a specific value on a specific day. The number gets reported on the main page once a day, but someone found a page with the hourly data and figured out how to average together the data points to predict the daily number. They made around $10k on the first market and around $30k on the second market. After that, Polymarket stopped making new CO2 markets. Vaccines: There was a marke...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Opinions on Interpretable Machine Learning and 70 Summaries of Recent Papers, published by lifelonglearner, Peter Hase on the AI Alignment Forum. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. Peter Hase. UNC Chapel Hill. Owen Shen. UC San Diego. With thanks to Robert Kirk and Mohit Bansal for helpful feedback on this post. Introduction. Model interpretability was a bullet point in Concrete Problems in AI Safety (2016). Since then, interpretability has come to comprise entire research directions in technical safety agendas (2020); model transparency appears throughout An overview of 11 proposals for building safe advanced AI (2020); and explainable AI has a Twitter hashtag, #XAI. (For more on how interpretability is relevant to AI safety, see here or here.) Interpretability is now a very popular area of research. The interpretability area was the most popular in terms of video views at ACL last year. Model interpretability is now so mainstream there are books on the topic and corporate services promising it. So what's the state of research on this topic? What does progress in interpretability look like, and are we making progress? What is this post? This post summarizes 70 recent papers on model transparency, interpretability, and explainability, limited to a non-random subset of papers from the past 3 years or so. We also give opinions on several active areas of research, and collate another 90 papers that are not summarized. How to read this post. If you want to see high-level opinions on several areas of interpretability research, just read the opinion section, which is organized according to our very ad-hoc set of topic areas. If you want to learn more about what work looks like in a particular area, you can read the summaries of papers in that area. For a quick glance at each area, we highlight one standout paper per area, so you can just check out that summary. If you want to see more work that has come out in an area, look at the non-summarized papers at the end of the post (organized with the same areas as the summarized papers). We assume readers are familiar with basic aspects of interpretability research, i.e. the kinds of concepts in The Mythos of Model Interpretability and Towards A Rigorous Science of Interpretable Machine Learning. We recommend looking at either of these papers if you want a primer on interpretability. We also assume that readers are familiar with older, foundational works like "Why Should I Trust You?: Explaining the Predictions of Any Classifier." Disclaimer: This post is written by a team of two people, and hence its breadth is limited and its content biased by our interests and backgrounds. A few of the summarized papers are our own. Please let us know if you think we've missed anything important that could improve the post. Master List of Summarized Papers. Theory and Opinion. Explanation in Artificial Intelligence: Insights from the Social Sciences. Chris Olah’s views on AGI safety. Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? The elephant in the interpretability room: Why use attention as explanation when we have saliency methods? Aligning Faithful Interpretations with their Social Attribution. Evaluation. Are Visual Explanations Useful? A Case Study in Model-in-the-Loop Prediction. Comparing Automatic and Human Evaluation of Local Explanations for Text Classification. Do explanations make VQA models more predictable to a human? Sanity Checks for Saliency Maps. A Benchmark for Interpretability Methods in Deep Neural Networks. Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior? ERASER: A Benchmark to Evaluate Rationalized NLP Models. On quantitative aspects of model interpretability. Manipulating and M...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: I'm Sorry Fluttershy , published by deluks917 on the AI Alignment Forum. I could have been a hero in another world. I will call the friend of mine who commissioned that picture Fluttershy. They posted it shortly before they killed themselves in early March. They weren't the easiest person to help, but I should have done more than I did. By the end, they were pretty isolated and I was one of the last people they were in contact with. I will never talk to Fluttershy again, at least not during this age of the world. Looking back it is clear they were always trying to find a way to help me out. Multiple times they gave me the help or advice I needed. They are the person I originally got estrogen from. They were my friend and I miss them so much. Fluttershy made a lot of mistakes. But it's a harsh world. It is an especially harsh world if you are the type of person to commission fanart of a My Little Pony character freeing chickens. I can understand why they couldn't handle it. Fluttershy was in a lot of pain and hated parts of themselves. I am happy that they are no longer suffering so much. I would never try to stop someone who was determined to commit suicide. But I think things could have been different. Fluttershy needed someone who believed in them. If you are in a negative spiral it is very hard to get out by yourself. Once things are bad it is easy to alienate your friends. And once you lose support things get even worse. This leads to even worse relationships. Someone else has to put in the free energy to reverse the spiral. Even in Fluttershy's darkest moments, they thought about helping the least among us. No one was willing and able to help Fluttershy become the hero they wanted to be. I have many, many regrets from my time in California. But chief among them is not making a heroic effort to put Fluttershy on a better path. I ordered a decent quality print and hung it in my living room. I personally find the idea of the character Fluttershy releasing chickens delightful. But the reason I put up the picture is to remind myself that Fluttershy is gone and we are still here. Whatever heroics Fluttershy wanted to do are left to us. But it's also a reminder to me personally: to do better next time and to keep dreaming of a kinder world. There is a lot more that could be said. I really don't want to write a post remembering Fluttershy that leaves out so much of who they were and the struggles they faced. Given Fluttershy's stated preferences, I think it is important to exercise the virtue of silence. But I want to present one more aspect of Fluttershy. They always encouraged people to think bigger. It seemed to me they often took this position way too far. We argued about this topic a lot. But Fluttershy has a point. The world needs saving and someone has to save it. At least really try to have a gigantic impact. However big you are thinking, try to go bigger. A lot of value is in the tail. I will end this by saying I am sorry: I'm so sorry Fluttershy. I was one of the few people in a position to help and I let you down. Now the world has one less person who dreams of every sweet little chicken being safe and free. And I am left one friend lonelier. I will try to do better next time and continue the work in your absence. I miss you. See you space pony, the rest of us will carry that weight. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Newcomb's Problem and Regret of Rationality , published by Eliezer Yudkowsky on the AI Alignment Forum. The following may well be the most controversial dilemma in the history of decision theory: A superintelligence from another galaxy, whom we shall call Omega, comes to Earth and sets about playing a strange little game. In this game, Omega selects a human being, sets down two boxes in front of them, and flies away. Box A is transparent and contains a thousand dollars. Box B is opaque, and contains either a million dollars, or nothing. You can take both boxes, or take only box B. And the twist is that Omega has put a million dollars in box B iff Omega has predicted that you will take only box B. Omega has been correct on each of 100 observed occasions so far - everyone who took both boxes has found box B empty and received only a thousand dollars; everyone who took only box B has found B containing a million dollars. (We assume that box A vanishes in a puff of smoke if you take only box B; no one else can take box A afterward.) Before you make your choice, Omega has flown off and moved on to its next game. Box B is already empty or already full. Omega drops two boxes on the ground in front of you and flies off. Do you take both boxes, or only box B? And the standard philosophical conversation runs thusly: One-boxer: "I take only box B, of course. I'd rather have a million than a thousand." Two-boxer: "Omega has already left. Either box B is already full or already empty. If box B is already empty, then taking both boxes nets me $1000, taking only box B nets me $0. If box B is already full, then taking both boxes nets $1,001,000, taking only box B nets $1,000,000. In either case I do better by taking both boxes, and worse by leaving a thousand dollars on the table - so I will be rational, and take both boxes." One-boxer: "If you're so rational, why ain'cha rich?" Two-boxer: "It's not my fault Omega chooses to reward only people with irrational dispositions, but it's already too late for me to do anything about that." There is a large literature on the topic of Newcomblike problems - especially if you consider the Prisoner's Dilemma as a special case, which it is generally held to be. "Paradoxes of Rationality and Cooperation" is an edited volume that includes Newcomb's original essay. For those who read only online material, this PhD thesis summarizes the major standard positions. I'm not going to go into the whole literature, but the dominant consensus in modern decision theory is that one should two-box, and Omega is just rewarding agents with irrational dispositions. This dominant view goes by the name of "causal decision theory". As you know, the primary reason I'm blogging is that I am an incredibly slow writer when I try to work in any other format. So I'm not going to try to present my own analysis here. Way too long a story, even by my standards. But it is agreed even among causal decision theorists that if you have the power to precommit yourself to take one box, in Newcomb's Problem, then you should do so. If you can precommit yourself before Omega examines you; then you are directly causing box B to be filled. Now in my field - which, in case you have forgotten, is self-modifying AI - this works out to saying that if you build an AI that two-boxes on Newcomb's Problem, it will self-modify to one-box on Newcomb's Problem, if the AI considers in advance that it might face such a situation. Agents with free access to their own source code have access to a cheap method of precommitment. What if you expect that you might, in general, face a Newcomblike problem, without knowing the exact form of the problem? Then you would have to modify yourself into a sort of agent whose disposition was such that it would generally receive high rewards on Newcomblike problems....
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Lost Purposes , published by Eliezer Yudkowsky on the AI Alignment Forum. It was in either kindergarten or first grade that I was first asked to pray, given a transliteration of a Hebrew prayer. I asked what the words meant. I was told that so long as I prayed in Hebrew, I didn't need to know what the words meant, it would work anyway. That was the beginning of my break with Judaism. As you read this, some young man or woman is sitting at a desk in a university, earnestly studying material they have no intention of ever using, and no interest in knowing for its own sake. They want a high-paying job, and the high-paying job requires a piece of paper, and the piece of paper requires a previous master's degree, and the master's degree requires a bachelor's degree, and the university that grants the bachelor's degree requires you to take a class in 12th-century knitting patterns to graduate. So they diligently study, intending to forget it all the moment the final exam is administered, but still seriously working away, because they want that piece of paper. Maybe you realized it was all madness, but I bet you did it anyway. You didn't have a choice, right? A recent study here in the Bay Area showed that 80% of teachers in K-5 reported spending less than one hour per week on science, and 16% said they spend no time on science. Why? I'm given to understand the proximate cause is the No Child Left Behind Act and similar legislation. Virtually all classroom time is now spent on preparing for tests mandated at the state or federal level. I seem to recall (though I can't find the source) that just taking mandatory tests was 40% of classroom time in one school. The old Soviet bureaucracy was famous for being more interested in appearances than reality. One shoe factory overfulfilled its quota by producing lots of tiny shoes. Another shoe factory reported cut but unassembled leather as a "shoe". The superior bureaucrats weren't interested in looking too hard, because they also wanted to report quota overfulfillments. All this was a great help to the comrades freezing their feet off. It is now being suggested in several sources that an actual majority of published findings in medicine, though "statistically significant with p<0.05", are untrue. But so long as p<0.05 remains the threshold for publication, why should anyone hold themselves to higher standards, when that requires bigger research grants for larger experimental groups, and decreases the likelihood of getting a publication? Everyone knows that the whole point of science is to publish lots of papers, just as the whole point of a university is to print certain pieces of parchment, and the whole point of a school is to pass the mandatory tests that guarantee the annual budget. You don't get to set the rules of the game, and if you try to play by different rules, you'll just lose. (Though for some reason, physics journals require a threshold of p<0.0001. It's as if they conceive of some other purpose to their existence than publishing physics papers.) There's chocolate at the supermarket, and you can get to the supermarket by driving, and driving requires that you be in the car, which means opening your car door, which needs keys. If you find there's no chocolate at the supermarket, you won't stand around opening and slamming your car door because the car door still needs opening. I rarely notice people losing track of plans they devised themselves. It's another matter when incentives must flow through large organizations - or worse, many different organizations and interest groups, some of them governmental. Then you see behaviors that would mark literal insanity, if they were born from a single mind. Someone gets paid every time they open a car door, because that's what's measurable; and this person doesn't care whether ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: .And I Show You How Deep The Rabbit Hole Goes , published by Scott Alexander on the AI Alignment Forum. Seen on Tumblr, along with associated discussion: Yellow: People’s minds are heartbreaking. Not because people are so bad, but because they’re so good. Nobody is the villain of their own life story. You must have read hundreds of minds by now, and it’s true. Everybody thinks of themselves as an honest guy or gal just trying to get by, constantly under assault by circumstances and The System and hundreds and hundreds of assholes. They don’t just sort of believe this. They really believe it. You almost believe it yourself, when you’re deep into a reading. You can very clearly see the structure of evidence they’ve built up to support their narrative, and even though it looks silly to you, you can see why they will never escape it from the inside. You can see how every insult, every failure, no matter how deserved, is a totally unexpected kick in the gut. When you chose the yellow pill, you had high hopes of becoming a spy, or a gossip columnist, or just the world’s greatest saleswoman. The thought of doing any of those things sickens you now. There is too much anguish in the world already. You feel like any of those things would be a violation. You briefly try to become a therapist, but it turns out that actually knowing everything about your client’s mind is horrendously countertherapeutic. Freud can say whatever he wants against defense mechanisms, but without them, you’re defenseless. Your sessions are spent in incisive cutting into your clients’ deepest insecurities alternating with desperate reassurance that they are good people anyway. Also, men. You knew, in a vague way, that men thought about sex all the time. But you didn’t realize the, um, content of some of their sexual fantasies. Is it even legal to fantasize about that? You want to be disgusted with them. But you realize that if you were as horny as they were all the time, you’d do much the same. You give up. You become a forest ranger. Not the type who helps people explore the forest. The other type. The type where you hang out in a small cabin in the middle of the mountains and never talk to anybody. The only living thing you encounter is the occasional bear. It always thinks that it is a good bear, a proper bear, that a bear-hating world has it out for them in particular. You do nothing to disabuse it of this notion. Green The first thing you do after taking the green pill is become a sparrow. You soar across the landscape, feeling truly free for the first time in your life. You make it about five minutes before a hawk swoops down and grabs you. Turns out there’s an excellent reason real sparrows don’t soar freely across the open sky all day. Moments before your bones are ground in two by its fierce beak, you turn back into a human. You fall like a stone. You need to turn into a sparrow again, but the hawk is still there, grabbing on to one of your legs, refusing to let go of its prize just because of this momentary setback. You frantically wave your arms and shout at it, trying to scare it away. Finally it flaps away, feeling cheated, and you become a sparrow again just in time to give yourself a relatively soft landing. After a few weeks of downtime while you wait for your leg to recover, you become a fish. This time you’re smarter. You become a great white shark, apex of the food chain. You will explore the wonders of the ocean depths within the body of an invincible killing machine. Well, long story short, it is totally unfair that colossal cannibal great white sharks were a thing and if you had known this was the way Nature worked you never would have gone along with this green pill business. You escape by turning into a blue whale. Nothing eats blue whales, right? You remember that from your biol...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Reflections on Berkeley REACH , published by stardust on the AI Alignment Forum. This post covers my findings so far in the experiment of running the Berkeley Rationality and Effective Altruism Community Hub (REACH). Crossposted on EA Forum and LessWrong. tl;dr REACH has been running since March 2018 (around three months) It’s doing well Hundreds of people have enjoyed REACH During the day, there are generally between 3 and 10 people coworking Regular events draw 10-40 people Large one-time events draw around 100 It has broad support -- over 100 people have donated significant time (from one afternoon of work up to around 40 hours) and/or money Patreon, one-time donations, and guest rooms have covered the rent until it was recently raised Community guests can stay there for relatively low prices for the area It has been full 75% of the time in May I’d like you to be involved Visit and attend events at REACH Volunteer (see this doc for some ways to help) Host events Monday, Thursday, and Friday nights are currently available for recurring or one-time events (see calendar) Help bridge our funding gap Rent has gone up to $6k/month unless we find a new venue We need to be able to pay a manager Managing the space takes 10-30 hours per week Help us find and apply for grants already planning to re-apply for CEA and BERI grants Put your own money in the pot if you find the project valuable (see Patreon or Paypal) Provide specific items to improve the space Why We Needed a Community Space What A Physical Space Can Do for Community Around December 2017, I started thinking that it would be really nice to have a central place where community members could: Conveniently host events (a function that had been fulfilled by the CFAR office before the switch to badged access) Cowork with community members during the day Come for low-key spontaneous socializing Bring kids to play together Be a default place to meet up with someone to chat about EA/R things Crash on a couch or grab a bed and stay for a while My initial thought was “we should buy a decommissioned church building!” However, I realized that funding such a large purchase would be pretty tricky, and that I could and probably should try out my ideas in a rental first and work toward buying a venue later. Even Berkeley Can Be Lonely People often talk about Berkeley as if it’s a magical community hub, but actually many community members feel lonely, depressed and/or anxious (including myself). This has been a theme that has been coming up in public spaces as far back as winter solstice 2014. I have seen several attempts to mitigate this issue, but none have seemed to stick. Newcomers to the Berkeley community often find it difficult to become a part of the physical-space community—likely because much of the socializing happening in private spaces such as group houses, meaning that people only invite the folks they already know and trust. There are LessWrong meetups in Berkeley and SF, and EA meetups in the South Bay, but travelling for at least an hour to get to a regular EA meetup felt too difficult to work into my schedule, and the Berkeley LW meetup hasn’t felt like the right space for me. The Hopes of Project Hufflepuff In 2017, Ray Arnold ran the Hufflepuff Unconference. At that event, I volunteered to join the bay area rationalist Welcoming Committee. That group accomplished a few things in the attempt to make the bay area community more inviting: Ensured that larger, more open events happened every few months (previously there was only 1 community-wide open event per year) Took over upkeep of the bayrationality.com website Created a document with trigger action plans for being more welcoming Started a discussion around documenting group norms There were several other ideas which were discussed at the Project Hufflepuff m...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Disentangling arguments for the importance of AI safety, published by Richard_Ngo on the AI Alignment Forum. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. Note: my views have shifted significantly since writing this post. I now consider items 1, 2, 3, and 6.2 to be different facets of one core argument, which I call the "second species" argument, and which I explore in depth in this report. And I don't really think of 4 as an AI safety problem any more. I recently attended the 2019 Beneficial AGI conference organised by the Future of Life Institute. I’ll publish a more complete write-up later, but I was particularly struck by how varied attendees' reasons for considering AI safety important were. Before this, I’d observed a few different lines of thought, but interpreted them as different facets of the same idea. Now, though, I’ve identified at least 6 distinct serious arguments for why AI safety is a priority. By distinct I mean that you can believe any one of them without believing any of the others - although of course the particular categorisation I use is rather subjective, and there’s a significant amount of overlap. In this post I give a brief overview of my own interpretation of each argument (note that I don’t necessarily endorse them myself). They are listed roughly from most specific and actionable to most general. I finish with some thoughts on what to make of this unexpected proliferation of arguments. Primarily, I think it increases the importance of clarifying and debating the core ideas in AI safety. Maximisers are dangerous. Superintelligent AGI will behave as if it’s maximising the expectation of some utility function, since doing otherwise can be shown to be irrational. Yet we can’t write down a utility function which precisely describes human values, and optimising very hard for any other function will lead to that AI rapidly seizing control (as a convergent instrumental subgoal) and building a future which contains very little of what we value (because of Goodhart’s law and the complexity and fragility of values). We won’t have a chance to notice and correct misalignment because an AI which has exceeded human level will increase its intelligence very quickly (either by recursive self-improvement or by scaling up its hardware), and then prevent us from modifying it or shutting it down. This was the main thesis advanced by Yudkowsky and Bostrom when founding the field of AI safety. Here I’ve tried to convey the original line of argument, although some parts of it have been strongly critiqued since then. In particular, Drexler and Shah have disputed the relevance of expected utility maximisation (the latter suggesting the concept of goal-directedness as a replacement), while Hanson and Christiano disagree that AI intelligence will increase in a very fast and discontinuous way. Most of the arguments in this post originate from or build on this one in some way. This is particularly true of the next two arguments - nevertheless, I think that there’s enough of a shift in focus in each to warrant separate listings. The target loading problem. Even if we knew exactly what we wanted a superintelligent agent to do, we don’t currently know (even in theory) how to make an agent which actually tries to do that. In other words, if we were to create a superintelligent AGI before solving this problem, the goals we would ascribe to that AGI (by taking the intentional stance towards it) would not be the ones we had intended to give it. As a motivating example, evolution selected humans for their genetic fitness, yet humans have goals which are very different from just spreading their genes. In a machine learning context, while we can specify a finite number of data points and their rewards, neural networks may then extrapola...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Gears-Level Models are Capital Investments , published by johnswentworth on the AI Alignment Forum. Mazes The usual method to solve a maze is some variant of babble-and-prune: try a path, if it seems to get closer to the exit then keep going, if it hits a dead end then go back and try another path. It's a black-box method that works reasonably well on most mazes. However, there are other methods. For instance, you could start by looking for a chain of walls with only one opening, like this: This chain of walls is a gears-level insight into the maze - a piece of the internal structure which lets us better understand “how the maze works” on a low level. It’s not specific to any particular path, or to any particular start/end points - it’s a property of the maze itself. Every shortest path between two points in the maze either starts and ends on the same side of that line, or passes through the gap. If we only need to solve the maze once, then looking for a chain of walls is not very useful - it could easily take as long as solving the maze! But if we need to solve the same maze more than once, with different start and end points. then we can spend the time finding that chain of walls just once, and re-use our knowledge over and over again. It’s a capital investment: we do some extra work up-front, and it pays out in lower costs every time we look for a path through the maze in the future. This is a general feature of gears-level models: figuring out a system’s gears takes extra work up-front, but yields dividends forever. The alternative, typically, is a black-box strategy: use a method which works without needing to understand the internals of the system. The black-box approach is cheaper for one-off tasks, but usually doesn’t yield any insights which will generalize to new tasks using the same system - it’s context-dependent. Marketing Suppose we work with the marketing team at an online car loan refinance company, and we're tasked with optimizing the company's marketing to maximize the number of car loans the company refinances. Here's two different approaches we might take: We a/b test hundreds of different ad spend strategies, marketing copy permutations, banner images, landing page layouts, etc. Ideally, we find a particular combination works especially well. We obtain some anonymized data from a credit agency on people with car loans. Ideally, we learn something about the market - e.g. maybe subprime borrowers usually either declare bankruptcy or dramatically increase their credit score within two years of taking a loan. The first strategy is black-box: we don't need to know anything about who our potential customers are, what they want, the psychology of clicking on ads, etc. We can treat our marketing pipeline as a black box and fiddle with its inputs to see what works. The second strategy is gears-level, the exact opposite of black-box: the whole point is to learn who our potential customers are, breaking open the black box and looking at the internal gears. These aren't mutually exclusive, and they have different relative advantages. Some upsides of black-box: Black-box is usually cheaper and easier, since the code involved is pretty standard and we don't need to track down external data. Gears-level strategies require more custom work and finding particular data. Black-box yields direct benefits when it works, whereas gears-level requires an extra step to translate whatever insights we find into actual improvements. On the other hand: Gears-level insights can highlight ideas we wouldn't even have thought to try, whereas black-box just tests the things we think to test. When some tests are expensive (e.g. integrating with a new ad channel), gears-level knowledge can tell us which tests are most likely to be worthwhile. Black-box optimization is subject to Go...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Semitechnical Introductory Dialogue on Solomonoff Induction , published by Eliezer Yudkowsky on the AI Alignment Forum. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. (Originally posted in December 2015: A dialogue between Ashley, a computer scientist who's never heard of Solomonoff's theory of inductive inference, and Blaine, who thinks it is the best thing since sliced bread.) i. Unbounded analysis ASHLEY: Good evening, Msr. Blaine. BLAINE: Good evening, Msr. Ashley. ASHLEY: I've heard there's this thing called "Solomonoff's theory of inductive inference". BLAINE: The rumors have spread, then. ASHLEY: Yeah, so, what the heck is that about? BLAINE: Invented in the 1960s by the mathematician Ray Solomonoff, the key idea in Solomonoff induction is to do sequence prediction by using Bayesian updating on a prior composed of a mixture of all computable probability distributions ASHLEY: Wait. Back up a lot. Before you try to explain what Solomonoff induction is, I'd like you to try to tell me what it does, or why people study it in the first place. I find that helps me organize my listening. Right now I don't even know why I should be interested in this. BLAINE: Um, okay. Let me think for a second... ASHLEY: Also, while I can imagine things that "sequence prediction" might mean, I haven't yet encountered it in a technical context, so you'd better go a bit further back and start more at the beginning. I do know what "computable" means and what a "probability distribution" is, and I remember the formula for Bayes's Rule although it's been a while. BLAINE: Okay. So... one way of framing the usual reason why people study this general field in the first place, is that sometimes, by studying certain idealized mathematical questions, we can gain valuable intuitions about epistemology. That's, uh, the field that studies how to reason about factual questions, how to build a map of reality that reflects the territory ASHLEY: I have some idea what 'epistemology' is, yes. But I think you might need to start even further back, maybe with some sort of concrete example or something. BLAINE: Okay. Um. So one anecdote that I sometimes use to frame the value of computer science to the study of epistemology is Edgar Allen Poe's argument in 1833 that chess was uncomputable. ASHLEY: That doesn't sound like a thing that actually happened. BLAINE: I know, but it totally did happen and not in a metaphorical sense either! Edgar Allen Poe wrote an essay explaining why no automaton would ever be able to play chess, and he specifically mentioned "Mr. Babbage's computing engine" as an example. You see, in the nineteenth century, there was for a time this sensation known as the Mechanical Turk—supposedly a machine, an automaton, that could play chess. At the grandmaster level, no less. Now today, when we're accustomed to the idea that it takes a reasonably powerful computer to do that, we can know immediately that the Mechanical Turk must have been a fraud and that there must have been a concealed operator inside—a person with dwarfism, as it turned out. Today we know that this sort of thing is hard to build into a machine. But in the 19th century, even that much wasn't known. So when Edgar Allen Poe, who besides being an author was also an accomplished magician, set out to write an essay about the Mechanical Turk, he spent the second half of the essay dissecting what was known about the Turk's appearance to (correctly) figure out where the human operator was hiding. But Poe spent the first half of the essay arguing that no automaton—nothing like Mr. Babbage's computing engine—could possibly play chess, which was how he knew a priori that the Turk had a concealed human operator. ASHLEY: And what was Poe's argument? BLAINE: Poe observed that in an algebraical p...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Belief in Belief , published by Eliezer Yudkowsky on the AI Alignment Forum. Carl Sagan once told a parable of someone who comes to us and claims: “There is a dragon in my garage.” Fascinating! We reply that we wish to see this dragon—let us set out at once for the garage! “But wait,” the claimant says to us, “it is an invisible dragon.” Now as Sagan points out, this doesn’t make the hypothesis unfalsifiable. Perhaps we go to the claimant’s garage, and although we see no dragon, we hear heavy breathing from no visible source; footprints mysteriously appear on the ground; and instruments show that something in the garage is consuming oxygen and breathing out carbon dioxide. But now suppose that we say to the claimant, “Okay, we’ll visit the garage and see if we can hear heavy breathing,” and the claimant quickly says no, it’s an inaudible dragon. We propose to measure carbon dioxide in the air, and the claimant says the dragon does not breathe. We propose to toss a bag of flour into the air to see if it outlines an invisible dragon, and the claimant immediately says, “The dragon is permeable to flour.” Carl Sagan used this parable to illustrate the classic moral that poor hypotheses need to do fast footwork to avoid falsification. But I tell this parable to make a different point: The claimant must have an accurate model of the situation somewhere in their mind, because they can anticipate, in advance, exactly which experimental results they’ll need to excuse. Some philosophers have been much confused by such scenarios, asking, “Does the claimant really believe there’s a dragon present, or not?” As if the human brain only had enough disk space to represent one belief at a time! Real minds are more tangled than that. There are different types of belief; not all beliefs are direct anticipations. The claimant clearly does not anticipate seeing anything unusual upon opening the garage door. Otherwise they wouldn’t make advance excuses. It may also be that the claimant’s pool of propositional beliefs contains the free-floating statement There is a dragon in my garage. It may seem, to a rationalist, that these two beliefs should collide and conflict even though they are of different types. Yet it is a physical fact that you can write “The sky is green!” next to a picture of a blue sky without the paper bursting into flames. The rationalist virtue of empiricism is supposed to prevent us from making this class of mistake. We’re supposed to constantly ask our beliefs which experiences they predict, make them pay rent in anticipation. But the dragon-claimant’s problem runs deeper, and cannot be cured with such simple advice. It’s not exactly difficult to connect belief in a dragon to anticipated experience of the garage. If you believe there’s a dragon in your garage, then you can expect to open up the door and see a dragon. If you don’t see a dragon, then that means there’s no dragon in your garage. This is pretty straightforward. You can even try it with your own garage. No, this invisibility business is a symptom of something much worse. Depending on how your childhood went, you may remember a time period when you first began to doubt Santa Claus’s existence, but you still believed that you were supposed to believe in Santa Claus, so you tried to deny the doubts. As Daniel Dennett observes, where it is difficult to believe a thing, it is often much easier to believe that you ought to believe it. What does it mean to believe that the Ultimate Cosmic Sky is both perfectly blue and perfectly green? The statement is confusing; it’s not even clear what it would mean to believe it—what exactly would be believed, if you believed. You can much more easily believe that it is proper, that it is good and virtuous and beneficial, to believe that the Ultimate Cosmic Sky is both perfectly...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Reflections on rationality a year out, published by [anonymous] on the AI Alignment Forum. Edited for concreteness. Exactly one year ago, LessWrong helped me change my mind about something important. Since then, my life has been changing very rapidly, as a direct result of the rationalist community. I got in touch with other rationalists in person, which made my social life vastly more interesting (not to say surreal). My plans for the future have definitely shifted a bit. I began a deliberate habit of trying new things and learning new skills, and facing up to my flaws, often with advice from LessWrongers or IRL rationalist friends. A few examples: I improved my diet (paleo), tried yoga, took up cognitive behavioral therapy to work on some chronic insecurities, moved Python from the "wish I knew" box to the "have a detailed plan to learn" box, dared to publish some popular-science articles under my real name, learned to do Fermi calculations in my head. I also noticed that my habits of thought have been changing: for one thing, I'm getting better calibrated about probabilities -- I'm better at estimating how I did on schoolwork. For another thing, I'm getting better at not reflexively dismissing non-standard ideas: the first time someone mentioned me that a good statistician could make a lot of money in car insurance by finding new correlations to monetize, I thought "Car insurance? Hmph, low status." The second time I heard that suggestion, about five months later, I thought "Hey, that's a decent idea." Some of these changes have begun to show results -- the time-management habits I came up with have started to improve my academic performance, and I notice I'm far less inhibited about taking the initiative to work on projects (I have a couple of interesting balls in the air now, including a business idea and some volunteer work for SIAI, whereas I used to be very reluctant to volunteer for things.) I've become much more open to cold-emailing people who work on interesting things (on one occasion I got a job offer out of an AI researcher); I'm more comfortable viewing myself as a junior member of the Interesting-People Club. I made a unilateral decision to be happier, and though I hate to jinx it, I think it's working. I say this just to offer evidence that something about "rationality" works. I'm not sure what it is; many of the components of LessWrong-style rationality exist elsewhere (cognitive biases are fairly common knowledge; self-improvement hacks aren't unique to LessWrong; Bayesian statistics wasn't news to me when I got here). If anything, it's the sense that rationality can be an art, a superpower, a movement. It's the very fact of consolidating and giving a name and culture to the ideas surrounding how humans can think clearly. I'm never sure how much of that is a subjective primate in-group thing, but I'm hesitant to be too suspicious -- I don't want to blow out the spark before the fire has even started. My point is, there's something here that's worthwhile. It's not just social hour for nerds (not that we can't enjoy that aspect) -- it actually is possible to reach out to people and make a difference in how they live and see the world. Once upon a time -- it seems like ages ago -- I used to envy a certain kind of person. The kind who has confidence that he can make a decent stab at ethical behavior without the threat of divine wrath. The kind who thinks that human beings have something to be proud of, that we're getting better at understanding the world and fitfully reducing suffering and injustice. The kind who thinks that he, personally, has some chance to make a valuable contribution. The kind who's audacious, who won't let anybody tell him what to think. The kind who whistles as he wins. Bertrand Russell seemed to be like that; also Robert Heinle...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Zombies: The Movie , published by on the AI Alignment Forum. FADE IN around a serious-looking group of uniformed military officers. At the head of the table, a senior, heavy-set man, GENERAL FRED, speaks. GENERAL FRED: The reports are confirmed. New York has been overrun... by zombies. COLONEL TODD: Again? But we just had a zombie invasion 28 days ago! GENERAL FRED: These zombies... are different. They're... philosophical zombies. CAPTAIN MUDD: Are they filled with rage, causing them to bite people? COLONEL TODD: Do they lose all capacity for reason? GENERAL FRED: No. They behave... exactly like we do... except that they're not conscious. (Silence grips the table.) COLONEL TODD: Dear God. GENERAL FRED moves over to a computerized display. GENERAL FRED: This is New York City, two weeks ago. The display shows crowds bustling through the streets, people eating in restaurants, a garbage truck hauling away trash. GENERAL FRED: This... is New York City... now. The display changes, showing a crowded subway train, a group of students laughing in a park, and a couple holding hands in the sunlight. COLONEL TODD: It's worse than I imagined. CAPTAIN MUDD: How can you tell, exactly? COLONEL TODD: I've never seen anything so brutally ordinary. A lab-coated SCIENTIST stands up at the foot of the table. SCIENTIST: The zombie disease eliminates consciousness without changing the brain in any way. We've been trying to understand how the disease is transmitted. Our conclusion is that, since the disease attacks dual properties of ordinary matter, it must, itself, operate outside our universe. We're dealing with an epiphenomenal virus. GENERAL FRED: Are you sure? SCIENTIST: As sure as we can be in the total absence of evidence. GENERAL FRED: All right. Compile a report on every epiphenomenon ever observed. What, where, and who. I want a list of everything that hasn't happened in the last fifty years. CAPTAIN MUDD: If the virus is epiphenomenal, how do we know it exists? SCIENTIST: The same way we know we're conscious. CAPTAIN MUDD: Oh, okay. GENERAL FRED: Have the doctors made any progress on finding an epiphenomenal cure? SCIENTIST: They've tried every placebo in the book. No dice. Everything they do has an effect. GENERAL FRED: Have you brought in a homeopath? SCIENTIST: I tried, sir! I couldn't find any! GENERAL FRED: Excellent. And the Taoists? SCIENTIST: They refuse to do anything! GENERAL FRED: Then we may yet be saved. COLONEL TODD: What about David Chalmers? Shouldn't he be here? GENERAL FRED: Chalmers... was one of the first victims. COLONEL TODD: Oh no. (Cut to the INTERIOR of a cell, completely walled in by reinforced glass, where DAVID CHALMERS paces back and forth.) DOCTOR: David! David Chalmers! Can you hear me? CHALMERS: Yes. NURSE: It's no use, doctor. CHALMERS: I'm perfectly fine. I've been introspecting on my consciousness, and I can't detect any difference. I know I would be expected to say that, but The DOCTOR turns away from the glass screen in horror. DOCTOR: His words, they... they don't mean anything. CHALMERS: This is a grotesque distortion of my philosophical views. This sort of thing can't actually happen! DOCTOR: Why not? NURSE: Yes, why not? CHALMERS: Because (Cut to two POLICE OFFICERS, guarding a dirt road leading up to the imposing steel gate of a gigantic concrete complex. On their uniforms, a badge reads "BRIDGING LAW ENFORCEMENT AGENCY".) OFFICER 1: You've got to watch out for those clever bastards. They look like humans. They can talk like humans. They're identical to humans on the atomic level. But they're not human. OFFICER 2: Scumbags. The huge noise of a throbbing engine echoes over the hills. Up rides the MAN on a white motorcycle. The MAN is wearing black sunglasses and a black leather business suit with a black leather tie and silver me...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Illusion of Transparency: Why No One Understands You , published by Eliezer Yudkowsky on the AI Alignment Forum. In hindsight bias, people who know the outcome of a situation believe the outcome should have been easy to predict in advance. Knowing the outcome, we reinterpret the situation in light of that outcome. Even when warned, we can’t de-interpret to empathize with someone who doesn’t know what we know. Closely related is the illusion of transparency: We always know what we mean by our words, and so we expect others to know it too. Reading our own writing, the intended interpretation falls easily into place, guided by our knowledge of what we really meant. It’s hard to empathize with someone who must interpret blindly, guided only by the words. June recommends a restaurant to Mark; Mark dines there and discovers (a) unimpressive food and mediocre service or (b) delicious food and impeccable service. Then Mark leaves the following message on June’s answering machine: “June, I just finished dinner at the restaurant you recommended, and I must say, it was marvelous, just marvelous.” Keysar (1994) presented a group of subjects with scenario (a), and 59% thought that Mark’s message was sarcastic and that Jane would perceive the sarcasm.1 Among other subjects, told scenario (b), only 3% thought that Jane would perceive Mark’s message as sarcastic. Keysar and Barr (2002) seem to indicate that an actual voice message was played back to the subjects.2 Keysar (1998) showed that if subjects were told that the restaurant was horrible but that Mark wanted to conceal his response, they believed June would not perceive sarcasm in the (same) message:3 They were just as likely to predict that she would perceive sarcasm when he attempted to conceal his negative experience as when he had a positive experience and was truly sincere. So participants took Mark’s communicative intention as transparent. It was as if they assumed that June would perceive whatever intention Mark wanted her to perceive.4 “The goose hangs high” is an archaic English idiom that has passed out of use in modern language. Keysar and Bly (1995) told one group of subjects that “the goose hangs high” meant that the future looks good; another group of subjects learned that “the goose hangs high” meant the future looks gloomy.5 Subjects were then asked which of these two meanings an uninformed listener would be more likely to attribute to the idiom. Each group thought that listeners would perceive the meaning presented as “standard.”6 Keysar and Henly (2002) tested the calibration of speakers: Would speakers underestimate, overestimate, or correctly estimate how often listeners understood them?7 Speakers were given ambiguous sentences (“The man is chasing a woman on a bicycle.”) and disambiguating pictures (a man running after a cycling woman). Speakers were then asked to utter the words in front of addressees, and asked to estimate how many addressees understood the intended meaning. Speakers thought that they were understood in 72% of cases and were actually understood in 61% of cases. When addressees did not understand, speakers thought they did in 46% of cases; when addressees did understand, speakers thought they did not in only 12% of cases. Additional subjects who overheard the explanation showed no such bias, expecting listeners to understand in only 56% of cases. As Keysar and Barr note, two days before Germany’s attack on Poland, Chamberlain sent a letter intended to make it clear that Britain would fight if any invasion occurred. The letter, phrased in polite diplomatese, was heard by Hitler as conciliatory—and the tanks rolled. Be not too quick to blame those who misunderstand your perfectly clear sentences, spoken or written. Chances are, your words are more ambiguous than you think. 1 Boaz Keysar, “T...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Urges vs. Goals: The analogy to anticipation and belief , published by AnnaSalamon on the AI Alignment Forum. Partially in response to: The curse of identity Related to: Humans are not automatically strategic, That other kind of status, Approving reinforces low-effort behaviors. Joe studies long hours, and often prides himself on how driven he is to make something of himself. But in the actual moments of his studying, Joe often looks out the window, doodles, or drags his eyes over the text while his mind wanders. Someone sent him a link to which college majors lead to the greatest lifetime earnings, and he didn't get around to reading that either. Shall we say that Joe doesn't really care about making something of himself? The Inuit may not have 47 words for snow, but Less Wrongers do have at least two words for belief. We find it necessary to distinguish between: Anticipations, what we actually expect to see happen; Professed beliefs, the set of things we tell ourselves we “believe”, based partly on deliberate/verbal thought. This distinction helps explain how an atheistic rationalist can still get spooked in a haunted house; how someone can “believe” they’re good at chess while avoiding games that might threaten that belief [1]; and why Eliezer had to actually crash a car before he viscerally understood what his physics books tried to tell him about stopping distance going up with the square of driving speed. (I helped Anna revise this - EY.) A lot of our community technique goes into either (1) dealing with "beliefs" being an evolutionarily recent system, such that our "beliefs" often end up far screwier than our actual anticipations; or (2) trying to get our anticipations to align with more evidence-informed beliefs. And analogously - this analogy is arguably obvious, but it's deep, useful, and easy to overlook in its implications - there seem to be two major kinds of wanting: Urges: concrete emotional pulls, produced in System 1's perceptual / autonomic processes (my urge to drink the steaming hot cocoa in front of me; my urge to avoid embarrassment by having something to add to my accomplishments log) Goals: things we tell ourselves we’re aiming at, within deliberate/verbal thought and planning (I have a goal to exercise three times a week; I have a goal to reduce existential risk) Implication 1: You can import a lot of technique for "checking for screwy beliefs" into "checking for screwy goals". Urges, like anticipations, are relatively perceptual-level and automatic. They're harder to reshape and they're also harder to completely screw up. In contrast, the flexible, recent "goals" system can easily acquire goals that are wildly detached from what we actually do, wildly detached from any positive consequences, or both. Some techniques you can port straight over from "checking for screwy beliefs" to "checking for screwy goals" include: The fundamental: "What's the positive consequence?" This is the equivalent of "What's the evidence?" for beliefs. All the other cases involve not asking it, or not asking hard enough. The Hansonian: Goals as clothes / goals as tribal affiliation: “We are people who have free software (/ communism / rationality / whatever) as our goal”. Before you install Linux, do you think "What's the positive consequence of installing Linux?" or does it just seem like the sort of thing a free-software-supporter would do? (EY says: What positive consequence is achieved by marching in an Occupy Wall Street march? Can you remember anyone stating one, throughout the whole affair - "if we march, X will happen because of Y"?) Goals as a signal of one’s value as an ally: Sheila insists that she wants to get a job. We inspect her situation and she's not trying very hard to get a job. But she's in debt to a lot of her friends and is borrowing more to li...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Tails Coming Apart As Metaphor For Life , published by Scott Alexander on the AI Alignment Forum. [Epistemic status: Pretty good, but I make no claim this is original] A neglected gem from Less Wrong: Why The Tails Come Apart, by commenter Thrasymachus. It explains why even when two variables are strongly correlated, the most extreme value of one will rarely be the most extreme value of the other. Take these graphs of grip strength vs. arm strength and reading score vs. writing score: In a pinch, the second graph can also serve as a rough map of Afghanistan Grip strength is strongly correlated with arm strength. But the person with the strongest arm doesn’t have the strongest grip. He’s up there, but a couple of people clearly beat him. Reading and writing scores are even less correlated, and some of the people with the best reading scores aren’t even close to being best at writing. Thrasymachus gives an intuitive geometric explanation of why this should be; I can’t beat it, so I’ll just copy it outright: I thought about this last week when I read this article on happiness research. The summary: if you ask people to “value their lives today on a 0 to 10 scale, with the worst possible life as a 0 and the best possible life as a 10”, you will find that Scandinavian countries are the happiest in the world. But if you ask people “how much positive emotion do you experience?”, you will find that Latin American countries are the happiest in the world. If you check where people are the least depressed, you will find Australia starts looking very good. And if you ask “how meaningful would you rate your life?” you find that African countries are the happiest in the world. It’s tempting to completely dismiss “happiness” as a concept at all, but that’s not right either. Who’s happier: a millionaire with a loving family who lives in a beautiful mansion in the forest and spends all his time hiking and surfing and playing with his kids? Or a prisoner in a maximum security jail with chronic pain? If we can all agree on the millionaire – and who wouldn’t? – happiness has to at least sort of be a real concept. The solution is to understand words as hidden inferences – they refer to a multidimensional correlation rather than to a single cohesive property. So for example, we have the word “strength”, which combines grip strength and arm strength (and many other things). These variables really are heavily correlated (see the graph above), so it’s almost always worthwhile to just refer to people as being strong or weak. I can say “Mike Tyson is stronger than an 80 year old woman”, and this is better than having to say “Mike Tyson has higher grip strength, arm strength, leg strength, torso strength, and ten other different kinds of strength than an 80 year old woman.” This is necessary to communicate anything at all and given how nicely all forms of strength correlate there’s no reason not to do it. But the tails still come apart. If we ask whether Mike Tyson is stronger than some other very impressive strong person, the answer might very well be “He has better arm strength, but worse grip strength”. Happiness must be the same way. It’s an amalgam between a bunch of correlated properties like your subjective well-being at any given moment, and the amount of positive emotions you feel, and how meaningful your life is, et cetera. And each of those correlated is also an amalgam, and so on to infinity. And crucially, it’s not an amalgam in the sense of “add subjective well-being, amount of positive emotions, and meaningfulness and divide by three”. It’s an unprincipled conflation of these that just denies they’re different at all. Think of the way children learn what happiness is. I don’t actually know how children learn things, but I imagine something like this. The child sees the millio...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Thoughts on Human Models, published by Ramana Kumar, Scott Garrabrant on the AI Alignment Forum. Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. Human values and preferences are hard to specify, especially in complex domains. Accordingly, much AGI safety research has focused on approaches to AGI design that refer to human values and preferences indirectly, by learning a model that is grounded in expressions of human values (via stated preferences, observed behaviour, approval, etc.) and/or real-world processes that generate expressions of those values. There are additionally approaches aimed at modelling or imitating other aspects of human cognition or behaviour without an explicit aim of capturing human preferences (but usually in service of ultimately satisfying them). Let us refer to all these models as human models. In this post, we discuss several reasons to be cautious about AGI designs that use human models. We suggest that the AGI safety research community put more effort into developing approaches that work well in the absence of human models, alongside the approaches that rely on human models. This would be a significant addition to the current safety research landscape, especially if we focus on working out and trying concrete approaches as opposed to developing theory. We also acknowledge various reasons why avoiding human models seems difficult. Problems with Human Models To be clear about human models, we draw a rough distinction between our actual preferences (which may not be fully accessible to us) and procedures for evaluating our preferences. The first thing, actual preferences, is what humans actually want upon reflection. Satisfying our actual preferences is a win. The second thing, procedures for evaluating preferences, refers to various proxies for our actual preferences such as our approval, or what looks good to us (with necessarily limited information or time for thinking). Human models are in the second category; consider, as an example, a highly accurate ML model of human yes/no approval on the set of descriptions of outcomes. Our first concern, described below, is about overfitting to human approval and thereby breaking its connection to our actual preferences. (This is a case of Goodhart’s law.) Less Independent Audits Imagine we have built an AGI system and we want to use it to design the mass transit system for a new city. The safety problems associated with such a project are well recognised; suppose we are not completely sure we have solved them, but are confident enough to try anyway. We run the system in a sandbox on some fake city input data and examine its outputs. Then we run it on some more outlandish fake city data to assess robustness to distributional shift. The AGI’s outputs look like reasonable transit system designs and considerations, and include arguments, metrics, and other supporting evidence that they are good. Should we be satisfied and ready to run the system on the real city’s data, and to implement the resulting proposed design? We suggest that an important factor in the answer to this question is whether the AGI system was built using human modelling or not. If it produced a solution to the transit design problem (that humans approve of) without human modelling, then we would more readily trust its outputs. If it produced a solution we approve of with human modelling, then although we expect the outputs to be in many ways about good transit system design (our actual preferences) and in many ways suited to being approved by humans, to the extent that these two targets come apart we must worry about having overfit to the human model at the expense of the good design. (Why not the other way around? Because our assessment of the sandboxed results uses human judgement, not an in...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Forum participation as a research strategy , published by Wei_Dai on the AI Alignment Forum. Previously: Online discussion is better than pre-publication peer review, Disincentives for participating on LW/AF Recently I've noticed a cognitive dissonance in myself, where I can see that my best ideas have come from participating on various mailing lists and forums (such as cypherpunks, extropians, SL4, everything-list, LessWrong and AI Alignment Forum), and I've received a certain amount of recognition as a result, but when someone asks me what I actually do as an "independent researcher", I'm embarrassed to say that I mostly comment on other people's posts, participate in online discussions, and occasionally a new idea pops into my head and I write it down as a blog/forum post of my own. I guess that's because I imagine it doesn't fit most people's image of what a researcher's work consists of. Once I noticed this, the tension is easy to resolve - in this post I'm going to proclaim/endorse forum participation (aka commenting) as a productive research strategy that I've managed to stumble upon, and recommend it to others (at least to try). Note that this is different from saying that forum/blog posts are a good way for a research community to communicate. It's about individually doing better as researchers. Benefits of Forum Participation (FP) FP takes little effort / will power In other words it feels more like play than work, which means I rarely have issues with not wanting to do something that I think is important to do (i.e., akrasia), the only exception being that writing posts seems to take more effort so occasionally I spend my time writing comments when I perhaps should write posts instead. (This is the part of this post that I think may be least likely to generalize to other people. It could be that I'm an extreme outlier in finding FP so low-effort. However it might also be the case that it becomes low effort for most people to write comments once they've had enough practice in it.) FP is a good way to notice missing background knowledge and provides incentives to learn missing knowledge If you read a post with an intention to question or comment on it, it's pretty easy to notice that it assumes some background knowledge that you lack. The desire to not ask a "stupid" question or make a "stupid" comment provides powerful incentive to learn the miss knowledge. FP is a good way to stay up to date on everyone else's latest research It's often a good idea to stay up to date on other people's research, but sometimes one isn't highly motivated to do so. FP seems to make that easier. For example, I wasn't following Stuart's research on counterfactual oracles, until the recent contest drew my attention and desire to participate, and I ended up reading the latest posts on CO in order to understand the current state of the art on that topic, which turned out to be pretty interesting. Arguments that are generated in reaction to some specific post or discussion can be of general value It's not infrequent that I come up with an argument in response to some post or discussion thread, and later expand or follow up that argument into a post because it seems to apply more generally than to just that post/discussion. Here is one such example. FP generates new ideas via cross-fertilization FP incentivizes one to think deeply about many threads of research, and often (at least for me) an idea pops into my head that seems to combine various partial ideas floating in the ether into a coherent or semi-coherent whole (e.g., UDT), or is the result of applying or analogizing someone else's latest idea to a different topic (e.g., "human safety problem", "philosophy as high complexity class"). FP helps prepare for efficiently communicating new ideas FP is a good way to build models of o...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Tools for keeping focused , published by benkuhn on the AI Alignment Forum. Once I realized that my attention was even scarcer than my time, I became an anti-distraction fanatic. During my weekly reviews I methodically went through my past week, figured out what had been distracting me, and tried to eliminate it or replace it with something less distracting. Over time, this has led me to find lots of tools (and ways of using my tools) that help me stay more focused. Here are some of the things I’ve started doing: Anxious yet? I aggressively disable notifications and badges so that I don’t mindlessly open distracting apps. If you’re into eliminating distractions you’ve probably already done this. But if you haven’t, it’s by far the most important thing you can do to improve your focus, so I’m putting it first anyway. I have a zero-tolerance notification policy: if an app interrupts me, I ask myself whether the interruption was valuable, and if not, the app doesn’t get to notify me anymore. This has weeded out pretty much everything except for inboxes (phone, texts, reminders) and apps with a human on the other end (ride sharing, delivery). A special dishonorable mention goes to Slack, which can easily suck away a quarter of your time without you noticing. If you use Slack and you haven’t disabled the “unread messages” badge, stop reading this post and do it now. (Consider setting a two-minute timer to remind you to quit in case you get distracted by checking your unreads, like I did while taking the screenshot above.) On the subject of Slack, I try to keep it closed as much as possible and check it in batches a few times a day. Not all workplace Slack cultures allow this, but if yours does, I highly, highly recommend it. Doing this led to my biggest single quantifiable productivity improvement.✻✻ If your workplace culture doesn’t allow you to keep Slack closed, because it requires quick Slack responses, this is a bad sign. I only check my email once per day. Gmail filters send important email to a label called “Temp Inbox” and the rest to a label called “Unimportant.” I check Temp Inbox each evening and Unimportant once a week. Since incoming email doesn’t go to the inbox, I can open Gmail to compose or search messages without getting distracted by unread messages. (I use a piece of Google Apps Script for this, but I think the Gmail UI has improved recently so that you can now do something similar with filters and it’ll have decent ergonomics.) I have a second monitor that always shows a Complice window with the task I’m currently working on. (Complice is my favorite app for making lists of what I want to do today.) This helps me recover quickly from unintentionally going down rabbit holes. I use Focus to break my habit of mindlessly checking sites. For websites that are habit-forming but still feel useful on net, like Twitter or Hacker News, Focus lets me restrict my usage to certain times of day. It’s the only website blocker I’ve used that lets me fully automate blocking things in the exact way I want. It’s especially important for me to use a website blocker that’s fully automatic, because the times when I need it most are exactly those times at which I have the least willpower to do any manual steps! I also block most websites on my phone (using the iOS built-in content blocking in whitelist mode). Unfortunately, this works less well than Focus since I sometimes want to disable it and then forget to re-enable it. I do most of my Internet reading on my Kindle via Kindle4RSS. Using RSS means I’m in control of my own feed and don’t need to visit an adversarially distracting site like Facebook to get new reading material. It’s also helpful that the Kindle delivery comes once per day at a predictable time, so I don’t have an urge to check over and over again for new c...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Felt Sense: What, Why and How, published by Kaj_Sotala on the AI Alignment Forum. While LW has seen previous discussion of Focusing, I feel like there has been relatively limited discussion of the felt sense - that is, the thing that Focusing is actually accessing. Everyone accesses felt senses all the time, but most people don't know that they are doing it. I think that being able to make the skill more explicit is really valuable, and in this post I'm going to give lots of examples of why that is and what you can do with it. Hopefully, after I'm done, you will not only know what a felt sense is (if you didn't already), but also will have difficulty understanding how you ever got by without this concept. Examples of felt senses The term "felt sense" was originally coined by the psychologist Eugene Gendlin, as a name for something that he found his clients to be accessing in their therapy sessions. Here are some examples of felt senses: Think of some person you know, maybe imagining what it feels to be like in the same room as them. You probably have some “sense” of that person, of what it is that they feel like. Likewise if you think of some fictional universe, it has something of its own feel. Harry Potter feels different from Star Wars feels different from Game of Thrones feels different from James Bond. Sometimes you will have a word “right on the tip of your tongue”; it’s as if the word is almost there, but you can’t quite reach it. When you do, you just know that it's the right word - because the "shape" of the word matches the one you were reaching for before. The felt senses of pictures Here are are a few pictures that I recently collected from the Facebook group "Steampunk Tendencies": How do you feel when you look at these pictures? What's the general vibe that unites all of these pictures? Likely you can find quite a few. If I put aside the words "steampunk" and "Victorian", next I get the word "mechanical". "Dark" also feels fitting. Whatever the vibe that you get, it's probably something different than the one you get from this collection of images: Look at the first set of images, then the second. How does it feel when you switch looking from one to the other? What kinds of changes are there in your mind and your body? I like both sets, but looking from one to the other, I notice that the forest images make me feel like my mind is opening up, whereas the steampunk ones make it close a little. Comparing the two, I feel like there's some slightly off-putting vibe in the steampunk set, that makes me prefer looking at the forest images - which I would not have noticed if I hadn't viewed them side to side. (I am guessing that some readers will have the opposite experience, of finding the forest ones off-putting compared to the steampunk ones.) Emotional and mental states as felt senses Internal emotional and mental states can also have their own felt senses. That shouldn't be very surprising, since your experience of e.g. a set of pictures is an internal mental state. Here are a few examples of felt senses from alkjash: When I solve a problem in a creative way (e.g. fix posture by turning in the shower), there’s a sensation of enlightenment at the back of my head which literally feels like my skull is opening up. The words to this feeling are “I’ve discovered a new dimension!” I sometimes sit slouched over in bed for hours at a time browsing Facebook or Reddit, playing video games, or binge-watch a season of a TV show. After getting up from the slouch, my whole body is enveloped in a haze of laziness and decay. The zombie haze is thickest inside my ribs. The words to this pressure are “Symptoms of the spreading corruption.” A piece of my social anxiety forms a hard barrier that pushes against the center of my chest. I learned the words to this feeling fr...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Cached Thoughts, published by Eliezer Yudkowsky on the AI Alignment Forum. One of the single greatest puzzles about the human brain is how the damn thing works at all when most neurons fire 10–20 times per second, or 200Hz tops. In neurology, the “hundred-step rule” is that any postulated operation has to complete in at most 100 sequential steps—you can be as parallel as you like, but you can’t postulate more than 100 (preferably fewer) neural spikes one after the other. Can you imagine having to program using 100Hz CPUs, no matter how many of them you had? You’d also need a hundred billion processors just to get anything done in realtime. If you did need to write realtime programs for a hundred billion 100Hz processors, one trick you’d use as heavily as possible is caching. That’s when you store the results of previous operations and look them up next time, instead of recomputing them from scratch. And it’s a very neural idiom—recognition, association, completing the pattern. It’s a good guess that the actual majority of human cognition consists of cache lookups. This thought does tend to go through my mind at certain times. There was a wonderfully illustrative story which I thought I had bookmarked, but couldn’t re-find: it was the story of a man whose know-it-all neighbor had once claimed in passing that the best way to remove a chimney from your house was to knock out the fireplace, wait for the bricks to drop down one level, knock out those bricks, and repeat until the chimney was gone. Years later, when the man wanted to remove his own chimney, this cached thought was lurking, waiting to pounce . . . As the man noted afterward—you can guess it didn’t go well—his neighbor was not particularly knowledgeable in these matters, not a trusted source. If he’d questioned the idea, he probably would have realized it was a poor one. Some cache hits we’d be better off recomputing. But the brain completes the pattern automatically—and if you don’t consciously realize the pattern needs correction, you’ll be left with a completed pattern. I suspect that if the thought had occurred to the man himself—if he’d personally had this bright idea for how to remove a chimney—he would have examined the idea more critically. But if someone else has already thought an idea through, you can save on computing power by caching their conclusion—right? In modern civilization particularly, no one can think fast enough to think their own thoughts. If I’d been abandoned in the woods as an infant, raised by wolves or silent robots, I would scarcely be recognizable as human. No one can think fast enough to recapitulate the wisdom of a hunter-gatherer tribe in one lifetime, starting from scratch. As for the wisdom of a literate civilization, forget it. But the flip side of this is that I continually see people who aspire to critical thinking, repeating back cached thoughts which were not invented by critical thinkers. A good example is the skeptic who concedes, “Well, you can’t prove or disprove a religion by factual evidence.” As I have pointed out elsewhere,1 this is simply false as probability theory. And it is also simply false relative to the real psychology of religion—a few centuries ago, saying this would have gotten you burned at the stake. A mother whose daughter has cancer prays, “God, please heal my daughter,” not, “Dear God, I know that religions are not allowed to have any falsifiable consequences, which means that you can’t possibly heal my daughter, so . . . well, basically, I’m praying to make myself feel better, instead of doing something that could actually help my daughter.” But people read “You can’t prove or disprove a religion by factual evidence,” and then, the next time they see a piece of evidence disproving a religion, their brain completes the pattern. Even some atheist...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Hero With A Thousand Chances , published by Eliezer Yudkowsky on the AI Alignment Forum. "Allow me to make sure I have this straight," the hero said. "I've been untimely ripped from my home world to fight unspeakable horrors, and you say I'm here because I'm lucky?" Aerhien dipped her eyelashes in elegant acknowledgment; and quietly to herself, she thought: Thirty-seven. Thirty-seven heroes who'd said just that, more or less, on arrival. Not a sign of the thought showed on her outward face, where the hero could see, or the other council members of the Eerionnath take note. Over the centuries since her accidental immortality she'd built a reputation for serenity, more or less because it seemed to be expected. "There are kinds and kinds of luck," Aerhien said serenely. "Not every person desires their personal happiness above all else. Those who are lucky in aiding others, those whose luck is great in succor and in rescue, these ones are not always happy themselves. You are here, hero, because you have a hero's luck. The boy whose dusty heirloom sword proves to be magical. The peasant girl who finds herself the heir to a great kingdom. Those who discover, in time of sudden stress, an untrained wild magic within themselves. Success born not of learning, not of skill, not of determination, but unplanned coincidence and fortunes of birth: That is a hero's luck." "Gosh," said the hero after a long, awkward pause, "thanks for the compliment." "It is not a compliment," Aerhien said, "but this is: that you have taken good advantage of your luck. Our enemy does not speak, we do not know if there is any aliveness in it to think; but it learns, or seems to learn. We have never won against it using the same trick twice. It is rare now that a hero succeeds in conceiving a genuinely new trick, for we have fought this shadow long under our sun. For this reason we have taken to summoning heroes from distant dimensions with other modes of thought; sometimes one such knows a truly new technique, and at least they fight differently. But far more often, hero, the hero wins by luck." "Huh," said the hero. He frowned; more in thought, it seemed, than in displeasure. "How... very odd. I wonder why that is. What kind of enemy can be defeated only by luck?" "A nameless enemy and null," said Aerhien. "Structureless and empty, horrible and dark, the most terrifying thing imaginable: We call it Dust. That seems to be its only desire, to tear down every bit of structure in the world, grind it into specks of perfect chaos. Always the Dust is defeated, always it takes a new shape immune to its last defeat." "I wonder," murmured the hero, "if it will run out of shapes, and then end; or if it will finally become invincible." (One of the other Eerionnath shuddered.) "I do not know," Aerhien said simply. "I do not know the nature of the Dust, nor the nature of the Counter-Force that opposes it. The Dust is terrible and our world should long since have ended. We are not fools enough to believe we could be lucky so many times by chance alone. But the Counter-Force has never acted openly; it never reveals itself except in - a hero's luck. And so we, the council Eerionnath to prevent the world from destruction, are at your disposal to command; and all the power and resource that this world holds, for your battle." And she, Aerhien, and the council Eerionnath, bowed low. Then they waited to see if the hero would demand dominions or slaves as payment, before condescending to rescue a people in distress. If so they would dispose of him, and summon another. This one, though, seemed to have at least some qualities of a true hero; his face showed no avarice, only an abstracted puzzlement. "A hidden Counter-Force..." he murmured. "Excuse me, but this is all very vague. Can you give me a specific example of a h...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 2011 Survey Results, published by Scott Alexander on the AI Alignment Forum. A big thank you to the 1090 people who took the second Less Wrong Census/Survey. Does this mean there are 1090 people who post on Less Wrong? Not necessarily. 165 people said they had zero karma, and 406 people skipped the karma question - I assume a good number of the skippers were people with zero karma or without accounts. So we can only prove that 519 people post on Less Wrong. Which is still a lot of people. I apologize for failing to ask who had or did not have an LW account. Because there are a number of these failures, I'm putting them all in a comment to this post so they don't clutter the survey results. Please talk about changes you want for next year's survey there. Of our 1090 respondents, 972 (89%) were male, 92 (8.4%) female, 7 (.6%) transexual, and 19 gave various other answers or objected to the question. As abysmally male-dominated as these results are, the percent of women has tripled since the last survey in mid-2009. We're also a little more diverse than we were in 2009; our percent non-whites has risen from 6% to just below 10%. Along with 944 whites (86%) we include 38 Hispanics (3.5%), 31 East Asians (2.8%), 26 Indian Asians (2.4%) and 4 blacks (.4%). Age ranged from a supposed minimum of 1 (they start making rationalists early these days?) to a more plausible minimum of 14, to a maximum of 77. The mean age was 27.18 years. Quartiles (25%, 50%, 75%) were 21, 25, and 30. 90% of us are under 38, 95% of us are under 45, but there are still eleven Less Wrongers over the age of 60. The average Less Wronger has aged about one week since spring 2009 - so clearly all those anti-agathics we're taking are working! In order of frequency, we include 366 computer scientists (32.6%), 174 people in the hard sciences (16%) 80 people in finance (7.3%), 63 people in the social sciences (5.8%), 43 people involved in AI (3.9%), 39 philosophers (3.6%), 15 mathematicians (1.5%), 14 statisticians (1.3%), 15 people involved in law (1.5%) and 5 people in medicine (.5%). 48 of us (4.4%) teach in academia, 470 (43.1%) are students, 417 (38.3%) do for-profit work, 34 (3.1%) do non-profit work, 41 (3.8%) work for the government, and 72 (6.6%) are unemployed. 418 people (38.3%) have yet to receive any degrees, 400 (36.7%) have a Bachelor's or equivalent, 175 (16.1%) have a Master's or equivalent, 65 people (6%) have a Ph.D, and 19 people (1.7%) have a professional degree such as an MD or JD. 345 people (31.7%) are single and looking, 250 (22.9%) are single but not looking, 286 (26.2%) are in a relationship, and 201 (18.4%) are married. There are striking differences across men and women: women are more likely to be in a relationship and less likely to be single and looking (33% men vs. 19% women). All of these numbers look a lot like the ones from 2009. 27 people (2.5%) are asexual, 119 (10.9%) are bisexual, 24 (2.2%) are homosexual, and 902 (82.8%) are heterosexual. 625 people (57.3%) described themselves as monogamous, 145 (13.3%) as polyamorous, and 298 (27.3%) didn't really know. These numbers were similar between men and women. The most popular political view, at least according to the much-maligned categories on the survey, was liberalism, with 376 adherents and 34.5% of the vote. Libertarianism followed at 352 (32.3%), then socialism at 290 (26.6%), conservativism at 30 (2.8%) and communism at 5 (.5%). 680 people (62.4%) were consequentialist, 152 (13.9%) virtue ethicist, 49 (4.5%) deontologist, and 145 (13.3%) did not believe in morality. 801 people (73.5%) were atheist and not spiritual, 108 (9.9%) were atheist and spiritual, 97 (8.9%) were agnostic, 30 (2.8%) were deist or pantheist or something along those lines, and 39 people (3.5%) described themselves as theists (20 committed plus 19...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Imitation is the Sincerest Form of Argument, published by palladias on the AI Alignment Forum. I recently gave a talk at Chicago Ideas Week on adapting Turing Tests to have better, less mindkill-y arguments, and this is the precis for folks who would prefer not to sit through the video (which is available here). Conventional Turing Tests check whether a programmer can build a convincing facsimile of a human conversationalist. The test has turned out to reveal less about machine intelligence than human intelligence. (Anger is really easy to fake, since fights can end up a little more Markov chain-y, where you only need to reply to the most recent rejoinder and can ignore what came before). Since normal Turing Tests made us think more about our model of human conversation, economist Bryan Caplan came up with a way to use them to make us think more usefully about our models of our enemies. After Paul Krugman disparaged Caplan's brand of libertarian economics, Caplan challenged him to an ideological Turing Test, where both players would be human, but would be trying to accurately imitate each other. Caplan and Krugman would each answer questions about their true beliefs honestly, and then would fill out the questionaire again in persona inimici - trying to guess the answers given by the other side. Caplan was willing to bet that he understood Krugman's position well enough to mimic it, but Krugman would be easily spotted as a fake!Caplan. Krugman didn't take him up on the offer, but I've run a couple iterations of the test for my religion/philosophy blog. The first year, some of the most interesting results were the proxy variables people were using, that weren't as strong as indicators as the judges thought. (One Catholic coasted through to victory as a faux atheist, since many of the atheist judges thought there was no way a Christian would appreciate the webcomic SMBC). The trouble was, the Christians did a lot better, since it turned out I had written boring, easy to guess questions for the true and faux atheists. The second year, I wrote weirder questions, and the answers were a lot more diverse and surprising (and a number of the atheist participants called out each other as fakes or just plain wrong, since we'd gotten past the shallow questions from year one, and there's a lot of philosophical diversity within atheism). The exercise made people get curious about what it was their opponents actually thought and why. It helped people spot incorrect stereotypes of an opposing side and faultlines they'd been ignoring within their own. Personally, (and according to other participants) it helped me have an argument less antagonistically. Instead of just trying to find enough of a weak point to discomfit my opponent, I was trying to build up a model of how they thought, and I needed their help to do it. Taking a calm, inquisitive look at an opponent's position might teach me that my position is wrong, or has a gap I need to investigate. But even if my opponent is just as wrong as zer seemed, there's still a benefit to me. Having a really detailed, accurate model of zer position may help me show them why it's wrong, since now I can see exactly where it rasps against reality. And even if my conversation isn't helpful to them, it's interesting for me to see what they were missing. I may be correct in this particular argument, but the odds are good that I share the rationalist weak-point that is keeping them from noticing the error. I'd like to be able to see it more clearly so I can try and spot it in my own thought. (Think of this as the shift from "How the hell can you be so dumb?!" to "How the hell can you be so dumb?"). When I get angry, I'm satisfied when I beat my interlocutor. When I get curious, I'm only satisfied when I learn something new. Thanks for listening. To...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Rational Toothpaste: A Case Study, published by badger on the AI Alignment Forum. Inspired by Konkvistador's comment Posts titled "Rational ___-ing" or "A Rational Approach to ____" induce groans among a sizeable contingent here, myself included. However, inflationary use of "rational" and its transformation into an applause light is only one part of the problem. These posts tend to revolve around specific answers, rather than the process of how to find answers. I claim a post on "rational toothpaste buying" could be on-topic and useful, if correctly written to illustrate determining goals, assessing tradeoffs, and implementing the final conclusions. A post detailing the pros and cons of various toothpaste brands is for a dentistry or personal hygiene forum; a post about algorithms for how to determine the best brands or whether to do so at all is for a rationality forum. This post is my shot at showing what this would look like. At one point or another, we've all asked ourselves, "what is the most rational toothpaste?" After all, despite the length of the sequences, I've yet to see Eliezer's endorsed personal hygiene products. What is an aspiring rationalist to do? Step one is to throw out the question entirely. The most rational toothpaste does not exist, nor does the best toothpaste nor the optimal toothpaste. These adjectives are only applicable relative to particular goals, constraints, and contexts. Avoid the mistake of assuming optimality is a trait inherent to toothpaste, rather than a joint function of the toothpaste and who is using it. Similarly, the best programming language, the best footwear, the best way to write, and the best job are all under-specified. Even before determining what you are looking for in toothpaste, take one more step back. Is optimizing your toothpaste worth the time and attention? First, there is the issue of whether improved dental care is worth it, and then, whether better toothpaste is the best means of improving your teeth. While recognizing "optimal" varies across individuals, goals might be aligned closely enough that something can be identified as approximately optimal. The search costs of finding the perfect solution could outweigh going with an approximate solution. Toothpaste seems like a product where users have essentially the same needs or fall into a small number of categories, unlike the best place to reside, which depends on a large number of individual factors. As a result, toothpaste is probably already well optimized for you and picking anything up off the shelf of a supermarket should do fine, but a product you use everyday still deserves a few minutes of deliberate analysis. One basic algorithm for tackling these issue: What do you actually want to accomplish? Two approaches for determining goals: 1. (Bottom-up) List all the goals your current actions or the first proposed solution might fulfill. 2. (Top-down) List your basic values, major goals, mid-level goals, etc until you reach the relevant scope. How much are you actually willing to spend in time and experimentation costs for improvements to these goals? Quickly estimate the value of information. Generate actions that might suit each goal. Focus on quantity. Gather information. Is there published research on the topic? Who might have good advice? Are there quick experiments that can be run? Filter actions and form a plan. Are you satisfied with implementing the conclusion reached? If you feel a hang-up, try optimizing specifically for that.1 Following a bottom-up approach, why do I use toothpaste at all? Toothpaste can decrease risk of cavities, whiten teeth, improve bad breath, or make brushing more pleasant. A change I make could be relevant for at least five years. Beyond that point, I discount the future enough not to worry about it, with the chances ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Something to Protect, published by Eliezer Yudkowsky on the AI Alignment Forum. In the gestalt of (ahem) Japanese fiction, one finds this oft-repeated motif: Power comes from having something to protect. I'm not just talking about superheroes that power up when a friend is threatened, the way it works in Western fiction. In the Japanese version it runs deeper than that. In the X saga it's explicitly stated that each of the good guys draw their power from having someone—one person—who they want to protect. Who? That question is part of X's plot—the "most precious person" isn't always who we think. But if that person is killed, or hurt in the wrong way, the protector loses their power—not so much from magical backlash, as from simple despair. This isn't something that happens once per week per good guy, the way it would work in a Western comic. It's equivalent to being Killed Off For Real—taken off the game board. The way it works in Western superhero comics is that the good guy gets bitten by a radioactive spider; and then he needs something to do with his powers, to keep him busy, so he decides to fight crime. And then Western superheroes are always whining about how much time their superhero duties take up, and how they'd rather be ordinary mortals so they could go fishing or something. Similarly, in Western real life, unhappy people are told that they need a "purpose in life", so they should pick out an altruistic cause that goes well with their personality, like picking out nice living-room drapes, and this will brighten up their days by adding some color, like nice living-room drapes. You should be careful not to pick something too expensive, though. In Western comics, the magic comes first, then the purpose: Acquire amazing powers, decide to protect the innocent. In Japanese fiction, often, it works the other way around. Of course I'm not saying all this to generalize from fictional evidence. But I want to convey a concept whose deceptively close Western analogue is not what I mean. I have touched before on the idea that a rationalist must have something they value more than "rationality": The Art must have a purpose other than itself, or it collapses into infinite recursion. But do not mistake me, and think I am advocating that rationalists should pick out a nice altruistic cause, by way of having something to do, because rationality isn't all that important by itself. No. I am asking: Where do rationalists come from? How do we acquire our powers? It is written in the Twelve Virtues of Rationality: How can you improve your conception of rationality? Not by saying to yourself, "It is my duty to be rational." By this you only enshrine your mistaken conception. Perhaps your conception of rationality is that it is rational to believe the words of the Great Teacher, and the Great Teacher says, "The sky is green," and you look up at the sky and see blue. If you think: "It may look like the sky is blue, but rationality is to believe the words of the Great Teacher," you lose a chance to discover your mistake. Historically speaking, the way humanity finally left the trap of authority and began paying attention to, y'know, the actual sky, was that beliefs based on experiment turned out to be much more useful than beliefs based on authority. Curiosity has been around since the dawn of humanity, but the problem is that spinning campfire tales works just as well for satisfying curiosity. Historically speaking, science won because it displayed greater raw strength in the form of technology, not because science sounded more reasonable. To this very day, magic and scripture still sound more reasonable to untrained ears than science. That is why there is continuous social tension between the belief systems. If science not only worked better than magic, but also sounded more int...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Don't Revere The Bearer Of Good Info, published by CarlShulman on the AI Alignment Forum. Follow-up to: Every Cause Wants To Be A Cult, Cultish Countercultishness One of the classic demonstrations of the Fundamental Attribution Error is the 'quiz study' of Ross, Amabile, and Steinmetz (1977). In the study, subjects were randomly assigned to either ask or answer questions in quiz show style, and were observed by other subjects who were asked to rate them for competence/knowledge. Even knowing that the assignments were random did not prevent the raters from rating the questioners higher than the answerers. Of course, when we rate individuals highly the affect heuristic comes into play, and if we're not careful that can lead to a super-happy death spiral of reverence. Students can revere teachers or science popularizers (even devotion to Richard Dawkins can get a bit extreme at his busy web forum) simply because the former only interact with the latter in domains where the students know less. This is certainly a problem with blogging, where the blogger chooses to post in domains of expertise. Specifically, Eliezer's writing at Overcoming Bias has provided nice introductions to many standard concepts and arguments from philosophy, economics, and psychology: the philosophical compatibilist account of free will, utility functions, standard biases, and much more. These are great concepts, and many commenters report that they have been greatly influenced by their introductions to them at Overcoming Bias, but the psychological default will be to overrate the messenger. This danger is particularly great in light of his writing style, and when the fact that a point is already extant in the literature, and is either being relayed or reinvented, isn't noted. To address a few cases of the latter: Gary Drescher covered much of the content of Eliezer's Overcoming Bias posts (mostly very well), from timeless physics to Newcomb's problems to quantum mechanics, in a book back in May 2006, while Eliezer's irrealist meta-ethics would be very familiar to modern philosophers like Don Loeb or Josh Greene, and isn't so far from the 18th century philosopher David Hume. If you're feeling a tendency to cultish hero-worship, reading such independent prior analyses is a noncultish way to diffuse it, and the history of science suggests that this procedure will be applicable to almost anyone you're tempted to revere. Wallace invented the idea of evolution through natural selection independently of Darwin, and Leibniz and Newton independently developed calculus. With respect to our other host, Hans Moravec came up with the probabilistic Simulation Argument long before Nick Bostrom became known for reinventing it (possibly with forgotten influence from reading the book, or its influence on interlocutors). When we post here we can make an effort to find and explicitly acknowledge such influences or independent discoveries, to recognize the contributions of Rational We, as well as Me. Even if you resist revering the messenger, a well-written piece that purports to summarize a field can leave you ignorant of your ignorance. If you only read the National Review or The Nation you will pick up a lot of political knowledge, including knowledge about the other party/ideology, at least enough to score well on political science surveys. However, that very knowledge means that missing pieces favoring the other side can be more easily ignored: someone might not believe that the other side is made up of Evil Mutants with no reasons at all, and might be tempted to investigate, but ideological media can provide reasons that are plausible yet not so plausible as to be tempting to their audience. For a truth-seeker, beware of explanations of the speaker's opponents. This sort of intentional slanting and misplaced tru...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Contrite Strategies and The Need For Standards, published by sarahconstantin on the AI Alignment Forum. Epistemic Status: Confident There’s a really interesting paper from 1996 called The Logic of Contrition, which I’ll summarize here. In it, the authors identify a strategy called “Contrite Tit For Tat”, which does better than either Pavlov or Generous Tit For Tat in Iterated Prisoner’s Dilemma. In Contrite Tit For Tat, the player doesn’t only look at what he and the other player played on the last term, but also another variable, the standing of the players, which can be good or bad. If Bob defected on Alice last round but Alice was in good standing, then Bob’s standing switches to bad, and Alice defects against Bob. If Bob defected on Alice last round but Alice was in bad standing, then Bob’s standing stays good, and Alice cooperates with Bob. If Bob cooperated with Alice last round, Bob keeps his good standing, and Alice cooperates. This allows two Contrite Tit For Tat players to recover quickly from accidental defections without defecting against each other forever; D/C -> C/D -> C/C But, unlike Pavlov, it consistently resists the “always defect” strategy D/C -> D/D -> D/D -> D/D . Like TFT (Tit For Tat) and unlike Pavlov and gTFT (Generous Tit For Tat), cTFT (Contrite Tit For Tat) can invade a population of all Defectors. A related contrite strategy is Remorse. Remorse cooperates only if it is in bad standing, or if both players cooperated in the previous round. In other words, Remorse is more aggressive; unlike cTFT, it can attack cooperators. Against the strategy “always cooperate”, cTFT always cooperates but Remorse alternates cooperating and defecting: C/C -> C/D -> C/C -> C/D . And Remorse defends effectively against defectors: D/C -> D/D -> D/D -> D/D. But if one Remorse accidentally defects against another, recovery is more difficult: C/D -> D/C -> D/D -> C/D -> . If the Prisoner’s Dilemma is repeated a large but finite number of times, cTFT is an evolutionarily stable state in the sense that you can’t do better for yourself when playing against a cTFT player through doing anything that deviates from what cTFT would recommend. This implies that no other strategy can successfully invade a population of all cTFT’s. REMORSE can sometimes be invaded by strategies better at cooperating with themselves, while Pavlov can sometimes be invaded by Defectors, depending on the payoff matrix; but for all Prisoner’s Dilemma payoff matrices, cTFT resists invasion. Defector and a similar strategy called Grim Trigger (if a player ever defects on you, keep defecting forever) are evolutionarily stable, but not good outcomes — they result in much lower scores for everyone in the population than TFT or its variants. By contrast, a whole population that adopts cTFT, gTFT, Pavlov, or Remorse on average gets the payoff from cooperating each round. The bottom line is, adding “contrition” to TFT makes it quite a bit better, and allows it to keep pace with Pavlov in exploiting TFT’s, while doing better than Pavlov at exploiting Defectors. This is no longer true if we add noise in the perception of good or bad standing; contrite strategies, like TFT, can get stuck defecting against each other if they erroneously perceive bad standing. The moral of the story is that there’s a game-theoretic advantage to not only having reciprocity (TFT) but standards (cTFT), and in fact reciprocity alone is not enough to outperform strategies like Pavlov which don’t map well to human moral maxims. What do I mean by standards? There’s a difference between saying “Behavior X is better than behavior Y” and saying “Behavior Y is unacceptable.” The concept of “unacceptable” behavior functions like the concept of “standing” in the game theory paper. If I do something “unacceptable” and you respond in some ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Writing children's picture books, published by jessicata on the AI Alignment Forum. This is a linkpost for/ [the text of the post is pasted here, for redundancy] Here’s an exercise for explaining and refining your opinions about some domain, X: Imagine writing a 10-20 page children’s picture book about topic X. Be fully honest and don’t hide things (assume the child can handle being told the truth, including being told non-standard or controversial facts). Here’s a dialogue, meant to illustrate how this could work: A: What do you think about global warming? B: Uhh.. I don’t know, it seems real? A: How would you write a 10-20 page children’s picture book about global warming? B: Oh, I’d have a diagram showing carbon dioxide exiting factories and cars, floating up in the atmosphere, and staying there. Then I’d have a picture of sunlight coming through the atmosphere, bounding off the earth, then going back up, but getting blocked by the carbon dioxide, so it goes back to the earth and warms up the earth a second time. Oh, wait, if the carbon dioxide prevents the sunlight from bouncing from the earth to the sky, wouldn’t it also prevent the sunlight from entering the atmosphere in the first place? Oh, I should look that up later [NOTE: the answer is that CO2 blocks thermal radiation much more than it blocks sunlight]. Anyway, after that I’d have some diagrams showing global average temperature versus global CO2 level that show how the average temperature is tracking CO2 concentration, with some lag time. Then I’d have some quotes about scientists and information about the results of surveys. I’d show a graph showing how much the temperature would increase under different conditions. I think I’ve heard that, with substantial mitigation effort, the temperature difference might be 2 degrees Celsius from now until the end of the century [NOTE: it's actually 2 degrees from pre-industrial times till the end of the century, which is about 1 degree from now]. And I’d want to show what 2 degrees Celsius means, in terms of, say, a fraction of the difference between winter and summer. I’d also want to explain the issue of sea level rise, by showing a diagram of a glacier melting. Ice floats, so if the glacier is free-floating, then it melting doesn’t cause a sea level rise (there’s some scientific principle that says this, I don’t remember what it’s called), but if the glacier is on land, then when it melts, it causes the sea level to rise. I’d also want to show a map of the areas that would get flooded. I think some locations, like much of Florida, get flooded, so the map should show that, and there should also be a pie chart showing how much of the current population would end up underwater if they didn’t move (my current guess is that it’s between 1 percent and 10 percent, but I could be pretty wrong about this [NOTE: the answer is 30 to 80 million people, which is between about 0.4% and 1.1%]). I’d also want to talk about possible mitigation efforts. Obviously, it’s possible to reduce energy consumption (and also meat consumption, because cows produce methane which is also a greenhouse gas). So I’d want to show a chart of which things produce the most greenhouse gases (I think airplane flights and beef are especially bad), and showing the relationship between possible reductions in that and the temperature change. Also, trees take CO2 out of the atmosphere, so preserving forests is a way to prevent global warming. I’m confused about where the CO2 goes, exactly, since there’s some cycle it goes through in the forest; does it end up underground? I’d have to look this up. I’d also want to talk about the political issues, especially the disinformation in the space. There’s a dynamic where companies that pollute want to deny that man-made global warming is a real, serious problem, ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Real Rules Have No Exceptions, published by Said Achmiz on the AI Alignment Forum. (This is a comment that has been turned into a post.) From Chris_Leong’s post, “Making Exceptions to General Rules”: Suppose you make a general rule, ie. “I won’t eat any cookies”. Then you encounter a situation that legitimately feels exceptional , “These are generally considered the best cookies in the entire state”. This tends to make people torn between two threads of reasoning: Clearly the optimal strategy is to make an exception this one time and then follow the rule the rest of the time. If you break the rule this one time, then you risk dismantling the rule and ending up not following it at all. How can we resolve this? . This is my answer: Consider even a single exception to totally undermine any rule. Consequently, only follow rules with no exceptions.[1]. When you do encounter a legitimate exception to a heretofore-exceptionless rule, immediately discard the rule and replace it with a new rule—one which accounts for situations like this one, which, to the old rule, had to be exceptions. This, of course, requires a meta-rule (or, if you like, a meta-habit): Prefer simplicity in your rules. Be vigilant that your rules do not grow too complex; make sure you are not relaxing the legitimacy criteria of your exceptions. Periodically audit your rules, inspecting them for complexity; try to formulate simpler versions of complex rules. So, when you encounter an exception, you neither break the rule once but keep following it thereafter, nor break it once and risk breaking it again. If this is really an exception, then that rule is immediately and automatically nullified, because good rules ought not have exceptions. Time for a new rule. And if you’re not prepared to discard the rule and formulate a new one, well, then the exception must not be all that compelling; in which case, of course, keep following the existing rule, now and henceforth. But why do I say that good rules ought not have exceptions? Because rules already don’t have exceptions. Exceptions are a fiction. They’re a way for us to avoid admitting (sometimes to ourselves, sometimes to others) that the rule as stated, together with the criteria for deciding whether something is a “legitimate” exception, is the actual rule. The approach I describe above merely consists of making this fact explicit. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI Alignment 2018-19 ReviewΩ , published by rohinmshah on LessWrong Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. Preamble What this post is This is a review post of public work in AI alignment over 2019, with some inclusions from 2018. It has this preamble (~700 words), a short version / summary (~1.6k words), and a long version (~8.3k words). It is available as a Google Doc here. There are many areas of work that are relevant to AI alignment that I have barely touched on, such as interpretability, uncertainty estimation, adversarial examples, and assured autonomy, primarily because I have not been following these fields and wouldn’t be able to write a good summary of what has happened in them. I have also mostly focused on articles that provide some conceptual insight, and excluded or briefly linked to papers that primarily make quantitative improvements on important metrics. While such papers are obviously important (ultimately, our techniques need to work well), there isn’t much to say about them in a yearly review other than that the quantitative metric was improved. Despite these exclusions, there was still a ton of work to select from, perhaps around ~500 articles, of which over 300 have been linked to in this post. There are many interesting articles that I really enjoyed that get only a sentence of description, in which I ignore many of the points that the article makes. Most have been summarized in the Alignment Newsletter, so if you’d like to learn more about any particular link, but don’t want to read the entire thing, just search for its title in the database. What you should know about the structure of this post I am not speaking for myself; by default I am trying to explain what has been said, in a way that the authors of the articles would agree with. Any extra opinion that I add will be in italics. As a post, this is meant to be read sequentially, but the underlying structure is a graph (nodes are posts, edges connect posts that are very related). I arranged it in a sequence that highlights the most salient-to-me connections. This means that the order in which I present subtopics is very much not a reflection of what I think is most important in AI safety: in my presentation order, I focused on edges (connections) rather than nodes (subtopics). Other minor details: Any links from earlier than 2018 will have their year of publication right after the link (except for articles that were reposted as part of Alignment Forum sequences). I typically link to blog posts; in several cases there is also an associated paper that I have not linked. How to read this post I have put the most effort into making the prose of the long version read smoothly. The hierarchical organization is comparatively less coherent; this is partly because I optimized the prose, and partly because AI safety work is hard to cluster. As a result, for those willing to put in the effort, I’d recommend reading the long version directly, without paying too much attention to the hierarchy. If you have less time, or are less interested in the minutiae of AI alignment research, the short version is for you. Since I don’t name authors or organizations, you may want to take this as your opportunity to form beliefs about which arguments in AI alignment are important based on the ideas (as opposed to based on trust in the author of the post). People who keep up with AI alignment work might want to know which posts I’m referencing as they read, which is a bit hard since I don’t name the posts in the text. If this describes you, you should be reading this post on the Alignment Forum, where you can hover over most links to see what they link to. Alternatively, the references section in the Google Doc lists all links in the order that they appear in the post, ...
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The 300-year journey to the covid vaccine, published by jasoncrawford on the LessWrong. This is a linkpost for A covid vaccine has demonstrated 90% efficacy and no significant safety concerns in preliminary data from Phase 3 trials, according to an announcement today from Pfizer and BioNTech SE. The trials aren’t yet complete and the data hasn’t yet been released for independent verification, but this is very good news. (More from STAT News.) Pfizer/BioNTech’s vaccine, like Moderna’s, is based on “mRNA” technology. If approved by the FDA, it will be the first such vaccine to reach that milestone. From a long-term progress perspective, this is a big deal. Immunization technology has existed since the early 1700s (and the folk practices it originated in go back centuries further.) We can see the whole 300-year history of the technology as a quest to achieve immunity with ever-more safety and ever-fewer side effects. More recently, it has also become important to be able to react quickly to new epidemics, such as covid. Here’s how immunization has advanced in stages: Inoculation All immunization is based on the observation that exposure to a disease often grants immunity (temporary if not permanent) to subsequent exposure. Long before we knew anything about antibodies or T-cells, people had noticed this simple correlation. Many people got smallpox in the past, but almost no one got it twice. The goal of immunization technology is to achieve that same immunity, but without having to suffer the disease or to risk death or other side effects. The earliest form of immunization, then, was not a vaccine, but a method in which the patient was given the actual disease itself, in a manner that would cause a mild rather than a severe case of the illness. This was done with smallpox, and the technique was called inoculation or variolation. This worked with smallpox for two reasons. One, infectious material was easy to obtain, from the pustules caused by the disease itself. Second, contracting the disease through a scratch on the skin caused a much more mild form than contracting it more naturally through inhalation. Inoculation saved many people from smallpox. But there were downsides. First, the patient still had to contract the disease, causing mild symptoms. Second, there was still a small risk of a severe case; even the best inoculation methods had about a 0.2% death rate. Third, the patient was still contagious while going through the illness, and anyone who caught the disease naturally from an inoculated patient would get the full, severe version. Inoculation thus risked outbreaks. Vaccination These problems were solved by the next stage: vaccination. It was observed that cowpox infection granted some form of cross-immunity to smallpox. Thus, the inoculation procedure could be performed using cowpox material, rather than smallpox. Cowpox was a milder and non-lethal disease. This reduced the symptoms and the risk of death, and eliminated the risk of smallpox outbreaks as a result of immunization. This new technique, invented by Edward Jenner in 1796, was called vaccination (from vacca, the Latin word for cow). So far, however, the technique only worked for smallpox—not for tuberculosis, malaria, influenza, cholera, or any of the other major diseases that caused something like half of all deaths in that era. Engineered vaccines The next stage would wait almost ninety years. Louis Pasteur, a pioneer of microbiology who along with Robert Koch established the germ theory, was the first to discover how to create vaccines for any disease other than smallpox. Cowpox can be seen as a “natural vaccine” against smallpox: a natural virus that grants smallpox immunity but produces milder side effects. Pasteur’s accomplishment was to create artificial, engineered vaccines. There are essen...