Kristian Rönn | The Darwinian Trap That Explains Our World

[00:00:00] BE: Welcome to a new episode of the Existential Hope Podcast. I'm your host, Beatrice Beatrice, and I co-host this podcast along with Allison Duettmann. Today, we are joined by Kristian Rönn, who is the CEO of the company Normative. Normative is a pioneering company in carbon accounting, where they are utilizing different innovative technologies to measure and reduce carbon emissions for businesses and organizations worldwide.

[00:00:23] BE: Kristian has a very multidisciplinary background in the fields of mathematics, philosophy, computer science, and AI, and I would really consider him a leading voice, not only when it comes to carbon accounting, but also on global catastrophic risk. He's also now written a book called The Darwinian Trap, where he explores different evolutionary pressures that shape our society. He introduces the concept of Darwinian demons, which is something similar to the term of Moloch that I think if you've listened to this podcast before, you're probably familiar with. He says that these are forces that drive individuals and organizations to pursue just too narrow goals. But he also introduces the concept of Darwinian angels, and that's something that we'll also be touching on.

[00:01:03] BE: He does have some very concrete proposals for how we can actually create a brighter future. So join us as we dive into this episode where we speak about evolution, human behavior, and different paths to our desired societal outcomes with Kristian.

[00:01:16] BE: But before we start, I will just remind you to visit existentialhope.com for more resources, a full transcript of the episode, and also to tell you to sign up to our newsletter to stay updated on everything that we do at Existential Hope. So now join me in welcoming Kristian to the Existential Hope podcast. I'm here with Kristian Rönn. Thank you so much, Kristian, for joining. Why don't you just start with telling us a little bit about yourself? Who are you? What are you working on and what got you started?

[00:01:45] Kristian's Philosophical Journey

[00:01:45] KR: Oh, that's such a broad and excellent question. Who am I? I would categorize myself as someone who takes philosophy extremely seriously and as someone who wants to minimize ethical contradiction in my life.

[00:02:05] KR: I think one of the most transformative experiences in my personal journey was when I decided to become a vegetarian at the age of seven. You could see that as the kickstart of my philosophical investigations. We had this petting zoo very close to where we lived in the countryside. I got to chill with cute goats and pigs.

[00:02:33] KR: And then there was this realization all of a sudden that the meat I was eating was actually made from those cute animals. And I was indirectly harming them by eating meat. That created this huge dissonance and contradiction that had to be resolved. I think most people could shove a contradiction like that, push it under the rug and ignore it.

[00:02:58] KR: But to me, it was a very painful experience. And the only way I knew how to proceed was to become a vegetarian. But that resulted in people asking me constantly, "Why are you a vegetarian?" So I almost got forced into philosophy and contemplation because of that reason. Then eventually a bit later when we got access to the internet, I found out about Peter Singer's work and realized, heck, there is a name for this thing. It's utilitarianism.

[00:03:28] KR: I think the next big step is when I found the works of Nick Bostrom and came to the realization that actually, if I care about the well-being of sentient beings, most of those beings will live in the future. Reducing global catastrophic risks and existential risks should be a top priority. I decided that, okay, I want to work together with Nick and I want to work for the Future of Humanity Institute, which I eventually did when I took my bachelor's.

[00:04:05] KR: But I came to realize quite quickly that I like to build things. So for the past 10 years, I've been running a business called Normative. It started as an attempt to create software tools for doing utilitarian calculations, almost creating a utilitarian way of accounting, which was hugely ambitious. But I had to narrow that down to doing carbon accounting. And that's what I've been doing with that business called Normative for the past 10 years.

[00:04:35] KR: So I don't know if that's a good summary, but hopefully that's a good overview of who I am, what I've done, and what motivates me.

[00:04:45] BE: Yeah, that's a great start. I feel like there are a lot of threads to pull at from there.

[00:04:49] Time at the Future of Humanity Institute

[00:04:49] BE: So let's just start with your mention of the Future of Humanity Institute. Do you want to tell us a bit about your time there? Like when were you there? What did you do there? It would be interesting, I think, because it happened this year that the Future of Humanity Institute doesn't exist anymore. And I think you were there in quite the early days, right?

[00:05:09] KR: Yeah, I was there in 2013 and 2014. So I wasn't there for a long period of time compared to a lot of other people.

[00:05:18] KR: What I did there was like academic project management. The way I saw myself was as a multiplier to all of the other brilliant people and colleagues, and that meant that I focused on two projects in particular. The first one was working together with Nick on the release of Superintelligence, doing research, giving feedback on the manuscript, but also some of the practical logistics involved in actually launching a book.

[00:05:54] KR: And then secondly, working together with Sean to establish the Center for the Study of Existential Risk. So some of the grant-making and mapping potential grants, building websites and a whole host of practical issues that are involved with establishing a completely new institute.

[00:06:17] BE: Yeah. And it would be interesting to hear since your time at FHI, a lot has happened in these fields. I guess the field of studying existential risk - what are some interesting culture shifts or any paradigm shifts that have happened since that time?

[00:06:35] The Rise of Artificial Intelligence

[00:06:35] KR: I think the main thing is that artificial intelligence is no longer fiction. It is here right now. I remember back in those days, I made the argument really early that neural networks are universal function approximators. So from first principles, you should be able to create AGI by just scaling the existing paradigm. People did not take that seriously at all whatsoever back then.

[00:06:55] KR: So there is a huge increase in awareness of risk from artificial intelligence, but also existential risks more broadly. Nick mentioned at that point in time this absolutely insane statistic: if you searched on Google Scholar or similar search engines on existential risk versus the dung beetle, there were more peer-reviewed papers published on the dung beetle than global catastrophic and existential risks that threaten all life on planet Earth, which is absolutely insane.

[00:07:51] KR: That says something about our priorities. And if you do the same search now, there are more articles on existential risk than the dung beetle, which is like a very low benchmark, by the way. But it shows that things have improved. And there's a lot happening on the policy side of artificial intelligence.

[00:08:12] KR: I've been involved in some of those projects. So yeah, I think a lot has happened to summarize.

[00:08:20] BE: That's some progress there. I was going to ask you if it had surpassed the dung beetle, but that's good to hear. Great.

[00:08:26] Introducing 'The Darwinian Trap'

[00:08:26] BE: And maybe now is a good time to dive into your book a little bit, because I think that relates a lot to this development in artificial intelligence right now.

[00:08:35] BE: You just wrote a book and it's coming out, I think, in the next month or so, when this episode is published. It's called, let me double-check so I'm not saying it wrong, The Darwinian Trap. So maybe tell us a bit about what this book is about. Why did you decide that you needed to write this book right now?

[00:08:56] Darwinian Demons and Global Challenges

[00:08:56] KR: Yeah, so the reason why I decided to write this book is that from my perspective, all of the big global challenges that humanity is facing, regardless if it is climate change, rogue AI, global pandemics, risk from nuclear war, all of those global catastrophic risks have the same root cause. And that is coordination problems amongst enterprises and nation states, and overall shortsightedness.

[00:09:31] KR: I've been to a lot of climate conferences being the CEO of a climate scale-up for the past 10 years. I'm officially a part of Sweden's delegation to the big UN climate conference. And when you go there, everyone is talking about the importance of cutting emissions.

[00:09:54] KR: All enterprises are talking about the importance of cutting emissions and yet nothing gets done. It's easy to say that this is due to selfishness or people not wanting the best for the world. But I think the root cause here is a type of shortsightedness because everyone is realizing that we're actually going to pay a bunch more money from adapting to climate change.

[00:10:24] KR: This will flood. We will have famines. We will have a whole host of horrible things. Same thing with arming and creating more and more nuclear weapons. We all know that we've been really close to nuclear war that could have potentially killed billions of people. And yet we're doing it. So I wanted to answer that question.

[00:10:46] KR: Like, why are we so shortsighted? And in the book, I'm doing that from an evolutionary lens. So I present this concept that I call a Darwinian demon, and that is essentially a selection pressure for a particular agent to act in a way that is net negative for the world. When you start looking for these Darwinian demons, you see them more or less everywhere.

[00:11:18] KR: You see them in nature, in a sense predators are a cause of Darwinian demons. It is a selection pressure for stealing someone else's nutrients by killing them instead of creating your own. That is simpler. And there is a selection pressure for that. Similarly, there are selection pressures for cancer and viruses and a whole host of things.

[00:11:41] KR: Or even in the workplace, let's say you're a car salesman. You might be choosing to be incredibly ethical in the way you approach it. You don't want to manipulate anyone, but all of a sudden you see your colleagues that might have more deceptive tactics for pushing car sales. And then you're going to lose out in the end of the day if you don't adapt to the same tactics.

[00:12:06] KR: And similarly, if you're an enterprise and you say, okay, we're going to cut emissions and actually take a cut in our profit margins as a result, you're not going to survive against the businesses that ruthlessly optimize for profit maximization. So there are these selection pressures out there and you can analyze all of the root causes for these global coordination problems from a framework of natural selection. So that's what I'm trying to do in the book to summarize a little bit.

[00:12:39] BE: Yeah. Thank you. Yeah. I read the book and I really highly recommend it. It was really well written and I would say a surprisingly fun read for such a heavy topic.

[00:12:49] BE: And yeah, we'll also make sure to share a link with this episode so people can pre-order it. I know it's published September 24th. It's available. Yes. So one of the concepts...

[00:13:01] Moloch vs. Darwinian Demons

[00:13:01] BE: In the book, you mention that you do this from an evolutionary perspective, but there is this recurring concept that I've heard a bit of, at least it's called Moloch. And you also mentioned that in the book, and there's the tragedy of the commons and you have this term Darwinian demons. So it would be interesting to hear you say if these are the same thing. Or are these the same concept? Or like, why are there so many different names? Would it not be better if we just have one name? Isn't that the first coordination issue? Do you have any thoughts on this?

[00:13:34] KR: Oh, that's a great point. There's almost a cluster of adjacent terms like Prisoner's Dilemma, Tragedy of the Commons, Perverse Incentives. We have Goodhart's Law, for instance. We have a bunch of adjacent concepts. And if you take something like the tragedy of the commons, for instance, it's a very specific example in game theory.

Let's imagine that you have a pasture, for instance, or an even simpler example might be that you have some sort of fishery. Different fishermen can choose to follow the quota and be okay. We all fish sustainably so that the fish stock has a chance to replenish itself from one year to another.

[00:14:23] KR: But if you look at it from a game theoretical perspective, it might make sense for one fisherman to defect and fish a little bit more because then they can make a higher profit. If everyone does the same, which they're incentivized to do, then the fishing stock will not replenish and they will all lose in the end.

[00:14:46] KR: So the tragedy of the commons and the prisoner's dilemma are very specific game theoretical examples. But on the other hand, I would say that the concept of Moloch is more or less exactly the same as Darwinian demons. So I could more or less have used that concept. The reason why I didn't is that it is not very descriptive in terms of what is actually going on.

[00:15:12] KR: And I think the easiest way of explaining the concept of Moloch, because it can be a little bit vague, maybe I should actually tell you the origin of Moloch. The origin of the concept of Moloch is a brilliant post from 2014 by Scott Alexander called "Meditations on Moloch."

[00:15:34] KR: That post had a huge influence on me. And Moloch is this God of destruction, more or less, a God of destruction that is always incentivizing people to do bad things. So it's very mythical in its nature. And I wanted something that was more descriptive because I think the best way of explaining what Moloch is is actually in terms of evolutionary game theory and to explain it as a specific type of selection pressure.

[00:16:05] KR: So that's why I decided on the name Darwinian demon instead of Moloch, because I think it's more descriptive of the phenomena and less mythological in a sense. But a friend of mine, Liv Boeree, she recently had a TED talk on Moloch. So she has actually popularized that term quite a bit more. But I wasn't aware of that TED talk happening.

[00:16:29] KR: I actually met her at TED as the talk was happening. And if I would redo it, I feel like it has gained a little bit of notoriety. So maybe I would have called it Moloch myself in the book as well. Probably not, but maybe because it has been more popular recently because of that TED talk and her overall efforts.

[00:16:52] BE: Yeah, I think I would agree with you that the Darwinian demons is more descriptive and has, I feel like, a larger potential to reach a wider audience. It feels more widely accessible than Moloch. Moloch seems like a very in-group term between rationalists and that type of people.

[00:17:11] KR: Exactly. And my intent here was to reach a greater audience. While still giving credit to the origin of the term, I have invented a few concepts in the book that I feel proud about, but 99 percent is me standing on the shoulders of giants. And one inspiration there is Scott Alexander for sure.

[00:17:36] Planetary Boundaries and Resource Arms Race

[00:17:36] BE: Yeah. And in terms of what you write in the book, I thought one thing that was interesting was the nine planetary boundaries. You wrote about the Darwinian demons and their relation to these planetary boundaries. It would be interesting to hear you expand a bit on what these boundaries are and also provide some examples of how Darwinian demons take place in our world right now in relation to these boundaries.

[00:18:09] KR: Yeah. So I talk about different types of Darwinian arms races in the book. I talk about the arms race for resource acquisition. And in that context, I talk about climate and planetary boundaries, but I also talk about the arms race for power. Because with limited resources, an adaptive strategy might be to just optimize for power so you can steal resources from somebody else.

[00:18:40] KR: And I think we can probably cover that later, but to zoom in on the planetary boundaries and resource arms race chapter, the point that I'm making is that there seems to be in environmental circles this myth that humans are the root cause of all the problems that we're facing. Because when you look at these planetary boundaries, there are certain system boundaries within our earth system.

[00:19:11] KR: You need the right balance between oxygen and carbon dioxide, for instance. You have nitrogen that needs to be constantly recycled. Same thing with water. You need a certain level of biodiversity. So we have all of these boundaries that we don't want to cross because if we cross them, we put the entire biosphere out of balance and that can create mass extinctions. In fact, we're facing a huge mass extinction right now.

[00:19:44] KR: Due to us crossing some of these planetary boundaries, like half of all species are likely to go extinct within our lifetime. Half of all species. So this is a mass extinction at an unprecedented scale. And it is easy to say that this is all humanity's fault, which it is if you look at just a very surface-level analysis.

[00:20:13] KR: But if you look deeper, the fact is that all life needs a metabolism. So in order to live, you need to ingest different resources and energy that needs to go somewhere, right? So every time I breathe out carbon dioxide and I inhale oxygen, and then I eat a bunch of nutrients and then I need to poop and piss as well, pardon my French, but that's what all life needs to do.

[00:20:41] KR: So you need to have a balanced system where the oxygen and carbon dioxide are in balance. Same thing, all waste from metabolism needs to be recycled. And in order to maintain a balance, this is not unique for humans. If you look at previous mass extinctions, I would argue that at least five mass extinctions have been caused by other forms of life.

[00:21:14] KR: So essentially bacteria or algae replicating super rapidly, taking over the oceans and affecting the biosphere in similar ways as we are doing right now. And then when you look at it from that perspective, there seems to be this Darwinian drive for fast and aggressive replication, and that's certainly what us humans are engaging in right now.

[00:21:42] KR: In order to win the game of building civilizations, you need to extract resources as quickly as possible. And there are societies that have lived in harmony with nature, but they're more or less extinct right now. Because they were outgunned by more aggressive societies that focused on quick resource exploitation.

[00:22:05] KR: And that is more or less the global norm right now. But similarly, there is a selection pressure for algae and bacteria and a whole host of creatures to expand quite aggressively and exploit resources quite aggressively. But unlike bacteria that might have caused previous mass extinctions, we humans actually have this ability of predicting consequences.

[00:22:31] KR: We're not just victims of our evolutionary programming. We can actually do something about it. So my take is that humans are not at fault. It is the Darwinian demons, the selection pressures that are causing these mass extinctions. And humanity is probably the only hope to save life from itself. And I call this concept the fragility of life hypothesis, that all life is inherently fragile.

[00:23:04] KR: I think that might be the most reasonable resolution to the Fermi paradox. We could perhaps go in that direction a little bit more in the conversation, but I also want to pause and hopefully, yeah, respond to your question adequately enough.

[00:23:20] BE: You certainly did, but yeah, keep going - fragility hypothesis more.

[00:23:26] The Fragility of Life Hypothesis

[00:23:26] KR: Fragility of life hypothesis. A lot of these ideas I've had for a long time. When I finished high school, I was obsessed with Python programming. It felt like magic almost, being able to tell a computer what to do. So I had this idea that I wanted to simulate life in the computer.

[00:23:50] KR: So I built some very early simulations of predator-prey dynamics and sort of simple evolution in terms of how fast predators and prey can run and a few other attributes. And what I found from these simulations is that life almost always died because all of a sudden the predators became too good at catching prey.

[00:24:19] KR: So they killed all of the prey and then they were out of nutrients and then they died themselves. Right before I did some of these early simulations, I read Nick Bostrom's dissertation on anthropic bias, and he's made the point from the anthropic principle that we, regardless of what happens, will always as observers find ourselves in a universe that allows for life to exist.

[00:24:50] KR: But that means that we're intrinsically biased. Whenever we look through the window, we see a really biased part of the world and we don't see all other possible worlds. And what I felt like I did when I ran these computer simulations was creating sort of alternative worlds. And what I saw was that in these alternative worlds, life tends to kill itself, at least in the very simple examples that I built.

[00:25:14] KR: So I got obsessed by this idea that maybe life contains the seed to its own destruction, similar to what I saw in my computer simulation. So I googled around, as you tend to do when you're curious. And then I found the concept of evolutionary suicide and kamikaze mutants, which are two related concepts.

[00:25:46] KR: To me, it's absurd that more research hasn't been performed in relation to that, but it seems to be like an observable fact that sometimes life sees mutations that are beneficial in the short term but in the long term, they lead to extinction, like similar to my predator and prey example. So this has actually been observed in multiple scenarios.

[00:26:13] KR: I think the easiest scenario where it's observed that we can all relate to are super aggressive viruses. The common flu is very successful because it's not aggressive enough to kill us. But there are viruses that are super aggressive that kill us, but they tend to kill themselves. But the interesting thing is that in the short term, being that aggressive can be adaptive because evolution is this trial and error algorithm where it just tries a couple of things and, oh, being super aggressive in terms of spreading as a virus can be super adaptive in the short term.

[00:26:48] KR: But in the long term, it might mean that you actually kill all of the hosts and then you die yourself. So this was just to give you one example. So the fragility of life hypothesis is essentially saying that what if these kamikaze mutations are a lot more common than we think. And the reason why we think that life is this incredibly stable process is because we're biased.

[00:27:16] KR: So in the book, I talk about an example that I call the warehouse universe, just to do a little bit of a thought experiment and exemplify this. So I talk about 10 billion soundproof rooms, and in each soundproof room in this warehouse, there might be living creatures. These living creatures are not aware of any of the other rooms.

[00:27:38] KR: All they know is the room that they live in. And let's say that in one out of six of these rooms, a mutation will occur from one generation to another that causes that creature to be a little bit too aggressive in terms of how they exploit the resources. And at first, such a trait might seem adaptive in this environment, because then you win sort of the competition against everyone else in the room, but it might result in you exploiting more resources than can be replenished.

[00:28:11] KR: And then you die. So you essentially die if you get that gene. And let's say one out of six is getting that gene, then it would only take 126 generations for there to only be one room left. If you're born in generation 126, you will by 100 percent probability find yourself inside of that room. And when you look at that room, you will look back and say, "Hey, like we have survived for 126 generations.

[00:28:39] KR: This looks super stable. This kamikaze mutation just seems to be a myth. And our empirical observation confirms that it is a myth." But if you would have, as an observer, access to all of the other rooms, then you would see the truth that it is not a myth. So then the question is what evidence can we find right now because we don't see other worlds, we don't see life on other worlds.

[00:29:07] KR: So is there any evidence for this fragility of life hypothesis to be true? And I think one piece of evidence is if we look through our geological record and in our geological record, we see that there have been plenty of instances of kamikaze mutations that have caused mass extinctions. I mentioned the algae and bacteria and their metabolism before, but I think also the Fermi paradox, the fact that we don't see the universe teeming with life should point in that direction as well.

[00:29:40] KR: So instead of one great filter, it might be that evolution through natural selection is just like one elongated gradient filter. So the fact that we don't see life on other planets should point as evidence in that direction. And I think if we would, for instance, observe on Mars or Europa or somewhere else that, you know, oh, actually very simple life existed at some point, which I actually think is quite likely.

[00:30:11] KR: I think biogenesis is probably just chemical reactions. And if you have the right chemical soup, it will be like a chemical inevitability that sort of simple life will occur.

[00:30:22] Fragility of Life Hypothesis

[00:30:22] KR: So if we see that on other planets, that would also be some evidence in the direction of the fragility of life hypothesis being true.

[00:30:31] BE: Yeah. So we have a survivor bias basically, most likely. Yeah. This is very interesting. I'd love to hear more about it, we probably don't have time to dive into it right now though. Oh, bud. You were saying there's not like that much research that has been done on this. 

[00:30:48] KR: Yeah, that's correct.

[00:30:49] Evolutionary Suicide and Kamikaze Mutants

[00:30:49] KR: If you Google evolutionary suicide and kamikaze mutants, there's like only five researchers or so, and just like a handful of papers, somewhere between 10 and 20, if I'm not mistaken. And I do think it's because we're all falling for this bias. We all tend to implicitly believe that life is this self regulating system that is super resilient, because that's what our empirical data tells us, but it doesn't paint the complete picture.

[00:31:23] KR: And I think if you look under the hood and start to do agent based modeling on predator prey dynamics, like virus host dynamics and parasites and all of these other phenomena and take that seriously, you will find that life is not particularly stable. But also if you look more closely, I suspect that a lot of traditional extinctions might be a result of [ ]  where, and there's examples, for instance, of, yeah, like in order to survive as a prey, you might want to optimize how agile and fast you run. And then there's this arms race where all of a sudden you need to spend all of your energy hiding from predators. And that means that you can spend less energy on reproduction.

[00:32:11] KR: And then it actually leads to a suicide over the longterm. So that's one example. Another example is some types of fish where actually becoming smaller helps in terms of avoiding predators. So that is selected for the gene for being small, but for some fish, it also makes you less fertile. So you're slowly killing yourself as well.

[00:32:37] KR: And some have argued that the big cod collapse in Canada was what was caused by that. And I think I bring that up in the book as well. So I would love to, there to be more research around like how stable is life actually, and are we diluting ourselves to some extent? 

[00:32:57] AI Development and Resource Arms Race

[00:32:57] BE: And I know that one risk that you're particularly worried about right now is in relation to AI development, so why is the scenario of uncontrolled AI evolution frightening to you? What are the potential dangers with that, that you can see? 

[00:33:15] KR: Yeah. So I want to take a step back and link that to the resource arms race that I just talked about. So we live on a planet with limited resources, right?

[00:33:28] KR: And there seems to be, it seems to be adaptive for some agents in the short term to evolve, to aggressively expand and exploit resources as quickly as possible. And that's what we're doing right now. The most successful countries are the ones that sort of crack the formula of converting fossil fuel and minerals and all of these things into economies.

[00:33:55] KR: And if you do that, you win, but in a world of limited resources. One adaptive strategy might be to steal resources from someone else. And that's what we have seen with all wars and conflicts throughout history. And that creates a selection pressure for developing more and more capable weapons over time.

[00:34:17] KR: In fact, you could see the birth of most empires from the backbone of new military innovations. The Mongols, for instance, perfected horseback archery so they could just mow over everyone else and create an empire. But over time you develop these more and more dangerous weapons because you need them in order to protect your own resources and take everyone else's resources.

[00:34:46] KR: So another example is British gunboats. It enabled them to build the biggest empire on earth. And they also built Gatling guns and on a whole host of things. And in world war two, we saw chemical weapons for the first time. That was like hundreds of thousands of times, more deadly than the arrows that the Mongols used.

[00:35:10] KR: And world war two, we saw nuclear weapons. And obviously. The U. S. has nuclear weapons. The Soviets need them as well, and everyone else needs nuclear weapons. So you see this proliferation of nuclear weapons. 

[00:35:25] Intelligence Arms Race and Alignment Problem

[00:35:25] KR: And I think the next part of that arms race is the intelligence arms race because there is a certain limit to how much power we can project due to limitations in intelligence that we intrinsically have as humans.

[00:35:42] KR: And typically speaking are. Ability to innovate is only as great as the biggest genius in whatever country, right? Everything from Maxwell to Einstein, et cetera, that, that enables a whole host of new sorts of technological innovations because of fundamental research. Intelligence, indeed, if you have enough.

[00:36:04] KR: Resources to fuel it seems to be a very adaptive trait, and that's why we're seeing this intelligence arms race right now, both between the U S and China, but also between a bunch of AI companies that want to win that arms race. And I think one thing that I talk about in the book that I think is important to point out is that a lot of people.

[00:36:30] KR: are rightfully obsessed with the so-called alignment problem. How can we build artificial intelligence that is aligned with human values? But what I claim in the book is that we will never be able to solve that unless we solve societal alignment first. And what I mean by that is, let's say that we actually build, figure out how to build aligned agents, which is a big if.

[00:37:00] KR: Because there's a bunch of impossibility, mathematical impossibility theorems in terms of proving by 100 percent probability that a particular AI system is aligned. Mathematically, it's somewhat impossible, at least in the edge case, but let's succeed in building an agent that is aligned by 99.999 percent probability. Is that the agent that will be deployed across the world? I don't think so, because of Darwinian demons again, as an enterprise. Will you deploy an agent that is maximizing some sort of human welfare, or will you deploy an agent that maximizes the profit for a particular enterprise? As a matter of fact, it would arguably be illegal for you to deploy an agent that doesn't maximize profit.

[00:37:52] KR: Like you would probably be voted out by the board. And if you're not voted out by the board, the shareholders certainly would vote you out. And let's say you're Israel and you want to defend yourself and you deploy AI technologies in the field. Do you want AI? That is optimizing your chances of winning the war rather than respecting all sort of human values and optimizing for that, probably.

[00:38:18] KR: And there is even research on this in terms of autonomous cars. When you ask people on the street, do you think autonomous vehicles Should take all lives into consideration and think about the least amount of harm it can do. Of course they say, yes, it sounds incredibly reasonable, but then you're asked the question of, okay, would you be okay with your self driving car, like killing you in order to save the life of let's say five pedestrians?

[00:38:53] KR: And people will say, of course not. Of course, I'm not going to buy that car. So there seems to be selection pressures for non aligned agents. And I think over time, there will even be selection pressures. For less and less responsible deployment of AI in the global economy, we have such things as tax havens, for instance, like it is adaptive for a country, some smaller state to say, Hey, we have super low taxes, please come here.

[00:39:26] KR: And that is a big problem for a lot of bigger states. And right now we might have a mentality, of course, we should always maintain a human in the loop when all of these autonomous agents are being deployed eventually. In the military, that is already the policy, right? Always maintain a human in the loop.

[00:39:45] KR: But I think over time, it might be adaptive for smaller states to say, please move all of your AI companies over here. We have zero regulation and you can build as automated systems as you want to, without any safeguards and humans in the loop, et cetera. And that might create a system where humans are marginalized over time, because AI will just evolve to outcompete us.

[00:40:15] KR: And countries that will remove safeguards will evolutionary outcompete those that don't so we really need to find and solve global coordination problems first in, in order to have aligned AI being deployed safely. 

[00:40:34] BE: Yeah I think we're definitely most people by now are on board with your like problem profile of Darwinian demons.

[00:40:43] BE: What, so what are the solutions? Are you suggesting in the book that accounting could be a potential solution to many of our challenges? And I know that's something you also have experience with, in practicality. So what does that mean? Like how can accounting help and how can you do that in a way that preserves, the integrity of everyone involved and also without that becoming like Goodhearts law hacked.

[00:41:08] KR: Oh, totally. Yeah. That's a great question. 

[00:41:11] Reputational Markets and Global Coordination

[00:41:11] KR: So first of all, maybe just briefly touch upon some other mechanisms that could solve this. One thing that I explore is more centralized mechanisms. Can we reform the UN? Because if we had the functional UN system that would penalize countries that are defecting in the global game of power and resources, then we wouldn't have this problem.

[00:41:37] KR: When you look more closely, there are very low incentives for sovereign countries to give up their sovereignty to a higher entity like the UN. The UN was a huge exception that was created due to two world wars where we felt like we didn't have another option. So to reform the UN will Take a lot of time and we'll be a really slow process.

[00:42:03] KR: And I think we probably need a bunch of warning shots first, where we have close calls, that will be a bit of a wake up call for that to happen. So I think the only way is to sign a more decentralized way of solving these global coordination problems. So one of the ideas that I present in the book is something that I call a Reputational market, and the idea is essentially what if we start to account for the things that we actually intrinsically value, for instance, being more nature or different people will have different intrinsic values, right?

[00:42:46] KR: But it seems reasonable that it would be in our interest to, to try to, to the best of our ability to maximize the things that we intrinsically value. But in order to do that, we would need to devise some sort of accounting system to measure what we intrinsically value. So that's the part where I think accounting can play an important role.

[00:43:10] KR: But obviously it's not the only solution. You also need to have the right incentive structures in place for all players to follow this type of accounting regime. So that's where I talk a lot about the distributed nature of the world economy and the global network of value chains and supply chains.

[00:43:33] KR: Because when you look at any dangerous technology like artificial intelligence or engineered pandemics. You have incredibly complex supply chains, everything from minerals to refinement of minerals, to creating subcomponents, to creating lithography machines, to creating the actual three nanometer chips that you install in data centers, et cetera, et cetera.

[00:43:58] KR: So there is, in fact, for even the most trivial things, like making a chicken sandwich, for instance, like there is. Probably thousands of people indirectly involved. I bring up this example of a guy on YouTube that wanted to create a chicken sandwich from scratch. It sounds easy, you bring the chicken from the fridge and then, take up the sandwich, take some butter on it or whatever, and then you're done.

[00:44:24] KR: But that's not making it from scratch, making it from scratch means growing the grain, milling it, making it into flour, making it into bread, feeding the chicken. So it took him like six months and 1, 500 US dollars just to create a single chicken sandwich. So what if we built a system where in fact you are incentivized to, or yeah, let's put the, take a step back.

[00:44:51] KR: What if you build a system where you could, for instance, like a responsible AI company would not get access to the latest computer chips? Or a responsible, irresponsible computer ship manufacturing wouldn't get access to core minerals or a responsive, irresponsible bio company wouldn't get access to the latest sort of lab equipment.

[00:45:17] KR: So that could potentially be like a mechanism designed to use these sorts of bottlenecks in global supply chains to govern these dangerous technologies. So then the question arises, like who, how do you know who is irresponsible and responsible? So what I propose in the book is something that I called a reputational market, which is based on the idea of prediction markets.

[00:45:44] KR: So essentially the way it works is that all participants in the market. Vote on what are the intrinsic values that I hold dear. Is it well being? Is it nature? Is it happiness or life satisfaction or whatever it might be? So that's step number one. But then in the marketplace, you can post predictions on how different entities affect these intrinsic values.

[00:46:10] KR: So you could, for instance, post a prediction, okay, will irresponsible AI Inc cause a major terrorism attack in 2025? So then you post that prediction. And that beautiful thing about prediction markets is that if they're well functioning, You're actually incentivized to make truthful predictions, like betting on different football games and the outcome of those games.

[00:46:36] KR: If you're making a correct prediction, then you will make money from doing so. So hopefully a reputational market would create incentives for everyone to collectively think about who is responsible and who is not responsible. This could be applied to climate change as well. Big Oil Inc, what will their carbon emissions be in 2025, for instance.

[00:46:59] KR: So you could post all of these predictions very organically, and then the market could react to different events, such as Irresponsible AI Inc releasing their new responsibility report or releasing their new model, or maybe some investigative journalism, for instance. So the idea is that if you aggregate all of those predictions and weight them by how much we intrinsically value the outcome of these predictions, you could assign dynamically a score between let's say zero and a hundred to, to every single entity on, on, on the planet.

[00:47:37] KR: And if you have a. So that's the general idea. And I know that we're running out of time, but I want to just complete that idea because the next question is going to be like, okay, fine. Every entity has a score in terms of how much they're expected to affect what we intrinsically value. Why should these entities care in the first place?

[00:48:00] KR: So one idea that I present in relation to that is: What if you get a lower score if you don't sanction someone with a low score in your supply chain? So one example would be, let's say Irresponsible AI Inc gets a horrible score because they just released the model and that model was jailbroken within the minute with disastrous consequences.

[00:48:28] KR: So the market would react to that. They would get a really low score, but as a consequence, The market, reputational market would automatically downgrade the NVIDIA if they give, if they sold and provided all of the compute chips to irresponsible AIing and NVIDIA, they might care about their score being downgraded because If their score is downgraded, they're afraid that they're not gonna get ships from TSMC, the biggest sort of ship manufacturer in the world, in Taiwan.

[00:49:03] KR: And TSMC, they might be a little bit worried because they won't get the lithography machines from ASLM, for instance. So you have this situation where if you don't sanction someone that you're doing business with, you will get a lower score yourself. And this sort of almost seems a little bit off. It seems like an infinite Russian nesting doll where someone cares about something, so he cares about something.

[00:49:29] KR: So there needs to be some sort of grounding where someone with enough power intrinsically care. And my take is that perhaps pension funds could be that grounding because they are already redefining their fiduciary duties in relation to climate change. The reason why most companies have net zero targets now that actually started with pension funds are responsible for 50 percent of all investment flows.

[00:49:56] KR: So if they say, Hey, we're actually gonna care about this TSMC and ALSM, and we're gonna care about these reputational ratings in general, then that would create a sense of caring. And I actually have seen this, like in practice with my business normative, there is this concept of value chain based emissions, where My carbon emissions, if I have a net zero target as a retailer in order to, for me to reach net zero, I need to ask my suppliers to go to net zero and they need to ask their suppliers to go to net zero.

[00:50:28] KR: And we have already seen these network effects in global supply chains with regards to net zero. And a lot of that is fueled by pension funds. So yeah, that was a bit of a long monologue, but hopefully it's a, it answered your original question. 

[00:50:45] BE: Yeah. I think there's definitely more to dive into and ask about these types of markets, but it's very interesting.

[00:50:52] BE: And I guess we just need to refer people to, to read the book because yeah, we do have not that much time left and more questions that I would like to ask you. 

[00:51:02] Hope for Humanity and Future Generations

[00:51:02] BE: This is the Existential Hope Podcast. So we've very much talked about these challenges that we've, we're facing now, but in terms of the future, are you positive?

[00:51:12] BE: Are you feeling optimistic about the future? Do you think that we'll be able to navigate this? What's your take on whether we'll be able to have a positive future in the next thousands of years? 

[00:51:25] KR: So that's, that is a tough question, right? I spent most of the talk so far presenting doom and gloom, why we have the fragility of life hypothesis and life might tend towards its own self destruction.

[00:51:39] KR: So where's the hope? I think hope is in humanity. Like we are in this incredibly exciting and unique place in history. Where we can decide and say, we're not going to let our societies be governed by this sort of default optimization algorithm of natural selection that just works through trial and error.

[00:52:05] KR: We can actually. Say, Hey, we're going to try and make predictions in the future and act accordingly. So I think humanity brings me a great deal of hope. And I think there's so much momentum around, for instance, the problem of climate change, technologies being deployed that brings me a great deal of hope as well.

[00:52:29] KR: But I also think if you look at. praised as artificial intelligence and how it can be governed. I think it's totally solvable because of the distributed nature of global supply chains. Like the fact of the matter is that if you don't have access to the supply chain, even such a simple thing as a chicken sandwich is fundamentally impossible.

[00:52:52] KR: We're all dependent. On one another, we're all interdependent. If we build a system of accounting and reputational ledgers through indirect reciprocity, I think we can leverage the fact that we're all interconnected into this almost super. Organism that we call the world economy. Sometimes it's called Gaia if you think about the entire ecosystem.

[00:53:20] KR: So that brings me a great deal of hope that actually not a single agent can screw us all over unless we agree to it. So let's not agree to it. Let's build transparent supply chains and hopefully we can. Generations from now, like our next generations are gonna look back at us with a great deal of amusement and be like, Holy shit, that was crazy.

[00:53:47] KR: We did not know how to. Stably cooperate and collaborate. And that was just a weird glitch. Hopefully it will also look back at us with a profound sense of respect and gratitude. We have survived for the past 4 billion years. by a lot of luck, if the fragility of life hypothesis is true. But for the first time, we might build a system where it's not luck anymore.

[00:54:16] KR: It's stable cooperation. And I think the time is now to solve all of this. And I think I like to think that in the future future generations will laugh at our early struggles to cooperate. 

[00:54:32] BE: Yeah. It makes me think of this Petrarca quote that, yeah, he said it like, I think a thousand years ago or something like that.

[00:54:40] BE: He was born to live in confusing times and that he was hoping for better for future generations. And I guess hopefully we're still in somewhat confusing times and that. they'll be able to look back on us and yeah be amazed that we made it through with such much luck. 

[00:54:56] Eucatastrophe and Reputational Markets

[00:54:56] BE: So one thing that we always ask also in this podcast is for an example of a eucatastrophe and by eucatastrophe, the opposite of a catastrophe.

[00:55:06] BE: So something that happens, an event where after it has happened, the world is much better off. There's much more value in the world. And it comes from this paper by Toby Ord and Owen Cotton Barret that actually came out of the Future of Humanity Institute a few years ago. And yeah, so if you had to like, think of the best eucatastrophe that you could think of, and also ideally, Think of it as something that, would be inspiring for people to see, because we're going to make a little art piece out of it, an AI generated art piece.

[00:55:37] BE: So what do you think would be something that would be an inspiring art piece of an event that made the world much better off? 

[00:55:46] KR: I think a successful rollout of the first reputational markets. Obviously I'm a little bit. biased, but I think if we roll this out seriously, like all you need to try and build this is just the small team with a few programmers to build the first prototype. And I think that will probably not be it. But if you could get the Norwegian sovereign wealth fund or some major players to say, Hey, we're actually going to pay attention to all of these predictions here. I think that would be a catalyst to make everyone else in the global supply chains fall in line and actually care about the overall reputation.

[00:56:36] KR: And I think it would almost lead to a collective intelligence, like nobody, not a single entity is in control of the, what is the definition of intrinsic values. We're collectively in control of that. Nobody's in control of who has a good or bad reputation. We're all collectively in control of that, depending on what predictions.

[00:57:00] KR: We may, so I think that would be a really pivotal moment overall, and maybe even an example of, okay, all of the sudden all companies are starting to pay attention and put significant resources into trying to do the right thing. So they will get a better reputation on the reputational markets as well.

[00:57:25] KR: Maybe the first clear sanction of someone saying, no, you have a bad reputation and you will not tarnish my reputation. So I am going to sanction you. And I think one sort of mental model that I have is the first piece of code encoding the blockchain or Bitcoin blockchain. It's just a simple piece of code, but it had the mechanism designed to become a global thing that millions, if not hundreds of millions of people all of a sudden care about, and they didn't have to do anything, any marketing.

[00:58:04] KR: They didn't have to do anything. Like it was built into the mechanism design of the protocol for this to spread globally. So maybe it would be pushing that first piece of code that create the incentives for everyone to care about the long term future and the long term instantiation of what we intrinsically value on a reputational market.

[00:58:29] BE: I think that's great. It's very concrete. I like it. It's also a new term, which is fun. And yeah, I would also just like to direct any listeners to the existentialhope. com website, because when we did the world building challenge, the winning team, that one, when they built out the world in 2045, had like a prediction market there as well.

[00:58:50] BE: It's called Cities of Orare and it's a type of, yeah, prediction market, but they use it for governance and it's very cool. We have to round off now. I'll ask you a few quick ones just to round off. 

[00:59:00] Final Thoughts and Recommendations

[00:59:00] BE: Do you have any recommendations other than your own book, of course, for listeners to read or watch or listen to? And it can be anything, like something that has made a difference for you when you come across it. 

[00:59:13] KR: So I would recommend people to also read The Meditation of Moloch by Scott Alexander. I would encourage people to subscribe to The Wind Podcast by Liv Boeree. I would encourage people to see talks and articles published by my friend Daniel Schmachtenberger. So those are some recommendations that I have. 

[00:59:40] BE:  Great. Last question. What is the best advice that you ever received? 

[00:59:47] KR: That is such a hard question to answer. I've received a lot of good advice, but I think the best advice that I've received is to have a fairly small ego. Like whenever you want to build something.

[01:00:08] KR: If you let your ego stand in the way, you might not bring the right people on board on the journey. I think realizing when I, when am I actually the bottleneck? That's typically the advice I give to people who are founders and creating their own businesses. So this advice might be a little bit biased towards founders, but I see that as a positive thing.

[01:00:31] KR: Common failure mode, founders not realizing their own limitations and think they're the best at everything. So they don't hire the best talent and realize when is it time to step aside. I think one other piece of advice that I think is really important as well is to focus on things that you can actually influence.

[01:00:52] KR: I think it's very easy to worry about the bunch of stuff happening in the world. It's. That is incredibly worrisome, but I think for your own sort of mental health, knowing the boundaries of what you can influence, and then try to push those boundaries constantly as well, to just be able to survive in a world with a lot of doom and gloom and not let that drain your energy and instead focus on the things that you can actually influence and expand that circle of influence so much. 

[01:01:25] BE: I couldn't agree more. And I guess that's also what the Existential Hope Project is about to a large extent, just like thinking about what we can do here and how we can make it as good as possible. With those words, thank you so much, Kristian, for coming on the podcast. 

[01:01:39] KR: Thanks a lot for having me, Beatrice.

[01:01:43] BE: Thank you so much for listening to this episode of the Existential Hope Podcast. Don't forget to subscribe to our podcast for more episodes like this one. You can also stay connected with us by subscribing to our sub stack and visit existentialhope.com If you want to learn more about our upcoming project events and explore additional resources on the existential opportunities and the risks of the world's most impactful technologies, I would recommend going to our existential hope library.

[01:02:08] BE: Thank you again for listening and we'll see you next time on the existential podcast.