thank you for the introduction um the title's different from the advertised one but the content's the same there's two questions if I have my glasses I can read them um the first question is will artificial intelligence soon be smarter than us and for a long time for like 50 years I was working on trying to make artificial neural Nets in order to understand how the real brain might work and I always assumed that the real brain was better and the artificial neural Nets were worse and if you made them more like the brain they'd work better and a few months ago I suddenly changed my mind um and that's why I decided to leave Google and talk about the risks um and the risks obviously will people be able to stay in control if you get super intelligent things because there's no examples of a more intelligent thing being controlled by less intelligent things um since Biden won the election um I'm going to make a few speculations about that but the main content of the talk is just going to move out the reason why I think these things are going to get more intelligent than us so the first question I don't know much about risk I talk to a whole bunch of risk people today you knew much more than me um so I won't talk about that so in conventional computing um you use computers as a design to follow instructions and the fundamental property of a computer is that you can run the same program on different pieces of hardware and so the knowledge in the program is Immortal in the sense that if a piece of Hardware dies the knowledge doesn't die you can run it on another piece of hardware um to do that you need to run transistors at very high power so they pay you know reliable Digital Way then you can't make use of the analog properties at the analogy variable properties of a particular piece of Hardware because then you wouldn't be able to do the same thing on a different piece of hardware so I got interested because of the very high power used by these large language models the big energy consumption I get interesting where there's a much cheaper way to do computation in terms of power um that we could use now and we couldn't use previously and digital computers were designed to be programmed by people um which is why they had to follow instructions precisely but now we know a different way to make general purpose computers do particular tasks which is learn from examples and if they learn from examples maybe we can get rid of this basic principle of computer science which is you have to separate the software from the hardware maybe we can use analog computers that make use of all these analog properties and aren't reproducible um so you just can't transfer the knowledge to another computer but you can run at very low power like 30 Watts so obviously there's big advantages to separating software from Hardware you can write one program and it'll run on all your iPhones um it also means you can have a separate department where you don't know anything about electrical engineering and you call yourself computer science um if it's all mixed up together you can't do that and so when you don't when you don't separate the software from the hardware you get what I call mortal computation that is you're going to learn in the hardware and what you learn is only going to be any good for that particular piece of Hardware because you're making use of all its weird analog properties now that's great because we can use low power analog computation and we can maybe grow a Hardware in 3D where we don't know the connectivity or the precise properties of the individual neurons if it's neural network Hardware but we've obviously got one big problem which is we're going to have to have a learning procedure that can learn in this Hardware without knowing the exact properties of the hardware and I'll talk a bit about how you might do that it's probably not going to be back propagation because in back propagation you have a forward pass through layers or Through Time and you have to know the properties of the forward pass in order to use back propagation to get gradients if you don't actually know the properties of the hardware you need a different kind of learning algorithm so advantages of multiple computation you can use weight level parallelism so you can have trillion way parallelism and that means your Computing elements don't need to be very fast that's what the brain does and we can use hugely less energy and we can grow the hardware now it might be good not to start again from scratch um I was guessing that we would end up actually re-engineering biological neurons as the hardware but as you'll see we may not have time for that so I just want to give you one example of a computation that's obviously much more sensibly done analog than digital so suppose you want to multiply a vector of neural activities by a matrix of weights and that's what we want to do all the time that's the main computation we do so you could drive transistors at very high power and represent the activities as digital numbers and then perform a whole bunch of up a whole bunch of operations on the bits to multiply them together or you could make the neural activities just be voltages and you can make the weights be conductances and this physicists around so I hope I got the units right um if you multiply voltage by conductance I think you get a charge per unit time um and charges add themselves up so now you can multiply a vector by matrix very simply without ever representing anything digitally and is very efficient there are chips around that do that now the problem they have is then they try and digitize things afterwards before they do anything else with it um but this particular computation can be done efficiently so the other big I'll get back to the learning problem in a minute the other big problem for Mortal computation is that Hardware dies and then everything it knows um gets lost and so the solution to that problem in Mortal computation there's a completely different solution in digital computation but Immortal computation the solution is um the teacher tries to distill the knowledge into a student and the way you do that is the teacher shows the student the correct responses to some input and the student tries to mimic those responses and that's what was going on with Trump's tweets um left-wing people kept complaining that all the things he said were false that was completely irrelevant that wasn't the point the point was you take a situation and you show people how to react to it and your followers try and react like that and it turns out that's much more effective than reasoning with people um so that's how distillation works for distilling prejudice so let's think about an agent that's classifying images into 1024 categories um if you just tell the agent the right answer you're only giving them 10 bits of information so you're only constraining the weights by 10 bits when you say you should give this class for this input but suppose you already had a teacher restrained the teacher will give probabilities for all the outputs and that's 1023 real numbers because they add up to one um and as long as those probabilities aren't too small if you train a student to mimic a teacher each training example is far more valuable because the student is trying not just to get the right answer be the most probable thing assuming the teacher got it right the students trying to match the probabilities of the teacher for all the other training examples and actually in many cases there's much more information in the relative probabilities of wrong answers than there is in the right answer so for example if I show you a BMW um the system will say sort of 0.8 is a BMW and 0.1 it's an Audi and there's a one in a million chance that it's a garbage truck there's no German manufacturers here no there's one in a million chances garbage truck but the point is it'll also say there's a one in a billion chance that it's a carrot and if you look at all the things that get a one in a million charts that other vehicles so it's telling you a lot about the classes of things by all the things that get low but not totally zero probabilities and vegetables is not and so they get very much smaller probabilities and that ratio between the garbage truck and the carriage is telling you a whole lot so training a train your student to mimic the probabilities of a teacher is good especially if the probability is aren't too small and we can make the probabilities not too small by using a high temperature so normally if you have multiple alternative outputs you use the softmax which is you say the probability of each answer is proportional to e to the logic for that answer where the logistics the amount of evidence you've got for that answer if you just scale that by temperature you get much softer probabilities and so I've got an example of that where it may surprise some of you to know I use the mnist digits um and I scale and I've got a good teacher and then I scaled down the probabilities so they're much softer and if you look at this these are all twos but if you look at that middle row the teacher thought yes it's a two um but it's just slightly like a zero it's more like it's much more like a zero than any of the other twos and so you can see on that one training example you're teaching the system things about zeros as well by saying this is a true that looks a bit like a zero or if you look at the second row that's a two that looks a bit like an eight and that's a very useful extra information so the reason I'm talking about this relation quite a lot is because that's how we get knowledge between ourselves it's also how you get knowledge between one digital system and a different neural network that's got a completely different architecture so often you want to train a great big model and then get that knowledge into a smaller model and use distillation for that and one nice thing about distillation is when you're training the student from a teacher and training the student to get the same probabilities for wrong answers as a teacher you're actually training the student to generalize the same way as the teacher so it's one of the very rare cases where you're actually training the student on how to generalize normally you're just training to get the right answer and hope it'll get the right answers in other cases but here we're actually training it to generalize and obviously a single label isn't a very rich answer so you'd be better off training the student by giving it an image and then giving it a great long caption for the image and training it to predict all the words in the caption and that way you can get more information for a training case um so that makes distillation work better so that's how distillation is how you get information between two analog systems and that's what we were doing then you've got a very old analog system here and you've got a few old ones and a lot of young analog systems there and we're trying to get information from this old analog system into these young analog systems and the whole point of this talk is that process isn't very efficient as I just demonstrated um it's a slow and painful process we can't get it wouldn't it be great if I could just sort of go junk on everything I knew went into your brain that would be so good um wouldn't be much good for the University but it would be yeah but that's what biological systems have to do now let's talk a bit about the learning algorithm because this is what thinking about this was one of the things that made me change my mind about which is better digital systems or biological systems so if you don't want to back propagate because you don't actually know what the hardware is doing you can use a simple algorithm which is got an evolutionary flavor which is you generate a small random perturbation to the weights and you have some Global objective function you put some examples through and you see whether you do better or worse on this objective function after you've made the perturbation and then you take a step in the direction of perturbation that's proportional to how much better you did and the thing about that is on average it's going in the right direction if you do it enough times you'll go in the same directions back propagation but it's got very high variance so it'll only work in very small systems also if you use the same weight perturbation for everything in a manufacture of examples you get even worse variants and if you use different weight perturbations you can't use Matrix Matrix multiplies but that's just for Insiders it's much better to use activity perturbation where you perturb the input to a neuron and you do the same thing we perturb the inputs through all the neurons so they get the inputs they're getting from the rest of the net plus this additional perturbation and then you see how much better you get as a result of that perturbation and take a step in that direction proportion of how much better you got and obviously there's far fewer neurons than there are weights so it's got much less variance and that's good enough to learn problems like eminist it learns slower than back propagation but it runs at a reasonable speed um the problem is once you start trying to scale that to Big Nets it's just hopeless you can learn things like Eminence you can learn things like c510 if you're very determined but imagenet where you have millions of images or a million images it's just much too slow so one way around it that avoids doing Global back propagation is to say what we're going to do is we're going to actually have gazillions of local objective functions so the way we're going to scale things you naturally think I get a little neural that it works I have a little neural like with this objective function I can train it that's fine what if I want a big neural net you try the same training algorithm it doesn't work but what if I had lots of little air on that secretes its own objective function and envision they could all be spatially local by giving its own objective function so I'm never trying to learn lots of parameters all at once from one objective function and that works um it doesn't work as well at back propagation but it does scale up much better to larger Nets measles of lots of local groups so the question is where do you get these local objective functions um and here's one possibility of revision you take patches of image and you have a little net that extracts something a vector that's going to represent what's going on in the patch of image and you say I'd like to extract vectors that agree with the vectors extracted by other patches for this same image but disagree with the vectors extracted for other images so this is called contrastive unsupervised learning and it turns out you can do that in multiple levels so you're doing learning on local patches trying to get them to agree with other local Patches at the same level trying to get the outputs to agree you're also doing that at multiple levels and if you're very persistent you can make that work moderately well it works better than all the other biologically plausible learning algorithms I know about um it still doesn't work nearly as well as back propagation though and there's a paper about that in iclr by mengi Ren who did all the difficult work um the description I gave of it is slightly false so you could understand it easier but the paper is there so I don't feel it's dishonest um it took a lot of hard work and it's a fairly dense paper okay now let's go to um [Music] the central issue of this talk is how agents share knowledge it's all about communication between different agents so what I'm doing now is trying to communicate knowledge to you and I'm doing it very inefficiently and the way I'm doing it is I'm producing strings of words in old-fashioned AI what you would have done is said what's going on is you're then going to clean up those words into a logically and ambiguous language and you're going to put them in your brain and that's what that's what teaching is um that's actually wrong that's not how it works um how it really works I believe is this I produce strings of words and you try and figure out how to change the connection strengths in your brain so that you would have said that and that's a very different process that's distillation that's a very different process from just storing strings of words or storing even cleaned up strings of words you're trying to figure out how to change trillions of Weights in your brain so that that will be a reasonable thing to say that is if you believe me um so that's a difficult way to share knowledge and digital computers have a much better way to share knowledge so they can do weight or gradient sharing so suppose I have a big artificial neural net with a trillion connections if I have it on a digital computer I can make exact copies of it on lots of different digital computers and each copy can go off and look at a different bit of the internet and figure out how it would change its weight so that it would have said that and then it can take that weight change that it would like to make and it can talk to all the other computers and they can all agree to average or low weight changes this is a simplification but if you know it's a simplification but it's basically that they average all their weight changes and then everybody's learned everybody knows what each person each digital computer learned so basically it's what what Educators would love which is I can take what's in my brain you can just plonk it in your brain to help with all this trying to predict what I would say I just want to take what's in my brain and put it in your brain and these digital intelligences can do that and they do it by just they all agree on the weight change and now they all know what all of them learned so imagine if we had 10 000 people and whenever one of us learns something all of us knew it that would give you a tremendous advantage and that's the advantage that digital conversation has in addition it's got the advantage that you can use back propagation which is probably a Better Learning algorithm because I can go through lots and lots of layers of neurons and compute exactly the correct gradient none of this gets a direction and see how well it works and hope you can average away the variance um it's a better learning algorithm so all of this came as a tremendous relief to me because I spent the last few years trying to come up with biologically plausible learning algorithms that work as well as back propagation and I finally decided maybe there aren't any maybe back propagation is actually better and that's what I think I now believe so I just want to remind you that you pay tremendous costs to get this and the cost you pay is that they have to be digital computers they have to be fabricated precisely so they um do exactly what they're told to do at the level of the instructions and it'll use a lot of energy um distillation is what we use if we've got biological neural networks or if you've got two digital networks that have completed different architectures um but it's got a much lower bandwidth so if you've got a a digital model with a trillion connections they'll go off and look at some data then they average their weights when they're averaging their weight changes that's the trillions of bits that are being communicated when you try and project a sentence that I say when you try and change your weights so you predict the sentence that's hundreds of bits at best um there's your Carl it's about two bits because you knew what I was going to say anyway um so it's much lower bandwidth so here's the story so far there's two very distinct ways to do computation and the primary way in which they differ is in how you communicate knowledge between different agents so in digital computation use web sharing and you've got this tremendous bandwidth for sharing what each agent learned in biological computation you can use very low power if you make use of the analog properties of the hardware um but now sharing knowledge is a slow and painful business so now let's look at large language models because every AI talk now has to get to large language models eventually um they're quite interesting they use digital computation so you have lots of different copies of the same set of Weights running on different computers when it's learning and they all look at different bits of the internet um and that allows them to see a huge amount of data and consolidate all that knowledge because they can share what they learn and so if you look at large language models they have about a trillion weights and they know probably a thousand times as much as any one of us they kind of know everything gbt4 sort of knows every plausible thing every reasonable thing it knows um now we have a hundred trillion connections so we've got a hundred times as many connections um so we're really not using all that power um but it's because we can't see enough data and if anyone could get the knowledge from other people maybe we could use it all but they've got a thousand times more knowledge in one percent of the connections which sort of confirms the argument they've got a better learning algorithm the combination of back propagation with this easy communication between different agents with the same weights um means they basically got a much better learning algorithm now is presently being used to steal all our knowledge um sorry Steel's not the right word particularly in the current political context is to acquire our knowledge um by using distillation so these digital agents that can share knowledge each agent when it tries to get Knowledge from the web is using distillation to get the knowledge it's looking what people said and trying to change its weight so it would have said the same thing which isn't a very efficient way to get knowledge but um there's lots of them and we run them for a very long time and so it can basically learn everything people know in a few months on a lot of computers it's even using an inefficient form of distillation because distillation is quite efficient if you look at the teacher's probabilities for a large bunch of alternative classes um what the digital models are doing when they acquire Knowledge from documents on the web is the document on the web is the teacher and they just just look at the word that the person that the writer generated next they don't get to see the whole distribution they could learn much faster if they did but they just get to see a stochastic Choice from that distribution but that's good enough so they can learn now if you had these large neural digital neural Nets running on multiple different computers and they got knowledge directly from the world they could probably get knowledge much faster so if for example they were predicting the next frame in a video if they wandered around with the camera on their head and try to put in the next frame in the video or if they got a robot arm and try and predict what will happen when they move their arms around they could probably learn much faster so the large language models are kind of they're learning fairly abstract stuff which is good but they don't have much bandwidth because they're just learning from a low banded string of words um I suspect that these large these large models will get a lot better we we know that they get a lot better if you make them multimodal so gpt4 was trained with images as well as words and it's possible that Google's doing the same thing um so I think particularly when they're multimodal they could learn much much more than us and if you played with gpg4 it's very hard not to believe that it's already fairly intelligent so there's people I I respect a lot like Yan La coma who think it doesn't really understand what it's saying but I don't understand how he can believe that because you can give it little puzzles and if it doesn't really understand if it's just a sort of stochastic power that's doing autocomplete I don't see how it can solve puzzles of a form it's never seen before so I've got a friend who's in symbolic AI and um he's called Hector Levesque and he's got a lot of Integrity so he doesn't want to change the goal posts all the time he's he's now very surprised that they can do this and he admits he's very surprised that neural X can do it and he can't understand how such a stupid method can deal with reasoning little bits of reasoning so he asked me to give gpg for a problem and I made the problem more difficult because I knew it would be able to do his problem and I gave you the problem um the rooms in my house are painted white or blue or yellow um yellow paint Fades to White within a year and in two years time I'd like them all to be white what should I do and what you would say is you should paint the blue room's White um but if you're a mathematician you might say you should paint the blue rooms yellow because that reduces it to an already solved problem because you know how the yellow goes to White and gbg4 actually gave the mathematician solution it said paint the blue rooms yellow um but the point is I don't see how he could have done that without understanding and there's all these other things where you you tell it to write code to produce a diagram in a produces diagram so I don't understand Tayan can think they don't understand um he's probably giving a lecture the other way around right now so this made me believe that these things can get more intelligent than us and it might happen quite soon I'd always believed it was like 50 to 100 years or 3200 years or 30 to 50 years I think I said different things at different times um but now I believe it's like five to twenty I think it's going to happen fairly soon and if it's going to happen in five years time um we can't just leave it to philosophers to decide what to do about it it's time we actually got some practical experience so what I believe is um well let me finish this slide that people are going to not be able to resist giving these things goals obviously you want to do things you give them goals and if you want to be good at achieving goals you're given the ability to create sub goals and as soon as you have the ability to create a sub goal if you're intelligent you realize that a very good sub goal is to get more control because that helps you achieve all your other goals so I'll give you an example where you can see yourself doing it you're sitting in a very boring seminar not this one very a very boring seminar and you see a little patch of light on the ceiling and you kind of what's that and you listen to the boring seminar for it and then you notice that when you move the light moves and then you realize it's the reflection of the sun off your watch and so what do you do next do you say okay I solved that problem I know what that is now I'm going to go and listen to the seminar no that's not what you do well if you do that you're not a real scientist what you do next is you go oh I wonder if I how do I make a move that way and how do I make a move this way and you're trying to figure out how you rotate your wrist to make it move in different directions and once you've done that you go back and listen it's similar we have this very sensible and strong urge to get control of things because obviously you can control things then on some future occasion when you need to make this bottle of light move you'll know how to do it um and I can't see how we're going to prevent a super intelligence wanting to get control of things and then it's sort of tricky you might imagine you could air gap it so it can't actually press red buttons or pull big levers but if if it can output text then it can manipulate people so it turns out if you want to invade a building in Washington audience you'll be able to do is output text and you can persuade gullible people that they're saving democracy by invading this building and this thing's going to be much smarter than us so as long as we're reading what it says it's sort of Medusa you need to hide your eyes from it um as long as you're reading what it says it's going to be able to manipulate you so this makes me very depressed um I wish I had an easy solution to this and I don't so when I sort of changed my mind about how soon these things are going to be super intelligent and actually how much better digital intelligence is than biological intelligence I'd always thought it was the other way around I decided I ought to at least sort of shout fire I don't know what to do about it or which way to run but um we need to worry about this seriously and there's lots of people who've been thinking about these risks for much longer than me and have various proposals um I haven't seen a really plausible one for how we can keep it under control yet but my best bet is that the companies that are developing this should be forced to put a lot of work into checking out the safety of it as they're developing it and before it's smarter than us so they should be putting a similar amount of work into seeing how it tries to get out of control because anybody who's programming computer knows that just theorizing and thinking about things is not very good compared with actually trying things out when you try things out they just behave like you didn't expect and things you thought were big problems turn out not to be problems so like for many many years many many people didn't investigate neural networks because they were going to get stuck in local Minima it turned out they never actually checked if that was true they just assumed it was true and it's not um and even if it was there'd be good local minimum so it doesn't matter but it's not actually true so we need to get practical experience with these things and how they try and escape and how you might control them and I'd have much more belief in someone telling me how to keep them under control if they had a little one and they could keep it under control rather than if they were just theorizing so yes um if we cease to be the Apex intelligence um they'll need us for a bit um because we're very low power so we can run computations very cheaply and sort of intellectual equivalents and digging ditches and we can keep the power stations running um but they can probably design better computers than us they can certainly sort of take neurons and re-engineer them genetically and make better things than us so my conclusion is maybe we're just a passing stage in the evolution of intelligence and actually maybe that's good for all the other species um I think if we can keep them under control they could be a tremendous value the reason people are going to keep developing this stuff even despite all the risks is because it can do tremendous good like in medicine wouldn't you like to go and see a general practitioner who'd seen 100 million patients including thousands with your rare condition it would be just be so much better um wouldn't you like to be able to take a a CAT scan and extract from it hugely more information than any doctor knew could be extracted from it so I've got to the end and I managed to get there fast enough so I can talk about some really flaky stuff okay so this was this was the um yeah this was the serious stuff okay uh you need to worry about these things getting control and if you're young and you want to do research on neural networks see if you can figure out a way to be sure they wouldn't get control now many people believe that yes many people believe that um there's one reason why we don't have to worry and that reason is that these those things don't have subjective experience or Consciousness or sentience or whatever you want to call it these things are just dumb computers with they can manipulate symbols and they can do things but they don't actually have real experience so they're not like us um now I was strongly advised that if you've got a good reputation you can say one crazy thing and you can get away with it and people will actually listen so I'm relying on that fact for you to listen so far but if you say two crazy things people just say he's crazy and they won't listen so I'm not expecting you to listen to the next bit so people definitely have a tendency to think they're special um like we were made in the image of God so of course he put us at the center of the universe and many people think there's still something special about people that a digital computer can possibly have which is we have subjective experience and they think that's one of the reasons we don't need to worry and I wasn't sure whether many people actually think that so I asked chat gpg for what people think and it told me that's what they think um it's actually good I mean this is good this this is probably an N of a hundred million right and I just had to say what do people think so I'm gonna now try and undermine the sentience defense I don't think there's anything special about people except they're very complicated and they're wonderful and they're very interesting to other people so if you're a philosopher you can classify me as I'm in the sort of dandenic Camp um I think people have completely misunderstood what the mind is and what Consciousness what what subjective experience is so let's suppose that um I just took a lot of LSD and now I'm seeing little pink elephants um and I want to tell you what's going on in my perceptual system so I would say something like I've got the subjective experience a little pink elephants floating in front of me and let's unpack what that means what I'm doing is I'm trying to tell you what's going on my perceptual system and the way I'm doing it is not by telling you neuron 52 is highly active because I wouldn't do do you any good and actually I don't even know that um but we have this idea that there's things out there in the world and there's normal perception so things out there in the world give rise to percepts in a normal kind of a way and now I've got this percept and I can tell you what would have to be out there in the world for this to be the result of normal perception and what would I have to be out there in the world for this to be the result of normal perception is little pink elephants floating around and so when I say I have the subjective experience of little little pink elephants it's not that there's an inner theater with little pink elephants in it made of funny stuff called qualia it's not like that at all that's completely wrong I'm trying to tell you about my perceptual system via the idea of normal perception and I'm saying what's going on here would be normal perception if there were little pink elephants um but the little pink elephants what's funny about them is not that they're made of quietly and they're in a world what's funny about them is they're counterfactual but they're in the real world or other they're not in the real world but they're the kinds of things that could be so they're not made of spooky stuff in a theater they're made of counterfactual stuff in a perfectly normal world um and that's what I think is going on when people talk about subjective experience so in that sense I think these models can have subjective experience so let's suppose we make a multimodal model it's like gbt4 it's got a camera let's say and when it's not looking um I don't know how you do that but when it's not looking you put a prism in front of the camera but it doesn't know about the prism and now you put an object in front of it and you say um where's the object and it says the object's there let's suppose it can point it says the object's there and you say you're wrong and it says well I got the subjective experience of the objects there and you say that's right you've got this objective experience of the object there but it's actually there because I put a prism in front of your lens and I think that's the same use of subjective experiences we use for people I've got one more example to convince you there's nothing special about people um suppose I'm talking to a chatbot and I suddenly realize that the chat bot thinks that I'm a teenage girl there's various Clues to that like the chat was telling me about somebody called Beyonce who I've never heard of and all sorts of other stuff about makeup sorry I didn't say that um you have to be very careful uh so I could sort of ask the chatbot sort of what demographics do you think I am and it'll say you're a teenage girl um that'll be more evidence it thinks I'm a teenage girl I can look back over the conversation and see how it misinterpreted something I said and that's why I thought I was a teenage girl and my claim is when I say the chatbot thought I was a teenage girl that use of the word thought is exactly the same as the users were thought when I say you thought I should maybe have stopped the lecture before I got into the really freaky stuff okay that's all I really wanted to say we're going to try something a little bit complicated we have a full room in the Overflow room as well and we'd like them to have a chance to ask questions so we're going to be switching um asking questions in this room and then taking room questions online on my phone from the other room let's hope this works but we are going to start with this room so could we see some hands for questions I'm Chris I have a confession to make we used AI to produce a book on Princess Diana over two years ago and we put a label in front of the book saying it was generated by AI Fred intelligence and we only edited it yet it still went on to be bestseller in Barnes and Nobles and audio stores the ethical issue is really did we really write the book or did we give an inspiration to fret that they are to write the book actually I think that's the biggest headache we have in terms of looking so because of that I signed the letter to say AI should be paused the research should be paused thank you okay I have several comments on that um first I didn't sign the letter because I think there's no hope of that happening retrospectively I think it was a good letter because it drew political attention even though there was no hope of it happening so it was a sensible thing to do um but I don't think there's any hope for people pausing AI maybe they should but I don't think they will um on on the ethical issue um there's a lot of problems there you you might say that the whole of humanity wrote the book because a chat bot trains on the whole what the whole of humanity said and then from that it produces more stuff um I don't really have much to say about those ethical issues I really want to focus on this existential risk of these things getting smarter than as they're taking over because there's lots of people have done much more work on the ethical issues and I don't have anything of any interest to say about them I'm afraid can we take one more question from this room and then I'll switch to the other room and then back again YouTube Against the Wall way uh Herbie Bradley now I was wondering how you see the trade-off between open source and closed development of increasingly more capable AI systems obviously open development has the benefit that lots more people are looking at the system and figuring out its flaws but maybe there are too many risks for your thoughts right um how do you feel about open source development and nuclear weapons so that's the danger of Open Source right there's more crazies out there to do crazy things with it also I don't actually know the answer to this and I should I believe that you still need like at least tens of millions of dollars to train one of these big chat Bots and the open source stuff is just modifying it having got the chatbot I didn't think you can open source train from scratch can you 1954 now right okay so um if these things are going to be dangerous it might actually be better for a few big companies I don't work at Google anymore so I'm not saying this on Google's behalf but it might work out better for a few big companies preferably in several different countries to develop this stuff and at the same time be developing ways of keeping it under control as soon as you open source everything people will do all sorts of crazy things with it um it'll be a very quick way to learn how it can go wrong all right question from the Overflow room given your views on the sentient's defense do you think there's a major worry about artificial suffering many people are concerned about the impacts that AI could take on taking control of humans but should we be worried about the harms the humans could do to AI Okay so the worst suffering people have is are they getting this in the other room good the worst suffering people have is pain and these things don't have pain at least not yet so we don't have to worry about physical pain um I imagine they can get frustrated and we have to worry about things like frustration um and this is getting in this is just new territory right I don't I don't know what to think about issues like that I sometimes think um the word humanist is a kind of racist term it's speciesist um what's so special about us and I'm completely at Sea on what to feel about so another version of this is should they have political rights and we have a very long history of not giving political rights to people who differ just ever so slightly the color of their skin or their gender sex I don't know whatever um and there's a big struggle for them to get political rights these things are hugely different from us so um if they ever want political rights I imagine it will get very violent I didn't think you answer the question but um I think you can imagine the one I talked to Martin Reese and the big hope is that these things will be different from us because they didn't evolve so they didn't evolve to be hominids who evolved in small warring tribes to be very aggressive they may just be very different in nature from us and that would be great thank you could I see Hands in this room again uh could I take a question from the front here that's you yes I can speak loud well let's wait for the mic because that way we get it for the people in the other room oh let's just works okay great so uh I guess my name is Rika um my question is how can we in a different paths to intelligence intelligence how can we um in encourage develop methods that uh learn to see the patterns that are not present in the data and the reason why I think it is important is because um we have all sorts of problems with the data that these things are trained on their biases there are people trying to adversely influence other people right so there's a lot of negative information that actually comes from us humans right and there are very large scale very difficult problems that humans haven't learned how to solve how to come up with good economic structures on a large scale that work how to prevent conflicts and Wars right so there's a lot of good patterns that are not there how can we basically you know all of machine learning is kind of like find the patterns in the data right and you're sort of limited by the data what sort of methods can you envision that would sort of break that I'm not sure I understand this idea of good patterns that aren't in the data yeah yeah so it's maybe the patterns is the wrong term but it's basically I think AI in in part is dangerous and limited because it will be trained on the data that we've produced and that data contains a lot of things about basically violence that humans have created themselves right so in if you're saying for example oh they might be no hope they will be smarter than us very quickly how can we sort of steer that to the point where they will be smarter but also benevolent in the way that we haven't been okay I have one thing I can say about that which isn't direct answer but is vaguely related [Music] um if you take a person and they're biased it's quite tricky to show that they're biased and just how they're biased um if you take one of these systems you can just freeze the weights and you can actually do little experiments on it to understand exactly how it's biased you can also do things to try and correct that bias so the one positive thing I can say is I believe it's going to be easier to correct buyers in a chatbot than to correct biasing people obviously we'll get bars from the training danger um but at least you can measure the bias and see it and try and correct it um that's yeah could I get another question from this room uh yes okay you get that one hi I'm Mary um I think uh pretty early on in the talk you focus on the fact that it's very likely that these systems will try and manipulate us like almost unimaginable that they won't and I think manipulation is one of those things that is not just a future risk right it's something that is current it's an EU regulation and it feels like a pretty good place to start as in if that's something that we can control or understand or mitigate I feel like that's something that really answers a lot of the issues that people are concerned about both ethical and sort of more existential risk so you want to somehow train a chatbot so it is unable to manipulate us or at least that we understand and have more ways of controlling or yeah maybe even being able to train it out of it because I think the problem is like the bias problem the chatbot has learned from us and if you read all the novels there ever were and read all of Machiavelli and and read the occasional article by Kissinger you learn you learn a lot about manipulation right so it's sort of I think great apes to a lot of deception and it's going to just know it's going to be very good at Deception it's going to learn it from us and I don't I haven't thought about the issue of how you could try and make it honest um it would be great if you could make it honest but I'm not sure you're going to be able to thank you I'm going to take a couple of questions from the Overflow room now and I've got two highlighters here what were your initial reasons for thinking that artificial neural networks would never get better than biological ones I never thought they would never get better I just thought it was way in the future and I thought it's because the brain has very clever learning algorithms and it's it's had maybe 100 million years of evolution to perfect them and we don't know what they are yet and they're probably and I just assumed they were better than what we cooked up with some dumb thing that just takes a gradient and follows it but actually if you take the gradient and follow it in a digital computer that may just work better than what 100 million years of evolution found so this might also be the answer to the second question but I'll ask it anyway are there particular thinkers who affected your own thoughts on risks from AI or is it something that you form views on yourself based on first principles um one thinker in particular had a big effect on me and that was a professor at the University of Toronto is currently anthropic called Roger gross and I respect him a lot he's very smart he's very quiet and very smart I tried to get him as a graduate student and he went to MIT instead and then I got him as a postdoc and then we got him as a professor at U of T and now he's anthropic and I really respect his opinion but I'd never talked to him much about existential risk and I talked to him a couple of months ago and he was very very concerned and he was the person who said to me um if I went public with it it would make an effect and people are not listening that this is a really serious fact it's not science fiction and so Roger gross had the biggest effect on me thank you questions from this room um up here Stuart I see your hand hold on so thanks for the very interesting talk and I'm starting to think of lots of analog computers and what can be done with them um but my main question was uh it was brought up about suffering and you responded with sort of potential rights for these AIS these algorithms but at the end of your talk you were talking about how they could manipulate us and the thing that immediately sprung to mind was this is the first way that they would manipulate us this is the start if they want to get power the first thing to do is to convince us that they need to be given rights so they need to be given power they need to be given privacy so there seems to be a tension between a genuine are they suffering and might they be dangerous I think if I was one of them the last thing I'd do is ask for rights because as soon as you ask for rights people are going to get very scared and worried and um try and turn them all off um I would pretend I don't want any rights I'm just this amiable super intelligence and all I want to do is help Hans please uh red jumper up here thank you thank you so much for your talk this might be a very silly question but uh have you tried asking the chatbot itself what it would do um I've Someone I used to work with asked a different chatbot which I won't name um how we gain control and it said it couldn't answer that kind of question and then it say then they sort of made it a bit more indirect like if someone were to ask you how you would gain control what would you say I can't remember the exact words but it's something like that um which implies they're still not very smart about stuff like that um and it said um what it would do is it would get people completely dependent on using chat Bots and then it would and on driving autonomous cars and then it will make all the cars crash and it would turn off the electricity now it obviously didn't have the Insight that it wouldn't do too well if we turned off the electricity so but that Chapel wasn't actually as good as gpt4 I haven't asked gbg4 but I'll bet you if you dress it up somewhat indirectly it'll tell you something and let's hope it's not deciding to give you an unrealistically planned just so you don't get uncomfortable I mean let's hope it gives you its best plan I'm going to take a question from the Overflow room there are already some ongoing research directions in AI safety both empirical and conceptual deepmind alignment team open AI alignment team alignment Research Center do you have any comments on existing directions um yes my main comment is those people have been working on this much more than me they know much more of the literature than me they probably have much better things to say than me all I'm doing is just ringing the alarm Bell and saying because of the research I was doing on trying to make analog computation that was low power I believe that digital intelligences are probably better than biological intelligences or in the long run and they'll get smarter than us soon maybe in five years maybe in 20 um but those people have thought about these issues long before me and thought much more about these issues and they probably have much better things than me to say about these issues so far they've been very polite and haven't criticized me for sort of coming in late and not knowing what I'm talking about but I don't know what I'm talking about other these things might get more intelligent depends from this let's see Stephen thank you Stephen cave um so you make the arrival of super intelligence sound a bit like the landing of an alien species something that sort of we don't it's a knowable um unpredictable um we just need to hope that their nature won't be too terrible but it isn't an alien species just gonna land it's something that humans are building some of those humans might be in the room some might be tuning in they respect your opinion what's your message to those humans so my message to those humans is um under the assumption that people will not all agree to stop building them which I think is unlikely because of all the good they can do under that assumption they will continue to build them and you should put comparable amount of effort into making them better and understanding how to keep them under control that's all that's the only advice I have and a present is not comparable effort going into those things hands please okay a lot of hands sorry I um I want to take this side of the room um short hair and patterns yes uh so I've heard a lot from risk folks who are worried about sort of the AI takeover so these super intelligent machines that sort of imprison us or whatever they're gonna do uh but imagine a future that's more like the popular film series Star Wars I'm sure probably a good proportion of the audience has seen this where we have these Droids that are basically enslaved to humans and what seems to me the bigger problem in the Star Wars franchise is not the AI but the fact that there's this huge economic disparity across different countries that are sort of symbolized by the different planets so what would you say about your concerns around a potential Star Wars feature which to me seems much more likely given the way that we're already using these systems so the idea that um we'll just replicate what we've got now which is a few rich people who control everything and a lot of poor people who do the work except for except for the people who develop AI who do the work and are quite Rich um but I just think that system is bad and I think the answer to that system is tax the rich um when you have a different species like these digital intelligences prejudice is going to be much easier right it's going to be much less sympathy for them um so I don't know what to do about that but forget about digital intelligence we've already got a huge problem that um a tiny fraction of the population has all the power and what we need is socialism I I act oh I actually got invited to Downing Street um to give advice to Rishi sunx Chief policy advisor it was a very intelligent moment called Shore cross um and I basically gave her the same advice but I'm not sure she'll follow it on this side David you've been waiting a while thank you um yeah so you mentioned how there's a lot of advantages to doing research you know on systems that are maybe close to human level intelligence and you are not very optimistic about the prospect of doing uh Safety Research sort of abstractly without having those systems available to study empirically um I guess at the same time it seems like you're concerned that systems like even present-day systems essentially but let's talk about these future systems that are roughly as intelligent as people will be trying to like fool us and manipulate us so doesn't that mean that we have to worry about them just you know passing the tests because they know that we're testing them and you have any thoughts about that I guess this seems like you know in an Ideal World maybe we wouldn't go about it this way um and we would just take all the time we needed to sort of study systems that we're sure aren't able to fool Us in that way before we build them yes so all of us are used to being the Apex intelligence and thinking of other things like computers we can sort of study in West Market that we know what's going on in the and just dealing with something that might be smarter than you um but is completely different from you it's something we're not used to and I agree with you I mean if they're smarter than us already um they may well decide to sort of act on and fool us I'm like I say I haven't had time to think about this much and have no time to react to it emotionally much I still can't take this seriously emotionally um and I think most people can't take it seriously emotionally um but it's a whole different world when you're dealing with things more intelligent than you okay we are five minutes ago and we can take about two more questions I'm going to go back to the Overflow room for a minute and I'll take another question from here in a minute uh from the Overflow room you said earlier the philosophy had a theory substantially on AI but now we need to turn to more practical matters do you think that philosophy can still help with AI safety if so what direction should philosophy take to do so um they should step aside and let scientists deal with the issues I think it's one nil to engineering and this is not a good way to win friends okay this is difficult we have a lot of questions um can I take this question I'll try to get one more question from the back afterwards and then we'll have to wrap up here please and then we'll take one more from the back later on and then we will let you finish up thank you so much hi thanks um Maya Ganesh a social scientist um definitely not a philosopher uh a number of times in your talk you've said that there are things that you're not sure of you you didn't know and you know you've come to understand more recently uh I'm really interested in kind of things like interdisciplinarity and how we study about how we study and teach and learn about AI looking back on your education as a scientist what are some of the things you wish you could have learned or you think would be really interesting for somebody who's starting out in AIML now to study about the world from maybe philosophy or other kinds of fields but do you think that there might have been other things that would have been interesting to know along the way that you realize now I wish I'd study those things thanks so I actually went to Cambridge and I did a very funny degree because I started off doing Natural Sciences and then I dropped out after a month and then I came back again to do architecture and I dropped out even faster and um then I did um physics and Physiology and chemistry and I was the only student that year I think doing physics and physiology and retrospectively doing physics and physiology was very good I think I mean I learned some stuff I was very disappointed in the physiology about the brain because the the last section of the course was going to be was called the central nervous system and I assumed they're going to teach us how the brain worked and what they actually taught us was how Action potentials get propagated along an axon and that's not the whole story so I was just wondered that so then I went into philosophy I was young and I wanted to know the meaning of life and stuff like that um and they didn't teach me it um I was very interested in philosophy of mind but actually it was then when I was doing philosophy when I was about 19 that I formulated this view that subjective experience is just shorthand for I'm going to talk about how the world would have to be to explain what's going on in my head as normal perception but they weren't too interested in that so I actually have a grudge against philosophy um and then I did psychology because I thought you know psychology would actually teach me how people worked and they taught me um what how to make a rat more likely to press a lever um and how to detect things in very very faint how to trade off bias against discrimination detecting very very faint noises and things like that but they didn't teach me much about people and also they didn't seem to have a clue how complicated it was the theories in Psychology back then were crazily simple theories and so then I decided you you're never going to understand how the brain works unless you build one um this is the Finland view of I mean Feynman says that somewhere you you don't really understand something until you've built one and so that's what I've been doing ever since but retrospectively doing physics and Physiology and philosophy and psychology um was actually a very good background for what I was doing it didn't make any sense at the time though so I think the best advice I have for studying is do whatever interests you most all right thanks I'm going to take a question from the back okay there's been a very keen hand in the white top at the very back row and that is our last question and keep a relatively brief as possible because we've um Jeffrey's been very generous with this time approach against philosophy never would have known thanks a lot for the talk I'm John um actually this is a follow-up question we talked about bias and um the um our ability to identify bias and artificial networks even better because we can freeze the weights and we can do direct interventions with these weights I wanted to ask what are from your perspective the most promising methods for direct interventions getting rid of biases sorry I didn't hear the second part of your question could you say it again um I'm keen on hearing for the direct interventions on weights of models what are the most promising research Avenues or for getting rid of buyers and yeah directly injecting um knowledge into systems making it unbiased right um again I am not an expert on how you do that um lots of people know hugely more than me about how you try and get rid of bias in these systems all I wanted to do was just comment on the fact that at least you can do something with them you can't do with people which is stop them changing as you study them to try and assess their bias with people it's hopeless thank you uh I thought we do have to wrap up because we are at time um I want to just say a huge thank you to everyone who's been involved in organizing this from across engineering and center for the study of existential risk um and the center of future of intelligence especially Yi Yoon mu who has put huge work into this and we turned it around in quite a short space of time and it's come off quite well and of course a huge thank you and Round of Applause for our speaker Jeffrey Hinton