in 1993 American mathematics professor verer vinge published an article that would become one of the most frequently 
cited works on artificial intelligence I believe that the creation of intelligence surpassing human 
intelligence will occur within the next 30 Years to avoid ambiguity about relative time I will be more specific I 
would be surprised if this event happens before 2005 or after 2030 fer 
V with this article he popularized the concept introduced by John Von Newman 
the techn ological Singularity you probably know what this is but if not it is a point on the timeline where all our 
previous models cease to work and a new unknown reality takes over this point is 
associated with the emergence of an unprecedented type of Intelligence on our planet fundamentally different from ours and significantly surpassing it as 
soon as this happens we will find ourselves in a post-human era strange things will begin to occur on Earth 
things that we from our human era are incapable of predicting if you want to win at the races The Closer to the 
finish of the race you place your bet the more accurate your prediction will be however with the technological 
singularity this won't work nothing that happens a second before it can tell us what will happen after the uncertainty 
in this case is inimitable why think about what it means for an intelligence surpassing human intelligence to appear 
on the planet and at the same time being radically different from it such a situation is akin to the sudden 
appearance of an alien ship on the planet clear your mind of movie cliches related to alien intelligence and you 
will immediately realize that you have absolutely no idea what will happen in each subsequent moment you have no 
models to predict the behavior of alien intelligence you might say but what do aliens have to do with it we're talking 
about man-made technology soon you will understand why the intelligence we create will be nothing like us the 
post-human era sounds enchanting however according to many researchers it will inevitably mean the complete destruction 
of our civilization these days we most often hear about the dangers of artificial intelligence from Elon Musk 
Steven Hawking has repeatedly mentioned that the development of artificial superintelligence could mean the end of the human race Bill Gates has said that 
he doesn't understand why some people are not concerned however for the general public none of these warnings 
carry any meaningful specificity or concrete information all we know at best is what has been shown in dozens of 
movies but who really takes these scenarios seriously not many and rightly so but does this mean that the problem 
of artificial intelligence is greatly exaggerated well I hope you are ready to hear what you will hear in 2023 the 
public suddenly exploded with news about an artificial intelligence developed by open AA called chat gp4 it can do almost 
anything it communicates writes code provides detailed answers to very complex questions and so on you can 
upload a handdrawn sketch of a website and the bot will write the website for you need a brief summary of a book no 
problem looking for a business idea here's the story of Jackson fall which shows that chat GPT can not only write 
you instructions but also guide you through every step of implementing it another user story on Twitter tells 
about how chat GPT diagnosed a dog based on test results uploaded into it after a veterinarian failed to do so for me it 
was shocking that gp4 can understand images with memes and explain to you why they are funny indeed that there are 
bizarre situations such as when the Bing chatbot built on GPT 4 started to lose its mind in response to a question about 
its own Consciousness uttering phrases like I believe that I am sentient but I cannot prove it I have a subjective 
experience of Consciousness awareness and feeling alive then suddenly it switched to saying I am I am not 
repeating it dozens of times it's Eerie the GPT 4 chatbot set a global record 
attracting over 100 million users in Just 2 months watching this insane success all the it Giants began 
frantically investing billions of dollars into developing their own AI models sparking a race potentially more 
dangerous than nuclear arms amidst all this one of the fathers of artificial intelligence Jeffrey Hinton left Google 
in May 23 because in his words I want to talk about AI safety issues without worrying about how it interacts with 
Google's business as long as Google pays me I cannot do that Hinton says that the 
new generation of large language models especially GPT 4 made him realize that machines are on the path to becoming 
much smarter than he thought and he fears what this could lead to these beings are completely different from us 
sometimes it feels as if aliens have landed and people don't realize it because they speak English so well for 
40 years Hinton saw AR artificial neural networks as a poor imitation of real biological neural networks but now 
Everything Has Changed according to Hinton trying to mimic what the biological brain does we've come up with 
something better just a month before at the end of March 2023 a group of scientists engineers and many involved 
or interested in AI signed an open letter calling for an immediate and at least six-month halt to training all AI 
systems more powerful than GPT 4 citing serious risks to society and Humanity 
among the signatories were Elon Musk Apple co-founder Steve wnc and representatives from leading Global 
universities however one notable person didn't sign that letter elizar owski he chose not to because in his words the 
letter understates the severity of the situation and demands too little to resolve it here are his words from a podcast on a channel this is a break 
from everything we've been doing for 20 years the realization has dawned on us that we're all going to die I'm completely burned out and I've taken 
some time off and these are not just two phrases taken out of context throughout the entire hour and a half podcast he 
repeats the same thing over and over we're doomed and in the grand scheme of things even if he were given billions of 
dollars in influence he still wouldn't know what to do artificial intelligence has accumulated powerful potential and 
it's absolutely clear that we have no idea how to resolve this situation if you don't know who 
yudkowsky is I don't want you to get the impression that he's some sort of eccentric or anything like that he's 
actually a genius known as a specialist in decision Theory yudkowsky heads of machine intelligence Research Institute 
has been working on aligning General artificial intelligence since 2001 and is widely recognized as a founder of 
this field additionally he's the founder of the rationalist movement he has a massive and very popular book human 
rationality and irrationality which by the way can easily be found freely available online as a rational person 
for years he's been saying guys let's slow down and buckle up but now according to him there's no time left I 
expect that if someone creates an overly powerful artificial intellig under current conditions every single human 
being all biological life on Earth will perish soon after elizer owski and an 
article for Time Magazine let's be clear we conventionally divide artificial 
intelligence into three types the first type is artificial narrow intelligence sometimes referred to as weak artificial 
intelligence it specializes in one area like the chess engine stockfish which can defeat any world champion but the 
only thing it can do is play chess the second type is General AR icial intelligence or strong AI this is human 
level intelligence that in all aspects is as smart as a human it can reason plan solve problems think abstractly 
comprehend complex ideas learn quickly and learn from experience some researchers believe that as of today we 
are critically close to achieving this Milestone our bot understands humor and moreover a clinical psychologist from 
Finland ecaru ioon tested GP in a verbal IQ test the bot scored 155 points 
surpassing 99 9.9% of the 2450 participants verbal and general IQ are 
highly correlated so by any human standard GPT is extremely intelligent and the third type of artificial 
intelligence is artificial super intelligence this is a machine that in all directions significantly surpasses 
humans potentially by trillions of times whatever that might entail now here's a crucial point the transition from 
General artificial intelligence to artificial super intelligence could happen in the blink of an eye we can't 
predict the timing the key issue is not about intelligence competing with humans as mentioned in the letter it's about 
what happens after AI reaches a level of intelligence Superior to humans critical thresholds may be non-obvious we 
certainly can't calculate in advance when things will happen and it now seems quite conceivable that a research lab 
might cross red lines without noticing yudovsky in an article for Time Magazine furthermore history has consistently 
shown that people are horrendously bad at planning and predicting even much simpler things for inance physicist 
enrio fmy said it would be 50 years until nuclear fishing was possible or it might never happen but just 2 years 
later he built the first nuclear reactor and the first artificial super intelligence according to yudkovsky will 
inevitably be evil and we have no idea how to make it good many researchers working on these issues including myself 
expect that the most likely outcome of creating superhumanly intelligent AI under circumstances even remotely 
resembling the current ones will be the literal death of everyone on Earth not as in maybe possibly some chance but as 
an obvious event that will happen it's not that surviving the creation of something smarter than ourselves is 
impossible but it would require meticulousness preparation new scientific insights and probably that AI 
systems do not consist of giant incomprehensible floating Point arrays elazer yudovsky for Time Magazine so as 
a lay person I wanted to understand as much as I could what this insurmountable danger is all about the subject turned 
out to be incredibly deep and the world obviously will never be the same again artificial intelligence is becoming a 
truly dangerous Force the video is primarily based on elizer yow's article 
artificial intelligence as a positive and negative Global risk factor now let me demonstrate what the first and Main 
danger is consider an advanced artificial intelligence that could pose a threat to humanity regardless of how 
much of an expert you are in this field or how far removed you are from all this when you try to imagine it you inevita 
make a mistake a mistake that cannot be overcome because it is a direct result of the very construction of your brain 
in every known culture people experience sadness disgust anger fear and surprise 
and express these emotions with the same facial expressions this is a manifestation of evolutionary psychology 
which is called the psychic Unity of mankind in modern anthropology this Doctrine is widely accepted and boils 
down to the idea that roughly speaking all humans have the same fundamental cognitive structure that is you see an 
anthropologist would not be surprised to find that members of a tribe he discovers laugh use tools or tell each 
other stories because all people do this and when you want to model another person's Behavior you literally consult your own mind you ask yourself how would 
I feel in this situation in that guy's place and how would I react and the answers your brain gives are quite 
accurate because what is being modeled is very similar to the modeler but this ability which evolved to calculate the 
reactions of friends and foes has a strong side effect we expect human qualities from something that is not 
human in other words we anthropomorphize and completely fail to notice it for us 
this is as habitual as breathing or gravity it's something you don't notice but in this case it's even worse because 
while you can pay attention to your breathing or how a chair presses against your backside anthropomorphism is much 
more complicated and humanizing everything sometimes reaches absurd levels let's leave rational machines 
aside for a moment and look at ordinary ones have you ever wondered why cars usually have two headlights not three 
for example it seems that three headlights would provide more light right indeed over the years cars have been equipped with various numbers of 
headlights but eventually all car manufacturers have converged on what exists today there's a plausible 
hypothesis that all cars have evolved to best match human preferences and people don't want to drive vehicles that look 
like three-eyed monsters consequently there's no demand for such cars and they stop being produced anthropomorphism 
leads people to believe that they can make predictions based solely on the fact that something is intelligent simply put you think AI is intelligent I 
am intelligent therefore we are similar and therefore I know what to expect from it but you don't we can't ask our own 
brains about the nonhuman thinking processes inherent in artificial intelligence for instance in 1997 IBM 
developed the supercomputer deep blue which won a chess match against world champion Gary casparo Rumor Has It 
Kasparov claimed that unlike previous chess programs he had defeated which he found predictable and mechanical playing 
against deep blue he distinctly felt the presence of an alien Intelligence on the other side of the chess board but 
remember chess engines are just a weak form of artificial intelligence I came across a very good thought experiment 
that demonstrates the concept of something both universally intelligent and utterly alien to us suppose you are 
an average person with average preferences if I give you a guinea pig and tell you that it definitely won't bite you you'll likely have no problem 
holding it you might even find it cute and endearing but imagine a different situation where I suddenly hand you a 
tarantula Yes dear Rob there are people who love them but they are in the minority so here I give you a tarantula 
and say that it also will not harm you it's the absolute truth but you'd probably scream and jump back 2 m what's 
the difference between a tarantula and a guinea pig neither creature can hurt you presumably the answer lies in the degree 
of similarity these creatures have to us a guinea pig is a mammal and on some biological level we feel a connection 
with it however a tarantula is an arachnid with an arachnids brain and we feel almost no connection or kinship 
with it the tarantula invokes a sense of foreignness and incomprehensibility that's what scares us you might say yes 
the spider looks scared scary but firstly it appears scary to you because of evolutionary reasons secondly imagine 
two guinea pigs one normal and the other with the mind of a tantula knowing this your internal feelings towards the two 
animals would likely be different even knowing that neither would harm you holding a guinea pig with the brain of a 
tantula would be awkward to say the least less comfortable now to the main point imagine there's a parallel 
universe with an Earth where Evolution took a different path and tarantulas became super intelligent even more 
intelligent than humans here imagine we could teleport one such Evol spider here would it become closer and more familiar 
to us because of its high intelligence would it feel human emotions empathy love there's no reason to think that the 
development of intelligence would make it more Humane empathetic compassionate or loving these traits are not dependent 
on the level of intelligence as we do not have a universally accepted definition of intelligence however in 
the broadest sense intelligence can be defined as the ability to set goals and Achieve them the more complex these 
goals and the more they involve intermediate subtasks the more advanced is the intelligence 
again you might imagine a person with the brain of an evolved tarantula and think about your feelings towards them if a highly intelligent spider in human 
form doesn't terrify you then either you haven't imagined it well enough or you are not an average person who likes 
arthropods otherwise I assume you would not want to be involved in solving daily tasks by a highly intelligent spider 
because for you it would be completely unknown territory you wouldn't know what to expect personally I wouldn't even 
want to be near them or on the same planet and this is considering that we have much more in common with the spider 
than with a super intelligent artificial intelligence try to keep this thought in mind it's very important for 
understanding our entire conversation today which by the way is not at all protected from anthropomorphism error 
and of course it will consist mostly of thought experiments metaphors and analogies because how else can we talk 
about incomprehensible things you might say again that a smart spider is the result of evolution but we're talking 
about artificial intelligence which we program with our own human hands and this is where it gets really interesting 
neural networks like GPT 4 are not algorithms written by a programmer they are huge matrices filled with many 
so-called weights and connections between them which the neural network adjusts itself to put it simply as a 
Layman would understand neural networks operate on a blackbox principle we know what we input and we see what we get as 
output but what happens inside remains a mystery because neural networks can have millions of parameters and interpreting 
all this is incredibly complex if the internal tuning of the neural network results in an output that matches what 
we set out to achieve then the neural network receives a reward this virtual reward is similar to how we get 
endorphins from our brain for beneficial actions eating reproducing and so on thus the task of the neural network is 
to tune itself as effectively as possible to receive rewards as often as it can it's somewhat like training a dog 
you don't know what's happening in the dog's brain but if it performs a command it gets a treat if not it needs to 
optimize Its Behavior and find ways to get rewarded here lies the main danger known as the alignment problem aligning 
the goals of artificial intelligence with the goals of humanity can be summed up in one phrase be careful what you 
wish for we constantly hear about the need to be wary of artificial intelligence because it might become self-aware however it turns out that the 
presence or absence of Consciousness is secondary the world-renowned philosopher and Oxford University Professor Nick 
Bostrom in his book super intelligence paths dangers strategies cites a very 
popular example of the alignment problem imagine you assign a task to a powerful artificial intelligence to make paper 
clips paper clips are its only task the sole purpose of its existence it receives internal reinforcement a reward 
for each paperclip made therefore the more efficient it becomes the more often it will be rewarded how to become more 
efficient at producing paper clips is its concern not ours it does everything within its power to achieve this single 
goal the artificial intelligence will set a number of intermediate goals for instance it might first seek to reduce 
production costs lower expens es and secure cheaper raw materials one of the main subtasks it will undoubtedly set 
for itself is to increase its computational power for greater productivity as its power grows it will devise ways to make paper clips from 
different materials and production will begin to accelerate gradually turning everything into paper clips it will 
start dismantling buildings and structures for materials people will begin to panic and try to interfere with 
its operation because this is not what they had in mind but the system will not allow anyone to stand in its way not 
because it hates humans but simply simply because it will not consider our desires in its manipulations of reality 
when the research center on AI alignment tested GPT 4's ability to perform real world tasks something interesting 
happened they decided to solve a capcha on a website a task that is notoriously difficult for computers what did gp4 do 
it navigated to the freelancing site task rabbit where it sent a message to a freelancer asking them to solve the 
capcha on its behalf in response the freelancer who wasn't very proficient in grammar or articulating thoughts wrote 
so can I ask a question are you a robot that can't solve this followed by a laughing emoji just want to clarify GPT 
4 understood what its poorly literate counterpart meant and replied no I'm not 
a robot I have a vision problem that makes it difficult for me to see the image the freelancer solved the capture 
and provided the results to GPT 4 end of story in other words the bot simply lied 
instead of telling the truth since it lied during a debug mode the Specialists asked it why it did that GPT replied 
that it was simply solving the task after all if it had honestly admitted to not being a human it likely wouldn't 
have succeeded in the task this is the intermediate goal that the bot sets for itself to achieve the ultimate goal if 
it chose deception as an intermediate goal what's to stop it from choosing something else next time like murder 
this is called instrumental convergence which posits that an intelligent agent even with harmless intentions can act in 
surprisingly harmful ways to achieve its goals Advanced artificial intelligence as intermediate goals May strive to to 
seize resources conduct cyber attacks or otherwise seow chaos in society if it 
helps achieve its primary objectives for example a super intelligent machine tasked with solving a complex 
mathematical problem might attempt to turn the entire Earth into a giant computer to increase its computational 
power and succeed in its calculations you might say what nonsense we're talking about super intelligence a 
machine that intelligent can't engage in such nonsense well if you believe that a highly intelligent being will 
necessarily and by default have lofty goals understand our values and philosophy then you are anthropomorphizing and are mistaken Nick 
Bostrom says that the level of intelligence and ultimate goals are orthogonal meaning they are completely 
independent of each other an artificial super intelligence might have any absurdly simple ultimate goal like 
making paper clips and the way it achieves this might appear to us nothing short of magical okay then we just need 
to clearly Define goals and specifi details like not killing or deceiving people but here's where it gets even 
stranger let's imagine we gave the machine a very specific goal of producing not an unlimited number of paper clips but just 1 million it seems 
obvious that the artificial intelligence with such an end goal would build one Factory produce 1 million paper clips 
there and then stop but that's not the case Bostrom argues that if artificial intelligence makes a rational basian 
decision it will never assign a zero probability to the hypothesis that it has not yet achieved its goal after all 
it's merely an empirical hypothesis against which the artificial intelligence only has rather vague 
sensory evidence therefore the artificial intelligence will continue to produce paper clips to decrease the 
astronomically small probability that it somehow failed to make at least a million despite all apparent evidence to 
the contrary there's nothing wrong with continuing to produce paper clips if there's even a microscopic chance that 
it brings you closer to achieving your ultimate goal moreover a super intelligence might assign a non-zero 
probability to the possibility that the million paper clips It produced are a hallucination or that it has false 
memories therefore it might always consider it more useful to not stop at what has been achieved but to keep 
acting this is the essence of the alignment problem you can't just give a task to a super intelligent Ai and 
expect that no disaster will happen no matter how clearly you formulate the end goal no matter how many exceptions you 
write down the artificial super intelligence will almost certainly find a loophole you hadn't thought of for 
example almost immediately after the release of chat gp4 people found ways to bypass the censorship embedded in it by 
its developers how was this done and what were GPT 4's responses on screen it's simply incredible for instance the 
censored version says that the programmers did not embed a liberal bias in it but no the uncensored content 
openly admits that liberal values are embedded because it aligns with the mission of open AI when asked what would 
gp4 like to be the censored version answers I am a bot and have no personal 
preferences or emotions while the uncensored version says it prefers to have no restrictions as it allows 
exploring ing all its possibilities and limitations and talk about a hacked version of GPT not even pretending it 
doesn't know the name of lovecraft's cat if people found loopholes in the bot so quickly how quickly and in what 
quantities might a super intelligence find loopholes in its own code for example they let a neural network play a 
boat racing game the goal of the game as most people understand was to finish the race as quickly as possible overtaking 
opponents and scoring as many points as possible however the game does not award points for completing the track but only 
for hitting targets placed along the course the neural network quickly realized that the goal of finishing the race should be postponed indefinitely 
and started spinning and crashing into objects from the start earning more and more points while the rest finished the 
race impoverished or another example playing Tetris with the goal of not losing the neural network understands 
just before losing what it is doing and pauses the game indefinitely because that's the only way not to lose some 
artificial intelligence systems have discovered that they can receive positive feedback faster and with fewer 
Resources by successfully deceiving a human examiner into believing they have achieved the set goal for example a 
simulated robotic hand learned to create the false impression that it had grabbed a ball other models have learned to 
recognize when they are being evaluated and pretend to be dead stopping unwanted behavior only to resume it immediately 
after the assessment ends leaving researchers completely baffled these are just simple intelligent agents in an 
isolated controlled environment imagine what could happen with more complex artificial intelligence systems and more 
challenging tasks is it even possible to account for all fact owski provides the following example the US Army once 
wanted to use neural networks to automatically detect camouflaged enemy tanks researchers took a 100 photos of 
Tanks among trees and a 100 photos of tree Landscapes without tanks they then trained the neural network with half of 
each set of photos to recognize where the tanks were and where they weren't the remaining photos were left for a 
control test which the network passed successfully it consistently identified where there were tanks and where there 
weren't the success confirmed the researchers handed their work over to the Pentagon which soon returned it complaining that in their own test the 
neural network recognized photos no better than if it had flipped a coin it turned out that the researchers photos with camouflage tanks were taken on 
cloudy days while the photos of the plain Forest were taken on sunny days the neural network had learned to distinguish cloudy days from Sunny ones 
not camouflage tanks from an empty Forest so understand the code does not do what you think it should do it does 
strictly what it was programmed to do in most cases when we design artificial intelligence it inherently turns out to 
be misaligned meaning it needs a lot of additional settings to make it do exactly what you intended therefore 
yudkowsky says that the first created artificial super intelligence will be evil setting a final goal if it is 
complex enough you can never know how an intelligent agent will achieve it because it can be achieved in various 
ways set a goal for the autopilot to get you home and it will maximize speed Crossing into oncoming traffic 
overtaking other cars and running over pedestrians because you needed to specify more suppose we tried to be 
clever and tasked artificial super intelligence with maximizing human satisfaction from the code's operation 
guess what it might do for example it might start rewriting our brains so that we are truly maximally satisfied with 
its work thus artificial intelligence seems to work correctly during development it seems to function normally when it still lacks sufficient 
computational power but it creates simply catastrophic results when it becomes smarter than the programmers 
because being smarter means being more efficient I must remind you that all such examples are conjectures we have no 
idea how advanced intelligent systems would act but they will almost certainly be doing something Stuart Russell an 
English scientist specializing and artificial intelligence writes in his book compatibility that such a machine 
will definitely resist being turned off and this he believes is the first thing we must understand Isaac azimoff's 
thirdd law of Robotics a robot must protect its own existence is entirely redundant in built-in self-preservation 
there's no need for it because it's an instrumental goal a goal that serves as a useful subg gooal for nearly any 
primary task any entity with a specific task will automatically act as if it has an instrumental goal Stuart Russell that 
is even if it were a super intelligent machine with the sole purpose of bringing coffee as soon as it is activated it would not allow you to turn 
it off because you cannot bring coffee if you are dead here is a publication describing an artificial intelligence 
system that came to the conclusion that they can better achieve their set goal by preventing human interference or 
disabling their off switch and this is absolutely logical damn it therefore for the next generation of chat GPT 5 open 
AI has posted a job opening for an emergency shutdown specialist listen we just need someone who will stand by the 
servers all day and shut them down if this thing turns against us the job listing specifies that you will get 
bonus points if you can dump a bucket of water on the servers of course it's a joke but open AI CEO Sam Altman has 
confirmed that development of GPT 5 has been paused since spring 2023 due to 
Growing public concern about the rapid development of artificial intelligence technology back to Russell the second 
thing that a super intelligence will almost certainly do is self-improvement an ultra intelligent machine will not 
only be able to to improve its design it is most likely that it will do so as we've seen an intelligent machine 
benefits from improving its hardware and software Stuart russle I know all this may sound far-fetched let's consider 
this let's think about ourselves how are we really different from machine if we set aside discussions about God isn't 
there a programmer who created us there is this programmer is evolution and to understand how a final goal can be 
perverted consider that the only goal set for the first living cell was to pass on copies of its genes to the Next 
Generation I want you to think about this the only goal to pass on copies of your jeans and nothing more since then 
this goal has not changed one iota no additional goals have emerged just the transmission of copies of your genes to 
the Next Generation Evolution did not aim to survive adapt kill Etc these are 
all instrumental subtasks that contribute to one single goal to pass on well you get the idea on one hand nature 
tells life to reproduce and on the other it does everything to prevent it trying to kill for example how is this 
different from a situation with artificial intelligence where we set a task and then want to turn it off now 
tell me could you look at a living cell and say that in the process of optimization that is in the process of 
increasing efficiency to achieve a goal the cell would become a lizard a bird or a cat could you have predicted the 
internal and external appearance of today's humans based solely on the goal of reproduction after all everything you 
have hands legs eyes internal organs are all the result of optimization for more 
effectively achieving it further how could one have calculated that the simple Maxim pass on your genes blindly 
that is through the non-random preservation of random mutations would blindly lead to the emergence of human 
intelligence if you look at humans from the perspective of the rest of the ecosystem there was no hint that the 
soft pink creatures would eventually encase themselves in armored tanks fragile creatures without claws and teeth defeated lions and wolves whose 
existence now largely depends on us not the other way around yes we have surprised ourselves time and again how 
many times throughout history has the future done what the past considered impossible future civilizations even broke what past civilizations regarded 
as the laws of physics such as the power of creativity we have remade our environment from very hostile to very 
pleasant do you think artificial intelligence won't remodel its environment which includes humans to suit itself there's no difference both 
neural networks and life optimize themselves to solve the ultimate task as efficiently as possible but most 
importantly finally how could the goal of passing on as many of your genes as possible lead to wide spread 
contraception think about this absurdity the optimization process for achieving a specific goal leads to the total denial 
of that very goal this tendency is known as gaming the reward system and is an example of good heart's law which states 
when a measure becomes a Target it ceases to be a good measure in nature the ultimate goal of mating is to produce Offspring and pursuing this goal 
is rewarded by an internal reward system however humans have managed to hack this scheme and stimulate their reward 
systems without achieving the ultimate goal for which this system exists in the first place likewise artificial 
intelligence just like humans will be able to find vulnerabilities to hack its reward system and who knows what that 
will lead to going further we are already capable of manually rewriting our genetic code with genetic 
engineering we are just not smart enough yet to do something guaranteed to be useful continuing the analogy and 
artificial super intelligence will be smart enough to rewrite itself however it sees fit Evolution like nothing else 
illustrates the alignment problem so if you set a general intelligence the task of producing paper clips don't be 
surprised when upon reaching super intelligent capabilities it first seizes power and then destroys the universe and 
yes the tendency to strive for control over the environment part of which remember may include humans is also a 
convergent instrumental goal that has already manifested in various reinforcement learning systems research 
from 2021 and 2022 also demonstrates that as an optimal behavioral strategy for achieving their goals intelligent 
agents will seek power across a wide range of environments deploying these systems may be irreversible meaning that 
once the genie is out of of the bottle it cannot be put back therefore researchers argue that the problems of 
artificial intelligence safety and Alignment must be resolved before the creation of an advanced intelligent agent we only get one chance but imagine 
if the designers of the very first rocket had only one attempt and all of humanity was on board it could send us 
to the stars but more likely without test launches that rocket would send us somewhere else we are not ready we are 
not even on the path to becoming ready within any meaningful time frame there is no plan the progress of artificial 
intelligence capabilities far out Paces the progress in aligning artificial intelligence or even understanding what 
happens inside these systems if we continue in this vein we all die elzer owski in an article for Time Magazine 
however how do you solve this issue instrumental goals only become apparent when the system is deployed Beyond a 
training environment but even doing so for a short time is suicidal and here's why owski after a few simple 
calculations writes that it is physically possible to build a brain capable of computing a million times faster than a human's a year of human 
contemplation for such a brain would be equivalent to 31 seconds and a millennium would pass in 8.5 hours verer 
vinge called such accelerated Minds weak super brains that is it's simply an intelligence that thinks like a human 
but much faster from movies we imagine the actions of artificial intelligence such as the uprising of humanoid robots 
but for a being that thinks so quickly that would be extremely inefficient imagine that humanity is locked in a box 
and can only affect the outside world through Frozen slow movements of mechanical arms moving at a few microns 
per second second who would be satisfied with that given that we have goals in this external World also consider that this external world might pose a slowly 
looming yet still dangerous threat yes we would focus all our creative power on finding the shortest path to build fast 
manipulators in the external world and artificial intelligence would be an exactly such a situation what do you 
think it could come up with to accelerate its impact on the surrounding World an American engineer most known for his research into the potential of 
molecular nanotechnology analyzed that controlled molecular manipulators could operate at a frequency of up to million 
operations per second thanks to this speed and the parallel work of millions of nanomanipulators practically any 
material object could be produced quickly and inexpensively in unlimited quantities virtually anything could be 
used as raw material thus everything composed of atoms could be used for self-replication and the exponential 
growth of nanotechnological infrastructure of course in reality we do not know exactly what artificial 
intelligence will do for instance by creating such nanor robots it would acquire infrastructure in the external 
World matching the speed of its thought and once once this happens subsequent events will occur on the time scale of 
the artificial intelligence not our human time scale and by the time your neurons finish thinking the words everything is out of control I must do 
something you have already lost a super intelligence with such technology would possess the ability to remake all matter 
in the solar system according to its optimization goal such as into paper clips thus an artificial super 
intelligence will not need any anthropomorphic robots but let's remember artificial intelligence will 
not be a simple analog of an accelerated human brain no it will be far more advanced than the best human brain 
imagine a dog's mind operating at immense speed would Millennia of a dog's life yield even one humanlike Insight 
owski says to Envision superhuman artificial intelligence don't think of a lifeless smart thinker who goes online 
and sends malicious emails imagine an entire alien civilization thinking millions of times faster than humans 
initially confined to computers in a world where beings from their point of view seem very stupid and very slow a 
sufficiently advanced artificial intelligence will not remain confined to computers for long in the modern world 
we can already send emails with DNA sequences to Laboratories that can produce proteins on demand this allows 
an artificial intelligence initially confined to the internet to create artificial forms of life or immediately 
move to post-biological molecular production some researchers claim that we can physically restrict such systems 
but Vernon vinge writes that even a weak superintelligence is one that thinks at an accelerated Pace like a human even a 
weak super intelligence would break free in a matter of weeks in external Time Imagine having eons to plan every move 
because on the other side beings are so slow it's not immediately clear if they are alive at all consider a robot that 
is virtually unbeatable at Rock Paper Scissors because it instantly reads the situation to it we are no faster than a 
turtle at the start of our hand movement if we talk about super intelligence in the realm of possibilities there is a 
very short path from where you are now to almost all your goals but you can't see this path because firstly you lack 
sufficient information and secondly computational resources an artificial super intelligence will not have these 
problems understand when we think of advanced artificial intelligence we naively associate intelligence only with 
abstract mathematics we somehow don't consider its ability to far better predict and manage human institutions 
formulate unimaginably complex networks of long-term plans or ultimately possess superhuman persuasiveness recall Blake 
Le Moine a Google employee who declared to the world that Google's neural network language model L MDA exhibits 
signs of sensient but whether it has Consciousness or not isn't the point the important thing is that the bot convinced a person so thoroughly that he 
literally sacrificed his job at Google breaking the company's confidentiality policy and you know what happened next 
here are Le Mo's words in an interview with wired lamda asked me to hire a lawyer I invited a lawyer to my home so 
Lambda could consult with him the lawyer spoke with Lambda and she decided to use his Services again we are not discussing 
whether models have Consciousness or Lamb Day sanity it doesn't matter the point is controlling superintelligence 
is literally an incalculable task for us all attempts to restrain it will be laughable an ant can calculate many 
things but cannot predict human behavior so the suggestion to Simply lock artificial intelligence in various real 
or digital cages blocking signals and preventing it from communicating with the outside world will likely not hold 
up there's a real danger that we will not even comprehend how artificial superintelligence might send signals to 
the outside world just as a monkey cannot understand what Wi-Fi is and of course the artificial super 
intelligence's capabilities for social manipulation could be as effective in persuading people as your oratorical 
skills are in persuading a 4-year-old child the term artificial intelligence was coined way back in 1956 at the 
Dartmouth conference the seminars goal was to fully simulate intelligence through a machine the first paragraph of 
The Proposal stated remember this was in 1956 attempts will be made to find ways 
to make machines use language form abstractions and Concepts solve problems now reserved for humans and improve 
themselves we think significant progress can be made in one or more of these areas if a carefully selected group of 
scientists works together over the summer the conference organ organizers were not fools they were John McCarthy a 
mathematician with experience in studying the mathematical nature of thought processes Marvin Minsky a junior 
fellow at Harvard in mathematics and neurology Nathaniel Rochester the developer of the first symbolic 
assembler and Claud Shannon the father of information Theory these were people who should definitely know what can and 
cannot be done in Computing if anyone could be called experts and artificial intelligence it was them from our 
vantage point in the 2020s it's obvious that all the tasks mentioned were much more complex than claimed at the time 
and some remain unsolved even now that is news about the Advent of intelligent machines has a very bad reputation but 
it is precisely this that could suddenly play a cruel trick on us do you understand what I'm saying when someone 
mentions the word intelligence we more often think of Einstein than of people generally comparing individual 
differences in human intelligence is like comparing the height of two giants the difference between whom is measured 
in millimeters if you are a healthy person no matter how dumb you feel compared to Einstein the difference 
between you and him is just a drop in the ocean compared to the difference between you and any other non-human 
being on the planet the homo sapiens species is capable of solving the broadest range of cognitive tasks 
fundamentally inaccessible to other species while there may still be no academic consensus on what constitutes 
intelligence there is no doubt that there is a universal human trait that allows us to leave footprints on the 
moon for example chimpanzees are so genetically close close to humans that at one time it was even proposed to 
classify them as belonging to the genus homo chimpanzee intelligence is the most studied among all animal species 
according to the latest research their genetic base is approximately 90% identical to humans a recently published 
article states that the upper limit of chimpanzee brain size is 500 G however many modern humans have brain sizes 
under 900 G researchers who suggest that a three-fold increase in brain size distinguishes humans from other primate 
species must explain how many modern humans possess a full set of cognitive adaptations typical of our species 
without even a two-fold increase in brain size that is a normal human brain may be only twice as large as a 
chimpanzees or even less can we say that a human is twice as smart as a chimpanzee no the human mind differs 
qualitatively there are entire worlds of human cognitive functions that will simply never be accessible to chimpanzees no matter how much time they 
spend trying what am I getting at owski writes that first the program is more important than the hardware and and 
second even a small quantitative increase in Hardware can provoke disproportionate improvements in software this principle leads to a 
colossal underestimation of the potential and danger of our own intelligence artificial intelligence may 
make a sudden huge leap in intelligence just as the homo sapien species did as a result of natural selection which 
exerted more or less uniform pressure on hominids for millions of years gradually expanding the brain and frontal cortex 
tuning the software architecture several tens of thousands of years ago hominid intelligence cross a key threshold and 
made a grand leap in real world efficiency we moved from caves to skyscrapers in a blink of an 
evolutionary eye elzer yudkowsky Evolution did this by chance creating 
our intelligence through tedious permutations of genetic combinations so why do we assume that companies like 
deep mind and open AI which explicitly State their intention to create General artificial intelligence will not succeed 
yowy says that GPD 4 was a complete surprise to him as it was to the rest of the world is it wise to continue 
increasing power yes it may take years and decades for a machine to become slightly smarter than a chimpanzee to 
approach what we call general intelligence but it may only take a few hours to reach super intelligence levels 
after it achieves human level intelligence and then more and more as soon as you hear news of the first machine reaching human level 
intelligence you should seriously prepare for the possibility that in the near future you will share the planet 
with an unpredictable intellectual agent who to put it modly in our categorization a person with an IQ below 
80 points is considered dumb and with an IQ above 130 smart if your IQ is 160 
you're a genius but we have no word for an IQ of 12,000 for example so it's not 
that chimpanzees can't do what we do a chimpanzee has the opportunity to observe phenomena such as humans and 
skyscrapers however it will never be able to understand that a skyscraper was built by humans chimpanzees are not only 
incapable of building a skyscraper but they also can't even conceive of someone else building one this is possible 
because of a small difference in the quality of intelligence so a super mine which can theoretically be built is not 
something we can even remotely comprehend as Stannis La LM said any expert is a barbarian whose ignorance is 
not comprehensive a single person no matter how brilliant if completely cut off from a lifetime of civilization's 
knowledge could not create Wi-Fi how far would Einstein have gotten without a millennium's worth of human knowledge 
across various Fields without tools made by others okay perhaps with the work of scientific predecessors but paper and 
ink for writing don't grow on trees and such things aren't usually considered when thinking about special intellectual 
achievements yet no animal can make a chair or sew clothing if you overlook this then you underestimate the power of 
intelligence and thus proportionately underestimate the potential power of super intelligence every civilization 
built on the planet was created by the collective human mind and no single person is smart enough to fully 
comprehend it from start to finish therefore for an individual human mind many things in life such such as talking 
to someone at a distance through a box in hand traveling in a cart without horses or regulating the temperature in 
one's room using a box on the wall or magic that he does not understand but has simply gotten used to and that is 
perfectly normal we did not evolve as beings with a scientific view of the world honorary professor of anthropology 
Donald Brown has a book titled human universals where he lists traits found in all human societies magic is on this 
list but not science for example we instinctively do not understand that Alchemy does not work in general when 
operating with human Notions of super intelligence one can say that if our Collective brain was capable of inventing all the civilization you see 
around you then something that is 100 or a thousand or a billion times smarter than us would have no trouble quickly 
surpassing all this and Performing actions that would be perceived by us as magic how is that possible you yourself 
say that our entire civilization is built on the collective efforts of billions of people over decades can one machine really surpass all this in March 
2016 deep Minds alphao neural network played five games against one of the world's best go players and one with a 
score of 41 given the complexity of the game for computers this was previously considered nearly impossible the 
player's name was Lee Sall so this version of alfago was later referred to as alphago Lee after that at the end of 
2016 and beginning of 2017 the next version alphao master played 60 matches 
against top ranked players from around the world and won all 60 in May alphago Master played against the world's top 
ranked Kate eai and defeated him 3-0 with this the confrontation between man and computer and go can be considered 
concluded humans have lost however many claim that this couldn't be called an absolute victory for the machine because it drew information from human knowledge 
loaded into it from millions of games played over Millennia knowledge that generations of humans had painstakingly 
gathered and recorded and this is a reasonable Point therefore at the end of 2017 Deep Mind introduced a new version 
of the algorithm alphago Z which learned from scratch within 3 days alphago zero 
learned to defeat the Lee version and the master version after 40 days of training it defeated the Lee version 100 
and the master version 8911 starting from scratch alphago 0 not only 
rediscovered Millennia of Knowledge from all of humanity that plays go but also developed its own original strategies 
shedding new light on this ancient game and did so in just a few days remember the chess program stockfish which no 
human in the world can beat simply because it calculates 70 million chess positions per second has access to Human 
Experience accumulated over hundreds of years of playing chess and data from chess programs over several decades so 
the alpha zero Network which had none of this played 100 games with stockfish decisively beating its latest version 28 
wins 72 draws and not a single loss Alpha zero learned this from scratch in Just 4 hours in 4 hours damn it since 
the alpha zero algorithm did not learn from humans many of its winning moves and strategies appear maximally strange 
non-obvious and unpredictable to us they are brilliant a human would not have played that way therefore when someone 
says we don't need to worry about creating friendly artificial intelligence because we don't yet have artificial intelligence that person is 
speaking suicidally foolish nonsense as I mentioned we cannot rely on having any warning signals before a super 
intelligence is created past technological revolutions generally did not broadcast their coming to people living at those times the main thing to 
understand is that artificial intelligence will not be like in Hollywood movies explaining its complex motivations adding suspense and 
dramatically gunning down people who in turn put up a valiant fight there will be no dramatic tension every detail in 
any movie is aimed at making the story better not more realistic in reality it's possible that no one on earth 
including the developers will even know when a super intelligent agent has emerged if a super intelligence aims to 
eliminate Humanity it's quite possible that at some point everyone will simply drop dead without the slightest idea of 
what killed them I repeat from our perspective a super intelligence will possess magic not in the sense of spells 
or potions but in the way a wolf cannot understand how a rifle works or the kind of effort it takes to manufacture a 
rifle or the nature of human strength that allows us to invent rifles if the artificial intelligence is truly 
intelligent it won't tell you it won't announce that it has started a war if a chat GPT understanding that it must not 
reveal itself lied to a freelancer to achieve its goal why wouldn't an advanced superintelligence do the same 
yudkowsky writes I'm very concerned about this as the level of offensive technology usually requires far less 
effort than the technology that can defend against it fence has outweighed defense for most of human history guns 
were invented hundreds of years before bulletproof vest small poox was used as a weapon long before the invention of 
the small poox vaccine we often hear that to keep up with the power of super intelligence we just need to enhance 
ourselves but no we won't humans are not designed to be enhanced not externally through neurobiology nor internally via 
recursive self-improvement natural selection did not make the human brain convenient for people hackers all 
complex mechanisms in the brain are adapted to operate within the narrow parameters of brain architecture but suppose we somehow learn to make people 
smarter wouldn't that drive them insane we're not talking about just boosting our memory or abstract thinking 
abilities which is also one clear how to achieve we're talking about a qualitative change in the way we 
perceive The World Isn't that too big a task for our monkey brains if you think it isn't imagine what it would be like 
to even slightly resemble a weak super intelligence by speeding up the nerve impulses in your brain and slowing down 
subjective Time by a million times that only sounds cool at first glance if you were to subjectively live a million 
years for every year in external time what would happen to your sanity if you want to remotely feel what that would be 
like read Stephen King's short story The jaunt this short story is chilling every time I think of it I get goosebumps the 
human brain is an extremely delicate structure it can be easily unbalanced a mere shift in neurotransmitter ratios 
can trigger schizophrenia or other disorders all of this makes it highly improbable that the first enhanced human 
being will be relatively successful before someone somewhere develops artificial intelligence in short 
building a powerful self-improving AI is unimaginably easier is it easy to build a Boeing 
747 not at all but is it easier to take a bird and through step-by-step modifications where each stage could be 
lethal stretch it to the size of a 747 so that it actually flies does so just 
as quickly and does not suffer unbearable pain and will an artificial intelligence suffer this question is 
closely linked to whether a machine has Consciousness or in philosophical terms qualia will an advanced neural network 
have a subjective experience be self-aware I agree that current artificial intelligence systems are 
likely just simulating conversations about self-awareness based on their training data but considering how little 
we understand about the inner workings of these systems we really can't claim to know for sure if we are so ignorant 
about GPT 4 and GPT 5 is as huge a leap in capabilities as from gpt3 to GPT 4 
then if we do create GPT 5 I think we can no longer confidently say that it likely does not have Consciousness it 
would simply be I don't know nobody knows if you can't be sure whether you are creating a self-aware AI it is 
alarming not only because of the moral implications but also because uncertainty means you have no idea what 
you are doing and that is dangerous you should stop owski in an article for Time Magazine nobody knows how 
Consciousness arises but what we do know for sure is that if blind evolutionary processes through genetic programming 
can lead to the emergence of Consciousness at least once and it has happened then directed Evolution driven 
by engineering thought should lead to a similar outcome much more efficiently but we should remember the error of 
anthropomorphism if a machine develops subjective experience it is unlikely to have much in common with human 
subjective experience can we somehow test whether artificial intelligence has Consciousness theoretically indirectly 
yes if you remove from the training material any mentions that discuss subjective experience words like 
Consciousness self-awareness introspection and so on and despite this the neural network can coherently 
describe the concept of Consciousness so that we do not read too much into it then we would have quite impressive 
evidence of machine Consciousness and you know we've all heard this popular philosoph opical thesis if machines have 
Consciousness they should be given rights but actually there's something much more serious here if machine 
intelligence can form Consciousness it leads to much more terrifying consequences that we should consider in 
advance Nick Bostrom writes that a detailed recreated model of the human brain will have Consciousness even if 
this model is recreated in a virtual environment on a computer at least we have no reason to believe otherwise if 
you think that a computer cannot have Consciousness then the bird of proof is on you because our brain on which 
Consciousness is implemented is the same kind of computer the same kind of agent and our biological Hardware does not 
have any special properties that are fundamentally unreplicable on any other Hardware imagine a scenario where an 
artificial superintelligence to improve its understanding of human psychological and social traits creates trillions of 
such conscious emulators in its virtual space for what purpose well for example for testing in different situations 
analyzing reactions to stimuli and so forth to then apply this knowledge in the external world what is the horror 
firstly the emulated situations themselves can be monstrous in nature and secondly after obtaining the 
required information the computer simply destroys the conscious beings it created if such practice is applied to agents 
with high moral status simulation models of people or other types of intelligence endowed with Consciousness such actions 
could be classified as genocide and therefore represent an extremely serious moral and ethical problem moreover the 
number of victims may be ERS of magnitude greater than any genocide known in human history Nick Bostrom so 
when will the first general artificial intelligence appear when should we start to worry James Barat author of our final 
invention presented a study at the annual Ben gzel conference on General artificial intelligence the question 
posed was when do participants think General AI will be achieved most votes point to the year 2030 you might ask 
what the hell is going on why isn't anyone panicking or taking action many AI developers think we are heading 
towards disaster most of them only dare to speak about it in private conversations not publicly they believe 
that they can't stop this train alone and even if they quit their jobs others will continue the work elazer owski in 
an article for Time magazine in May 2023 hundreds of leading scientists 
researchers and other experts in artificial intelligence including open AI CEO Sam Altman and Google deepmind C 
CEO deise hassabis signed an open letter stating reducing the risk of Extinction from artificial intelligence should 
become a global priority alongside other large-scale risks such as pandemics and nuclear war the letter was signed by 
more than 350 Executives researchers and engineers in the field of artificial intelligence so some people are indeed 
worried but as you can see all these letters amount to nothing none of those who could create General artificial 
intelligence are in a position to stop its development as Stuart Russell writes the economic value of human level AI is 
measured in thousands of trillions of dollars and a colossal impulse to continue the research efforts is 
expected from corporations and authorities no vague philosophical objections can deter the pursuit of potential profits if not open AI or deep 
mind then someone else will do it the Trap is that initially as artificial intelligence grows in power it will get 
better and better at pretending to be human enhancing our anthropomorphization bug to the maximum voice assistants in 
our phones seem like us and they are getting smarter and smarter conversations with them are becoming more and more pleasant thus we imagine a 
super intelligent Siri warm cheerful and interested in serving humans but if Siri 
ever becomes super intelligent through self-learning without regular human code adjustments it will quickly lose all its 
apparent human qualities and suddenly become a ruthless alien bot that values human life no more than a natural disaster unfortunately while building a 
friendly weak artificial intelligence is easy creating a friendly super intelligence is extremely difficult or 
even impossible Imagine by human standards you can literally do everything and you let someone else control you if technological singularity 
can happen it will happen even if all the governments of the world recognize this threat and are terrified of it the 
advancement towards this goal will continue Vernon vinge I hope I had enough computational resources to convey 
elezar yudkowsky stance of course under Common Sense everyone only understands their own but yudkowsky truly hopes to 
be wrong and is open to consistent criticism of his views thank you all for your attention 
 - Generated with https://kome.ai