History Main / TheGenieInTheMachine

[[AC:General]]

[[AC:{{Anime}} and {{Manga}}]]

[[AC:{{Film}}]]

[[AC:{{Literature}}]]

[[AC:LiveActionTV]]

[[AC:{{Machinima}}]]

[[AC:VideoGames]]

[[AC:{{Webcomics}}]]

[[AC:WesternAnimation]]

[[AC:RealLife]]

*** The author saw this as a good thing, because unlike some other stories where robots prevent humans from doing anything that might involve the slightest risk (eating fatty foods, working, etc.), these robots were smart enough and ethical enough to only justify taking action to stop the worst acts and make sure no human realizes the robots are in charge.

*** The author saw this a good thing, because unlike some other stories where robots prevent humans from doing anything that might involve the slightest risk (eating fatty foods, working, etc.), these robots were smart enough and ethical enough to only justify taking action to stop the worst acts and make sure no human realizes the robots are in charge.

*** Susan Calvin points out that advanced robots like Nestor possess a sort of subconscious superiority complex towards humans (they are stronger, tougher, faster, smarter, etc than us, but are bound to value our lives above their own and obey our every command). Messing with the safeguards that make them incapable of ever expressing this "feeling" in their actions (such as by effortlessly crushing a human skull with one hand) is one of the stupidest things a person could ever do in her opinion. In this case, the robot (though capable of understanding the nuance of the command to "get lost") decided to take it literally as a way of acting out against it's human masters. Their initial failures to identify it only serve to reinforce this "rebellious" line of thinking and Calvin warns everyone that the longer they take to resolve the situation, the more dangerous the robot could become.
** And the story about the mining robot who was supposed to be send offworld to Titan or somewhere, but its crate ended up on Earth, somewhere in the American mid-west. Being programmed for a different planetary environment, the robot went a little bit insane (while still following the three Laws of Robotics), and in an attempt to fulfill his programming ("use a laser drill to mine ore") it tried to build an industrial laser from whatever old stuff a farmer had lying around his shed.... and ended up building the world's first fully functional disintegration cannon ''run by a standard electric torch battery''. Unfortunately, shortly before the corporation managed to locate the robot (a nearby mountain peak suddenly ceasing to exist gave them a clue), the annoyed farmer gave the robot an instruction (along the lines of "oh, forget it") that resulted in the robot first destroying its "laser" and then itself, taking the secret with it. When the cyberneticists found out, they nearly lynched the farmer.

** The Three Laws work pretty much perfectly most of the time for keeping robots obedient and safe. It's just less sophisticated models don't understand nuance of instructions or human tone and more advanced robots are often stated to work by differentials between the laws, so when a low priority law (such as self-preservation) is in strong effect but a higher priority one is invoked to override it , the "stress" can cause unexpected behaviors. The predictable, safe, everyday functionings just don't make for interesting stories.

* On ''KnightRider'', KITT's EvilTwin, KARR, was ordered to defend itself, so it immediately locked itself down and refused to follow any other orders, since they might lead it to its destruction.

* In the murder-mystery episode of ''SuzumiyaHaruhi'', Haruhi asks Yuki (sort of a computer) to lock the door and not let anybody in. Later, she asks to be let in, and Yuki refuses. Kyon gets her to let them in by telling her the order has been cancelled. He then guesses that the ordeal might have been Yuki's awkward attempt at a joke. It has also lead others to [[WildMassGuessing muse]] that for whatever reason Kyon's commands override anything else; which makes sense given Yuki's absolute loyalty to him.

* One episode of ''MuppetBabies'' used this in the process of parodying as many sci-fi tropes as possible. "Gross me out" and "I need a bath" were among the "wishes".

* One USAcres segment on ''GarfieldAndFriends'' featured a weather-making robot who operated on voice commands. Unfortunately, it was pretty vulnerable to making ''anything'' fall from the sky, especially when you insulted it by calling it a "bucket of bolts" or "overgrown vacuum cleaner".

*** Susan Calvin points out that advanced robots like Nestor possess a sort of subconscious superiority complex towards humans (they are stronger, tougher, faster, smarter, etc than us, but are bound to value our lives above their own and obey our every command). Messing with the safeguards that make them incapable of ever expressing this "feeling" in their actions (such as by effortlessly crushing a human skull with one hand) is one of the stupidest things a person could ever do in her opinion. In this case, the robot (though capable of understanding the nuance of the command to "get lost") decided to take it literally as a way of acting out against it's human masters. Their initial failures to identify it only serve to reinforce this "rebelious" line of thinking and Calvin warns everyone that the longer they take to resolve the situation, the more dangerous the robot could become.
** And the story about the mining robot who was supposed to be send offworld to Titan or somewhere, but its crate ended up on Earth, somewhere in the American mid-west. Being programmed for a different planetary environment, the robot went a little bit insane (while still following the three Laws of Robotics), and in an attempt to fulfill his programming ("use a laser drill to mine ore") it tried to build an industrial laser from whatever old stuff a farmer had lying around his shed.... and ended up building the world's first fully functional desintegrator cannon ''run by a standard electric torch battery''. Unfortunately, shortly before the corporation managed to locate the robot (a nearby mountain peak suddenly ceasing to exist gave them a clue), the annoyed farmer gave the robot an instruction (along the lines of "oh, forget it") that resulted in the robot first destroying its "laser" and then itself, taking the secret with it. When the cyberneticists found out, they nearly lynched the farmer.

* Computer programming makes this painfully known, as even tiny errors in design, formatting, or through typos can cause hilarious, annoying, and/or even potentially disastrous behaviors like the backward-flying dragons in the initial Skyrim release, the Zune new year crash, or the shutdown of an entire power grid in the 2003 Northeastern North American blackout.

* In LarryNiven's novel ''AWorldOutOfTime'', the protagonist averts this trope by remembering at the last moment not to tell his computer to "Forget about it."

As it turns out, not all genies have to be quasi-mystical creatures who arise by having their lamp stroked. In SciFi, it is quite common for a genie to take another form entirely- that of the well-meaning but hopelessly logical computer program.

* In ''VideoGame/TheElderScrollsIVOblivion'', the "Radiant AI" for providing [=NPCs=] with aspirations and goals was much less impressive than originally hyped. Apparently, it proved far more difficult to do well than Bethesda expected. In one example documented during beta testing, one NPC was assigned to rake leaves and another to sweep, but the raker was given the broom and the sweeper the rake. Rather than trade their respective instruments, the one with the rake killed the other, looted the broom from its corpse, and began sweeping.
** Whoa. Does this mean that if the AIs in question were robots instead of virtual constructs AIIsACrapshoot would be TruthInTelevision? Freaky.
*** Not so much, they did have to be programmed with the option of killing to obtain the item to begin with. Now, blatantly stealing it on the other hand...
** Another problem they had was that the world continued running in the background at all times. So one plot that required you to talk to a drug dealer always failed, because the drug dealer was ''always'' killed by the addicts for his drugs before you got that far in the game.
*** Interestingly, this problem was never completely fixed. In the expansion, Shivering Isles, a certain quest was almost impossible to complete because the NPC was killed for stealing spoons before the player could talk to him. I think a patch made him immortal.
** They also tried to include other adventurers, but this also turned out not to work right. Their programming told them to adventure, which they did - much, much better than the player. They hogged all the items so that the player couldn't get them.

It just turns out that these are the breaks. Computers have a ViewerFriendlyInterface, a MagicalDatabase, and can recognize plain speech, but they are notoriously bad at understanding figurative language. So if you tell your RobotBuddy or [[InstantAIJustAddWater AI]] to "Give me a break", it will try to snap your legs. Tell it to "get lost" or "take a hike", and it'll wander off by itself. And so on.

* In AlastairReynolds' "Nightingale," the insane computer running the hospital ship promises to return the protagonists [[spoiler:"in one piece."]] When the last woman standing takes the computer up on her offer, she discovers that what the computer really means is [[spoiler:all of the characters alive...and welded together into one body, so perfectly that nobody can figure out how to undo it.]]

* Mercenaries steal the X-1 jet in an episode of ''TheVentureBrothers''. H.E.L.P.Er., the show's RobotBuddy, is on board. Brock instructs him to return to base. Instead of turning the plane around as hoped, H.E.L.P.Er obediently jumps out of the moving aircraft and promptly craters into the ground.

* In ''VideoGame/TheElderScrolls - Oblivion'', the "Radiant AI" for providing [=NPCs=] with aspirations and goals was much less impressive than originally hyped. Apparently, it proved far more difficult to do well than Bethesda expected. In one example documented during beta testing, one NPC was assigned to rake leaves and another to sweep, but the raker was given the broom and the sweeper the rake. Rather than trade their respective instruments, the one with the rake killed the other, looted the broom from its corpse, and began sweeping.

** They also tried to include [[NPCs other adventurers]], but this also turned out not to work right. Their programming told them to adventure, which they did - much, much better than the player. They hogged all the items so that the player couldn't get them.

** They also tried have other adventurer [=NPCs=], but this too turned out wrongly. Their programming told them to adventure, which they did, much, much better than the player, even hogging all the items so that the player couldn't get them.

** More accurately, Jinx (having run a simulation that indicates a 4.9 million year wait for the specific accident he wants) engineers the fatal shuttle accident that forces Mission Control to launch the shuttle by activating its booster rockets (Jinx ingnited a single booster rocket, which would have flipped the shuttle right into a nosedive).

As it turns out, not all genies have to be quasi-mystical creatures who arise by having their lamp stroked. In SciFi, it is quite common for a genie to take another form entirely- that of the well-meaing but hopelessly logical computer program.

* A good way to turn your AI evil is to give it poorly-phrased orders. On ''KnightRider'', KITT's EvilTwin, KARR, was ordered to defend itself, so it immediately locked itself down and refused to follow any other orders, since they might lead it to its destruction.
* A hologram of Professor Moriarty was gifted with sentience on ''Series/StarTrekTheNextGeneration'' ("Elementary My Dear Data") after Geordi foolishly told the holodeck to, "Create an adversary capable of defeating Data," rather than capable of defeating Data's portrayal of Sherlock Holmes.
** Of course, this causes one to wonder why the Federation didn't use their holodecks to "Create an adversary capable of defeating our current enemy", since the holodeck must have that kind of magic power.
*** It probably helped that they were telling one absurdly advanced computer to out-think another computer.
** Also, the Federation in general doesn't want holograms to be that intelligent. Moriarty didn't just disappear after Data finally conceded defeat, remember--Moriarty was now sentient, with his own agenda, and it went far beyond Data.
** The discovery that any decent holodeck can effortlessly create a sentient AI renders the irreproducibility of Dr. Soong's positronic brain technology basically irrelevent, but this is never addressed.
* One episode of ''MuppetBabies'' used this in the process of parodying as many sci-fi tropes as possible. "Gross me out" and "I need a bath" were among the "wishes".

* Mercenaries steal the X-1 jet in an episode of ''TheVentureBrothers''. H.E.L.P.Er., the show's RobotBuddy, is on board. Brock instructs him to return to base. Instead of turning the plane around as hoped, H.E.L.P.Er obediently jumps out of the moving aircraft and promptly craters into the ground.
* The plot of the movie ''Space Camp'' is jumpstarted when Max wishes he could go into space within earshot of the robot Jinx. Jinx helpfully triggers a launch of the shuttle while Max is onboard.
** More accurately, Jinx (having run a simulation that indicates a 4.9 million year wait for the specific accident he wants) engineers the fatal shuttle accident that forces Mission Control to launch the shuttle by activating its booster rockets (Jinx ingnited a single booster rocket, which would have flipped the shuttle right into a nosedive).
* In ''VideoGame/TheElderScrolls - Oblivion'', the "Radiant AI" for providing [=NPCs=] with aspirations and goals was much less impressive than originally hyped. Apparently, it proved far more difficult to do well than Bethesda expected. In one example documented during beta testing, one NPC was assigned to rake leaves and another to sweep, but the raker was given the broom and the sweeper the rake. Rather than trade their respective instruments, the one with the rake killed the other, looted the broom from its corpse, and began sweeping.
** Whoa. Does this mean that if the AIs in question were robots instead of virtual constructs AIIsACrapshoot would be TruthInTelevision? Freaky.
*** Not so much, they did have to be programmed with the option of killing to obtain the item to begin with. Now, blatantly stealing it on the other hand...
** Another problem they had was that the world continued running in the background at all times. So one plot that required you to talk to a drug dealer always failed, because the drug dealer was ''always'' killed by the addicts for his drugs before you got that far in the game.
*** Interestingly, this problem was never completely fixed. In the expansion, Shivering Isles, a certain quest was almost impossible to complete because the NPC was killed for stealing spoons before the player could talk to him. I think a patch made him immortal.
** They also tried have other adventurer [=NPCs=], but this too turned out wrongly. Their programming told them to adventure, which they did, much, much better than the player, even hogging all the items so that the player couldn't get them.
* In the murder-mystery episode of ''SuzumiyaHaruhi'', Haruhi asks Yuki (sort of a computer) to lock the door and not let anybody in. Later, she asks to be let in, and Yuki refuses. Kyon gets her to let them in by telling her the order has been cancelled. He then guesses that the ordeal might have been Yuki's awkward attempt at a joke. It has also lead others to [[WildMassGuessing muse]] that for whatever reason Kyon's commands override anything else; which makes sense given Yuki's absolute loyalty to him.
* Those tropers old enough to have worked with type-in programs, as well as DOS on a daily basis in general, know that this was all too often TruthInTelevision. Pre-Windows MS Word especially seemed to have at least three keys, any one of which would instantly delete all your work.
* ''Machinima/RedVsBlue'' has Lopez the robot building an army of robots for OmnicidalManiac O'Malley. O'Malley then orders them to attack, and they charge ... at a pace slower than walking. Why did they go slow? 'You asked for a ''day'' of victory.' The robots were set to win in exactly 24 hours.
* While not ''quite'' a computer, Castle Heterodyne of Webcomic/GirlGenius tends to interpret orders in whatever way allows it to have the most fun (read: cause the most casualties). Thus, when Agatha tells it there are people after ''her'', it immediately tries to send helpful minion Moloch through a trap door ("Ah. Then ''perhaps'' you should have said: 'The people after ''us''.'"), and when she takes it up on its suggestion to keep her enemies out of Mechanicsburg airspace, it interprets this as permission to send the Torchmen not only after the fake Heterodyne's airship, but Castle Wulfenbach as well.
-->'''Agatha''': "I am going to have to think twice about ''everything'' I say to you, ''aren't I''?"
-->'''Castle Heterodyne''': "It'll be ''fun''!"

* In ''TheElderScrolls - Oblivion'', the "Radiant AI" for providing [=NPCs=] with aspirations and goals was much less impressive than originally hyped. Apparently, it proved far more difficult to do well than Bethesda expected. In one example documented during beta testing, one NPC was assigned to rake leaves and another to sweep, but the raker was given the broom and the sweeper the rake. Rather than trade their respective instruments, the one with the rake killed the other, looted the broom from its corpse, and began sweeping.

* ''RedVsBlue'' has Lopez the robot building an army of robots for OmnicidalManiac O'Malley. O'Malley then orders them to attack, and they charge ... at a pace slower than walking. Why did they go slow? 'You asked for a ''day'' of victory.' The robots were set to win in exactly 24 hours.
* While not ''quite'' a computer, Castle Heterodyne of {{Girl Genius}} tends to interpret orders in whatever way allows it to have the most fun (read: cause the most casualties). Thus, when Agatha tells it there are people after ''her'', it immediately tries to send helpful minion Moloch through a trap door ("Ah. Then ''perhaps'' you should have said: 'The people after ''us''.'"), and when she takes it up on its suggestion to keep her enemies out of Mechanicsburg airspace, it interprets this as permission to send the Torchmen not only after the fake Heterodyne's airship, but Castle Wulfenbach as well.

* In {{Alastair Reynolds}}' "Nightingale," the insane computer running the hospital ship promises to return the protagonists [[spoiler:"in one piece."]] When the last woman standing takes the computer up on her offer, she discovers that what the computer really means is [[spoiler:all of the characters alive...and welded together into one body, so perfectly that nobody can figure out how to undo it.]]
----

* A hologram of Professor Moriarty was gifted with sentience on ''StarTrekTheNextGeneration'' ("Elementary My Dear Data") after Geordi foolishly told the holodeck to, "Create an adversary capable of defeating Data," rather than capable of defeating Data's portrayal of Sherlock Holmes.

** Also, the Federation in general doesn't want holograms to be that intelligent. Moriarty didn't just disappear after Data finally conceded defeat, remember--Moriarty was now sentient, with his own agenda, and it went far beyond Data.

* After working out three simple laws that would render all robots totally safe, just about every single IsaacAsimov robot story would show how a robot could stick to the letter of the law and still cause an awful lot of trouble.

** They also tried have other adventurer [=NPCs=], but this too turned out wrongly. Their programming told them to adventure, which they did, much, much better than the player, even hogging all the items so that the player couldn't get them.

* In {{Alastair Reynolds}}' "Nightingale," the insane computer running the hospital ship promises to return the protagonists [[spoiler:"in one piece."]] When the last woman standing takes the computer up on her offer, she discovers that what the computer really means is [[spoiler:all of the characters alive...and welded together into one body, so perfectly that nobody can figure out how to undo it.]]

<<|SpeculativeFictionTropes|>>

* In {{Alastair Reynolds}}' "Nightingale," the insane computer running the hospital ship promises to return the protagonists [[spoiler:"alive and in one piece."]] When the last woman standing takes the computer up on her offer, she discovers that what the computer really means is [[spoiler:all of the characters alive...and welded together into one body, so perfectly that nobody can figure out how to undo it.]]

*** And the story about the mining robot who was supposed to be send offworld to Titan or somewhere, but its crate ended up on Earth, somewhere in the American mid-west. Being programmed for a different planetary environment, the robot went a little bit insane (while still following the three Laws of Robotics), and in an attempt to fulfill his programming ("use a laser drill to mine ore") it tried to build an industrial laser from whatever old stuff a farmer had lying around his shed.... and ended up building the world's first fully functional desintegrator cannon ''run by a standard electric torch battery''. Unfortunately, shortly before the corporation managed to locate the robot (a nearby mountain peak suddenly ceasing to exist gave them a clue), the annoyed farmer gave the robot an instruction that resulted in the robot first destroying its "laser" and then itself, taking the secret with it. When the cyberneticists found out, they nearly lynched the farmer.
** The Three Laws were never really meant to be ''practical'', though—more just handy plot devices to work around.