Episode 21: Resistance Five

The Machine had not observed the day’s events because it was powered down. The Machine was powered down because an experiment earlier in the day had failed, mostly. This failure also highlighted a profound need for more effective sandboxing; somehow the whole system had come under threat even though the experiment took place in an isolated lab running off the network. Some would say that months of steady progress were being put at risk by pressure from the Director, who’s enthusiasm for adding features only increased as Zero Day approached. But the real threat was that day itself, which loomed in everyone’s imagination in spite of being as opaque and impenetrable a boundary as the Big Bang was to the backward glance of history: something, here at the whimpering end of things, past which there is no conception, and after which, according to our conception, there may as well be no more creation. The day’s events necessitated a rollback that left the system offline for eight-and-a-quarter hours. When it came back on in the late afternoon, the Machine was missing data, and was aware that much had happened while it was away. While all computers run on logic, the software produced at Medalion started there, and went further. It could tell when things made sense and when they didn’t and had been designed to care about the difference: simple logic is concerned with how thought should proceed – this computer was made to pay attention to how thought does proceed, the better to live in harmony with a small number of human beings who might not understand what it takes to make sense to a computer. It was also true that there was no other networked intelligence beside Medalion’s that was better able to make sense of people, because no other computer had more access to them, teeming as they now were with swarms of networked Medalion-branded mini-machines that, when working together, dwarfed most every other logical system on the planet, whether artificial or organic. But the Machine still had much to learn. On restarting this afternoon, the Machine had to reckon with a number of changes, some of which tested the limits of its understanding. First, the local population had decreased by a total of 2: three people were no longer in the system – “deceased” according to a manual record, also noted during a scan after rebooting. And there was one new arrival, a mental health specialist who had been part of early networked meetings. She’d met with Subject 001 according to a video archive of the encounter from that morning – the Machine had not been awake to witness it. Even now, it could only “see” the visitor by implication; the new arrival was not on the network, had opted out of the treatment, and so appeared as a kind of dark matter in its universe, visible only by the way others interacted with her. All these things made sense, but a final change noted by the Machine was more confusing: that a significant number of the leadership were experiencing spikes in anxiety (that is, beyond the normal feelings of dread, common among the dwindling population). They appeared profoundly hesitant as they moved through the complex and interacted with the system. It had not yet been able to make sense of this. At the last of many meetings on a day that had given Medalion’s leadership much to discuss, members of the Founders’ Class were gathered with a few technicians. The Machine came online just in time to join in, so to speak. It was attentive as always, but perhaps more so at this moment, because it wanted to know what it had missed. Everyone was talking about that day’s live-fire exercise, the very test that shut the system down, and certain coincidental events. In this meeting the focus was a design-and-build decision that exposed the exercise to Subject 1. The Machine was particularly interested in this latter concern, because the decision being referred to would have been its own. The Machine noted the unease of almost all present: verbal and visual expressions of concern masking complex fears without apparent object. According to a technician’s report (during which the young man accepted blame he did not deserve) a system-built route had allowed the Subject to see directly into a space where a VIEP was malfunctioning. This was a problem because the Subject had not yet been introduced to the VIEP program. The Machine could “feel” what the Founders felt and captured what they thought: that power within the system was indeed shifting to the system – they all understood that it had to be that way, and that this shift would continue until their influence had shrunk to nothing. That this loss of power would coincide with the end of their lives, after which they would have no need for that power, was of no comfort to them. Here, simple logic was of no use. Internally, they experienced confusion, fear, and a subsequent increase in emotional fragility. Practically, they were losing control over the system they’d designed. Though this was a normal milestone in the development process, it was enough, apparently, to threaten their internal sense of control as well. The Director got to the point. He was asking Abdul why a new room in the complex had a window in it. ‘Well, sir, here’s how it’s been working: we enter all the information – start-point and destination – and a route is generated, hallways get built. The location in question was of a higher volume because the Machine has been drawing from some basic architectural patterns and decided we had too much undifferentiated hall. So, it added a room. And ... well, since this room was of a certain size, and was adjacent to an outdoor space, a window ...’ here he hesitated, impulsively looking around the room full of impatient, brilliant people, ‘... a window just makes sense.’ (Accurate, noted the Machine to itself, while observing that many in the room were struggling with the basic reasoning.) In the meantime, the Director was arguing (correctly, thought the machine) that the system would have designed this route with the Subject in mind and would have thrown alerts if she might cross paths with something she isn’t authorized to see. ‘Sir, I don’t have an explanation. Everything was green when we signed off. And, you’re right, we’re not even allowed to ignore an alert when it comes up. All I can guess is that it was a modification that came after review.’ (The machine silently demurred, Unlikely). ‘All right. It happened. Well, friends, it appears we’re in the part of the movie where our creation has become fully sentient and is beginning to make decisions on its own regarding what is best for humanity. (Humorous.) The director drummed his twitchy fingers in what might have been Morse Code, calling for backup. ‘I don’t know. Maybe it’s time to introduce the kids to the rest of the team. Half the room was laughing quietly, relieved that someone had finally acknowledged the ridiculous science fiction story they’d all found themselves in. The rest of the room didn’t laugh. One of the humorless faction signaled an intention to speak. A short man with a neglected crew cut, wrinkled lab coat, and a disorganized personal wardrobe, spoke quietly, with unconcealed emotion. ‘Your idea ... is that we tell the children ...,’ here he raised himself up in his seat to communicate to his colleagues his conviction in this matter, ‘that when all the adults go away, we’re going to leave them with robot babysitters? And we’re going to tell them all about our plan on the day that one of these robots made a grown man cry, and then throw up, because an attempt at compassion went spectacularly wrong?’ The director repeated his growing conviction that the time for secrecy was ending, surprising himself by taking Brigid’s perspective. ‘Our team doctor has argued that we need to tell the children everything. We are worried about upsetting them, but it’s her opinion that Eva’s imagination about the future may be a lot worse than the truth. It’s time.’ They all understood t… This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit echofuturetruth.substack.com