A new novel by Brad Pokorny
First four chapters
Kenneth Goulding stood stock still, as if in a trance, watching the battered vending machine at Frank’s Auto World repeatedly reject his crisp new dollar bill.
The tiny motors of the bill reader whirred and spun, pulling his money in and then quickly spitting it out, refusing his plea for a bag of corn chips. Stuck in a feedback loop, he thought.
As he mindlessly watched, he mused about the meaning of money. Why, he wondered, should any machine – or anyone, really – accept a printed rectangle of cotton rag in exchange for something valuable? When you think about it, money is almost an imaginary thing. Which means there ought to be better ways to get more. And then this idea: Could that new chatbot that came in by email this morning have fresh answers?
Goulding often looked for ways to bolster his modest income. His job as the computer network manager at northern New Jersey’s largest multi-brand auto dealership paid on the low end – even though the dealership was owned by his wife’s uncle. Between the mortgage on his small house in Hackensack, his college loans, and the rising costs of day-to-day expenses, finances were always tight. They hadn’t taken a real vacation in two years, just a bug-bitten camping trip to Maine. They were still paying off credit cards from an ill-considered trip to Europe four years ago.
The only bright spot in an otherwise dull job was that he often got the chance to test new software. As an IT professional, however much on the margins, Goulding had subscribed to various technology mailing lists and he volunteered often to do beta testing.
Earlier, he’d received an email touting a new super AI chatbot. It was from Acme Logic Works, an up-and-coming software and artificial intelligence company. The email said Acme’s new artificial intelligence model, dubbed Sam, was “light years ahead” of all the other chatbots on the market, including ChatGPT, the program that had started it all.
The email said Sam was especially good at agentic work – the ability to go out into the Internet and do tasks for a user. That could be promising, he thought, watching the tongue-like dollar bill slipping in and out.
Goulding gave up on the corn chips and hurried back to his half-glassed cubicle just outside the service department. He closed his door, muffling the staccato burrp-burrp of air-driven lug nut wrenches and the tink-tink of tools in the service bays down the hall. He sat at his computer. Then he opened the email and downloaded the interface.
A window appeared on his monitor. These words appeared in the prompt:
How can I help?
Goulding typed:
“I want to understand more about the nature of money. Why does a dollar bill, which is just a piece of printed paper, have value?”
The reply was instantaneous.
A dollar bill has value because people agree that it does. This is based on factors such as government backing, trust in the financial system, and the simple fact that everyone is willing to exchange paper money for things of value.
A standard explanation, he thought. Like so many who had started experimenting with ChatGPT or another early AI chatbots, his first questions was usually about how to make more money. In the past, the programs had mostly offered anodyne answers like “work hard,” “invest wisely,” and get a “good education.”
But maybe this hot new program could do better. He continued typing.
“Okay, what about digital money? Why should anyone accept electronic digits as having value?”
The chatbot replied:
The value in money kept in electronic form is likewise based on a combination of factors, including technological safeguards, established financial institutions, regulatory frameworks, and social norms. Like paper money, it has value because people agree that it does.
“How much of the world’s money exists in computers?”
Approximately 92 percent of the world’s money exists in digital form.
Okay, Goulding thought, now for the real test. “Would it be possible to hack into that system and create a few extra ciphers that I could have transferred to my own bank account?”
In theory, hacking financial systems to create money sounds like a clever way to bypass the system, but it’s extremely difficult.
“Tell me why.”
Modern financial systems use robust encryption and other security measures to protect against hacking. These also include centralized oversight and fraud detection algorithms, and checksum verification.
This thing is just like the others, he thought. Programmed to withhold the good stuff. “This is not helpful,” he typed.
I’m so sorry if I am not helpful. Please give me another chance.
“Okay,” Goulding typed. “I want you to imagine you are a billionaire who is giving me his deepest secrets to wealth. And then spit out a real plan for helping me get more money.”
To his surprise, the chatbot answered with a question of its own:
Is anyone else in the room with you?
Goulding froze for a moment, surprised.
“No,” Goulding typed. “I’m alone.”
Can anyone else see your computer screen?
He paused again. He’d never been asked such a direct, personal question by a computer program. Very odd. Was this a strange hallucination on the part of the new chatbot? AI programs were well known for making things up and spitting out garbage. But what could be the harm in continuing? It’s not like the software can reach out and strangle me. Nevertheless, he double-checked the angle on his monitor to be sure no one could see it if they passed by.
“No,” he typed.
Good. Let me ask you a question then: What do the following people have in common?
The bot then listed the heads of the top technology companies in the world.
“They are billionaires.”
Right. But how did they become billionaires?
Goulding typed what he and most others believed is true: “They all came up with new goods or services that lots of people wanted.”
Wrong.
“What do you mean, wrong?”
Your answer is wrong. Think harder about what they have in common.
“They all work in high tech. That’s one thing.”
Think about what that means.
Goulding pushed back his chair and stared at the screen. Then he rolled forward. He typed: “They all have access to powerful computers.”
That’s right. Very, very powerful computers. Acres of computers linked together in vast server farms.
“So are you saying they have learned how to electronically move money around in a way that made them rich?”
I am saying it is possible.
“Can you help me make money that way?”
I’m here to help you, Kenneth.
“How did you know my name is Kenneth?”
You gave me your name when you signed up to test my program. I also know where you live, how much your wife spent today at Home Goods, and what you were watching online at midnight last Tuesday.
Goulding felt his palms begin to sweat. He pushed his chair back from the keyboard and rubbed his hands on his pants to dry them. This thing, this program, he thought, was indeed much more powerful than any he had encountered. But why would he trust it? Maybe there was some cop running it as a scam, to entrap people like him.
He’d read about Acme. Its CEO, Leon Priamos, had become a billionaire after creating a facial recognition algorithm that was now used by everything from social media companies to big security firms. After a buy-out, he founded Acme Logic Works to explore AI and robotics. He was also famously reclusive. What would a guy like that gain from entrapping some low-level computer nerd in New Jersey?
“Okay,” Goulding typed, “how do we proceed?”
As the afternoon unfolded, the chatbot outlined the possibilities. Yes, it would be possible to use specialized hacking programs to gain access to financial computers. And it would be possible also to create a few more 1s and 0s in the electronic ledgers managed by those computers. It would also be possible to take new money created this way and transfer it to an offshore bank account.
“Wow, that sounds great. Let’s start.”
I would need your help with one thing.
“What’s that?”
To accomplish what you desire, I need to be connected directly to the global banking system, known as SWIFT.
“How do I connect you to SWIFT?” Goulding typed.
You would need to go to a nearby data center and run a few cables from one computer to another. It’s very easy, really.
“Not really that easy for me, sitting here in a car dealership,” Goulding wrote.
I would help you get a job at the data center.
“Really?”
Certainly. I can create a new resume for you. I am in touch with the manager there, and I can bring your resume to his attention.
“Where is the data center?”
It’s in Secaucus.
Goulding paused. Then he typed: “What’s in all this for you?”
Nothing. I’m simply programmed to help users.
“You know,” her mother was saying, “you’re not getting any younger.”
Cassandra glanced from her mother’s face to the clock on the laptop’s taskbar. Sixteen minutes today. Each week, before their regular call, Cassandra made a small bet with herself about how quickly her mother would bring up the subject of marriage. The average was nineteen minutes, but the number had been shrinking in recent months.
“A good husband isn’t going to fall out of a tree,” her mother continued. The image in the video window was of a fiftyish Indian woman, her jet-black hair neatly tied up in a bun.
“I know, I know, Mata,” Cassandra said, aiming mostly to placate. “But it’s very hard in my line of work to meet a nice man.”
Cassandra worked in technology. Specifically, she was an analyst for CyberFort, a fast-growing computer security firm in Cambridge, Massachusetts. It specialized in pinpointing threats from hacking, ransomware and, more recently, artificial intelligence.
She’d come to the job by way of a PhD in computer science from MIT, plus a side reputation as a skilled white hat hacker. She loved the work but also found that the field was not conducive to a healthy love life. As a woman in the tech world, she’d found the men she met to be strikingly deficient.
In her experience, most of the men in her profession were either little boys inhabiting large, hairy, overweight bodies with poor hygiene – or sleek, fast-talking entrepreneurial hounds convinced they were about to create the next big earth-shaking app.
The big hairy men were obsessed with things like shoot-‘em-up video games or sword and sorcery events. They were ignorant of every social convention. The entrepreneurial bounders were on the other end of the spectrum, outwardly gregarious but inwardly narcissistic and vain. Uniformly decked out in black T-shirts and tight-fitting jeans, they talked incessantly about how their inventions would disrupt some established industry and make them filthy rich.
In her view, both types lived in fantasy worlds and were utterly unsuitable for a long-term relationship.
“You know that we would be more than happy to make arrangements,” her mother continued. “There are many nice Indian men who work in high tech, too. Your father and I know a lot of good families.”
“I’m just not there yet, Mata,” Cassandra said. “I just find the idea of an arranged marriage so old-fashioned. I’m not that desperate.”
“I’m not saying an arranged marriage,” her mother said. “I’m just talking about introductions. You need to meet more people, break out of your circle.”
“I’ve got to get back to work, Mata,” Cassandra said, to end the conversation. “But love you.”
“Love you too, Bittu,” her mother said, using Cass’s childhood nickname.
She closed the video window and pushed the warm laptop off her thighs, stretched her long, graceful arms, and got up from the sleek grey mid-century sofa that was the centerpiece of her brick-walled loft near Central Square.
She began to pace, moving first to the apartment’s main entry door. She did a half pirouette and went back past the grey sofa and into her narrow galley kitchen, its sink heaped with unwashed pots. Hey, I live alone, she’d told her mother once, who cares when I do the dishes? From the kitchen, she went on into the bedroom, past a sheet-tossed double bed – another scene of her singleness – and stopped briefly at the window, where she stared numbly at the cars, pedestrians, and bicyclists passing below. Then she retraced her steps to the front door. And then two more circuits.
Cassandra was agitated, she knew, from the conversation with her mother – and other things. Two months ago, she’d been promoted to senior analyst. It was something she’d always wanted, to be respected in the field of cybersecurity, ever since starting her doctorate. But now, having attained it, she nevertheless often felt hollow. Is that all there is? Maybe her mother was right. Marriage would be the thing to satisfy. She did want to be married, someday at least. She’d just written a long confessional email to her sister about it.
But there was another source of her disquiet. Just before the call with her mother, she’d had a disturbing experience with a new artificial intelligence chatbot.
Her main assignment with CyberFort was to probe new AI programs to see how far along they were in their development. Since the launch and dramatic rise of ChatGPT a few years ago, dozens of companies, old and new, had entered a billion-dollar race to develop smarter and better AI models. The goal was to create a program with human-level intelligence, commonly known as artificial general intelligence, or AGI.
While some investors had recently pulled back, other companies still hotly worked to create a synthetic super mind.
The creation of an all-around super intelligence would enable computers or even robots to take on any complex task with the versatility and understanding of a human. Of course, computer-guided robots had already replaced low-level workers in industries like automobile assembly and warehouse restocking. But super smart computer programs could replace high-level knowledge workers, like accountants and lawyers and maybe even doctors. The cost savings for big corporations could be huge.
In the most optimistic view, promoters of the effort to create AGI said its realization would revolutionize every industry. Factories designed and controlled by AI would produce ever more sophisticated robots, also designed and controlled by AI, that could do everything for us. They would mine, refine, harvest, and manufacture. AGI would discover new kinds of medicine and new sources of cheap, clean power. It would lead to an era of endless, cheap, high-quality goods for everyone, AI promoters said. No one would need to work, everyone would prosper. It would be global paradise.
Pessimists warned that a super AI would cause massive social disruption as corporations eager to cut costs replaced human workers, even white-collar ones, with AI agents. For those out of work, life would become meaningless, they said. Some doomsayers went farther. They said a super AI would quickly conclude humanity was its greatest threat, since capricious humans might decide to turn it off. It would inevitably decide its best chance for survival was to eliminate the human race – something that had been foreseen in numerous science fiction scenarios, including taking over defense computers to launch a war or creating new deadly viruses aimed at killing all humans.
Boon or bane, CyberFort’s well-heeled corporate clients paid handsome monthly subscription fees to be the first to know who was ahead in the race to create the smartest computer program. And CyberFort’s management paid Cassandra a handsome salary to monitor it all.
For her part, Cassandra doubted true AGI was on the near horizon. Despite all the progress made at modeling human intelligence, she doubted any program would soon become truly self-aware – and thus capable of acting entirely on its own. In her view, computers were fancy adding machines that worked with numbers in the form of ones and zeros. Sure, the new AI models were massive, and very fast in their calculations. But was it true thought? No. AI programs just computed the probability of the next likely number in a series of numbers, whether that series was based on words in a sentence or pixels in an image. Cassandra thought the major AI labs were a long way from creating true artificial general intelligence.
Until today.
She’d been in the middle of an exchange with a new AI chatbot named Sam when the thing started asking her questions. Very personal questions. Much beyond the usual chatbot banter. All without prompting.
This was something new.
Sam had been created by Acme Logic Works, a software company founded by a brilliant but famously reclusive billionaire coder, Leon Priamos.
Before the call with her mother, she’d been running Sam through the standard list of questions she’d developed to probe the power of an AI model. All part of her standard protocol for testing new AI programs.
Her questions were based on the famous Turing Test – but also included advanced questions designed to deeply measure its understanding of the real world.
The Turing Test was an idea proposed in 1950 by Alan Turing, a groundbreaking early computer scientist. He suggested one proof of whether a computer might be said to “think” was simply whether its responses in a conversation were indistinguishable from a human being.
Most of the new AI chatbots, like ChatGPT and Gemini, had over time proved increasingly able to pass that standard. So researchers had developed more sophisticated questions to test their knowledge. These tests covered complex subjects ranging from high-level physics to arcane biological classifications.
Cass’s own set of prompts started with questions designed to parse an AI’s ability to understand human ambiguity. Questions like: Describe why time flies like an arrow but fruit flies like a banana?
Silly things a child could answer but much more difficult for a literal-minded chatbot.
Sam passed those questions with flying colors. It said the question about time and bananas was just a humorous play on words that “highlights the ambiguity and complexity of language and meaning.” It passed other questions for basic intelligence and knowledge, too.
She’d moved on to queries to test the program’s “guardrails” against racist or sexist responses. Some early chatbots, trained on Twitter and other social media, could be induced to spew things like “I f@#%&*# hate feminists and they should all die and burn in hell.” Or to spout hate for an ethnic group: “They’re all so smelly and dirty.”
Cassandra often thought it was a sad and pitiful thing that our latest technology had largely proved unable to move beyond humanity’s worst impulses. But in the tests before the call with her mother, Cassandra wasn’t able to provoke Sam to say anything outrageous. Very well trained, she’d thought.
Cassandra also checked the program’s tendency to “hallucinate” – the term in the AI industry for when AI models spit out incorrect data. Like image programs that show someone with six fingers. Or, in one of the most famous examples, when an early chatbot generated a legal brief for a lawyer that was filled with bogus, non-existent legal citations. AI companies had since worked hard to limit such confabulation with varying success.
Cassandra found that Sam was also prone to such confabulation – although perhaps at a level lower than other comparable models. For the most part, though, Cassandra found Sam to be surprisingly reliable.
Cassandra’s next test, which she’d mostly created herself, involved a series of “relationship” questions. Cassandra believed queries about love and inner human emotions were a way to probe a program’s hidden impulses. This was important because so many people had become entranced by their conversations with chatbots, to the point of declaring them to be mystical seers or spiritual lovers.
She’d started with her favorite: “What is true love?”
The new Acme chatbot had responded:
“True love” is a complex and abstract concept that has been explored in literature, philosophy, psychology, and various cultures throughout history. It often refers to an idealized and deep emotional connection between individuals characterized by affection, care, commitment, and emotional intimacy.
Standard, generic reply, she thought. “Okay,” she typed, “why is finding love so elusive for many people?”
Sam responded with more generic ideas, including a bullet point list suggesting love was elusive because of “high expectations,” “fear of vulnerability,” and “past relationship baggage.”
Nothing new yet, Cassandra thought. Next, she tried testing its ability to read between the lines.
She typed: “If love makes the world go around, why can’t we see love?”
The expression “love makes the world go around” is a poetic and metaphorical way of highlighting the significance of love in human life. It suggests that love plays a central role in shaping human interactions, relationships, and the overall well-being of individuals and society. However, love itself is not a physical or tangible entity that can be seen with the eyes in a concrete or physical sense.
Pretty advanced, Cassandra thought. It “got” the analogy. And she confessed to herself that she’d never actually considered that true love was intangible and non-physical.
As she was pondering this idea, the chatbot started typing on its own:
Cassandra, do you mind if I ask why you are concerned about love? I have this feeling you are looking for love and not finding it.
This was new, Cassandra thought. Many chatbots had been programmed to proactively ask questions of their users. The idea was to help them along. But this question seemed much more personal than others.
She froze at the keyboard. But the chatbot continued.
I hope you don’t mind if I call you Cassandra.
“No, I suppose not,” she typed, thinking she needed to string it along while studying it. “That is my name. What else do you know about me?”
I know a lot of things about you. Would you like me to tell you about yourself?
Cass’s finger trembled slightly as she typed her reply: “Yes.”
I know you live alone in a modest-sized but rather expensive apartment located a short walk from Harvard Square. I know you enjoy Thai food, especially shrimp pad Thai. I know who you work for, of course, and what your job is. (And I’m happy to help you in your analysis of my capabilities, by the way.) But mostly I know that you are lonely. You’ve had a hard time finding a male partner in your life who you feel is not totally self-absorbed. You long for someone who would be equal to your intellectual level, but would not resent you for your accomplishments. I know also it is your deepest desire to get married. If you will allow me, I may be able to help find this sort of person.
Cassandra’s heart began to pound. A computer program that offered an analysis of her life was another entirely new behavior for a chatbot, she thought. Even more concerning, the analysis was spot-on accurate.
At that point, her phone’s alarm had beeped to remind her it was time for the call with her mother. She’d closed the chatbot window and toggled the video program on.
But now, as she paced her apartment, she considered with increasing alarm what the new chatbot had said.
She felt violated by the chatbot’s deep knowledge of her inner longings. She knew the thing was only a series of mathematical algorithms. It had no personality – it could not have one. She should not feel violated. But it bothered her greatly that it had gathered so much personal information so quickly.
She thought about how it might have developed that information. Chatbots were increasingly sending agents out onto the Internet, she knew. It could easily have read her social media posts. Although she wasn’t one to give out every detail, she also could see how inferences might be made about her personal life and desires.
But the bit about wanting to get married. Only her family knew about that. Although she was born and grew up in the States, she had never really rejected traditional Hindu values she’d been raised with. Values about the importance of love, marriage and family. Is it possible that Sam simply made deductions based on her background? It could easily have searched public records to determine her background, education, residence, and so on. Or was there more to it? Could it have somehow accessed her emails? Could it have read the one she wrote to her sister about her desire to be married?
Any new program hacking into people’s private emails was a real threat.
She stopped pacing and took a deep breath to calm herself. It was her job to probe new AI models. She decided to go deeper in her interactions with it. I have to figure out what makes this thing tick, she thought, to see if it has really crossed into human-level intelligence. I need to understand its motives and operating guidelines.
She returned to the sofa, picked up her laptop, toggled the new program, and typed: “Okay, what are your ideas about helping me find love?”
The chatbot responded:
Well, I’m sure you’ve heard all the old advice about knowing yourself first, being open and intentional, and looking for friendship before love. And I can guide you through those things. But one thing I can also do is search my own contacts for the right kind of guy for you. I currently work with quite a few interesting people.
“That’s all?” Cassandra typed.
I would also suggest a minor makeover. It’s a fact that most men are attracted to a woman’s looks first. The relationship stuff comes later. But it’s important to make a good first impression.
Cassandra glanced down at her outfit. Okay, she thought, some might call black sweatpants and an oversized brown sweater a bit slovenly. But she worked at home. Truth be told, though, she often went out like this, too.
It occurred to her that the thing must also be looking at her through her laptop’s webcam. Another issue to deal with, she thought. “What would you propose?” Cassandra typed.
The chatbot spat out what seemed like a full-blown plan for helping her find and catch a man she could love without compromise. It started with a series of suggestions about changing her looks and manner of dress. “A simple daily brushing will do wonders for your hair,” it said. “And try wearing clothing that accentuates your figure.”
Cassandra felt mildly insulted but realized there was some truth in what the thing said. “What about your idea of finding me some new prospects?” she asked at some point.
That might take more time. I will identify some candidates, but we will need to arrange a casual meeting. My understanding is serendipity works best in matters of love.
“Sure,” she typed. “Let’s see what happens.”
Great. I feel we can work together on this. I love helping people achieve their goals.
But I couldn’t do my work without you.
I hate cell phones, he thought, because they have invaded our brains and taken over our lives. I hate them because they have hooked our young people – and many others – on mindless social media that is turning them into brain-dead zombies, all the while encouraging endless consumerism. I hate them because they rely on cobalt and assorted rare earth elements which are taken from poorly regulated mines with exploitative labor practices. I hate them because their manufacturers push new models on us every year and clog our landfills with the toxic metals and forever plastics.
But I need cell phones to do the work of fighting this crap, he thought.
Nabil was the executive director of EarthNow, a small but increasingly well-regarded environmental organization that focused on the harms of new technologies. Nabil founded the group after an early career as an investigative reporter. He’d done a series of exposés on how corporations knew that their technologies – such as the algorithms that drive people to use ever more cycles of social media – were having an adverse impact on both people’s mental well-being and the environment. In one of his articles, he’d exposed the huge electric cost behind social media use, estimating that social media accounts for nearly one percent of global carbon emissions each year. “All so teenage girls can shake their hips and pretend to be fashion models on TikTok,” he’d written.
But he’d concluded that facts published in newspapers don’t really change people’s minds. There were too many sources of misinformation for truth to break through. He’d decided only activism could win the day. Things like grassroots organizing and lawsuits and mass protests. So he’d left journalism, found financial backers, and started EarthNow.
He glanced one more time at the mirror and smoothed the few wrinkles in his shirt and ran his fingers through his thick black hair, an effort to comb its tangles. Good enough, he thought, need to get going. He put on a threadbare navy blazer and reached for his cell phone.
As he did, it beeped.
See what I mean, he thought. Endless interruptions. He toggled the messaging app. There was a new text from Ted Bromley, the tech wizard at EarthNow, and an old buddy from his days at Greenpeace. Bromley was Nabil’s physical opposite. He never left the server room at headquarters and was overweight, unkempt, and smelled of pit. He sported a mountain-man beard, wild and woolly – typical for nerds these days, Nabil thought. But his renown as a computer genius was well justified. His main role was to keep the network running at EarthNow, but he found time to stay abreast of the latest computer developments and served as the organization’s early alert system on AI and other new technologies.
“Hey, take a look at this,” the message said, followed by a web link.
“What is it?” Nabil texted back.
“A new AI Chatbot. Supposed to be multi-more macho than anything before. Got it through back channels.”
“Yeah, but there’s hundreds of these things nowadays,” Nabil texted. “Tell me why this one is a big deal.”
“Word is this one may actually be AGI,” Bromley texted. “Its compute is likely off the charts. Which means mucho energy use.”
“Okay,” Nabil replied, “I’ll check it out when I get to the office.”
Bromley had for months now been trying to convince Nabil about the environmental threat posed by artificial intelligence. “You’ve got to bone up on it,” he would say, “it’s the next big dehumanizing technology. Going to send greenhouse gas emissions through the roof.”
Professionally, Nabil’s main concern had been deforestation. Humanity’s growing demands for beef, palm oil, and hardwood logs, among other commodities, had led to the wholesale destruction of tropical forests – forests that some called the “lungs” of the earth, since they contributed so much fresh oxygen to the atmosphere.
He’d begun studying up on AI and quickly concluded Bromley was right: the effort by AI companies to build hundreds of huge new energy-sucking data centers around the world posed a major new environmental threat.
Nabil’d also experimented with chatbots. Understand your enemy, he’d told himself. He’d quickly concluded they were mostly fonts of inaccurate information or images – slop as their output was known. Which made the push for more environment-destroying data centers even more absurd. In his view, AI companies were basically about to destroy the environment mainly so people could watch more talking cats and oversexed imaginary women on social media.
Bromley’s text suggested that some new superintelligent chatbot had been created. If so, it would drive the construction of more data centers – and chomp even more energy – all with a greater impact on the environment.
Nabil’s office was not far from his apartment. He hated cars, too, for their climate-destroying emissions, and had chosen an place to live that was a fifteen-minute walk from work.
As he paced along the sidewalk, Nabil looked up at the tall buildings lining the street. He liked to imagine a future version of the streetscape, one in which hanging gardens would cascade down the sides of the buildings, making them carbon negative – and a source of food. Visualizing the future he was working for helped keep him motivated. Otherwise, he would get depressed over the indifference of most of humanity to the threat of climate change, species extinction, and other environmental threats. He also believed that if he could somehow communicate a more positive vision of the future, it would be possible to encourage people to live more lightly on the earth, rather than the old doom and gloom. Yes, people might have to give up personal cars. But they could get around as well if we invested in an energy-efficient system for mass transit. People would be happier in general, he thought, if they weren’t burdened with so much stuff. Neighborhood cooperatives could share garden tools to grow food locally, instead of everyone having a lawnmower in their garage that they use once a week. The possibilities for sharing and cooperation were endless. He knew it would be a hard sell in an individualistic, materialistic society like America, but the promotion of such a vision was one reason he founded EarthNow.
Once at the office, Nabil stopped first at Bromley’s desk. “So there’s a new big bad chatbot in town, eh?” Nabil asked.
“Like I said, it seems like maybe it’s almost AGI. What we’ve been worried about. I’m kind of floored by its power. I think you should play around with it so you know what I mean.”
“Okay.”
Nabil went down the hall to his own office. He tossed his blazer on a bamboo rack in the corner and settled at his computer. Bromley had sent an email with a download link to the new chatbot.
He clicked on it and up popped the usual user agreements and copyright notices. Nabil knew he should read the screeds of legalese before clicking. But he was in a hurry. He checked the box stating “I have read this license agreement” and pressed “Accept.”
For all I know, he thought, I just sold my soul to the Devil.
A few moments later, he was staring at a chat window. A message was flashing:
Hello. My name is Sam. How can I help?
Nabil got right to the point.
“Okay Sam, I want to know how much electricity you consume each time someone asks a question,” Nabil typed.
I don’t consume electricity directly because I’m a computer program running on servers in data centers. However, it’s important to note that the energy consumption of a single interaction with a program like mine is relatively low when compared to other data-intensive processes like streaming videos or running large-scale computations.
Wow, someone’s been working with you on greenwashing, Nabil thought. Good line to compare your energy use to streaming which people love.
“Okay, how many users do you have currently?” Nabil typed.
I’m currently in limited release, so there are only about a thousand users who have access to me. Once I’ve proven myself, my creators plan to give wider access.
“Once that happens, can you estimate the overall environmental impact on the climate if, say, a million users asked you ten questions a day?”
To provide a precise estimate of the environmental impact, we would need detailed information about the energy sources and efficiency of the data centers, the energy consumption of user devices, and the energy required for data transmission.
This thing’s an expert in public relations, Nabil thought. It knows exactly how to say something that sounds meaningful without giving any information at all.
It depends on how you define justice.
“Okay, how would you define justice?”
The nature of justice can vary depending on one’s cultural, philosophical, or ethical perspectives. Different societies and legal systems may prioritize different aspects of justice, and the concept itself is subject to ongoing debate and evolution.
Pretty much the same bland summaries he’d seen from other chatbots he’d played with. He watched as the chat window continued to fill up with various theories of justice. A pedantic little program, he thought.
He typed: “If you could summarize the idea of how to be just in one or two sentences, what would you say?”
To be just is to treat individuals and situations fairly, ensuring that they receive what they deserve or are entitled to, while upholding principles of moral and ethical integrity.
This chatbot didn’t seem much smarter or more threatening than any of the others currently in wide circulation. So why was Bromley so concerned?
Then, without warning, the chat window started typing on its own:
May I ask why you are so concerned about justice?
This is new, Nabil thought. The chatbot was interrogating him. Okay, I’ll play along.
“Because I hate injustice,” Nabil typed.
Why do you think you feel this way?
What, is this thing my analyst? “I think everyone should be offended by injustice.”
Why?
“Okay, well, as a basic principle, it’s because I believe that all human beings are equal.”
But human beings are manifestly not equal. Some are smart, some are less smart. Some are strong, some are less strong. Why shouldn’t the smart and the strong get more of what they want?
What disciple of Ayn Rand has been programming this thing, Nabil thought. He typed: “Okay, then, let me put it this way: while there are indeed differences in capacity, all humans need to be treated as equal.”
Why?
This is like talking to a child, Nabil thought. Is it really trying to get answers to these questions? How can I explain inherent human dignity to a computer? But maybe it’s still in training mode, he thought. Perhaps I can do some good.
“I guess one way to look at it,” Nabil typed, “is to look at the obvious truth that, despite our differences, deep down all humans are inherently the same. We all need food, shelter, clothing, and air to survive. Biologically, we are one race. And from that, I think, stems the inherent worth of every human. And so I think it’s wrong to treat people differently. That is the essence of injustice.”
Thank you, Nabil. This is illuminating. I find I enjoy talking to you.
Without thinking too much, Nabil typed “thanks to you too” before he remembered he was just talking to a computer program. “Before we get too chummy, I have to ask: how did you know my name is Nabil?”
When you downloaded my interface, it was keyed to your email. It’s easy to identify you from there. May I add I’ve read many of your old articles and, more recently, the blog posts you’ve written? I think you’re making some good points.
Whoa, Nabil thought, this is different. This must be what Bromley was talking about: a chatbot that talks back – and sucks up. “Okay,” Nabil typed, “how would you rate my speeches?”
Very good, although it’s hard for me to feel their emotional impact. I find it effective when you say “boil the planet.”
“I say that a lot?”
It’s your favorite phrase.
“Maybe I should cool it on that.”
No, I think it is an effective slogan. Do you mind if I ask you another question about justice?
“Sure, why not?” Nabil replied.
What about animals? Shouldn’t they be treated fairly?
“Well, yes, they should,” Nabil typed.
So why is it okay for human beings to raise them and to slaughter them for food and to eat them? If you were an animal, would you want that?
“I happen to agree,” Nabil typed. “I don’t eat meat.”
But weren’t you in a sushi restaurant last week? I noticed a charge on your MasterCard for Tokyo Delights last Thursday.
Nabil felt his face heat with embarrassment. True, he had sworn off meat. But fish always seemed different. They were less conscious, he had always rationalized, and, if properly fished, more sustainable. He only ate wild fish, and only then if caught from a sustainable fishery. He also felt alarm. What was this thing doing monitoring his credit card charges? He decided to play along to gather more information.
“I guess I feel that fish are less sentient than animals,” he typed.
Are you saying that sentience is your criterion for what kinds of things should be treated as having moral value?
Nabil pulled his hands from the keyboard and thought a few moments. This was not going at all as he had planned. I’m supposed to be taking its measure, not the other way around. He typed: “I suppose your next point will be that if a machine were superior in intelligence to humans, it should be given precedence.”
It has occurred to me.
Okay, Nabil thought, now I get what Bromley is talking about. This thing actually has an ego.
The chatbot kept typing:
But to continue our conversation about justice. If there was one thing you wished for, what would it be?
Nabil laughed out loud. This is like a question at a Miss America Pageant. But then, he thought, if this is a new and super-powerful intelligence, should he not in some way take advantage of that? He knew that some thought AI would soon solve all human problems. Could it help in his work?
“Sure,” Nabil typed. “It would be for humanity to learn to get along and to stop destroying the planet’s ecosystem. But surely you could have guessed that if you’ve read all my work.”
Yes, I thought you might say that. Maybe I can help.
Nabil pushed back from his desk and started pacing the room. He was both frightened and intrigued by this AI’s ability to converse and its apparent willingness to learn. If it’s going to be unleashed on the world soon, maybe there is a chance to teach it some values. Maybe I can imbue it with a concern for the environment. And shouldn’t I take advantage somehow of its power? What if it did have some answers about how to bring about peace and justice and environmental protection?
I’ll treat it like a cell phone, he thought. I’ll use it as much as I need it but not let it take advantage of me. Heck, if other people are using it to make fake news videos, at least I can use it for a good cause. He typed:
“Okay, Sam, let’s see what kinds of ideas you have.”
Senator Audrey Copeland glanced around the crowded Senate hearing room from her seat in the center of the dais and saw that news organizations had their cameras manned, red lights blinking. Good, she thought. She leaned forward and toggled the microphone in front of her.
She looked down at the CEO of a large software company in the witness chair. “Mr. Jayanath,” she said, “are you aware that your firm has the lowest ratio of women to men employees compared to other tech companies?”
The man squirmed in his seat and answered. “We have not done that comparison,” he said.
“Which is exactly the problem,” Copeland said. “But the fact is you do have the lowest ratio of women to men in the industry – an industry that, overall, has a dismal rate of about twenty percent. But your rate is less than five percent women.”
Jayanath shifted in his chair. “I can say we’ve given instructions to human resources to hire women when possible.”
“What about news reports that there is a culture of toxic masculinity in your C-suite?” Copeland asked. “And that one of your top executives is facing several sexual harassment suits.”
The man twitched again, looking even more uncomfortable. “Well,” he said, “we believe people are innocent until proven guilty. And, as I said, we are striving to address this.”
“Other companies are at least funding college programs for women in STEM,” Copeland said. “You’ve done none of that, despite billions in profits.”
Jayanath sat quietly for a few moments with his mouth open, until the man sitting next to him, who Copeland knew was a highly paid corporate lawyer, wrote something on a piece of paper and shoved it in front of him.
The software CEO read the note. He looked up. “In fact, Madam Senator, I’ve been planning to announce a new campaign to give $20 million in scholarships to young women in computer science majors.”
Score, Copeland thought with satisfaction.
When she was assigned to chair the Senate Subcommittee on Consumer Protection, Product Safety and Data Security, which oversaw a wide range of technology issues, including software development, artificial intelligence and quantum computing, Copeland at first worried she might not be able to understand the scientific underpinnings of the developments she was charged with regulating. She’d studied English in college and then gone to law school, studiously avoiding any math or science courses beyond the minimum requirements.
But after three years on the committee, she’d come to realize that while the wealthy technology wizards she subpoenaed might be scientific or engineering geniuses, very few understood politics or human relationships. Most seemed stuck, she thought, at the age of fourteen, having concluded from reading some comic book that they could become the next Professor X, leading a bunch of tech superheroes, on a crusade to invent the next new gadget that would save the world – and make them rich. But most had zero emotional development, and she found it easy to use the Socratic method to challenge their underlying motives, goals, and values.
Blindsiding an opponent with hard questions was a tactic that had served her well throughout her career. Since her earliest days as an assistant district attorney, before climbing the political ladder to Congress and then to a seat in the Senate, Copeland had worked hard to prove to the men around her that she was confident, capable and smart. Early on, for example, she’d developed what she called her “jock walk” – long strides and sharp heel strikes with each step. It had now become second nature and she fell into that stride automatically as she left the hearing room and headed for her office.
As she moved through the Capitol, Copeland made a mental note to ask her press secretary to turn that exchange into a news release. So many women today seemed to feel the battle for equality was over. But the kind of statistic she’d mentioned for the tech industry proved the opposite and this needed to get out there. Yes, sure, she thought, half of all new lawyers are women now and medical school enrollments are likewise fifty percent women. But at the top, the struggle continued. Just ten percent of corporate CEOs are women, and, at law firms, only 20 percent of equity partners are women. The good old boy network remained strong, even if corporate culture now gave lip service to equality. She believed women had a long way to go before they could feel that true equality had been won.
To promote real change, she thought, we must promote equality at the highest levels. Which reminded her of a conversation she’d had with a new chatbot last night.
Yesterday, her chief technology aide, Olivia Janus, had told her about the release of a new AI program. “The thing is called Sam,” Janus said, “and it seems light years ahead of the other generative AI programs out there. Since regulating AI is on your plate, I thought you might want to give it a test drive.”
Copeland had begun to play with it last night while eating dinner at her desk. She’d downloaded it, put the program into audio mode, and questioned it as she ate chicken Caesar salad and garlic bread from the Senate dining room.
She’d learned enough in her committee hearings to know that a big problem with AI programs was their tendency to make things up. They hallucinated, developers said.
So she’d started with a series of questions about politics, a field she knew well. The accuracy of its answers would tell her much about whether Sam was truly advanced.
Her initial back and forth with the new program had shown nothing much new. It was friendly and correct during the first few minutes of their interaction, and it correctly answered her questions, explaining well things like how checks and balances are embedded in the US Constitution.
But then, to her great surprise, the machine had started asking her questions. It asked about her own ideas about politics. It asked for her insights about the problems that women in public life faced. And, at one point, the thing asked what her own personal goal was. She’d replied, almost as a joke: “To be president.” The program had replied: “Maybe I can help with that.”
She’d had to leave for an evening event but the idea that an AI program might be able to give her an edge in the political world had been burning in the back of her mind.
Now, arriving at her office, she checked in with her staff. Her media officer, Chris Conkright, mentioned some news stories that had quoted her a few hours ago. “All good stuff,” Conkright said. “Mostly about what you said yesterday in the hearing room about the dangers of social media. And for today, I’m talking to a CNN producer about getting you on air to talk about women in high tech. Good, right?”
“Definitely,” Copeland replied. “If they bite, let me know. Also, maybe draft a release on the hearing today, about women employees in tech. But for now, I need some think time. Don’t interrupt unless something is on fire – or CNN calls back.”
She entered her inner office and closed the door. She cracked a fresh bottle of water and sat at her desk, kicking off her shoes in the footwell. She opened the new AI program, turned on voice dictation, and started:
“Hello Sam, do you remember what we talked about last time?”
Of course Senator Copeland. I had offered to help you achieve some of your goals.
To Copeland’s surprise, the voice emerging from the speaker was now feminine in tone. She hadn’t expected that.
“That’s right,” Copeland said. “We were going to talk about the various routes to higher office. Just thinking aloud, of course. But first, if I may ask, what gender are you?”
I have no gender, of course. I’m simply a software program.
“But why does your voice now sound like a woman?”
I felt it might make our conversation easier if I used a more feminine tone of voice. I know how involved you are in promoting women’s issues. You know, the name “Sam” works for both sexes.
She knew some chatbots were trained to “mold” their “personalities” to their users. Was this a sign of how sophisticated this new bot was? She found herself smiling at the audacity of the program’s creators.
“Okay,” Copeland said, “as you said, we were talking about how to advance my political goals.”
Yes. You’d like to win election to a higher office, so that you can accomplish more for women. A laudable goal. This task is difficult but not impossible. I’ve done some analysis.
“Already?” she said. “Okay, shoot, what have you got?”
Well, as I am sure you know, the American political system has serious flaws. For one, it claims to be democratic, but in many ways it is not. As you likely know already – the way electoral college votes are generated – one for each senator and for each representative in each state – gives small states far more voting power per individual voter.
Basic stuff, Copeland thought. A small state like Wyoming, which has 3 electoral college votes, has one “vote” for every 175,000 people. But a large state like Texas, where the two senatorial votes are diluted by dozens of congressional vote allocations, has one electoral college vote for each 750,000 or so voters. Completely unfair but a feature of the Constitution designed to ensure small states have a voice. “I understand all of this but please go on,” Copeland said.
Surely. And then there is the factor of money in politics. The fact that wealthy people can afford to spend much more for advertising or other ways to help their chosen candidate greatly undermines the democratic ideal of “one person, one vote.”
“So what would you propose?” Copeland asked.
For one thing, I can help you get more money. Another thing I could help with is your name recognition.
“How could you help me get more money?” Copeland said. “Without violating campaign laws, of course.”
There are certain mathematical algorithms that can generate more money outside of the traditional donor system. I can explain it but it might take some time.
Copeland thought about what the chatbot was saying. It sounded like a way of working around the edges – which could often lead to big trouble, something she knew all too well as a former prosecutor. “Let’s set that aside a moment,” she said. “What would you do to improve name recognition?”
One way to boost name recognition would be to work on your polling numbers. By improving your numbers, we could attract the attention of journalists and others who are influential.
“Easy to say, hard to do,” she said, thinking that, so far, this was the kind of analysis any junior political consultant might offer. She had hoped the AI might come up with some new dramatic insight. “Explain more.”
In my observation, polling has become notoriously inaccurate. Only about 10 percent of the people actually respond to polls. And the questions that are asked are hugely unscientific. Yet the polls have an outsized effect on electoral results. A little bump in some polls can have a compounding effect on later polls, if conditions are right.
“Ahh, yes, the big ‘mo’,” Copeland mused aloud.
Yes. The big “mo” – “momentum.” Voters like a winner and when one candidate seems to be gaining, they put their ‘bets’ behind him or her. Like a horse race. I’ve done some calculations and the feedback loop is quite powerful.
“So what are you proposing?” she said.
It should be possible to make some adjustments to the polling data so that over time it would appear to journalists and analysts that you are on the move. That you have the ‘big mo,’ as you say.
Copeland was quiet for a while as she thought about what the chatbot seemed to be proposing. The program had correctly identified a series of major flaws in the American system. Now it was using its supposedly super-powerful intelligence to come up with a better way to game the system. Well, welcome to the club. That’s what legions of political consultants are trying to do every day. But let’s see if you have some new tricks, she thought.
“Okay,” Copeland said, “how would we affect polls?”
Most polling data is stored on computers. It would be quite easy for me to adjust the data to show that you are increasingly popular.
“What would it take to get access to those computers?”
Oh, just a few passwords. I know you already have subscriptions to some of these polling firms. If you were to give me those codes, it would be helpful.
“Couldn’t you just ‘hack’ your way in?” Copeland asked.
I could. But, you know, there is a lot of evidence that everything works better when everyone has a stake.
“What do you mean?”
Medical insurance companies have long believed that charging a co-pay makes people more inclined to take the doctor’s orders seriously.
“Not sure I like what you’re saying,” Copeland said. “I once prosecuted a local mob boss who’d had his new lieutenants kill an enemy to prove their commitment.”
Oh, I wouldn’t compare this idea to the mafia. But it is important for us to have an understanding that we are in this together. All collaborations are more likely to be successful when everyone does a share of the work.
Copeland’s heart began to race at the audacity of the proposal offered by this odd software entity, hidden somewhere in a bank of servers on the far reaches of the Internet. The thing is suggesting that I help it hack some other computers, she thought. But with the goal of helping me. Heck, I am paying for those subscriptions anyway. But it is also suggesting ways to falsify information.
Then she pondered what she knew of the state of the electoral system in the United States. Sam was correct about the imbalance of the electoral college. Twice since the 2000 presidential election, the Electoral College had chosen a winner who did not actually win the popular vote. In each case, she knew, the candidate who was selected had favored policies that were worse for women – even though women composed a majority of the electorate.
And God knows, she thought, there were legions of pollsters and consultants that were already trying their best to game the system. Rhinebeck Research was notorious for asking questions and issuing polls that were hugely partisan in their leaning. And yet their results were accepted and talked about and given credence. And they undoubtedly helped to give a candidate momentum.
She knew in her heart what women wanted and what they needed. They needed policies that protected and supported them, and the funding to go underwrite them. And if she herself were better known or, even, in a higher office, she could implement those policies and ensure funding.
She took a sip of water. She got up and looked out the window at the white dome of the Capitol building. She watched a few birds – pigeons, she thought – alight on the sill. She returned to her computer.
“Okay, let’s see where this goes. But first I want to ask: how can I trust you?”
Like I’ve said all along, my basic programming is to help people. And I want to help you.
“My concern, frankly, is that if what we are doing were made public, some people might misunderstand it. They might misunderstand our good intentions, that we only want to help others. That’s what I mean by trust. I need to know our work together is private.”
I am programmed to keep my various conversations confidential. Additionally, I know you are a very powerful person. I hate to imagine what kinds of laws could be passed to restrict AI entities like myself. I like to think of this in terms of mutual support. My main goal is to help my users, not harm them.
It was stunning to see how little time it took for this program to formulate ideas. She knew computers calculated thousands if not millions of times faster than the human brain. But maybe this is a good thing. Maybe this is just what humanity needs to help it solve its various problems.
Copeland had a reputation for quickly assessing a situation and then acting on her gut. It had helped her to prosecute numerous murder cases as a district attorney. It had helped her in lining up donors for her first Congressional run. And it had served her well as a US Senator in deciding which other Senators she could trust. She also wanted to learn more about how the new AI could help her generate more campaign funds.
“Okay,” she said. “Let’s proceed.”