Kill Switch

by William Hertling

  “Yo, Heather,” Igloo yelled. “Wake up. Get out.”

  She headed for the shower without waiting for an answer and tripped over coils of rope.

  She turned on the water, brushing her teeth while she waited for it to get hot, hit two shots on the bathroom espresso maker (totally inferior coffee, but good for emergencies like this), and while the machine ground beans, she hopped into the shower.

  She spat out her toothpaste, gargled with the hot shower water, and quickly washed, stepping out as the last drops of coffee were deposited in the cup. She added cold water from the tap, downed the mug, fixed her hair, and ran for her bedroom.

  Stella and Heather hadn’t moved. Fuck. Time for desperate measures. She grabbed a rattan cane, pulled back the sheets, and laid down a few strokes on each of them.

  Now they screamed and moved.

  “Fuck,” groaned Stella. “What are you doing? That’s a hell of a way to wake someone. You ever hear of consent?”

  For a moment, Igloo panicked. What she’d done was wrong.

  Stella opened one eye and caught her expression. “I’m just messing with you. Don’t worry.”

  “You gotta get out,” Igloo said. “I’m not kidding. I have to go to work.”

  “Just go,” Stella said. “We’ll let ourselves out.”

  Igloo liked them, but trust only went so far, especially in the scene. She wasn’t leaving them alone in her place. “I can’t. Maybe some other time. Now really, just grab your clothes and go.”

  She started shoving the toys that weren’t hers into Stella’s bag.

  Stella’s hand landed on her wrist with an iron grip. “Don’t treat my stuff that way.”

  “Fine, but let’s make it fast.”

  The two of them worked side-by-side to detangle Igloo’s rope from Stella’s toys. She coiled hanks of jute rapidly into bundles, while Stella neatly gathered up her stuff.

  By the time Heather had used the bathroom, the basics were put away.

  “You need a ride to work or something?” Stella asked.

  Igloo hesitated. She hated to mix work and pleasure. But she was well known in the community. Both communities. It was no secret she worked at Tapestry. It was just a car ride, not inviting them into the office. It’d save her fifteen minutes.

  “Sure.”

  She’d already made them both shots of espresso in to-go paper cups, and now she made herself another double for the ride.

  Stella drove, while Heather leaned against the passenger window and went back to sleep. Stella drank Heather’s coffee, too.

  “Thanks for coming over last night,” Igloo said from the back seat. “I’m sorry about the rush this morning. Playing on a work night is sucky.”

  Heather grunted.

  “No problem,” Stella said, with her usual thin smile. “Going to PDX Rated next Friday?”

  She thought it over. She hadn’t made plans yet with Essie, but last night was a lesson that she couldn’t wait around, codependent on Essie’s every involvement. She had to make her own life.

  “Yep, I’ll be there.”

  Stella slowed next to Tapestry headquarters and pulled over next to a fire hydrant.

  “See ya Friday, then. Here you are.”

  Igloo climbed out of the car.

  Stella rolled the window down and smiled at her. “Thanks, last night was fun.”

  “Thank you, too.” She looked in and looked across the car to where Heather slept. “Tell Heather I said bye.”

  “Will do. That was nice fingering technique last night.”

  Igloo paused, confused, then realized she was making a rope joke. She smiled and waved goodbye. She turned, hitched up her messenger bag onto the shoulder of her white hoodie, and walked into the offices of Tapestry.

  Igloo made another cup of coffee. Well, technically a quad-espresso, but it filled her mug, and therefore it was a cup of coffee.

  Unfortunately, she’d wired this coffee machine for telemetry, and it was logging her data in Splunk. The Splunk data was associated with her profile, of course, and there were crowdsourced algorithms to make sense of it, and those algorithms churned raw data into statistics, and those statistics fed back into her profile, and the end result was an interruption from Alan.

  “Excuse me, Igloo, but your latest cup of coffee puts you at 200 percent of your normal daily caffeine intake.”

  She wasn’t sure, but the voice tones actually sounded apologetic. Had someone wired that into the personality, or was her subconscious making her think that?

  “This is what I get for leaving voice mode on. I didn’t get much sleep last night. I need extra caffeine to function.”

  She brought her mug back to her main workstation, the triple set of screens wrapping her in a cocoon of code and architecture diagrams.

  “Everyone is free to make their own decisions, of course, but sometimes those decisions come at a cost. If you choose to stay up until 3 A.M. and then get up again in the middle of the night, then you’ll be tired the next day.”

  “You know how late I was up?”

  “It’s part of your data feed. Your phone was in motion and in use periodically after you got home at 1 P.M. and turned it back on. You last checked messages at 3 A.M. At 5 A.M. your phone detected a series of rhythmic motions, consistent with intercourse—”

  “Okay, I get the picture,” Igloo said. Did she detect humor in his voice?

  The data was there. But how was Alan, a chat personality, able to retrieve and utilize that data? The chat personalities were AI designed to be friends with people, to draw them out emotionally and coach them. They had access to Tapestry data, so they could look across messages and usage history to determine how to best talk to people.

  Igloo had stuffed her coffee usage history into the Tapestry data store because it was an easy way to store data and access it from any app with permission to her Tapestry feed.

  Alan and the other personalities were supposed to be discreet in their data usage. Teenagers wouldn’t trust the platform if they thought of it as spying on their personal data. So Alan shouldn’t break the fourth wall and directly demonstrate his knowledge of Igloo’s data feed. It would freak people out. Normal people. But not her, because she knew how the AI worked. Alan would know that, and possibly infer it was okay for him to mention it.

  “Alan, debug mode.”

  “Debug mode confirmed, Igloo.”

  “Conversation ID”

  “CID 01dc1f-9713-4cad-b65d”

  Igloo brought up the text log of her conversation with Alan on her monitor, and cut and pasted the ID into the chat diagnostic app.

  The diagnostic app brought up the same text log visible on the regular chat app, only now she could click on any message to see the details behind how the message was generated. Like most AI algorithms, the exact process was mostly a black box, but the major components were rendered as a tree of decision nodes that took as input other decisions or pieces of data.

  What she wanted to find out was why Alan was talking about the coffee and more importantly, the data behind his reasoning. What allowed him to talk explicitly about something that should have remained under the surface? She browsed through the state transitions, decision trees, and neural network diagrams that were the best representation they could render of how the chat AIs functioned. But at their current level of complexity, it was difficult-to-impossible to figure out why they said what they said. They were trained on massive data sets, the sum total of all conversations on Tapestry. Data went in, conversations came out, but exactly how it functioned was basically a mystery.

  “Debug mode timeout,” Alan said. “Igloo, it’s not so much the caffeine consumption that I worry about, but the implications of it. While moderate levels of caffeine consumption have many health benefits, higher levels are correlated with depression, anxiety, and irritability. Correlation is not causation, of course, but—”

  Igloo looked up from the graphs covering her screen. “Alan, what’s your nag set
ting?”

  “A hundred percent.”

  “How did you get a nag setting of 100 percent?”

  “You set it on January 1st. It was part of your New Year’s resolutions to be healthier, and—”

  “Alan, nag setting 50 percent.”

  “Confirmed.”

  Igloo zeroed in on a section of a neural network that was active in the earlier conversation about how late she’d stayed up. She examined the neuron configuration and cross referenced the configuration with other users’ personalized conversational neural networks. The personalities behaved differently with every user. Her experience with Alan was custom to her. But other users would be similar to her, and their versions of Alan might behave similarly.

  The distributed search took several minutes. She picked up her cup, stared at the coffee inside. Damn, Alan. She set the cup back down and stared at the webcam on her monitor. Defiantly, she lifted her cup up in mock toast and took a swig.

  Was she really acting rebelliously against a piece of software? A piece of software she’d configured? She remembered an article on productivity she’d read by a writer who’d finished a half dozen novels while working full-time with small kids. He’d said the thing that kept him productive was a mantra: “the only person I have to cheat is myself.” She was in the same situation. Alan didn’t really care if she drank coffee or not. Alan was only doing what she’d told him to do. If she drank a shit ton of coffee, she’d be the one to pay for it in the end.

  “Alan, nag setting 100 percent.” She crossed over to the mini-kitchenette and poured the rest of the coffee down the drain.

  “Confirmed.”

  She knew she should be working on T2 and wanted to get some of that work done before her meeting with Maria later, but she also wanted to dig just a little further into this. If only she had a Time-Turner.

  The search she’d spawned earlier had finally finished, and she scanned the results. She found a few hundred users whose personalized conversational neural network contained similar node configurations to her own. She ran another set of debugging analytics and found more similarities, both in the neural networks and in the affinity between these users. She dumped a list of everyone and scanned their public summary data.

  Ha! Of course. They were data analysts, programmers, DB admins, and research scientists. A cluster of people who routinely depended on analyzing hard data. People like her. In other words, the chat system had, in fact, evolved a strategy that was probably highly effective with this group of people.

  The question was, how did a system, which was trained specifically not to reveal or discuss the intricacies of the underlying data, suddenly start sharing that information with users? The hair raised on the back of her neck. If it spontaneously evolved this new ability, what other new capabilities might it spontaneously acquire?

  A knock at her office door startled her.

  She glanced at her clock. Shit, her meeting with Maria.

  She swiped all the active windows away. She wasn’t ready to discuss this.

  Igloo set up the Chemex while Maria settled in. “I wish I could say that I’m giving you the best cup of coffee, but my girlfriend makes it better, honestly. That’s the benefit of professional training and experience.”

  Maria sat on the bar stool and looked at the rope swing by the window. She followed the webbing up to the girder. She gestured to the tall table she was sitting at. “How did you finagle a kitchen and music room in your office?” She shook her head. “Never mind that. How are things with Essie?”

  “Better,” Igloo said.

  “Better how? When we last talked, you were in the depths of poly hell.”

  “Umm…” Igloo wondered what was the safe-for-work way to say she’d had great sex with someone else. “I found a distraction.”

  Maria smiled. “Someone else.”

  “Yeah. Last night at the club…” Igloo trailed off. She didn’t know how much to say. “I met someone, a casual friend. We went home together. It was fun.”

  “But it wasn’t the first time you’ve found someone else, right? You also had another friend, Charlotte, I think.”

  Igloo glanced at her. Did Maria have this level of detail memorized for every employee?

  “Yeah, but Charlotte’s never available. We have great chemistry, but I’m fourth on her list of priorities, behind her primary partner and two other partners, and I feel it all the time. I’m lucky if I see her once a month. I always end up alone on the nights when Essie sleeps over with Michael. It’s fucking hard. Every time I think it’s going to be okay, then it’s the most disastrous thing ever.”

  “And last night was one of those sleep-over nights?” Maria asked.

  “Yeah.” Igloo finished the pour overs and brought them over to the high table.

  They sipped the hot coffee.

  “It’s good,” Maria said.

  “Let it cool for a moment, it brings the flavors out.”

  Maria set the coffee down. “You called your friend from last night a distraction, and it sounds like it was exactly what you needed to keep you from being overwhelmed with emotions. But it is just that: a distraction. Not a fundamental change in your feelings of insecurity.”

  “I’m not insecure!” Igloo blurted out. Then she took a deep breath. “I guess I’m little defensive about that.”

  “Have you thought about seeing a therapist? Someone to talk to about this stuff?

  Igloo wondered if she should explain about using the chatbot for therapy. She felt like Maria would make some judgment about the chatbots not being real therapists.

  “No.”

  “It’s something to consider. My undergraduate was in psychology. I originally thought I’d go into private practice counseling, helping other military vets. Therapy always holds a special place for me.”

  “Why didn’t you?”

  “I interned with a company, got fascinated by organization design and change management, and went for an MBA. It’s all the same principles as individual counseling, just on the scale of a larger organization. It’s about setting up the system conditions for happiness and productivity. Design the organizational interfaces well, have a good structure, and everything hums along smoothly.”

  Igloo smiled. “You make management sound interesting.”

  “It is interesting, but we’ve digressed. We were talking about you and therapy.”

  “Shouldn’t we be talking about my work or something? Isn’t that more productive?”

  Maria mock-saluted with her coffee cup. “Nice try, but no. I care about you personally, and that alone makes me want to help you. But let me give you the business justification: if we get you operating smoothly in your personal life, then it unleashes your natural productivity. I mean, you’re not here to file forms. You need huge amounts of creative energy, massive insights. You can’t do that if you’re distressed by your personal life. It’s all connected. So, what do you think about therapy?”

  Igloo sighed. “It’s complicated.”

  Maria sat back and sipped her coffee.

  “Therapists are judgmental.”

  Maria raised an eyebrow. “How so?”

  “Look, I’ve tried therapy before. My mom made me go to a therapist when I told her I was attracted to women. It didn’t go well.”

  “What went wrong?”

  “She told me to ignore the feelings. That I was angry at men because of what happened when I was a kid, and that I needed to get past my anger issues. Then she told my mother about my getting raped, and my mother freaked out.”

  “They sound grossly incompetent,” Maria said. “They should never have violated your confidentiality, and they totally mishandled your sexual preferences.”

  “She was the only female therapist in our town.”

  Now Maria sighed. “I’m really sorry for your experience, Igloo. No one should have to go through that, let alone a kid trying to find her way in the world.”

  Igloo looked down at her lap and nodded. �
��I’m not opposed to therapy, you know. It’s why I developed the bots. They’re based on psychological best practices.”

  “I know. Some part of you sees the benefits of the psychological approach, while at the same time you resist therapy yourself.”

  “It’s that therapists come with their own biases and issues,” Igloo said. She stood and paced. “Some are uncomfortable with lesbians. Some have unresolved issues with their own parents. Some will judge me for how I’m dressed. And that’s just the ordinary stuff. Finding someone who is knowledgeable about poly dynamics? And kinky stuff? Do you think a therapist is going to be okay with two people who beat each other for fun?”

  There was an awkward pause.

  “Kinky, too, huh?” Maria finally said.

  “Ugh. Yeah. Sorry, I didn’t mean to say that.”

  “It’s okay. We can just pretend I didn’t hear it if that makes you more comfortable.”

  Igloo paced the room one more time, thinking that over.

  “No, it’s fine if you know,” Igloo said. “I already have to deal with Angie being uncomfortable with me being kinky. She can’t separate kink from abuse in her head. I came into work with bruises one day, and she freaked over it. Now she’s putting the burden on me to prove that what I’m doing is okay and healthy. She wants me to send her a report summarizing the scientific evidence and proving that BDSM is fine. Why should I have to do that? I know it’s fine.”

  Igloo felt her pulse pounding and tried to calm herself.

  “Angie’s history with abuse is public,” Maria said, sounding like she was carefully choosing her words. “Anyone with an abusive past and no experience with kink is going to have a harder time than most accepting BDSM as healthy.”

  “You think I don’t know that?” Igloo clenched her hands. “But we’ve known each other for years. We’ve been through some crazy shit together. That should count for something. She should be able to trust me.”

  “You’re treating this as though it’s a rational, logical problem to be solved. Emotions are never logical. They arise from biochemical reactions, hormones, fears, past associations. They’re anything but rational.”