The Silicon Jungle

by Shumeet Baluja

  Over the years, the common queries that everyone typed at some point in their web-searching lives, “Money,” “Houses,” “Sex,” “Adult Pictures,” “Politics,” “Music,” were handled extremely well at Ubatoo, as they were at every web search engine. What set Ubatoo apart, and imbedded Ubatoo firmly in everyone’s daily routine, was the absolutely brilliant job it did when those common queries were discarded in favor of those that held more personal meaning. At that point, users joined what was mathematically known as the “long tail.”

  The queries in the tail are those that aren’t asked very often. The reason it’s often called the “long tail” is that the majority of queries submitted to Ubatoo aren’t common. They are simply what make you, you. They are what make you unique to Ubatoo. Everyone queries for “sex,” and everyone queries for “money.” In contrast, all those requests for information that you, and only a few others in the entire world would think of querying, such as “did Freddie Krueger’s gloves have four blades or five,” are what distinguish you from the rest of Ubatoo’s users. These are things you care about.

  The moment you discovered that even a few of these queries could be answered well, a vital seed was planted. This was a place to turn to whenever information was needed—no matter how strange the topic. Without knowing what information you would eventually search for, Ubatoo, seemingly magically, was able to find an answer, blazing through tens of thousands of machines, all sorting and sifting through the unfathomably monumental number of words ever written. If you asked anybody at Ubatoo how they did it, or how they possibly figured out what ads to show you when you asked such esoteric queries, the answer was always, it’s the intelligence built into our Algorithms. More precisely, but perhaps with less marketability, it was all about “ranking”—how one result was chosen to appear above another. How one web page was more about what you wanted to see than some other.

  Many, many companies (on the order of hundreds), people (on the order of tens of thousands), and dollars (on the order of billions) were devoted, and continue to be devoted, to ranking. Ubatoo’s army of Ph.D.s working on just this problem, who were versed in everything from computer science, mathematics, statistics, linguistics, cognition, perception, and psychology, kept Ubatoo on the cutting edge of this problem. These people were affectionately called “Tinkers” within Ubatoo. Their sole job was to tinker with the ranking procedures in small ways that kept improving the order in which you saw the results. This excruciatingly tightly focused attention to ranking was crucial to Ubatoo’s success. When this was married with Ubatoo’s already huge head start and foresight into building enormous computational resources into its computing cloud, the combination ensured that Ubatoo kept, at the very least, decades ahead of their nearest competition.

  Every infinitesimal change to the ranking mixture was tested with thousands of users. Every click, word typed, query entered, was tracked to decide whether you were just a tiny bit more satisfied with the changes. Of course, this could never be done for the majority of “long tail” queries; too many of them existed. Instead, Ubatoo relied on the fact that if their algorithms and ranking functions worked well on the queries they tested, they would also work well for the queries that were important to you. That assumption was dead-on accurate. These ranking procedures were never adjusted without extensive testing; too many errors arose when humans were allowed to use their intuition. It also made sure nobody was able to bias the results to benefit any web site that was their personal favorite.

  It was because of this dogma that getting more people to Molly’s site, EasternDiscussions.com, was a terribly daunting task—even for someone who worked at Ubatoo. Nonetheless, this was precisely what Stephen needed to do. He needed to ensure that Molly’s web site appeared as close as possible to the top of the search results when someone typed anything related to politics and religion in the Middle East. The higher her web site appeared, the more people would see it, the more people would visit her site, the more people would contribute to its discussion forms, and the more chances Molly would have of gathering enough subjects for her thesis. The problem was, of course, that at least dogmatically, there should be no way to make this happen.

  Molly had rarely been able to make it to Ubatoo to join Stephen for lunch. The shifts she worked at GreeneSmart and the interviews and lunch meetings Stephen often had to attend made lunchtime get-togethers almost impossible. Because of this, in the few instances when lunch did happen, they attempted to sit by themselves. Besides, Stephen didn’t relish the prospect of having to keep the conversation interesting to both his work friends, who would invariably talk about work, and Molly, who would understandably be bored to tears. Occasionally, when someone didn’t get the hint that the couple didn’t want to be social, circumstances and politeness left little choice. On one of the rare days when Molly met Stephen for lunch, Andrew was at the Asiatique Café in line immediately behind them. The amiable conversation in line hadn’t ended with an invitation for him to join them, but when Stephen and Molly went to find a seat, Andrew hurriedly followed them to their table.

  Before either Andrew or Molly had a chance to go beyond introducing themselves, Stephen plunged into a conversation that had a chance of interesting everyone at the table. He described Molly’s predicament, getting subjects onto her web site, and admitted he was lost on how to help her.

  “If you could get your web site to appear in the top ten results for even just a few popular queries, that would drive enough users to you, I’d imagine,” said Andrew, talking directly to Molly.

  “I think the question is how I do that. I don’t have any budget and I need to have this happen sooner rather than later. I just need traffic, even if it’s for a little while, just long enough for me to finish my experiments.”

  “Do you know anyone in the ranking group? They might have some insights,” Andrew replied, this time talking directly to Stephen.

  “I don’t think they would help though, do you? I figure that with all the religion we have here on never hand-tuning results, and being purely algorithm-driven . . . blah, blah, blah,” Stephen said with exasperation. “Nothing is hand-tuned, right? I think it’s pretty unlikely they would help.”

  “I’ve done it before, though, for a friend’s site. I mean it was last year, when I was an intern in the Tinkers group, so things might have changed since then. I’m not even sure it could be done again,” Andrew said.

  “No, no. What I’m trying to do is to move just her site up a few notches. You couldn’t have done that, right?” Stephen asked.

  “It wasn’t really anything big. When I was an intern last year, we were encouraged to run some test users through some tweaks to the ranking system. We ran the experiments from start to finish, and then presented the results to the ranking team.”

  “So you made up results to help your friend? Really?”

  “The results weren’t really that made up. I didn’t just lie about them. I might have slightly enhanced the changes that boosted my friend’s site higher and presented only those, though,” Andrew reasoned, with a grin.

  “Nobody noticed?”

  “Oh, relax, Stephen. It was just for a week or so. I’m sure that if anybody noticed it was probably put down to a statistical anomaly. Things like that happen. And it sounds like that’s what Molly needs, anyway.”

  Andrew was using off-hand comments about statistics with this crowd? “No, Andrew, actually, things like that don’t just happen. Andrew, the kind of statistical errors you’re talking about are so many standard deviations away from the expected that the chances are negligible, you should know that.” Stephen was only willing to suspend disbelief for so long.

  “Fine. Maybe it was put down to ‘stupid intern can’t do anything right.’ Whatever, it worked. Ubatoo’s fine, my friend’s site got more traffic—he’s fine, and I’m fine. It’s all fine,” Andrew replied defensively.

  “What does your friend do, anyway?”

  “It’s really just a frien
d of my parents. My family is into buying houses and the whole real estate thing. This guy has been our real estate agent for years. The only thing I did was make sure his name popped up higher in the results. Now he gives us early access to his properties, foreclosures, whatever else, before he tells others. It’s paid off.”

  “Unbelievable. The whole thing is unbelievable,” Stephen was starting to rant. “We literally decide who gets to see what and when, and with all the religion we have here on being fair . . . and you bend all our rules for a real estate agent? It’s a pretty massive letdown, don’t you think? I could think of a few good reasons to bend our rules, orchestrating world peace, solving world hunger, but for a real estate agent?” Underneath the table, Molly’s silent kicks had been growing increasingly more violent for the last few minutes, trying to get him to end this outburst. Andrew, meanwhile, hadn’t said anything.

  Stephen opened his mouth to speak, but Molly sharply pinched the back of his neck to silence him. “Can someone help me do the same thing for EasternDiscussions?” she asked hopefully.

  Andrew didn’t say anything at first, finishing off his Coke and moving onto the coffee that was getting cold. He made a show of the long breath he exhaled. “If it’s still possible to do, I suppose that I could try,” he said finally. “But, listen, do me a favor, okay?” Andrew cautiously asked Stephen.

  “What?”

  “If you come up with something decent when you’re working on the Touchpoints project, make sure my name is attached to it, okay? Just say I gave you some ideas for the e-mail part if there is one, or make something up. Every little bit of goodwill helps. I really want to get an offer here this year. Otherwise, I’m on the job market again, at the end of the summer . . . ”

  Stephen flashed back to the time when he was heading his own group at SteelXchange, and he thought about how lucky he was to never have gone through this tit-for-tat negotiating there. He and Arthur had been cofounders; there wasn’t this trading of misdemeanors. He wondered whether his team had played these games, too. He had never seen it, but now, he doubted they would have been any different. Here, at Ubatoo, he was just as much a part of the reality show–like alliances as Andrew was. “If there are any breakthroughs, I’ll bring you in.”

  “Thanks, Stephen,” Andrew replied without looking at him. “Just get me the details later today. I can’t control when the next tests are, but it shouldn’t be more than a week or two.”

  “I thought you said you weren’t even sure it worked the same way as last year,” Stephen challenged, only half joking. Molly kicked him again.

  Hand in hand, Stephen walked Molly back to her car from the Asiatique Café. Lunch had been productive, but it was hard not to feel cheated out of their time together. “Play hooky with me?” Molly urged.

  “I’d like to, but I can’t. I’ve got too much to do today. Can I take a rain check?”

  “Come on,” she pleaded, still holding on to his hand. “Nobody cares when you’re here and when you’re not. Nobody’s even going to notice. Besides, even if they did notice, who knows, they might actually think you’re human if you take an afternoon off sometime.”

  “Don’t you have to work at GreeneSmart today?”

  “I took the day off to work at home. Tell you what, I’ll give you a few options of what we can do, and I’ll let you decide which one you want.”

  “I’m listening.”

  “We could go into San Francisco and ride the cable cars and then spend the rest of the day at the art museums. That’s option one. Second, maybe a leisurely drive up the coast on 101—and take the next few days off and stay at bed-and-breakfasts we find along the way. Third, we could go outlet mall shopping. What do you think?”

  “Really, all fascinating choices. But I’m still going to have to—”

  “Or,” she interrupted him, “we could go to a movie and watch the most brainless, gory, horror movie we can find and gorge ourselves on a bucket of popcorn. There’s always that option, too.”

  She had him. If they hadn’t met at GreeneSmart, the only other place they ever might have had even the smallest chance of meeting—if the stars aligned just right—was at an otherwise completely empty B-rate horror movie. Anything and everything that looked like it might be scary, campy, bloody, over the top, they watched—from aliens to demonic possessions to chainsaws.

  Before he could agree, even though it was already abundantly clear by his expression that he would, she sweetened the deal. “I’ll even throw in a quickie back at home. You know, just in case you still aren’t sure.”

  It’s good to be a tough negotiator.

  -NEGOTIATIONS AND

  HERDING CATS-

  July 13, 2009.

  “How are you enjoying being a manager, Jaan?” Atiq asked over a boiling cup of milky Teh Tarik—pulled, frothy, black tea—made by special request for Atiq by one of Ubatoo’s sous-chefs from Malaysia.

  With the countless recruiting trips and press interviews, it had been several weeks since Jaan and Atiq had a chance for their usual one-on-one meetings. “Managing is not for me, Atiq. I don’t know how you do it. Personally, I’d rather be left alone. I’ve been working on my new project for a month now, but being a manager is really slowing it down,” Jaan replied.

  “What project is this? Completing Touchpoints, I hope?”

  “Nope, it’s brand new. I’ve been thinking about where our next breakthrough will come from. The more I think about it, the more I realize Ubatoo already has a massive amount of information about people who are online. I want to see how much we can find out about people not online.” Jaan bounced up to the whiteboard and grabbed a marker, ready to write the moment the need arose. “My mom—she never touches a computer. In fact, she’s afraid of them. Normally, you’d think that Ubatoo couldn’t get any information about her, right?”

  “Sure,” Atiq shrugged.

  “But I, and my brothers and sisters, write e-mails about my mom and my parents all the time. In fact, the pictures we upload, quite a few have her in them. It’s too easy. Atiq, I don’t know why we haven’t done this before.” Jaan was drawing stick figures in rectangles representing photographs on the board with labels of “Me,” “Brothers,” “Mom.” He continued, excitedly, “Try this, just go to any photo sharing site, ours or any of the others, look at how many pictures have labels on them like ‘Me and mom,’ ‘My mom,’ or even ‘Mom and grandma.’ ” Another stick figure of grandma. “You’ll get hundreds of thousands of images. If we recognize the woman in a few pictures that I uploaded with this label, voilà, we’ve got her.” A bull’s-eye drawn over the figure of “Mom.” “We know what she looks like, who she’s with in the picture, and what her relation is to the person who uploaded the picture—me. We know she’s my mom. If we know something about me, then we know something about her, too. If we know something about my brothers and sisters, which we do, then that tells us even more about her. With just a little bit more work with the data we already have, we already know a lot about my mom. We just have to put it together.”

  “Why would we, though?” Atiq asked skeptically.

  But Jaan wasn’t listening. “It’s not even that hard. When I buy a present for my mom, what do I do? I buy it online and have it shipped to her. Sometimes I even attach a note, like ‘Happy Birthday’ or ‘Hope you feel better,’ or ‘Thought you could use this.’ Think how much information Ubatoo has on my mom. Birthdays, illnesses, her likes and dislikes, her address, hobbies, her kids—and never once did she touch a computer.”

  Atiq hopped out of his chair and grabbed the marker from Jaan’s hand. He drew a few more boxes with “father” and “classmates” written in them, and arrows pointing haphazardly around the board. “It’s not just you and your brothers and sisters. If your mom’s friends, old classmates, or even your father, for that matter, are online and they talk about your mom or buy her presents, we have all of their profiles to use as well. With all your mom’s relationships, we’d certainly be able to pro
file her. There are old class rosters and yearbooks online, too; they must have information to mine as well.” Atiq darkened the circles of the bull’s-eye centered on “Mom.” “Very nice, Jaan. We should have done this earlier. It’s so obvious.”

  Jaan smiled a very satisfied smile.

  Atiq stared at the whiteboard a moment longer, imagining connections drawn between the hundreds of stick figures that would eventually be connected to “grandma,” new sources of information being brought online from the mass of photos their users had uploaded, and profiles being created on people who had never touched any of their products. “That’s clever, Jaan. I’ll take it upon myself to find some people to work with you on this,” Atiq promised. Then he added, “Might want to keep it quiet until we give our PR department a chance to develop a positive spin on this before news of it leaks, though, okay?”

  Focus, he had to remind himself, focus. “But listen, Jaan, I really need you to wrap up Touchpoints, too. I’m counting on you for that. We need to deploy it across the company, and it’s not going to happen without you.”

  “Don’t worry so much. I’ve got an eye on it. The interns are already using the system every day, and the new hires are creating additional features. It’ll be done and fully deployed this summer. I don’t need to supervise it as closely anymore.”

  Struggling not to let his frustration show, Atiq replied, “Jaan, you’re in charge of it. Keep a close eye on it, okay?” Good scientists don’t always make good managers, Atiq thought to himself. He’d have to remember to get someone to take over Jaan’s management responsibilities soon.

  “How are the interns doing, by the way?” Atiq asked. This was the first time they had spoken about them since the conference call about Aarti and William on the second day of intern season.