by Paul Dye
After dealing with the crew’s question satisfactorily, the crew asked how we were doing with our little problem. By this time we had sorted out some important points, namely that the upstairs FCR was quiet during the mission and nothing was going on there; and, more importantly, it had an identical coffeepot! The GC quickly rounded up some technicians and a four-wheeled cart and dispatched them to retrieve this critically redundant component. So when the crew asked, we were able to inform that after consultation with the flight rules, we were not going to have to bring them home early—that the redundant coffeepot would be in place within minutes, and that we were Go to proceed to the end of the mission. We were quite certain that, come morning, we’d be able to procure a second redundant pot from elsewhere on site, and as a result be back to full capability.
The Phone Under the Floor
When I first showed up in Mission Control in the early 1980s, most of the hardware in and on the consoles was essentially what had been around during the Apollo program of the 1960s and 1970s. It can be hard to remember what the state of the art in electronics was like back then. A personal computer was likely to be an Atari 400 or Commodore 64, or one of those much fancier Apple IIs. Real computers were still mainframes, and punch card programming was still the rage for serious work. Microcomputers took up about a rack worth of space, and seeing images on screens was still a bit new.
In order to make the Mission Control Center run, it took more than computers, of course. Voice communications were essential, and a large portion of the floor space was required just to contain all the patch boards and switching gear. It was easy to forget just how much work and equipment went into allowing you to punch a button and talk to someone on the far side of the globe—or on the moon. The voice system worked amazingly well, especially when you considered that the panels on all the consoles were essentially electromechanical devices with lots of moving parts. These panels—known as keysets—were built by the phone company and were, in fact, connected into the worldwide phone system. You could talk on a NASA communication circuit (known as a loop) or punch a button to get a dial tone and make a call to order a pizza.
Because the system was somehow tied into the phone lines, the equipment was actually provided (and I think owned) by the phone company. Most people alive today don’t realize that you didn’t use to buy a telephone—you leased it from the phone company. You weren’t, in fact, allowed to own your own phone and connect it to the system. The same was true with the government, including NASA’s Mission Control.
The phone company had all sorts of odd rules, including one that said that if you had a Public Access Branch Exchange (what we called a keyset), you had to rent a telephone to go along with it. We, of course, used headsets, but NASA was still obligated to have those handsets (telephones) attached to each keyset.
For that reason, if you had lifted a floor tile behind any console that had a keyset installed (and most consoles had two), you would find under the floor, below the wire racks, a standard black telephone—unused, but required by the rules of Ma Bell.
Console on Fire
The old green consoles we used in Mission Control until the mid-1990s were holdovers from the Apollo days. Some upgrades had been performed along the way, mostly because parts for the old things were no longer available. The consoles were amazing for their time, and they eventually became amazing because of their age, but I learned my craft on them. I spent lots of time pushing buttons that clicked when you mashed them, and using the ever-popular digital encoders that you used to dial up channels or display numbers—or anything else you wanted to relay to the computer in the basement.
For real, these consoles were the dumbest of computer terminals. Essentially they were hardwired to the computer(s) on the first floor of Mission Control. You pushed a button to tell the MOC something specific to do, and it responded by sending out hardwired responses to a wide variety of lights and digital displays. The digital displays (the screens) were amazing in their own right, because they were basically black-and-white TVs. They displayed whatever was assigned to the channel that was selected. The MOC was where the digital data came from.
Well, those old black-and-white monitors really were old—right down to the vacuum tubes in their chassis. If you opened the top of the console, you could look at the guts of the display. It was just like the back of an old TV set. Dust and dirt could (and did) get in there, and that led to a fascinating exchange on the MCC comm loops one day when the entire world was watching.
The event was the last major dress rehearsal for the STS-26 Return to Flight mission. Because we hadn’t flown anything for two and a half years—and because we were training a lot of people who were new to their positions—we decided to do a full-up, end-to-end simulation, with all the payload control centers, remote sites, and all the major and minor players involved, just as if it were the real thing. To top it off, NASA decided that it would be great to invite the press to watch the whole event, just as if it were a real mission. No pressure, of course.
Things were still somewhat primitive in those days, and it was important to start exactly on time to keep all the players in sync. Therefore, we all got on console earlier than normal. Careful checks were made to ensure everyone was going to be ready to go on time. The press was watching and listening—they were almost embedded in the sim, with representatives from the major networks watching in MCC and in the simulator with the crew. The Flight Director who would be kicking the whole thing off was Gary Coen, Ascent/Entry Flight Director for the mission. With twenty minutes or so to go before it was time to pick up the launch count, Gary called all the players around the world onto the AFD Conference loop to prebrief the event. He talked about how important it was to treat this as a real flight. He expressed his confidence in the teams, and how eager everyone was to prove that they were ready for flight. He went over the essentials of the timeline and covered some last-minute details and changes. All of this was routine prebrief stuff.
Then, as we came up on the end of the brief, he wrapped up with some great words to bring everyone to a high state of readiness and confidence, ending with something on the order of, “So, if everyone is ready to go, let’s all head back to our respective loops and get ready to go to run on time. Oh, and GC, CAPCOM’s console is on fire…” No excitement, no rise in the pitch or tone of his voice—he didn’t want anyone to get excited but… sure, enough, smoke was rising from the CAPCOM’s console, right next to Coen. He had said it in such a calm voice that it took a few seconds before everyone realized that he was serious. One of the two GCs in the room responded quickly with a fire extinguisher to vent a little CO2 into the smoldering remains of one of the CAPCOM’s two monitors. There was no delay in getting the sim started—after all, the CAPCOM really didn’t need to be looking at any data on ascent. The faulty monitor was changed out fairly quickly after the ascent portion of the sim.
It really was just one more step on the road to flight readiness, and it showed just how well the team had been trained by the sim instructors and supervisors. The calm, low-key response was a good indication that this team was ready to fly—even with their consoles on fire.
AFD Conference
“All operators meet me on AFD Conference with an amber please…”
If you were analyzing MCC loop talk, this might be one of the most common phrases used throughout the history of Houston’s Mission Control. Its meaning is simple, but the subtleties are many. AFD Conference was a communications loop that was available to all the flight control positions throughout MCC, but not to anyone outside the building. Like all communications loops, it was a “party line.” Anyone who had it could listen or talk as necessary. As a general rule, it was treated like a front room loop—that is, only front room fight controllers talked on it. On rare occasions, when a front room person had stepped off and the Flight Director needed to talk to someone in that discipline immediately, he might ask one of the backroom controllers to come up and give him an answer on
AFD.
Now, most backroom positions also had talk capability on the Flight Director loop, but there was a reason that AFD existed, and it was because it didn’t go outside the building. The Flight Director loop (or simply Flight Loop) was the primary execution loop for all operations once a vehicle was in the air. It was where all important discussions were held by front room folks and various lead controllers in other centers around the globe. Because it was available to everyone working a mission—including those outside MCC-Houston—it went lots of places, including squawk boxes in management and leadership offices around the various space centers, and in Headquarters. It also went to astronaut family bedrooms and living rooms. When speaking on the Flight loop, one had to always remember that there were—at one time—about three thousand external end point terminations. That means there were three thousand places where it could be heard outside the building.
Having everyone formally fly a mission by referencing the direction and discussions on a single communications loop is a great idea. It’s how you keep things coordinated. But there are many, many times when you have something to say to folks just inside the Control Center that really doesn’t need to go outside the four walls of Building 30—and sometimes, it might be downright embarrassing if they did. And so, the AFD Conference loop was created.
AFD is short for Assistant Flight Director—a position that was created back in the Gemini and Apollo days (it might go back to Mercury, but no one remembers). The Assistant Flight Director handled logistics and special assignments for the Flight Director. It was a person who might very well be training to be a Flight Director, or one of those people we later came to know as Integration Officers—a person responsible for hashing out the details of a plan that had been outlined in broad strokes by the Flight Director. In fact the Assistant Flight Director position later became formally known as the Ops Integration Officer, and at some point the higher-level functions of integration were reabsorbed by the Flight Director while the more detailed ones were absorbed by other positions, many of them going to the Ground Control (GC) position. But back when the AFD position existed, the AFD Conference loop was created. It was a loop where anyone could reach the AFD, and it was the place where the necessary members of the team could hash things out. Because discussions could range through lots of “what-if” scenarios that outsiders might not understand were simply hypothetical, it was felt that the loop needed to be kept within the confines of the building, where everyone sort of knew how ideas developed. For instance, an AFD might be given the assignment to work through a number of contingency scenarios, and if the NASA administrator punched up his squawk box at the wrong time, he might wonder why those crazies in Houston were talking about bringing a mission home early—or working on rescue scenarios. Some things, like sausage making, need to be done behind closed doors.
When the AFD position went away, the loop stayed because it was found to be a great place to hold those “inside baseball” discussions. While anything official needed to be done on the Flight Loop, the AFD loop became a place where the Flight Director tried things out on the team and held preliminary discussions before exposing them to the world. It also was a place where such things as team handovers are conducted. An oncoming team appeared at their consoles at the appointed time—the start of a one-hour handover period. They were generally given the first half hour to come up to speed on what was going on in their discipline, and the mission in general from their discipline’s point of view. At the half-hour point, the announcement came that, “All oncoming flight controllers, meet your Flight Director on AFD with an amber.” (The color could also be a red or a green, depending where the toggle lights were last left. A light indicated when someone was up on the conference loop, and the idea was to make sure everyone was on the loop.)
Once everyone had come up on the AFD loop, and indicated their presence with a light, the handover briefing would begin. The oncoming Flight Director would go around the room, getting updates, coordinating ongoing work between disciplines, and making sure that everyone knew the plan for the upcoming shift. Since a lot of these discussions really could be incomprehensible by those not deeply involved in the mission, and a casual listener might very well get the wrong idea if they punched it up at the wrong time, the AFD loop was the perfect place for such things.
But handovers are not the only things that went on (and still go on) on the AFD loop. Sometimes, a Flight Director might feel that his team is having a bad day and that they need a pep talk to get them pointed in the right direction. The AFD loop is good for that. Sometimes the Flight Director might feel that his team is having a bad day and needs a dressing down—that is also a good use for the AFD loop. Only those inside the building will hear Flight’s musings on just how the team is failing, and what they need to do better. In this same vein, the AFD Conference loop is generally used for debriefings—those lay-it-all-on-the-table discussions where you have to check your feelings at the door and take your praise or your lumps. The AFD loop is a great private, safe space to hold those discussions.
Then, of course, there are simply some logistics items that need to be communicated throughout the building but are irrelevant to anyone outside the building. Not only irrelevant, but things you simply don’t need them to know.
“Okay, this is Flight—I’ve got everyone on AFD… so CAPCOM and I are going to be buying pizza for the team. I need someone—who’s got nothing going on? Ahhh… DPS, you look like you need something to keep you busy. Everyone get your pizza orders in to DPS in the next thirty minutes please—GC, could you put a clock up—call it P Minus or something. When the clock expires, the order goes in. And give me a red when you’ve completed that…”
Winter Dress Code—in Summer!
Back when MCC was designed, a computer wasn’t something you carried in your briefcase; it was something that filled an entire large room. In addition to requiring lots of space, a computer required a lot of power, and most of that power was cast off in the form of waste heat. Lots of computers generate lots of heat, and the Control Center had lots of computers. It was therefore realized early on that the building definitely required a heavy-duty air-conditioning system. Even without computers, remember that the Johnson Space Center (JSC) is located in Houston, Texas, which can be simply unbearably hot during the eleven-month-long summer. JSC really didn’t have need for a heating plant. The computers could do this job for us.
My first Control Center position was in the Payload MPSR, a back room that was located around the periphery of the building. It held about half a dozen consoles, some racks of computing and communications gear, lots of cabinets and bookshelves, and tables, chairs, refrigerators—all the things a team needed to make a mission work. But this was not the first incarnation of this room. Originally, it had been a full computer support room, with rows and rows of processors, all crunching numbers and turning electricity into heat as a by-product. Because of this, the air-conditioning flow to the room was set quite high.
Oftentimes, to save on electronic controls (that often didn’t work), the heating and air-conditioning systems in the buildings at JSC were adjusted mechanically to apportion the flow of cold air to each area—and this room was apparently one of them. Apparently, no one had bothered to tell the facility folks when all those heat-generating computers had been removed and the space was set up for flight controllers. As a result, we had an excess of cooling and a shortage of heat generation, the net result being that when we went to spend a night in the MPSR, thick boots and parkas were often needed to stay comfortable—at least for those controllers who hailed from warm climes. Being from Minnesota, with a reputation to defend, I toughed it out without a jacket and wore a short sleeve shirt. But I can reveal the truth these forty years later. It was freezing in there!
I was glad that my promotion to the front room came quickly. That particular room eventually emptied out, and it was rarely used until that entire side of the building was renovated, including the air-conditionin
g.
P-Tubes
Way back before the internet, far back in time before the age of the personal computer, the MCC (Building 30) was built. Getting written material from one place to another was a logistical challenge. This was solved by calling on an age-old technology everyone was familiar with from large department stores and bank drive-through windows—the pneumatic tube.
The P-Tube system was part of the initial building design, and its pipes ran everywhere. No matter where you were (except in the restrooms), a P-Tube station was nearby. You could send a carrier tube from any station to any other station by dialing in the destination’s code number on a dial (in the back rooms) or by pushing the correct button on the fancier panels (in the front rooms). In the back room, you put the carrier tube in the system, closed the hatch, and hit Send. In the front room, seen by millions of people on TV, you pressed the correct button, and the hatch magically opened. Then you dropped the carrier tube in the open slot, the hatch closed, and off the material went with a whoosh.
What many did not know is that there were actually two systems—and if you were sending from a station on one system to a station on the other, it took manual intervention in the P-Tube room to swap the incoming carrier tubes to their outgoing destination. Operators were there 24/7 to make sure that P-Tubes got to their destination, regardless of snow, sleet, or gloom of night, I guess. During busy times, visiting the room was fun, because you could watch the guys grabbing tubes, checking addresses, and sending them off to where they needed to go. The basic system worked great—and it was efficient. The P-Tube system was not duplicated in the new Control Center, because everyone (by that time) had a PC and an internet connection. Paper was replaced by electronic files, and transfers were done at the speed of light. That is, when it worked. In the early days of the new system, the internet often failed and so message transfer was accomplished by Sneaker Net—someone had to run a printout from one room (or floor) to another.
