by Paul Dye
I reported that first day along with a bunch of other co-op students and new employees. We spent the day going through paperwork, selecting insurance options, getting badge photos taken, and receiving a general orientation to working for the government. In the afternoon, we were finally taken out to meet our new supervisors and see where we’d be working—and what we’d be working on. Up until that point, I had no earthly idea what I’d be doing; I just hoped it would be interesting.
It turned out that the way the co-op program worked was that applications were passed around to the various organizations that had the need for a little extra help (that wouldn’t count against their limited staffing numbers), and supervisors flagged the folks who looked like they might be a good fit. I remember sitting with my first supervisor, Dick Koos (I later found out he had been one of the simulation supervisors back during the Apollo missions), and we talked a little about my background. He was interested in my diving and flying much more than my grades. This was probably good, because my GPA at that point was not all that impressive. What had impressed him, apparently, was that I had real-world experience in operations—taking equipment into the field, the water, and the air, and dealing with what happens in real time to alter your plans to accomplish a mission.
The group I was assigned to, the one Dick supervised, was the Payload Computer Section of the Payload Operations Division. The Shuttle had not yet flown, but the Flight Operations Directorate was already organized to not only test-fly the Shuttle but to prepare for the coming era of repeated access to space for scientific and commercial purposes. One of the major ways in which the Shuttle was to support these efforts was with the complement of Spacelab hardware—a modular package of systems designed to provide a shirt-sleeve environment in the Shuttle payload bay. The payload operations team would also support all the various experiments with power, cooling, data processing, and other needs.
Our section oversaw all the data processing equipment—the Spacelab Command and Data Management System, or CDMS. If it was doing digital stuff in the Spacelab, it was ours. There were two major computer systems—identical except in function. One supported the Spacelab subsystems, and the other supported the experiments. They were operated in parallel and they shared resources, like display units and interfaces to the command system. Using the command system, the ground could send up instructions to operate the various systems in the Spacelab so that the crew could concentrate on their science. The heart of both of the computer systems was the Input/Output (I/O) unit for each one. While they were internally redundant, there was only one of each, and they were essential to making the Spacelab work. If either the Subsystem I/O unit or the Experiment I/O unit went up in smoke, then half of the CDMS was out of business, and you were done.
There was a computer for each I/O unit, plus a backup computer that could be swung either way. The computers were simple machines with, as I recall, a Motorola 6502 CPU and 128K of RAM. Interestingly enough, the processor was the same as what was used in the Atari 400/800 consumer-grade computer you could buy in the stores in the early 1980s. Back then, the IBM PC had not yet come out, and you had your choice of Atari, Commodore, or the mighty Apple II—the really pricey option that a co-op student like me wasn’t going to be able to buy. In fact, as a co-op student, I wasn’t going to buy anything more expensive than a pocket calculator. It wasn’t until I came on full time that I was able to buy my first Atari—and I did that only so I could teach myself machine language programming. Well, that and play Missile Command, as I recall… But back to the CDMS—there were a number of subsystems I had to learn, but the real thing they wanted me to work on was the Spacelab Instrument Pointing System (IPS).
The IPS was a payload-bay-mounted, gimballed platform to which you could mount astronomical telescopes and science packages that needed to be pointed very precisely. Think of this as a precursor to the Hubble Space Telescope, which was in development but was years off from launching. The idea behind the IPS was that you could mount telescopes to it, carry them up in the Shuttle, do a week or two of science, and then bring them back. The next time up, the IPS could carry improved versions of those experiments, or something entirely new. Because the Shuttle itself was a fairly crude pointing platform, with stability measured in degrees and rather abrupt jet firings that jiggled and jilted the structure, the IPS acted sort of like a Steadicam—sensing its position in inertial space and isolating the telescopes and experiment packages from the Shuttle’s crude motions.
The heart of the IPS was a gyroscope package, which did the basic stabilization. There was also an accelerometer package mounted near the base that sensed motions before they got to the payload, so the system could effectively damp those motions out. There was also a package of three-star trackers that looked out at the universe to find known navigation stars and determine exactly where (in inertial space) the payload was pointing. This combination of sensors provided data to the software that computed necessary commands to the motorized gimbals to keep the whole thing pointed where you wanted—with an accuracy and stability of approximately an arc-second. The entire package was built in Europe—as was the entire Spacelab. NASA had written the original specifications for what the IPS (and the Spacelab) should be able to do, and how it should work—but then it was up to the European Space Agency to design and build the thing. Then it all came back over to our side of the pond, and we had to figure out how to actually make it work for us in space.
This was actually a good working example of what Flight Operations—and flight control in general—was all about. Design engineers built amazing hardware and software to accomplish a mission, but it was up to us to figure out how to make it do what we wanted, and how it could fail. The first step was to do system-level drawings that could help us understand its functionality, as well as provide reference materials for troubleshooting in simulations and flights. A system-level drawing is not really a schematic—it isn’t designed to show actual physical wires—but it is far more complex than a block diagram.
System drawings show how power, data, and commands flow through a black box, or a whole system. These flows give the flight controllers the information they need to watch over the function of the system and figure out where failures might be occurring when things aren’t working as they should. These system-level drawings are done by flight controllers, for flight controllers—and also for the crew and the folks who will train both the crew and the flight controllers. Using the original schematics as a basis, along with requirements documents and long discussions with the original builders, these drawings generally become the basis for all in-flight communication about failures and how systems work. I always said that real value in these drawings was the work that went into producing them.
On my first co-op tour, I was assigned a desk in an office with a couple of Apollo veterans—James Bodmer and Glenn Cress—who were now working on the IPS. Both were experienced flight controllers, and I think they had both been trained as electrical engineers. There I was, a traditional aeronautical engineering student—wings, tails, control surfaces, and engines being my interest and forte—being asked to produce a drawing of an electronic accelerometer package! This package featured solid-state accelerometers and the signal conditioning equipment and power supply circuitry necessary to generate acceleration data and feed it to the computer to use in the overall control loop. Now I wasn’t completely uneducated when it came to electronics—after all, I did have my electronics merit badge from my scouting days, and I had built a few things using transistors, resistors, and capacitors. But here I was suddenly immersed in the world of digital electronics and Boolean algebra—something I had never had a chance to study. I think my introduction (taught by Bodmer) took about thirty minutes. Then I was marched over to a neighboring office where another engineer had a collection of Texas Instruments reference books for all the available chips that were being used around the world at that time. They had each chip listed by number, a drawing, circuit information, an
d—most importantly—a truth table that showed what you’d get on the output side for any combination of inputs. These weren’t programmable microprocessors with a wide variety of possible outputs—these were simply logic gates that made circuits work.
So my job was to study the available schematics of the accelerometer package, along with all the notes and design requirements. I’d then look up the various chips on the schematics and figure out how they worked. I’d then figure out how they were being used in the circuits, and what they were doing—then distill that information down into a box on the drawing that showed what inputs led to what outputs. It was a satisfying game of cat and mouse, figuring out exactly what the designer had intended and how things worked. It was all logic and tracing signals—and it taught me what I really needed to know to understand how systems worked. Before my first six-month tour was up, I had figured out the IPS accelerometer package was a warm-up to tackling the Optical Sensor Package (OSP)—a set of three-star trackers that were pretty complicated (for a rookie). The really fun thing about the OSP was that it was built by Sodern, a French company, and all the drawings we had were in French, a language with which I only had a passing familiarity from working with French diving gear. But technical language is full of English cognates, and once I figured out a few key words, the rest started to make sense.
It was a fun winter, and I learned a lot. But the best part was that, every once in a while, I was able to stick my head up and realize just where I was—I was getting ready to work in Mission Control! As a matter of fact, our division’s offices were on the third floor of Building 4, and the other half of the floor was the Astronaut Office. I could hardly believe that I was sharing the stairs, elevators, and restrooms with our astronaut corps—a heady experience for someone who had been watching the space program since early childhood. I never really thought that NASA would be something I could be a part of. But now here I was, rubbing shoulders with the men and women who were going to fly on the Space Shuttle, and a couple who had walked on the moon.
It was during this first tour of duty that I was shown the training library—essentially a small storeroom filled with shelves and shelves of documents on the second floor of our building. It was here that I fell in love with the idea that I could really learn a whole lot more as a co-op student than as a college student in class back at the University of Minnesota. Here I found a treasure trove of material that I wanted to learn. There was an entire series of workbooks on orbital mechanics. Another series was dedicated to teaching the ins and outs of telemetry and command systems. There was a whole stack of training manuals on how to operate MCC consoles and other hardware and software tools I’d use as a flight controller. And then there were the huge number of workbooks on Shuttle systems—including the massive DPS HW/SW 2101—everything that could be pulled together about the computer systems that ran the Shuttle. It was a couple inches thick, with closely spaced text and lots of system-level drawings.
When I was first introduced to the library, I picked up a few books that quickly thrilled me, and I soon began filling my extra hours—including lunchtimes and evenings—reading those books. By the end of the six months, I had worked my way through almost all the Shuttle systems, all the MCC training manuals, as well as books on the basics of spaceflight. I probably stuffed more new knowledge into my head during that half year than in any other time of my life—but it was all stuff that I wanted to learn, and it came easily because of that.
Mission Control was really a computer system masquerading as a large building. Essentially three stories tall (very tall stories), the first floor was where all the computers and communications systems lived. The second and third floors were control centers—a Mission Operations Control Room (MOCR) in the middle, and smaller support rooms all around. The floors and ceilings were massive raceways for wiring. When you went inside, you really were in a computing complex. By today’s standards, that complex was exceedingly primitive—but in its day, it was truly state of the art. Most of the technology that the flight controller interfaced with was left over from Apollo. The consoles were really dumb terminals in a way—they displayed data on light panels and black and white TV monitors. The controllers could interface with the Mission Operations Computer (MOC) on the first floor with educated buttons that called for specific functions to occur.
Changing console functionality really meant writing a set of specific definition requirements and sending them to the organization that configured and ran the building. After a long approval process, someone pretty much had to go out with a soldering iron and make new connections from the consoles to the MOC. Specific consoles had unique buttons that could tell the computer to initiate particular pieces of computer code. Nothing was really reconfigurable in real time. You could select a function for a specific event light panel by dialing in a code that told the MOC what program to run. The MOC would then activate a certain set of lights on the panel. Then you had to pull an envelope of films out of the console drawer, select the film for the configuration you had selected on the MOC, and slide it into a holder in front of the lights. If you got the wrong film, the lights you’d see would make no sense at all.
Some of the control buttons on each console were standard—a panel to select TV channels and display numbers, for instance, was basic. Every controller had one. You could use it by dialing in a number—one digit at a time—and pushing a button to tell the MOC that you wanted to look at a specific display, or you could peer in on a specific display channel that someone else had already selected. At one time, there were eighty channels available in the building, and each console position had a certain number allocated to them.
So a particular controller might bring up their main display, and the MOC would put it on an empty TV channel. Anyone in the building could now look at that display by bringing up the channel. The console that originally selected the display had control of that channel, however—so if they flipped to something else, anyone watching that channel would see whatever new display was brought up. As the space vehicles got more complex, the number of parameters to be monitored went up. That meant that the total number of displays went up. But the number of available channels remained the same—making for full channels. This would force the Flight Director to ask, “All flight controllers, drop to your allocated channels!” You see, each discipline was allowed the exclusive use of a certain number of channels for any flight phase, but they were on the honor system to stay within that allocation. What do they say about honor among thieves? Well, it was pretty common to “accidentally” go over your allocation, especially when you got busy solving problems. Then someone would complain, and Flight would “request” that everyone check how many they were using. If that didn’t create space, the channel police would start calling every discipline one by one until they got everyone back in bounds. Of course, that took time, and if you just needed to check that additional display for a few more seconds… you were unlikely to get caught. It was a game, and some played it pretty well.
The MOC was sort of a virtual thing—there were five computers located on the first floor, any one of which might be the MOC on a specific day or simulation (sim). The other computers might be supporting other flight activities, a sim, or development work—and they got used as required. In the simplest terms, the data stream from the vehicle was fed into the MOC in a serial stream—all the bits in a single line. Imagine, for instance, a bunch of eight-bit words, all lined up in a row. Each word represents a measurement. As the data stream came into the MOC, it was “chopped up” like a bunch of sausages coming off an assembly line, and each measurement was stuffed into its own little mailbox. The various programs running on the MOC could then say, “Hey, I need the cabin pressure measurement from sensor number one,” and they would go to that mailbox and get the latest value of the measurement and use it for its computation cycle. The mailboxes were refreshed at various rates. Essentially, there was a 128KB stream coming into the building for regular Orbiter telemet
ry, and some parameters were updated every second, some at broader intervals, and some more frequently, such that they updated multiple times in each second.
The telemetry formats could be easily changed to support different flight phases. For instance, most of the ascent flight control data were unnecessary once the vehicle got on orbit, so you switched to a format that supported orbit operations. The key was that when the Orbiter changed format, the MOC had to follow—basically to make sure each side was using the right “decoder ring” so to speak. It took quite an army of people to keep the whole thing running and in sync, which was one of the many things that changed as the Control Center was modernized and modularized. The once-busy back rooms of the ground controllers (the folks who made the building work) became empty rooms of quiet computers, with nary a tech to be found. But that was much later in the program—in the early days, when I was a fresh-faced controller, the ground controllers making it all work probably outnumbered the flight controllers!
