Mission Control
AT 8:15 A.M. EASTERN TIME ON SATURDAY, FEBRUARY 1, THE space shuttle Columbia , 175 miles above the Indian Ocean, fired its engines to begin its hour-long descent to Cape Canaveral. With its onboard computers closely monitored by the Mission Control Center in Houston and directing its maneuvers perfectly, the seven men and women aboard had little to do but watch their instrument panels. Over Hawaii, the craft began to glow pink as parts of its surface heated to as high as 3,000 degrees—still all normal. The first sign of trouble appeared as Columbia passed over California. Four temperature sensors in the left wing failed, and temperatures in a brake line began to rise. A minute later, over Nevada, temperatures on the left side of the fuselage went up. At 8:59, Houston and the crew began discussing what looked like a handful of glitches.
The Houston communicator, Charlie Hobaugh, said, “ Columbia , Houston, we see your tire-pressure messages and we did not copy your last.”
The shuttle’s commander, Col. Rick Husband, calmly replied, “Roger, uh….”
After that, in the words of one aerospace expert, “the data bank in Houston” became “the biggest black box in the world.” Mission Control has been dealing with near-disasters—and, much more rarely, disasters—since space flight began. But except in the extraordinary case of sudden catastrophe as happened with Columbia , a spacecraft doesn’t face it all alone. It’s not like science fiction. If a disaster occurs in science fiction, as it inevitably must, the crew has to cope without help. Real manned space flight hasn’t been like that and won’t be for decades or centuries. Real missions require constant contact with trained personnel back on earth who monitor and command the spacecraft, assist the crew with their planned tasks, help them troubleshoot when problems occur, and generally provide a lifeline of communication and expertise to the astronauts. This indispensable role is what the MCC was built for.
A flight-control or mission-operations team is, in the words of the former controller Sarah Kirby, “an orchestra trying to perform a single piece of music called a space flight.” Under-lying that orchestra is a philosophy, established through decades of success and tragedy, that Failure Is Not an Option. Milt Heflin, chief of the Flight Director’s Office at NASA’s Lyndon B. Johnson Space Center (JSC), of which the MCC is a part, says, “Whatever the problem, however large or small, the flightcontrol culture says that we have to fix it and do it right.” The MCC has been doing things right for more than 38 years, expanding knowledge of innumerable engineering and scientific fields along the way.
The MCC had not yet been built when NASA’s first manned space flight program, Mercury (1961-63), took place, so Mercury was controlled from the Cape Canaveral launch facility (later renamed the Kennedy Space Center). Canaveral could support the simple Mercury flights—launch, brief suborbital or orbital missions, and touchdown—but it wouldn’t be able to handle the complexities of the upcoming Gemini and Apollo missions. Bales recalls those early days: “The only way we could get trajectory parameters was to call Goddard Space Flight Center [in Maryland] by phone. Even then, Goddard didn’t have all the software capability required for rendezvous.”
The MCC learned a lot from the experience of Cape Canaveral, but while the Florida facility was an invaluable training ground in many ways, it had no on-site computers or terminals. Controllers read data directly from strip-chart recorders and analogue meters. So when flight control moved to the MCC in Houston, with its powerful mainframes, controllers had to learn to use display consoles when most of them had never even seen a computer.
When it opened in March 1965, the new MCC was a monument to American technological prowess. It had the most powerful computing complex the world had ever seen, with five IBM 7094s. These mighty mainframe computers sat on the ground floor, controlling all the major and minor systems in the facility. Everything that appeared on the controllers’ consoles was driven directly by one of the mainframes, which was designated the Mission Operations Computer (MOC). There was always a second mainframe running the same software as a “hot backup.” If anything went wrong, it could take over in seconds.
It is common to read that some past computer system, awe-inspiring in its time, had less processing power than a typical desktop model today. That is true of the MOC—a 7094 had about twice as much memory as a Commodore 64 from the early 1980s—but even more important to the lives of the programmers was its severe lack of flexibility.
User-friendliness was not even a concept yet. There were no images or icons; instead, each console had one to three small green screens that displayed rigidly defined and tightly crowded rows and columns of white letters and numbers. Each controller was assigned to a specific spacecraft discipline, like guidance and propulsion or computer systems. With its tiny amount of computing power, all the system could do was display the data that was transmitted from space. Any analysis of that data had to be performed by humans.
The displays were not created electronically, as they would be today, but mechanically. For each display there was a static background, which was stored on a slide. Dynamic data was processed by the MOC, mixed optically with the background, and sent out to the displays. The system was far from perfect. According to Jim Stokes, who headed the behindthe-scenes group that programmed and maintained all the MCC’s computers, “When you dialed up the display number, the computer would go out, pick the right slide, and a mechanical arm would get the slide into place. It jammed all the time.”
Displays were programmed in assembly code, which consisted of strings of ones and zeros, and changing them required a huge effort. Even more troublesome were the event lights, which lit up when something happened that required a controller’s attention, and the command buttons, which the controller pushed to send orders to the spacecraft’s systems via the MOC. Stokes says, “Each event light was hard-wired to the computer. If the controllers wanted to change event lights, it required going in and rewiring thousands of event wires. It took months.” The rewiring was a tedious, never-ending process, especially in the Apollo program, which kept changing its requirements.
“It was incredible,” says Bales. “Each mission in Apollo was different. Apollo 8 required a translunar trajectory, which wasn’t needed before. Apollo 9 required landing-module trajectory, but not translunar trajectory. Apollo 10 needed translunar trajectory and landing and command-module trajectory processing, but not the final-descent software. Apollo 11 needed it all. And 12 had requirements for a precision landing.”
Also, while flight control for previous missions had focused on real-time events, the Apollo flights required more preflight planning for what-ifs or system failures or emergencies. These varying missions placed great demands on the programmers of the MOCs, who were very much a distinct group from the users. According to Bales, “We hardly ever met the people who did the coding.” He would write down the requirements for software and “never worry about them again until we flew the mission.”
In place of the obsolescent IBM 7094s, says Stokes, for Apollo “we got new IBM 360775s, and the first two had serial numbers 1 and 2.” But even the new computers weren’t fast enough: “We were constantly tweaking the machines for faster performance…. For example, if one parameter only needed an output every five seconds but another needed an output every two seconds, we would reorder priorities to get the crucial data output first.”
Sometimes the need for data processing was so great that it exceeded the capacity of the mainframes. The computer code for the lunar missions was two million lines long, all written in assembly language, and even that wasn’t enough. “We had a few unique but necessary applications, like the descent fuel computation for the lunar landing,” says Gene Kranz, who made history as the flight director for Apollo 11 and Apollo 13 . “By the time we had the data, we didn’t have time to get it in the mainframe.”
The solution was to obtain two small Olivetti computers to perform the calculations. These were kept in the Flight Operations Directorate offices in Building 4, a short walk from the MCC. “We pushed them in a market basket between Buildings 4 and 30,” Kranz says. “One day as we were preparing for a mission, the market basket hit a crack in the sidewalk, sending a computer crashing to the pavement. These computers weren’t easy to come by. I had to go to the manufacturer and get one shipped straight from the assembly line. I designated it as mission critical, and it took about a week to get it.”
THE REASON NASA WAS IN SUCH A HURRY TO GET TO THE moon was that it was racing not only the Soviet Union but President John F. Kennedy’s arbitrary deadline of the end of the 1960s. One factor that helped NASA deal with the pressure was the presence of a number of recent college graduates who had worked with computers in laboratories. The National Defense Student Loan Program let federal facilities hire from specified parts of the country. “Each of the new NASA space centers was assigned a recruiting area and JSC’s was mostly in the Midwest,” explains Kranz. “The NDSLP provided college opportunities to a new group of young people. We usually got the first in the family to go to college. They were from farms, with a dawn-to-dusk work ethic, and they could fix anything. They had marvelous talent, and we mixed them with the Mercury veterans.”
Those veterans came from the military flight-test community (like Kranz) or had other all-around technical experience (like Ed Fendell, who led remote site teams for both Gemini and Apollo). Kranz had been a fighter pilot and a flighttest engineer, while Fendell, who had graduated from junior college with an associate’s degree in merchandising, wound up at NASA by way of air-traffic control in military and commercial aviation and launch and re-entry support at Cape Canaveral with a contractor.
With Gemini missions every couple of months and the forthcoming Apollo flights, the pace was brutal. “People worked 16,18, 20 hours a day,” Fendell says. “It was such an experience that if you didn’t want to do that, you disappeared. If you screwed up and you didn’t want to admit it, you disappeared. We just worked. We didn’t think about it. We were wrapped up in trying to do the job. We weren’t racing the Russians; management was racing the Russians, and that set the schedule, but the working troops didn’t think about that. We didn’t even know if we got paid overtime. It was fantastic.”
The launch of Gemini 3 on March 23, 1965, marked the MCC’s first “live” operation as a backup for Cape Canaveral. The control centers’ positions were reversed for Gemini 4 in June, after which the Florida center was decommissioned and the MCC became NASA’s sole control center for manned space flight (though Canaveral continued to handle all launches).
The original monolithic MCC, also known as Building 30, was three stories high, enclosing the flight-control rooms, back rooms, and computers. The famous, oft-televised room with the big screen in front is the Flight Control Room (FCR, pronounced “ficker”). It is also known as the “front room.” There were actually two identical FCRs, one on each of the second and third floors. One FCR was used for missions while the other was used for training. Warrens of smaller rooms—the back rooms—surround the FCRs. Junior flight controllers staffed these during missions, supporting the senior personnel.
Data processing was not the only technology that limited the MCC’s ability to do its job. Another was communication. The two primary reasons for the MCC’s existence are telemetry (collecting signals from the spacecraft) and command (sending signals to the spacecraft). Both these functions involved a far-flung network of relay stations. Today we think nothing of instantaneous communication from the other side of the planet, but when the MCC began, there were no cell phones, virtually no communications satellites, no fiber optics, and, in many places, no ordinary phone lines.
Gene Kranz recalls: “Global communications consisted of submarine cables to London, Hawaii, and Australia. From there a 60-word-per-minute Teletype network, which dated from the days of America’s Pony Express, stretched to tracking stations in Bermuda, the Canary Islands, Africa, Australia, two tracking ships, and Canton Island in the South Pacific Ocean. Communications was a daily crapshoot. Interruptions were often caused by weather, solar activity, mechanical failure, and local construction.”
Since each mission needed human observation and command throughout the flight, even when the spacecraft was on the other side of the world from America, the only solution was to send small flight-control teams to remote sites like Carnarvon, Australia (600 miles north of Perth), the Canary Islands, and Nigeria. The remote sites and ships were equipped with antennas to track the spacecraft’s brief overhead flight, receive telemetry, and send commands and voice communications. The teams included a “capcom” (who led the remote team and spoke directly with the astronauts), two systems engineers, two doctors, and sometimes an observer.
As soon as the antenna “acquired” the spacecraft, controllers sprang into intense action. “The passes lasted 4 to 12 minutes for Mercury and Gemini,” Fendell says. “We’d get the data, send commands, do voice contact with the crew, and talk via voice and Teletype to Houston.” The remote crews had no computers; their consoles were equipped with strip-chart recorders and analogue meters, as in the old control center at Cape Canaveral. Controllers marked meter positions at specific times with colored grease pencils. After the overhead pass, controllers collected strip charts, notes, and meter readings, analyzed them, and sent formatted summaries to the MCC —via the routes Kranz described—by 60-word-perminute Teletype.
The MCC’s communications system was upgraded for Apollo with a trio of Univac 494s, and the remote sites got computers that could automatically process telemetry and commands and send the telemetry to the MCC. The remote sites also got better antennas and newer, faster communications links. This allowed the remote flight-control teams to return to the MCC after six unmanned Apollo test flights.
Working at the MCC had its ups and downs. Gemini was an exhilarating success, but the Apollo 1 fire that killed Ed White, Gus Grissom, and Roger Chaffee is still a quiet, painful memory for anyone who was in the program. Kranz explains how the MCC pulled through: “With Mercury, people were used to flight testing. They had seen pilots die, and they knew we could have a ‘bad day.’ They were aware of it. It let us, the leaders and managers, carry the young people forward.”
The fire created a yearlong delay while the pure-oxygen atmosphere of the spacecraft and other design problems were changed. Afterward the program returned to its hectic pace for both flight controllers and programmers. Says Stokes, “We got a call from Chris Kraft (the director of flight operations for Apollo), who said, ‘On Apollo 8 , we’re going to the moon.’ We had planned to have that software ready several flights downstream. Initially we thought it was impossible, but I said, ‘OK, we’ll try.’ We worked day and night, around the clock, many times. In those days, if you took a Sunday off, you felt guilty.”
All that hard work paid off when Apollo 11 landed on the moon—and, later, when Apollo 13 didn’t. Tack Knight, an Apollo flight controller, recalls the moment when Apollo 11 touched down: “I was in the back room with my team, and it was a moment of incredible joy and pride and elation. Then we went back to the business of looking at everything in our systems to be sure nothing was happening to cause a problem, just like we had been trained to do. We had two minutes to make up our minds to whether it could stay or not until the command module came back around.”
WHEN ONE OF THE MAIN OXYGEN TANKS EXPLODED on Apollo 13 , Knight was at a softball game after his shift in the MCC. “I heard about it as I drove home but thought perhaps it was news-radio overreaction. I called in as soon as I got home and was told it was ‘pretty serious.’” Of that harrowing ordeal, Knight recalls: “It was an incredibly long time from the start of blackout until someone saw the parachutes, but it was perhaps the most emotional moment I have ever had in my professional life, comparable to the Apollo 11 landing, but in a different way.” Other events that Knight includes in his career highlights are “ Apollo 16 , when Charlie Duke tried to do a somersault on the lunar surface and landed on his head,” and STS-I (the first space-shuttle mission in 1981).
Mission Control’s computer systems had included the best and latest hardware available at the time they were installed. As Apollo ended and NASA moved on to other projects, however, more and more tasks were grafted onto the original systems. The MOCs were replaced again in 1976, but to maintain continuity, the form of the displays remained the same through Skylab, the Apollo-Soyuz Test Project, and the early days of the space shuttle.
But their number increased dramatically. By the time of the space-shuttle program, there were 104 displays, which were needed because the shuttle had many more systems than earlier spacecraft. This was reflected in its 4,000 direct telemetry parameters, as well as 12,000 derived parameters, that controllers needed to monitor (though not all at once), compared with 97 for Mercury, 338 for Gemini, and 403 for Apollo. With all these parameters being monitored, getting the data from space to earth, processing it, and sending it where it was needed became an increasingly monumental task.
Skylab, launched in May 1973, sent its data at a rate of 250 kilobytes per second (kbps). The fastest communications circuits available at the time could handle 19.6 kbps, so the MCC’s communications and computing equipment risked being overwhelmed. The solution, which has since become a standard technique in data processing, was to “compress” the data by comparing each parameter with its previous value before sending it to the computers. If the parameter had not changed, it was not transmitted. Says Stokes: “The approach was very innovative, and it was quite a challenge to solve the timing and dynamics of the data compression.”
As monitoring became more complicated, so did the task of changing the flight-control software. By the early 1980s it took an average of 35 days to change one line in a display, since each display change required reprogramming the mainframes and rerouting many wires. More extensive changes could take years. Sarah Kirby joined the MCC in 1983 to work on the shuttle’s propulsion system. Her group submitted a request for a calculation to derive the fuel leak rate, a number they had been calculating by hand.
“We’d submit requirements, go through the Change Board, and get the requirements approved,” she says. Then came a six-month period in which programmers would work on the requirements without any interaction with the controllers. “Invariably, an error would be made in the interpretation of the requirements. So it took several iterations to get it right.” Kirby decided to start discussing the project regularly with the programmer working on the leak-detection calculation. “I figure that I saved about a year this way,” she says. The calculation was finally delivered in 1985.
The greatest calamity in NASA’s history before this year came on January 28, 1986, when the space shuttle Challenger exploded shortly after takeoff, killing its seven-member crew. For the personnel of the MCC it was a tragedy but not a crisis, since once the explosion happened, there was nothing they could do. John Muratore, who was working “on console” at the time, recalls the aftermath: “People’s response was to go to work on the problem. This is pounded into you: to go to work on the problem . It’s like a car slipping into gear. But with the Challenger , there was no place to go.”
At first, the MCC staffers were not sure what had happened: “During a flight we don’t have the TVs on except for a few places because we want to focus on the data and not get distracted by the video. When we saw no data coming down, we switched to video. Almost everyone expected a vehicle to come flying out of there.”
They did what they could to ensure that they would be able to learn from the event: “All the logbooks, every command, everything was locked down to make sure nothing was lost…. Then we looked at the data for days to see if there was any hint or clue. There just wasn’t. Some operational issues are controlled operationally; some are controlled by design. We don’t train for scenarios that we can’t do anything about.”
MCC controllers used the downtime after Challenger to develop, justify, and document the rules and procedures that would be used during the space-shuttle program. Another important event during the two-year layoff after Challenger was the replacement in 1986 of the IBM 360s with a new suite of mainframes. These new computers were upgraded in 1987. But perhaps the biggest change during this period was the entry of workstations and desktop computers into the MCC for the first time.
SOME CONTROLLERS, FRUSTRATED WITH THE BUREAU -cratic back-and-forth of software development, were already taking advantage of the growing power of desktop computers to write their own applications and design their own displays outside official channels. They set up free-Standing offline personal computers next to their consoles, entered data from the consoles into the PCs by hand, and ran it through homemade software to calculate key parameters. It was a makeshift solution, but once controllers got a taste of being in charge of their own applications, there was no going back.
Even before the Challenger disaster, the MCC’s bosses had begun their own official effort to integrate workstations. This project was led by Pat Duffin. The original intent was to completely replace consoles with workstations, but after the tragedy, Duffin says, “it was decided that the risks of change at the time NASA was re-engineering the shuttle’s solid rocket boosters and everything else was too great.” Those who had been writing software for unapproved PCs started writing it for the new workstations, but with the aim of supplementing the mainframe/console system rather than replacing it.
This new approach promised to speed things up, but old hands saw danger. The creaky development system certainly had room for improvement, but there was a reason it was in place. Says Kranz: “Mission controllers’ decisions are time-critical and irreversible. The young controllers, often fresh out of college, did not yet understand the risks of using software that had not been independently verified and tested over a wide range of mission conditions. We do our work in full view of the world, and our mission actions are closely scrutinized. An error that led to aborting a mission would lead to standing up in front of Congress to explain our actions. There are no excuses if we are wrong.”
All these changes reduced reliance on mainframes and consoles somewhat, but controllers continued to use the consoles for much of their work. As flight-control tasks became more complicated, however, the old system of displaying data as columns of numbers became too primitive to continue. For example, the space shuttle’s remote manipulator system (RMS), or robot arm, has six joints—two at the shoulder, one at the elbow, and three at the wrist. It operates basically the same way as a human arm, except that it’s 50 feet long. Controllers need to apply intense physics and mathematics to find the right instructions to move the RMS. It’s the kind of problem where a minus sign can be critical, making the difference between banging a payload into the shuttle’s cargo hold or gliding it into open space. As long as the MCC relied on its 1960s-era consoles, RMS controllers had to take the jumble of angles and distances that appeared on their screens and convert them into three-dimensional motions in their heads.
The solution was to eliminate the consoles completely from flight-control displays. The leader of this effort was John Muratore, a communications officer who had previously worked at Kennedy Space Center in Florida. The KSC had an impressive automated launch-monitoring system, and Muratore encountered quite a difference when he came to the MCC. “MCC had greater computational ability, but less monitoring capability,” he says. “I sat there for hours and hours going through simple checks and comparisons that I knew a computer could do. I wanted to free up my time to do more thinking tasks.”
Under Muratore’s direction, a small group of controller/programmers created real-time displays that featured graphics and color. RMS controllers, for example, could now see the shuttle’s robotic arm moving in three dimensions. The MCC still depended on its mainframes for processing telemetry and calculating and sending vehicle commands; indeed, the mainframes were upgraded yet again in 1989 specifically for these tasks. But for user displays, at least, the MCC had finally entered the modern world of distributed, workstation-based computing—just in time for the MCC’s next and last era.
That final era began in 1990, when construction started on the new five-story, 60,000-square-foot Space Station Control Center (SSCC). The plan was for the new SSCC to handle spacestation flights while the MCC continued to support the space shuttle. The SSCC would have it all over the MCC, with the most modern facilities in computing and communications and no mainframes. However, the space-station program was not politically popular in Washington, and Congress kept cutting its budget. Finally, in 1992, a 40 percent cut made it impossible for the program to have its own control center. Instead there would be a dual-use Control Center Complex (CCC).
The CCC took the place of the MCC, but it could never have opened on time without the lessons about software development learned in the old facility. Instead of the hierarchical, step-by-step “waterfall” approach, a new “spiral” method was put in place. Muratore explains: “Our approach to requirements was that you don’t have to write everything down. Get a high-level statement of the requirement and have the contractor write the code. Test the heck out of it, find the problems, and provide fixes back to the contractor. The contractor fixes it. Document it after it’s built.” This procedure reduced development cycles from an average of two to four years down to five or six months.
Every piece of hardware, and much of the software, in the new CCC was a standard commercial product, rather than the custom-built machinery and programming that had characterized the MCC for so many years. The system was much more flexible because its logic was embodied in the software, which could be changed easily, rather than in the hardware. The CCC began operations in December 1994 in backup and simulation modes. It took over as the sole control center in May 1996, at which time its name reverted to simply Mission Control Center. The new MCC has two separate Flight Control Rooms, the White FCR for shuttle operations and the Blue FCR for space-station operations.
But the old MCC was not quite through yet. By the time of the takeover, more than three million lines of code had been transferred from the old MCC’s mainframes to the new MCC’s workstations, but there was some that hadn’t yet managed to get there. It took another two years to convert the trajectory code the flight controllers use to send commands to the space shuttle.
Trajectory includes a lot. For the shuttle, it’s basically everything related to a flight except the launch itself. For the space station, trajectory includes boosting the station when it dips too low into the earth’s atmosphere, avoiding orbital debris, and more.
“Changing the trajectory subsystem architecture carries inherent high risk,” says Ron Epps, chief of flight design and dynamics in the MOD (Mission Operations Directorate). “It is loaded with extremely complex math and associated logic that must meet real-time constraints for critical-phase highspeed data processing. It has about a million lines of computer code that is made up of more than 75 number-crunching applications. To pull the project off, we had to have skills in systems engineering, aerospace engineering, and physics, as well as assembly and C programming.”
Assembly language was, of course, the first programming language developed, back in the days of the earliest computers. The number of people who can still program in it is rapidly diminishing. The main goal was to convert the assembly trajectory code to C, a modern language, but project leaders took the opportunity to improve many of their trajectory applications.
In early 2002, all the rehosting and certification of the trajectory software—except the lunar mission code, which has been archived—was completed. The new code was completely tested and fully operational by October 2002. The MOCs had finally been unplugged. The old MCC is now a National Historic Landmark and is regularly visited by tourists. The thirdfloor FCR, where flight control for Apollo 11 was conducted, has been restored to just the way it was in 1969, when the Eagle landed on the moon.
The original MCC never delayed a flight—or failed to support one—in its more than 30 years of existence. The closest it came to a failure was in November 1982, during the shuttle flight STS-5, when a main power bus burned in half, taking a number of critical systems with it. Flight controllers had only voice communication with the shuttle crew for an hour until the MCC engineers and technicians fixed the problem.
After nearly four decades, with a new building in place and countless technological generations having passed, is the original MCC spirit still there? Jack Knight thinks it is: “The flight-control culture is maintained, as in most organizations, through the leadership, the stories, the discipline groups, and the flightcontrol-team simulation and flight performance. The certification process, including the on-the-job training process and the ‘tribal knowledge’ of the groups, is also a strong contributor. ” Times may have changed, but “the hardware and software are serenely unconcerned about diversity, political correctness, and anyone’s self-esteem.”
IN MURATORE’S VIEW, CONTROLLERS ARE MADE AND THE MCC’s spirit is maintained in the crucible of its grueling simulations: “When I came to Houston, I knew I wanted to work in flight operations. But I didn’t understand why there were so many sims. During the sims we kept getting hit with five or six failures, and it didn’t seem realistic. A couple of months later, I was working, and I had a bad cold. When that happens, the whole discipline gets sick at once. I realized that the strength of the system was that you were trained so well that you could come in during the middle of the night and be sick and still do an excellent job.”
Muratore sums up the flight controller’s ethic: “The equipment is just a tool—just a matter of how much work we can expect people to do. The human side of flight control has been, still is, and always will be the relatively junior person in the middle of the night, whose car broke down coming to work, who ate a rotten burrito for lunch, and who is still sitting there doing an excellent job because he really cares about the people and the space mission. He has stars in his eyes, he is devoted to human space flight, and he is prepared to do whatever it takes to make the mission succeed.”
