Jon Krosnick - Work in Survey Methodology - National Aviation Operations Monitoring Service

National Aviation Operations Monitoring Service

The National Aviation Operations Monitoring Service (NAOMS) was an $11.5 million research and development project by NASA using survey methods to measure aviation safety. The program was created in response to the goal set by the White House Commission on Aviation Safety and Security in 1996 to reduce the risk of air travel accidents by 80 percent over the next 10 years. Krosnick was the lead consultant in developing and implementing the NAOMS suvey methodology.

While plane crashes remain rare, NAOMS sought to identify and reduce accident precursors and potential safety issues by regularly surveying commercial pilots, general aviation pilots, ground and flight crew members, and air traffic controllers. The project was designed to provide broad, long-term measures on trends and to measure the effects of new technologies and aviation safety policies. The project implemented a survey with an 80 percent response rate, interviewing a random sample of pilots about safety incidents.

In 2004, NAOMS researchers finished collecting data on the first cohort of pilots, having conducted about 24,000 interviews. To some observers, preliminary findings suggested that some safety-related problems were occurring at astonishingly high rates, in some cases as much as four times the amount previously reported by the FAA. The FAA was “extremely unhappy” with the results and called for the program to be shut down. NASA soon cancelled the program. The House Committee on Science and Technology Subcommittee on Investigations and Oversight later investigated FAA's role in the ending of NAOMS. Chairman Brad Miller (D-NC) stated the subcommittee found that the FAA did not support NAOMS. In 2006, Associated Press reporter Rita Beamish filed a Freedom of Information Act request for the NAOMS’s data. For 14 months, NASA rejected the request.

In a final denial letter to the AP, Thomas Luedtke, senior NASA official, indicated the data would not be released because the findings could damage the public's confidence in airlines and affect airline profits. Luedtke acknowledged that the NAOMS’s results "present a comprehensive picture of certain aspects of the U.S. commercial aviation industry." Significant criticism from the public over NASA’s refusal to release the data and its handling of NAOMS prompted Congress to launch an investigation into the matter. Members of Congress from both sides were very critical of NASA’s handling of the matter and demanded NASA release NAOMS’s results. During an oversight hearing, NASA's administrator, Michael D. Griffin, testified that Luedtke’s reasoning was a mistake and NASA would release the data. However, Griffin cast doubts on the reliability of the NAOMS’s data, cautioning that the data was never validated. Griffin warned, "there may be reason to question the validity of the methodology." On January 1, 2007, Griffin released some of the NAOMS data.

Many refuted Griffin's criticism and defended NAOMS. The NAOMS survey methods were extensively peer reviewed, and the methods were adapted from proven survey methods. Krosnick and others had used such methods in similar contexts in published scientific studies that had been extensively peer reviewed. In addition, NAOMS had also been thoroughly reviewed by internal and external experts. The International Federation of Professional and Technical Engineers (IFPTE) sent a letter to Congressman Bart Gordon, then chairman of the House Committee on Science, Space and Technology stating, “there was no valid scientific basis for the Administrator's technical criticism of the NAOMS project.” In a National Academy of Sciences 2004 report, NAS officially recommended “NASA should combine NAOMS methodology and resources with the ASRS program data to identify aviation safety trends." After thorough review, the Office of Management and Budget, which reviews all federal survey projects to ensure they are optimally designed, approved NAOMS. The union representing the majority of commercial pilots in the United States deemed NAOMS “tremendously valuable.” In 2009, the Government Accountability Office investigated the NAOMS survey methodology and found “the project was planned and developed in accordance with generally accepted principles of survey planning and design... as a research and development project, NAOMS was a successful proof of concept.”