Airline Quality Ratings: Can You Trust Them?

A nationally recognized reporter, writer, and consumer advocate, Ed Perkins focuses on how travelers can find the best deals and avoid scams.

He is the author of "Online Travel" (2000) and "Business Travel: When It's Your Money" (2004), the first step-by-step guide specifically written for small business and self-employed professional travelers. He was also the co-author of the annual "Best Travel Deals" series from Consumers Union.

Perkins' advice for business travelers is featured on MyBusinessTravel.com, a website devoted to helping small business and self-employed professional travelers find the best value for their travel dollars.

Perkins was founding editor of Consumer Reports Travel Letter, one of the country's most influential travel publications, from which he retired in 1998. He has also written for Business Traveller magazine (London).

Perkins' travel expertise has led to frequent television appearances, including ABC's "Good Morning America" and "This Week with David Brinkley," "The CBS Evening News with Dan Rather," CNN, and numerous local TV and radio stations.

Before editing Consumer Reports Travel Letter, Perkins spent 25 years in travel research and consulting with assignments ranging from national tourism development strategies to the design of computer-based tourism models.

Born in Evanston, Illinois, Perkins lives in Ashland, Oregon with his wife.

See recent posts by Ed Perkins

Want to receive stories like this every day? Subscribe to our free Deal Alert newsletter.

AirTran edged Hawaiian out of the top spot in this year’s annual Airline Quality Rating (AQR) scores. And the combined quality score for the country’s 16 largest airlines again crept up a bit over the previous year. Nevertheless—although AQR doesn’t indicate it—airline travel remains a miserable experience most of the time.

AQR scores are a composite of four objective measures: on-time performance, how many travelers were denied boarding (“bumped”), how many bags the line mishandled, and how many complaints were logged by the Department of Transportation (DOT)’s consumer program, all adjusted to a consistent per-passenger index. Because three of the four factors are negative (the higher the number, the lower the quality), all AQR scores are negative numbers. Some highlights:

The complete list, top to bottom, is AirTran, Hawaiian, JetBlue, Alaska, Southwest, US Airways, Delta, Continental, Frontier, American, SkyWest, United, Mesa, Comair, Atlantic Southwest, and American Eagle. Other lines are too small to provide adequate data.
As before, regional lines score better than the giants—the top four are all relatively small.
Again as before, five of the six lowest scoring lines are commuters, with United the lone giant line in that dubious collection.
Scores for the middle group—Southwest to American— show a relatively small spread.
Top lines for on-time performance are Hawaiian, Alaska, and United; the bottom three are Comair, JetBlue, and American Eagle.
JetBlue and Hawaiian bumped fewer than five per million travelers; American Eagle bumped 402 per million and Mesa and Frontier bumped more than 200 per million.
AirTran mishandled only 1.63 bags per 1,000 travelers; American Eagle mishandled 7.15.
Southwest, Mesa, SkyWest, and Atlantic Southeast generated fewer than 6.1 complaints per million travelers; Delta generated 20.

Current-year AQR scores appear to be a reasonably good predictor of future performance. Although the order of scores changed somewhat from 2009 figures, both winners and losers tend to be fairly consistent from year to year.

Nevertheless, I have a big problem with the AQR scores. As I’ve noted in previous analyses, “quality” for any product or service has two distinct components: how good the product is, as promised, and how well the supplier delivers on what was promised.

The AQR scores reflect only the delivery component of quality—how closely each airline comes to meeting basic obligations. Thus, for the most part, the scores reflect more the absence of problems rather than any positive experience. They have nothing to say about the many other factors that affect how much you might enjoy a flight—seat space, onboard service, in-flight entertainment, and such—nor do they account for the generosity of the various lines’ frequent flyer programs or the presence/absence of annoying fees and charges.

The problem with delivery-based scores is, of course, that an airline admittedly offering a lousy product—Ryanair is the prototype—could earn a top AQR score by delivering its lousy product consistently.

From the beginning, AQR has touted its scores as unique in that they’re based entirely on “objective” statistical data. True enough—many of the “how good is the product” quality features would be difficult to measure without resorting to surveys, with their unavoidable biases. Nevertheless, when I select an airline, those “how good is the product” factors far outweigh the AQR factors, especially given the relatively small spread between top and bottom AQR scores. And for the “how good” element of quality, my take (and that of many surveys) is that JetBlue offers by far the best flight experience of any large domestic airline. Its high AQR scores are icing on the cake.

The annual AQR release usually generates more attention than I’ve seen so far, probably because of the current focus on Southwest’s 737 problems. Even so, you’ll probably still see quite a bit about AQR. And when you do, keep in mind that AQR measures only one part—and, to me, the less important part—of the overall airline quality picture.

We hand-pick everything we recommend and select items through testing and reviews. Some products are sent to us free of charge with no incentive to offer a favorable review. We offer our unbiased opinions and do not accept compensation to review products. All items are in stock and prices are accurate at the time of publication. If you buy something through our links, we may earn a commission.