Subjective vs Objective Debate

INTRO: Given the strong reaction to some of my articles here are some thought’s on what’s likely behind the more emotional responses. The world of high-end audio can be almost religious and divided something like the Republicans and Democrats. In this case it’s the Subjectivists versus the Objectivists. It’s been called the “Great Debate” a “Holy War” and more. (photo: Francis Carnauba)

HIGH-END AUDIO’s DIRTY SECRET: What if I told you there was a proven way to evaluate gear using your own ears that highlights even the smallest audible differences between two pieces of gear? Compared to typical listening methods it’s been found to be far more reliable and revealing of subtle differences. More on this later!

THE SUBJECTIVISTS: The hardcore Subjectivists trust their own ears above all else and often ignore, downplay, or sometimes even actively discredit objective efforts. Some argue they have superior hearing and/or listening skills and more refined tastes. That sometimes creates at least a whiff of an elitist “club” that some are drawn to (think Robb Report). But, regardless, their genuine passion for audio is to be admired. And I believe at least some of them do have superior listening skills compared to the Average Joe. Despite their more emotional left brains, which might imply a greater love of music, there’s some consensus Subjectivists spend more of their time tweaking and evaluating their hardware than a typical objectivist. Stereophile’s Michael Fremer is generally considered a strong subjectivist.

THE OBJECTIVISTS: This group tends to prefer some sort of science, measurements, or objective listening tests to back up claims of “A is better than B”. When reading a gear review they’re more likely to skip to the measurements section (if there is one) than read subjective impressions. They tend to be skeptical of outrageous claims and ultra high priced gear. They also tend to buy less expensive gear, less often, than subjectivists making them less attractive to manufactures. As mentioned above, they tend to be more satisfied with their systems so the spend more time just listening to music rather than the gear. Some have speculated this is because they’re confident more of their hardware is already “good enough.” Peter Aczel and the late Julian Hirsch are classic audio objectivists. And a lot of the folks at Hydrogenaudio fall in this category.

THE MODERATES: Just as with politics and religion, it’s not black and white. Some have a foot firmly in both the objective and subjective side of things. Some examples are John Atkinson at Stereophile, John Siau at Benchmark Media, and to some degree, myself. We value objective measurements but also trust our ears and just because we may not hear a difference we accept someone else might. I believe those in the middle are generally the most open minded.

CREDIBILITY: Wine critics need credibility and trusted taste buds to discern all the subtle details of wine. Subjective audio reviewers are expected to have good hearing and highly developed listening skills. But objective geeks only have to make proper measurements others can verify. Even Grandpa with his hearing aids could do it. (photo: heatheronhertravels)

ACCOUNTABILITY: The subjective reviewers have it easy. If someone doesn’t agree with one of their reviews, excuses are plentiful. When subjective reviewers are questioned, I have seriously seen or heard variations of all of these responses:

• While you might not like it I preferred the slightly more recessed presentation of the UberDAC Black Edition
• The UberDAC is a better match with my ultra expensive reference system than your more modest gear
• Did you use the UberLink Reverse Twisted Unobtanium cables I recommended for the UberDAC? 
• I was in a noisy restaurant for lunch immediately before reviewing the UberDAC and my ears hadn’t fully recovered
• I didn’t know it at the time, but it turns out I was in the early stages of a head cold when I reviewed the UberDAC
• I had too much wine the night before (my personal favorite)

OBJECTIVE ACCOUNTABILITY: Objective reviewers have it hard. We either publish reasonably accurate measurements or we get caught with our pants down. The whole idea is to publish numbers in a way someone else can reproduce (or come close enough). So the quality of our measurements determines our credibility. An error is an error. It requires a lot of wine before 1+1 = 3. We can’t simply ignore or change the well established principals of audio engineering. We have to admit when we’re wrong or join the Planet Earth Is Flat Society.

JUST MESSENGERS: Independent objective reviewers, me included, just test gear and publish the numbers. We don’t make the gear, we usually don’t make up the measurements, and we certainly better not make up the results. We didn’t invent THD, the decibel or Ohm’s Law. So the numbers are what they are. If we’re doing our job right, we’re just messengers delivering numbers we have little control over. Of course nothing but numbers is boring and only useful to hardcore geeks and engineers. So it’s best when the numbers are presented in a relatively understandable way.

YACA (Yet Another Car Analogy): Say you’re in the market for a new fast car and you’re comparing them online. Chevy and Ford both have all new versions of the Camaro SS and Mustang GT. They’re so new nobody has done any track testing yet but here are the factory’s published numbers:

Specification	Ford Mustang GT	Chevy Camaro SS
Curb Weight	3605 Pounds	3860 Pounds
Horsepower	412 HP	426 HP
Torque	390 ft-lbs	420 ft-lbs
Performance 0-60 MPH	4.8 seconds	4.9 seconds
Gas Mileage (city)	18 miles/gal	16 miles/gal

Based on the numbers, the Mustang is lighter, quicker and uses less gas so you check it out at the dealer. It looks great and seems to have enough power so you buy it. When you get home you find the latest issue of Road & Track in your mailbox. They just tested the your Mustang GT and here’s what they found versus Ford’s numbers:

Specification	Ford’s Spec	R&T’s Measurement
Curb Weight	3605 Pounds	3910 Pounds
Horsepower	412 HP	290 HP (on dyno)
Torque	390 ft-lbs	275 ft-lbs (on dyno)
Performance 0-60 MPH	4.8 seconds	9.1 seconds (on track)
Gas Mileage (city)	18 miles/gal	14 miles/gal (test loop)

MARKETING MEETS REALITY: It turns out Ford’s marketing team wasn’t even close to accurate. The car is way heavier, has a lot less power, drinks more gas, and a Prius with a full charge might give you a good run at a stoplight. This is what you just paid $35,000 for? Faced with the bad news, here are some possible options: (photo: Ford Motor Company)

• Take the car back to the dealer, show them the article, and ask what’s going on
• No longer trust Ford and buy a Chevy 
• Offer your buddy with the dynamometer a case of beer if he’ll test your Mustang’s horsepower and torque
• Try a few of your own 0-60 runs to see if it’s really closer to 9.1 seconds than 4.8 seconds
• Burn the issue of Road & Track as you don’t really care about numbers anyway
• Fire off an angry email to Road & Track accusing them of being incompetent without ever trying to verify if their measurements are even correct

BUGS ON THE WINDSHIELD: If we treat cars like audio, it seems most who already own the Mustang prefer the last choice above along with some or all of the following thrown in for good measure:

• Shoot The Messenger! He’s clearly an idiot!
• I trust Ford is less biased than some guy at a magazine!
• His V8 Mustang was only running on 5 cylinders!
• He can’t tell the big hand from the little hand on his stopwatch!
• Those bugs on the windshield were slowing it down!

CARS vs AUDIO: Of course you never see automotive numbers off by the huge margins shown above. Why not? Because magazines like Road & Track keep the car manufactures honest. If they know their cars will be track tested, dyno tested, etc. it’s in their best interest to publish reasonably accurate data. If they didn’t it would be obvious. So why should audio companies be different or exempt from being held similarly accountable? (photo: dbaldwin)

FLAK JACKET REQUIRED: Not only is audio myth-busting a relatively thankless job, it sometimes requires protection from an angry mob of subjectivists. Some wonder why I’m relatively anonymous. First of all, it’s recommended Bloggers write under a pen name. And it’s a good thing as I’ve been threatened, called all sorts of names, accused of having other agendas, and much more. Apparently it’s dangerous work being a messenger, reporting real numbers, and challenging audio claims with real engineering! Isn’t Ford responsible for publishing misleading numbers? Why would someone attack Road & Track for helping expose the truth? The same is true of industry standard PC benchmarks. If the Dell claims their laptop is faster than the competing HP, but it’s really the other way around on a dozen different tests, would you attack the guy doing the review? There’s something odd going on with audio.

IT GETS PERSONAL: There’s a long history of attacking objectivists who try to clarify what matters and/or bust audio myths. Peter Aczel of the Audio Critic is a long time example and a newer one is Meyer and Moran for their SACD hi-res audio work. Unlike cars or PCs, much of high-end audio has little basis in fact. When someone tries to bring facts into the mix, some take it as a challenge to their personal beliefs, personal hobby, etc. And, unfortunately, some go on the defensive and try to discredit the messenger. It’s not too far removed from creationists attacking the science behind evolution.

FOLLOW THE MONEY: They say to figure out what’s really going in our messed up political system you just have to follow the money. It turns out, you can largely do the same thing in high-end audio. A lot of money gets spent based on highly biased subjective evaluations of audio gear. In fact, the more expensive the gear, the more likely it’s bought entirely based on subjective criteria. I show an example below in Subjective Report Cards. If you look at what most influences buying decisions--websites, magazines, the largest forums, etc.--you’ll find nearly all of them are largely bought and paid for by the companies making the gear. So it’s hardly surprising few do objective testing. And what objective tests they conduct often give the equipment the benefit of doubt. A classic example are A/V receiver tests where the manufacture’s power claims are rarely directly challenged. Instead they typically run a couple of power tests done in such a way to not highlight the fact a $1000 “120 watt x 7” receiver might manage only 28 watts/ch with all 7 channels operating.

SUBJECTIVE BIAS: Some interesting studies have been done about subjective bias in audio. Tom Nousaine published a 1991 AES paper titled Can You Trust Your Ears? It included several different tests, but one of the more interesting involved listeners evaluating (unknown to them) identical musical selections. They were asked if they preferred A, B or had no preference. 76% of them expressed a preference despite the selections being identical. It showed people readily hear differences when none exist. That’s not good news for someone who just replaced their $300 DAC with a $3000 DAC because they thought the more expensive one sounded better. The two may really sound the same. For more on this I recommend: Dishonesty of Sighted Listening Tests by Sean Olive. (photo: Sean Olive)

HEARING BIAS: There’s an entertaining and excellent AES Audio Myths Workshop Video with some very interesting observations by experts in their respective fields. One talks about how the brain and ear work together. Human's are not like an ideal microphone treating everything the same. As with the rest of our senses, the brain adapts our hearing for the needs of the moment. If you’re trying to pick out a conversation across a crowded room you naturally filter out everyone else talking without even realizing you’re doing so. This same filtering goes on when listening to audio gear. You might listen critically to the bass one moment and the highs the next, but you can’t listen to both at once and glean as much information. We do the same thing with our vision. There are televised examples of naked people running across football fields in plain view during key plays and few people ever even saw them. Their senses were heavily biased toward the players. They also talk in the video about replacing the circuitry of a high-end audiophile amplifier with a really low-end amp that no audiophile would be caught dead listening to. And, guess what, audiophiles still loved the amp because their eyes were telling them it was a high-end amp so that’s what their ears heard.

INVOLUNTARY BIAS (added 5/23): Subjective audiophiles often claim they’re not affected by the sort of sighted listening bias documented by Toole & Olive. They argue bias is only an issue for untrained listeners. But what if it’s genuinely involuntary? There’s a well understood phenomena called the McGurk Effect. It shows how certain knowledge, such as what we see, influences what we hear. And, even more significant, the bias is involuntary. Around the 2 minute mark in the video linked below they talk about how, even when your conscious brain knows what the truth is, your subconscious brain still alters your hearing in ways you cannot control. The researcher says he’s been studying this effect for 25 years and it still affects him just as much as an untrained listener. The McGurk effect goes away if you close your eyes. But in evaluating gear just closing your eyes isn’t enough if your brain still knows what you’re listening to. You need a blind test to eliminate the bias. Try it yourself with this fascinating video: (photo: BBC)

• Try The McGurk Effect! - BBC 3.5 Minute Video

FOLLOW THE MONEY PART 2: It’s human nature if you go out and spend your hard earned cash on some new piece of gear you want it to be worth the investment. This feeling is compounded by all the subjective reviews you read where others raved about the same piece of gear. In multiple ways your subconscious is already wired to hear a nice improvement even when there isn’t any improvement. This isn’t far removed for hearing “Fa” when the guy in the video above is clearly saying “Ba”. Your brain and senses are just telling you what they think you want to hear.

THE NEED TO BELIEVE: There are some negative comments in my Mini3 review saying I lost all credibility by comparing it to the $20 FiiO E5. If you look at the measurements, the two amps are relatively similar in many areas. But there are obviously some who need to believe a $180 amp is a lot better than a $20 one. They probably already have a Mini3 so it’s an insult for someone (me in this case) to indirectly suggest they spent many times more than necessary. I received similar comments for suggesting the $29 Behringer UCA202 measurements were respectable—very likely from people who spent a lot more for their USB DAC. Again, in their need to believe, these gear owners would rather shoot the messenger. It’s a kind of denial.

JUDGMENT DAY: Tens of thousands of “believers” launched massive campaigns to warn of “May 21st 2011 Judgment Day”. The world was supposed to start self destructing with massive earthquakes and more. Scientists tried to point out the facts but the believers didn’t want to hear any of it. If you’re reading those masses of fierce believers were obviously wrong. Even in 2011 lots of people fall under the spell of others, give in to peer pressure, myth, etc. They’re surprisingly willing to believe things with little or no basis in fact. Once these beliefs reach a critical mass, those within the group are very difficult to persuade they might be wrong. But they don’t have a good track record. From those who believed the earth was flat, to those who thought it was going to self destruct in 2011, the science geeks are the ones who are far more often correct. (photo: Scott Beal/Laughing Squid)

BELIEVING = BIAS: If the guy who thinks I’m an idiot for comparing the Mini3 to the E5 were to sit down and listen to both side-by-side which do you think he’d say sounds better? There’s almost zero chance he’d choose the E5. This same bias is widespread in audio. You have a $300 DAC, you arrange to listen to a $3000 DAC, and even if they sound exactly the same, your brain and hearing are “wired” to think the $3000 DAC sounds better. So how do we get around this problem?

BED SHEETS & TESTING: As you probably guessed, blind testing is the “dirty secret” I referred to at the start of the article. Matrix Audio conducted a relatively simple and eye opening example. The photo at the right shows the test set up with two different systems under a bed sheet sharing a pair of high-end speakers. Volunteers stood behind the speakers and swapped the high-end cables. There were no switch boxes involved. The result, if you haven’t seen it elsewhere, is the listeners couldn’t tell a high-end $12,000 stack of gear from a $700 (I’d say closer to $400) set up with a pro-sound power amp, bargain basement CD player, and a cheap obscenely long RCA cable connecting the two. You can read all about it here: (photo: Matrix Audio)

• Matrix Audio Test

LONG TERM LISTENING: A lot of blind testing involves switching between A and B, or replaying music tracks after something is changed. Critics of these tests argue that’s not the best way to evaluate audio gear. They say you must live with it for a while to appreciate the differences (never mind most of them claim to swap out a piece of gear and hear immediate and obvious differences). David Clark and Laurence Greenhill came up with a clever idea. They made a bunch of sealed black boxes where some had a direct connection inside while others distorted the audio signal to a significant degree. They were built with high-end connectors, etc. They sent the boxes home with members of a local audiophile club to live with and decide if they had a “straight wire” box or one that did ugly things to the audio. Despite living with them for a while, the audiophiles who took the boxes home failed to determine which was which. The same boxes, however, were identified with relative ease in a blind A/B/X test. This demonstrated the exact opposite of what many audiophiles claim: Long term listening is less sensitive than A/B/X testing. This test, and others, are summarized in Ten Years of ABX Testing.

A NEW WINE ANALOGY: Many have probably heard the analogy before, but here’s a short entertaining article by a wine critic describing blind testing. He rated the $2.50 Charles Shaw wine very poorly in sighted tasting. But he discovered, with brown paper wrappers on the bottles, it was “not going to be easy”. I won’t spoil the outcome but his experience is exactly analogous to what usually happens in blind audio testing. Suddenly that $20 FiiO or $30 Behringer is a lot harder to pick out. At least this critic was humble enough to go public with his experience:

• Wellesley Wine Press Blind Tasting

BLIND BANNED: The largest headphone forum around, Head-Fi, prohibits the discussion of blind testing in all but one of their 20+ forums. They only, seemingly grudgingly, allow it in the back-of-the-bus Sound Science forum that’s all but ignored by the mainstream. Why? Could it be their many sponsors, say Qables selling iPod cables priced at many times the iPod itself ($1000+ for a 6 inch iPod dock cable!), don’t approve of having their products debunked with blind test results? You won’t see many blind tests in the audiophile magazines or on ad-supported websites. It seems an excellent tool has been strategically marginalized, swept under the rug, and discredited over the last decade by the “industry”. It’s time for more people to start asking why.

SUBJECTIVE HAS ITS PLACE: When choosing a car, new laptop, and audio gear, the subjective side matters. It’s not all numbers. Things like ease of use, aesthetics, and build quality are all important. When it comes to the sound of different speakers and headphones, subjective opinions are often what matter most. Are you a basshead, like it bright and detailed, laid back, or as accurate as possible? I’m not trying to dispute subjective preferences. If you like the sound of tube gear, even if it measures poorly, that’s your business. If it puts a smile on your face that’s what matters most. And some buy high-end gear for the quality, looks, status, etc. My concern is misleading objective data, objective claims with no basis in reality, marketing “pseudo-science”, and when the line between subjective and objective is intentionally blurred.

SUBJECTIVE REPORT CARDS: Stereophile magazine assigns letter grades in their annual Recommended Components issue. In the April 2011 issue the Vitus Audio MP-P201 phono preamp, for a paltry $60,000.00, rated an “A+” while the boring $199 NAD PP-3 rated a sorry “D”—the kid who didn’t study for the test. Stereophile is to be commended for conducting measurements of some of the gear they review. In this case, the bargain NAD measured better than the uber-expensive Vitus. The NAD had notably lower distortion and lower noise—an especially important parameter in a phono preamp. So is it safe to assume the extra $59,801.00 of value in the Vitus must be purely subjective?

WHEN OPINION BECOMES FACT: In the example above one person, Michael Fremer, apparently decided the $60K Vitus was vastly better based on his personal opinion of how it sounds. From what I gather, he didn’t conduct any sort of rigorous blind listening tests that included others. Nor did John Atkinson’s inferior measurements ultimately carry much weight. I gather Fremer listened to the Vitus in his particular system, using his phono cartridge(s), with his particular tastes in music, and decided it’s worth the astronomical price tag. Is the rest of the world to believe they would also prefer the Vitus over the NAD even with their different personal preferences, phono cartridge(s), and music collections? This often happens at all price levels—someone else’s highly subjective (and nearly always biased) opinion becomes objective “proof” that Gear X is better than Gear Y. So lots of other people buy Gear X even though they might have different tastes or not hear any difference at all. There are many things wrong with this—especially when seemingly objective ratings like Stereophile’s  “A”, “B”, etc. or another magazine’s 5 stars, imply some clear criteria.

PEER PRESSURE: The Stereophile ratings are a lot like Wine Spectator scores. They create a sort of “peer pressure”--much like wanting to serve wine with a high score regardless of the wine buyer’s own personal tastes. And even if a critic enjoyed a particular wine with his Italian food, it might be a lousy match with someone else’s Sushi. The same can be said for Michael Fremer’s reviews. Just because the Vitus sounded good for him, using his gear, music, etc., doesn’t mean it’s audio nirvana for someone else. And how much was Fremer unavoidably biased by the $60K price tag and similar factors? Did someone from Vitus fill his head with hyperbole beforehand over a gourmet lunch? Perhaps most serious of all: What if the Vitus really sounds just like the cheap but well engineered NAD? That outcome is far more likely than most realize or want to admit.

BACK TO ACCOUNTABILITY: If Michael Fremer can go around recommending $60,000 gear based on his listening abilities, some might reasonably want proof he’s qualified. And being a good sport, Mr Fremer broke rank and participated in at least a few blind listening tests. The result was a rather mixed bag and at least one included lots of hand waving. Not surprisingly, few want to follow in his footsteps. Today it’s even more difficult to find listeners with a public reputation willing to participate in a blind test. The same people who publish hearing “immediate and obvious” differences in everything from cables to power conditioners typically make all sorts of questionable excuses when asked to do so with brown bags or bed sheets concealing the gear. Personally, I suspect Fremer probably can hear things 99% of the population would have trouble hearing. If anything, he’s a “ringer” for the subjectivists and I have genuine respect for his listening abilities. So it’s especially a shame he, and other skilled critics like him, won’t participate in more blind tests.

WIRED WISDOM (updated 6/3): Tom Nousaine published a great article in Sound and Vision called Wired Wisdom. The goal was to see if audiophiles, in their own homes using their own familiar high-end systems, could hear differences between cheap and expensive cables. In all three trials, they could not. The cable myth suffered a serious blow from reality. The second link compares expensive versus cheap speaker cables with similar results:

• Sound & Vision Wired Wisdom Article
• Observations of a controlled Cable Test (AVS Forum)

CREATIONISM vs EVOLUTION: Alan Lofft, the editor of Sound and Vision, tried to dance around the Wired Wisdom article--likely to appease their cable advertisers. He talked of creationists and evolution arguing both had an important role. And he compared high-end cables to “audio jewelry” that some buy for aesthetics and status rather than sound quality. Lofft did what much of the high-end audio industry does. He tried to soften the truth, not offend too many people, and make sure the status quo (along with their advertisers) remained relatively unscathed. See: Follow The Money. His column is on the last page of the Wired Wisdom article. I’m getting flak for not performing a similar delicate dance with AMB, NuForce, etc.. It seems nobody is supposed to rock the boat too much however factual their concerns. Is this audio or political foreign relations?

HIGH RES (SACD) vs CD: Multiple tests have been published comparing standard 16 bit 44.1 Khz CD quality audio to higher resolution formats such as 24 bit 96 Khz and SACD. The most famous is probably this one:

• Audibility of a CD-Standard A/D/A Loop Inserted into High-Resolution Audio Playback

Meyer and Moran played SACD content with the ability to switch an A/D –> D/A pair operating at 16 bits and 44 Khz into the signal path. In other words, the high resolution SACD audio was sometimes “down converted” to CD quality. They designed the test to give the listeners “every opportunity” to detect a difference. The testing lasted a year and included 60 members of the Boston Audio Society, many professional recording engineers, fresh eared college students, and a whopping 554 listening trials. After all that, the only way anyone could identify a consistent difference was by cranking the volume unrealistically high during quiet passages exposing the higher noise floor of the 16 bit conversion.

Think about the implications of the above. Most subjective audiophiles claim to hear differences between CD players, DACs, and indeed most anything that performs a digital to analog conversion. They also consider SACD and other high resolution formats as being plainly superior. Why can’t audiophiles detect any difference at all when the music is subjected to an extra A/D and then another extra D/A process when they don’t know that’s happening?

SHOOT THE MESSENGER (again): Not surprisingly, many have tried to discredit Meyer and Moran claiming they didn’t use the right source material, etc. The study authors have responded to much of the criticism in this little cited follow up. It’s my personal opinion nobody over the last 4 years, despite plenty of attempts and desire, has invalidated the overall results. If nothing else the study demonstrates just how genuinely transparent 16/44 digital audio can be. The best challenge I’ve seen is a single 2010 study finding a tiny minority of expert listeners, under very specific circumstances, could discern very slight differences. There are many other interesting references about SACD vs CD as well as a summary of the above test here:

• Wikipedia SACD Audible Differences

THINK ABOUT IT: Two guys come along and the most respected objective audio organization in the world publishes their paper that threatens to destroy the entire SACD audio industry and also does damage to high resolution audio formats of any kind. Meyer and Moran attempted to demonstrate CD quality audio really is good enough. If Philips, Sony and the music labels behind SACD knew the study was flawed, and SACD was audibly superior, it would have been pocket change for them to fund a study demonstrating where Meyer and Moran were wrong. But, surprise surprise, that never happened. The closest was the 2010 paper on Sampling Rate Discrimination mentioned above (if you want to geek out on a lengthy discussion of the 2010 study check out this Hydrogenaudio thread).

RINSE LATHER AND REPEAT WITH VINYL: Lots of claims are made for the analog nature of vinyl LPs and a small fortune is spent on esoteric turntables and phono gear as an analog source for pure analog high-end systems. When playing vintage analog-mastered music the audio never suffers the indignity of being whacked up into a bunch of numerical values and put back together. Much like the SACD test above, there have been various tests demonstrating even devout vinyl lovers can’t tell when you slip an A/D –> D/A loop into their otherwise all analog signal chain. Here’s a link to one but there are some better ones I’ll work on finding the links for. I’ve also done my own informal blind vinyl test on the sly. The vinyl lover wasn’t even aware he’d been listening to digital for several days on an extremely high-end all analog system. And this guy really hates digital anything. Don’t get me wrong, I own a nice turntable and I listen to vinyl. But for me it’s mostly about music that’s only available on vinyl. I don’t pretend it’s a technically superior format.

• Vinyl Myths on Hydrogenaudio (added 5/31)

OBJECTIVE ISN’T EVERYTHING: I said above subjective stuff matters, and I’m including the reverse just to be clear. The numbers only tell part of the story. They make a convenient way to compare some things—especially say power output, output impedance, how suitable a given source/amp is for a particular headphone, frequency response, etc. But there are limits. If nothing else, great measurements provide a significant piece of mind for some people. They can relax and enjoy the music knowing their gear is among the most transparent available. And blind tests “wrap” subjective listening in a controlled, and more objective, environment free of the usual bias while keeping score. The two complement each other well.

WHEN SPECS ARE NOT ENOUGH:  First the easy part. For speakers, headphone and phono cartridges I think everyone agrees it’s tough to look at the specs and know exactly what they will sound like. You can still make some valid comparisons but the specs only give you a partial idea of the sound. With speakers and headphones the acoustics are a big part of the listening experience—all rooms and ear/head geometries are different. The sound of cartridges are altered by the tonearm and turntable geometry they’re used in (effective arm length, VTA, arm resonance, damping, etc.).  They also perform very differently playing worn vinyl as stylus tips come in an almost endless variety of shapes and sizes. So cartridge A rides in a different part of the groove than cartridge B. And measurements are limited by the relatively low resolution of vinyl test albums.  So, in other words, your mileage may vary and caveat emptor. You have to listen to speakers, headphones and cartridges to fully evaluate them. But that’s much less true with electronics.

GRAY AREA (added 5/31): There’s a solid consensus on the subjective nature of speakers, headphones and phono cartridges but what about everything else? Most objectivists will tell you a $20 well designed interconnect and a $200 well designed interconnect will sound the same. And that’s been demonstrated many times (see Wired Wisdom above). Hardcore objectivists (such as Peter Aczel) argue any amp that measures sufficiently well and is operated well within its limits will be indistinguishable from any other amp. And that’s been demonstrated in countless blind listening tests. But what about when an amp nears its limits? What if you have difficult to drive speakers for example that are 2 ohms at some frequencies? Will the cheap amp still sound just like the high-end model? Perhaps not. Some of these behaviors can at least be partly measured but some are more difficult. And what defines “measures sufficiently well”? Enough studies have been done it’s fairly safe to make several generalizations, but there’s still room for discussion and further research in some areas. I’ll hopefully be publishing a future blog article on the topic of correlating specs with listening observations.

MAGICAL THINKING: As someone pointed out in the comments, some audiophiles are in this for the “mysticism, magical thinking, and never ending quest” (their words) and I certainly know a few in that category. The purveyors of tube products tend to be rather clever in their marketing. They rarely make boastful performance claims and some offer hardly any specs at all. They know certain people enjoy their products and they make an appropriately subjective sales pitch. That’s hard to argue with. But when someone makes objective claims, and they’re far from being realistic, that’s just deceptive and wrong—regardless of the buyer’s priorities. It’s easy for some to get “sucked in” to the hype, myths, and mania when the real facts are constantly being swept under the rug. It’s not unlike Wine Spectator scores. See Peer Pressure above.

MORE INFO: If you’re seeking more interesting info check out:

• AES Audio Myths Workshop Video This should be required viewing! - (credit: “Anonymous” on 5/18)
• AES Audio Myths Audio Files Courtesy Ethan Winer companion files to the video link above
• Douglas Self: Audio Subjectivists Another (very brilliant) engineer’s view of subjectivists
• Hydrogenaudio Links to Blind Listening Tests Lots of related links old and new

TO BE CONTINUED… I have lots more to share on this topic, but for now, I’ll see what sort of response this brings.