Header

Objective Reviews & Commentary - An Engineer's Perspective

May 18, 2011

Subjective vs Objective Debate

argument by Francis CarnaubaINTRO: Given the strong reaction to some of my articles here are some thought’s on what’s likely behind the more emotional responses. The world of high-end audio can be almost religious and divided something like the Republicans and Democrats. In this case it’s the Subjectivists versus the Objectivists. It’s been called the “Great Debate” a “Holy War” and more. (photo: Francis Carnauba)

HIGH-END AUDIO’s DIRTY SECRET: What if I told you there was a proven way to evaluate gear using your own ears that highlights even the smallest audible differences between two pieces of gear? Compared to typical listening methods it’s been found to be far more reliable and revealing of subtle differences. More on this later!

THE SUBJECTIVISTS: The hardcore Subjectivists trust their own ears above all else and often ignore, downplay, or sometimes even actively discredit objective efforts. Some argue they have superior hearing and/or listening skills and more refined tastes. That sometimes creates at least a whiff of an elitist “club” that some are drawn to (think Robb Report). But, regardless, their genuine passion for audio is to be admired. And I believe at least some of them do have superior listening skills compared to the Average Joe. Despite their more emotional left brains, which might imply a greater love of music, there’s some consensus Subjectivists spend more of their time tweaking and evaluating their hardware than a typical objectivist. Stereophile’s Michael Fremer is generally considered a strong subjectivist.

THE OBJECTIVISTS: This group tends to prefer some sort of science, measurements, or objective listening tests to back up claims of “A is better than B”. When reading a gear review they’re more likely to skip to the measurements section (if there is one) than read subjective impressions. They tend to be skeptical of outrageous claims and ultra high priced gear. They also tend to buy less expensive gear, less often, than subjectivists making them less attractive to manufactures. As mentioned above, they tend to be more satisfied with their systems so the spend more time just listening to music rather than the gear. Some have speculated this is because they’re confident more of their hardware is already “good enough.” Peter Aczel and the late Julian Hirsch are classic audio objectivists. And a lot of the folks at Hydrogenaudio fall in this category.

THE MODERATES: Just as with politics and religion, it’s not black and white. Some have a foot firmly in both the objective and subjective side of things. Some examples are John Atkinson at Stereophile, John Siau at Benchmark Media, and to some degree, myself. We value objective measurements but also trust our ears and just because we may not hear a difference we accept someone else might. I believe those in the middle are generally the most open minded.

wine heatheronhertravelsCREDIBILITY: Wine critics need credibility and trusted taste buds to discern all the subtle details of wine. Subjective audio reviewers are expected to have good hearing and highly developed listening skills. But objective geeks only have to make proper measurements others can verify. Even Grandpa with his hearing aids could do it. (photo: heatheronhertravels)

ACCOUNTABILITY: The subjective reviewers have it easy. If someone doesn’t agree with one of their reviews, excuses are plentiful. When subjective reviewers are questioned, I have seriously seen or heard variations of all of these responses:

  • While you might not like it I preferred the slightly more recessed presentation of the UberDAC Black Edition
  • The UberDAC is a better match with my ultra expensive reference system than your more modest gear
  • Did you use the UberLink Reverse Twisted Unobtanium cables I recommended for the UberDAC? 
  • I was in a noisy restaurant for lunch immediately before reviewing the UberDAC and my ears hadn’t fully recovered
  • I didn’t know it at the time, but it turns out I was in the early stages of a head cold when I reviewed the UberDAC
  • I had too much wine the night before (my personal favorite)

OBJECTIVE ACCOUNTABILITY: Objective reviewers have it hard. We either publish reasonably accurate measurements or we get caught with our pants down. The whole idea is to publish numbers in a way someone else can reproduce (or come close enough). So the quality of our measurements determines our credibility. An error is an error. It requires a lot of wine before 1+1 = 3. We can’t simply ignore or change the well established principals of audio engineering. We have to admit when we’re wrong or join the Planet Earth Is Flat Society.

JUST MESSENGERS: Independent objective reviewers, me included, just test gear and publish the numbers. We don’t make the gear, we usually don’t make up the measurements, and we certainly better not make up the results. We didn’t invent THD, the decibel or Ohm’s Law. So the numbers are what they are. If we’re doing our job right, we’re just messengers delivering numbers we have little control over. Of course nothing but numbers is boring and only useful to hardcore geeks and engineers. So it’s best when the numbers are presented in a relatively understandable way.

YACA (Yet Another Car Analogy): Say you’re in the market for a new fast car and you’re comparing them online. Chevy and Ford both have all new versions of the Camaro SS and Mustang GT. They’re so new nobody has done any track testing yet but here are the factory’s published numbers:

Specification Ford Mustang GT Chevy Camaro SS
Curb Weight 3605 Pounds 3860 Pounds
Horsepower 412 HP 426 HP
Torque 390 ft-lbs 420 ft-lbs
Performance 0-60 MPH 4.8 seconds 4.9 seconds
Gas Mileage (city) 18 miles/gal 16 miles/gal

Based on the numbers, the Mustang is lighter, quicker and uses less gas so you check it out at the dealer. It looks great and seems to have enough power so you buy it. When you get home you find the latest issue of Road & Track in your mailbox. They just tested the your Mustang GT and here’s what they found versus Ford’s numbers:

Specification Ford’s Spec R&T’s Measurement
Curb Weight 3605 Pounds 3910 Pounds
Horsepower 412 HP 290 HP (on dyno)
Torque 390 ft-lbs 275 ft-lbs (on dyno)
Performance 0-60 MPH 4.8 seconds 9.1 seconds (on track)
Gas Mileage (city) 18 miles/gal 14 miles/gal (test loop)

mustang 500MARKETING MEETS REALITY: It turns out Ford’s marketing team wasn’t even close to accurate. The car is way heavier, has a lot less power, drinks more gas, and a Prius with a full charge might give you a good run at a stoplight. This is what you just paid $35,000 for? Faced with the bad news, here are some possible options: (photo: Ford Motor Company)

  • Take the car back to the dealer, show them the article, and ask what’s going on
  • No longer trust Ford and buy a Chevy 
  • Offer your buddy with the dynamometer a case of beer if he’ll test your Mustang’s horsepower and torque
  • Try a few of your own 0-60 runs to see if it’s really closer to 9.1 seconds than 4.8 seconds
  • Burn the issue of Road & Track as you don’t really care about numbers anyway
  • Fire off an angry email to Road & Track accusing them of being incompetent without ever trying to verify if their measurements are even correct

BUGS ON THE WINDSHIELD: If we treat cars like audio, it seems most who already own the Mustang prefer the last choice above along with some or all of the following thrown in for good measure:

  • Shoot The Messenger! He’s clearly an idiot!
  • I trust Ford is less biased than some guy at a magazine!
  • His V8 Mustang was only running on 5 cylinders!
  • He can’t tell the big hand from the little hand on his stopwatch!
  • Those bugs on the windshield were slowing it down!

snake-oil_thumb2

CARS vs AUDIO: Of course you never see automotive numbers off by the huge margins shown above. Why not? Because magazines like Road & Track keep the car manufactures honest. If they know their cars will be track tested, dyno tested, etc. it’s in their best interest to publish reasonably accurate data. If they didn’t it would be obvious. So why should audio companies be different or exempt from being held similarly accountable? (photo: dbaldwin)

FLAK JACKET REQUIRED: Not only is audio myth-busting a relatively thankless job, it sometimes requires protection from an angry mob of subjectivists. Some wonder why I’m relatively anonymous. First of all, it’s recommended Bloggers write under a pen name. And it’s a good thing as I’ve been threatened, called all sorts of names, accused of having other agendas, and much more. Apparently it’s dangerous work being a messenger, reporting real numbers, and challenging audio claims with real engineering! Isn’t Ford responsible for publishing misleading numbers? Why would someone attack Road & Track for helping expose the truth? The same is true of industry standard PC benchmarks. If the Dell claims their laptop is faster than the competing HP, but it’s really the other way around on a dozen different tests, would you attack the guy doing the review? There’s something odd going on with audio.

IT GETS PERSONAL: There’s a long history of attacking objectivists who try to clarify what matters and/or bust audio myths. Peter Aczel of the Audio Critic is a long time example and a newer one is Meyer and Moran for their SACD hi-res audio work. Unlike cars or PCs, much of high-end audio has little basis in fact. When someone tries to bring facts into the mix, some take it as a challenge to their personal beliefs, personal hobby, etc. And, unfortunately, some go on the defensive and try to discredit the messenger. It’s not too far removed from creationists attacking the science behind evolution.

FOLLOW THE MONEY: They say to figure out what’s really going in our messed up political system you just have to follow the money. It turns out, you can largely do the same thing in high-end audio. A lot of money gets spent based on highly biased subjective evaluations of audio gear. In fact, the more expensive the gear, the more likely it’s bought entirely based on subjective criteria. I show an example below in Subjective Report Cards. If you look at what most influences buying decisions--websites, magazines, the largest forums, etc.--you’ll find nearly all of them are largely bought and paid for by the companies making the gear. So it’s hardly surprising few do objective testing. And what objective tests they conduct often give the equipment the benefit of doubt. A classic example are A/V receiver tests where the manufacture’s power claims are rarely directly challenged. Instead they typically run a couple of power tests done in such a way to not highlight the fact a $1000 “120 watt x 7” receiver might manage only 28 watts/ch with all 7 channels operating.

BlindVsSightedMeanLoudspeakerRatingsSUBJECTIVE BIAS: Some interesting studies have been done about subjective bias in audio. Tom Nousaine published a 1991 AES paper titled Can You Trust Your Ears? It included several different tests, but one of the more interesting involved listeners evaluating (unknown to them) identical musical selections. They were asked if they preferred A, B or had no preference. 76% of them expressed a preference despite the selections being identical. It showed people readily hear differences when none exist. That’s not good news for someone who just replaced their $300 DAC with a $3000 DAC because they thought the more expensive one sounded better. The two may really sound the same. For more on this I recommend: Dishonesty of Sighted Listening Tests by Sean Olive. (photo: Sean Olive)

HEARING BIAS: There’s an entertaining and excellent AES Audio Myths Workshop Video with some very interesting observations by experts in their respective fields. One talks about how the brain and ear work together. Human's are not like an ideal microphone treating everything the same. As with the rest of our senses, the brain adapts our hearing for the needs of the moment. If you’re trying to pick out a conversation across a crowded room you naturally filter out everyone else talking without even realizing you’re doing so. This same filtering goes on when listening to audio gear. You might listen critically to the bass one moment and the highs the next, but you can’t listen to both at once and glean as much information. We do the same thing with our vision. There are televised examples of naked people running across football fields in plain view during key plays and few people ever even saw them. Their senses were heavily biased toward the players. They also talk in the video about replacing the circuitry of a high-end audiophile amplifier with a really low-end amp that no audiophile would be caught dead listening to. And, guess what, audiophiles still loved the amp because their eyes were telling them it was a high-end amp so that’s what their ears heard.

bbc mcgurkINVOLUNTARY BIAS (added 5/23): Subjective audiophiles often claim they’re not affected by the sort of sighted listening bias documented by Toole & Olive. They argue bias is only an issue for untrained listeners. But what if it’s genuinely involuntary? There’s a well understood phenomena called the McGurk Effect. It shows how certain knowledge, such as what we see, influences what we hear. And, even more significant, the bias is involuntary. Around the 2 minute mark in the video linked below they talk about how, even when your conscious brain knows what the truth is, your subconscious brain still alters your hearing in ways you cannot control. The researcher says he’s been studying this effect for 25 years and it still affects him just as much as an untrained listener. The McGurk effect goes away if you close your eyes. But in evaluating gear just closing your eyes isn’t enough if your brain still knows what you’re listening to. You need a blind test to eliminate the bias. Try it yourself with this fascinating video: (photo: BBC)

FOLLOW THE MONEY PART 2: It’s human nature if you go out and spend your hard earned cash on some new piece of gear you want it to be worth the investment. This feeling is compounded by all the subjective reviews you read where others raved about the same piece of gear. In multiple ways your subconscious is already wired to hear a nice improvement even when there isn’t any improvement. This isn’t far removed for hearing “Fa” when the guy in the video above is clearly saying “Ba”. Your brain and senses are just telling you what they think you want to hear.

fiio e5 test

THE NEED TO BELIEVE: There are some negative comments in my Mini3 review saying I lost all credibility by comparing it to the $20 FiiO E5. If you look at the measurements, the two amps are relatively similar in many areas. But there are obviously some who need to believe a $180 amp is a lot better than a $20 one. They probably already have a Mini3 so it’s an insult for someone (me in this case) to indirectly suggest they spent many times more than necessary. I received similar comments for suggesting the $29 Behringer UCA202 measurements were respectable—very likely from people who spent a lot more for their USB DAC. Again, in their need to believe, these gear owners would rather shoot the messenger. It’s a kind of denial.

photo by <a href="http://laughingsquid.com">Scott Beale / Laughing Squid</a><br /><br />This photo is licensed under a Creative Commons license. If you use this photo within the terms of the license or make special arrangements to use the photo, please list the photo credit as "Scott Beale / Laughing Squid" and link the credit to http://laughingsquid.com.JUDGMENT DAY: Tens of thousands of “believers” launched massive campaigns to warn of “May 21st 2011 Judgment Day”. The world was supposed to start self destructing with massive earthquakes and more. Scientists tried to point out the facts but the believers didn’t want to hear any of it. If you’re reading those masses of fierce believers were obviously wrong. Even in 2011 lots of people fall under the spell of others, give in to peer pressure, myth, etc. They’re surprisingly willing to believe things with little or no basis in fact. Once these beliefs reach a critical mass, those within the group are very difficult to persuade they might be wrong. But they don’t have a good track record. From those who believed the earth was flat, to those who thought it was going to self destruct in 2011, the science geeks are the ones who are far more often correct. (photo: Scott Beal/Laughing Squid)

BELIEVING = BIAS: If the guy who thinks I’m an idiot for comparing the Mini3 to the E5 were to sit down and listen to both side-by-side which do you think he’d say sounds better? There’s almost zero chance he’d choose the E5. This same bias is widespread in audio. You have a $300 DAC, you arrange to listen to a $3000 DAC, and even if they sound exactly the same, your brain and hearing are “wired” to think the $3000 DAC sounds better. So how do we get around this problem?

BED SHEETS & TESTING: As you probably guessed, blind testing is the “dirty secret” I referred to at the start of the article. Matrix Audio conducted a relatively simple and eye opening example. The photo at the right shows the test set up with two different systems under a bed sheet sharing a pair of high-end speakers. Volunteers stood behind the speakers and swapped the high-end cables. There were no switch boxes involved. The result, if you haven’t seen it elsewhere, is the listeners couldn’t tell a high-end $12,000 stack of gear from a $700 (I’d say closer to $400) set up with a pro-sound power amp, bargain basement CD player, and a cheap obscenely long RCA cable connecting the two. You can read all about it here: (photo: Matrix Audio)

LONG TERM LISTENING: A lot of blind testing involves switching between A and B, or replaying music tracks after something is changed. Critics of these tests argue that’s not the best way to evaluate audio gear. They say you must live with it for a while to appreciate the differences (never mind most of them claim to swap out a piece of gear and hear immediate and obvious differences). David Clark and Laurence Greenhill came up with a clever idea. They made a bunch of sealed black boxes where some had a direct connection inside while others distorted the audio signal to a significant degree. They were built with high-end connectors, etc. They sent the boxes home with members of a local audiophile club to live with and decide if they had a “straight wire” box or one that did ugly things to the audio. Despite living with them for a while, the audiophiles who took the boxes home failed to determine which was which. The same boxes, however, were identified with relative ease in a blind A/B/X test. This demonstrated the exact opposite of what many audiophiles claim: Long term listening is less sensitive than A/B/X testing. This test, and others, are summarized in Ten Years of ABX Testing.

A NEW WINE ANALOGY: Many have probably heard the analogy before, but here’s a short entertaining article by a wine critic describing blind testing. He rated the $2.50 Charles Shaw wine very poorly in sighted tasting. But he discovered, with brown paper wrappers on the bottles, it was “not going to be easy”. I won’t spoil the outcome but his experience is exactly analogous to what usually happens in blind audio testing. Suddenly that $20 FiiO or $30 Behringer is a lot harder to pick out. At least this critic was humble enough to go public with his experience:

blind listening tests nwavguyBLIND BANNED: The largest headphone forum around, Head-Fi, prohibits the discussion of blind testing in all but one of their 20+ forums. They only, seemingly grudgingly, allow it in the back-of-the-bus Sound Science forum that’s all but ignored by the mainstream. Why? Could it be their many sponsors, say Qables selling iPod cables priced at many times the iPod itself ($1000+ for a 6 inch iPod dock cable!), don’t approve of having their products debunked with blind test results? You won’t see many blind tests in the audiophile magazines or on ad-supported websites. It seems an excellent tool has been strategically marginalized, swept under the rug, and discredited over the last decade by the “industry”. It’s time for more people to start asking why.

SUBJECTIVE HAS ITS PLACE: When choosing a car, new laptop, and audio gear, the subjective side matters. It’s not all numbers. Things like ease of use, aesthetics, and build quality are all important. When it comes to the sound of different speakers and headphones, subjective opinions are often what matter most. Are you a basshead, like it bright and detailed, laid back, or as accurate as possible? I’m not trying to dispute subjective preferences. If you like the sound of tube gear, even if it measures poorly, that’s your business. If it puts a smile on your face that’s what matters most. And some buy high-end gear for the quality, looks, status, etc. My concern is misleading objective data, objective claims with no basis in reality, marketing “pseudo-science”, and when the line between subjective and objective is intentionally blurred.

SUBJECTIVE REPORT CARDS: Stereophile magazine assigns letter grades in their annual Recommended Components issue. In the April 2011 issue the Vitus Audio MP-P201 phono preamp, for a paltry $60,000.00, rated an “A+” while the boring $199 NAD PP-3 rated a sorry “D”—the kid who didn’t study for the test. Stereophile is to be commended for conducting measurements of some of the gear they review. In this case, the bargain NAD measured better than the uber-expensive Vitus. The NAD had notably lower distortion and lower noise—an especially important parameter in a phono preamp. So is it safe to assume the extra $59,801.00 of value in the Vitus must be purely subjective?

WHEN OPINION BECOMES FACT: In the example above one person, Michael Fremer, apparently decided the $60K Vitus was vastly better based on his personal opinion of how it sounds. From what I gather, he didn’t conduct any sort of rigorous blind listening tests that included others. Nor did John Atkinson’s inferior measurements ultimately carry much weight. I gather Fremer listened to the Vitus in his particular system, using his phono cartridge(s), with his particular tastes in music, and decided it’s worth the astronomical price tag. Is the rest of the world to believe they would also prefer the Vitus over the NAD even with their different personal preferences, phono cartridge(s), and music collections? This often happens at all price levels—someone else’s highly subjective (and nearly always biased) opinion becomes objective “proof” that Gear X is better than Gear Y. So lots of other people buy Gear X even though they might have different tastes or not hear any difference at all. There are many things wrong with this—especially when seemingly objective ratings like Stereophile’s  “A”, “B”, etc. or another magazine’s 5 stars, imply some clear criteria.

PEER PRESSURE: The Stereophile ratings are a lot like Wine Spectator scores. They create a sort of “peer pressure”--much like wanting to serve wine with a high score regardless of the wine buyer’s own personal tastes. And even if a critic enjoyed a particular wine with his Italian food, it might be a lousy match with someone else’s Sushi. The same can be said for Michael Fremer’s reviews. Just because the Vitus sounded good for him, using his gear, music, etc., doesn’t mean it’s audio nirvana for someone else. And how much was Fremer unavoidably biased by the $60K price tag and similar factors? Did someone from Vitus fill his head with hyperbole beforehand over a gourmet lunch? Perhaps most serious of all: What if the Vitus really sounds just like the cheap but well engineered NAD? That outcome is far more likely than most realize or want to admit.

BACK TO ACCOUNTABILITY: If Michael Fremer can go around recommending $60,000 gear based on his listening abilities, some might reasonably want proof he’s qualified. And being a good sport, Mr Fremer broke rank and participated in at least a few blind listening tests. The result was a rather mixed bag and at least one included lots of hand waving. Not surprisingly, few want to follow in his footsteps. Today it’s even more difficult to find listeners with a public reputation willing to participate in a blind test. The same people who publish hearing “immediate and obvious” differences in everything from cables to power conditioners typically make all sorts of questionable excuses when asked to do so with brown bags or bed sheets concealing the gear. Personally, I suspect Fremer probably can hear things 99% of the population would have trouble hearing. If anything, he’s a “ringer” for the subjectivists and I have genuine respect for his listening abilities. So it’s especially a shame he, and other skilled critics like him, won’t participate in more blind tests.

rca cablesWIRED WISDOM (updated 6/3): Tom Nousaine published a great article in Sound and Vision called Wired Wisdom. The goal was to see if audiophiles, in their own homes using their own familiar high-end systems, could hear differences between cheap and expensive cables. In all three trials, they could not. The cable myth suffered a serious blow from reality. The second link compares expensive versus cheap speaker cables with similar results:

CREATIONISM vs EVOLUTION: Alan Lofft, the editor of Sound and Vision, tried to dance around the Wired Wisdom article--likely to appease their cable advertisers. He talked of creationists and evolution arguing both had an important role. And he compared high-end cables to “audio jewelry” that some buy for aesthetics and status rather than sound quality. Lofft did what much of the high-end audio industry does. He tried to soften the truth, not offend too many people, and make sure the status quo (along with their advertisers) remained relatively unscathed. See: Follow The Money. His column is on the last page of the Wired Wisdom article. I’m getting flak for not performing a similar delicate dance with AMB, NuForce, etc.. It seems nobody is supposed to rock the boat too much however factual their concerns. Is this audio or political foreign relations?

HIGH RES (SACD) vs CD: Multiple tests have been published comparing standard 16 bit 44.1 Khz CD quality audio to higher resolution formats such as 24 bit 96 Khz and SACD. The most famous is probably this one:

Meyer and Moran played SACD content with the ability to switch an A/D –> D/A pair operating at 16 bits and 44 Khz into the signal path. In other words, the high resolution SACD audio was sometimes “down converted” to CD quality. They designed the test to give the listeners “every opportunity” to detect a difference. The testing lasted a year and included 60 members of the Boston Audio Society, many professional recording engineers, fresh eared college students, and a whopping 554 listening trials. After all that, the only way anyone could identify a consistent difference was by cranking the volume unrealistically high during quiet passages exposing the higher noise floor of the 16 bit conversion.

Think about the implications of the above. Most subjective audiophiles claim to hear differences between CD players, DACs, and indeed most anything that performs a digital to analog conversion. They also consider SACD and other high resolution formats as being plainly superior. Why can’t audiophiles detect any difference at all when the music is subjected to an extra A/D and then another extra D/A process when they don’t know that’s happening?

SHOOT THE MESSENGER (again): Not surprisingly, many have tried to discredit Meyer and Moran claiming they didn’t use the right source material, etc. The study authors have responded to much of the criticism in this little cited follow up. It’s my personal opinion nobody over the last 4 years, despite plenty of attempts and desire, has invalidated the overall results. If nothing else the study demonstrates just how genuinely transparent 16/44 digital audio can be. The best challenge I’ve seen is a single 2010 study finding a tiny minority of expert listeners, under very specific circumstances, could discern very slight differences. There are many other interesting references about SACD vs CD as well as a summary of the above test here:

THINK ABOUT IT: Two guys come along and the most respected objective audio organization in the world publishes their paper that threatens to destroy the entire SACD audio industry and also does damage to high resolution audio formats of any kind. Meyer and Moran attempted to demonstrate CD quality audio really is good enough. If Philips, Sony and the music labels behind SACD knew the study was flawed, and SACD was audibly superior, it would have been pocket change for them to fund a study demonstrating where Meyer and Moran were wrong. But, surprise surprise, that never happened. The closest was the 2010 paper on Sampling Rate Discrimination mentioned above (if you want to geek out on a lengthy discussion of the 2010 study check out this Hydrogenaudio thread).

RINSE LATHER AND REPEAT WITH VINYL: Lots of claims are made for the analog nature of vinyl LPs and a small fortune is spent on esoteric turntables and phono gear as an analog source for pure analog high-end systems. When playing vintage analog-mastered music the audio never suffers the indignity of being whacked up into a bunch of numerical values and put back together. Much like the SACD test above, there have been various tests demonstrating even devout vinyl lovers can’t tell when you slip an A/D –> D/A loop into their otherwise all analog signal chain. Here’s a link to one but there are some better ones I’ll work on finding the links for. I’ve also done my own informal blind vinyl test on the sly. The vinyl lover wasn’t even aware he’d been listening to digital for several days on an extremely high-end all analog system. And this guy really hates digital anything. Don’t get me wrong, I own a nice turntable and I listen to vinyl. But for me it’s mostly about music that’s only available on vinyl. I don’t pretend it’s a technically superior format.

OBJECTIVE ISN’T EVERYTHING: I said above subjective stuff matters, and I’m including the reverse just to be clear. The numbers only tell part of the story. They make a convenient way to compare some things—especially say power output, output impedance, how suitable a given source/amp is for a particular headphone, frequency response, etc. But there are limits. If nothing else, great measurements provide a significant piece of mind for some people. They can relax and enjoy the music knowing their gear is among the most transparent available. And blind tests “wrap” subjective listening in a controlled, and more objective, environment free of the usual bias while keeping score. The two complement each other well.

WHEN SPECS ARE NOT ENOUGH:  First the easy part. For speakers, headphone and phono cartridges I think everyone agrees it’s tough to look at the specs and know exactly what they will sound like. You can still make some valid comparisons but the specs only give you a partial idea of the sound. With speakers and headphones the acoustics are a big part of the listening experience—all rooms and ear/head geometries are different. The sound of cartridges are altered by the tonearm and turntable geometry they’re used in (effective arm length, VTA, arm resonance, damping, etc.).  They also perform very differently playing worn vinyl as stylus tips come in an almost endless variety of shapes and sizes. So cartridge A rides in a different part of the groove than cartridge B. And measurements are limited by the relatively low resolution of vinyl test albums.  So, in other words, your mileage may vary and caveat emptor. You have to listen to speakers, headphones and cartridges to fully evaluate them. But that’s much less true with electronics.

GRAY AREA (added 5/31): There’s a solid consensus on the subjective nature of speakers, headphones and phono cartridges but what about everything else? Most objectivists will tell you a $20 well designed interconnect and a $200 well designed interconnect will sound the same. And that’s been demonstrated many times (see Wired Wisdom above). Hardcore objectivists (such as Peter Aczel) argue any amp that measures sufficiently well and is operated well within its limits will be indistinguishable from any other amp. And that’s been demonstrated in countless blind listening tests. But what about when an amp nears its limits? What if you have difficult to drive speakers for example that are 2 ohms at some frequencies? Will the cheap amp still sound just like the high-end model? Perhaps not. Some of these behaviors can at least be partly measured but some are more difficult. And what defines “measures sufficiently well”? Enough studies have been done it’s fairly safe to make several generalizations, but there’s still room for discussion and further research in some areas. I’ll hopefully be publishing a future blog article on the topic of correlating specs with listening observations.

MAGICAL THINKING: As someone pointed out in the comments, some audiophiles are in this for the “mysticism, magical thinking, and never ending quest” (their words) and I certainly know a few in that category. The purveyors of tube products tend to be rather clever in their marketing. They rarely make boastful performance claims and some offer hardly any specs at all. They know certain people enjoy their products and they make an appropriately subjective sales pitch. That’s hard to argue with. But when someone makes objective claims, and they’re far from being realistic, that’s just deceptive and wrong—regardless of the buyer’s priorities. It’s easy for some to get “sucked in” to the hype, myths, and mania when the real facts are constantly being swept under the rug. It’s not unlike Wine Spectator scores. See Peer Pressure above.

MORE INFO: If you’re seeking more interesting info check out:

TO BE CONTINUED… I have lots more to share on this topic, but for now, I’ll see what sort of response this brings.

101 comments:

  1. Thank you and well written.
    Feedback bullets:
    1. Just started reading, I appreciate the facts, can't suggest anything. Maybe a overall score rating system for gear?
    2. Audio gear/hifi/etc is still young and growing rapidly?
    3. If tests are by human ears, it is subjective. But subjective has it's place.
    So what is the proven method from paragraph two? Blind testing?
    Overall, please continue these awesome posts!

    ReplyDelete
  2. Thanks for the encouragement. Yes, blind testing is the "proven method". The test I described under Long Term Listening was especially telling as it demonstrated the reverse of what most subjective audiophiles believe--that A/B/X testing is actually more sensitive to audible differences than more typical extended listening tests. I suspect the "10 years" AES paper is floating around out there somewhere on the web. It's a good read if you can find it.

    ReplyDelete
  3. Regarding the personal attacks, circular arguments and attempts of diversion you have faced on certain forums, I applaud your ability to stay reasonably polite and calm.

    I consider myself an "objectivist" because my ideal of any recording and reproduction device is transparency (i.e., DACs, amps, etc. shouldn't have an audible "signature"), because good audio measurement instruments, including microphones, can discern much finer subtleties than any human (although wrong methodologies can still mislead), and because I, as a resource-constrained student, am always looking for a good price/performance ratio. I see audio devices merely as a means to an end and not as an end in itself, like some "subjectivists".
    Yet, I myself do not like to participate in these blind tests because I find such ultra-critical listening quite stressful and fatiguing, and tend to doubt my own perception in the course of it.

    Your blog is worthwhile, at least for me. But why do you personally keep buying all this gear, spend your time measuring them and writing articles even though you already have a more than adequate Benchmark DAC1 with integrated headphone amp?

    ReplyDelete
  4. Good points and a summary of the state of knowledge in the field.

    Measuring instruments tend not to suffer from expectation bias, and other factors that our own "wetware" can be subject to.

    The video of the Audio Myths workshop http://www.youtube.com/watch?v=BYTlN6wjcvQ
    and Ethan Winer's support files http://www.ethanwiner.com/aes/ shine some light on the matter.

    Perception may, indeed, be reality for the subjectivists. But, as Dr. Poppy Crum shows, your perception may not be what you think it is.

    ReplyDelete
  5. Continuing from comment #1 and the response. I was reading up on blind and ABX testing the other day and unless I'm mistaken blind testing and ABX are not the same. Plus double blind. Personally I would love to ABX all my purchases but I'm not Bill Gates. So I am thrilled to find your blog and hope you keep it up. I look forward to some analysis of gear I own.
    For example, currently, I am having a hard time finding the best setup for K701's. Maybe I will figure out the perfect amp for them and not buy snake oil thanks to your efforts.

    ReplyDelete
  6. Thanks for the last 3 comments. You're right about some blind listening being fatiguing when the differences are subtle. And we're of similar minds about accuracy.

    The AES Myths video seems like some good stuff but I've only watched part of it so far. I'll add the links to the article. Understanding how the brain hears things is a fascinating topic. It's like the naked person running across a football field during a key play. Few people even notice the naked person because their brain is wired to watch the game. We're the same way listening to gear. We subconsciously "wire" ourselves in ways that bias our listening.

    For the video, putting a junk solid state amp inside a tube McIntosh chassis and having audiophiles praise the sound quality is the sort of thing people like to laugh about and dismiss as a rare fluke. But there's something much more serious there. And such incidents are more common than most realize. It's amazing to me how that kind of thing isn't taken more seriously. What do examples like that say about the collective listening credibility of audiophiles?

    Audiophiles likes to think everyone involved in such embarrassing examples were just having an off day, had bad hearing, etc. They don't believe it would happen to them. But that argument falls apart really fast when you look at the vast number of people involved.

    As for "blind" and ABX you're correct there are some differences. I tried to keep this article more about the big picture and less about all the geeky details. But that's a topic for a future article.

    As for "why do you personally keep buying all this gear, spend your time measuring them and writing articles", that's a fair question--particularly in light of the recent unpleasantness.

    The short answer: I don't think enough of this information is getting out there and lots of money is wasted that would be much better spent elsewhere. I know "resource constrained students" eating boxed Mac & Cheese for dinner so they can afford their next expensive gear upgrade based entirely on biased subjective reviews and misleading hype masquerading as fact. I don't expect to make a huge difference, but I'd like to at least help get more facts out there and not dance around the 800 pound subjective industry gorilla in the room.

    ReplyDelete
  7. You seem to view this primarily as a case of innocent people being defrauded by an unscrupulous industry. I view this differently.
    I think that most "subjectivists" who have made buying, (subjectively) evaluating and collecting audio devices their hobby actually do have a desire for mysticism, magical thinking and a quest that must never end. These people would be bored, maybe disillusioned, once they had acquired (objective) reference devices and could therefore call it quits.
    One doesn't need to be an expert in audio engineering to recognize that most consumer audio magazines are actually advertorials given the kind of informal advertising language they employ to incite unrealistic expectations. It would only take critical thinking and a bit of heuristics.

    ReplyDelete
  8. That's a valid point and one I should probably make in the article as I largely agree with you. But it's the "most subjectivists" part that's significant to me.

    Plenty can afford the hobby but many more--especially newcomers--spend a rather stunning amount of their very limited disposable income chasing elusive "better sound" that's largely driven by hype and a sort of online "peer pressure".

    As for "unscrupulous industry" as I keep saying, numbers are numbers. And if the "industry" is going to publish specs and make factual claims they should either be close to correct or they shouldn't bother. I don't care who your target customers are. The objective stuff should be reasonably factual and not fabricated out of nowhere.

    I'm not trying to reach the "Robb Report" audiophiles or ruin their fun. I doubt many even find their way to this blog let alone read it.

    ReplyDelete
  9. Hello M8. Another great article. About the requested feedback:
    1-What would make this blog more useful?
    As I already talked with you, you could start adding information on the mods you sometimes make to the gear you review.

    2-Is this stuff worthwhile?

    Well, do YOU enjoy doing it? I like reading it. My chrome was an open tab to this blog and I do a refresh every day :) (stopped using rss a few years ago :) )

    3-Why do you think objective testing is so widely accepted for cars and PCs but not audio gear?

    IN PCs it is really "easy". You are testing mainly speed. It is very hard to make speed a subjective topic :) But it is happening. Hybrid hard drives reviews, the benchmarks say one thing, but the review normally says something like: "The numbers are not that great BUT...."

    4-What do you think of blind testing?

    Did them twice with my fathers gear (he is the subjective one). And once saved him 5-7K€ in an amp dispute (Musical Fidelity A3.5 vs McIntosh Pre+Power). And another time with some Supra speaker cables VS cheap (80 cents/meter cheap) cables. So for me they ARE the way to test gear.

    ReplyDelete
  10. Well said. Subjectivism? Objectivism? The bottom line is that we are using our brains to listen. What would '10ps of jitter' would mean if you can't just hear it? The psychoacoustic verification, such as DB-ABX test, is what the manufacturers should go after, if they were to make a real difference.

    BTW, Meyer and Moran's article in full text: http://drewdaniels.com/audible.pdf

    And its addendum on BAS:
    http://www.bostonaudiosociety.org/explanation.htm

    ReplyDelete
  11. Great post.
    1) I'd love to see more tests/reviews and comparisons (amps, DACs, DAPs, audio interfaces etc.). It's fun to see how cheap stuff can beat expensive high-end equipment.

    2) I think the main problem is that it's hard to relate audio measurements to what we hear. Combined with outrageous claims by people with golden ears, that have been accepted more and more in certain communities over the years, people have a hard time making sense of both subjective and objective data combined.

    3) What I think of blind testing? It can be a real eye-opener, but very tiring as well. :P

    ReplyDelete
  12. "Hearing is Believing vs. Believing is Hearing: Blind vs. Sighted Listening Tests, and Other Interesting Things" by Toole and Olive is an eye-opening AES paper showing how important blinding is even in the face of *obvious* differences. If you have not read it, highly recommended.

    W/R to credibility and reviewers having good listening skills, I attended an invite event with Damien Martin / Keith Johnson for their newish Entec subwoofers. Playing DAFOS master tapes on a five-figure custom transport, associated insane gear.

    A VERY well known golden ear reviewer that was present, hearing a performer fiddling with their instrument between takes asked a puzzled presenter what the strange sound was. Couldn't even identify that is was the same electric bass guitar that had been used in the just finished take, being 'chimed'. Appalling :-/

    Rob

    ReplyDelete
  13. BTW- I take umbrage ay your categorization of my eponymous magazine as 'elitist'.

    Watch it, or I may have to reconsider my advertisement on your site for my nine-sigma oxygen free cryogenic unidirectional quantum flux modified USB cable with vicuña insulation and Yangtze river dolphin sheathing (gamma radiated snakewood cable lift stands included at no extra charge, as is the oil to polish them...)

    Rob

    ReplyDelete
  14. Dear NwAvGuy, sure we all appreciate this blog, it is indeed very difficult to find people with such skills and not taking part into the audio industry BS.

    The only mistake you've done so far is not to approach the DIY in a more friendly manner, I think most errors they make are non-intentional and come from a lack of test gear and expertise.

    What would make this blog more useful?

    So far it is being very good. I'd miss more structure from the reviews. Also a list of updates would be great, so far it is difficult to track post's modifications. You also seem to delete some comments that have appeared (to incorporate them in the article), this is confusing.

    Is this stuff worthwhile?

    Totally yes. You'll help a lot of people to fall into audio BS. Also some objectiveness is needed in order to compare some competing audio design, like in the 3 channels case (just waiting you test a higher-end 3 channel amp, I wish I could send you one)

    Why do you think objective testing is so widely accepted for cars and PCs but not audio gear?

    For cars you have issue too of magazines which are supported by advertising. They often aren't very "precise" about the testing. IMHO, objective testing for audio gear didn't take off as almost no-one have been doing it in an appropriate manner.

    Also the audio scene would be very boring by using a "perfect" setup. People like differentiation, we don't all wear the same clothes. I guess the same happens with audio, and indeed, the "sonic signature" of a piece of gear is almost impossible to measure with current technology.

    What do you think of blind testing?

    I think it is a good way to un-mask fraudsters, both in the equipment scene and in the review scene.

    ReplyDelete
  15. As somebody with an electrical engineering degree and in the process of getting another one (specialty in communications but not on the RF/circuit/electromagnetics side, so analog electronics definitely not my area), there's no way I wouldn't find this relevant or interesting.


    The only blind testing I've done, and the easiest for anybody to do at home, is some personal A/B/X testing between lossless 16-bit 44.1 kHz and a couple compressed formats at different bitrates. What I find is that it's remarkably easy to imagine differences when none exist, even when blinded!

    The other conclusion from those tests was that personally on my gear and ears, mp3 encoded with modern LAME -V2 is not always transparent. It usually is, for me, but not quite on all passages. I've never been able to tell LAME -V0 from lossless CD audio on any real music sample I tried though. I likewise found modern Vorbis to do better on most tracks at equivalent bitrates than even LAME mp3.

    I guess Lossy vs. lossless is another one of the great holy wars, although it's easy enough for people to test for themselves what they need there. It's much harder for people to evaluate gear like in this blog, since few have the equipment, technical skills, and patience to run all the benchmarks you do.

    ReplyDelete
  16. Yet another in-depth and hence, outstanding article, that pin-points the big problem that the majority of audiophiles ignore (purposefully or ignorantly). Personally, I am massively frustrated with the advice given to new people at head-fi, especially headphone amps and needing more expensive amps to power headphones. Oh and then there's "synergy" with neutral amps - some more unaccountable bullshit.

    ReplyDelete
  17. This might be your best article yet and the one that might save your readers the most money. Your tone is a bit defensive at times, but given what others have thrown at you, that's totally understandable.

    I knew about some of the links (like Matrix) but had no idea about the other stuff you pointed out and still reading some of them.

    Your questions.....

    1) More reviews of affordable gear. I think one of the best things you have done is show price is a poor indicator of quality. I also would like to see more DIY stuff. I have not done any myself but I am tempted. And perhaps some of your own blind tests (more formal than you did with nuforce)?

    2) I dunno why this stuff is so suppressed. I guess its like the guy who drinks expensive wine and won't even taste the cheap stuff. Its his loss if he wants to get ripped off but it sucks those guys are trying to keep the rest of us in the dark.

    3) Blind testing rocks! Like I said I hope you do some of your own.

    ReplyDelete
  18. On a side note, a couple of years ago I tried hard but could no longer find a PS3 with SACD playback. The Meyer and Moran study that I did know made me happier for not having got one and spend money on SACD discs :)

    ReplyDelete
  19. Great article! Addresses reason and common sense that your critics obviously lack. IMHO, you ought to just ignore those fools in future. Just a waste of time.

    Keep up the outstanding work. I'm sure that beside me there are many who appreciate your that what you do (for us). And we stand behind you!


    As for your questions:

    1. This blog is fine as it is IMHO. Wouldn't know what to add or improve anyways. (Oh, there is one thing. I would love to see much more reviews. *cough, cough* ^_~)


    2. I would say that in the field of audio gear the narrow-minded subjectivist outnumber the objectivists by far. I personally -always- want facts and numbers before making a buying decision.


    3. Blind testing is a great thing! Always makes me happy when it turns out that my affordable gear sounds the same or even better than the exorbitantly priced guru stuff.

    ReplyDelete
  20. I don't know if you ever visit Head-Fi (anymore), but I found this article interesting and made me think of this blog post: http://www.head-fi.org/forum/thread/554205/dbt-why-the-stigma

    1. What would make this blog more useful? Is this stuff worthwhile?
    Every article is highly informative, and definitely is worth-while. This site just needs more advertisement (probably through word of mouth) so more people are able too see this side of the subjective vs. objective argument.

    2. Why do you think objective testing is so widely accepted for cars and PCs but not audio gear?
    See the above thread link for some ideas. It's more of a fantasy land that the majority have been inducted into (see:brain-washing). It seems that people wish to buy more expensive audiophile equipment to change their perception of sound, and not the physical properties of the sound wave itself. People need to be told both sides so they can make up their own mind.

    3. What do you think of blind testing?
    It's the only true, accountable way of proving audio equipment really is better than others. Headphones and such were created through the use of science, so to then measure the equipment through non-scientific means is illogical.

    ReplyDelete
  21. What would make this blog more useful? Is this stuff worthwhile?
    >Of course this is worthwhile,where would we go for this kind of data ?

    Why do you think objective testing is so widely accepted for cars and PCs but not audio gear?
    >probably because the facts/figures in cars and pcs are easire to relate to and have been made popular and widely available by the manufacturers themselves..

    What do you think of blind testing?
    >what better way to test ?

    ReplyDelete
  22. A very nice write. And really at a very interesting day when i read a Stereophile review of the Hifiman 601 that rated it as great while the measurements published were so bad that the review even included a manufacturer explanation of measurements vs designing by ear.

    Since i can not believe that science can lie and evolved ignoring figures count me as an objectivist.

    Your blog is the best web resource to learn about audio gear. Thus i hope to see more geat measured (current iPod Classic i.e.;)

    ReplyDelete
  23. 1) Your blog is totally worthwhile as it is. It would be better if you reviewed and measured more stuff but given the time it takes and the fact that few companies are going to send you review samples no matter how popular you get I don't think its anything that anyone will hold against you. Even if AMB and NuForce put out a hit on you and you publish no more you'll still have been a public service.

    2) I think subjective reviews are prevalent with audio because they're easier to do and because the market is relatively small. Cars are expensive enough and popular enough that lots of companies can make a profit reviewing them and afford all the proper test gear. Anyone can run some free or cheap benchmark software on a PC and make a website about it. I'd love to do what you do but it would only be a hobby so I couldn't shell out for that kind of equipment and I figure there are a lot of other people in a similar position.

    3) Blind testing is the best tool we have to determine what is and isn't audible but the fact that we need large sample sizes and the participation of self proclaimed "golden ears" to make solid conclusions makes it so difficult to use when evaluating physical equipment that measurements are usually more practical.

    Blind testing is almost impossible for headphones though. With only a few exceptions (eg Beyerdynamics of varying impedances) there is no practical way to blind such a test because you *have* to touch them and they will all feel different.

    Blind testing remains very useful on a personal level to evaluate bitrates and codecs. I tend to avoid audio codec testing though. If I become familiar with audio compression artifacts I'll never be able to ignore them and just listen. I already ruined myself for video codecs that way and I don't want to repeat it.

    ReplyDelete
  24. A huge THANK YOU for all the positive comments in the last day or so! Every comment so far has been published (no censorship) and I'm a bit surprised there are not at least a few trying to attack parts of the article (i.e. blind testing).

    NullZero said "the site needs more advertisement" through word of mouth (or keyboard) and that's something anyone reading this can help out with. I'll do my best to keep coming up with content people find useful and everyone's suggestions above really help.

    ReplyDelete
  25. I love this blog and keep coming back to it since that whole Nuforce debacle over at Head-Fi. I really wish more people over at Head-Fi would read your articles. Their wallet would thank them and I believe it will greatly help the DIY community as well. For me, DIY is the way to go in audio. There is so much more bang for your buck to be had there, as well as an understanding of the inner workings of audio equipment, thus making it easier not to fall into audio marketing BS.

    I'd love to see more reviews from you on good audio gear. Stuff you approve of, equipment that measures well, doesn't break the bank and most of all SOUNDS great.

    Another thing, how do YOU experience listening to music? What are your expectations from the gear you use and how do you set up your rig?

    Great blog, keep it up! I really like the writing style as well. You should do columns over at Headfonia or something. Just to counterbalance all the esoterics. :)

    ReplyDelete
  26. Thanks Erik for all the kind words. If a site with little or no commercial bias wants to give me column space, I'd happily provide content.

    The "how I experience music" question is a good topic for a future article. I'll work on that. The short answer: It's complicated! :)

    ReplyDelete
  27. By the way NwAvGuy, I hope you test cables. Statements like this

    http://www.amazon.com/Sennheiser-Open-Air-Audiophile-Headphone-Adapter/product-reviews/B0039L0O7G/ref=cm_cr_dp_all_helpful?ie=UTF8&showViewpoints=1&sortBy=bySubmissionDateDescending

    seems nonsense for me. Thanks for your blog.

    ReplyDelete
  28. The headphone cable thing is out of control. Most higher end headphones already use 4 wires in the cable. This prevents the cable from contributing significant crosstalk or other problems between the two channels.

    This is yet another "grain of truth" myth. A low impedance high-end headphone with a 3 wire cable (which are very rare) might benefit very slightly from an upgraded 4 wire cable. But upgrading the 4 wire Sennheisers linked above, with a Cardas cable is just a huge waste of money. The problem is finding willing listeners to prove it.

    It's not something you can measure as any sane cable will measure identically to the factory cable within the range of hearing on any test you can run. So, at best, you can only verify a cable upgrade is no worse than the factory cable.

    I'd be happy to blindfold someone and swap the factory Sennheiser HD650 cable for any of the Cardas upgrades (you provide the cable as I'm not blowing $200+ on pure snake oil) and see if they could tell which was which reliably. If the blind test is done properly I'm very confident they won't be able to tell. Anyone want to take that challenge and have the result published on YouTube?

    ReplyDelete
  29. Amen! Praise Jesus! Where were you 20 years ago when audio fantasy first became more popular than reality? You might have been able to stop the trend that's lead to an epic waste of money.

    I found a link to this article and just spent way too much time at work checking out some of your other articles. I'll be following this blog and hope you keep telling it like it is. Nice work!

    ReplyDelete
  30. What do you think about DACs in general, NwAvGuy? Do you believe that something like the EMU0404 which apparently measures very well is better than older vintage DACs such as the Assemblage or Parasound units which are very popular amongst the seniors in Head-Fi?

    ReplyDelete
  31. Hi,

    Thanks for all the articles (I am an objective type myself). However, you and I are not just about listening to music and part of the hobby is to "upgrade" our gear. I think your blog will be more useful (it is already very useful) if you continue to review/recommend low cost upgrades to low cost gear and backed with measurements. Often we don't care if we can't hear it, but it is fun to improve the specs.
    Thanks and keep up the good work.

    ReplyDelete
  32. Thanks for your reply, NwAvGuy.

    Then, if a non-stock cable will measure identically in all the usual tests (THD, response, etc...), how could it sound different?

    I thought a Cardas cable would measure differently.

    ReplyDelete
  33. The cable guys would like you to believe their cables measure different but, at audio frequencies, they rarely do.

    The ones that DO measure different at audio frequencies are intentionally DISTORTING the signal to help make the cable sound different (it might roll off the high frequencies for example to make it sound "warmer"). The Cardas cables, to my knowledge, don't use any of those techniques. So they will measure the same.

    The other part of your question is harder to answer. Some believe even if they measure the same they will sound the same. Others insist there are things we can't measure that can affect the sound. That's what blind tests are for. If those "other things" exist, and make a difference, they should show up in blind listening.

    ReplyDelete
  34. To Anonymous at 9:13 AM, Devices like the E-Mu 0404 (I have one) are more oriented towards recording than playback. The 0404 requires proprietary software drivers, has pro-audio connectors, and makes an especially poor headphone DAC. It's a very different product than most audiophile DACs.

    There are plenty of inexpensive DAC chips that can perform very well these days. But the implementation is where a lot of companies get something wrong--like NuForce did with the uDAC-2. So without proper measurements, you really don't know if a company got it right or not--regardless of what chips they used.

    For example, there are relatively large measurable differences in jitter performance between DACs--especially via USB. Some people seem more sensitive to jitter than others and it's not clear at what level jitter becomes audible. So I would be inclined to choose a DAC with lower jitter--as verified by independent measurements (versus the manufactures claims).

    ReplyDelete
  35. Why did you stop posting on HeadFi?

    ReplyDelete
  36. Everything was fine at Head-Fi until I said bad things about the uDAC-2 when NuForce is a Head-Fi sponsor. It's a long story but it ended with Jude, the owner, censoring several of my posts, and the posts of others trying to support me.

    Head-Fi is easily the most commercial popular audio forum with an abnormally large number of posts related to their sponsor's gear. Apparently those who want to share the objective truth are not very welcome there. Most of their sponsors sell gear almost entirely based on subjective reviews. Guys like me can be bad for the snake oil business.

    ReplyDelete
  37. May I ask exactly what Jude censored?

    Are you still allowed to post in the Sound Science forum? If so, please continue.

    ReplyDelete
  38. First Jude censored my profile and removed any mention of my blog followed by deleting any new posts with a link to this blog (even posts made by people other than me). Then he went nuts and censored any post that even mentioned this blog (no links at all). Finally he started censoring the posts from various people asking about the missing posts. I have screen shots of many posts he deleted before I gave up.

    None of the posts I made, or the ones made by others I have screen shots of, violated the Head-Fi Terms of Service. But apparently there's a general purpose "Jude can do whatever is best for Head-Fi" clause in the TOS. As many others have pointed out to me since, it's a very autocratic forum. And Jude's idea of "what's best for Head-Fi" is apparently what results in the most ad revenue.

    Bottom Line: I got the message loud and clear and Head-Fi doesn't seem like an organization I want to contribute to.

    ReplyDelete
  39. That really sucks.

    So why not continue to post in the Sound Science forum?

    Certainly your 3 channel review doesn't bother any of his sponsors. Can you post that without being censored?

    I'd love to see an M^3 review.

    ReplyDelete
  40. Sounds like an un-biased discussion place needs to be started (i.e forum). :P

    It's sad and quite frankly disappointing to see a forum like that run with commercial interest.

    ReplyDelete
  41. I agree about Head-Fi's blatant commercialism. But there are plenty of alternatives already out there and the masses still seem to mostly hang out at Head-Fi.

    Above all else, I'm a realist. If people want to have 200+ post threads about iPod LOD cables on Head-Fi, that's their business. I just hope at least some realize the bias at Head-Fi is very much towards the companies who profit from selling mostly snake oil.

    ReplyDelete
  42. A great read!

    questions:
    1)idk, good stuff tho.
    2)Because no one cares about sound!
    3)very useful, but a pain in the ass. I do generally trust my ears but if I can't ever ABX something I will accept there is no real difference.

    I have to say - I'm becoming more of an objectivist all the time, but I'll probably never be a complete objectivist (who always tend to buy really cheap gear).

    This might be a good link to add to the vinyl section:

    http://wiki.hydrogenaudio.org/index.php?title=Myths_(Vinyl)

    Keep up the good work!

    -Satellite

    ReplyDelete
  43. Hey Satellite, thanks for the comments. I added the link.

    I wouldn't say "complete objectivists" always buy really cheap gear. I own a Benchmark DAC1 Pre and other spendy gear. And I know plenty of hardcore objectivists who have invested well into 5 figure territory. It's not so much about cost as it is paying for tangible benefits--the Benchmark has better measurements, for example, than anything I know of for under $1600. Someone else might pay for something because it's well made in the USA and has a long warranty.

    And, as I said in the article, some things remain very subjective--especially speakers, headphones. and phono cartridges for turntables--they all sound different so it's difficult to choose one on specs alone.

    ReplyDelete
  44. Somebody will just tell you you need an Esoteric K-01 with an external cesium clock instead and that anything less is crap and an insult to be used with high end transducers...

    ReplyDelete
  45. As a recording artists it can be somewhat disheartening to see people give so much credit of good sound quality to various exotic devices. I think this blog will allow more people to just sit back, trust that their affordable equipment is transparent enough and just enjoy the music.

    You're doing us a great service!

    To answer your questions...

    1 - Keep testing affordable equipment, that's the best way to stay relevant to a greater amount of people. I would love to see amps with feature sets reviewed, such as the Practical Devices XM6 and Headstage Arrow 12HE. Worthwhile? You betcha!

    2 - A number of reasons. Car and PC specs can easily be verified by the consumers, but to properly test audio equipment requires expensive testing equipment and schooling. Manufacturers get away with whatever BS they make up - with your blog and the spreading of knowledge, this is slowly changing.

    3 - Blind testing is the only way to crack the golden eared, dogmatic audiophile who distrusts measurements. It can be equally useful for someone who doesn't know where his/her threshold lies - knowing that you can't hear a difference between 320kbps and uncompressed can be a relief.

    ReplyDelete
  46. Hi, interesting article.

    What would make this blog more useful? Is this stuff worthwhile?

    Yes, this stuff is worthwhile! What I would find more useful would be a wider selection of devices being tested. I think that a hard separation of measurements from interpretation and/or discusion may help communication wise. It's a style choice though.

    Why do you think objective testing is so widely accepted for cars and PCs but not audio gear?

    What I see is that in these two cases; objectivity is given it's well deserved place in the domain it belongs, and subjectivity is given it's own as well. In a PC the performance can be benchmarked, but the software (OS) and how people experience the use of the computer lies in the domain of the subjective.

    In a car a dyno will tell you the horsepower, but only sitting in it will let you know what the feeling of acceleration is, if it's got a blind spot backing up, or if the thing understeers going into a corner.

    We accept the objective testing easily in these cases because the subjective and objective elements are easily differentiated from one another.

    With audio gear we've got a mess, it's hard for people to differentiate between the objective and subjective domains. It seems people either fuse the two domains together, or completely disassociate the two.

    My suspicion (bit of an awkward statement ahead) is that because audio gear deals directly with one of our primarily senses people have a very difficult time differentiating between the objective and subjective parts.

    I don't think that the PC and the car have the same kind of problem because we relate to and experience what they 'do' in a much different way than audio gear's performance. Our experience of audio gear is both much 'deeper' and at the same time simpler. Attempts to be rational about it is just going to be much harder to pull off.

    What do you think of blind testing?

    I think it's great. It tests the perception of the listener, hile the technical empirical measurement of the thing doesn't even have to enter the picture. It's not perfect or beyond criticism, but it does meet our human subjectivity and perception head on.

    ReplyDelete
  47. Thank you very much for sharing your wisdom/experiences and those articles.

    1- This blog is definitely worthwhile.You can test equipments those are well known/popular amongst "audiophile" communities.Like cables , capacitors , opamp miracles etc.

    2- Analyzing audio systems isn't easy as PC's or Cars.It requires good testing equipment and theory behind it.People spend their money on "audiophile" gears rather than testing equipment and technical books.And so we see the result.

    ReplyDelete
  48. So... is this better than the Mini^3? I suspect it is. I'm sure according to NwAvGuy this giant killer should be better than M^3 and Beta22 too!

    ReplyDelete
  49. Thanks to maverickronin, b0ck3n, jadeeast and pea for your comments. To the last Anon post, I'm not sure what you're referring to being better than the various AMB designs?

    I haven't tested the M3 or Beta22 but I have shown, in multiple ways, 3 Channel Designs are inherently inferior in several objective ways. Several of the benefits claimed for 3 channel designs are simply false.

    ReplyDelete
  50. I'm listening to music from my computer through a goodie that arrived in the mail today. Thanks for helping me pick one!

    Originally I wanted to get another model, that you also reviewed, but decided against it after looking through the more concrete findings that are helpful, unlike those superficial superlatives "warmth" "depth" (so many words that say so little about what I'm BUYING) found in other reviews.

    Just curious -- how easily discernible is the difference from a 128kbps file to a 256kbps or even 190kbps? It seems noticeable even playing straight through my 4-year-old MacBook audio-out, but I don't want to say it's because I'm biased against the lower resolution files and just tricking myself :-(

    Also, just to point out from one of your linked "Interesting Sites":
    [quote]
    "(Note that the opposite of Subjectivist is not "Objectivist". I understand this term refers to the followers- if any- of the philosophies of Ayn Rand)
    [/quote]
    -- from http://douglas-self.com/ampins/pseudo/subjectv.htm

    ReplyDelete
  51. Dave, thanks for the comments. You can answer your own question about bit rates and lossy compression using Foobar 2000 and the ABX add-on for it. I provide some links in the first NuForce uDAC Listening Challenge article (March).

    Foobar ABX lets you play say a 192K version ("A") and 256K version ("B") of the same track. You can then click A to hear the lower bit rate, click B to hear the higher bit rate, and go back and forth as much as you want. When you think you're ready, click "X" and the software plays either A or B but you don't know which. You vote and it keeps score. It removes all the subjective bias.

    ReplyDelete
  52. Thanks, I'll definitely give that a try!

    Nice, just noticed Sennheiser 650 article's up -- you wouldn't happen to do Grado as well? I've a set of 325s coming in which I'll use for the ABX test.

    ReplyDelete
  53. Thanks for this article, very interesting and well written.

    ReplyDelete
  54. Superb article MwAvGuy.

    I am proud of my contribution to the 'objective' cause which is the Testing Audiophile Myths and Claims thread on Head-fi, the first time blind/ABX tests had been gathered together in one place.

    At least to Head-fi's credit, the thread has been allowed to run and has survived a few attempts to ruin it when mods have stepped in at my request to stop and remove trolls.

    I see the objectivist side as serving two benefits to the audiophile community. We are showing how great sound can be had for not a huge amount of money and we are taking away a lot of 'subjectivist' fretting about their hifi's sound quality. It is ironic that the objectivists are often told to leave the measurements and learn to listen and enjoy the music, yet I am sure the objectivists are more satisfied with their hifi's sound quality than others.

    ReplyDelete
  55. Thanks Prog Rock Man. We need more threads like yours, on more forums, and perhaps eventually blind testing will become more popular again. It's really a shame it's been marginalized the way it has. But if you follow the money, it's hardly surprising. Any effort to help educate others is going to have be largely non-commercial and grass roots.

    As for Head-Fi, I think once something has a large enough critical mass there Jude and crowd would have a difficult time trying to squash it. If you look at the stats, the Sound Science forum is a drop in the bucket for page views. So they probably figure 98% of Head-Fi visitors will never find your thread. And they're probably right. For more about Head-Fi's priorities some might want to check out: Banned at Head-Fi

    ReplyDelete
  56. I take the view that no measure (written, that is) is too extreme when it comes to attacking an irrational position. I see no evidence of any restraint being exercised by the subjectivists and in my view their position has only gained it's current credence due to the failure of objectivists to adopt similar tactics, perhaps because of an aversion to confrontation.

    Anyway, I like your site a lot.

    I've been working on a no-compromise headphone amp for a while now, but I see you have beaten me to the punch, you are a lot better resourced in terms of testgear than I. I look forward to seeing the technical details of what you have done.

    w

    ReplyDelete
  57. I haven't seen your original comments on the (or any) NuForce DACs, but I bought the original NuForce USB DAC a few years ago based on the hype on the internet. EVERYONE raved about how good these were, especially for slightly under $100. I did not see one bad review! All the reviewers were astounded at how good it was. So I bought one to use at work (to plug into my computer's USB port), plugged in my headphones, and listened, expecting glorious sound. What I hear were a few very small differences in tonality here and there. Certainly not "amazing" or "astounding" or "revealing". I bought it through Amazon and returned it to the manufacturer for a refund and posted a review on the web site of the store that sold it. My reply got a reply from someone who worked for NuForce saying I should try the newer version of the unit or maybe one it's big brothers. I never did, though. I just sent it back. I got my money back and no complaints from the manufacturer - no problems - but I felt like I had definitely been misled by the raves on the internet. Now when I read rave reviews all over the internet for a product, I take it with a grain of salt. I have a very strong feeling there are lot of shills for a lot of these companies out there posting great reviews of those companies' products. I read in the news recently something about a number of product review -type posts on Amazon were written by people who didn't even own the product.

    I read magazines and web sites of U.S. audio magazines and they seem to be very reluctant to call product "x" a piece of junk even if it is. I read a review of an audio product a few days ago where they said the bass sounded sloppy and limited, the highs seemed rolled off, etc. and in the wrap-up at the end of the article they praise it for being such good product! What kind of b.s. is that?

    I recently stumbled onto a British web site for "What Hi-Fi" magazine. I've seen their magazine in the racks at book stores before but only browsed them occasionally as I did various other U.S. audio mags I could find. Well, the product reviews on their web site were interesting - they seemed to be very direct about a piece of equipment they reviewed. Their reviews are pretty short compared to some other mags/sites. They'll review some expensive amp (for example) and maybe say in so many words that it doesn't sound good to them and that you should spend your money on something else instead. I've seen a number of two star ratings for products they didn't like. I much prefer that to the American magazine "suck-up" style where they say, maybe, "the highs are rolled off" and then insert the weasel word "but...".

    I also enjoy photography and noticed this different in American vs. British camera reviews (before I noticed it with audio reviews). The American mags seem to like everything, even things that don't measure up. The British mags will say that the product doesn't measure up and recommend you try other (unnamed)products instead.

    I think, at least in America, these mags and their advertisers may be "in cahoots", as we say in America, or as some would say, "in bed together". I can't prove it; I don't say that is true; I say only that it just a strong suspicion on my part.

    ReplyDelete
  58. On the whole objective vs. subjective thing, I listened to a number of different CD players in my home system a few years ago when my CD player started dying. I could definitely hear differences between the various CD players - between each other and between each one and mine. The differences weren't huge and vast and stunning, though. There was no "1,000 veils removed" kind of thing. All were priced around $1000-$2000 U.S. One I listened to had such a forward midrange that it almost drove me out of the room. Another I listened to was more "normal"(?) in the midrange, and had deep bass but the mid-bass I like seemed lacking.

    I listened to a Triode brand 40w/ch. tube integrated amp at a dealer recently (I've heard all the hype about tubes from tube lovers and wanted to hear for myself what it was all about). I compared it to a Naim solid-state integrated amp in the same general price range using the same speakers. There were definitely obvious differences in the sound to me. On some songs, the Naim sounded, by comparison, rather flat and boring. On some songs, the tube sounded "better" to me in that the sense of instruments in space seemed noticeable better (not on everything, though).

    Through my headphones, though, a lot of music I played through the tube amp seemed much less pleasing, to my ear, than what i had at home.

    Objectivists measure electronics and tell you what they measure. Subjectivists listen and tell you what they hear. They don't have to be mutually exclusive clubs. Just because the subjectivist says that this amp sounds different than that amp (or CD player, or preamp, or DAC) and the objectivist measures the electronics and says there is no measurable difference doesn't prove that there truly is no difference. Perhaps the objectivist isn't measuring the right thing, or never thought to measure "x" also. The subjectivist, too, is subject too mood (music sounds much better and more fun to me when I'm happy and after a couple of margaritas!). When I'm in a crummy mood or tired, I really notice the flaws in the reproduction and say to myself, "this sounds like crap!", when, in reality, nothing has changed but my attitude.

    I think there are definitely differences in sound between some amps, preamps, CD players, etc. I haven't heard enough of these various electronics to say that they -all- sound different. It does seem reasonable to me, as a non-engineer, that "stuff" between the source of the music (CD or LP) and the speaker could change the way it sounds.

    ReplyDelete
  59. Audio equipment makers usually want their equipment to as aesthetically pleasing as possible. Some even go to great expense to give the equipment a lot of "bling". Reviewer always, always, always comment on how "pretty" a piece of equipment looks.

    Reviewers also value physical weight of a product. If the product being reviewed weighs 100 lbs. they say, wow, this thing is built like a tank and must be on hell of a work of engineering, as if the weight of the product has any bearing on how well it works or sounds. They always seem to take the cover off the electronics and peer around inside. If it's very tidy, has a physically large in size power supply, has famous name parts, they go "ooh", "ahhh", and I think that probably influences them a lot. One review I read (by someone not a professional reviewer) said they were so impressed with the way a piece of equipment looked that it was for sure going to end up in their audio system - before even listening to it.

    I think audio reviewers should have someone else unbox the product to be reviewed, set it up in a room, cover it with a sheet, hook everything up, and then let the reviewer listen to it. That way the reviewer is not influenced by the looks or build or size or weight of the product or even the manufacturer. They are just told that they have a new amp or preamp or CD player or whatever to listen to. They listen to it as much as they want, take notes, write up their review, and then they get to know who the manufacturer is and can poke at it all they want.

    I don't know how to prevent manufacturers from influencing the results of the reviews, though. If Conrad-Johnson loans to a mega-bucks amp to review and they happen to be one of your magazine's biggest advertisers and you slam their amp, what's to say CJ won't pull their ads from your magazine? If they do that, you lose a lot of money. Okay, so the CJ gets a good review.

    ReplyDelete
  60. To Anon at 2:28 PM about CD players and amps sounding different. I fear you either missed, or are ignoring, a huge part of this article: Sighted listening, as you did in the store and at home with the CD players, is ALWAYS biased.

    If you watch the Mcgurk Effect video linked, you'll see how our brain deceives us with what we hear. Even the researcher in the video, who's been studying this stuff for years, isn't immune to it. It's involuntary.

    And the CD players in the Matrix HiFi test linked were massively different. One a cheapo Sony from WalMart and the other a Wadia with a 4 figure price tag. Yet nobody in that group could tell them apart when they didn't know which was which. So think what you want, but until you do it blind, your brain is deceiving you.

    People think sonic bricks change the sound if they expect them to. That doesn't mean the bricks are really changing anything besides your bank account. Put them under a bedsheet as in the Matrix test and see what happens.

    It is possible the tube amp sounded different because tube gear tends to alter the sound in audible ways. You're sometimes literally listening to the amp playing along with your music. It's like having someone murmuring in the background along with your music. If you like that sort of thing, fine. But the tube amp is actually less true to the music and what the recording engineer intended.

    ReplyDelete
  61. A question for NwAvGuy:

    I'm not an electrical engineer and know nothing about circuit design, or how to build an amplifer, or CD player, etc, etc. All I know is that an amp is a box of electronic "gizmos" that somehow increase the power out of a source so it can be heard through speakers.

    So isn't it possible that one box of "gizmos" could sound different than another box of "gizmos"? I mean, I read all the time about power supplies, capacitor quality, amplifier topology (there seem to be a lot of different ways to build an amplifier from what the reviewers say). Don't most amplifiers include computer chips now too? Couldn't the programs running on the chip have potentially significant influence on the final sound?

    Does anyone ever measure harmonics when measuring audiophile electronics? I can't say I can define "harmonics" but I have read that there are even-order harmonics and odd-order harmonics and, according to the reviewers, they supposedly affect the sound of equipment.

    ReplyDelete
  62. To anon above, Yes I measure harmonics in all my reviews on this blog. And I often point out the even-order and odd-order ones in the spectrum graphs. But if they're below a certain threshold they're generally considered inaudible and that's been demonstrated in many different studies. So the key is designing an amplifier that's "good enough". I talk about what's "good enough" in a headphone amp more in my O2 Headphone Amp article.

    And yes different boxes of gizmos can sound different. But they often sound the same. Consider 6 different calculators on your desk. You punch the same numbers into them and they'll all give you the exact same answer. They all have different gizmos in them too but they perform their intended task the same. If you had really advanced equipment you might discover some put the answer on the display a little faster than others, but for their intended use that makes no difference. They're all "fast enough" and yield the same result. A human can't tell them apart.

    And no, headphone amplifiers don't usually have computer chips in them. Products like the FiiO E7 do because they're also a DAC with other features. But pure amps usually don't.

    The bottom line is guys like anon above are just ignoring all the studies, blind tests, the Mcgurk effect, etc. and throwing out really broad general arguments like "there are different parts in them so they must sound different". But it doesn't work that way. A lot of parts don't really have a "sound" and it comes down to the skill of the designer.

    And if the differences are there, why do they disappear when you simply throw a bed sheet over the stack of equipment leaving just the speakers exposed (the Matrix Audio test and many more like it)?

    ReplyDelete
  63. Not sure if you've linked to this elsewhere - an excellent piece by Roger Russell on the wire game:
    http://www.roger-russell.com/wire/wire.htm

    Links therein have a couple articles on damping on full-size home systems.

    ReplyDelete
  64. Good article. A few points would need some comment, but it would take almost the same amounts of characters :)

    Therefore, i'll comment only on one thing: methododogy.
    I like and have done blind tests, but i hate the abx methodology, unless i'm trying to only recognize if two pieces are different, and i know everything else very well (my system, and my source material). Otherwise the mental fatigue and anxiety would inhibit any result.
    I have done a blind test abx with cables along with friends and 1 f 4 had a % statistically sufficient to recognize a difference - reason the source material was his.

    For qualitative evaluation, between two pieces that i dont know like my pockets i prefer A/B, blind better, but i consider myself one of those little subsceptible by the looks.
    I liked the paragraph about long-term listening - i have never believed it and i'm glad it's been debunked here. Some people need a very long time to understand and judge the sonics of a piece, and then they can AB it. I need from 20s to a few minutes.
    On a funny side, i never ask the pricetag before doing any comparison -another thing out of the equation for the complicated mind.

    ReplyDelete
  65. Thanks for your comments Telstar. It's rare anyone gets a valid result from blind cable testing but it has happened and there can be some valid, and measurable, reasons for it.

    I also agree about fatigue and that ABX is best to determine if there's any difference at all. If differences are found other tests can help sort out which one is preferred. And blind testing of any sort is best with familiar source material and as much else familiar as possible.

    That's why I like tests like Wired Wisdom (the cable test linked in the article). It was done in the homes of the audiophiles using their systems, their source material, etc. It's just too bad it's not as easy as slipping brown paper bags on bottles of wine. :)

    ReplyDelete
  66. Well to be honest, I can't even discern the difference between high bitrate mp3/aac vs lossless files but some claim that Flac just sounds better due to the dynamic range and what-not. I suppose to some it just makes them feel better knowing that they have the best sample (lossless).

    Now to the main point of my post, I've been reading up on a couple of pricey portable amps/dacs. The Fostex HP-P1 ( http://www.fostexinternational.com/docs/products/HP-P1.shtml ) and the Cypher Labs Algorhythm Solo ( http://cypherlabs.com/blog/post/why_buy_a_solo ) had pretty interesting descriptions.

    Apparently these two devices have the ability to unlock iDevices' potential and produce bit-perfect playback. Cypher Labs claimed that most DAPs have their true potential locked away (or compressed, something along those lines) and what the CLAS does is unlock these ... ... ...

    I know nothing about the technical aspect of things but the descriptions of the hp-p1 and CLAS sound like a lot of the usual advertising mumbo jumbo.

    Sometimes I wonder to what lengths will these "audiophiles" go to achieve their audio nirvana...

    ReplyDelete
  67. I am interested in the CLAS and the Fostex. After reading this blog I am quite turned off but I wanted to get your opinion , NWAVGUY, on the subject. Currently I have a iPod classic 240gb :) using westone 4 IEMS. Compared to my galaxy S2 phone the sound is not quite there and the sound is great once i hear music loudly. There seems to be a jump in sound once I pass a certain threshold ( But of course listening to anything loud is bad for you). I used to have the audinst HUD-MX1 and I liked the AMP/DAC combo much better than out of my comp. But since I'm on the move more than normal I wante to get that sound I once had in portable form.

    ReplyDelete
  68. Any opinions on the CLAS or HP-P1? I used to have the audinst HUD-MX1 but I sold it since I'm on the move nowadays. I miss the better sound I used to have on my laptop and want to recreate it. I have an ipod classic with the westone 4 IEM and the sound is alot weaker than it is on my Galaxy S2.

    ReplyDelete
  69. If you want a good on-the-go amp for your Westones I would suggest the FiiO E5 (or wait for my E6 review coming soon as it's supposedly even better). It has a low output impedance and plenty of power for the Westones. It's also really small and easy to travel with. Westones, being balanced armature, require a very low output impedance (under 2 ohms) which many devices don't have. With higher output imepdances their sound will significantly change. That likely accounts for most of the differences you're hearing. See my February articles on impedance.

    ReplyDelete
  70. Thanks I read you article! Your a god send. I went into the other articles also and I'm found a DIYer to build me an o2 amp! I'm in Korea so somethings are harder to come by but thanks for having this blog!

    ReplyDelete
  71. Hi NwAvGuy, greetings from MatrixHiFi. Having noticed you had spent some time reading our tests, I'd encourage you and anyone reading to take a minute to check the test of all tests when talking about CD players: The much praised dCS Verdi/Delius transport/DAC combo vs a dated Denon from 1987... you can imagine the results even if reading the (for the time being) Spanish version: http://www.matrixhifi.com/pc_scala_denon.htm

    Regards, MatrixHiFi

    ReplyDelete
  72. I'm a UK consumer journalist working in a totally different field (moto) but we test bikes, tyre performance and suspension and the objective/subjective gray area here is a similar sort of car crash...

    If someone spends a wad of cash on a part, they'd be brave to say it was crap, whether its wheels or a DAC.

    And as for the pleas to 'test more', well, it strikes me that most consumers want to be taken by the hand and led to the ATM/shop. Like we always say, if you like your bike, what the hell do you care what we think, because we're not riding it.

    Nice work sir, carry on.

    ReplyDelete
  73. Thank You!

    Unbelievably refreshing...

    ReplyDelete
  74. An eye opener and a well written blog. I ended up here for I'm concern about buying an amp and dac. Well , I want to buy AKG 702 and ATH-ES29A for my sister. I'm planning to hook it up either on the PS3 or PC. I visited many popular audiophile sites. Most of the forums post said that these headphones needs a dac and an amp but almost all the products they suggest for the headphones are like $300 plus each. I really don't know if I really needed those expensive amps and dacs or would a sub $300 amps and dacs sufficient enough to drive these headphones.....Any suggestion will be appreciated.....***sorry for my English***

    ReplyDelete
  75. @Anon, I don't know about the ES29A, but the 702 does need an amp for most applications. Consider an assembled O2 Amplifier which is well under $300. If you need a DAC or not depends on your source. You might be happy with the FiiO E10. This is off topic for this article.

    ReplyDelete
  76. @NwAvGuy

    THanks for your prompt interesting reply. I think I have to spend more time reading your 02 Amplifier article because its a bit too long and technical for me.Probably I'll allocate my weekend for reading it.Hopefully my little knowledge about electronics will help me understand the article......God Bless and hopefully more people will visit and read your blog..

    ReplyDelete
  77. I wanted to thank you for your practical and informative articles. I feel like I owe you since my reading here has essentially saved me from an unwise purchase (Little Dot MK VIII SE) I was about the pull the trigger on JUST before I found your blog. (I did a Google search for 'hd-280-pro vs DT770' which yielded your DT770 review page as the 5th result, and the 1st result not from head-fi.org, which I had been absorbing the last few weeks a bit too credulously it seems).

    Thanks again!

    ReplyDelete
  78. Reading this post reminds me of Audion vs. SoundJam, two competing music player programs. This was back when MP3 players were just starting to become popular, and before iTunes was released.
    http://panic.com/extras/audionstory/

    It's an interesting read. Many people claimed Audion sounded "better" even though they were both using standard MP3 decoding software without any kind of special filtering or enhancements.

    ReplyDelete
  79. Regarding Tom Nousaine's Wired Wisdom article in Sound and Vision, you characterize Alan Lofft's subsequent editorial as an attempt to "dance around the Wired Wisdom article--likely to appease their cable advertisers." I did not notice this at all in reading his editorial. He seemed to accept the fact that there is no audible difference between high-end and cheap speaker cables, and likened the audiophile community's insistence to the contrary to a religious-like desire to believe in the "mystical" over the "mundane" -- essentially because it is more "fun."

    Yes, he was tactful in his presentation, but his conclusion was much the same as your own: that the laws of physics are predictable and authoritative, and that "true believers" will be unswayed by the evidence for reasons that "parallel" religion. His final two sentences posit a placebo effect and offer a bit of a jibe to the subjectivists: "Now, where are those crystals..." (referencing his earlier remark: "an eighties yuppie seduced by crystals from California and the nutball babbling of Shirley Maclaine...") This doesn't sound particularly "appeasing" to me. Perhaps your only real gripe with Alan Lofft is that he should have been less tactful? My speculation is that your own difficulties in dealing with many subjectivists may simply be a matter of tact more than content. Possible?

    ReplyDelete
    Replies
    1. Thanks for your comments, you're not the first to "read" Alan's editorial differently than I did. I'm not a psychologist and I'm biased by having worked in the industry for a while. Alan wasn't denying the cables sounded the same, but I got the impression he was trying to "sugar coat" the outcome so as not to offend those who want to spend money on upgraded cables. I think it's partly tact and partly trying to keep readers spending money with advertisers for "audio jewelry".

      Delete
  80. For those interested: http://www.headfonia.com/on-sound/

    The statement that immediately grabbed me is the following: "the human sense is far superior than any of the currently available machinery when it comes to observing the physical world" Perhaps this is true to some extent; the ability of the human brain to sift through the vast amount of sensory data that constantly bombards it is truly remarkable -- arguably making it the most advanced 'machine' in existence -- but claiming human sense is "far superior" is a gross overstatement, in my opinion.

    Of course, given that I consider myself a man of science, I am, perhaps, susceptible to some bias. I am curious what others make of this article in the context of objective vs. subjective.

    Oh, and as this is my first post, I'd like to say 'thanks' to everyone (especially NwAvGuy!) for creating and fostering this environment. I've learned more on this site in the last week that the last 5 years on head-fi :)

    ReplyDelete
  81. The "human sense" when it comes to high quality audio reproduction has been fairly well tested and documented. There are several links in this article to such documentation and many more to be found at AES and elsewhere.

    Many writers, magazines, and websites like Headphonia like to wax poetic about how humans can hear a pin drop across a crowded football stadium, but the fact is, when you put a blindfold on those same humans to remove the bias, everything changes. It's been very well documented over and over.

    I welcome anyone from Headfonia to take any of my blind listening challenges. I'll give $500 to the charity of their choice if they can hear what they frequently claim to easily hear when they're listening blind.

    ReplyDelete
  82. I'm curious how the perceived nature of a format like SACD, could influence the production/mixing of the SACD compared to a CD, maybe SACD will sound better because it's been given a different EQ and less compression since it's expected someones playing it on a dedicated equipment, compared to a car radio with heaps of cabin noise, broadcast over FM radio, or played on an average mp3 player with $5 headphones on that noisy bus ride to school.

    ReplyDelete
    Replies
    1. Meyer & Moran conducted a well run study lasting more than a year. They inserted a 16/44 A/D and D/A into a high resolution SACD signal path and, not surprisingly, nobody could tell the difference.

      That said, many SACDs are arguably better mixed/mastered than their CD counterparts (or even their CD layers/sides). It's no secret many SACD's are mastered to a different standard than mainstream music. The real benefit of SACD is in the sometimes improved mastering rather than any technical advantage of the format itself.

      Delete
  83. I'm sorry to disappoint you NwAvGuy, but you've got it all wrong. Thing is that you can only make useful comparisons once you've got the basic setup. For instance only vinyl source can be used, And then the vinyl temperature needs to be stabilised at 21-22c. And the entire equipment needs to be switched on and stabilised for at least 24 hrs. And of course oxygen-free copper, hardly the latest of gimmicks is it. And that's just the basics. I could go on here but I don't feel like I'm getting a "blind test" hearing from you.

    ReplyDelete
    Replies
    1. And don't forget to orient the amplifier with the output jacks towards magnetic north and to only test during the correct phase of the moon! I'm not sure if your post is a sarcastic joke, or sincere, but either way it's representative of some of the more deluded members of the audiophile community.

      Delete
    2. I don't know about you NwAvGuy . . . but I personally think that RobDob was joking.

      Delete
  84. Hey, I would like to know if most people would notice a difference between something like this

    http://www.amazon.com/SendStation-PDLO-MiU5-PocketDock-Adapter-iPhone/dp/B002ACNZFI/ref=sr_1_1?s=electronics&ie=UTF8&qid=1331562160&sr=1-1

    and just plugging in your headphones straight into the standard jack built-in to an iPod. Has anybody conducted a blind test for something like this? If you hear a difference when not doing a blind test is it likely that you are imagining the difference?

    ReplyDelete
    Replies
    1. That's nothing more than a Line Out Dock adapter with a USB jack for charging. I have measured my iPod Touch 3G using a LOD cable and it does measure slightly better than the headphone jack. You do, however, lose use of the volume control on the iPod.

      I haven't done a blind comparison as that would be somewhat tricky. But if you were to say feed an O2 amp from the iPod headphone jack vs the Line Out I suspect you would have a hard time hearing much difference under most conditions. But the LOD does, in theory, have a small advantage when connecting an iPod to an amplifier, powered speakers, receiver, etc.

      Delete
    2. Hmm, I just got an idea . . . by using something like that product could you avoid amplifying the amp inside an iPod when using a pair of noise cancelling headphones? If you were able to do this and just have the digital signal that is your music go directly into the internal amp of your noise cancelling cans that would be just awesome cause I could avoid amplifying (and thus making more obvious) the imperfections of the amp in the iPod . . . I believe that you think that the amps inside an iPod are sufficient enough or even good [I'm not arguing that as I haven't even been an 'audiophile' (I'm don't really consider myself a hardcore audiophile . . . I just have some audiophile like traits. Besides, I think that hardcore audiophiles are ridiculous) for a year yet so I wouldn't know] but still I know that every amp has it's imperfections and I would like to avoid amplifying the imperfections. This whole post may just be ridiculous but in theory, if you could bypass the amp in your iPod and just use the amp in your cans . . . wouldn't that remove (or at least make less obvious) the audible hissing and other things associated with noise cancelling headphones?

      Delete
    3. You're partly correct. Ideally you don't want any extra amplifier stages in the signal path. But, in practice, most music we listen to has already been through many (often dozens) amplifier stages before it ends up on a CD. So adding one more stage is not a big deal.

      And there are some problems with what you suggest. As I said, the iPods volume control doesn't work on the Line Out (LOD). So your powered noise canceling headphones would have to have their own volume control and you would have to use it and not the one on the iPod. They also would have to be designed to work with only 0.5 volts of input.

      In every noise cancelling headphone I've heard, the hiss from the noise cancelling electronics is worse than the hiss from the iPod's headphone amp. Some of the headphones have gain--they boost the volume so they also boost any hiss by the same amount. I doubt using the LOD would make the hiss go away although it might slightly help in some circumstances.

      The headphone amp in the iPod is respectably quiet. In fact, because the headphone jack is capable of more output, in terms of pure Signal to Noise Ratio, the headphone jack might be slightly quieter than the LOD when you compare the noise against the maximum output of each.

      Also, what you describe is not using "the digital signal". The line output is still an analog signal much like what comes out of the headphone jack. It's just taken slightly "upstream" in the iPod's analog signal path. Many iPods do have a digital output but it's proprietary and that would require even more specialized powered headphones that had their own DAC built in. There are outboard headphone DACs that use the digital signal from the iPod. But I'm skeptical if they sound any better than the DAC in the iPod. And many of them cost as much or more than the iPod itself.

      The main benefit of the digital output is for use with home and car audio systems that convert everything to digital anyway. They already have their own DACs so it saves a D/A and A/D conversion to use the iPod's digital output. But that doesn't apply to headphone listening where you're better off using the analog outputs.

      An iPod Touch 3G, from the Line Out or headphone jack, is capable of amazingly good sound quality if you use a decent headphone amp to lower the output impedance and provide more power (if your headphones need more). A lot of the products that claim to improve the iPod's sound are snake oil.

      Delete
  85. Sorry, as I said in my comment, I don't know much . . . and thank you.

    ReplyDelete
  86. I just wanted to thank you for this blog. Having read and participated in many many subjectivist vs. objectivist audio debates over the years, yours is one of the best summaries I've ever read. You and I know that the fervent audio "true believers" are immune to objective evidence, but efforts like yours are useful in opening the eyes of those who have yet to choose mysticism over reason.

    ReplyDelete
  87. From an economic viewpoint I believe your case for Objetivists is not as strong as it could be. Cars in any market segmet are all but virtually indistinguishable. All relevant data are easily measurable, probably accurate and readily published. Still you'll find the same lyrical language in car reviews as in audio brochures. It's an emotional affair, perfectly human, not bad at all.

    Snakeoil buyers are too easy a target. More likely there's evidence for a law of diminishing returns. Meaning beyond a certain price level spending ten times more on equipment will not make it ten times better, but only slightly. It's up to this level where your reasoning is valid imho. Still going cheap is almost surely not the best choice under all circumstances.

    A Subjectivists long term blind test I'd like to see done is feeding listeners with a signal from different sources, so that they can conveniently switch between them at their homes. Usage should be tracked. My thesis is, there will be a consistent revealed preference over time.

    ReplyDelete
  88. I would like to start off with saying I think this blog is amazing, but I think you were a bit too "nice" with regards to discussing speakers/headphones general lack of objectivity. Here is my take on the whole situation/some thoughts I had. You seem like an intelligent and reasonable human with regards to their passion for audio, so I would value your opinion on them.

    While I agree it is near impossible to look at the specs of a speaker system and judge its quality and sonic signatures; however, I believe the cause of this is just due to lack of measurements. Generally, there is information provided about frequency response and output level, at one single operating point or input power level, although if you are lucky, you may get some off-axis response measurements as well. Why not display these parameters vs input level? Additionally, very few speaker manufacturers will publish distorion measurements at any input level. I understand that measuring speakers is complicated by the fact the environment they are placed in plays such a huge role in the final sound, but these test can still be done to give objective people some basis of comparison.

    Also, why does the audio industry and consumers (myself included) go nuts over wanting the electrical components of their systems to have amazing distortion characteristics. From what I understand, a good loudspeaker/headphone would be hard-pressed to obtain under 1% THD, and that is for under 100dB SPL, and good luck for SPLs above that. This is orders of magnitude greater than what a competently built amplifier can achieve, so why worry about the electrical components when speakers appear to be the clear bottleneck?

    I think I may have an idea why. I read the article about transparency, and it claims less than 0.05% distortion is transparent based on listening tests, and above this level it is audible....through speakers. Does this measurement actually mean "at a background level of 1% THD (or whatever the speaker happens to be distorting at), 0.05% THD is audible"? This would actually infer that humans can probably discern levels of distortion lower than 0.05%. Additionally, it likely means if you are listening through speakers with different distortion characteristics than those used to determine the 0.05% threshold, it would change.

    Anyways, just some thoughts I had.

    ReplyDelete
    Replies
    1. To address the distortion question first, there are some good reasons why you still want low distortion electronics. A lot of this blog deals with headphones, and there are headphones that have measured distortion well below that of many headphone amplifiers (and below 0.1%). And some high quality speakers can also have distortion similar to, or below that, of many amplifiers at modest listening levels.

      Distortion is also additive and comes in many flavors. Second order distortion, for example, is relatively benign. So a speaker could have a lot of second order distortion and yet much smaller amounts of odd order or dissonant distortion from the electronics may still be audible and unpleasant.

      Testing has been done using real humans and typical high quality speakers and headphones. In these tests distortion in amplifiers has been audible down to 0.05% under some conditions. Several argue, for some good reasons, for a threshold of 0.01% (-80 dB) and, a few, even make a case for 0.001% (-100 dB) as it's below the noise floor of a CD recording.

      Finally, in my What We Hear article there's a link to an Audio Myths video and some notes in the Tech Section on how to download recordings with various amounts of added distortion and listen for yourself. Using his method the "synthetic" distortion is still audible, with good ears and high quality playback hardware, at -70 dB which is 0.03%.

      As someone who has done commercial speaker design I can tell you most of what you ask isn't done for "dirty laundry" reasons. Unless everyone does it, those products that publish high SPL data will look bad to many. All speakers have dynamic compression that gets worse at higher peak levels. And typically distortion gets dramatically and progressively worse above a certain peak level as well.

      The other reason is the measurement method, equipment, and facility used, can make a big difference in some of those measurements. It would be great to require every speaker sold be measured by the same lab on the same equipment under carefully controlled conditions--sort of like how the government measures fuel economy for cars--but that's just a fantasy.

      What you would get instead would be questionable measurements with little accountability. If Stereophile obtained significantly different acoustic measurements than say Polk published, Polk could justifiably claim it was because they used a different room, different microphone, different speaker/mic position, etc. In that situation the manufacture's measurements have a lot less value as they can get "creative" with what they publish and it's difficult to hold them accountable.

      Delete
    2. Thanks for your reply, it does clear up some things for me, and gives me some more understanding on the issue of transparency and distortion. Oddly, I looked up one of my headphones, HD600s, and they are listed at 0.1% THD (lower than the 1% I remembered, which sparked my initial post), which I will assume is at their rated output of 97dB at 1mW and constant at all frequencies (not sure about IMD). I can very much see the need for 0.05% distortion (total sum) electronics in this case, as above that would create a greater than 1dB increase in distortion over the background speaker level (I think I added those distortions correctly...), and I suppose a well trained ear might be able to pick that up. I noticed HD800s are spec'ed at <0.02% at 102dB/1khz, so I would think the case for 0.01% or even less would make sense. I did not realize headphones could achieve these types of distortion numbers. And I did vaguely remember reading about 2nd order harmonics being not as intrusive as other forms, although I never do remember solid numbers being attached (which may be impossible) such as people can tolerate ~6dB more 2nd order than 3rd order etc, and speakers/electronics producing different levels of each, thus complicating it more.

      Ya, that is quite a fantasy world that I described in terms of speaker measurements. There are more parameters to be specified for a given loudspeaker test to make it reproducable which makes it easier for companies to be very handy-wavy when their product produces bad results, and then there is the question of what room/mic/position would be comparable to someone sitting at home. I still think some attempts are better than none. And yes, I knew the answer already, the numbers do get ugly, and whoever was the first to publish them may not be selling very many speakers after they do so. But just like for electronics, we should push manufactures to do so it becomes an accepted norm and there are at least some basis of comparison. Maybe while I am at it, I can convince the record industry to stop releasing CDs with horribly compressed masters.

      I guess then I am left with the question how do I objectively select a speaker/headphone? I would think first thing is frequency response, but really, any small deviations could just be EQed. I could consider sensitivity/impedance, but that really just dictates the amp I need/have. Maybe in the case of headphones, some distortion data would be present, but other than that, without any other measurements to go on, I am stuck. I could listen to the speakers and choose with my ears, but that is shown to be a very bad idea if you care about objective performance (I would assume this is what most manufacturers do in their design process). I could blind test various speakers (somehow I dont think many stores would want me doing this), but that will only tell me if I have a preference between the two I am testing, not which one is necessarily objectively better (although, I would think this is where a trained ear would come in handy to try and steer towards objectiveness). And this is where it seems to come down to subjectivity in the end, somewhat making all the objectivity up to here pointless. Maybe that is the point though, make all the electronics transparent, and then the loudspeaker is chosen based on your personal preferences, such that, its the only unknown you are working with as it is the most difficult to fully objectively measure. It would be easy to just change 1 component to get the sound you like, as oppose to trying to match non-transparent electronics to speakers.

      Anyways, I much appreciate allowing me to pick your brain. I think what you have demonstrated and accomplished with the O2, ODAC, and (hopefully soon) ODA is an amazing thing for the audio loving world.

      Delete
  89. You're welcome. The nice thing about measuring (or blind testing) electronics is it's possible to do it in very reproducible and verifiable ways. So it's relatively easy to keep manufacturers honest. Sadly, that's not the case with headphones and even less true with speakers.

    My best suggestion for choosing headphones/speakers objectively is to find independent tests, like those at InnerFidelity, and compare models that were tested in (hopefully) nearly identical ways. The obvious downside is that limits you to a much smaller pool of choices but at least many of the most popular have been tested.

    There's really not much you can determine from manufacturers specs except perhaps when there is a large enough difference it's not likely related to the test methods. For example if one speaker is 82 dB/watt and another is 92 dB/watt it's a safe bet the second one will need less power. But, if it's 82 dB from a Polk and 85 dB from a Paradigm, you really don't which is more efficient as testing differences could create 3 dB of variation.

    If I ever get my Timex vs Rolex article published it goes into more detail about what matters, how to choose products, where to find reviews, etc.

    ReplyDelete
  90. Thanks again. I had known about headphone.com, and actually selected my headphones as they seemed to have the lowest measured distortion in the price range on that site, but InnerFidelity seems to have more comprehensive testing, which is great.

    That does make sense about the specs, and their variations between different manufacturers. I suppose its just like treadwear ratings on tires. When comparing within the same manufacturer they have some value, although not so much when comparing multiple brands. And since it is difficult to measure these parameters, manufacturers have less reason to be honest.

    When/if that new article gets published, I will be sure to have a read through it.

    ReplyDelete
  91. Thanks for the article: I must checkout the McGurk effect.
    As an electronic engineer, I find your comments very easy to accept; they accord with my experience.
    You may be interested in my purchase of some headphones a while back. I found a nice little hifi shop that stocked items with various levels of quality, and aimed to spend about £50 on headphones (15 years ago!). I went armed with a test CD with pure tones at various frequencies and an extra track with some white noise, and asked the sales assistant/manager if he would let me try several headphones. At first, he looked pretty skeptical about my approach, but it was gratifying to see him appreciate that I did seem to know what I was doing. Anyway, I particularly wanted a better bass response than my existing headphones, so I put the tones through the headphones to see what I felt was best. Interestingly, I found the noise track to be more useful, because I had the impression that I could get a quick idea of the bass response all in one hearing. I was relieved to find that my price range included headphones with acceptable bass response. Now of course I wanted to check the rest of the headphones' response, and make sure I didn't loose any treble. The Senheiser headphones I tried had a good treble response, but I was very surprised that I didn't like that much crispness. Most unexpected, but I shrugged my shoulders, accepted my preference, and got on with the job. The model I settled on was not in stock, so the manager offered to let me have the slightly more expensive model at the cheaper price so that I could avoid any delay, but I couldn't bring myself to have the alternative because I preferred "my" model's response when listening to noise. I said "That's the wrong answer, isn't it?!". So he ordered "my" model (deliberately emotive term!), and I have been delighted with it ever since.
    I look back on this process quizzically, having otherwise relied on sig gens, spec ans, net ans, component specs, calculations etc. Your comments about people's heads and preferences being different make good sense of course. So, how would I avoid the McGurk effect in such a situation?! ;-)

    Ian

    PS "Subjectivism": Doctrine that knowledge is merely subjective and that there is no external or objective test of truth (COD 1976). So my question is: how can they justify persuading anyone about anything, be it hifi quality or subjectivism itself?

    ReplyDelete

PLEASE NOTE! If you see "LOAD MORE..." OR "LOADING..." just above this paragraph, you need to click that link to load the newest comments. It may take several seconds. When all the comments are displayed the Load More/Loading link disappears.

COMMENT APPROVAL: Comments may take anywhere from a few minutes to a few days to be approved. You might want to save a copy (highlight Ctrl-C) of your comments before submitting in case something goes wrong. The approval process is necessary to prevent obscenity, etc. Please keep comments on topic. Comments promoting commercial interests will not be approved and may be reported as SPAM.