Pages

February 2, 2011

RightMark Audio Analyzer

ROSE COLORED GLASSES (updated 3/3/11): RightMark Audio Analyzer ( aka RMAA from rightmark.org ), in some ways, is impressive software for the price (free!). A lot of people, websites, and organizations, are using RMAA to publish audio measurements. It's a useful tool when used properly and the user is aware of its many limitations. Considering it’s available for free the developer should be congratulated. But RMAA has many limitations and potential problems. And a lot of people use it incorrectly and, often unknowingly, publish misleading RMAA results.

THE SHORT VERSION FOR NON-RMAA USERS:  You can rarely compare one person's RMAA results to another person's RMAA results. This is especially true for evaluating anything other than a PC audio interface (aka “soundcard”). It's very easy to have a better piece of gear actually show worse results on RMAA because of different test conditions, settings, etc. So don't put too much trust in any RMAA results unless you made them yourself under carefully controlled conditions. Unless you know the absolute levels used, load applied, PC sound hardware used, and settings of the PC and device being tested, you simply can’t trust the results. Changing any of these things can cause a greater change in the results than testing a different audio device.

SETUP IS EVERYTHING: With RMAA there are many things to get wrong. Examples include hidden mixer controls, sound processing settings in the PC sound driver, sample rate conversion by the operating system or sound hardware, level settings—both at the PC and for the device being tested, loading, ground loops, cabling, etc. And if you’re doing loopback testing, you have no way of knowing what performance is related to the A/D, the D/A, or both. Even having a cell phone nearby can corrupt RMAA measurements with no error indication at all—just erroneous measurements. RMAA testing has lots of “blind” aspects that users are often unaware of. It has very poor internal consistency and error checking. Instead of giving a warning something might be wrong, it more often just produces false results.

15 IMPORTANT THINGS RMAA DOES NOT MEASURE (revised April 2011): There are many things that can make an audible difference in sound quality RMAA does not test for. Many are difficult or impossible to measurel with a soundcard-based test setup even using other software, specialized hardware, etc. They include:

  • MAXIMUM CLEAN OUTPUT LEVEL - RMAA has no concept of absolute levels. It can't measure voltages, power outputs, etc. So you have no way of knowing, for example, how loud that portable player or headphone amp can play without obvious distortion. You also have no way of knowing what level you’re testing it at. You could be measuring a device right on the edge of clipping. Or the level might be so low you’re measuring more noise than distortion. RMAA can’t tell you if you’re testing at 2 volts of audio signal or 0.02 volts. 
  • OUTPUT IMPEDANCE - Anything designed for driving headphones including PC's, portable players, headphone amps, USB headphone DACs, audio interfaces, pro gear, etc. has an output impedance. It varies widely and can have a huge impact on the sound quality--especially with certain types of headphones (see Headphones Amp Impedance). RMAA does not measure output impedance at all. This is a huge limitation as it’s one of the biggest factors in the sound quality of headphone sources.
  • DAC LINEARITY – Some DACs exhibit considerable non-linearity at low levels. RMAA doesn’t measure low level linearity, and because it can’t measure absolute levels, everything in RMAA is only relative to an unknown reference. 
  • SQUARE WAVE PERFORMANCE – A 1 Khz square wave reveals a lot of information about analog and digital audio components such as stability, bandwidth, rise time, compensation, and for digital devices, the type of digital filtering used. One or more of these can have an audible impact on device performance. But RMAA doesn't have a square wave test. And even if you use oscilloscope software for a soundcard, it will only measure up to the cut off frequency of the PC’s sound interface. The bandwidth of PC sound hardware is usually too low to see ultrasonic/RF, ringing, oscillations, switching power supply noise, Class-D amp noise, or instabilities. The very limited bandwidth is like looking through rose colored classes. The result can easily appear much better than the real performance.
  • JITTER - Jitter has been proven to be audible in some circumstances. RMAA has no jitter test.
  • VARIOUS TWIN TONE TESTS – RMAA has a single IMD test that’s similar to the SMPTE test but it’s not clear if it follows the SMPTE standard. And there’s no capability for other important twin tone tests such as the popular CCIF 19 Khz/20 Khz which is very revealing of high frequency performance. From what I’ve seen, RMAA doesn’t even calculate SMPTE distortion properly. The results are often artificially optimistic.
  • THD20 – It’s very useful to measure THD at 20 Khz for a variety of reasons. It’s a good indicator of stability, feedback loop design, and more. To perform this measurement properly, you should be able to measure the first 3 harmonics at 40 Khz, 60 Khz and 80 Khz. Even if you get RMAA to work with at a 192 Khz sampling rate, it still can’t perform a proper THD20 measurement. And it has no option to adjust the measurement bandwidth of its THD sweeps.
  • LOW FREQUENCY DISTORTION – It’s also useful to measure THD+N at very low frequencies such as 5 to 10 hz to expose thermal modulation effects, power supply issues, and more. RMAA doesn’t allow arbitrary low frequency testing.
  • POWER vs THD: RMAA cannot perform the classic measurement of output power versus THD. This is a standard benchmark amplifier test that’s widely performed as it’s very revealing of the amplifier’s behavior at different levels. RMAA has no way to perform this critical measurement nor am I aware of any other software that offers this sort of measurement with a sound card.
  • SELECTIVE SPECTRUM TESTS: It’s very useful to perform different spectrum testing with various input signals. RMAA doesn’t give you any control over its signal generator or its analysis. You cannot control the FFT points, averaging, weighting, filtering, etc. This is very limiting for audio testing. You’re stuck with the ‘canned’ results RMAA provides and many of those don’t follow industry standards (or it’s not clear how they’re calculated).
  • RESIDUAL ANALYSIS – Analyzing the residual distortion products can be very revealing—for example it can determine if an amplifier suffers excessive crossover distortion. RMAA has no way to do this.
  • REAL TIME RESULTS - Many problems may only show up briefly or intermittently. And many adjustments (like finding the clipping point) are best performed in real time. For example, maximum output is commonly defined as 1% THD. So with a real time audio analyzer you just raise the level until it reads, in real time, 1% distortion. With RMAA it takes a long time to run a test sequence which makes if very difficult to see what affect adjustments have on the results. It might take 10 or more passes to set the level correctly for even close to 1% THD. And if you have a bad cable, or some other intermittent problem such as outside noise or interference during the test, you will likely never know because you just get a dump of results at the end. I’ve had a cell phone randomly mess up RMAA measurements in ways that just made the product being tested look bad with no other indication the results were invalid.
  • SLEW RATE – RMAA cannot measure slew rate which, especially for audio power amplifiers, can be an important measurement. PC sound hardware is bandwidth limited making these sorts of measurements difficult or impossible with any PC software.
  • DELAY & LATENCY – RMAA cannot measure delay and latency for digital hardware.  This can be critical for live audio recording, studio work, gaming, and for lip sync with video.
  • HARDWARE LIMITATIONS – Nearly all PC sound hardware has some severe limitations that restrict RMAA and any other software. For example bridged and certain other amplifiers cannot have any of their output terminals grounded or connected together. But typical PC sound interfaces have common grounded inputs. To make matters worse, these grounds often go back to the PC’s  AC power grounds which creates even more problems. Some of these issues can seriously harm the device being tested and/or your PC’s sound interface or even the PC itself. USB and Firewire sound hardware is also grounded back to the PC. Most of these devices can only work with up to a few volts of input. Higher levels will cause overload and even damage the hardware. As mentioned earlier, the bandwidth, best case, is usually limited to about 90 Khz and more often only about 20 Khz which precludes many measurements and often hides other problems. There are also inherent issues with bit resolution, gain matching, channel balance matching, absolute levels, and more.

 

ALL THE NASTY DETAILS (non-geeks probably want to skip this section):

If you're not familiar with RMAA, here’s a typical screenshot. Note the 3rd column in the results labeled "RMAA Problems" and the highly questionable numbers. More on that later (click for larger):
 

 
PC HARDWARE ISSUES: RMAA results are only as good as your soundcard and how RMAA is used. Lots of people are running RMAA on whatever sound hardware their computer came with. And that's often a serious limitation. Built-in sound hardware is prone to all sorts of problems--especially the A/D section which is often only used for digitizing voice these days so manufactures don't put much effort into making the built-in A/D signal path perform well.

Nearly all built-in sound hardware amounts to little more than a cheap CODEC chip stuffed onto a crowded and electrically very noisy motherboard. Modern CPUs create spikes of current that can exceed 100 amps. That generates a lot of electrical noise and it's hard to keep it all out of the nearby audio circuitry where it can confuse RMAA measurements.

Even with external sound hardware, other applications or your operating system can play random sounds while you're running the test (like when you get new email). Even CPU loading and hard drive activity can affect the noise floor of built-in audio hardware--especially the line inputs and A/D. And while RMAA should log an error if extra sounds are played during the analysis, it just corrupts the results in a non-obvious way (more on that later).

Some PC's (especially laptops) don't even have a line level input. They only have a microphone input. Trying to use a mic input for line level audio is a disaster and will yield very misleading results. Yet some still do it and publish the results with no indication of how they obtained the results and hence no clue the results might be invalid.

Lots of PC's and sound cards perform various audio processing in the digital domain. It's sometimes impossible to turn all this processing off hence many PC sound devices are not "bit accurate". Some perform internal sample rate conversion, for example, regardless of how they're configured (Creative's Sound Blaster and X-Fi devices, for example, infamously perform conversions). Others attempt to optimize or enhance the audio in various ways. This internal processing affects the accuracy of RMAA in unpredictable ways.

Finally, even lots of outboard PC sound devices have problems--especially with their A/D hardwaer. Some have microphone preamps always in the circuit which seriously compromises their performance for line levels. Others have very inexpensive A/D chips and circuits in them. Some are not bit accurate, have noisy USB-derived power supplies, and/or problematic level controls, etc.

GAIN MATCHING: With many PC sound interfaces, the mixer settings operate in the digital domain. And when you lower the level settings you may lose digital bit resolution. The effective number of bits might be 16 bits with the controls maxed, but if you have to reduce the gain, you may get less than 16 bit resolution. At some point this can become the limiting factor in the test setup (insufficient ENOB). This is especially critical for measuring very low levels like noise, distortion, or crosstalk (channel separation). 

And changing levels is made more difficult because PC sound hardware levels controls are rarely calibrated in known increments (like dB). And even the few I have seen that are calibrated in dB (in software) don’t “track” well with the actual hardware settings. For example, some steps are close to 1 dB but near the end of the range can be as high as 3 dB. The manufacture never intended for their hardware to be used as a lab instrument so they probably didn’t much care about such accuracy as it’s not a big deal for 99% of applications.

So, ultimately, you have to verify levels with an external meter and not just any meter (more on that later). Any changes in the controls or settings usually requires establishing the levels with the external meter all over again.  This can get very tedious when dealing with audio signals that often vary over a wide range during testing. And if you don’t re-adjust the level controls to operate the A/D near full scale, you can lose lots of resolution and dynamic range making the device being tested seem to perform much worse than it really does.

Level setting also depends on the topology of your PC sound hardware. Some of the external USB or Firewire devices have their own physical pots for levels controls. I’ve never seen these calibrated in any meaningful way. They typically cover an extremely wide range of gain (i.e. 65 dB for the E-mu products). These controls, because of their wide range, are extremely “touchy” and it’s just about impossible to make precise small adjustments or get the two channels precisely balanced. And the settings can even change on their own which can really throw off RMAA if you don’t realize it happened during a test.

Better devices use programmable gain amplifiers (PGAs) ahead of the A/D converters and control these PGAs from software. This can be a better solution, but only if you’re given calibrated control in sufficiently fine steps, and the PGAs themselves are accurate. I’ve yet to find any interfaces that meet these criteria. Many of the less expensive ones are not terribly uniform in their steps over their full range. PC sound hardware is not designed with absolute precise values in mind. But that’s exactly what’s needed for a good measurement set up.

Some PC sound hardware uses a fixed gain amplifier and some sort of “digital pot”. These are often less accurate than the PGAs above. And often have fewer, and more coarse, non-linear steps. And the high fixed gain amps can have more noise issues. Digital pots often use a linear rather than logarithmic scale. This can make the step size rather random.

Some PC sound hardware lacks any sort of attenuator or PGA on the inputs. They simply design the circuit so the maximum expected input corresponds to roughly 0 dBFS in the A/D and then do everything in software from there. While this is fine for a person’s voice for Skype, it’s far from ideal for measurement purposes. As the signal level is reduced, the effective resolution of the A/D goes down with it. If your source happens to have a maximum level close to the maximum of the sound hardware input you’re in fairly good shape (say 2 Vrms the Redbook standard for digital audio). But if you’re trying to measure a portable player with a maximum output of 0.4 volts you just lost 14 dB of dynamic range from your measurements.

The gain increments are often too large, variable and/or unknown. For example, if you want to remove a 0.7 dB channel balance error, but only have 2 dB steps to work with, you’re stuck. And the steps often don’t correspond to any convenient number of dB because they’re derived from whatever the hardware supports. Most electronic volume control chips, for example, only have 64, 128 or 256 steps. So what you get is the full range divided by 64 uneven increments—whatever that happens to work out to.

Proper audio analyzers have calibrated analog stepped attenuation built into the input and output circuitry so regardless of the signal level they maintain the resolution of the D/A and/or A/D—to always be within a few dB of full scale. You can also set levels in their software to a resolution of at least 0.01 dB. I’m not aware of any PC audio devices that can do anything even close.

NO ABSOLUTE LEVELS: Even if you run RMAA on a decent PC audio interface with good specs, you still have no idea of the absolute levels. Are you testing too low? Too high? Near the device’s clipping point? This also prevents knowing the true output capability of the device being tested (i.e. dBu, dBv, dBm, volts, watts, milliwatts, etc.). It also means the channels may not be properly balanced which affects RMAA's measurements—the RMAA calibration routine compensates for channel imbalance in your device when you really want to be measuring that imbalance. The actual levels can make a BIG difference in the RMAA numbers you get. Testing a device at 1 volt versus 0.1 volts will yield a 20 dB difference in noise and distortion levels. That’s huge.

AN EXAMPLE: Joe posts his RMAA results for Player A and tested at a relatively low level. This will yield noise numbers significantly worse than the device is capable of. Bill posts his results for Player B but, unknown to Bill, the level was set too high and it was on the edge of clipping. So Bill’s noise numbers look really good but his distortion numbers look bad because the player was on the verge of overload. Hopefully you can see how it’s impossible to draw many valid conclusions from the published results because the tests used very different levels. But that doesn’t show up anywhere in the RMAA report or results. Player B might really be the better player, but Bill had no idea it was starting to clip so the high distortion results make it look like the inferior product.

REFERENCES ARE CRITICAL: The proper way to do audio measurements is to use standard reference values. For example  0 dBu is 0.775 volts RMS and a standard for consumer audio. Without knowing the actual levels of the signals being measured, you have no way to use proper references or even know what the levels are so you can test the next device, or even retest the current device, at the same level. And even if you use an external meter (most of which are designed for 60hz AC power and not even close to accurate across the audio band) you're still likely stuck with the un-calibrated mixer/level controls of your PC audio device. So the instant you change any of the controls, your meter measurements are rendered useless. Without absolute levels, and a known reference, it’s usually impossible to duplicate the results.

NO STANDARDS: RMAA isn’t very clear about what standards it conforms to, if any. Did the guy who wrote the software build in A Weighting to the noise measurement? If you use a higher sampling rate, are the THD values only calculated over the audio spectrum or do they also include ultrasonic noise and distortion? How is the IMD value calculated? There are lots of unanswered questions about how RMAA works internally and it makes comparing RMAA data to real measurements that conform to accepted standards difficult or impossible.

TEST LOADS ARE CRITICAL: Loading is often overlooked. For example, portable MP3 players often measure far better unloaded driving just the line input of your PC audio device than when properly loaded with real headphones or a suitable test load. Nearly all the RMAA measurements I see published never mention anything about what the device was loaded with so I'm guessing it's usually nothing. Unloaded tests are often meaningless. Some measurements, like distortion and crosstalk, are hugely affected by the load. And if you're trying to measure a line level output (rather than a headphone output) most PC sound devices don't have a well defined input impedance. Here again, this makes repeatability, comparing, and verifying results difficult—especially at higher levels of performance.

USING HEADPHONES AS THE LOAD: This can be good and bad. The good news is it’s a realistic non-linear load--especially if they're the headphones the user intends to pair with the device. The bad news is someone else needs the exact same headphones (which they likely don't have) to conduct a fair test that's valid for comparison. And for many devices the headphones will alter the measured frequency response in ways more related to the headphones than the device (see the example below). And, unknown to many, headphones also act as microphones. So, for example, they pick up background noise in your room and this can raise the noise floor measured by RMAA making for misleading noise and distortion measurements. A proper resistive test load is better if you want reproducible results that can be more fairly compared with other results. It also avoids the "microphone effect".

Here are the RMAA results using a 15 ohm resistive load, and 2 different kinds of headphones--the UE SuperFi 5 Pro's have a really wide impedance swing from about 10 ohms to 85 ohms and a pair of Sony MDR-EX76's that only vary by a few ohms at higher frequencies. The levels were within 2dB of clipping at the worst case frequency using a Benchmark ADC1:


The best numbers are with the 15 ohm resistive load. When using the Sony's the THD more than triples and the noise floor rises up a few dB. And what's with those SuperFi 5 numbers? Does the otherwise low distortion source suddenly produce 4% THD? No, but RMAA makes you believe it does! What likely happened is the signal was clipped because of the frequency response swings. But you have no way of directly knowing that, and from the average person’s point of view, the levels were properly calibrated using RMAA’s calibration routine. Here's a "zoomed in" graph showing what effect the Sony headphones have on the frequency response:


So if Joe tested with a resistive 15 ohm load you'd see the white graph in his results. If Bill used his Sony headphones you'd see the green graph published and probably think less of the product being tested. And if Bill used his SuperFi 's here's what you'd see (in green)—a whopping 15 dB of response deviation:


Here's a Sansa e260 MP3 player with no load and a 15 ohm load:


Notice it has more than 8 times as much THD, more than 4 times as much IMD, the crosstalk is far worse, and the frequency response much worse when loaded. Here's the frequency response difference:


There's likely a coupling capacitor in the output amplifier of the Sansa. And, unloaded it has little effect, but with a typical load it's down -3dB at 40 hz. There's also a weird rise at high frequencies likely caused by marginal feedback design in the amplifier. Here's the swept IMD vs frequency:


Unloaded the distortion is down below -72dB worst case, but loaded it's more like -57dB which is likely audible. With no load, as many use RMAA, the Sansa rivals some of the better players out there. Loaded, however, it's closer to the bottom of the pack. It makes a huge difference! These are typical examples. It makes a very big real world difference.

GROUNDING: Even an external PC audio interface is grounded to your electrically noisy PC via the USB/Firewire connection. And the inputs and outputs share a common ground. These both can create potential ground loops, noise sources, or worse, with whatever device you're trying to test. If the device has bridged outputs it may harm the device to ground one side of the output. And many even conventional (non bridged) amplifiers can oscillate or even self destruct if you connect their input and output grounds together externally. You can use transformers to isolate the device, but they introduce lots non-linear distortion on their own so you end up measuring mostly the transformer. Proper audio analyzers have differential and/or floating inputs and outputs that are not only isolated from ground but from each other.

HIDDEN GROUND PROBLEMS: Ground loops, especially where PCs are involved, can create noise in non-obvious ways. It’s very likely a lot of RMAA results are badly corrupted by ground issues but those making the measurements are likely totally unaware of the problem. They just see results that are much worse than what the device is capable of when it’s used normally. You have to remember RMAA was mainly designed to test sound cards in loop back where grounding typically isn’t an issue.

OVERLOAD: If you're trying to measure gear intended to drive speakers, it's likely to overload and possibly damage your PC hardware if you directly connect the output to the line input. Such inputs are usually only designed to handle a few volts maximum and some even less.  But speakers usually require 10+ volts. So you have to make an external divider network which is another potential source of problems and inaccuracy. It also makes it more difficult to compare RMAA measurements made by others unless they’re using the same divider network.  And there’s still the grounding problem mentioned above which can literally damage the gear you're trying to test as well as your PC hardware.

SAMPLING RATE: RMAA’s measurement range is limited by the sampling rate of your audio hardware and what the drivers support. Often it doesn't work (or work right) above a 44 or 48 Khz sampling rate. This restricts the bandwidth to around 20 Khz. Many devices may have instabilities that show up at ultrasonic or even RF frequencies. RMAA is usually completely “blind” to these problems. See Testing Methods for an interesting real world example. And Class-D amplifiers and/or switching power supplies are increasingly being used used in everything from MP3 players to A/V receivers. These switch at high frequencies and often create large amounts of noise that’s entirely missed by typical PC audio inputs. Even if you don't think it's audible, the amount of out-of-band noise can still be a useful indication of the quality of the device being tested. And stability issues often are audible.

BIT DEPTH AND NOISE FLOOR: RMAA is typically limited to 16 bits resolution by either the PC audio hardware and/or driver limitations. But you often can't use the full 16 bit range because of gain limitations (see GAIN MATCHING above). So, in reality, you might end up with less (or similar) useable dynamic range than the device you’re trying to test.  Ideally the test set up should have far better dynamic range than whatever you’re testing. Even if you're lucky enough to get RMAA to work correctly with 24 bit drivers, you won't get anywhere near 24 bit performance from typical 24/96 or 24/192 PC sound hardware--you're lucky to get 17 or 18 bits of effective resolution (ENOB) due to the noise floor, power supply noise, grounding issues, typical A/D and D/A converters, etc.  Often this level of resolution is inferior to what you’re trying to test. ideally your measurement hardware should be at least 10 db better than the gear you’re trying to measure. That often requires expensive hardware which often doesn’t work with RMAA correctly anyway.

FALSE SECURITY: People often do a "loop back" test to first test their PC sound hardware. And, more often than not, RMAA reports fairly impressive results. So they think they're good to go. But there are often problems being masked, or caused, by one or more of the above issues. For example, level, loading, and grounding issues won't show up in a loop back test. And don’t forget the previous list of 15+ things RMAA doesn't test for at all.

RMAA DEFECTS: On top of all the above issues, the RMAA software itself is buggy and prone to problems. The last release was several years ago and development has apparently been abandoned. Some of the FFT/math used to calculate the results is apparently wrong and/or has serious limitations that are neither properly documented nor obvious. In my experience the results often disagree with a professional audi analyzer. Plus there are lots of quirks and some obvious bugs. For example, try right clicking on any of the icons for the combined results graphs--instant total crash and you lose all your results.

Perhaps most important, RMAA often just outputs bad data instead of indicating an error condition with the measurement. To use an extreme example, you can stop playback of the test file half way through the tests, and instead of issuing an error, RMAA goes right on calculating and randomly spews out half truths and half garbage. It's obvious the developer didn't bother to put in many checks for validity of the input and output data. This can take much more subtle forms such as bad cabling, random noise (RFI from a cell phone, sound events from your PC, etc.),  not being detected during the test.

Here's an example of RMAA inventing impossible results out of nothing. It was run with no test file at all and, instead of timing out, it ran without a single error or warning and here's the result:


Wow, 245% IMD, that's pretty bad! I didn't know you could have more than 100%. But the THD is “only” 1.7%. The frequency response doesn't look so good though. I joke, but you get the idea. This doesn't inspire confidence in the software. And then there are messages like this one:


So clipping occurred but 0.000% of the samples were actually clipped? Hmmm. The funny thing is, in this case, there was zero clipping by the ADC (my Benchmark ADC1 has clip lights that stay on if it even clips a single sample until you reset the LEDs). And what's even more interesting is lowering the level by several dB and running the test again (as the error suggests doing) yields the exact same error! It appears to just be a case of RMAA getting confused trying to test this particular device (a device that tests just fine on my Prism dScope). Again, this does nothing to boost confidence in RMAA results.

RMAA DESIGNED BACKWARDS? I’ve seen enough weird results from RMAA I have to wonder if the developer designed it backwards to match the specs of a reference soundcard? Whatever test signals and analysis being used might have been “tweaked” until he got numbers that roughly matched the specs of some respected soundcard. If true, this would explain a lot. Ideally RMAA would just do all the math correctly and the results would be verified against a professional audio analyzer to make sure the math was accurate. But it often doesn’t agree and I’m at a loss to otherwise explain why unless there are either bugs in the calculations or it was designed to produce the expected numbers when testing a certain soundcard.


RMAA TIPS: So how is a person supposed to make RMAA measurements more accurately? First, use the best PC audio hardware you can. A good device would be something like the RME Fireface UC but, in my experience, proprietary low latency drivers designed for multi-track recording don’t work well (or at all) with RMAA. Example popular devices that do work are the E-Mu USB audio interfaces:

  • Creative/E-Mu 0202 - This device has decent A/D and D/A performance but it has un-calibrated level controls that cover a wide range of gain (65 dB!) and are very "touchy". This makes setting the level the same for both channels just about impossible. So you can’t know the actual channel balance of the device you’re trying to test and certain other tests can be inaccurate. The 0202 also has a mic preamp in the signal path on just one channel. Not surprisingly, that channel has higher noise than the line-in only channel. So any device you test with the 0202 is prone to rather lopsided results, weird crosstalk numbers, etc. It's also at the mercy of the sometimes noisy USB power supply. And it has serious issues on some PCs with its proprietary buggy drivers.
  • Creative/E-Mu 0404 - This device is better than the 0202 in that it has its own dedicated power, and both channels have the same signal path (unfortunately both have mic preamps). It also has slightly better metering but still suffers from the "touchy" gain control problem and the inability to exactly match the channels without some external help. See the section below on Levels. Another downside of the 0404 is it requires the same special drivers as the 0202. The drivers haven't been updated in years and especially can have issues with Windows 7. As far as I know there only buggy beta 64 bit drivers.

SETTING LEVELS: This is critical for many reasons. First, it's useful to know if the device you’re testing has a channel imbalance. But if you use the meters in RMAA to say adjust your E-Mu 0404, you will remove any channel imbalance anywhere in the signal chain. You can either feed known matched levels (i.e. verified with a meter) into both inputs and then run the RMAA calibration, or use a "Y" cable to temporarily split one channel into two outputs to set the levels. That way you know any channel imbalance that shows up during testing is the device, not your set up.

MEASURING LEVELS: As discussed earlier, it's important to know what levels you're really testing at. So it's best to use an external true-RMS meter that’s accurate across audio frequencies. Most cheap DMM's, even many that claim true-RMS, are only designed to be accurate around 50 – 400 hz. They often "roll off" dramatically or behave unpredictably at higher, or lower, audio frequencies. True RMS calculation is non-trivial and it’s expensive to have it be accurate at higher frequencies. So most reasonably priced meters don’t bother as they’re not intended for audio use. So if you're testing at say 1 Khz, they might read only half the real value. Looks at the specs for the meter, or if you already have one, test it on the output of your sound interface with test tones across the audio spectrum to see how flat it is (or isn't). It's a hassle to do the math at different frequencies to apply a correction factor but it’s cheaper than buying a more expensive meter.

CHOOSING A TEST LEVEL: If you're testing a device with a volume control, how far do you turn up the volume? I would suggest picking a value that's  well below the maximum output to help assure it won't be close to clipping. But if you test at too low of a value, the noise and distortion numbers will be much worse because noise will dominate the measurement. If you have a meter (see above) and the maximum output level is specified for the device a good starting point is about 70% of that value (roughly –3 dB). But this may also depend on the load being used. if the maximum output is not specified, see the next paragraph.

REFERENCE LEVELS: Here are some guidelines for reference levels:

  • Headphone Outputs - A good guideline for headphone outputs is at least 1mW into 32 ohms as that's used in the manufacture's specs of many devices and a value nearly any device can manage without clipping. P = (V*V)/R which means V = SquareRoot(P*R). So the square root of (.001v * 32 ohms) is 0.179 volts or about 180 mV RMS. This also happens to be about the typical power level most listen to their music at with typical low impedance headphones. So use your meter to set the player/amp to 180 mV RMS while playing a 0dBFS 1 Khz reference file. Then set your levels on your PC sound interface/mixer controls to be just under clipping (0 dB) while monitoring the 180 mV signal. But, beware, such low reference levels may challenge your PC hardware when making noise and crosstalk measurements. Many headphone outputs can deliver 300 – 500 mV into 32 ohms at low distortion so you might want to run a test at several different levels to find where the distortion starts to dramatically increase and the run the test just below that level.
  • Line Level Outputs – The Redbook standard for home digital audio gear is 2 Vrms. That’s what you’ll get from most home CD players playing a 0 dBFS signal. But portable and USB powered line outputs are often lower—typically in the range of 0.5 volts to 1.6 volts. And some gear can be above 2 V rms.
  • Speaker Outputs – As mentioned earlier, you need to use a voltage divider to avoid overloading and likely damaging the line level input of your RMAA sound hardware. This is not something I want to cover as if someone gets it wrong it could cause expensive damage. But for those in the know, the standard level for checking noise and distortion is typically 1 watt into 8 ohms which is 2.8 Vrms.


RUN A LOOPBACK TEST AT THE SAME LEVEL: Regardless of what sound hardware you're using, test it with a "loop back" test by connecting the inputs to outputs as instructed on the RMAA website to know what the "baseline" performance is at that level. Try to do this test at the same absolute level you're going to use for testing your piece of gear (i.e. the 180 mV mentioned above, etc). The signal to noise ratio, distortion, crosstalk, etc. of your PC interface will change at different test levels.

USE A PROPER LOAD: See the earlier section on loading. Ideally use a resistive test load. For devices intended to drive headphones, use resistors in the range of 15 – 600 ohms. A good single value is 32 ohms. Or test at several different impedances if you want. Even small 1/4 watt resistors are fine. They should be carbon film, and not wire wound. Use caution using headphones as a load both because they may be damaged by test tones at high levels, and they can also pick up room noise and create erroneous results due to the their impedance variations with frequency. For line level outputs the (often unknown) load of your RMAA sound hardware is probably sufficient. For speaker outputs the normal load is 8 ohms and it needs to be rated to handle whatever power level you’re going to test at.


VERIFY YOUR CABLES:  You don’t need $100 audiophile cables, but $1 ones from eBay are not a good idea either. Because of the way RMAA works, if you have a cable problem you may never know and just get marginal numbers. So test all the cables in the signal path by listening to a test tone while moving them around. If you hear static, or worse, replace the cable.  And, when testing crosstalk, cables become a huge problem if there’s any extra shared ground wiring.

RUN THE TESTS MULTIPLE TIMES: A good way to help rule out intermittent problems is to run the tests multiple times. If you get consistent results, they’re more likely to be valid. If the results vary by more than 5%, something is likely wrong.

USE A REAL OSCILLOSCOPE:  If possible, it’s worth checking the output, especially driving a real world load, with an oscilloscope that has a bandwidth out to at least 5 Mhz. This may show problems not seen in RMAA using a sound card limited to 20 Khz – 96 Khz. You can also measure the slew rate on a real scope and better evaluate square wave and impulse performance. Beware most scopes have the same grounding issues as PC sound hardware.

HOW I DO IT: For RMAA testing I use a Benchmark ADC1 for the A/D, a Benchmark DAC1 Pre for D/A, a 6 1/2 digit Agilent bench DMM that's extremely accurate and flat from 10 hz - 100 Khz for exact levels and other measurements, and one of several oscilloscopes with 60+ Mhz bandwidth. But, far more often, I use a dScope audio analyzer as it can make far more measurements, conforms to international testing standards, and produces far better and more accurate results. See my blog post Testing Methods for more information.

26 comments:

  1. Can you suggest better/equal software that is free?

    ReplyDelete
  2. To Anonymous, this is a classic case of "you get what you pay for". For the price, RMAA, is a useful tool. But, beyond the suggestions in the article regarding how to get the most out of it, I'm not sure what to suggest that doesn't involve spending some serious money.

    Unfortunately, any PC audio device--soundcard, firewire, or USB--is going to have serious limitations. To move significantly beyond those limitations involves spending lots of money (typically at least $5000).

    The audio measurement business is very low volume. So it's not likely anyone is going to invest the massive amounts of time required to develop a low cost/free solution if not many are even interested.

    Freeware is great, but it tends to work best where it has broad appeal. That increases the odds that you'll find at least a few generous soles willing to invest hundreds (or thousands) of hours of their time for free in exchange for widespread benefit and fame. Audio measurements, in contrast, are a very narrow niche. I doubt if Audio Precision and Prism combined have sold more than 9000 units ever. Contrast that to say the iPod with ~40 million units sold.

    ReplyDelete
  3. I actually loved your blog, it is very professional and I have not found any place with a professional point of view regarding sound quality.

    I have one question I'd like to make. The thing is I have an ipod nano 6th and and ipad 2, I ripped a couple of cds to apple lossless and synced them in the ipod and the ipad. To my surprise I felt the ipad had a better audio quality, the thing is I'd like to test that, but I don't know how (i'm not even an engineer to do it), so can you please give me some tips. Also I read that the last MD factory is closing and I felt very sad because I was very keen on them, and I was wondering, how were those player's audio quality compared to the actual mp3 players. I don't know, I have many questions, but all I have found are opinions and I'd like to have hard answers, so please help me!!!!

    ReplyDelete
  4. I loved your blog, very professional in a sea of opinions.

    I'd like to ask if you can give me some tips to test the audio quality of an Ipod 6th and an Ipad 2 (i have both). The thing is i tried some apple lossless and felt that the audio was better in the ipad (strange), so I'd like to test that.

    ReplyDelete
  5. Hey Edo, thanks for the praise. It is an iPad 1G or 2G? I have the 1G and plan to test it. I don't have a 6G Nano, however. Considering how tiny the Nano is, I would expect the iPad to have the edge. The Nano, like several Apple products, is all about form over function. It's more like a piece of jewelry than serious audio gear.

    If you check out the Sansa Clip+ review on this blog you'll find many tests of the iPod Touch 3G mixed into the results. The Touch 3G performed impressively well.

    I'm not sure what other tips to offer besides reading more of the articles here and perhaps seraching the web for test results on the gear you're interested in. But if you read the RMAA article, you know you can't usually trust published RMAA results.

    ReplyDelete
  6. Correct, i read your posts, very interesting indeed. In fact, i am not an apple fan, but for weird circumstances i got an ipod 6th half price (in fact the design is very comfortable) an an ipad 2 with 200 dollars in accesories for free so i thought it was good deal and got one (i'm good bargaining, that is what i do for living).

    Anyways, although the stuff is fairly good, I was disappointed with the sound quality of the iPod (I still rather prefer my old md player). Then I got the iPad, and it sounded fine, but it is pretty useless.

    As an average user, but with a scientific mind, I'd like to know if my impression is correct. Hopefully more people was interested in the facts more than marketing, people get astray very easily.

    Regarding the Sansa, it will be my next portable player, as for now (and since I paid over $100 for the iPod) I am looking for a nice and affordable stereo, I am moving to Japan and I'd like to get a good stereo for under $600 ( i'm going back to school, grad student). I've been thinking in Sony or Panasonic ( they have a very good variety over there), but I am not sure....

    ReplyDelete
  7. Most of issue do not connected with RMAA anyhow. It's more likely common problem for ANY measurement on PC without Audio Precision gears.

    "Some of the FFT/math used to calculate the results is apparently wrong" - that's completely UNTRUE. RMAA made by Ph.D and uses Intel IPP library. There are no FTT/Math errors inside Intel IPP. We can trust it entirely.

    Maxim Liadov, RMAA Project Manager.

    ReplyDelete
  8. @Maxim, please provide documentation how the math is done in RMAA? There is much that is not explained as there is very little good documentation for RMAA and how the software really works. There are many RMAA results that do not agree with professional instrumentation (even measurements within the capability of the PC sound hardware used).

    Many times, RMAA does not even agree with itself. The results from the RMAA THD sweep vs frequency are often very different from the RMAA FFT results. The same is true for frequency response.

    Just because the Intel FFT library was used doesn't mean it was used correctly. That's like saying because a program is written using the Intel C Compiler it has no bugs. The devil is in the details.

    I have shown examples of how you CANNOT "trust it entirely" and plan to show more in the future. RMAA often delivers erroneous and misleading results with no explanation why.

    ReplyDelete
  9. There may be a lot of things why measurements are wrong, even if you use Audio Precision $25k gear. So if it's FFT problem inside Audio Precision DSP of or just common problem of your hardware/software setup? We made direct comparison of results RMAA vs. APS2 with E-MU engineers. There was no significant difference. Numbers were the same, a little bit lower in loopback as it was expected. That is why they add RMAA presets to E-MU professional interfaces. They trust us enough for that.

    But that's true only if you real audio engineer with proper setup. Not just ordinary PC user, and you do not understand anything, but pressing key "Start".

    ReplyDelete
  10. I'm looking for software like Rightmark to test a digital system by playing out a reference file and looping it back to record another file then analyse it. Rightmark does this, but seems a bit flakey. I don't mind paying for it - but I have not been able to find anything suitable. Any pointers?

    ReplyDelete
  11. @Anon, sorry but I'm not aware of much besides some spectrum analyzer and acoustic software between RMAA and the professional products like Audio Precision, dScope, etc. That's especially true for non-realtime analysis as you describe (playing a file).

    The professional choices are mainly Audio Precision and the Prism Sound dScope that I use. But they start at around $5000 for the most basic versions. See my February Testing Methods article if you haven't already.

    ReplyDelete
    Replies
    1. Sorry to bump this article.
      I was reading through your Testing methods article, and was wondering. If someone is an electrical/computer engineer student (cough, me cough) and has an interest in DIY audio, for the testing equipment alone must we spend a large amount? So if we see this as a future hobby, and want to buy something as an AP ATS for 8.5k for example, would you recommend that for a DIY enthusiast. I'm still trying to figure out if this is a worthwhile hobby, seems pricey.

      Delete
    2. Bumps are welcome.... That's a tough question. If you just want to explore audio as a casual hobby, you can either stick with proven DIY designs, manufacturer reference designs, or try to get the most out of RMAA and a decent USB audio interface to measure your own designs.

      If you want to go further with audio design, especially to design something that outperforms a lot of commercial mainstream gear, it's best to use an audio analyzer. There are just too many things RMAA doesn't measure and too many limitations of RMAA compatible sound hardware.

      The AP ATS is really limited. The new APx series, or a dScope, are much better options. There really isn't anything that's much less expensive that offers better performance than RMAA and a decent USB audio interface. And a lot of the older used gear is worse.

      Delete
  12. Isn't it the case that a mobile phone can cause erroneous measurements in many pieces of audio gear regardless of what equipment is being used to measure them?

    Why bring it up in the context of RMAA then?

    You're exaggerating the case against RMAA and other soundcard-based software measuring systems to make your own equipment and techniques look better by comparison. You haven't taken the time to look at the others, such as ARTA, or to evaluate what can be achieved with them.

    You're operating on prejudice and your own self-generated mythology.

    It's nice to see an attempt at objectivism, but sadly, you've erred in the opposite direction and made targets of individuals who are as committed to objectivism as you but don't have the money to throw at problems that you evidently do.

    Close, but no cigar.

    ReplyDelete
    Replies
    1. Because RMAA doesn't show you anything (besides the calibration level) in realtime you can't see intermittent problems--like those created by a mobile phone. You just get corrupted data when RMAA finishes. With a real audio analyzer you can get real time data and intermittent problems are usually obvious as you see the readings, spectrum, etc. fluctuate in real time.

      You're entitled to your opinion but you're criticizing my objectivity while not objectively showing where I've make a mistake. There's a ton of bad RMAA data out there and the main purpose of this article was to show why everyone needs to be suspicious of published RMAA results and why it's often invalid to compare them to other RMAA results. At the end of the article I offer several tips for how to get the most out of RMAA to improve the results for those who can't afford or justify an expensive audio analyzer.

      Delete
  13. 'Because RMAA doesn't show you anything (besides the calibration level) in realtime you can't see intermittent problems--like those created by a mobile phone. You just get corrupted data when RMAA finishes. With a real audio analyzer you can get real time data and intermittent problems are usually obvious as you see the readings, spectrum, etc. fluctuate in real time.'

    Yes, but the problems arising because of radio interference only make the DUT look bad, they don't make it look good. Anyone suffering this problem will either run the test again or give up, which is a problem for them, but doesn't result in overly optimistic measurements being published.

    If results were being published saying that X piece of equipment measured better than Y well known piece of equipment in a given situation THEN there would arise the possibility that an overly optimistic impression were being created, but I have never seen results presented in this fashion.

    ReplyDelete
    Replies
    1. That's true but bad results are bad results. And I've documented several other ways in which RMAA results can be, and often are, invalid for comparison purposes. The cell phone problem is but the tip of a much bigger iceberg of problems hidden under the water.

      Delete
  14. NwAvGuy, you are being a little hard on RMAA. The issues you point out are not limited to RMAA and most will apply to any PC soundcard based system. My serious analogue testing goes back to Radford LDO & Distortion test set which dates me a bit.

    I went bush for more than a decade and was pleased to find, on emerging computer power so cheap & available, that many test methods I only dreamed about, are now easily possible. (I design speakers & microphones)

    While I agree with some of your criticisms, many are easily overcome, eg Calibrate the input & output of your soundcard.

    I think you missed a golden opportunity with Maxim to influence the progress of RMAA. eg better documentation including explanations of their noise, distortion & spectrum plots. Or even have an edited version of this thread on their website.

    IMveryHO, I'm very pleased that RMAA is available and as a beach bum, it allows me to do stuff without a dScope or Audio Precision. I only wish there was similar acoustic software. (I use Audacity to measure and my own DOS software to process acoustic stuff)

    ReplyDelete
    Replies
    1. Thanks for your comments. I stated at the beginning of the article I think RMAA can be useful and the price is right. Towards the end I talk about calibration and how to work around some of the limitations. And throughout I point out that many of the limitations are related to typical PC audio hardware and not the software.

      The main point I was trying to get across is when you see published RMAA results, more often than not, they cannot be trusted or used for comparison purposes. Few people seem to be using RMAA in ways that allow for valid comparisons of test results unless you're just testing the soundcard itself in loopback.

      Delete
  15. RMAA (at least versions 6.2.3 and 6.2.4 at 96 kHz/24 bit, I did not test 6.2.5 yet) seems to report noise and dynamic range values that are about 2.5 dB better than they really should be, and the crosstalk is also "improved" by about 6 dB. These may be random bugs, or the result of "designing backwards".

    ReplyDelete
    Replies
    1. More on RMAA (v6.2.4):
      - it is not entirely true that "there’s no capability for other important twin tone tests such as the popular CCIF 19 Khz/20 Khz", since the "Intermodulation distortion + noise (swept freqs)" test actually includes that. This test generates two tones at equal amplitude (about -3 dBFS total), with a +/- 500 Hz offset relative to the frequency being tested. Therefore, at 19.5 kHz, the CCIF IMD + noise is shown, although I did not check if it is calculated correctly
      - distortion tests in general are performed at ~-3 dBFS peak amplitude (thus, in the case of the SMPTE IMD, the 60 Hz tone is at ~-5 dBFS)
      - noise tests have a band limit of 20 kHz
      - unweighted RMS noise and dynamic range are referenced to a 0 dBFS square wave
      - A-weighted noise and dynamic range are probably meant to be referenced to a 0 dBFS sine wave, but the developers forgot to add the +2 dB offset so that the filter has 0 dB gain at 1 kHz, making the values printed overly optimistic
      - the A-weighting filter is not very accurate, and at low frequencies it does not agree between the noise and dynamic range test
      - in addition to being "off" by 6 dB, the crosstalk test averages the dB results for the left and right channels to get an overall value; if the difference is large (fortunately it is usually not), the result will be even more optimistic
      - the ability to generate and analyze WAV files is also useful for testing the program by adding known artificial quality degradation to the WAV file and then checking if the results printed by RMAA are correct
      - the loopback recording is saved to a "Receive.wav" file. This file can be checked with an audio editor for playback/recording problems, or for more detailed analysis

      Delete
    2. "and at low frequencies it does not agree between the noise and dynamic range test"

      Well, this actually turned out to be not a filter problem; the dynamic range test has a "noise floor" itself that is higher than that of the noise test, but it is not high enough to be an issue when testing real analog hardware.

      Delete
  16. One possible workaround to the grounding problems that is not mentioned, although it might not be practical for the casual RMAA user, is to build a simple differential amplifier. This can also be implemented with different gains to extend the input voltage range. Of course, the amplifier itself also introduces some degradation to the signal, but this can easily be outweighed by the advantages of eliminating/reducing ground loops. Even with an inexpensive op amp like the NE5532, if it is reasonably implemented, most of the noise and distortion is added by the sound card in a loopback test. I do not know how this method compares to a reasonably priced transformer (which obviously isolates much better), though.

    ReplyDelete
    Replies
    1. That could be a partial workaround. The trick is having enough common mode rejection in a cheap DIY diff amp as the op amp will be operating with substantial common mode signals. Any common mode garbage has to be below the noise floor of whatever you're trying to measure and that's harder to pull off than just low noise and distortion with a single-ended input op amp. Most differential op amp configurations also have around 6 dB higher noise than their single-ended counterparts.

      Delete
    2. Of course, it is not a perfect solution, and probably not many people would build and use it, but it can make a useful improvement.
      For example, I tested the noise level of a sound card, recording its output with another in the same PC (all the following dB values are A-weighted, referenced to a 2 Vrms full scale signal, and have a bandwidth of 22 kHz), and without the differential amplifier, the measured level was -91 to -92 dB, and it also depended on the system activity. That is rather poor, considering that the sound card output tested is specified to have an SNR of 116 dB, while it is 118 dB for the input (I measured ~117.2 dB under the same conditions as above, but with no input signal). Adding the differential amplifier to the loopback improved the noise to -113.25 dB, by more than 20 dB. With shorted inputs, it was -115.4 dB.
      When testing crosstalk with a low impedance load, the accuracy of the results was significantly improved as well (compared against what was measured with a DMM).

      I tried to test the common mode rejection, and (with matched resistors), it was about 60 dB for most of the audio band, but I am not sure if this is correct. In any case, 30-40 dB can already be good enough to make the difference between a useless result, and a reasonably accurate one. However, a single ended source with high output impedance can degrade the effectiveness of the circuit.

      With an input impedance of 5.4 (2 * 2.7) kiloohms, 0 dB gain, and NE5532AP op amp, the noise is about -120.3 dB in theory. That is not very good, but can easily be outweighed when a ground loop is eliminated. It is also possible to compensate the measured noise levels to some extent.

      Delete
  17. Sorry disturbing your extremely interesting discussion. But I do not even get RMAA 6.x running on none of my PCs or Noteboosks (XP/Win7) after installation. It immediately crashes due to application errors,

    Reiner

    ReplyDelete

PLEASE NOTE! If you see "LOAD MORE..." OR "LOADING..." just above this paragraph, you need to click that link to load the newest comments. It may take several seconds. When all the comments are displayed the Load More/Loading link disappears.

COMMENT APPROVAL: Comments may take anywhere from a few minutes to a few days to be approved. You might want to save a copy (highlight Ctrl-C) of your comments before submitting in case something goes wrong. The approval process is necessary to prevent obscenity, etc. Please keep comments on topic. Comments promoting commercial interests will not be approved and may be reported as SPAM.