RightMark Audio Analyzer

ROSE COLORED GLASSES (updated 3/3/11): RightMark Audio Analyzer ( aka RMAA from rightmark.org ), in some ways, is impressive software for the price (free!). A lot of people, websites, and organizations, are using RMAA to publish audio measurements. It's a useful tool when used properly and the user is aware of its many limitations. Considering it’s available for free the developer should be congratulated. But RMAA has many limitations and potential problems. And a lot of people use it incorrectly and, often unknowingly, publish misleading RMAA results.

THE SHORT VERSION FOR NON-RMAA USERS:  You can rarely compare one person's RMAA results to another person's RMAA results. This is especially true for evaluating anything other than a PC audio interface (aka “soundcard”). It's very easy to have a better piece of gear actually show worse results on RMAA because of different test conditions, settings, etc. So don't put too much trust in any RMAA results unless you made them yourself under carefully controlled conditions. Unless you know the absolute levels used, load applied, PC sound hardware used, and settings of the PC and device being tested, you simply can’t trust the results. Changing any of these things can cause a greater change in the results than testing a different audio device.

SETUP IS EVERYTHING: With RMAA there are many things to get wrong. Examples include hidden mixer controls, sound processing settings in the PC sound driver, sample rate conversion by the operating system or sound hardware, level settings—both at the PC and for the device being tested, loading, ground loops, cabling, etc. And if you’re doing loopback testing, you have no way of knowing what performance is related to the A/D, the D/A, or both. Even having a cell phone nearby can corrupt RMAA measurements with no error indication at all—just erroneous measurements. RMAA testing has lots of “blind” aspects that users are often unaware of. It has very poor internal consistency and error checking. Instead of giving a warning something might be wrong, it more often just produces false results.

15 IMPORTANT THINGS RMAA DOES NOT MEASURE (revised April 2011): There are many things that can make an audible difference in sound quality RMAA does not test for. Many are difficult or impossible to measurel with a soundcard-based test setup even using other software, specialized hardware, etc. They include:

• MAXIMUM CLEAN OUTPUT LEVEL - RMAA has no concept of absolute levels. It can't measure voltages, power outputs, etc. So you have no way of knowing, for example, how loud that portable player or headphone amp can play without obvious distortion. You also have no way of knowing what level you’re testing it at. You could be measuring a device right on the edge of clipping. Or the level might be so low you’re measuring more noise than distortion. RMAA can’t tell you if you’re testing at 2 volts of audio signal or 0.02 volts. 
• OUTPUT IMPEDANCE - Anything designed for driving headphones including PC's, portable players, headphone amps, USB headphone DACs, audio interfaces, pro gear, etc. has an output impedance. It varies widely and can have a huge impact on the sound quality--especially with certain types of headphones (see Headphones Amp Impedance). RMAA does not measure output impedance at all. This is a huge limitation as it’s one of the biggest factors in the sound quality of headphone sources.
• DAC LINEARITY – Some DACs exhibit considerable non-linearity at low levels. RMAA doesn’t measure low level linearity, and because it can’t measure absolute levels, everything in RMAA is only relative to an unknown reference. 
• SQUARE WAVE PERFORMANCE – A 1 Khz square wave reveals a lot of information about analog and digital audio components such as stability, bandwidth, rise time, compensation, and for digital devices, the type of digital filtering used. One or more of these can have an audible impact on device performance. But RMAA doesn't have a square wave test. And even if you use oscilloscope software for a soundcard, it will only measure up to the cut off frequency of the PC’s sound interface. The bandwidth of PC sound hardware is usually too low to see ultrasonic/RF, ringing, oscillations, switching power supply noise, Class-D amp noise, or instabilities. The very limited bandwidth is like looking through rose colored classes. The result can easily appear much better than the real performance.
• JITTER - Jitter has been proven to be audible in some circumstances. RMAA has no jitter test.
• VARIOUS TWIN TONE TESTS – RMAA has a single IMD test that’s similar to the SMPTE test but it’s not clear if it follows the SMPTE standard. And there’s no capability for other important twin tone tests such as the popular CCIF 19 Khz/20 Khz which is very revealing of high frequency performance. From what I’ve seen, RMAA doesn’t even calculate SMPTE distortion properly. The results are often artificially optimistic.
• THD20 – It’s very useful to measure THD at 20 Khz for a variety of reasons. It’s a good indicator of stability, feedback loop design, and more. To perform this measurement properly, you should be able to measure the first 3 harmonics at 40 Khz, 60 Khz and 80 Khz. Even if you get RMAA to work with at a 192 Khz sampling rate, it still can’t perform a proper THD20 measurement. And it has no option to adjust the measurement bandwidth of its THD sweeps.
• LOW FREQUENCY DISTORTION – It’s also useful to measure THD+N at very low frequencies such as 5 to 10 hz to expose thermal modulation effects, power supply issues, and more. RMAA doesn’t allow arbitrary low frequency testing.
• POWER vs THD: RMAA cannot perform the classic measurement of output power versus THD. This is a standard benchmark amplifier test that’s widely performed as it’s very revealing of the amplifier’s behavior at different levels. RMAA has no way to perform this critical measurement nor am I aware of any other software that offers this sort of measurement with a sound card.
• SELECTIVE SPECTRUM TESTS: It’s very useful to perform different spectrum testing with various input signals. RMAA doesn’t give you any control over its signal generator or its analysis. You cannot control the FFT points, averaging, weighting, filtering, etc. This is very limiting for audio testing. You’re stuck with the ‘canned’ results RMAA provides and many of those don’t follow industry standards (or it’s not clear how they’re calculated).
• RESIDUAL ANALYSIS – Analyzing the residual distortion products can be very revealing—for example it can determine if an amplifier suffers excessive crossover distortion. RMAA has no way to do this.
• REAL TIME RESULTS - Many problems may only show up briefly or intermittently. And many adjustments (like finding the clipping point) are best performed in real time. For example, maximum output is commonly defined as 1% THD. So with a real time audio analyzer you just raise the level until it reads, in real time, 1% distortion. With RMAA it takes a long time to run a test sequence which makes if very difficult to see what affect adjustments have on the results. It might take 10 or more passes to set the level correctly for even close to 1% THD. And if you have a bad cable, or some other intermittent problem such as outside noise or interference during the test, you will likely never know because you just get a dump of results at the end. I’ve had a cell phone randomly mess up RMAA measurements in ways that just made the product being tested look bad with no other indication the results were invalid.
• SLEW RATE – RMAA cannot measure slew rate which, especially for audio power amplifiers, can be an important measurement. PC sound hardware is bandwidth limited making these sorts of measurements difficult or impossible with any PC software.
• DELAY & LATENCY – RMAA cannot measure delay and latency for digital hardware.  This can be critical for live audio recording, studio work, gaming, and for lip sync with video.
• HARDWARE LIMITATIONS – Nearly all PC sound hardware has some severe limitations that restrict RMAA and any other software. For example bridged and certain other amplifiers cannot have any of their output terminals grounded or connected together. But typical PC sound interfaces have common grounded inputs. To make matters worse, these grounds often go back to the PC’s  AC power grounds which creates even more problems. Some of these issues can seriously harm the device being tested and/or your PC’s sound interface or even the PC itself. USB and Firewire sound hardware is also grounded back to the PC. Most of these devices can only work with up to a few volts of input. Higher levels will cause overload and even damage the hardware. As mentioned earlier, the bandwidth, best case, is usually limited to about 90 Khz and more often only about 20 Khz which precludes many measurements and often hides other problems. There are also inherent issues with bit resolution, gain matching, channel balance matching, absolute levels, and more.

 

ALL THE NASTY DETAILS (non-geeks probably want to skip this section):

If you're not familiar with RMAA, here’s a typical screenshot. Note the 3rd column in the results labeled "RMAA Problems" and the highly questionable numbers. More on that later (click for larger):
 

 
PC HARDWARE ISSUES: RMAA results are only as good as your soundcard and how RMAA is used. Lots of people are running RMAA on whatever sound hardware their computer came with. And that's often a serious limitation. Built-in sound hardware is prone to all sorts of problems--especially the A/D section which is often only used for digitizing voice these days so manufactures don't put much effort into making the built-in A/D signal path perform well.

Nearly all built-in sound hardware amounts to little more than a cheap CODEC chip stuffed onto a crowded and electrically very noisy motherboard. Modern CPUs create spikes of current that can exceed 100 amps. That generates a lot of electrical noise and it's hard to keep it all out of the nearby audio circuitry where it can confuse RMAA measurements.

Even with external sound hardware, other applications or your operating system can play random sounds while you're running the test (like when you get new email). Even CPU loading and hard drive activity can affect the noise floor of built-in audio hardware--especially the line inputs and A/D. And while RMAA should log an error if extra sounds are played during the analysis, it just corrupts the results in a non-obvious way (more on that later).

Some PC's (especially laptops) don't even have a line level input. They only have a microphone input. Trying to use a mic input for line level audio is a disaster and will yield very misleading results. Yet some still do it and publish the results with no indication of how they obtained the results and hence no clue the results might be invalid.

Lots of PC's and sound cards perform various audio processing in the digital domain. It's sometimes impossible to turn all this processing off hence many PC sound devices are not "bit accurate". Some perform internal sample rate conversion, for example, regardless of how they're configured (Creative's Sound Blaster and X-Fi devices, for example, infamously perform conversions). Others attempt to optimize or enhance the audio in various ways. This internal processing affects the accuracy of RMAA in unpredictable ways.

Finally, even lots of outboard PC sound devices have problems--especially with their A/D hardwaer. Some have microphone preamps always in the circuit which seriously compromises their performance for line levels. Others have very inexpensive A/D chips and circuits in them. Some are not bit accurate, have noisy USB-derived power supplies, and/or problematic level controls, etc.

GAIN MATCHING: With many PC sound interfaces, the mixer settings operate in the digital domain. And when you lower the level settings you may lose digital bit resolution. The effective number of bits might be 16 bits with the controls maxed, but if you have to reduce the gain, you may get less than 16 bit resolution. At some point this can become the limiting factor in the test setup (insufficient ENOB). This is especially critical for measuring very low levels like noise, distortion, or crosstalk (channel separation). 

And changing levels is made more difficult because PC sound hardware levels controls are rarely calibrated in known increments (like dB). And even the few I have seen that are calibrated in dB (in software) don’t “track” well with the actual hardware settings. For example, some steps are close to 1 dB but near the end of the range can be as high as 3 dB. The manufacture never intended for their hardware to be used as a lab instrument so they probably didn’t much care about such accuracy as it’s not a big deal for 99% of applications.

So, ultimately, you have to verify levels with an external meter and not just any meter (more on that later). Any changes in the controls or settings usually requires establishing the levels with the external meter all over again.  This can get very tedious when dealing with audio signals that often vary over a wide range during testing. And if you don’t re-adjust the level controls to operate the A/D near full scale, you can lose lots of resolution and dynamic range making the device being tested seem to perform much worse than it really does.

Level setting also depends on the topology of your PC sound hardware. Some of the external USB or Firewire devices have their own physical pots for levels controls. I’ve never seen these calibrated in any meaningful way. They typically cover an extremely wide range of gain (i.e. 65 dB for the E-mu products). These controls, because of their wide range, are extremely “touchy” and it’s just about impossible to make precise small adjustments or get the two channels precisely balanced. And the settings can even change on their own which can really throw off RMAA if you don’t realize it happened during a test.

Better devices use programmable gain amplifiers (PGAs) ahead of the A/D converters and control these PGAs from software. This can be a better solution, but only if you’re given calibrated control in sufficiently fine steps, and the PGAs themselves are accurate. I’ve yet to find any interfaces that meet these criteria. Many of the less expensive ones are not terribly uniform in their steps over their full range. PC sound hardware is not designed with absolute precise values in mind. But that’s exactly what’s needed for a good measurement set up.

Some PC sound hardware uses a fixed gain amplifier and some sort of “digital pot”. These are often less accurate than the PGAs above. And often have fewer, and more coarse, non-linear steps. And the high fixed gain amps can have more noise issues. Digital pots often use a linear rather than logarithmic scale. This can make the step size rather random.

Some PC sound hardware lacks any sort of attenuator or PGA on the inputs. They simply design the circuit so the maximum expected input corresponds to roughly 0 dBFS in the A/D and then do everything in software from there. While this is fine for a person’s voice for Skype, it’s far from ideal for measurement purposes. As the signal level is reduced, the effective resolution of the A/D goes down with it. If your source happens to have a maximum level close to the maximum of the sound hardware input you’re in fairly good shape (say 2 Vrms the Redbook standard for digital audio). But if you’re trying to measure a portable player with a maximum output of 0.4 volts you just lost 14 dB of dynamic range from your measurements.

The gain increments are often too large, variable and/or unknown. For example, if you want to remove a 0.7 dB channel balance error, but only have 2 dB steps to work with, you’re stuck. And the steps often don’t correspond to any convenient number of dB because they’re derived from whatever the hardware supports. Most electronic volume control chips, for example, only have 64, 128 or 256 steps. So what you get is the full range divided by 64 uneven increments—whatever that happens to work out to.

Proper audio analyzers have calibrated analog stepped attenuation built into the input and output circuitry so regardless of the signal level they maintain the resolution of the D/A and/or A/D—to always be within a few dB of full scale. You can also set levels in their software to a resolution of at least 0.01 dB. I’m not aware of any PC audio devices that can do anything even close.

NO ABSOLUTE LEVELS: Even if you run RMAA on a decent PC audio interface with good specs, you still have no idea of the absolute levels. Are you testing too low? Too high? Near the device’s clipping point? This also prevents knowing the true output capability of the device being tested (i.e. dBu, dBv, dBm, volts, watts, milliwatts, etc.). It also means the channels may not be properly balanced which affects RMAA's measurements—the RMAA calibration routine compensates for channel imbalance in your device when you really want to be measuring that imbalance. The actual levels can make a BIG difference in the RMAA numbers you get. Testing a device at 1 volt versus 0.1 volts will yield a 20 dB difference in noise and distortion levels. That’s huge.

AN EXAMPLE: Joe posts his RMAA results for Player A and tested at a relatively low level. This will yield noise numbers significantly worse than the device is capable of. Bill posts his results for Player B but, unknown to Bill, the level was set too high and it was on the edge of clipping. So Bill’s noise numbers look really good but his distortion numbers look bad because the player was on the verge of overload. Hopefully you can see how it’s impossible to draw many valid conclusions from the published results because the tests used very different levels. But that doesn’t show up anywhere in the RMAA report or results. Player B might really be the better player, but Bill had no idea it was starting to clip so the high distortion results make it look like the inferior product.

REFERENCES ARE CRITICAL: The proper way to do audio measurements is to use standard reference values. For example  0 dBu is 0.775 volts RMS and a standard for consumer audio. Without knowing the actual levels of the signals being measured, you have no way to use proper references or even know what the levels are so you can test the next device, or even retest the current device, at the same level. And even if you use an external meter (most of which are designed for 60hz AC power and not even close to accurate across the audio band) you're still likely stuck with the un-calibrated mixer/level controls of your PC audio device. So the instant you change any of the controls, your meter measurements are rendered useless. Without absolute levels, and a known reference, it’s usually impossible to duplicate the results.

NO STANDARDS: RMAA isn’t very clear about what standards it conforms to, if any. Did the guy who wrote the software build in A Weighting to the noise measurement? If you use a higher sampling rate, are the THD values only calculated over the audio spectrum or do they also include ultrasonic noise and distortion? How is the IMD value calculated? There are lots of unanswered questions about how RMAA works internally and it makes comparing RMAA data to real measurements that conform to accepted standards difficult or impossible.

TEST LOADS ARE CRITICAL: Loading is often overlooked. For example, portable MP3 players often measure far better unloaded driving just the line input of your PC audio device than when properly loaded with real headphones or a suitable test load. Nearly all the RMAA measurements I see published never mention anything about what the device was loaded with so I'm guessing it's usually nothing. Unloaded tests are often meaningless. Some measurements, like distortion and crosstalk, are hugely affected by the load. And if you're trying to measure a line level output (rather than a headphone output) most PC sound devices don't have a well defined input impedance. Here again, this makes repeatability, comparing, and verifying results difficult—especially at higher levels of performance.

USING HEADPHONES AS THE LOAD: This can be good and bad. The good news is it’s a realistic non-linear load--especially if they're the headphones the user intends to pair with the device. The bad news is someone else needs the exact same headphones (which they likely don't have) to conduct a fair test that's valid for comparison. And for many devices the headphones will alter the measured frequency response in ways more related to the headphones than the device (see the example below). And, unknown to many, headphones also act as microphones. So, for example, they pick up background noise in your room and this can raise the noise floor measured by RMAA making for misleading noise and distortion measurements. A proper resistive test load is better if you want reproducible results that can be more fairly compared with other results. It also avoids the "microphone effect".

Here are the RMAA results using a 15 ohm resistive load, and 2 different kinds of headphones--the UE SuperFi 5 Pro's have a really wide impedance swing from about 10 ohms to 85 ohms and a pair of Sony MDR-EX76's that only vary by a few ohms at higher frequencies. The levels were within 2dB of clipping at the worst case frequency using a Benchmark ADC1:

The best numbers are with the 15 ohm resistive load. When using the Sony's the THD more than triples and the noise floor rises up a few dB. And what's with those SuperFi 5 numbers? Does the otherwise low distortion source suddenly produce 4% THD? No, but RMAA makes you believe it does! What likely happened is the signal was clipped because of the frequency response swings. But you have no way of directly knowing that, and from the average person’s point of view, the levels were properly calibrated using RMAA’s calibration routine. Here's a "zoomed in" graph showing what effect the Sony headphones have on the frequency response:

So if Joe tested with a resistive 15 ohm load you'd see the white graph in his results. If Bill used his Sony headphones you'd see the green graph published and probably think less of the product being tested. And if Bill used his SuperFi 's here's what you'd see (in green)—a whopping 15 dB of response deviation:

Here's a Sansa e260 MP3 player with no load and a 15 ohm load:

Notice it has more than 8 times as much THD, more than 4 times as much IMD, the crosstalk is far worse, and the frequency response much worse when loaded. Here's the frequency response difference:

There's likely a coupling capacitor in the output amplifier of the Sansa. And, unloaded it has little effect, but with a typical load it's down -3dB at 40 hz. There's also a weird rise at high frequencies likely caused by marginal feedback design in the amplifier. Here's the swept IMD vs frequency:

Unloaded the distortion is down below -72dB worst case, but loaded it's more like -57dB which is likely audible. With no load, as many use RMAA, the Sansa rivals some of the better players out there. Loaded, however, it's closer to the bottom of the pack. It makes a huge difference! These are typical examples. It makes a very big real world difference.

GROUNDING: Even an external PC audio interface is grounded to your electrically noisy PC via the USB/Firewire connection. And the inputs and outputs share a common ground. These both can create potential ground loops, noise sources, or worse, with whatever device you're trying to test. If the device has bridged outputs it may harm the device to ground one side of the output. And many even conventional (non bridged) amplifiers can oscillate or even self destruct if you connect their input and output grounds together externally. You can use transformers to isolate the device, but they introduce lots non-linear distortion on their own so you end up measuring mostly the transformer. Proper audio analyzers have differential and/or floating inputs and outputs that are not only isolated from ground but from each other.

HIDDEN GROUND PROBLEMS: Ground loops, especially where PCs are involved, can create noise in non-obvious ways. It’s very likely a lot of RMAA results are badly corrupted by ground issues but those making the measurements are likely totally unaware of the problem. They just see results that are much worse than what the device is capable of when it’s used normally. You have to remember RMAA was mainly designed to test sound cards in loop back where grounding typically isn’t an issue.

OVERLOAD: If you're trying to measure gear intended to drive speakers, it's likely to overload and possibly damage your PC hardware if you directly connect the output to the line input. Such inputs are usually only designed to handle a few volts maximum and some even less.  But speakers usually require 10+ volts. So you have to make an external divider network which is another potential source of problems and inaccuracy. It also makes it more difficult to compare RMAA measurements made by others unless they’re using the same divider network.  And there’s still the grounding problem mentioned above which can literally damage the gear you're trying to test as well as your PC hardware.

SAMPLING RATE: RMAA’s measurement range is limited by the sampling rate of your audio hardware and what the drivers support. Often it doesn't work (or work right) above a 44 or 48 Khz sampling rate. This restricts the bandwidth to around 20 Khz. Many devices may have instabilities that show up at ultrasonic or even RF frequencies. RMAA is usually completely “blind” to these problems. See Testing Methods for an interesting real world example. And Class-D amplifiers and/or switching power supplies are increasingly being used used in everything from MP3 players to A/V receivers. These switch at high frequencies and often create large amounts of noise that’s entirely missed by typical PC audio inputs. Even if you don't think it's audible, the amount of out-of-band noise can still be a useful indication of the quality of the device being tested. And stability issues often are audible.

BIT DEPTH AND NOISE FLOOR: RMAA is typically limited to 16 bits resolution by either the PC audio hardware and/or driver limitations. But you often can't use the full 16 bit range because of gain limitations (see GAIN MATCHING above). So, in reality, you might end up with less (or similar) useable dynamic range than the device you’re trying to test.  Ideally the test set up should have far better dynamic range than whatever you’re testing. Even if you're lucky enough to get RMAA to work correctly with 24 bit drivers, you won't get anywhere near 24 bit performance from typical 24/96 or 24/192 PC sound hardware--you're lucky to get 17 or 18 bits of effective resolution (ENOB) due to the noise floor, power supply noise, grounding issues, typical A/D and D/A converters, etc.  Often this level of resolution is inferior to what you’re trying to test. ideally your measurement hardware should be at least 10 db better than the gear you’re trying to measure. That often requires expensive hardware which often doesn’t work with RMAA correctly anyway.

FALSE SECURITY: People often do a "loop back" test to first test their PC sound hardware. And, more often than not, RMAA reports fairly impressive results. So they think they're good to go. But there are often problems being masked, or caused, by one or more of the above issues. For example, level, loading, and grounding issues won't show up in a loop back test. And don’t forget the previous list of 15+ things RMAA doesn't test for at all.

RMAA DEFECTS: On top of all the above issues, the RMAA software itself is buggy and prone to problems. The last release was several years ago and development has apparently been abandoned. Some of the FFT/math used to calculate the results is apparently wrong and/or has serious limitations that are neither properly documented nor obvious. In my experience the results often disagree with a professional audi analyzer. Plus there are lots of quirks and some obvious bugs. For example, try right clicking on any of the icons for the combined results graphs--instant total crash and you lose all your results.

Perhaps most important, RMAA often just outputs bad data instead of indicating an error condition with the measurement. To use an extreme example, you can stop playback of the test file half way through the tests, and instead of issuing an error, RMAA goes right on calculating and randomly spews out half truths and half garbage. It's obvious the developer didn't bother to put in many checks for validity of the input and output data. This can take much more subtle forms such as bad cabling, random noise (RFI from a cell phone, sound events from your PC, etc.),  not being detected during the test.

Here's an example of RMAA inventing impossible results out of nothing. It was run with no test file at all and, instead of timing out, it ran without a single error or warning and here's the result:

Wow, 245% IMD, that's pretty bad! I didn't know you could have more than 100%. But the THD is “only” 1.7%. The frequency response doesn't look so good though. I joke, but you get the idea. This doesn't inspire confidence in the software. And then there are messages like this one:

So clipping occurred but 0.000% of the samples were actually clipped? Hmmm. The funny thing is, in this case, there was zero clipping by the ADC (my Benchmark ADC1 has clip lights that stay on if it even clips a single sample until you reset the LEDs). And what's even more interesting is lowering the level by several dB and running the test again (as the error suggests doing) yields the exact same error! It appears to just be a case of RMAA getting confused trying to test this particular device (a device that tests just fine on my Prism dScope). Again, this does nothing to boost confidence in RMAA results.

RMAA DESIGNED BACKWARDS? I’ve seen enough weird results from RMAA I have to wonder if the developer designed it backwards to match the specs of a reference soundcard? Whatever test signals and analysis being used might have been “tweaked” until he got numbers that roughly matched the specs of some respected soundcard. If true, this would explain a lot. Ideally RMAA would just do all the math correctly and the results would be verified against a professional audio analyzer to make sure the math was accurate. But it often doesn’t agree and I’m at a loss to otherwise explain why unless there are either bugs in the calculations or it was designed to produce the expected numbers when testing a certain soundcard.

RMAA TIPS: So how is a person supposed to make RMAA measurements more accurately? First, use the best PC audio hardware you can. A good device would be something like the RME Fireface UC but, in my experience, proprietary low latency drivers designed for multi-track recording don’t work well (or at all) with RMAA. Example popular devices that do work are the E-Mu USB audio interfaces:

• Creative/E-Mu 0202 - This device has decent A/D and D/A performance but it has un-calibrated level controls that cover a wide range of gain (65 dB!) and are very "touchy". This makes setting the level the same for both channels just about impossible. So you can’t know the actual channel balance of the device you’re trying to test and certain other tests can be inaccurate. The 0202 also has a mic preamp in the signal path on just one channel. Not surprisingly, that channel has higher noise than the line-in only channel. So any device you test with the 0202 is prone to rather lopsided results, weird crosstalk numbers, etc. It's also at the mercy of the sometimes noisy USB power supply. And it has serious issues on some PCs with its proprietary buggy drivers.
  
• Creative/E-Mu 0404 - This device is better than the 0202 in that it has its own dedicated power, and both channels have the same signal path (unfortunately both have mic preamps). It also has slightly better metering but still suffers from the "touchy" gain control problem and the inability to exactly match the channels without some external help. See the section below on Levels. Another downside of the 0404 is it requires the same special drivers as the 0202. The drivers haven't been updated in years and especially can have issues with Windows 7. As far as I know there only buggy beta 64 bit drivers.

SETTING LEVELS: This is critical for many reasons. First, it's useful to know if the device you’re testing has a channel imbalance. But if you use the meters in RMAA to say adjust your E-Mu 0404, you will remove any channel imbalance anywhere in the signal chain. You can either feed known matched levels (i.e. verified with a meter) into both inputs and then run the RMAA calibration, or use a "Y" cable to temporarily split one channel into two outputs to set the levels. That way you know any channel imbalance that shows up during testing is the device, not your set up.

MEASURING LEVELS: As discussed earlier, it's important to know what levels you're really testing at. So it's best to use an external true-RMS meter that’s accurate across audio frequencies. Most cheap DMM's, even many that claim true-RMS, are only designed to be accurate around 50 – 400 hz. They often "roll off" dramatically or behave unpredictably at higher, or lower, audio frequencies. True RMS calculation is non-trivial and it’s expensive to have it be accurate at higher frequencies. So most reasonably priced meters don’t bother as they’re not intended for audio use. So if you're testing at say 1 Khz, they might read only half the real value. Looks at the specs for the meter, or if you already have one, test it on the output of your sound interface with test tones across the audio spectrum to see how flat it is (or isn't). It's a hassle to do the math at different frequencies to apply a correction factor but it’s cheaper than buying a more expensive meter.

CHOOSING A TEST LEVEL: If you're testing a device with a volume control, how far do you turn up the volume? I would suggest picking a value that's  well below the maximum output to help assure it won't be close to clipping. But if you test at too low of a value, the noise and distortion numbers will be much worse because noise will dominate the measurement. If you have a meter (see above) and the maximum output level is specified for the device a good starting point is about 70% of that value (roughly –3 dB). But this may also depend on the load being used. if the maximum output is not specified, see the next paragraph.

REFERENCE LEVELS: Here are some guidelines for reference levels:

• Headphone Outputs - A good guideline for headphone outputs is at least 1mW into 32 ohms as that's used in the manufacture's specs of many devices and a value nearly any device can manage without clipping. P = (V*V)/R which means V = SquareRoot(P*R). So the square root of (.001v * 32 ohms) is 0.179 volts or about 180 mV RMS. This also happens to be about the typical power level most listen to their music at with typical low impedance headphones. So use your meter to set the player/amp to 180 mV RMS while playing a 0dBFS 1 Khz reference file. Then set your levels on your PC sound interface/mixer controls to be just under clipping (0 dB) while monitoring the 180 mV signal. But, beware, such low reference levels may challenge your PC hardware when making noise and crosstalk measurements. Many headphone outputs can deliver 300 – 500 mV into 32 ohms at low distortion so you might want to run a test at several different levels to find where the distortion starts to dramatically increase and the run the test just below that level.
• Line Level Outputs – The Redbook standard for home digital audio gear is 2 Vrms. That’s what you’ll get from most home CD players playing a 0 dBFS signal. But portable and USB powered line outputs are often lower—typically in the range of 0.5 volts to 1.6 volts. And some gear can be above 2 V rms.
• Speaker Outputs – As mentioned earlier, you need to use a voltage divider to avoid overloading and likely damaging the line level input of your RMAA sound hardware. This is not something I want to cover as if someone gets it wrong it could cause expensive damage. But for those in the know, the standard level for checking noise and distortion is typically 1 watt into 8 ohms which is 2.8 Vrms.

RUN A LOOPBACK TEST AT THE SAME LEVEL: Regardless of what sound hardware you're using, test it with a "loop back" test by connecting the inputs to outputs as instructed on the RMAA website to know what the "baseline" performance is at that level. Try to do this test at the same absolute level you're going to use for testing your piece of gear (i.e. the 180 mV mentioned above, etc). The signal to noise ratio, distortion, crosstalk, etc. of your PC interface will change at different test levels.

USE A PROPER LOAD: See the earlier section on loading. Ideally use a resistive test load. For devices intended to drive headphones, use resistors in the range of 15 – 600 ohms. A good single value is 32 ohms. Or test at several different impedances if you want. Even small 1/4 watt resistors are fine. They should be carbon film, and not wire wound. Use caution using headphones as a load both because they may be damaged by test tones at high levels, and they can also pick up room noise and create erroneous results due to the their impedance variations with frequency. For line level outputs the (often unknown) load of your RMAA sound hardware is probably sufficient. For speaker outputs the normal load is 8 ohms and it needs to be rated to handle whatever power level you’re going to test at.

VERIFY YOUR CABLES:  You don’t need $100 audiophile cables, but $1 ones from eBay are not a good idea either. Because of the way RMAA works, if you have a cable problem you may never know and just get marginal numbers. So test all the cables in the signal path by listening to a test tone while moving them around. If you hear static, or worse, replace the cable.  And, when testing crosstalk, cables become a huge problem if there’s any extra shared ground wiring.

RUN THE TESTS MULTIPLE TIMES: A good way to help rule out intermittent problems is to run the tests multiple times. If you get consistent results, they’re more likely to be valid. If the results vary by more than 5%, something is likely wrong.

USE A REAL OSCILLOSCOPE:  If possible, it’s worth checking the output, especially driving a real world load, with an oscilloscope that has a bandwidth out to at least 5 Mhz. This may show problems not seen in RMAA using a sound card limited to 20 Khz – 96 Khz. You can also measure the slew rate on a real scope and better evaluate square wave and impulse performance. Beware most scopes have the same grounding issues as PC sound hardware.

HOW I DO IT: For RMAA testing I use a Benchmark ADC1 for the A/D, a Benchmark DAC1 Pre for D/A, a 6 1/2 digit Agilent bench DMM that's extremely accurate and flat from 10 hz - 100 Khz for exact levels and other measurements, and one of several oscilloscopes with 60+ Mhz bandwidth. But, far more often, I use a dScope audio analyzer as it can make far more measurements, conforms to international testing standards, and produces far better and more accurate results. See my blog post Testing Methods for more information.