4 Comments

There’s much to love about YouTube (Rick Beato, Rahsaan Roland Kirk playing “I Say a Little Prayer”), but it’s coming at a price: the dumbing down of the audio industry. I got some inkling of this future in 1990, the first year of my audio career, when an acquaintance asked me, “How are those Bose speakers? I heard them on a TV commercial, and they sounded pretty good.” He was in the oil biz, not the audio biz, so I didn’t blame him for failing to grasp that he was hearing not a Bose system, but recorded music played through his TV speakers. Yet thanks to the Internet’s negligible barriers to entry, people who claim to be audio experts are now making the same mistake he did.

I’ve noticed an increasing desire among manufacturers, retailers, and reviewers to demo audio products over the Internet—streaming some representation of the sound of a headphone, a speaker, or even an amplifier through a webpage, YouTube, etc., with the idea that you can listen to it from home and judge the sound quality for yourself. In an era when it’s nearly impossible to get a demo in person, the appeal of this method is obvious. However, the people who do it rarely consider the problems involved.

Brent Butterworth

A good example is a YouTube video I saw the other day, where a reviewer described a high-end audio system and then said, “Let’s give it a listen, shall we?”—and played a recording through the system. Any audio pundit worthy of respect would know that you’re not hearing just that featured system—you’re also hearing the acoustics of the room it’s in, plus the response of whatever unidentified microphone was used for the recording, plus the artifacts of the audio codec used to stream the video, plus the characteristics of whatever audio gear and room you’re using to listen to the YouTube video. It tells you nothing about the performance of the audio gear featured in the video. It’s as absurd as trying to judge the quality of a wine by first mixing it with two or three other wines before you drink it.

In the desire to alert readers to the inherent flaws of these demos—and in the hope that they might someday be conducted with more thought and care—I thought I’d use this column to highlight the only laudable online demo effort I’ve seen, and explain where other attempts went wrong.

The one good effort

By far, the most serious attempt I’ve seen at doing audio demos on the Internet is the SpeakerCompare feature on the Crutchfield.com website, which lets you compare speakers online through headphones. SpeakerCompare attempts to eliminate any sonic changes your headphones make to the sound, using compensation curves based on Crutchfield’s own measurements of numerous headphones. Before being streamed through the website, the music selections are processed using Crutchfield’s measurements of the speakers, which were made in an anechoic chamber. Further processing simulates the effect that a typical room will have on the speakers’ sounds.

SpeakerCompare

The system is wonderfully simple to use: select your headphones, choose a music track from one of several genres, and then click the onscreen Play button under the speaker you want to hear. It’s easy to switch between speakers and listen at whatever level is comfortable for you.

This effort couldn’t have been more earnest. I’d guess it cost Crutchfield between half a million and a million dollars, including consultation by a couple of PhDs. Even so, I’d rank it as only modestly successful.

I happen to have Klipsch Reference R-51Ms, which are among the speakers included in SpeakerCompare, so I was able to compare the sound of the actual speakers in my listening room (playing the same tunes, but from Spotify) to the sound of the speakers simulated through Audeze LCD-X headphones. I’d say the simulation got the midrange about right—in both presentations, it had that “little bit bright” sound characteristic of Klipsches in this price range. But the bass was considerably elevated in the simulation, which made the treble sound subjectively softer, so I wasn’t getting an accurate idea of what the speaker sounded like.

SpeakerCompare

Comparing the simulated R-51M to a simulation of the Jamo S 801, I could tell that the Jamo sounded a little bassier and a little softer—no surprise, as it has a conventional dome tweeter versus the R-51M’s horn-loaded tweeter. But because of the overall bass-heavy coloration of both presentations, I couldn’t gauge which speaker I’d like best. My results were almost the same when I switched to the AKG N60 NC headphones, which are included as a listening option in SpeakerCompare. The differences between the simulation and my live, in-room listening should come as no surprise, as it’s unlikely the acoustics of my room match the profile Crutchfield used to create the simulation. Perhaps I’d have gotten better results had I popped $25 to rent the SpeakerCompare Listening Kit, which includes a set of Sennheiser headphones that I assume were chosen to give optimum results.

Despite the considerable effort Crutchfield put into developing SpeakerCompare, the company doesn’t claim it’s a substitute for a live demo, and Crutchfield encourages customers to make their final speaker choice only after a free in-home trial. Sadly, none of the other people I’ve seen offering online audio demos are so dedicated . . . or anywhere near as wise.

The many bad ones

On the flip side, there’s an unfortunate trend in YouTube reviews to simply record the sound coming from headphones or speakers in some convenient and inexpensive manner, and stream the result as a representation of that product’s performance, with little or no regard for what changes the sound might go through between the product’s transducers and the listener’s ears.

In one example I recently saw, a reviewer on YouTube was attempting to demonstrate the sonic characteristics of headphones by playing music through them, recording the result through a miniDSP EARS headphone measurement rig, and streaming that out to his viewers. Presumably, the viewers would listen to the result on headphones, although I can’t recall whether the reviewer made any particular suggestions in this area. So they could be listening on a $50,000 stereo system, through the speaker built into their phone, or anything in between.

To understand the problem here, remember that the actual experience in a real-world evaluation of a set of headphones has two fundamental components:

  1. The characteristics of the headphones under test
  2. The acoustical and psychoacoustical characteristics of your ears and brain

When you’re listening to the video described above, you’re hearing not just two components but four fundamental components:

  1. The characteristics of the headphones under test
  2. The characteristics of the ear simulator used for the recording (in this case, the miniDSP EARS)
  3. The characteristics of the headphones (or speakers) and electronics used for monitoring
  4. The acoustical and psychoacoustical characteristics of your ears and brain

Obviously, what you’re hearing from the audio stream, through your headphones or speakers, will be very different from what you hear when you listen directly to the headphones being demonstrated. You may be able to hear a difference between two products demonstrated in this fashion, but you won’t be able to put it in perspective. How can you tell if a headphone with a 6dB peak at 3kHz sounds better than a headphone with an 8dB peak at 2kHz if you’re listening through an artificial ear that doesn’t conform to any industry standards, and then through your headphones, which contribute their own peaks and dips?

To illustrate how much the sound of headphones can change in an online demo, I measured the frequency response of a set of AKG K371 headphones mounted on a GRAS Model 43AG ear/cheek simulator, using white noise. Then I mounted the K371s on a miniDSP EARS, ran that sound into a computer, and monitored it through Beyerdynamic T5 (3rd generation) headphones, which I then mounted on the 43AG.

Comparison

You can see the result on the chart above. It’s a huge difference, and it was unpleasant to listen to—yet this is actually better than what a typical listener will experience, because the audio hasn’t gone through one of the various audio-compression codecs YouTube uses, and the T5s are a better monitoring system than probably 99% of YouTube users will have.

I’ve often seen people who do online audio demos (and who measure using the miniDSP EARS) admitting their system isn’t perfect but saying, “I think it’s pretty close.” Yet this is just their gut feeling; I’ve never seen any of them present any sort of methodology they used to come to their “pretty close” judgment, and few, if any, have the equipment or the expertise to create such a methodology. Their gut feeling that their result is “pretty close”—without, of course, having the slightest clue what equipment their viewers are using to listen—may satisfy some YouTube viewers, but anyone with a grasp of audio engineering will reflexively roll their eyes.

I think it’s possible to do halfway decent online audio demos, at least of headphones. One way you might actually get “pretty close” is to mount a popular set of headphones, such as Sony’s MDR-7506, on the miniDSP EARS and measure the combined transfer function—i.e., the total change made in the frequency response by the EARS and the Sonys. Use an equalizer or an audio processing plug-in to create an inverted version of that transfer function, and use that curve to process the sound on the video. Now put whatever headphones you're testing on the EARS, and stream that audio, processed with the equalizer. When the listener puts on the MDR-7506 headphones (which many headphone enthusiasts and audio pros already own, and which typically cost only about $99), in theory, at least, they’ll be hearing the frequency response of the headphones under test, minus the sonic effects of the EARS and the Sonys. Of course, it would work correctly only with the MDR-7506 headphones, and maybe tolerably well for a few headphones with a similar response. Ideally, you’d offer listening options for several popular headphones, as Crutchfield does, although that would be practical only on a dedicated website—on YouTube, you’d have to offer a separate video for each headphone model.

I did something similar to this for the demo videos accompanying my article “How Much Noise Do Your Headphones Really Block?” It was complicated, but it worked fairly well—although I was demoing noise levels, not the subtleties of an audio product’s performance, so the frequency response of my viewers’ audio systems didn’t matter much.

I hope that if reviewers and retailers who do online demos consider how much effort and expense Crutchfield put into SpeakerCompare, they’ll realize how amateurish it is for someone to just hook up a microphone or a cheap headphone measurement rig and hope for the best. And I hope they’ll invest some time, money, and creative (but informed) thinking in making their online demos more useful. But in a subculture where an unsupported assertion that something’s “pretty close” satisfies enough viewers for the presenter to turn a profit, I’m not optimistic that this will happen.

. . . Brent Butterworth
This email address is being protected from spambots. You need JavaScript enabled to view it.

Say something here...
Log in with ( Sign Up ? )
or post as a guest
People in conversation:
Loading comment... The comment will be refreshed after 00:00.
  • This commment is unpublished.
    Allan · 3 years ago
    I found SpeakerCompare to be a decent comparative tool.  As a way to increase sales and decrease returns it was probably a good investment.
  • This commment is unpublished.
    nat w · 3 years ago
    I found the Crutchfield system pretty compelling when I first tried it about six months ago. No, it is not the same as hearing the speakers in ones own room. But it was possible to differentiate some aspects of tone.

    Lately, their code is broken and one cannot switch between models; but to a lay person, it just seems like all the samples sound the same since the UI makes it look like you are switching between models! Sure hope they fix it! Otherwise, it is worse than half way working but undermines the idea itself.

    And that would be a shame!

    This is an idea that had been floating around for a while and none other than Genelec more than a decade ago has said that it works well for critical testing of their high end speakers and monitors (which are industry standards for a reason). https://assets.ctfassets.net/4zjnzn055a4v/4SO5nkzx17qwEUpnOG1gsN/c554ff26c49f0984b8493f79228e7c41/2008_hiekkanen_makivirta_karjalainen.pdf

    And of course things like the REALISER from Smyth Research has for quite a while now given consumers access to something similar.
    • This commment is unpublished.
      nat w · 3 years ago
      This is the paper from the research team associated with Genelec:
      "Virtualized Listening Tests for Loudspeakers"
      Audio Engineering Society Convention Paper 7367
      Presented at the 124th Convention
      2008 May 17–20 Amsterdam, The Netherlands
  • This commment is unpublished.
    Mauro · 3 years ago
    Nice scientific approach!
    YouTube reviewers are usually good entertainers.. no one has a scientific background afaik as Tyll did (engineer) or as you have come to develop.
    I think it just depends on the expectation of the viewer.

    As you mention ‘profit’ and I would add visibility, they do have some advantages anyway.
    In case of the main YouTube reviewers, they usually have a lot of headphones for comparison as Tyll did. And this I think it might be good, but not everyone is taking good advantage of it :?

    So.. you might want to take a leap and jump into the YouTube world?! (happy)
    • This commment is unpublished.
      Brent Butterworth · 3 years ago
      Agreed. Too many of them lack basic knowledge of the sort you can get by reading a few books (although that's always been true of reviewers), but I think perhaps more important is that they lack the experience of working with a listening panel. Once you solicit the opinions of other listeners, especially with headphones, you realize your opinions are not reliable. So you invest in measurement gear, read AES papers, learn from as many people as you can, and your opinions evolve and become wiser as your knowledge grows. Tyll set the example, and I try to follow.

      But I understand the appeal of personality -- I have watched a zillion Davie504 videos, even though I have learned nothing from them about music or playing bass. :)
      • This commment is unpublished.
        Mauro · 3 years ago
        That would be great: videos of Brent à la Davie504!
        That would be the definitive headphone review format!!
        Epic! (bomb)

        SoundStage should hire Davide and make a format right now! :)
  • This commment is unpublished.
    oluv · 3 years ago
    It’s actually more than “pretty close” when done right, talking bullshit is also “pretty easy”:
    • This commment is unpublished.
      Dealux · 3 years ago
      If you record it on your head it will take on the characteristics of your ears. The GRAS rig is an average of many people and likely closer to the truth.

      Also, the Sonarworks target is not neutral (it's too warm).
      • This commment is unpublished.
        Andres · 3 years ago
        I have used Sonarworks with HD600, HD800, dt990, Focal Clear, MDR7506 etc. and for me it is quite neutral. Especially HD600, cause it does not have huge narrow peaks or dips.
    • This commment is unpublished.
      Brent Butterworth · 3 years ago
      I've made my case, and you've responded by saying I'm "talking BS." Oluv, you can say what you want about another reviewer, we're all fair game. But in your last video, you're suggesting Sean Olive has a hearing problem, and you seem to think you're smarter than the people who developed the IEC 60268-7 standard, all the people who have developed professional headphone measurement gear, and multiple past presidents of the AES. This would make you either the greatest audio genius alive, or today's best example of Dunning-Kruger syndrome. There are incredibly knowledgeable and dedicated people on YouTube (Erin's Audio Corner is the best example), but doing what he does takes a big investment in money and learning. I encourage you to follow suit.

Latest Comments

gzost 1 years ago Yamaha YH-5000SE Headphones
https://www.superbestaudiofriends.org/index.php?threads/yamaha-yh5000se-orthodynamic-headphones-hp-1-successor.12712/page-3 has a screengrab of the only measurements out there at the moment. 
I'd not call ...
todd 1 years ago Yamaha YH-5000SE Headphones
Why not use the Focal 2022 for comparison? You have them reviewed here. Would be ...
Dustin 1 years ago Yamaha YH-5000SE Headphones
Why use the LCD 3 as a reference?  It's frequency response is far off from ...
This post is very informative and helped me lot for getting a understand of harman ...
Geoffrey Morrison 1 years ago Sennheiser HD 660S2 Headphones
@RagavThat's a lot of Sennheisers to review! I need to cover other brands too, but ...
Fantastic review! Could you also review the venerable Sennheiser HD 600 as also the HD ...
Geoffrey Morrison 1 years ago Why Do People Think Headphones Are Magic?
@Doug SchneiderYeah, that's definitely the same idea. 
Geoffrey Morrison 1 years ago Why Do People Think Headphones Are Magic?
@Doug SchneiderThat's another good one!
Doug Schneider 1 years ago Why Do People Think Headphones Are Magic?
You have to have Facebook access to see the link below to a post by ...
Doug Schneider 1 years ago Why Do People Think Headphones Are Magic?
Headphone things aren't jumping to mind -- but they could. But the thing that's always ...