The X Files (merged cable topic)

anything and nothing
ChrisfromRI
senior member
senior member
United States of America
Posts: 466
Joined: 13 May 2004 04:17
Location: New England

Why Blind Testing isn't a Panacea

Post by ChrisfromRI » 22 Apr 2019 19:02

"Because it was scientifically proven to be useless more than 60 years ago.

A speech scientist by the name of Irwin Pollack have conducted an experiment in the early 1950s. In a blind ABX listening test, he asked people to distinguish minimal pairs of consonants (like “r” and “l”, or “t” and “p”).

He found out that listeners had no problem telling these consonants apart when they were played back immediately one after the other. But as he increased the pause between the playbacks, the listener’s ability to distinguish between them diminished. Once the time separating the sounds exceeded 10-15 milliseconds (approximately 1/100th of a second), people had a really hard time telling obviously different sounds apart. Their answers became statistically no better than a random guess.

Since then, the experiment was repeated many times (last major update in 2000, Reliability of a dichotic consonant-vowel pairs task using an ABX procedure.)

So reliably recognizing the difference between similar sounds in an ABX environment is impossible. 15ms playback gap, and the listener’s guess becomes no better than random. This happens because humans don't have any meaningful waveform memory. We cannot exactly recall the sound itself, and rely on various mental models for comparison. It takes time and effort to develop these models, thus making us really bad at playing "spot the sonic difference right now and here" game.

Also, please note that the experimenters were using the sounds of speech. Human ears have significantly better resolution and discrimination in the speech spectrum. If a comparison method is not working well with speech, it would not work at all with music.

So the “double blind testing” crowd is worshiping an ABX protocol that was scientifically proven more than 60 years ago to be completely unsuitable for telling similar sounds apart. And they insist all the other methods are “unscientific.”

The irony seems to be lost on them."

The author of the above short synopsis is a gentleman named Alexey Kornienkov, and I read his view and then partially researched it coming to the same conclusion he so carefully described above.

I believe that there are good tests and bad tests, regardless of whether they are done blind or sighted.

ChrisfromRI
senior member
senior member
United States of America
Posts: 466
Joined: 13 May 2004 04:17
Location: New England

Re: Why Blind Testing isn't a Panacea

Post by ChrisfromRI » 22 Apr 2019 19:53

OTOH, I can tell about a great example of how sighted testing absolutely gave a perfect result.

I have been a ham radio operator for a real long time. 40+ years ago I began experimenting with how to copy Morse Code better during demanding band conditions with interference and poor signal to noise ratios - when the desired signal was weak, at or even dipping below the noise level on the band and due to high activity there was also some interference in the form of nearby in frequency signals as well. I determined that even though my receiver was a mono source that I could wire my headphones out of phase and somehow I seemed to be able to concentrate better on the desired signal I was trying to copy, while not being disturbed as much by the interference and noise. I used these headphones to make successful moon-bounce contacts with signals that were right at my noise floor. My conclusion was that when wired in phase the sound of all of the noise, interference, and the desired signal was just outside of my ears spatially and challenging to concentrate on just the desired signal. OTOH, wired out of phase the signals were spread out along a line between my ears inside my head, and I could concentrate better on the desired signal because of these small spatial differences.

So, I started looking to buy headphones that could easily be wired out of phase and my whole hamshack had multiple sets of headphones connected to different receivers that were all wired out of phase. During the Walkman era I was working in NYC and would walk past street vendors with tables full of "New In Package" headphones for use with a Walkman and had a separate wire coming down from each ear, and selling for only $3 or $4. Maybe they were knock-offs or whatever, but they were lightweight and I could cut the mini stereo connector off the end of the cable, and wire the two headphone wires out of phase into a proper 1/4 inch ham radio headphone jack. I had a whole bunch of them around my hamshack.

A ham friend of mine was on his way to a DX-pedition where a group of dozen or so seasoned hams and excellent high speed Morse Code operators were to travel by plane and then by boat to a remote island that was only mostly out of the water part of the year. That island was considered a rare country code "ham call letter prefix" by hams and the hams that were collecting and competing on how many "countries" they had talked to in the world hadn't had the opportunity to even try to talk to this remote place for several decades. There was much pent up demand to activate the remote island, and my friend was going. It was in the days of early computer logging and he needed me to quickly make him a cable to go from a computer he would log contacts on, and which would automatically key the Morse Code input of the radio and speed the operation (the goal was 3 contacts with 3 different calling stations per minute). He was on his way to the airport and needed this cable urgently so I immediately made the cable for him, and also quickly threw a couple of extra RF and Audio adapters and small bits in his travel bag, including one set of those $3 headphones wired out of phase. I didn't bother to tell him they were wired in any particular way because it was standard practice for me by that point, despite being an unknown practice, and I just mentioned they were very lightweight and comfortable to wear for hours at a time as they didn't put a lot of pressure against the ears.

After a few weeks he returned and held the headphones out to me with a big smile on his face and said to me they had no less than 20 sets of headphones on that island, including expensive airplane pilot headphones, and every operator soon began positioning to use my "Good" headphones. They became know by the group as the "Good" ones that could allow the operator to run a faster rate of contacts (recorded by computer data) and just hear better than any of the other headphones. In many cases they sat at the radio with 2 operators listening via a Y-Adapter with 2 sets of headphones, helping each other to copy the weaker signals, and whoever wore my "Good" headphones could always hear better. They concluded that my "Good" headphones were in fact the best headphones anyone had ever used and they all wanted to buy a set. My friend apologized to me that the foam pads had gotten a bit worn due to heavy use, and his jaw dropped when I said "oh, don't worry about it, you can just keep them". He immediately said "no, I could never take these for free, they are too amazing". We both had a good laugh when I told him they were cheap $3 headphones, bought off the NYC streets and probably knock-offs, that I had just wired out of phase to take advantage of what I felt was a psychoacoustic phenomenom allowing me to hear better.

So it was an accidental sighted test of out of phase wired headphones, and every operator that wore them knew exactly which headphones they were wearing, and each agreed they heard better with the chintzy ones I sent as well as being comfortable for hours. They had no idea why they heard better, just that my set of headphones were indeed better for everyone, and there were even arguments about who should be wearing the "Good" headphones when they were operating. I never intended to create a test, just to toss something useful into my friends travel bag. Funny how it proves that sighted testing reveals differences without confirmation bias...

hobie1dog
senior member
senior member
United States of America
Posts: 330
Joined: 01 Oct 2008 20:41
Location: Cornelius, NC

Re: Why Blind Testing isn't a Panacea

Post by hobie1dog » 22 Apr 2019 23:05

What's your call sign?

W4SEX

Woodbrains
senior member
senior member
England
Posts: 373
Joined: 23 Feb 2018 18:18
Location: Liverpool

Re: Why Blind Testing isn't a Panacea

Post by Woodbrains » 23 Apr 2019 00:05

ChrisfromRI wrote:
22 Apr 2019 19:02
"Because it was scientifically proven to be useless more than 60 years ago.

A speech scientist by the name of Irwin Pollack have conducted an experiment in the early 1950s. In a blind ABX listening test, he asked people to distinguish minimal pairs of consonants (like “r” and “l”, or “t” and “p”).

He found out that listeners had no problem telling these consonants apart when they were played back immediately one after the other. But as he increased the pause between the playbacks, the listener’s ability to distinguish between them diminished. Once the time separating the sounds exceeded 10-15 milliseconds (approximately 1/100th of a second), people had a really hard time telling obviously different sounds apart. Their answers became statistically no better than a random guess.

Since then, the experiment was repeated many times (last major update in 2000, Reliability of a dichotic consonant-vowel pairs task using an ABX procedure.)

So reliably recognizing the difference between similar sounds in an ABX environment is impossible. 15ms playback gap, and the listener’s guess becomes no better than random. This happens because humans don't have any meaningful waveform memory. We cannot exactly recall the sound itself, and rely on various mental models for comparison. It takes time and effort to develop these models, thus making us really bad at playing "spot the sonic difference right now and here" game.

Also, please note that the experimenters were using the sounds of speech. Human ears have significantly better resolution and discrimination in the speech spectrum. If a comparison method is not working well with speech, it would not work at all with music.

So the “double blind testing” crowd is worshiping an ABX protocol that was scientifically proven more than 60 years ago to be completely unsuitable for telling similar sounds apart. And they insist all the other methods are “unscientific.”

The irony seems to be lost on them."

The author of the above short synopsis is a gentleman named Alexey Kornienkov, and I read his view and then partially researched it coming to the same conclusion he so carefully described above.

I believe that there are good tests and bad tests, regardless of whether they are done blind or sighted.
Hello,

This post does not disprove the validity of double blind testing, I'm not sure why you think it does. It might tell us things about the human capacity of auditory processing abilities and limitations, but nothing about how the testing method was 'flawed'.

Ironically, you have indulged yourself a confirmation bias here, something that double blind testing takes pains to eliminate. Having a notion that double blind testing is somehow flawed and then finding one isolated article that (incorrectly assumed, as it happens) confirms what you think is not a valid device.

I'm not sure why you want to disprove the validity of double blind testing, anyway. Your second post can never be a reliable source of information just because it wasn't double bind tested. It may well be all true, but only the rigour of the double blind test can eliminate the spurious artifacts that will always exist when taking the results. How do we know the tester did not imply which way the subjects should decide by body language? How do we know that one subject did not influence another subject which way to decide, either subliminally or purposely? And dozens of other artifacts that will taint the data and cause any conclusion to be dismissed out of hand.

Mike.

ChrisfromRI
senior member
senior member
United States of America
Posts: 466
Joined: 13 May 2004 04:17
Location: New England

Re: Why Blind Testing isn't a Panacea

Post by ChrisfromRI » 23 Apr 2019 03:05

My point is that any test can more easily be invalid than valid, whether blind or sighted. Just because a test is done blind doesn't make it any more valid than if it were done sighted. Personally, I take anyone else's test result with a grain of salt.

WRT the sighted headphone test example, the FACT that the out-of-phase headphone I sent was better was PROVED by the QSO (successful contact) rates logged in the computer in the first couple of days of the expedition, and evident in the computer report. More contacts per minute/hour were logged, with fewer resends for confirmation of something missed, fewer broken call signs, all proving just plain better receiving with the out-of-phase headphones. These dozen folks were world class Morse Code operators that were selected to go on this expedition, and they all preferred my cheapo bought off the street headphones to all of the other headphones they brought along - despite having an assortment of some very expensive fancy headphones available to use (including their own personal favorites). They didn't know why there was a difference, just that there was a difference, and it was clearly evident in the computer log report data.

In the end, I believe the "rigour of the double blind test" is too often a fallacy.

JaS
engine room
engine room
Posts: 11235
Joined: 12 Feb 2002 16:32

Re: Why Blind Testing isn't a Panacea

Post by JaS » 23 Apr 2019 14:56

While the discussion of Double Blind Testing in audio isn't entirely focused on differences between cables, it's essentially part of the same argument so the DBT thread has been merged with the cable topic, it's spiritual home.

cafe latte
vinyl addict
vinyl addict
Australia
Posts: 11151
Joined: 11 Oct 2009 04:27
Location: Cattle property near Ravenshoe Qld Australia

Re: Why Blind Testing isn't a Panacea

Post by cafe latte » 23 Apr 2019 23:46

ChrisfromRI wrote:
23 Apr 2019 03:05
My point is that any test can more easily be invalid than valid, whether blind or sighted. Just because a test is done blind doesn't make it any more valid than if it were done sighted. Personally, I take anyone else's test result with a grain of salt.

WRT the sighted headphone test example, the FACT that the out-of-phase headphone I sent was better was PROVED by the QSO (successful contact) rates logged in the computer in the first couple of days of the expedition, and evident in the computer report. More contacts per minute/hour were logged, with fewer resends for confirmation of something missed, fewer broken call signs, all proving just plain better receiving with the out-of-phase headphones. These dozen folks were world class Morse Code operators that were selected to go on this expedition, and they all preferred my cheapo bought off the street headphones to all of the other headphones they brought along - despite having an assortment of some very expensive fancy headphones available to use (including their own personal favorites). They didn't know why there was a difference, just that there was a difference, and it was clearly evident in the computer log report data.

In the end, I believe the "rigour of the double blind test" is too often a fallacy.
Sometimes a blind test takes preconceived ideas out of the equation. Twice I have used blind testing to make a point with someone who was rutted in a belief. Once someone I was working with went on and on how vinyl sounded terrible and how he could instantly tell vinyl even from a good system, this was more than I could resits. I selected a pop free album and recorded onto a cd a couple of tracks. Took it to his house and knowing he liked this artist I asked him to put the cd on. He loved it, asked about the quality great he replied, I said sure there is nothing wrong with it? No it sounds amazing he said, this is when I told him it was recorded off a vinyl record :lol: He was stunned and stoof their with his mouth open.
The other time was with cables someone who was convinced there was a night and day difference with cables I swapped back and forth with him blind to the swaps and asked each time what he thought. I did the same when he swapped, neither of us could determine the differences blind, yet he thought there was obvious differences in cables and he was convinced he would hear the difference easily blind.
Blind testing has a place it removes expectation bias which we need to remove to get a fair test.
Chris

Sunwire
long player
long player
United States of America
Posts: 2038
Joined: 07 Oct 2004 21:54
Location: New York

Re: The X Files (merged cable topic)

Post by Sunwire » 24 Apr 2019 04:08

The example of the headphones doesn't seem like a counter example to double blind testing.
In fact, it seems more like an example of the efficacy of blind testing.
The ham guys were not aware of the out of phase wiring of the headphones. They were blind to the actual difference between those headphones and the others. Yet they expressed a clear preference for them.
So, this "experiment" confirmed the advantage of out of phase wiring for this application, even though the "testers" didn't even know what they were "testing".

The idea that Pollack's study disproves the efficacy of ABX testing makes no sense to me.

There is a much more complete discussion of this subject here, with arguments on both sides:
https://www.quora.com/Why-do-so-many-au ... components

Post Reply