WE ARE AUDIOPHILES AND ARE PROUND OF IT:
TEST METHOD

'd like to invite you to read this text that explains the way I test audio products. I have been receiving this type of questions, sometimes asked indirectly, from Readers who want to know how my tests are performed and how assessments are formed. I hope this editorial will explain a thing or two for you :)

TEST METHOD

It is quite interesting that the more information about how I perform my test I share with Readers the less understanding for my method I actually feel from your side. I might be wrong but it seems to me that maybe you don't fully understand the method. Or maybe it just needs to be repeated and explained every now and then? In fact I wrote about it already a few times, usually in one of “We are audiophiles and are proud of it” editorials. Still, I'd like you to feel comfortable when reading reviews in „High Fidelity” and I'd like for you to be able to benefit from them as much as possible – to make it easier for you I am going to present my test method in a comprehensive way.

The word 'test' is defined as:

A procedure for critical evaluation; a means of determining the presence, quality, or truth of something; a trial.

But a text describing an audio device is not only a 'test' but also a 'review' that might be described as following:

Review is an opinion of a competent person on a book, movie, some product, and so on.

Tests of audio product is performed to find out as much as possible about its design, designer's idea for particular product, about its aesthetics and finally its performance. A journalist performing such a test is sort of a sensor or measuring device, whose purpose is to describe and to assess said product; obviously it doesn't have to be a journalist, but as long as his/her findings are published in some sort of printed or web-based magazine, he or she becomes one.

In the above quoted definitions one should notice different focus. Test in this understanding is something more related to technical side of the device, something performed in a more automatic way, meaning less dependent on a contest. On the other hand a review is always done in some contest and is related to an 'artistic' side of a reviewed product. But for a review to bear any real meaning it should be performed by a 'competent person'. So an audio component's test always covers two aspects: a literal one and a metaphoric one; a formalized one and an arbitrary one.

Test

In most of my test apart from 'preface', 'sound' and 'design' sections I also add one called 'test method'. It might seem pretentious, pompous even but I believe it is necessary for you to understand the section about the 'sound'. An audio test is namely a description of device's design and performance. It is a comparison, and that means comparing what I can hear to what my own reference system presents. In other words – listening to (for example) amplifier X I compare its performance to my reference amplifier, which is either my own, or another one also on loan for another test. Any test is surely more comprehensive when I can do more then one comparison, but the necessary condition is a head-to-head duel versus my own reference component of particular kind.

Why is it so important? For a test to have a certain universal value it has to be performed using some standard procedure. Repeating the same procedure for different devices makes results of such test more credible. To explain my method even better I decided to describe a whole procedure of a test of the Orpheus power amplifier made by Bulgarian company TR Audio Design, that you can find in the same issue of „High Fidelity”.

Step 1

A test of any audio component has three stages: a preparatory stage, an actual test and a final stage. First I have to choose a product for a test. There are two options – some products are offered by distributors and I choose something among proposed models, some products I find myself – that's how I came by Orpheus. I found RT Audio Design while doing some research on the Internet, contacted the owner and asked for some information on company and its products. I found their designing approach particularly interesting, their products looked really good on available photos, so I requested a DAC for a review.

This time it was even simpler as I already knew company's owner, so I just asked him, Mr Цветан Ценков, about what he was working on. As it turned out he was finishing his new class A power amplifier, with a stabilized power supply that could be turned into 250W beast in AB class if needed. Wow! I thought, that sounded really interesting. So we agreed a review and few days later courier brought me two parcels. After arrival of any product I have to unpack it, check if it wasn't damaged during transport and if it is functioning properly. If such a check-up turns out OK., usually the next step is a photo session. It's also a perfect opportunity to take a closer look at external and internal design of the component.

Step 2

Usually there is a few days break before I can actually start phase 2 as firstly I need to finish some current review. The new arrival might be used in the meantime to check how it works with component/s under review. Finally the actual test may begin – I can start comparing the product under review with my reference component of the same kind. In this particular case my reference device is a 150.000 PLN Soulution 710 power amp. Now a very important piece of information that some people commenting my reviews seem not to understand: I do not compare a component under review to another one representing similar class of performance, nor similarly priced.

A 2500 PLN amplifier, say Clones Audio 25iR reviewed for the same HF issue, worked in my system where each single component, including cables and anti-vibration elements, costs more than this particular amp. I don't know if you ever had anything to do with measuring procedures, but you should know that it is required from measuring instrument that it featured better parameters than a measured object/process and at least by an order of magnitude. Otherwise one would measure errors of both elements and wouldn't be able to separate ones from the others. A test of said amplifier driving 40.000 PLN loudspeakers placed on stands that cost the same and connected using cables that cost several thousands zloty is the only way to find out what the reviewed amplifier introduces to the system.

If I decided to use said amplifier with speakers from its price range I wouldn't be able to assess the amplifier, because I would actually listen to an amplifier/loudspeakers system and not amplifier itself. And I am supposed to perform a test of an amplifier, mind you. Theoretically I could listen to the amplifier with few different pairs of speakers but it still wouldn't tell me much about amplifier itself. The only choice is to listen to it being a part of a much more expensive system. The higher quality and pricier the component the more difficult this becomes because at some point performance of the component under review matches or even outperforms my reference system. At this point procedure itself isn't enough anymore – knowledge and experience of the reviewer comes into play – it is up to him whether he manages to assess and describe product's performance properly. Usually such devices that outperform my own allow me to learn something new and are a great source of experience that might come handy in future.

Step 3

After the actual test there is still time to combine the reviewed item with some others that wait for their turn. Later I have to pack it and send it back to distributor or manufacturer and I can finally sit down and write my review. There you have, that's how it looks like in a real life.

Point of reference

An important element here is a point of reference. A reference device is one of the possibilities. The other one is live sound. Which one is more important? It's been discussed for years and there were periods when one or the other option prevailed. For a long time the option of live music being an ultimate reference for audio system performance was considered to be a better one. So people while listening to the music on their systems looked for similarities between what they could hear and what they remembered from live concerts.

In 1960ties and 1970ties many audio magazines changed their approach and decided to use measurement results as the ultimate assessment criterion. Listening sessions became almost irrelevant. Many audio journalist claimed that differences between amplifiers were at most insignificant or even non-existing. Such an approach almost killed the whole audio industry, because it contradicted what audiophiles experienced in their own systems. Luckily first „Stereophile” and then also „The Absolute Sound” abandoned this approach and started to use listening sessions as a foundation of their assessments. These magazines (people working for them) also for a long time believed that a proper reference for music reproduction is a live performance.

I have my own opinion on the subject. I believe that there is no such thing as an objective, 'external' point of reference one might use. There is an intermediary element between live performance and sound reproduced by audio system – a recording. Several elements form the ultimate result - a recording – actual recording process, mixing, mastering and finally release form and quality.

Nobody ever managed to record music in such a way that it could sound exactly as live music, so each recording is only an attempt to achieve as good approximation of live event as possible. What we can hear is a result of choices of people responsible for the whole process. Everything matters from particular setting of microphones, through choice of particular equipment used in the process, the way material is mixed, and even the way it is released. And even if we had a perfectly transparent system at our disposal and if we were able to record and release a perfect 1:1 copy of a live event still to be able to assess a reproduction via audio system we would have to be firstly present in a studio during recording and secondly remember exactly all the aspects of this live performance. Each room “sounds” differently, each band, instrument, and even one performance differs from the next one.

So one would have to accept some sort of compromise, assume some sort of “standard” sound for each instrument, for example for a piano instead of for the piano played by the pianist in a particular room at particular time. And what about electronic or rock music? How could one assess system's performance using these genres of music? There is no one standard sound one could compare it to. Mind you, not all music lovers listen to jazz and classical music only.

A/B comparison

The only reliable, repeatable comparison is the A/B one, where A is a reference component and B is a reviewed one. Sure, going to live concerts helps to develop our sound sensitivity. But it is only an auxiliary element that might come handy when performing comparisons, also helping to train one's listening skills. But the key element remains a head-to-head comparison between two components.

In order to even consider comparing test results from, say, a year ago, with another one performed today, all conditions of both tests have remain unchanged. It is an extremely difficult task as we do change over time, our system does and the environment around the system too. But one should limit changes to absolute minimum and if they are implemented to the system they should be a part of a long-term process. So when I decide to replace my trusted speaker cable with another one I have to use both, older and new one simultaneously during the same test, because it allows me to recognize changes introduced by the new one, adjust to them and understand them.

For years now I have been performing tests at the same time of the day – I need to be sure that different time of the day doesn't influence test results. I make as few changes to my reference system as possible. Even if I do introduce some change I usually replace my component with another, higher class one made by the same manufacturer. I've been using Soulution 710 power amplifier already for four years despite the fact that already two years ago manufacturer released a newer model, 711, which I'm sure, offers even better performance. But still, replacing one with the other would mean some change in the system – that's why I'm reluctant to do it. Same goes for my loudspeakers – a year ago Harbeth released new, improved version, M40.2. I have to listen to them in my own system before I make my decision, but I'm also not in a hurry to do that.

And so on – I've been using Ancient Audio CD Player for a few years already, and before I used its less expensive version, Prime. I've been using Tara Labs speaker cables for five years and only now I'm considering a higher version, Evolution. Each change introduced to the system we know and like very much is quite a traumatic experience, even if it is change for better.

Let me repeat one more time: the very foundation of a reliable test is comparison. In my case the only comparison acceptable is an A/B one with A and B known. The ABX, or so called „double blind test” that is preferred by many engineers, who often don't even listen to the music on regular basis, does not offer reliable results (in my opinion). Those engineers who also do listen to the music regularly and know how to listen to it don't use ABX test – chief editor of „Stereophile”, John Atkinson, is a great example. You can find a whole series of articles discussing this topic HERE, HERE and HERE. Also Paul Miller, a chief editor of „Hi-Fi News” and Martin Colloms of „HIFICRITIC” share this opinion. My conclusion was the same and it was based on my experience gained behind mixing console, and from testing audio components and listening to live music.

The A/B comparison is about switching between components – listening to one for while and then switching to the other and listening to exactly the same music. And repeat the process until I can form an opinion about the component under review. I know what I listen to and what I compare it to. I can see reviewed components. For a limited time they become a part of my life just as they will of those who will purchase them. What one buys is not sound but a product. Of course one might treat the issue differently. One might treat an audio component as a 'black box' that is supposed to play sound and all its other features are irrelevant. Although I used the term 'black box' as a metaphor, many audio devices actually are just that. The other approach is focusing on product's appearance, functionality and ease of use; when that's a case performance plays a lesser role.

For my comparisons I use fragments of carefully selected tracks. In our 12 years long history of Cracow Sonic Society we have worked out a consensus about length of such music pieces – these should be no shorter then 1,5 and no longer then 2 minutes. And I follow this rule to the letter. When performing ABX tests fragments should be no longer then 10 seconds otherwise comparison becomes unreliable. I performed such (ABX) tests multiple times, with tables, statistical analysis and so on. They take a lot of time and effort and are simply tiresome. Their weakest link is... men performing them. When longer samples were used participants quickly got confused. After the main part of my test I also listen to whole tracks and albums. When reviewing sources (CD Players, DACs, turntables, cartridges, phonostages), interconnects, power cables, platforms and so on I indulge my headphone passion and spend long night hours listening to the music, usually whole albums, using favorite cans.

Because music is ultimately the most important element of audio. When assessing products I follow the rule “first, do no harm”. When reviewed component communicates with me, appeals to my emotions, teases my curiosity, encourages me to buy new albums, it means that it offers “my kind” of sound. And I mean “my kind” and not some “neutral” one, as there is no such thing. If there is no single 'external' reference point, and I tried to explain that there isn't, everything is relative. When reading reviews one should notice what are other components the reviewed one was paired with, what component it was compared to, what music was used for test, and what are personal preferences of the person conducting the test.

My sound

„My” sound seems to be dark. During my recent visit in Jacek Gawłowski's, Grammy Award winner, man responsible for Niemen albums remastering and now working on the whole Polish Jazz catalog, studio I learned that I wasn't the only one feeling this way about sound. Also Janusz, a member of Cracow Sonic Society, thinks the same way. “My” sound is dense, with solid bass foundation, that is needed not to pump air but to “release” midrange, let it truly shine, create large phantom images. It is also most resolving sound, that might appear to lack selectivity. It is difficult to point out instruments and planes because these are unite with the whole presentation, creating a cohesive, logical story. It is music above all.

If I am to choose between transparent, very precise sound but without properly developed lower midrange and a darker, but less resolving one I choose the latter. Over extended period of time it will be a better choice, more “human-friendly'. And it will be easier to improve such performance, because many high quality cables offer a very good resolution, many anti-vibration platforms might be used for a better imaging. I hate bright sound. Hissing sounds make me seek. In a real world and played by high quality systems sound of percussion cymbals is very energetic but it's not disturbing/aggressive. When they do focus listener's attention, when non-musical elements of the presentation steal the show, they automatically diverse listener's attention away from what is really important, from music – and that's not a good thing. Compromise should involve presenting bit less of this huge energy for a better coherence of the performance.

I don't like smeared, boomy bass. Despite the fact that I always choose warm sound over dry one, a boring presentation will always be boring to me. I won't complain much about such sounding product but I won't praise it either. That's why when above mentioned RT Audio Design amplifier offered a warm sound, remarkably resolving and with fantastic micro-dynamics I loved it. Such a presentation was highly satisfying and allowed me to enjoy my music collection.

I am not really interested in all these 'wars' between fans of tubes and solid-states, digital and analogue sources, large and small speakers – I am interested in the results of implementing one or the other technology and these differ from case to case. I know many expensive, but poorly performing tube components and as many remarkable solid-state ones. I know hopeless turntables that sound like CD Players from 1980ties, and CD Players sounding like high quality turntables. So I suggest that you keep an open mind – don't allow yourself to be biased by supposed superiority of one technology over the other; it all depends on the application of said technology. Obviously each of us has some preferences, and that's OK.

That's what this is all about, that's what true audio is: a fully aware perception of music with all of its richness coming from all elements involved - the notes, performance, recording process and reproduction by audio system. Only when all these elements come together, music is truly whole. It's a goal worth pursuing because life is short and there is so much wonderful music waiting for discovery. Once you discover it and want to listen to it do everything in your power to let it truly shine. I'd rather listen to one high quality recording played in a proper way than to fifty shitty ones. I hope I'm not the only one who thinks this way.

WOJCIECH PACUŁA
Chief editor