Best attention tests? 

Ga HIER naar de Nederlandse versie van deze pagina

On this page there will be an extensive review of several attention tests available. Using a new evaluation system, you can more easily compare the attention tests and decide which ones are better suited to your needs. 

This review will be expanding every now and then, to discuss as many attention tests as possible. 

The new evaluation system can be found on this page: A new evaluation of neuropsychological tests. Enjoy the reading!

Attention tests: a short introduction

Most attention tests require that you have to discover something between quite a few (distracting) stimuli. For example, finding 4 dots in groups of 3 or 5 dots, or discovering the number 4 among all the other numbers (0-9) shown one after the other on a computer screen. In such tests you have to focus or concentrate on 1 task and try not to be distracted by other tasks. In this sense, attention is similar to the word 'focus' or 'concentration'. 

These attention tests are designed to examine concentration (see also my page on Attention Problems and an explanation of the different types of attention, coming soon). Unfortunately, such tests do not really represent daily life activities and it is therefore questionable how accurately your concentration skills are measured.

However, one thing is quite certain: attention tests certainly require the necessary concentration, probably much more than with activities in daily life.

Test of Selective Sustained Attention (TOSSA)

The first attention test that I have to review here, to be very honest, is no surprise because that is the test I, as the smallest test publisher in the Netherlands, have released for almost 15 years now. It is not only that I am a bit biased here, but I do hope to explain why this is a very good and useful attention test in a clinic. 

The TOSSA is an 8-minute auditory selective sustained attention test in which you will hear 240 groups of 2, 3 or 4 beeps. You must concentrate on the 3 beeps and always respond by pressing a space bar when you hear the 3 beeps. Reaction times are not measured (that cannot be accurately done with the help of a space bar), but your reactions are stored of course. You can respond correctly to 3 beeps, you can miss them, but you can also press incorrectly on 2 or 4 beeps.

Responding incorrectly is seen as not concentrating properly on what to do. Failure to respond is also regarded as such. The test lasts long enough to detect the slightest concentration problems. And it's not too long so that patients don't get very frustrated. They usually find the test a bit annoying, but not to such an extent that they stop during the test.

The TOSSA is a so-called continuous performance test: you have to concentrate for several minutes in a row. Other good examples of such tests are the TOVA (Test of Variables of Attention), the IVA (Integrated Visual and Auditory CPT) and the Connors CPT (reviews of them will be forthcoming). They are all quite long, from 4 to 20 minutes.

The TOSSA differs from these CPTs in several respects. It is a very simple test: most healthy people score very high on the TOSSA (median is 93.6%). This suggests that thousands of healthy people are not needed for a large norm group, since almost all of them will have high scores.

The TOSSA is also one of the shortest CPTs: only 8 minutes. The TOSSA is also fully auditory, no visual concentration is needed. And the TOSSA does not measure reaction times. Unfortunately, the TOSSA is less well normed in healthy people, but it is very well normed in a large group of different neurological patients (n=1019). And the TOSSA is only suitable for Windows computers.

Psychometric properties of the TOSSA are sufficient and clinically relevant. The reliability of the main indices ranged from .88 (test-retest reliability) to .95 (internal consistency) in a sample of 94 patients and 9 healthy controls. The convergent validity is sufficient, ranging from .33 to .57. The divergent validity ranges from .17 to .45. 

The TOSSA is a completely standardized (computer test) with clear and short instructions to administer, the reactions are all automatically saved and test results are calculated reliably and quickly.

The TOSSA is also sufficiently patient friendly or clinically useful. Almost all patients can take the test without becoming too frustrated or irritated. The TOSSA can also be administered in one-handed patients and in severely visually impaired patients.

The costs of the TOSSA are comparable to those of other attention tests: 175 euros ex. VAT for 1 year licenses on 2 computers.

The TOSSA has already shown good sensitivity and specificity in several pilot studies and it can detect attention problems faster than some other (widely used) tests without higher false positives. It can also distinguish between different patient groups somewhat better than some other tests. The test is easily available via a download link from www.pyramidproductions.nl. If you want to know more about the TOSSA please click HERE for the manual of the TOSSA.

Below you can also find two figures: a TOSSA results screen in which the blue line represents the patient's concentration and a red line that shows the degree of distraction (the further away from 0, the more distracted). The 2nd figure shows what the patient (black dot) is compared with: there are 5 norm groups.

TOSSA_resultsScreen

Above the TOSSA results for a client. Further information to be found in the manual.

TOSSA_NormgroupsGraph

Above the TOSSA indices are compared to the 5 different norm groups for a fictive patient Kovas.

Critical evaluation of the TOSSA

Below you will find the new evaluation table for the TOSSA. As you can see the TOSSA gets a score of 29 out of 37. There are several points to improve, as in all tests. 

The TOSSA seems to be a good attention test. In particular, it is broadly standardized for different neurological patient groups, it is relatively short and it can be administered to severely visually impaired patients and also to patients with aphasia.

But of course several improvements are welcome: 

  1. The norming of healthy people is not optimal. More healthy people should do the TOSSA so that it can actually be shown that the test is very easy for healthy people, young or old. The age range should be expanded and enlarged. For example, it would also be interesting to administer the TOSSA to children, for example from the age of 7 to 16, capable of doing the test.
  2. The storage of the raw data could be expanded to an excel file to gather a lot of data for more research.
TOSSA_EvaluationTable

The Integrated Visual and Auditory Continuous Performance Test (IVA plus)

The IVA plus is an improved version of the IVA (www.braintrain.com), a continuous performance test (CPT) that uses both visual and auditory stimuli. It is a computer test and takes 13 minutes to complete. Despite the fact that a review in Strauss, Sherman and Spreen (2006) was rather critical, there are certainly some positive things to report about the IVA. Unfortunately I am not familiar with this test myself, so I have collected most of the criticism from books and summarized it here.

First of all, the IVA is readily available at www.braintrain.com. It is completely standardized. Moreover, it has clear potential to detect attention problems. Precisely because it lasts 13 minutes, it can test sustained attention (and fatigue). It is well thought out: it uses a varying Inter Stimulus Interval (ISI), which makes for varying demands on attention. A varying target-to-distracter ratio is also used to measure impulsivity and concentration more reliably.

However, there are also some points of criticism. As with the TOSSA, the large norm group of 1700 individuals is rather randomly collected, not stratified (unfortunately like in many neuropsychological tests). The age range is from 6 to 99 years, but this is not evenly distributed across all age groups.

Much more serious, however, is the lack of test-retest reliability of some very important indices such as the Full-scale Response Control Quotient and the Auditory Response Control Quotient (<.59). The Full-scale Attention Quotient fluctuates around .66 and .75. Unfortunately, this has been tested in only 70 (!) healthy volunteers, whereby it is often the case that these test-retest correlations are often lower in healthy subjects than in patients. These correlations may be higher in patient groups (this is certainly the case with the TOSSA). Other reliability analyzes are not mentioned in the manual.

Other important research on validity, in particular convergent and divergent validity, is not sufficiently mentioned in the manual. This lack of correlations between the IVA and other known attention tests is a serious shortcoming because it remains unclear to what extent attention problems can actually be detected with the IVA.

Based on how the test is composed, it is almost inevitable that the IVA has the potential to detect attention problems well. However, this must be determined by independent studies, especially in neurological patient groups. Because the TOSSA has tested such a large neurological group of patients, it would be interesting to compare the IVA with the TOSSA in the same group of patients.

A practical problem of the IVA may be its length of 13 minutes. However, the complete test administration with exercises and instructions takes up to 20 minutes and with neurological patients who are slower, possibly even more. This seems trivial but certainly hinders the use of the IVA in clinical settings where time pressure is often high.

Below you will find the new Evaluation table for the IVA plus.

IVA_evaluationTable

The Test of Variables of Attention (TOVA)

Another computerized continuous performance test is the TOVA (Test of Variables of Attention). As can be seen in the Evaluation Table below, the TOVA has some interesting advantages. It is therefore all the more surprising why it is not widely used in neuropsychological studies.

It is possible that practical considerations play a role here: the TOVA test takes at least 22 minutes. It comes in 2 versions: either a visual or an auditory form. The stimuli are very simple and the hardware requires a special button box that has to be linked to the computer so that 1ms measurements are guaranteed to be reliable. The trend that only network computers are installed in many hospitals and institutions suggests that the TOVA should be developed for such a network version.

Unfortunately, the norms of the auditory TOVA (TOVA-A) are only present for children (6 to 19 years, n=2551!), while there are norms for the visual version for both children and adults (n=1596).

The test construction has been nicely done, just like that of the IVA, because both commission errors (due to fast and target high-frequency condition) and omission errors (slow and target low-frequency condition) are scored. Reaction times per response are also measured.

The fact that the TOVA takes so long is both an advantage and a disadvantage. The advantage is that it is about the only test that can measure an endurance load, much better than many other much shorter attention tests (such as the Symbol Digit Modalities Test or the Trail Making Test). The downside is that quite a few patients may not be able to stick with the test to the end and become very frustrated with it as it is very boring. If a patient falls asleep halfway through or becomes irritable, the test results are much less reliable.

That does not mean that it would still be very interesting, just like with the IVA, if this TOVA is administered more often in brain injury patients. This test, together with, for example, the TOSSA, would be worthwhile in particular for milder brain injuries. What are the False Positive Rates? Because this also seems a bit high with the TOVA. The test can be used for visually impaired people, but there are no standards for adults.

Below you will find the new Evaluation table for the TOVA.

TOVA_EvaluationTable

The Test of Divided Attention (TODA)

This TODA test has been developed by myself, so again I am a bit biased here. The test has been around for some 12 years now but it has been a real challenge to norm it sufficiently. Nowadays, it still is being used in some clinics because I do think it is potentially a good attention test.

The TODA is a new attention test developed to measure divided attention: the ability to perform 2 or more things at the same time. There aren't many tests that measure this aspect of attention.

We have, of course, the PASAT (Paced Auditory Serial Addition Task), but there are serious problems with it, despite the fact that this test keeps popping up in publications. First of all, the PASAT is very difficult and frustrating for many neurological patients, but also for quite a few healthy people. Particularly if the shortest interstimulus interval times are used (1.2, 1.4), someone's frustration increases to such an extent that the test has to be stopped prematurely on a regular basis. The question then remains how reliable the results are.

Moreover, can a good distinction be made between healthy people and neurological patients? The test is also not available in a standardized form. Usually a tape is used, but voices change per tape and quite a few tapes are of inferior quality. The norms of the PASAT are questionable, often only a handful of healthy individuals or patients with brain injury, and there are serious test-retest effects: 18 to 20% progress (steep learning curve) can be made if the test is repeated for the 2nd time within a short period of time.

The TODA is designed as a better alternative to the PASAT. It is much more customer-friendly, both for patients and for the researcher/clinician. Of course, frustration can also increase with this more difficult test, but this remains within limits. Also and most importantly, the TODA is not very difficult for healthy people, not even for the much older healthy people. In this way, far fewer false positives are found, while the test is still able to detect subtle attentional problems. The test appears to be sensitive enough for brain injury and divided attention problems in particular.

The TODA is available as a very well standardized computer test (for Windows computers). It is easily available via www.pyramidproductions.nl but unfortunately only in Dutch. An English language version has still to be released.

The TODA itself takes 9 minutes, 11 minutes to take the test including exercise. Usually it is recommended to do the TODA after the TOSSA has been performed. If the TOSSA does not find concentration problems, then it may be that there are divided attention problems that can only be detected by the TODA. This often confirms the complaints that patients have.

The TODA requires a response to each presented stimulus, namely pressing 1 of the 3 arrow keys on the keyboard. The stimulus is always a sum (e.g. 2+3=5) vertically on the screen and a group of 2, 3, or 4 beeps. See the Figure below for a stimulus from the TODA.

If the sum is correct and 3 beeps are heard, press the left arrow key (with one hand, using 3 fingers as if playing the piano). However, if the sum is incorrect or there are 2 or 4 beeps, the middle arrow key must be pressed. The right arrow key is when both the beeps and the sum are wrong. After an exercise (maximum 2x practice), most patients understand the instructions and the real test administration can begin.

The TODA has 2 speeds: one fast, and one slow where the sequence of stimuli is exactly the same. This makes it easy to calculate how speed, duration and learning effects have an effect on the results. No response times are measured because they are unreliable via a keyboard. Only the number of responses is measured.

The TODA measures divided or varying attention. Possibly also mental flexibility because of constantly having to switch between answers. The test has a learning effect, but if there is a sufficient number of months (minimum 2) between the test administrations, this learning effect is probably not large or significant. This still needs to be investigated further.

TODA_stimulus

Above the stimulus screen of the TODA. A patient has to press a specific key to signal whether this sum is correct or incorrect, and the number of beeps (3) is correct.

The Trail Making Test (TMT)

The Trail Making Test is one of the most widely used tests in the Western world. If not the most used attention test. 

Probably not only because the TMT is very quick and easy to administer (within 5 minutes) but also because at least 46 standardization studies have been carried out. It has also reached the status of the Nintendo DS where it is included in the Brain Training game. This will make many people familiar with the test for a long time and this learning effect is not favorable for use as a neuropsychological test.

Although very popular in clinical practice and research studies, I certainly have reservations about this test. The TMT scores fair to good on reliability, standardization, availability and ease of use. This concerns the TMT-A and B version.

Reliability

The TMT seems reasonably reliable (test-retest reliability). In several studies in neurological patient groups, the reliability coefficient of TMT-A is somewhat lower than that of TMT-B, around .55 to .70. It varies quite a bit across populations and time intervals (as do all neuropsychological tests, really).

Validity

Several studies show a moderate correlation between the TMT and other concentration tests and tests that measure speed of information processing, such as the PASAT and the Symbol-Digit Modalities Test. Factor analysis studies reveal factors such as attention, visual search processes, speed of information processing, and executive control, all of which play a role in TMT. However, it is important which variable is taken as the most significant. Most studies use the TMT-B/TMT-A ratio as a valid measure of problems in executive and divided attention. However, it is not clear what the cut-off score should be (largely also dependent on age and level of education). Usually a ratio of 3 or higher is considered abnormal.

In my own study, in which I compare several tests such as the TMT, Stroop, d2, Digit Span and the newer tests TOSSA and TODA (see the Manual of the TOSSA test), it appears that the ratio score is not really sensitive. That is also my clinical experience with more than 3000 patients tested. More recent research suggests that the TMTB-TMTA difference score is a better and more valid measure of attention problems. This is also apparent in my study (for more details, see the TOSSA manual, which can be downloaded here for free).

Moreover, the TMT has serious validity problems when you look at its content. The time difference between A and B would only be pure if all other differences were not there. Both the length and spatial positioning of the stimuli differ between versions A and B, so that a comparison in seconds is not entirely fair. In particular, TMT-B can be more difficult because it also requires different motor and visuo-spatial skills than TMT-A.

Another serious problem of the TMT is that it cannot be used reliably in all patients: visually severely affected patient groups with, for example, visual field loss or visual neglect or visual agnosia cannot perform the test properly for that reason. People also need to know the alphabet and that is certainly not always the case. There are certainly test-retest effects with the TMT, so there will have to be several months between 2 measurements.

In summary: the TMT certainly says something about divided attention problems, speed of information processing and visual search processes. But when trying to detect attention problems, this test cannot be used exclusively. The chance of missing attention problems is then really too high, so that complaints from patients may be taken less seriously. The TMT could serve well as a rapid screening test if no other (better) attention test is available. But if the TMT finds no problems, I definitely recommend using other (better) tests with more sensitive attention tests, especially if patients have serious complaints that indicate attention problems.

Below you will find the new Evaluation table for the TMT.

TMT_EvaluationTable

Have A Great Story About This Topic?

Do you have a great story, remarks or any additions to or about this? One that could help other people as well and above all is constructive? Then please share it!

[ ? ]

Upload 1-4 Pictures or Graphics (optional)[ ? ]

 

Click here to upload more images (optional)

Author Information (optional)

To receive credit as the author, enter your information below.

(first or full name)

(e.g., City, State, Country)

Submit Your Contribution

  •  submission guidelines.


(You can preview and edit on the next page)

DISCLAIMER

I will not take any responsibility for how the information on this website will affect you. It always remains your responsibility to handle all information with care and in case of medical or mental problems you should ALWAYS consult a professional in your neighbourhood!

Ik neem geen enkele verantwoordelijkheid voor hoe de informatie op deze site u zal beïnvloeden. Het blijft altijd uw verantwoordelijkheid om al deze informatie zorgvuldig te bekijken. In het geval van lichamelijke en/of mentale problemen dient u ALTIJD een professional in uw directe omgeving te waarschuwen!