Introduction

Autism Spectrum Disorders (ASDs) DSM-5 criteria (American Psychiatric Association, 2013) include symptoms from one broad domains of deficits in social communication and social interaction which in DSM-IV (American Psychiatric Association, 2000) were two independent domains. Also, DSM-5 requires two symptoms of the last domain of restricted, repetitive patterns of behavior, interests, or activities with the inclusion of hyper- or hyporeactivity to sensory input or unusual interest in sensory aspects of the environmental. In DSM-IV-TR only one criterion was required and there was no consideration on sensory atypical interests.

Even though we have seen progress in timely recognition, diagnostic delay in countries with medium and low income persist (Montiel-Nava, Chacín, & González-Ávila, 2017; Tapia Guillen et al., 2019; Zavaleta-Ramírez et al., 2020). For example, in México, the prevalence in León, Guanajuato, was .87% (95% CI [.62, 1.1]) (Fombonne et al., 2016), consistent with other recent studies of prevalence (Pedersen et al., 2012). Fombonne et al. (2016) showed that most Mexican children (77.8%) were diagnosed after three years old, even when their parents detected the first symptoms before that age, and most families (61.2%) had earlier contact with health services. Some obstacles for recognizing early signs of autism are lack of training among the health care professionals and sub-optimal performance of the few valid screening instruments’.

Validity studies of the M-CHAT (Modified Checklist for Autism in Toddlers) (Robins, Fein, Barton, & Green, 2001), the SRS (Social Responsiveness Scale) (Constantino & Gruber, 2005), and the ABC (Autism Behavior Checklist) (Krug, Arick, & Almond, 1980) show cultural bias and inconsistent psychometric properties (Albores-Gallo et al., 2012) requiring adjustments in the cut-off scores (Fombonne, Marcin, Bruno, Tinoco, & Marquez, 2012; Living-Varela & Albores-Gallo, 2012). On the other hand, gold standards such as the ADI-R (Le Couteur, Haden, Hammal, & McConachie, 2008) and the ADOS-G (Lord, Rutter, Dilavore, & Risi, 1999) are valuable but expensive and hard to use in busy clinical settings.

CARS (Child Autism Rating Scale) Schopler, Reichler, DeVellis, and Daly (1980) is a diagnostic observational assessment that has consistently shown good psychometric properties. In the original validity study, the internal consistency was very high (Cronbach’s alfa α = .94), and the inter-rater reliability showed a concordance of .71, and the concurrent validity with the clinical assessment showed high correlation r = .84, p = < .001 and r = .80, p = < .001.

In 2013, Breidbord and Croudace (2013), performed a meta-analysis of the CARS validity studies published between 1980 and 2012. The inter-item reliability was estimated (k = .19, M = .896, IC 95% [.877, .913], Q = 99.66) as well as the inter-rater reliability measured by the Cohen’s kappa coefficient and the Cochran’s Q (k = .33, M = .796, IC 95% [.736, .844], Q = 183.36), and the test-retest (k = 0.8, M = .756, IC 95% [.690, .814], Q = 7.52).

In Mexico, CARS is the most used tool in centers providing therapeutic services for children with autism (Harris & Barton, 2016). Nonetheless, the Mexican version’s psychometric properties are unknown. This study aims to compare the instrument performance with the DSM-IV and DSM-5 criteria and investigate some of the psychometric properties of the Mexican version of the CARS (Childhood Autism Rating Scale) in a clinical sample of children and adolescents seeking attention in a child psychiatric hospital.

Method

Participants (N = 137) with an age range between 2 and 18 years seeking the diagnosis of Autism Spectrum Disorder in a Psychiatric Hospital. We excluded children with any genetic syndrome associated with Autism Spectrum Disorder.

Instruments

ADI-R (Lord, Rutter, & Le Couteur, 1994)

The Autism Diagnostic Interview, Revised (ADI-R), is a semi-structured interview administered by an experienced clinician for individuals with a mental aged above any genetic syndrome associated with Autism Spectrum Disorder. The interview assesses three domains of symptoms, consistent with the ICD-10 and DSM-IV criteria: language/communication, reciprocal social interactions, restricted, repetitive, and stereotyped behaviors and interests. It also evaluates the period between ages four and five (when symptoms are the most intense). It consists of 93 items, and its results are categorical. The inter-rater reliability was k = .63 to .89 for each item; the intraclass correlation coefficients (ICC) were above .92 for all the domains and subdomains. This study used the algorithm for current behavior, comprising four categories (A, B, C, D), which assess reciprocal social interaction, communication and restricted, repetitive, and stereotyped behavior patterns, and age of developmental abnormalities.

M-CHAT (Robins et al., 2001)

The M-CHAT is a modification to the Checklist for Autism in Toddlers (Baron-Cohen, Allen, & Gillberg, 1992). It consists of 23 dichotomous response items (yes/no) from the parent report. The original validity study used a sample of 1,293 children ages 16 to 30 months of the community. The discriminant function identified six critical items due to the higher coefficients of the discriminating function: items 2, 7, 9, 13, 14, and 15.

Total internal reliability and relation to the critical questions were adequate (α = .85 and α = .83, respectively). Failure in three of the 23 items of the M-CHAT or two from six critical items reported sensitivity of 87%, an specificity of 99%, a positive predictive value of 80%, and negative predictive value of 99%.

In Mexico, Albores-Gallo et al. (2012) validated the Mexican version with a mixed sample (clinical and community) of 456 children. The Mexican version of the M-CHAT’s internal consistency for the total items (23 items) was KR = .75 and .70 for the six critical items. The convergent validity of the M-CHAT total with the CBCL/1.5-5/PPD reported a Spearman coefficient rho = .66, p = .01; correlations with the ADI-R were within a range of rho = .23-.66. The criterion validity between the CARS and the ADI-R (dimensions A, verbal B, and non-verbal B, and C) was k = .17 to .61. High concordance with the non-verbal B dimension was found, k = .61 p = .0001, contrasting with the verbal dimension k = .29 p = .004.

ABC (Krug, Arick, & Almond, 1993)

The Autism Behavior Checklist (ABC) assesses individuals from 18 months to 35 years of age. It explores 57 atypical behaviors or items organized in five subscales: sensory stimuli, relationships, the use of the body and objects, language, and socialization/self-help. All patients had completed the ABC at the moment of the study, so we used the ABC’s sensory items to match the DSM-5 criteria. The selected items of the sensory stimuli subscale were: 1. Misuse of visual discrimination when learning (attention is fixed on parts of the objects, such as size, color, position), 2. Appears not to hear (despite normal hearing tests), 3. Sometimes the individual will not show “startle response” to loud noises, 4. The individual will sometimes not react to painful stimuli (cuts, pinches, contusions), 5. The individual will often do not blink when bright lights are directed to the eyes, 6. Will cover his/her ears to many sounds, 7. Will frown, squint or cover eyes to natural lights, 8. Will frequently not visually react to “new” people, and 9. Stares at no particular object for long periods. The internal consistency reported for this subscale is an alpha coefficient of .59.

Childhood Autism Rating (CARS) (Schopler, Reichler, & Renner, 1988)

CARS is an observational diagnostic instrument that assesses symptoms by comparing a child likely to be autistic with a neurotypical individual of the same age. The behavior observed in different settings; during a clinical evaluation or in the classroom, through videos recorded by the parents, or by a combination of resources. The scale consist of 15 items: - Relationship with people, mimicking behaviors, emotional response, use of own body, use of objects, adaptation to changes, visual response, auditory response, response to and use of taste, smell, and touch, fear or nervousness, verbal communication, non-verbal communication, activity levels, level and consistency of intellectual response, general impression.

The scores range from one (the individual behaves within normal limits for the age group) to four (the individual’s behavior is severely abnormal for the age group). Mid-point scores are assigned (1.5, 2.5, and 3.5) when the assessed behavior falls between two items. Results reflect normal to severely abnormal punctuation in a continuum. A cut-off score of 30 or above classifies as autism; the range from 30 to 36.5 defines mild to moderate autism, and 37 to 60 defines severe autism. The original study published in 1980 (Schopler et al., 1980) reported internal consistency of α = .94 for the instrument’s total. Inter-rater reliability demonstrated a concordance of .71. Concurrent validity between the instrument and the clinical judgment were between r = .84, p = < .001 and r = .80, p = < .001.

Procedure

A convenience sample of patients seeking a diagnosis for autism in the Children’s Psychiatric Hospital Juan N. Navarro clinic (Hospital Psiquiátrico Infantil Dr. Juan N. Navarro). Initially, the parents of the children completed the M-CHAT and the ABC instruments. Trained professionals conducted a semi-structured clinical interview based on the DSM-IV criteria (gold standard), and the ADI-R (gold standard) interview was completed in approximately 3.5 hours by an experienced child psychiatrist.

Researchers blinded to the ADI-R, M-CHAT, and the DSM-IV clinical interview results completed the CARS.

We used ADI-R and the DSM-IV interview results to assign patients to three groups: autism, asperger’s syndrome, or pervasive developmental disorder not otherwise specified (PDD-NOS).

Statistical analysis

The variables were expressed as means, standard deviations, and proportions. Reliability and internal consistency were analyzed using Cronbach’s alfa coefficient for the total sample and the subsamples by gender.

Concurrent validity was calculated with Spearman’s Rho coefficients of correlation between the CARS (total score) and the clinical diagnosis score, using DSM-IV criteria.

CARS Concordance with the DSM-IV and DSM-5 criteria

To infer the DSM-5 diagnosis we combined the DSM-IV domains of social interaction and communication deficits to match the DSM-5 social communication domain. We selected the ABC items that best-represented hyper or hypo activity sensory stimuli or hypersensitivity and added restrictive and repetitive behavior to match the DSM-5 criteria.

1. Sometimes an individual will not show “startle response” too loud noises; 2. the individual will sometimes not react to painful stimuli (cuts, pinches, contusions); 3. the individual will frequently not blink when bright lights are directed to the eyes; 4. will cover ears to many sounds; 5. will frown, squint or cover eyes to natural lights; 6. will touch, smell, or taste close objects.

We used the kappa coefficient to calculate the concordance rate of DSM-IV and DSM-5 diagnosis with the CARS (cut-off = 33).

All the instruments’ information was gathered in an Excel database and then transferred to the SPSS statistical program, version 17.0.

Discriminant validity

To study the instrument’s discriminant validity, we conducted a discriminant analysis with all the CARS items to analyze which items predict the membership to the diagnostic categories of Autism, Asperger, and PDDNOS.

Ethical considerations

This study received approval from the ethical committee of the institution (Hospital Psiquiátrico Infantil Dr. Juan N. Navarro) with the following ID number II3-03-1112.

Results

Sample characteristics

An experienced child psychiatrist assessed 137 children; we eliminated 13 cases from the analysis because they were incomplete. The final sample consisted of children (N = 124), 78.8% (N = 99) male. The children’s average age was 8.5 (SD = 3.8), ranging from ages 2 to 18 years. The mean age of fathers was 37.9 (SD = 7.8), and the mean age of mothers was 34.2 (SD = 7.2). The total mean score of the CARS was M = 32.3, SD = 7.3, ranging between 14 and 50.

Internal consistency

Internal consistency for the total items analyzed by Cronbach’s alfa coefficient was α = .88. By gender, internal consistency and mean score for males (N = 79) was α = .87, M = 50.9, SD = 13, and for females (N = 20) was α = .90, M = 49.8, SD = 15.8.

Concurrent validity

Spearman’s Rho coefficient between the CARS total score and the DSM-IV diagnosis criteria demonstrated moderate correlation (r^s = .62, p < .0001).

Convergent validity

Spearman’s coefficients of correlation between the CARS (total score) and the ADI-R subscales are shown in Table 1.

view

The highest correlation between the CARS total score and the ADI-R was r^s = .61 for the B domain of the communication. The lowest correlation was r^s = .32 for the C domain of restricted, repetitive, and stereotyped behavior patterns.

The kappa coefficient between the categorical diagnosis of the CARS (cut-off score of 33) and the M-CHAT (cut-off: 2/6 critical items) demonstrated a kappa coefficient of k = .66, p = < .001.

Concordance rate with DSM-IV and DSM-5 diagnosis criteria

The concordance rate between the CARS and the categorical diagnosis by DSM-IV and DSM-5 was 71% and 84.5%, respectively (Table 2).

view

Discriminant analysis

A discriminant analysis was conducted to identify which CARS items could better predict the membership to the diagnostic groups of Autism, Asperger’s Syndrome, and PDD-NOS. Two discriminant functions were calculated, with two canonical correlations, r = .627 and r = .563; both functions explain the 58.2% and the 41.8% of the variance with effect size of 1.0 and .31, respectively. For the combination of both functions (with all the items) the Wilks’ lambda was significant Λ = .415, χ² = 71.282, p < .0001; when removing the first function, the Wilks’ lambda remained significant Λ = .683, χ² = 30.887, p = .006, so both functions are relevant.

The centroid positions are shown in Figure 1 / Table 3. The CARS sensitivity analysis for diagnosing the subgroups shows that CARS discriminates between Autism and PDD-NOS, correctly identified 97% and 70% of the cases, respectively. However, its effectiveness in assessing Asperger’s Syndrome is lower, classifying only 12.5% of the cases (Table 3).

view

view

For the second function, only the intellectual response level and consistency, adaptation to changes, non-verbal communication, and relationship with people were significant (Table 4).

view

Discussion and conclusion

In this study, we analyzed the CARS Mexican version’s psychometric properties as an observational instrument for diagnosing autism.

The sample was predominantly male (78.8%), which is consistent with the higher ratio between male and females and with most of the reliability and validity studies from countries such as Japan (77.2% and 75.8%) (Kurita, Miyake, & Katsuno, 1989; Tachimori, Osada, & Kurita, 2003), United States (82%) (Sturmey, Matson, & Sevin, 1992), and Spain (74.5%) (García-Villamisar & Muela, 2000).

The instrument showed excellent internal consistency for the total of items, demonstrated with a Cronbach alfa coefficient α = .87. This result is higher than coefficients reported for children between ages two and six years, α = .79, and teenagers 13 to 22 years, α = .73 in the US (Garfin, McCallon, & Cox, 1988), and those reported in India, α = .79 (Russell et al., 2010), and Brazil, α = .82 (Pereira, Riesgo, & Wagner, 2008), both with mixed samples of children and adolescents. Our coefficient is lower than the reported in the original validity study (α = .95), but similar to studies in Korea α = .87 (Shin & Kim, 1998), and Spain α = .89 (García-Villamisar & Muela, 2000).

Most of the validity studies showed excellent internal consistency with Cronbach coefficients similar to the original study, α = .94 (Tachimori et al., 2003; Nordin, Gillberg, & Nydén, 1998; Saemundsen, Magnússon, Smári, & Sigurdardottir, 2003); only Akoury-Dirani, Alameddine, and Salamoun (2013), and Polaino-Lorente, García-Villamisar, and Muela (1991), reported higher Cronbach coefficients, α = .95 and α = .98, respectively.

Convergent validity

We demonstrated CARS has convergent validity with DSM-IV criteria through a high correlation coefficient Rho = .62 and k = .33, sensitivity (58.1%), specificity (76.6%), and concordance (71%). Our results are lower than others like Chlebowski, Green, Barton, and Fein, (2010), who reported k = .57, and Perry, Condillac, Freeman, Dunn-Geier, and Belair, (2005) who showed a concordance rate (88%). These results show good agreement with DSM-IV.

We also demonstrated evidence for convergent validity between CARS and ADI-R. The highest correlation with the ADI-R and domain B (r^s = .61) and moderate for the domains A, B, and C (r^s = .50). Our results are similar to those of Saemundsen et al. (2003), who showed a moderate correlation with domain B (.60), but, contrary to our results they showed a higher correlation with domain A (.81). Also, they found a 66.7% concordance using a cut-off CARS score of 30 with k = .40. Other studies compared the autism disorder diagnosis through the ADI-R and the CARS categorical diagnosis, reporting concordance of 85.7%, with k = .36 (Ventola et al., 2006), and k = .095, p = .486 (Rellini, Tortolani, Trillo, Carbone, & Montecchi, 2004).

Differences in the instrument administration, as a structured interview with parents as informants or as an observational tool to code the patient behavior, can explain low correlations between instruments (Saemundsen et al., 2003; Ventola et al., 2006; Pilowsky, Yirmiya, Shulman, & Dover, 1998). For the same reason, higher kappa values k = .55 and k = .60 are achieved when comparing the CARS and ADOS (both observational instruments) (Chlebowski et al., 2010; Ventola et al., 2006).

Our study demonstrated convergent validity through a high concordance of CARS and the DSM-5 (84.5%), high specificity (90.5%) but low sensitivity (46.7%), and low kappa (.36). These results are inconsistent with other studies showing better kappa coefficient such as Mayes et al. (2014) and Dawkins, Meyer, and Van Bourgondien (2016) and high levels of sensitivity (ranging from 81% to 100%) and specificity (ranging from 70% to 100%) for the CARS when using the DSM-5 compared to DSM-IV criteria. However, both studies were prospective, as Dawkins et al. (2016) included adults, and clinicians were not blind to the clinical diagnoses.

Overall our results show that even though CARS was developed before the DSM classification, it has convergent validity with both DSM-IV and DSM-5 classifications and provides evidence for an increased specificity with DSM-5 compared to the DSM-IV, which was a primary goal of the recent DSM-5 classification.

The discriminant analysis demonstrated the instrument better performance identifying autism and PDD-NOS groups. However, it is less useful in identifying the Asperger’s syndrome. This result is consistent with Rellini et al. (2004) and García-López and Narbona, (2014) as it shows that the CARS did not discriminate between children without autism and children with Asperger’s syndrome and a has a less discriminant power for PDD-NOS. However, the analysis of discriminant functions’ centroids of our study supports the capacity to discriminate between autism and PDD-NOS, reporting that three items make the difference: non-verbal communication, level, and consistency of intellectual response and relationship with people.

Some authors have demonstrated that the total score can help distinguish between groups. For example, Chlebowski et al. (2010) demonstrated that children with classic autism scored higher than the PDD-NOS group M = 35.1, SD = 4.2 and M = 29, SD = 4, respectively. In turn, Tachimori et al. (2003) suggest that cut-off scores of 27 and 27.5 are better discriminating PDD-NOS from Asperger’s syndrome, with a sensitivity of .65 and .71, and a cut-off CARS score of 30 to 30.5 differentiates autism well from PDD-NOS, with a sensitivity of .68 and .75 for each, for adolescents and adults. Mayes et al. (2009) suggested that a cut score of 25.5 is useful discriminating high functioning autism, with 95% sensitivity and 97% specificity.

A limitation of our study was its retrospective design, so the generalizability of the results is not possible.

In addition, the lack of IQ for the participants, decreased our ability to study the discriminant validity between severe autism and intellectual disabilities. Future studies should study the interobserver reliability.

This study shows the validity evidence of the Mexican Spanish version of the CARS, which is the most widely used instrument in Mexico, making this scale a valuable tool in the often-busy clinical environment. A new version of CARS2 (Schopler, Van Bourgondien, Wellman, & Love, 2010) is available and has the same items and scoring standards as the original CARS (Schopler et al., 1988). Also, Sanchez and Constantino (2020) provided a protocol to standardize the observation showing incremental validity of the instrument making it suitable for telehealth applications with potential use for assessing children in remote areas or during the pandemia of COVID-19.