Sleep disorders are among the most common comorbidities in children with Autism Spectrum Disorder (ASD), and subjectively defined sleep disturbances have been related to ASD symptom severity. However, no study has investigated the differential impact of objectively measured sleep and circadian rhythm disturbances on behavioral difficulties in this population. Fifty-two children with ASD aged 3–10 years underwent assessments of sleep and circadian rest–activity rhythms objectively with actigraphy and subjectively with the Children’s Sleep Habits Questionnaire. Behavioral difficulties were assessed using the ABC-C. Group comparison analyses were used to compare sleep and circadian rhythm parameters of children with higher and lower behavioral difficulties and dominance analysis to rank predictors and address multicollinearity. Children with high irritability had a shorter continuous sleep period compared to those with lower irritability (−60 min, p = 0.04), as well as those with high stereotypic behaviors compared to children with less stereotypies (−75 min, p = 0.006). Objective circadian and sleep disturbances accounted together for, respectively, 17%, 18% and 36% of the variance in social withdrawal, irritability and stereotypic behaviors. The identification of both sleep and circadian rhythm disturbances as explanatory factors for behavioral difficulties warrants their inclusion in the existing behavioral management strategies for children with ASD.