Inhibition of Return (IOR) refers to slower reaction time to a target presented at the same location as a preceding stimulus. Here, we examine reflexive attention orienting via the saccadic IOR using a shift in gaze direction (i.e. from averted to direct) in faces presented as a peripheral cue, in upright and inverted orientations, in adults with Autism Spectrum Disorder (ASD) and typically developed comparison participants. While both groups showed an IOR in the inverted face condition, this effect was reduced in participants with ASD in the upright face condition, as compared to comparison participants, suggesting that moving eyes do not trigger reflexive exogenous orienting in individuals with ASD. Impaired reflexive orienting to eye gaze might severely compromise the later development of social functions in ASD, such as joint attention, face emotion recognition and mindreading.