The visual occipito-temporal cortex is composed of several distinct regions specialized in the identification of different object kinds such as tools and bodies. Its organization appears to reflect not only the visual characteristics of the inputs but also the behavior that can be achieved with them. For example, there are spatially overlapping responses for viewing hands and tools, which is likely due to their common role in object-directed actions. How dependent is occipito-temporal cortex organization on object manipulation and motor experience? To investigate this question, we studied five individuals born without hands (individuals with upper limb dysplasia), who use tools with their feet. Using fMRI, we found the typical selective hand–tool overlap (HTO) not only in typically developed control participants but also in four of the five dysplasics. Functional connectivity of the HTO in the dysplasics also showed a largely similar pattern as in the controls. The preservation of functional organization in the dysplasics suggests that occipito-temporal cortex specialization is driven largely by inherited connectivity constraints that do not require sensorimotor experience. These findings complement discoveries of intact functional organization of the occipito-temporal cortex in people born blind, supporting an organization largely independent of any one specific sensory or motor experience.