Infants who underwent more exclusive and longer durations of breastfeeding have displayed greater cognitive abilities when tested as children and adolescents. However, this association between breastfeeding and brain development remains unclear. 85 typically developing children born to 83 mothers underwent multiple diffusion tensor imaging (DTI) scans which provided us with 331 datasets. Breastfeeding information was collected from the mothers at their child’s scan, and at 3, 6, and 12 months postpartum. The results showed that breastfeeding exclusivity and duration were associated with global and regional white matter microstructure, whilst controlling for perinatal and sociodemographic factors. Greater breastfeeding exclusivity showed sex differences between females and males. In conclusion, the findings suggest white matter microstructure differences associated with breastfeeding that differ by sex. Differences may come from the divergent development of white matter between males and females in early childhood and may indicate potential long-term white matter differences related with breastfeeding. Overall, this study can help us understand the importance of breastfeeding in developing healthy brain structures.