The CXCR4 receptor and its ligand CXCL12 (also named stromal cell-derived factor 1, SDF1) have a critical role in chemotaxis and homing, key steps in cancer metastasis. Although myofibroblasts expressing CXCL12 are associated with the presence of axillary metastases in HER2 breast cancers (BC), the therapeutic interest of targeting CXCR4/CXCL12 axis in the different BC subtypes remains unclear. Here, we investigate this question by testing antitumor activity of CXCR4 inhibitors in patient-derived xenografts (PDX), which faithfully reproduce human tumor properties. We observed that two CXCR4 inhibitors, AMD3100 and TN14003, efficiently impair tumor growth and metastasis dissemination in both Herceptin-sensitive and Herceptin-resistant HER2 BC. Conversely, blocking CXCR4/CXCL12 pathway in triple-negative (TN) BC does not reduce tumor growth, and can even increase metastatic spread. Moreover, although CXCR4 inhibitors significantly reduce myofibroblast content in all BC subtypes, they decrease angiogenesis only in HER2 BC. Thus, our findings suggest that targeting CXCR4 could provide some therapeutic interest for HER2 BC patients, whereas it has no impact or could even be detrimental for TN BC patients.