The gold standard material for bone regeneration is still autologous bone, a mesenchymal tissue that consists mainly of extracellular matrix (ECM) (90% v/v) and little cellular content (10% v/v). However, the fact that decellularized allogenic bone grafts often present a clinical performance comparable to autologous bone grafts demonstrates the crucial role of ECM in bone regeneration. For long, the mechanism by which bone allografts function was not clear, but recent research has unveiled many unique characteristics of ECM that seem to play a key role in tissue regeneration. This is further confirmed by the fact that synthetic biomaterials with composition and properties resembling bone ECM present excellent bone regeneration properties. In this context, ECM molecules such as glycosaminoglycans (GAGs) and self-assembly peptides (SAPs) can improve the performance of bone regeneration biomaterials. Moreover, decellularized ECM derived either from native tissues such as bone, cartilage, skin, and tooth germs or from cells such as osteoblasts, chondrocytes, and stem cells has shown promising results in bone regeneration applications. Understanding the role of ECM in bone regeneration is crucial for the development of the next generation of biomaterials for bone tissue engineering. In this sense, this review addresses the state-of-the-art on this subject matter.