Human Respiratory syncytial virus (hRSV) and human metapneumovirus (hMPV) are the two major etiological viral agents of lower respiratory tract diseases, affecting mainly infants, young children and the elderly. Although the infection of both viruses trigger an antiviral immune response that mediate viral clearance and disease resolution in immunocompetent individuals, the promotion of long-term immunity appears to be deficient and reinfection are common throughout life. A possible explanation for this phenomenon is that hRSV and hMPV, can induce aberrant T cell responses, which leads to exacerbated lung inflammation and poor T and B cell memory immunity. The modulation of immune response exerted by both viruses include different strategies such as, impairment of immunological synapse mediated by viral proteins or soluble factors, and the induction of pro-inflammatory cytokines by epithelial cells, among others. All these viral strategies contribute to the alteration of the adaptive immunity in order to increase the susceptibility to reinfections. In this review, we discuss current research related to the mechanisms underlying the impairment of T and B cell immune responses induced by hRSV and hMPV infection. In addition, we described the role each virulence factor involved in immune modulation caused by these viruses.