We consider models based on multivariate counting processes, including multi-state models. These models are specified semi-parametrically by a set of functions and real parameters. We consider inference for these models based on coarsened observations, focusing on families of smooth estimators such as produced by penalized likelihood. An important issue is the choice of model structure, for instance the choice between a Markov and some non-Markov models. We define in a general context the expected Kullback-Leibler criterion and we show that the likelihood based cross-validation ($LCV$) is a nearly unbiased estimator of it. We give a general form of an approximate of the leave-one-out $LCV$. The approach is studied in simulation and illustrated by estimating Markov and two semi-Markov illness-death models with application on dementia using data of a large cohort study.