The time-of-flight (TOF) feature of PET scanners has been used for a long time in PET reconstruction, but many implementational aspects are still incomplete or ambiguous in the literature. Here we formalize and present theoretical and practical implementation details for the reconstruction of clinical TOF histogram and list-mode data using ML-EM. Relevant aspects include the computation of the TOF component of the system matrix, the processing of TOF bins, the use of estimations of random and scattered coincidences, and differences between histogram and list-mode ML-EM TOF reconstruction. Several approaches and approximations have been implemented in the CASToR platform and compared for OSEM reconstructions of patient data from the GE Signa PET/MR scanner. Differences between implementations are not larger than the typical bias in clinical data reconstruction. The largest difference and contrast loss occur when the processing of histogram TOF bins is simplified, and list-mode reconstruction is most sensitive to the truncation of the Gaussian TOF probability distribution.