We introduce a class of semiparametric time series models (SemiParTS) driven by a latent factor process. The proposed SemiParTS class is flexible because, given the latent process, only the conditional mean and variance of the time series are specified. These are the primary features of SemiParTS: (i) no parametric form is assumed for the conditional distribution of the time series given the latent process; (ii) it is suitable for a wide range of data: non-negative, count, bounded, binary, and real-valued time series; (iii) it does not constrain the dispersion parameter to be known. The quasi-likelihood inference is employed in order to estimate the parameters in the mean function. Here, we derive explicit expressions for the marginal moments and for the autocorrelation function of the time series process so that a method of moments can be employed to estimate the dispersion parameter and also the parameters related to the latent process. Simulated results that aim to check the proposed estimation procedure are presented. Forecasting procedures are proposed and evaluated in simulated and real data. Analyses of the number of admissions in a hospital due to asthma and a total insolation time series illustrate the potential for practical situations that involve the proposed models.