To address the issue of data sparsity and cold-start in recommender system, social information (e.g., user-user trust links) has been introduced to complement rating data for improving the performances of traditional model-based recommendation techniques such as matrix factorization (MF) and Bayesian personalized ranking (BPR). Although effective, the utilization of the explicit user-user relationships extracted directly from such social information has three main limitations. First, it is difficult to obtain explicit and reliable social links. Only a small portion of users indicate explicitly their trusted friends in recommender systems. Second, the "cold-start" users are "cold" not only on rating but also on socializing. There is no significant amount of explicit social information that can be useful for "cold-start" users. Third, an active user can be socially connected with others who have different taste/preference. Direct usage of explicit social links may mislead recommendation. To address these issues, we propose to extract implicit and reliable social information from user feedbacks and identify top-k semantic friends for each user. We incorporate the top-k semantic friends information into MF and BPR frameworks to solve the problems of ratings prediction and items ranking, respectively. The experimental results on three real-world datasets show that our proposed approaches achieve better results than the state-of-the-art MF with explicit social links (with 3.0% improvement on RMSE), and social BPR (with 9.1% improvement on AUC).