This paper researches the application of grey system theory in cost forecasting of the coal mine. The grey model (GM(1.1)) is widely used in forecasting in business and industrial systems with advantages of minimal data, a short time and little fluctuation. Also, the model fits exponentially with increasing data more precisely than other prediction techniques. However, the traditional GM(1.1) model suffers from the poor anti-interference ability. Aimed at the flaws of the conventional GM(1.1) model, this paper proposes a novel dynamic forecasting model with the theory of background value optimization and Fourier-series residual error correction based on the traditional GM(1.1) model. The new model applies the golden segmentation optimization method to optimize the background value and Fourier-series theory to extract periodic information in the grey forecasting model for correcting the residual error. In the proposed dynamic model, the newest data is gradually added while the oldest is removed from the original data sequence. To test the new model’s forecasting performance, it was applied to the prediction of unit costs in coal mining, and the results show that the prediction accuracy is improved compared with other grey forecasting models. The new model gives a MAPE & C value of 0.14% and 0.02, respectively, compared to 1.75% and 0.37 respectively for the traditional GM(1.1) model. Thus, the new GM(1.1) model proposed in this paper, with advantages of practical application and high accuracy, provides a new method for cost forecasting in coal mining, and then help decision makers to make more scientific decisions for the mining operation.