Forecasting the trend of tuberculosis incidence in Anhui Province based on machine learning optimization algorithm, 2013–2023

Abstract

Tuberculosis has been one of the most common communicable diseases raising global concerns. Accurately predicting the incidence of Tuberculosis remains challenging. Here we constructed a time-series analysis and fusion tool using multi-source data, and aimed to more accurately predict the incidence trend of tuberculosis of Anhui Province from 2013 to 2023. Random forest algorithm (RF), Feature Recursive Elimination (RFE) and Least absolute shrinkage and selection operator (LASSO) were implemented to improve the derivation of features related to infectious diseases and feature work. Based on the characteristics of infectious disease data, a model of RF-RFE-LASSO integrated particle swarm optimization multiple inputs long short term memory recurrent neural network (RRL-PSO-MiLSTM) was created to perform more accurate prediction. Results showed that the PSO-MiLSTM achieved excellent prediction results compared with common single-input and multi-input time-series models (test set MSE:42.3555, MAE: 59.3333, RMSE: 146.7237, MAPE: 2.1133, R²: 0.8634). PSO-MiLSTM enriches and complements the methodological research content of calibrating the time-series predictive analysis of infectious diseases using multi-source data, and can be used as a brand-new benchmark for the analysis of influencing factors and trend prediction of infectious diseases at the public health level in the future, as well as providing a reference for incidence rate prediction of infectious diseases.

Introduction

Tuberculosis (TB), an infectious disease of the lungs caused by Mycobacterium tuberculosis, is a common infectious disease worldwide. According to the World Health Organization (WHO), the global incidence of TB decreased by 1.5% between 2000 and 2015, but 200 million people are still infected with Mycobacterium tuberculosis, and 1.5 million people die from TB every year [1]. Mathematical models or artificial intelligence models can effectively predict the future epidemiological trends of infectious diseases and develop appropriate prevention and control measures based on the results [2, 3]. Time series methods are often used in previous studies to predict the incidence of infectious diseases. Among them, linear methods are represented by seasonal autoregressive integrated moving average (SARIMA) [4], and nonlinear methods are represented by non-autoregressive (NAR) [5], long short-term memory (LSTM) [6], and recurrent neural network (RNN) [7], recurrent neural network (long short term memory, LSTM) [6] and nonlinear autoregressive with exogenous input (NARX) [7]. These models usually utilize historical incidence data to make predictions of future incidence and involve the developmental patterns of infectious diseases themselves.

However, the factors affecting the incidence of infectious diseases are complex, and their correlation characteristics involve multiple aspects. Usually, the outbreak of an infectious disease will lead to an increase in news media coverage of the type [8], fluctuations in the stocks of companies or sectors associated with it [9], and even some changes in the socio-economic structure [10]. Studies have shown that changes in climate and pollutant indicators also have a complex impact on the development of infectious diseases [11, 12]. With the development of satellite remote sensing and Internet technologies, it has become possible to access and utilize multi-source data related to infectious diseases. The effective utilization of these data has a high application value and helps to improve the effectiveness of early warning research on infectious diseases. Currently, some previous studies have successfully applied Internet data in the prediction of infectious diseases and revealed the precedence and relevance of Internet data in the prediction of infectious diseases [13, 14]. Therefore, integrated utilization of multi-source data can significantly improve the effectiveness and accuracy of prediction models.

Feature engineering is a necessary step to ensure the accuracy of machine learning methods, and feature derivation, feature extraction, and feature screening can effectively solve the problem of mismatch on the time scale of variables. However, feature extraction categories are complex and feature screening methods are diverse, and choosing appropriate methods to screen out independent variables with large influence factors is one of the difficulties in establishing efficient prediction models [15, 16]. On the one hand, in order to make full use of the data, in addition to the commonly used statistical features (mean, extreme deviation, etc.), the features in the time domain and frequency domain of the time series data are also extracted [17, 18]; on the other hand, for the characteristics of the derived features with high dimensionality, the feature screening study is carried out. Recursive feature elimination method eliminates weak features by repeatedly training the model to avoid overfitting problem and improve the generalization ability of the model [19]. Random forest and LASSO algorithms also have unique advantages in feature screening [20, 21]. By comparing the performance of the algorithms and targeting the characteristics of the variables, we adopt the fusion of the above three methods for screening in order to determine the input variables of the model, aiming to improve the accuracy of the model prediction.

In China, tuberculosis belongs to one of the higher categories of statutory infectious diseases and has obvious time series characteristics [22]. Therefore, based on the TB dataset in Anhui Province, the data structure, input method, feature engineering and parameter optimization of the incidence prediction model were explored for validity and versatility, and innovations were made in the following aspects: (1) Multi-source data can be fully explored for the factors related to the development of infectious diseases, to correct the prediction results of infectious disease data and to improve the model’s prediction ability for the perturbation part. (2) By performing comprehensive feature derivation and adopting multiple methods for feature screening, consistency on the time scale of variables is accomplished while achieving full utilization of data. (3) The constructed PSO-MiLSTM model obtains better disease prediction results, which can provide a reference basis for the prediction of the incidence of various infectious diseases.

Methods and materials

Figure 1 shows the technical route of the data and methods in this study.

Fig. 1

Full text technology roadmap

Data Sourcing and Cleaning

The data on the number of monthly incidence of tuberculosis in Anhui Province were obtained from the Chinese Center for Disease Control and Prevention (CCDC), the Internet search index was obtained from the Baidu search engine, the data on atmospheric pollutants were obtained from the official website of the China Environmental Monitoring Station (CEMS), the data on climate were obtained from the National Meteorological Information Center (NMIC), and the data on stock fluctuations were obtained from the Flush Data Center. The overall time span of the data is from December 2013 to October 2023, a total of 119 months, covering 35 initial variables, mainly through the database for keyword co-occurrence analysis, expert consensus and experience to determine the keyword information of high relevance to the infectious diseases to be analyzed. Stock information is selected primarily from local comprehensive sector indices. Atmospheric pollutant data and climate data are from NEPA and Meteorological Agency respectively, incorporating the specific information of the data as shown in Table 1.

Table 1 General information of the experimental datasets

All the research data have been reasonably collected,

Among them, pollutant information and search index have some missing data, and the percentage of their missing is less than 0.1%, which meets the inclusion criteria, and the simulation filling comparison study was carried out by using mean filling, median filling, proximity filling, and linear interpolation, and it was finally concluded that using the proximity of the backward filling has the smallest impact on the sequence characteristics, so the proximity filling was used to fill the missing data of the pollutant and search index sequences in data cleaning [23], and cross-validation was used to further explore the stability of the data after filling. Therefore, proximity filling is used in data cleaning to fill in the missing data of pollutant and search index series, and cross validation is used to further explore the stability of the filled data. When matching the data, in order to achieve the purpose of early prediction, the independent variable is the previous 1 month data corresponding to the dependent variable. For example, when predicting the number of tuberculosis cases in October 2023, the input data of the independent variable is actually the index of Baidu in September 2023 and other indicators.

Feature extraction and screening

As can be seen from Table 1 and Table 2, there is a mismatch in the time scale of the incorporated features, and the features present more complex change patterns and trends. Through time series feature extraction, patterns and regularities in the data, such as trends, periodicity and noise, can be found. The accuracy and reliability of the algorithm can be significantly improved by applying the extracted information features as independent variables in the model [24]. In this study, 21 statistical indicators such as the mean, variance, and waveform factor of each underlying variable are mainly extracted.

Table 2 Basic features and indicators for extracting information

After feature extraction, the number of features will be significantly increased, if directly input into the prediction model there will be data imbalance phenomenon, thus increasing the risk of overfitting. Therefore, feature screening is an essential part of the process, and commonly used feature screening methods include: Recursive Feature Elimination (RFE), Random Forest and Lasso regression (see Fig. 2).

Fig. 2

Flowchart of feature extraction and filtering

Recursive Feature Elimination (RFE) is mainly used to select the optimal subset of features by recursively training the model and eliminating the features. The method trains the model with the current optimal feature subset during each iteration and evaluates the model performance on new data. Then, based on the model performance and the importance of the features, the least important features are selected and removed from the feature set. This process is repeated recursively until a predefined stopping condition is met, such as reaching the maximum number of iterations or reaching the desired number of features [25].

Random forest is an integrated learning method that improves prediction performance by constructing multiple decision trees and combining their predictions. In random forest, feature selection is an important step that helps us to find the features that have the greatest impact on the prediction results. For each decision tree, during the construction process, Random Forest will use the self-help method (bootstrap) to extract samples from the original dataset, and then perform feature selection for each sample; after the training of all the decision trees is completed, Random Forest will use the voting method (voting) to synthesize the prediction results of each decision tree [26].

Lasso regression is a classical feature selection and regression analysis method, which achieves feature selection by adding an L1 regular term to the loss function so that some coefficients are close to zero. Similarly, for feature selection, Lasso regression is able to deal with the problem of multicollinearity, and Lasso regression is interpretable and can output the importance of each feature. It can also fulfill its unique advantages when dealing with large-scale datasets and high-dimensional data [27, 28].

Model building and optimization

There are more method models for forecasting time series data, the common ones are STL model, ARIMA model, and Holt-Winters model, etc. [29] With the rise of machine learning, more and more machine learning models are also used to forecast and analyze the time series data, and better results have been achieved [30].

SARIMA (Seasonal Autoregressive Integrated Moving Average) model is a classical model used for time series forecasting, which is composed of seasonal autoregressive model (SAR), differential integration (I), and moving average (MA).The SARIMA model is mainly used to forecast the time series by capturing the seasonality and trend to predict future values.The parameters of SARIMA model can be determined by autocorrelation function (ACF) and partial autocorrelation function (PACF) [31]. Its basic form can be expressed as:

$$\text{SARIMA }(\text{p d q})(\text{P D Q}{)}_{\text{s}}$$

Model is a time series prediction model based on neural network, which was proposed by Japanese scholar Nishiguchi Taro in 2017. The neural network model combines neural network and autoregressive model (AR). The neural network is used to learn the latent structure of time series, and the AR model is used to control the long-term dependence of time series. The main idea of the model is to decompose the time series data into three parts: trend, seasonality and noise, and then use the neural network to learn the pattern of trend and seasonality, and use the autoregressive model to model the noise component [32]. Based on the neural network model, for multivariate prediction, the researchers constructed a neural network model, Combining the external input with the internal state to predict the output at the next time point, the neural network model model is usually used to solve the system problem with nonlinear dynamic characteristics, has good generalization ability, and can handle complex time series data. [33]. Long short-term memory) neural network is a special neural network (recurrent neural network, recurrent neural network) structure, First proposed by Hawklett and Schmidhuber in 1997, long short-term memory has better long-term dependence characteristics than traditional neural networks, and can effectively deal with long-term dependence in time series data (see Fig. 3). The main features of the least square method include: 1. Memory cell: Memory cell contains a special structure called “memory cell,” which is used to store and update the information about the sequence; 2. Gated structure: The system contains three gates (input gate, forget gate and output gate), which are used to control the flow of information and memory update; 3. Vanishing gradient problem: By introducing memory unit and gating structure in the least square model, the vanishing gradient problem in the network is effectively solved, thus improving the training effect of the model [34].

Fig. 3

Schematic diagram of the LSTM model

Particle Swarm Optimization (PSO) is an optimization algorithm based on the behavior of a flock, first proposed by Eberhart and Kennedy in 1995. The algorithm is inspired by the foraging behavior of bird flocks and finds the optimal solution by simulating the interaction of individuals in the flock. The particle swarm optimization algorithm considers both global and individual optimal solutions. Each particle uses the information of both the global optimal solution and the individual optimal solution during the updating process, which enables it to converge to the optimal solution faster. In addition, exploration and convergence are balanced by inertia weights and learning factors, thus maintaining the diversity of the population during the search process. This helps to avoid the algorithm from falling into local optimal solutions. Particle swarm optimization algorithms have been widely used in a variety of optimization problems, such as function optimization, constrained optimization, combinatorial optimization, and machine learning [35].

Results

In the results section, the analysis is first carried out to evaluate the model performance by univariate autoregressive modeling. The model evaluation indexes mainly include MSE (Mean Square Error), MAE (Mean Absolute Error), RMSE (Root Mean Square Error), MAPE (Mean Absolute Percentage Error), and R² (Coefficient of Determination), and the smaller the first four items are, the better the model performance is, and the closer the R² is to 1, the better the model prediction performance is.The evaluation of the results is based on the size of the R² value of the test set as the main basis, and the rest of the test set indicators as the secondary basis.

Univariate time series modeling

After collecting data on the number of TB cases from December 2013 to October 2023 in Anhui Province, the serial data were modeled using the SARIMA model as well as the NAR model, respectively. The model was constructed using the data from December 2013 to October 2022 to predict the number of monthly incidence from November 2022 to October 2023, and the results were compared with the real results, which are shown in Fig. 4 and Table 3.

Fig. 4

Model fitting results (SARIMA, NAR)

Table 3 Model evaluation indexes

Among them, according to the autocorrelation and partial autocorrelation regression results, using the AIC value to compare with the BIC value, the final will of the SARIMA model has the p, d, and q of 1, 0, and 1, respectively, and the values of P, D, and Q of 1, 1, and 1, respectively. the NAR model has the step size of 2, the number of neurons of 20, and the trainFcn is set to Levenberg–Marquardt backpropagation.

From the above, it can be seen that both the SARIMA model and the NAR model have poor predictive performance, and the inclusion of multivariate for infectious disease analysis is an important part to improve the accuracy of the model.

Evaluation of feature screening methods

As shown in Table 2, 21 monthly statistical features can be extracted for each base variable to achieve consistency between the independent and dependent variables on the time scale. After incorporating 35 basic variables for feature extraction derived as 735 features, 12 important variables are screened out through RFE-RF method, and the data set named S1 is constructed with the number of morbidity; 9 important variables are screened out by using LASSO, and the data set named S2 is constructed; the intersection variables of the two screening processes are 5, and the S3 data set is constructed with the dependent variable; and the concatenated set of variables, totaling 14, is formed. S4 dataset is formed. The NARX model is used to predict and analyze these four data sets and compare the final results. The division of the data sets is consistent with 2.1, and the model parameters are adjusted by using five-fold cross-validation. In order to facilitate the adjustment of model parameters, the data from November 2021 to October 2022 were set as the validation set data, and the model parameters were adjusted using the validation set model evaluation indicators. The results are shown in Fig. 5 and Table 4.

Fig. 5

Model fitting results of each dataset

Table 4 Model evaluation indexes

The step size of the NARX model is 12, the number of Shenjing elements is 50, and trainFcn is also set to Levenberg–Marquardt backpropagation. The results of correlation analysis show that the S4 dataset has the best effect, and the corresponding variables are 14. The results of correlation analysis show that the correlation between variables is small and there is no collinearity (see Fig. 6).

Fig. 6

Results of correlation analysis

Number indicates the number of TB cases, and the rest of the features are expressed using A-B, where A indicates the specific indicator and B indicates the extracted information, e.g. BCG-RMS indicates the valid values extracted from the BCG monthly index data

Multi-input time series modeling

Using the S4 dataset constructed in Part 2.2 for modeling analysis, PSO-NARX, MiLSTM and PSO-MiLSTM were constructed respectively. The training set, validation set and test set are divided in the same way as in Sect. “Feature extraction and screening”, and the results of the validation set are used as the objective function of the optimisation algorithm. The result of five-fold cross-validation was used as the objective function. The results are shown in Fig. 7 and Table 5.

Fig. 7

Fitting results of each model

Table 5 Evaluation indexes of each model

After combining the intelligent algorithm for hyper-parameter optimisation, the complexity of MiLSTM model parameter adjustment can be effectively solved through cross-validation, and PSO-MiLSTM achieves the most excellent prediction performance, with step size by 12, maximum number of iterations by 236; initial learning rate by 0.001; learning rate decline factor by 0.01. The model test set MSE is 42.3555, MAE is 59.3333, RMSE is 146.7237, MAPE is 2.1133, and R² is 0.8634.

Discussion

The spread and development of infectious diseases can lead to serious health problems and death. For example, the new coronavirus (COVID-19) can cause serious diseases such as pneumonia, acute respiratory distress syndrome, multi-organ failure, and even death. It can also have an impact on many aspects such as socioeconomics, public safety, education, and tourism [36]. Mycobacterium tuberculosis is highly contagious and easily spreads among the population, especially in confined and crowded environments, such as schools, factories, and prisons. The prevention and treatment of TB consumes a large amount of medical resources and funds, and imposes a heavy economic burden on individuals, families, and society. TB epidemics may lead to labor shortages and decreased production capacity, thus affecting socioeconomic stability and development [37]. Forecasting and analyzing the development trend of tuberculosis epidemics in various geographic regions can provide an adequate objective basis for the development of preventive and control measures. In the analysis process, it is crucial to establish a stable and accurate prediction model by utilizing various methods [38].

With the increase of external interfering factors, using only historical data of infectious diseases to predict future trends has greater limitations. Data from multiple sources were used for feature analysis, and the variables with the highest correlation with the number of incidence cases were screened as input data, which were combined with the autoregressive part of the time-series data to fit the number of TB incidence cases through the MiLSTM model, which is conducive to improving the accuracy of the model. Among them, the steps of feature extraction, feature screening, base model and parameter optimization are the key links to improve the accuracy and generalization of the model, and the diversification of feature extraction can achieve the purpose of making full use of the data, but it will bring the problem of too high feature dimension, which needs to be solved by choosing the appropriate feature screening method. The complexity of the deep learning model structure can more accurately capture the trend of the dependent variable, but compared with the ordinary model, the number of hyperparameters increases significantly, and it is necessary to choose the appropriate optimization algorithm for automatic optimization [38,39,40].

Therefore, in the analysis and prediction of tuberculosis incidence in Anhui Province, we conducted a comparative study to evaluate two univariate input models, four feature screening approaches and three multi-input models. On this basis, we proposed the RRL feature screening strategy. To address the difficulty in tuning the model performance due to the increase in parameters, the MiLSTM model performance was further improved using the PSO optimisation algorithm. Finally, we successfully constructed the optimal model PSO-MiLSTM and achieved the best prediction performance on the test set.

Although this study is reasonable and rigorous in both design and execution, there is still some room for improvement and enhancement. We look forward to future research directions. First, a multi-dimensional and broad field analysis should be conducted in terms of features such as spatial distribution, population distribution and causative factors for better implementation of early warning. Second, theoretical research should be conducted on the principles of formulaic methods such as MSE, huber loss function, and quantile loss function in terms of loss function construction, so as to construct a loss function that conforms to the infectious disease data, and to further improve the prediction effect of the model. Finally, the prediction effect of the model should be verified on multiple infectious disease datasets to verify its generalization ability.

In summary, this study uses data on atmospheric pollutants and climate as external input variables, combined with the time-series characteristics of the number of infectious disease cases, to carry out trend prediction analysis of the number of tuberculosis infections in Anhui Province. Based on the NAR model and NARX model, the S4 feature dataset consistent with the predictive analysis of infectious diseases was constructed by performing feature extraction and feature screening, and the excellence of the predictive performance of PSO-MiLSTM was revealed by empirical studies. Therefore, the feature engineering method as well as the prediction model proposed in this study can effectively analyse the development trend of the occurrence of tuberculosis epidemics, and by monitoring the change trend of the screened features, it can accurately determine the change trend of the number of tuberculosis cases in the next 1 month, and provide a powerful guide for the prevention and control of other infectious diseases.

In future studies, it is planned to incorporate data from more geographical areas, including regions with different climatic, economic and socio-cultural backgrounds, to test the stability and accuracy of the model in different environments. It is also planned to apply the model to more types of infectious diseases, such as other respiratory diseases, gastrointestinal diseases, and insect-borne infectious diseases, in order to explore the potential value and application of the model in the prediction of different diseases.

Conclusion

In summary, this study utilized data on atmospheric pollutants and climate as external input variables, and combined the time-series characteristics of the number of infectious disease incidence with the trend prediction analysis of the number of tuberculosis infections in Anhui Province. On the basis of NAR model and NARX model, the S4 feature dataset conforming to the predictive analysis of infectious diseases was constructed by performing feature extraction and feature screening, and the excellence of the predictive performance of PSO-MiLSTM was revealed by empirical studies. Therefore, the feature engineering method as well as the prediction model proposed in this study can effectively analyze the development trend of the occurrence of tuberculosis epidemics, and by monitoring the change trend of the screened features, it can accurately determine the change trend of the number of tuberculosis cases in the next 1 month, and provide a powerful guide for the prevention and control of other infectious diseases.