Inconsistencies in predictive models based on exhaled volatile organic compounds for distinguishing between benign pulmonary nodules and lung cancer: a systematic review

Abstract

Background

There is a general rise in incidentally found pulmonary nodules (PNs) requiring follow-up due to increased CT use. Biopsy and repeated CT scan are the most useful methods for distinguishing between benign PNs and lung cancer, while they are either invasive or involves radiation exposure. Therefore, there has been increasing interest in the analysis of exhaled volatile organic compounds (VOCs) to distinguish between benign PNs and lung cancer because it’s cheap, noninvasive, efficient, and easy-to-use. However, the exact value of breath analysis in this regard remains unclear.

Methods

A PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses)-oriented systematic search was performed to include studies that established exhaled VOC-based predictive models to distinguish between benign PNs and lung cancer and reported the exact VOCs used. Data regarding study characteristics, performance of the models, which predictors were incorporated, and methodologies for breath collection and analysis were independently extracted by two researchers. The exhaled VOCs incorporated into the predictive models were narratively synthesized, and those compounds that were reported in > 2 studies and reportedly exhibited consistent associations with lung cancer were considered key breath biomarkers. A quality assessment was independently performed by two researchers using both the Newcastle-Ottawa Scale (NOS) and the Prediction Model Risk of Bias Assessment Tool (PROBAST).

Results

A total of 11 articles reporting on 46 VOC-based predictive models were included. The majority relied solely on exhaled VOCs (n = 44), while two incorporated VOCs, demographical factors, and radiological signs. The variation in the sensitivity, specificity, and AUC indicators of the models that incorporated multiple factors was lower compared with those of the models that relied solely on exhaled VOCs. A total of 84 VOCs were incorporated. Of these, 2-butanone, 3-hydroxy-2-butanone, and 2-hydroxyacetaldehyde were identified as key predictors that had significantly higher concentrations in the exhaled breath samples of patients with lung cancer. Substantial heterogeneity was observed in terms of the modeling and validation methods used, as well as the approaches to breath collection and analysis. Many of the reports were missing certain key pieces of clinical and methodological information.

Conclusions

Although exhaled VOC-based models for predicting cancer risk might be a conceivable role as monitoring tools for PNs risk, there has been little overall change in the accuracy of these tests over time, and their role in routine clinical practice has not yet been established.

Clinical trial number

PROSPERO registration number CRD42023381458.

Introduction

Lung cancer is the second most common cancer and the leading cause of cancer-related deaths worldwide [1, 2]. Approximately half of patients with lung cancer are diagnosed at an advanced stage, missing the optimal treatment window. This contributes significantly to the high mortality rate associated with the malignancy [3]. Therefore, the early detection and diagnosis of lung cancer are of great significance to both clinical practice and general public health. Chest radiography and sputum cytology have been used for lung cancer screening since the 1970s. However, the sensitivity levels of these modalities are low [4, 5]. Over the past several decades, the US National Lung Cancer Screening Trial (NLST) and several other trials in Europe have demonstrated that low-dose computed tomography (LDCT) scan effectively reduces lung cancer mortality by facilitating earlier-stage diagnoses [6,7,8,9]. There is a general rise in incidentally found pulmonary nodules (PNs) requiring follow-up due to increased CT use [10]. However, only a small fraction of these nodules are actually lung cancer [11,12,13,14]. Therefore, it is essential to develop efficient and easy-to-use techniques for distinguishing between benign PNs and lung cancer, as this is crucial for guiding clinical decision-making.

Breath analysis, a simple and noninvasive approach, has shown great potential in diagnosing various pulmonary diseases [15]. Breath is a rich medium comprising gas-phase organic and inorganic compounds, as well as aerosols [16]. In the gas phase, there are hundreds of volatile organic compounds (VOCs) of diverse chemical natures that may be present in trace quantities [16]. Although no specific compounds have been identified whose presence or absence in exhaled breath can reliably indicate lung cancer to date, various prediction models based on exhaled VOCs can accurately classify lung cancer patients and healthy volunteers [17,18,19]. However, the value of breath analysis to distinguish between benign PNs and lung cancer remains unclear.

Published studies on predictive models for distinguishing between benign PNs and lung cancer based on exhaled VOCs have reported conflicting results. First, there is some inconsistency regarding the types and quantities of breath biomarkers incorporated into the models. For example, Peled et al. first developed a diagnostic model based on a single compound (1-octene) to distinguish patients with benign vs. malignant PNs, in their prospective trial [20]. By contrast, a team from the University of Louisville developed PN predictive models by incorporating four compounds: 2-butanone, 3-hydroxy-2-butanone, 2-hydroxyacetaldehyde, and 4-hydroxy-2-hexenal [21,22,23]. Based on these studies, researchers from Zhejiang University established predictive models by incorporating 19 VOCs to distinguish between benign PNs and lung cancer [24]. It is worth noting that diagnosing lung cancer using a single VOC is challenging, highlighting the importance of incorporating multiple VOCs to conduct more accurate predictions [25, 26]. Second, the performance metrics of existing models have been inconsistent. Two clinical studies have demonstrated that exhaled VOCs can be used to distinguish lung cancers confirmed by biopsy analysis from suspicious PNs observed on repeated LDCT, with high sensitivity and acceptable specificity [23, 27]. However, Liao et al. found that predictive models based solely on exhaled VOCs are not sufficient to accurately identify patients with lung cancer and those with benign PNs and reported that the performance of those models must be improved by combining them with additional factors such as demographic characteristics and radiological findings [28].

This review summarizes the current knowledge regarding exhaled VOC-based predictive models for distinguishing between benign PNs and lung cancer and assesses the overall value of these models in this regard.

Methods

This systematic review was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) reporting checklist [29]. The protocol was registered with the International Prospective Register of Systematic Reviews (PROSPERO) (registration number CRD42023381458).

Search strategy

A PRISMA-oriented systematic search was performed in PubMed, Embase, Web of Science, Cochrane Library, the Chinese National Knowledge Infrastructure (CNKI), China Biology Medicine disc (CBM), Wanfang, and CQVIP. We initially searched databases from their inceptions until 23 July 2022, with an update on 30 September 2023. The search was conducted through the combination of the Medical Subject Headings (MeSH) terms and keywords ‘lung neoplasms’ ‘lung cancer’ ‘multiple pulmonary nodules’ ‘solitary pulmonary nodule’ ‘volatile organic compounds’ ‘breath’ and ‘exhaled’. A list of the detailed search strategy used is described in the Appendix 1.

Inclusion and exclusion criteria

We included studies that reported on the development and validation of exhaled VOC-based risk-predictive models to distinguish between benign PNs and lung cancer. Studies were included if they were conducted in patients with LC confirmed via biopsy analysis, as well as in patients with benign PNs confirmed via biopsy analysis or detected by repeated radiological scans [30]. The included studies needed to identify and report specifically which VOC biomarkers were used. The predictors incorporated into the models were permitted to have been either exhaled VOCs alone or combined with other factors such as demographic characteristics and radiological signs. Articles published in English or Chinese were included, without any restrictions on study design.

Studies that analyzed exogenous VOCs or compounds in breath condensate and biofluids such as serum, urine, feces, and gastric content were excluded. Studies were also excluded if they were not carried out in humans or were not relevant to distinguishing between benign PNs and lung cancer. Studies limited to identifying risk predictors, reviews, letters, comments, and conference abstracts were also excluded.

Study selection

The electronic reports identified were imported into the reference manager Endnote and duplicates removed. Screening title and abstract of studies identified was performed by two researchers (ZXS and GYL) independently, and then full texts were reviewed to determine eligibility for inclusion. During this procedure, potential studies from the reference lists of original articles on this issue were screened and reviewed thoroughly for eligibility.

Data extraction and analysis

A pre-defined table was designed to extract the variables through a panel discussion with experts and epidemiologists. Information was extracted from each study includes: (1) the characteristics of study (title, first author, publication year, study country, study design, and study participants); (2) the characteristics of prediction models (candidate variables of the models, variables incorporated into the models, statistical method, modeling method, internal validation, external validation, sensitivity, specificity, accuracy, and area under the receiver operating characteristics (ROC) curves); (3) detailed methodologies (breath test environments, patient physiological conditions, sample collection methods, and VOC analysis and identification). Data extraction was independently performed by two researchers (ZXS and GYL). When encountering disagreements, our research team would discuss the article and reach a consistent agreement.

Quality assessment

The adapted Newcastle-Ottawa Scale (aNOS) used as a tool for risk of bias assessment of the included articles (Appendix 2) [31]. Moreover, the models’ applicability to the intended population and setting were assessed by the Prediction Model Risk of Bias Assessment Tool (PROBAST) (Appendix 3) [32, 33]. Two researchers conducted a critical appraisal of the studies, with any disagreements re-solved through consensus.

Results

Characteristics of the included studies

A total of 2,288 studies were identified. Among them, 2,240 were published in English, while 48 were in Chinese. In addition, five additional studies have been preliminary added according to their titles through other sources, including reference and website [28, 34,35,36,37]. After removing 1303 duplicates, a total of 990 studies underwent title and abstract screening, before 450 studies were screened for full-text. A total of 439 studies were excluded for various reasons, as outlined in Appendix 8. Our final analysis included 11 studies (Fig. 1 and Appendix 7). These studies reported 46 VOC-based predictive models that had been used to distinguish a total of 597 patients with benign PNs from 1,700 patients with lung cancer. The number of participants involved in establishing the predictive models ranged between 72 and 768 per study. Of the 11 total studies, seven (63.64%) were conducted in the US and four (36.36%) were carried out in China. Nine (81.82%) were cross-sectional studies, one was a prospective trial, and one was a cohort study. The characteristics of the included studies are listed in Table 1.

Fig. 1

Selection process of PRISMA flow diagram

Table 1 Characteristics of the included prediction model studies

Breath collection and analysis methods

The most commonly used methodology for collecting exhaled breath involved the use of Tedlar bags (n = 8 studies, 72.73%), although the sample volume ranged between 450 and 1,000 mL. All of the studies used MS-based techniques such as gas chromatography-mass spectrometry (GC-MS; n = 5, 45.45%), Fourier-transform ion cyclotron resonance mass spectrometry (FT-ICR-MS; n = 5, 45.45%), and high-pressure photon ionization time-of-flight mass spectrometry (HPPI-TOFMS; n = 1, 9.1%). Notably, solid-phase microextraction (SPME; n = 3, 27.27%) and thermal desorption (TD; n = 2, 18.18%) were the most frequently used methods for pre-concentrating the samples. Regarding the identification of the chemical structures of the VOCs, four of the studies (36.36%) referred to the National Institute of Standards and Technology Library (NIST), while two (18.18%) relied on MS techniques and retention times.

The factors that influenced exhaled VOC concentrations were divided into three categories: breath test environments, patient physiological conditions, and breath collection and analysis methods. A total of seven studies (63.64%) described their breath test environments, while seven (63.64%) focused on the physiological conditions of the patients analyzed. All of the included studies discussed their breath collection and analysis methods. Detailed information regarding these factors is presented in Table 2.

Table 2 Summary of factors reported to influence volatile organic compounds within exhaled breath

Incorporated predictors

The number of predictors incorporated into the models ranged between 1 and 35 (Table 3). Out of 46 total predictive models, 44 (95.64%) were solely based on VOC biomarkers. Only one model (2.18%) incorporated both VOC biomarkers and patient ages. Another model (2.18%) considered VOC biomarkers, age, and radiological signs (including nodule size, count, type, and spiculation).

A total of 84 VOCs were incorporated into the predictive models. The most commonly reported VOCs were 2-hydroxyacetaldehyde (n = 5 studies), 2-butanone (n = 4), 3-hydroxy-2-butanone (n = 4), and butyric acid (n = 3). Among these VOCs, the concentrations of 2-hydroxyacetaldehyde, 2-butanone, and 3-hydroxy-2-butanone were reported to have been significantly higher in the exhaled breath samples of patients with lung cancer. The associations between butyric acid concentration and lung cancer were not reported among any of the included studies. Table 4 presents a comprehensive summary of all of the VOCs that were reported on in ≥ 2 of the studies. The VOCs that were reported on in only one study are listed in Appendix 4, along with their chemical classes and CAS registry numbers.

Table 3 Predictors, methods, and performance of the assessed development models

Table 4 Volatile organic compounds reported in at least two studies

Modeling methods

Table 3 presents detailed information regarding the modeling methods used in the included studies. The most commonly used modeling methods were machine learning—which included support vector machine classification models (n = 9, 19.57%), logistic regression analysis (n = 6, 13.04%), random forest algorithm (n = 3, 6.52%), discriminant analysis (n = 3, 6.52%), artificial neural network (n = 2, 4.35%), and partial least squares analysis (n = 1, 2.17%). The statistical modeling methods used for 22 of the models (47.83%) were unclear.

Predictive performance

Each study included in our analysis reported at least one measure of predictive performance (Table 3). Nine of the studies reported classification measures of 20 predictive models. Of these, the sensitivity and specificity values of nine of the models (45%) were reported to both be > 70%. Six of the studies reported index of prediction accuracy values that ranged between 54.15% and 94.6%. Eight of the studies assessed discrimination using AUC values, which ranged between 0.625 and 0.986. Notably, none of the studies reported on calibration measures.

The variation in terms of sensitivity, specificity, and AUC values of the models that incorporated multiple factors was lower compared with those of the models that were based solely on exhaled VOCs. Among the models based solely on exhaled VOC biomarkers, the model with the highest sensitivity (100%) exhibited a specificity of 81.8%—while the model with the highest specificity (100%) showed a sensitivity of 28%. When exhaled VOC biomarkers were combined with other factors, the model with the highest sensitivity (80.80%) had a specificity of 60.50%, while the model with the highest specificity (68.3%) had a sensitivity of 78.7%. The models based solely on exhaled VOCs had AUC values ranging between 0.625 and 0.986, while those that incorporated additional factors such as age and radiological signs maintained stable AUC values of 0.776–0.781.

Validation approaches

Out of the 46 developed models, 19 (41.3%) exclusively underwent internal validation using methods such as K-fold cross-validation (n = 13, 28.3%), leave-group-out cross-validation (LGOCV; n = 5, 10.9%), and leave-one-out cross-validation (LOOCV; n = 1, 2.2%). Two of the models (4.3%) were internally validated through K-fold cross-validation before being subjected to blinded validation at two independent laboratories. Only one study validated two models (4.3%) using a new patient sample.

Study quality assessment

The aNOS scores of the included studies ranged between 6 and 9 (Appendix 5). Of the 11 total studies, five were rated very good, five were rated good, and one was rated satisfactory. The one study that received satisfactory ratings did not report on the representativeness levels of their patient samples.

The PROBAST tool was used to evaluate the risk of bias associated with the included studies. As shown in Fig. 2 and Appendix 6, all were found to have a high risk of bias in terms of participants, predictors, analysis, and overall domains. This high risk of bias may be attributable to the inappropriate inclusion criteria or small sample sizes of the studies. Predictors were also evaluated based on outcome information, and the selection of predictors using univariable analysis was done without reporting on measures of discrimination or calibration for model performance. Furthermore, the risk of bias in the outcome domains was unclear for all 11 of the included studies.

In terms of the applicability of the models, nine of the studies had low risks of bias in the participant domain, whereas two had unclear risks of bias. All of the models developed had unclear risks of bias in the predictors, outcomes, and overall domains.

Fig. 2

Risk of bias and applicability assessment according to the PROBAST

Discussion

This systematic review assessed the performance of 46 predictive models for lung cancer, based on VOCs detected in exhaled breath samples, reported in 11 studies and involving a total of 2,297 patients. Although the findings indicated that there was significant heterogeneity in the predictive performances of the models in terms of distinguishing between benign PNs and lung cancer, the models that incorporated additional factors such as demographic characteristics and radiological signs showed better performance metrics. Several compounds—including 2-butanone, 3-hydroxy-2-butanone, and 2-hydroxyacetaldehyde—were identified as being the most significant VOCs for distinguishing between benign PNs and lung cancer. Despite the potential of VOC breath analysis, there has been little overall change in the accuracy of these tests over time, and their role in routine clinical practice requires further research to be established.

Although the concentrations of exhaled 2-butanone, 3-hydroxy-2-butanone, and 2-hydroxyacetaldehyde have been identified as significant biomarkers for distinguishing between benign pulmonary nodules (PNs) and lung cancer, the endogenous origins of these volatile organic compounds (VOCs) remain largely unknown or speculative. In lung cancer patients, elevated levels of ketones have been observed, which may result from fatty acid oxidation known to increase the production of 2-butanone [38, 39]. A previous study has demonstrated that exhaled 3-hydroxy-2-butanone is likely an endogenous product resulting from the degradation of 2-butanone [40]. Similar research in rodents has shown that 2-butanone is oxidatively metabolized to 3-hydroxy-2-butanone, presumably by cytochrome P-450-dependent monooxygenases [41, 42]. Additionally, 3-hydroxy-2-butanone is produced during the detoxification of acetaldehyde and may participate in pulmonary redox cycling, potentially generating toxic reactive oxygen species (ROS) that can damage lung tissue [43,44,45]. Endogenous 2-hydroxyacetaldehyde is formed through the oxidative degradation of glucose, as well as from glycated proteins, lipid peroxidation, and the oxidation of amino acids [46]. This compound, produced by human neutrophils during phagocytosis, can be a potential source of ROS and may play an important role in tumor development and progression [46]. It is also important to note that exhaled isoprene has been identified as a marker for lung cancers [47]. Initially, it was believed to originate from hepatic cholesterogenesis; however, recent study has revealed that exhaled isoprene originates from muscular lipolytic cholesterol metabolism, as determined by the IDI2 gene [48]. Therefore, further research is needed to uncover the human metabolic origins of endogenous VOCs in lung cancer to promote clinical validation for diagnosis.

Our literature review indicated that, in addition to VOCs alone, incorporating multiple factors, such as patient characteristics and radiological signs, can greatly improve the stability of predictive models based on breath samples. Although the majority of the studies included in this review established predictive models based solely on exhaled VOCs, two of the ones that exhibited superior levels of performance also incorporated patient ages and radiological signs [28, 49]. When considering additional factors, variables such as age and smoking status should be considered first, as these represent known predictors of lung cancer risk [50, 51]. Furthermore, researchers have shown that diagnosing lung cancer using a single VOC is challenging and that incorporating several VOCs may be necessary to enhance the performance of such predictive models [25, 26]. However, including too many variables may increase the risk of model overfitting and spurious relationships [52]. Without proper statistical corrections, spurious correlations can be found even in entirely nonsensical contexts, referred to as Voodoo correlations [53]. This issue is further exacerbated by a relatively small sample size, which reduces the power of statistical tests and increases the likelihood of false positives. Therefore, well-established techniques such as the Bonferroni correction or False Discovery Rate (FDR) control that can and should be used for the reducing the likelihood of spurious correlations [54, 55].

Performance of predictive models varied significantly depending on the algorithm of machine learning. The main application field of logistic regression (LR) and support vector machine (SVM) is binary classification which makes these algorithms attractive to solve these clinical tasks [56, 57]. Xie SH, Liao PQ, and Chen X et al. utilized LR algorithm to develop predictive models based on exhaled VOCs to distinguish between benign PNs and lung cancer [24, 28, 49]. The predictive performance was almost the same for these models, with acceptable AUC values. Rai et al. trained SVM to establish their relevance in lung cancer patients’ classification which also achieved an acceptable accuracy [58]. The Random forest (RF) algorithm is more flexible because it can identify a broader scope of possible relationships between the model predictors and the disease status [59]. For example, Ding, X, et al. demonstrated the efficacy of the RF models based 16 exhaled VOCs for discriminating lung cancer from benign PNs [60]. In addition, validation approaches also varied widely between studies, including K-fold cross-validation [24, 28, 58] leave-group-out cross-validation [61], and leave-one-out cross-validation [20], which makes replication of results between studies difficult.

Our review further highlighted that sample collection and measurement techniques are hard to rationalize in clinical perspective. Many researchers have optimized method of exhaled breath analysis before the study of real subjects, but the results are varied. For example, some studies have required participants to fast for 6–12 h before breath collection [24, 28, 49], while Li MX. et al. conducted in patients without any diet controls [61]. In addition, some researchers have utilized Tedlar bags, Mylar bags [20,21,22,23, 61], and Bio-VOC breath samplers [49, 62] to collect breath samples. While others preferred portable breath sample collection devices or self-made collection devices [63, 64]. Only the end-tidal phase of a breath represents systemic concentrations of VOCs [65], while the lack of alveolar sampling in the included studies is biased via various confounding effects [21,22,23, 58, 60, 61]. Physio-metabolic and analytical confounders are to be minimized to realize actual pathophysiological effects on exhaled VOC concentrations [66]. Therefore, to standardize the procedure for exhaled breath analysis, expert opinions can be gathered through various methods, such as the Delphi process [67].

MS-based techniques are generally considered the gold standard for analyzing VOCs, owing to their ability to determine the molecular masses and possible chemical structures of individual VOCs [68]. However, this technique is subject to certain limitations. First, the analysis of VOC biomarkers using MS-based techniques requires pre-concentration and a high level of expertise, making them too expensive and complex for many clinical application [69, 70]. Moreover, pre-concentration methods may selectively enhance the signals of certain VOCs while simultaneously leading to the loss of others [20]. Second, breath samples are recommended to be analyzed within six hours of collection to best preserve sample compositions [71]. However, it is important to consider that background pollutants from the sampling equipment may potentially alter breath sample compositions as well [71, 72]. From a clinical perspective, clinicians are more interested in identifying “treatable traits” by stratifying patients based on clinically relevant qualities such as diagnosis, prognosis, and treatment response [73, 74]. In this scenario, simpler more cost-effective techniques have been considered and developed, such as the electronic nose (or “e-nose”) [75]. Many researchers have concluded that e-noses have the potential to become promising diagnostic tools in everyday clinical practices [76, 77].

Although existing VOC-based predictive models have shown acceptable levels of performance for distinguishing between benign PNs and lung cancer, the majority of the existing predictive models for distinguishing between benign PNs and lung cancer have not undergone validation, which hinders their clinical implementation. Ideally, external validations of VOC-based predictive models should be performed in large observational cohorts that have been carefully designed to be accurately representative of real-world patients with benign PNs and lung cancer [78]. To better inform clinical practices, future studies should carefully consider the heterogeneity of prediction effects and conduct model validations to develop predictive models that are of value to clinicians.

Limitations

This review summarized the current knowledge on exhaled VOC-based predictive models for distinguishing between benign PNs and lung cancer. However, it was subject to certain limitations worth noting. First, there was substantial heterogeneity among the included studies in terms of the methods used for their establishment, as well as for patient breath sample collection and analysis. Consequently, no quantitative meta-analysis has been conducted on the results of the studies. Second, all of the included studies were found to have a high risk of bias, likely caused by inappropriate inclusion criteria, the selection of predictors using univariate analyses, or a lack of indicators that could be used to evaluate calibration risk in the models. Third, most included studies in this review were conducted with cross-sectional study design, which conducted in patients with mid-/late-stage LC. The difference in performance of models based on breath VOCs among benign PNs and early LC demand further investigation with a large sample. In addition, eight of the studies were conducted in small groups, which may have limited the internal validity of their methods and the generalizability of their models to the general population. Finally, studies that used sensor- and pattern-based recognition technologies without reporting on the volatile biomarkers analyzed were not included in this review. Nevertheless, we believe that our report allows for a comprehensive review of exhaled VOC-based predictive models for distinguishing between benign PNs and lung cancer.

Conclusion

Exhaled 2-butanone, 3-hydroxy-2-butanone, and 2-hydroxyacetaldehyde might be significant breath markers for distinguishing between benign PNs and lung cancer. Moreover, predictive models that incorporate multiple factors alongside exhaled VOCs, such as demographic characteristics and radiological signs, have shown superior levels of performance compared with those based solely on exhaled VOCs. These observations highlight a conceivable role for VOCs as monitoring tools for PNs risk. However, the role of exhaled VOC-based predictive models in routine clinical practice has not yet been established, owing to various constraints associated with breath collection and analysis methods, as well as a general lack of external validation. Further investigations are therefore warranted to standardize the sample collection and measurement techniques for this approach, as well as to enhance the reliability and generalizability of such models.