Fully Automated and Standardized Segmentation of Adipose Tissue Compartments via Deep Learning in 3D Whole-Body MRI of Epidemiologic Cohort Studies

Introduction

MRI is a widely used imaging modality that enables highly resolved anatomic and functional depiction of organs, tissues, and disease processes. The acquisition of imaging data has become an integral aspect of reliable and fast disease assessment, staging, therapy, and treatment monitoring. In addition to conducting individual assessment of disease using MRI, large-scale cohort studies provide insight into factors influencing pathogenesis of various diseases (1,2).

The increasing prevalence of type 2 diabetes mellitus demands additional research in large-cohort MRI studies and fast automatic phenotyping to determine suitable biomarkers for risk assessment, as well as to deliver personalized lifestyle intervention treatment for prevention. The adipose tissue distribution in the body has been shown to be a key indicator of the pathogenesis of insulin resistance and type 2 diabetes mellitus (3–5). The use of MRI for whole-body assessment of adipose tissue distribution has proved to be a promising noninvasive method for screening in large-scale cohort studies (6–8). One crucial step in the evaluation of medical image content is the recognition and segmentation of specific organs or tissues, that is, performing a voxelwise classification known as semantic segmentation. Several methods for automated segmentation of MR images have been proposed, including those relying on explicit models (9), general correspondence (10), and random forest models (11), as well as deep learning (12) approaches, including convolutional neural networks (CNNs) (13–15). Most CNN-based segmentation architectures are derived from the U-Net (15), owing to its good generalizability and performance (16,17). Landmark detection (18,19) or anatomic object localization (20) have already been demonstrated and can be an important preprocessing step for segmentation. Deep learning–based approaches for use with MRI data have been proven effective for segmentation of brain matter (21,22), brain tumors (23,24), the liver (25), and cardiac ventricles (26).

Approaches for automatic adipose tissue segmentation have been proposed mainly by employing machine learning techniques (27). Suitable contrast in MRI for differentiation of adipose tissue from all other tissue types is provided by the markedly T1-weighted imaging leading to hyperintense adipose tissue signal with respect to lean tissue (LT). An alternative approach for fat-water imaging is phase sensitive to multiecho gradient-echo imaging (Dixon technique), which exploits the different chemical shifts from water to methylene and methyl signals from triglycerides, a principle first described by Dixon (28). The multiecho acquisition enables extraction of fat-water separated images and other confounding factors, such as fat fraction and R2* maps (29–31). This information can be used to preprocess the data (eg, bias field correction [32]), or it can be incorporated into the segmentation. Adipose tissue segmentations include multiatlas-based machine learning for fat images of regional muscles (33), contour-based segmentations (34), and machine learning clustering methods (35). More recently, two-dimensional (2D) deep learning networks for abdominal adipose tissue segmentation using the Dixon MRI technique (36,37) were proposed. Adipose tissue segmentation on whole-body MRI enabled the use of adipose tissue profiling (38) to investigate metabolic risks for cardiac disease (39), type 2 diabetes mellitus (40), and cancer (41).

The aforementioned studies, focused on either a single organ or a body region, were performed in smaller-scale or single cohorts, or provided only 2D processing. In this work, we propose a semantic segmentation network for whole-body adipose tissue segmentation that operates on single-parameter and multiparametric three-dimensional (3D) whole-body MRI from different multicenter epidemiologic patient cohort studies. The aim of this work is to provide a segmentation network that is robust to datasets acquired in different imaging sites, with varying scanner types, field strengths, and receiver coil set-ups, and for changing imaging sequences with varying imaging resolutions and different patient positioning.

This study had four main objectives: (a) to develop a 3D deep learning architecture as a combination of a densely connected network with merge-and-run mappings for attention-based multiresolution focusing in an encoder-decoder segmentation together with a relative positional encoding of input patches; (b) to determine the performance of this architecture in terms of robustness, sensitivity, specificity, precision, accuracy, and Dice overlap in comparison with a 3D U-Net segmentation; (c) to assess the possibility of transfer learning between different epidemiologic cohorts for training database composition and effect on its performance; and (d) to integrate the model into clinical workflow with automated reporting of adipose tissue head-feet profiles to enable profiling and studying in an epidemiologic setting.

Materials and Methods

A CNN is proposed for 3D semantic segmentation of whole-body adipose tissue into subcutaneous adipose tissue (SAT), abdominal visceral adipose tissue (VAT), LT, and background (BG). To cope with varying tissue appearances and distribution, the network uses a combination of architectural designs (14,15,42,43) and proposes the inclusion of positional encoding for attention-based (44) multiresolution learning (45) with an enhanced receptive field of view (14,15).

The network is trained and tested on whole-body MRI data from different multicenter epidemiologic patient databases with varying image contrast, imaging dimensionality, scanners, field strengths, patient positioning, and coil arrangement.

Epidemiologic Patient Databases

Database compositions, as well as scanner and acquisition parameters, are summarized in Table 1. Studies are approved by the ethics committees, and individuals gave written consent. Inclusion and exclusion criteria and further study information are stated in Bamberg et al (2) and Machann et al (6).

Tuebingen Family Study and German Center for Diabetes Research databases.—The Tuebingen Family Study (TUEF) and the studies performed in the framework of the German Center for Diabetes Research (DZD) aim to measure fat distribution in White individuals at increased risk of type 2 diabetes owing to weight (body mass index >27 kg/m²), having a first-degree relative with type 2 diabetes, impaired glucose tolerance, and/or gestational diabetes prior to and after lifestyle intervention (6). Data were acquired in a single-center study (TUEF) and multicenter studies (DZD) using 1.5- and 3.0-T scanners with whole-body T1-weighted multiple breath-hold and multislice 2D fast spin-echo sequences in the axial direction. Individuals were placed in the prone position (extended arms) with one rearrangement (the first step was head first, the second was feet first). Data were recorded using the whole-body coil of the MRI unit. From the approximately 2000 scanned individuals in four sites, 500 1.5-T cases and 500 3.0-T cases were labeled and included in this study.

German National Cohort database.—The German National Cohort (NAKO Gesundheitsstudie) aims to understand the natural history of a broad set of diseases and to potentially identify novel imaging biomarkers (2). The study randomly recruits individuals for MRI screening at five imaging centers distributed throughout Germany. From the imaging protocol, we evaluate the whole-body multiple breath-hold 3D dual gradient-echo chemical shift (Dixon) sequence, which was acquired with a 3.0-T scanner in the axial orientation. Individuals were placed in the head-first supine position. In this set-up, data recording was performed with local array receiver coils on the front and back of the individual. The scanner was equipped with an 18-channel body coil and a 32-channel spine coil from the manufacturer. Approximately 15 000 individuals had been examined at the time of this study, and MR images from 300 of them were labeled and included in this study.

Ground Truth Labeling

Training data are obtained with a semi-supervised labeling technique (34). Automatic fuzzy c-means clustering was used to presegment the whole-body MRI data based on intensity histograms after normalization and partial volume correction into BG, LT, and adipose tissue. A subsequent snake algorithm divides the adipose tissue into an SAT region and a VAT region. The obtained masks (BG, SAT, VAT, LT) were manually inspected and corrected, if necessary, by two trained experts with 9 (S.G.) and 17 (J.M.) years of experience in whole-body MRI on an in-house–developed graphic user interface to ensure proper data curation.

Proposed Architecture

The CNN-based segmentation is inspired by the concepts developed in U-Net (15) for pixelwise localization, V-Net (14) for volumetric medical image segmentation, ResNet (43) to cope with vanishing gradients and the degradation problem, DenseNet (42) to enable deep supervision, and merge-and-run mapping (46) to provide attention-based multiresolution focusing. The proposed network architecture is depicted in Figure 1. It combines the aforementioned schemes, which result in a densely connected CNN (ie, DCNet). Specific details on the DCNet architecture are described in Appendix E1 (supplement).

Figure 1:

Download as PowerPoint

In total, the proposed DCNet consists of 154 layers, which results in approximately 12 million trainable parameters for a single- or dual-channel 3D input. Each 3D volume is first normalized into a range of 0–1 and then cropped with a sliding window into overlapping 3D input patches with a size of 32 × 32 × 32 × C, where C ε[1,2] is given by the multiparametric input. For the T1-weighted fast spin-echo data, C equals 1, whereas for the multiecho chemical shift (Dixon technique) data, C equals 2 (fat and water image). RMSProp with multiclass focal loss (47) and multimetric loss (true-positive rate and Jaccard distance) is applied during training for a batch size of 48. Training was conducted over 100 epochs with early stopping. The network was implemented in TensorFlow 1.14 (https://www.tensorflow.org/) and trained on a Tesla V100 GPU (Nvidia, Santa Clara, Calif).

Experiments

The proposed method aims to reliably work in all cohorts and for changing input data (imaging sequences, contrasts, sites, scanners, field strengths, coil, and patient positioning). Therefore, robustness and generalizability of the proposed architecture toward changing training and test data are investigated. Robustness and reliability were investigated for all experiments by means of fourfold cross validation for the proposed DCNet and compared against a 3D U-Net segmentation (Appendix E2 [supplement]).

Differing training, validation, and test set composition among and within epidemiologic cohorts was created to determine if and what information can be exchanged. Furthermore, a transfer learning scheme was applied to the proposed DCNet to conduct the database domain change and was compared against multidatabase training. The proposed network was trained on database A and tested on database B or C, in the following description denoted by A → B|C. The following four scenarios were investigated: (a) A → A: intradatabase training and testing; (b) A → B: interdatabase testing to infer generalizability; (c) A+B → A|B: transductive transfer learning with pretraining on database A (first 40 epochs), followed by fine-tuning of training on database B (remaining epochs) and testing on A or B; and (d) A and B → A|B: multidatabase learning for randomly shuffled samples from databases A and B with testing on A or B. The latter case also serves as a comparison if a guided learning approach (ie, transfer learning) is superior to multidatabase learning. Data compositions between the different epidemiologic cohorts (TUEF/DZD and NAKO) as well as between the 1.5- and 3.0-T scans of the TUEF/DZD database are investigated to infer influence on prediction for changing imaging sequences and contrast (T1-weighted fast spin-echo technique vs multiecho chemical shift Dixon technique), imaging sites and scanners, coil positioning, patient positioning (prone vs supine), and field strengths (1.5 T vs 3.0 T).

Training datasets for the training folds and smaller subsets are randomly selected from the labeled cohort to provide uniform distribution over phenotypic factors (sex, age, and body mass index). For all experiments, datasets were split into 70% for training, 10% for validation, and 20% for the test set. For each of the 1.5- and 3.0-T measurements (500 individuals for each) from TUEF/DZD, there were 350 training, 50 validation, and 100 test sets. For the NAKO set of 300 individuals, cases were split into 210 training, 30 validation, and 60 test sets. For each cross-validation run, a different fixed training, validation, and test set was created.

Statistical Analysis

Unless stated otherwise, all reported results represent the mean and two-sided standard deviation on all four cross-validation runs and the test case. Performance was evaluated qualitatively and quantitatively in terms of specificity, sensitivity, precision, and accuracy (equations are found in Appendix E3 [supplement]), which are derived from the confusion matrix between the predicted segmentation, S, and the labeled ground truth, G, of the four classes (SAT, VAT, LT, BG) on a voxel-by-voxel comparison in a one-against-rest approach (ie, the target class is compared against the sum of all remaining classes). Segmentation overlaps are evaluated with the Dice coefficient (equation in Appendix E3 [supplement]) between S and G. Statistical significance for the performance metrics was determined by using a paired Welch t test (significance level of α = .05) under the null hypothesis of equal means for unequal variances performed in Python (version 3.6; https://www.python.org/) (open source) with SciPy (version 1.4.0; https://www.scipy.org/) (open source). The source code is available at https://github.com/lab-midas/med_segmentation.

Results

Axial slices of the TUEF/DZD databases for individuals examined using 1.5- and 3.0-T scanners are shown in Figure 2. The T1-weighted fast spin-echo MR images can be affected by spatial inhomogeneities of the B₁ field and by inhomogeneous sensitivity characteristics of the receiver coil, as indicated by the arrows in Figure 2. The segmentation is robust toward these artifacts and is in close agreement with the ground truth.

Figure 2:

Download as PowerPoint

The same network architecture is also generalizable to the NAKO database with individuals scanned with 3.0-T scanners at multiple imaging sites, as qualitatively depicted in Figure 3. Moreover, extension of the network input dimension from single-channel (T1-weighted fast spin-echo) to multichannel (multiecho chemical shift; Dixon technique) imagers preserves a stable segmentation, indicating a robust architecture. Quantitative evaluation of these experiments is shown in Table 2. Low standard deviations for the proposed DCNet indicate good robustness for changing training data folds. On average, for all segmented tissues and databases, sensitivity of 96.6%, specificity of 95.1%, precision of 92.1%, accuracy of 98.4%, and Dice overlap of 0.94 were achieved. The proposed DCNet significantly outperforms a standard 3D U-Net by 30.2% for sensitivity, 40.1% (P < .001) for specificity, 151.6% (P < .001) for precision, and 46.9% (P < .001) for Dice score, with the lone exception of accuracy (16.0%, P = .14) (Figure E1 [supplement]). Training of the proposed DCNet required approximately 25 hours, whereas only 5–7 seconds are needed for prediction after dataset loading.

Figure 3:

Download as PowerPoint

The influence of changing input data can be examined for changing scanners and imaging field strengths (1.5 and 3.0 T) in Figure 4 and for different cohorts (TUEF/DZD vs NAKO, fast spin-echo vs Dixon, 2D vs 3D, prone vs supine) and for different scanners and receiver coil arrangements in Figure 5. Intradatabase training and testing (A → A) performs better than interdatabase testing (A → B). Transfer (A+B → A|B) and multidatabase (A&B→A|B) learning overcome segmentation limitations of the interdatabase experiments, such as magnetic field inhomogeneities (Fig 4) and different through-plane resolutions (Fig 5; TUEF/DZD [10 mm] vs NAKO [3 mm]), which did not exist in the respective training database. Quantitative scores for changing scanners and imaging field strengths are shown in Table 3 and for changing cohorts in Table 4, which substantiate these findings. The best performing experiment and hence the best data use was obtained via transfer learning (A+B), which was not better than multidatabase learning (A&B) for sensitivity (P = .94 and P = .91), specificity (P = .98 and P = .90), precision (P = .66 and P = .95), accuracy (P = .82 and P = .67), and Dice overlap (P = .86 and P = .84), as shown in Tables 3 and 4, respectively.

Figure 4:

Download as PowerPoint

Figure 5:

Download as PowerPoint

The segmented images enable derivation of a head-feet adipose tissue profile, which helps to quickly visualize the adipose tissue distribution as exemplary (Fig E2 [supplement]). These profiles, together with the segmented masks, are automatically reported back to the referring physician. The adipose tissue profiles also enable phenotypic characterization, as shown for different test cases grouped by age and sex from the TUEF/DZD database, as shown in Figure 6, or grouped by body mass index and sex (Figure E3 [supplement]). The respective adipose tissue profiles of the NAKO database are shown in Figures E4 and E5 (supplement). In general, LT was more elevated in men than in women. LT and SAT show similar percentage distribution in women, whereas in men the percentage distribution of LT is mostly higher than that of SAT.

Figure 6:

Download as PowerPoint

Discussion

In this work, we present a 3D adipose tissue segmentation network for whole-body MRI. The proposed network combines different architectural concepts and receives the positional input of the 3D patches to learn multiresolution attention focusing. Reliable adipose tissue segmentation can be obtained with high Dice overlap (0.94), sensitivity (96.6%), specificity (95.1%), precision (92.1%), and accuracy (98.4%), which is also significantly better than a comparable U-Net segmentation (except accuracy). A fast whole-body adipose tissue segmentation can be conducted within 5–7 seconds, which enables integration into clinical workflow. The returned segmentation masks and adipose tissue profiles enable a quick phenotypic assessment. We investigated the performance of the network for varying training databases (TUEF/DZD and NAKO) with changing scanners and imaging field strengths (1.5 and 3.0 T), receiver coil set-ups, imaging sequences (multislice 2D T1-weighted fast spin-echo and 3D multiecho chemical shift Dixon methods), and patient positioning (prone and supine). The effect of changing training and test data, as well as information sharing between and within epidemiologic cohort studies by means of transfer learning, is investigated. The aim was to provide a robust architecture that generalizes well.

The proposed network deals well with changing input data, such as single- and multicontrast images, as well as varying resolutions, scanners, and imaging field strengths. The network performance can vary in testing if the image content differs strongly from the trained cases (eg, TUEF/DZD → NAKO) because the visual appearance of adipose tissue in fast spin-echo and Dixon images is diverse. This obstacle can be mitigated by transfer or joint learning on the data. We observed a minor improvement of transfer learning (A+B) over multidatabase learning (A and B) (ie, guiding the network training with a priori knowledge improves its performance). Moreover, use of complementary information can help mitigate segmentation errors (eg, magnetic field inhomogeneities in 3.0-T cases, as observed in Fig 4). If the network is trained on artifact-affected images, segmentation on good cases becomes more challenging (eg, 3.0 T → 1.5 T). These results suggest that a careful selection and training database composition is important for reliable and robust segmentation.

Segmentation of BG performs equally well in all experiments with high sensitivity, specificity, and precision. Because of its patient-dependent appearance and irregular structure, VAT is more challenging to segment than SAT. Nevertheless, high metrics were achieved as well. Performance of 1.5-T cases in TUEF/DZD was slightly better because of the decreased likelihood of disturbance by magnetic field inhomogeneities.

The proposed architecture outperforms a standard U-Net, owing to its multiresolution sharing and attention focusing. For U-Net, we observed repetitive patterns of misclassification that might indicate insufficient network depth for given input dimensionality. The applied 3D patching in DCNet provides invariance to epidemiologic parameters, such as height and weight, and no additional preprocessing (eg, image alignment) is required. Furthermore, invariance to patient positioning (prone vs supine) was observed. Overall good generalization to different scan conditions (imaging sequences, scanners, coil, and patient positioning) and anthropometric features was obtained. Results from cross-validation runs with different training and test cases showed small standard deviations, indicating reliable and robust training. On the basis of these experiences, we anticipate good reproducibility for repeated predictions and measurements.

Our study had limitations. The obtained results are specific to the imaging set-up and MR sequence design of the study at hand. The segmented tissues in our study are rather large structures and thus may be easier to segment compared with more complicated anatomic structures. Thus, generalizability to other segmentation tasks and methods will be investigated in future studies. Currently, the network is not trained to distinguish bone marrow, which leads to misclassifications. However, for the underlying task of adipose tissue profiling, these misclassifications are attributed to a small and almost constant amount among individuals. In future studies, we plan to extend the labeling and classification with a bone marrow class.

In conclusion, automatic 3D adipose tissue segmentation and standardized topography mapping based on whole-body MRI data with the proposed DCNet is feasible. The proposed architecture uses merge-and-run mapping blocks, dense connections, and patch-input encoding to provide multiresolution attention focusing. The architecture is robust and generalizable to different imaging sequences, contrasts, sites, scanners, field strengths, and examination set-ups (coil arrangements and patient positioning). Segmented adipose tissue compartments (subcutaneous and visceral fat) and head-feet profiles are automatically generated and provide feedback to the referring physician.

Disclosures of Conflicts of Interest: T.K. disclosed no relevant relationships. T.H. disclosed no relevant relationships. M.F. disclosed no relevant relationships. M.S. disclosed no relevant relationships. A.F. disclosed no relevant relationships. H.U.H. disclosed no relevant relationships. K.N. Activities related to the present article: disclosed no relevant relationships. Activities not related to the present article: is on the speakers bureau of Siemens, Bayer, and Bracco. Other relationships: disclosed no relevant relationships. F.B. Activities related to the present article: disclosed no relevant relationships. Activities not related to the present article: institution received a research grant from Siemens Healthineers; is on the speakers bureau of Siemens Healthineers. Other relationships: disclosed no relevant relationships. B.Y. disclosed no relevant relationships. F.S. disclosed no relevant relationships. S.G. disclosed no relevant relationships. J.M. disclosed no relevant relationships.

Author Contributions

Author contributions: Guarantor of integrity of entire study, J.M.; study concepts/study design or data acquisition or data analysis/interpretation, all authors; manuscript drafting or manuscript revision for important intellectual content, all authors; approval of final version of submitted manuscript, all authors; agrees to ensure any questions related to the work are appropriately resolved, all authors; literature research, T.K., T.H., F.B., S.G., J.M.; clinical studies, T.K., A.F., H.U., F.B., F.S.; statistical analysis, T.K., T.H.; and manuscript editing, T.K., T.H., M.F., M.S., A.F., K.N., F.B., B.Y., F.S., S.G., J.M.