Most recently proposed methods for fully-automated segmentation can be categorized into model-based and atlas-based approaches. Model-based approaches apply a prior statistical shape model of the image object and try to best match the model to the target image. On the other hand, in atlas-based approaches, one or multiple references are generated by aligning and merging manually segmented images into specific atlas coordinate spaces. Despite promising results by model-based and atlas-based segmentation methods, both approaches perform poorly when there is high subject variability and significant differences in local features. In addition, both approaches rely on a priori knowledge of object shapes and require high computation costs and relatively long segmentation times. Other semi-automated techniques have been used for image segmentation using “region growing,” “live wire,” “edge detection,” and “active contour” methods. Although these semi-automated methods can achieve good results, they generally require a great deal of user interaction and, therefore, can be time-consuming and effort-demanding compared to fully-automated methods.

In early 2017, our group was among the first to explore deep learning for medical image analysis and developed an a-first-of-this-kind convolutional neural network-based method to perform high-performance knee MRI segmentation. We proposed a deep learning technique for segmenting bone and cartilage on knee MR images using a convolutional encoder-decoder network, which provides accurate multi-class tissue labels for patellofemoral bone and cartilage. A 3D simplex deformable modeling is applied to refine the output from the neural network to preserve the overall shape and maintain a desirable smooth surface for musculoskeletal structure. After benchmarking using publicly available knee image datasets to compare with other competing segmentation methods, our fully-automated approach led to remarkably improved accuracy and efficiency for segmenting 3D patellofemoral bone and cartilage with significantly reduced computing time (less than one minute processing time in our method versus hours in other conventional methods). Our approach demonstrated consistent segmentation performance on morphologic and quantitative MR images acquired using different sequences and spatial resolutions, including T1-weighted images, proton-density weighted images, fat-suppressed fast spin-echo images, and T2 maps.

Later, we extended the segmentation framework to incorporate generative adversarial networks, achieving unsupervised multi-contrast image conversion and segmentation in knee MRI. We also improved our method to allow automated full knee structure segmentation and parcellation so that efficient and highly reproducible analysis can be performed for regional quantification on all knee anatomies to better characterize the regional effect.

• Liu F, Zhou Z, Jang H, Zhao G, Samsonov A, Kijowski R: Deep Convolutional Neural Network and 3D Deformable Approach for Tissue Segmentation in Musculoskeletal Magnetic Resonance Imaging. Magn Reson Med. 2017; 79 (4), 2379-2391. (Editor’s Pick of Magnetic Resonance in Medicine)
• Liu F: SUSAN: Segment Unannotated image Structure using Adversarial Network. Magn Reson Med. 2018; 81 (5), 3330-3345. (Editor’s Pick of Magnetic Resonance in Medicine)
• Zhou Z, Zhao G, Kijowski R, Liu F: Deep Convolutional Neural Network for Segmentation of Knee Joint Anatomy. Magn Reson Med. 2018; 80 (6), 2759-2770.

MRI of the chest can provide a regional assessment of lung ventilation by inhaling oxygen, hyperpolarized noble gases, or fluorinated gas. Oxygen-enhanced MRI using a three-dimensional radial ultrashort echo time (UTE) sequence supports quantitative differentiation of diseased vs. healthy lungs using whole lung ventilation defect percent.

Despite these rapid advances in pulmonary structural and functional imaging using MRI, developing a fast, reproducible, and reliable quantification tool for extracting potential biomarkers and regional image features has lagged. Segmentation of lung parenchyma from proton MRI is challenging due to modality-specific complexities, including coil inhomogeneity, arbitrary intensity values, local magnetic susceptibility, and reduced proton density due to the large fraction of air space.

We developed and evaluated a deep learning framework for automated lung segmentation from functional lung imaging using UTE proton MRI to support fast functional quantification. Secondarily, to understand the disease-related structural alterations, the parenchymal signal-intensity assessments in the upper, middle, and lower lung regions automatically separated from the whole lung mask were compared in the diseased vs. normal groups. We demonstrated that lung segmentation using deep learning would enable accurate and faster volumetric and functional measurements relative to the reference method of segmentation using supervised region growing.

• Zha W, Fain S, Nagle S, Schiebler M, Liu F: Deep Convolutional Neural Networks with Multi-plane Consensus Labeling for Lung Function Quantification Using UTE Proton MRI. J Magn Reson Imaging. 2019; 50 (4), 1169-1181.

Brain extraction or skull stripping of MRI is an essential step in neuroimaging studies, the accuracy of which can severely affect subsequent image processing procedures. Current automatic brain extraction methods demonstrate good results on human brains but are often far from satisfactory on nonhuman primates, which are a necessary part of neuroscience research.

To overcome the challenges of brain extraction in nonhuman primates, we propose a fully-automated brain extraction pipeline combining a deep Bayesian convolutional neural network and a fully connected three-dimensional (3D) conditional random field (CRF). The deep Bayesian CNN, the Bayesian network, is the core segmentation engine. As a probabilistic network, it is not only able to perform accurate high-resolution pixel-wise brain segmentation but is also capable of measuring the model uncertainty by Monte Carlo sampling with dropout in the testing stage. Then, fully connected 3D CRF is used to refine the probability result from the Bayesian network in the whole 3D context of the brain volume. The proposed method was evaluated with a manually brain-extracted dataset comprising T1w images of 100 nonhuman primates. Our method outperforms six popular publicly available brain extraction packages and three well-established deep learning-based methods. A better performance against all the compared methods was verified by statistical tests. The maximum uncertainty of the model on nonhuman primate brain extraction has a mean value of 0.116 across all 100 subjects. The behavior of the uncertainty was also studied, which shows the uncertainty increases as the training set size decreases, the number of inconsistent labels in the training set increases, or the inconsistency between the training set and the testing set increases.

• Zhao G, Liu F, Oler JA, Meyerand B, Kalin NH, Birn R: Bayesian Convolutional Neural Network Based MRI Brain Extraction on Nonhuman Primates. NeuroImage. 2018; 175:32-44.

Video Demo for ISMRM 2019 Open-Source Software Tools Weekend Course:

• Educational Talk
• Demo Talk
• Example #1: a gradient echo image formation in MRI https://youtu.be/ol3wCrbIS_4
• Example #2: echo-planar imaging (EPI) https://youtu.be/7eVc5ztdAy4
• Example #3: non-Cartesian radial K-space acquisition https://youtu.be/IGl-MJcG2yg
• Example #4: balanced steady-state free precession acquisition https://youtu.be/9XRfmGKX0jY
• Example #5: multi-channel receiving for parallel imaging https://youtu.be/EI3LoWrWo_o