A Simulation Paradigm for Evaluation of Subtle Liver Lesions at Pediatric CT: Performance and Confidence

Published Online:https://doi.org/10.1148/rycan.2019190027

Abstract

Purpose

To create and validate a systematic observer performance platform for evaluation of simulated liver lesions at pediatric CT and to test this paradigm to measure the effect of radiation dose reduction on detection performance and reader confidence.

Materials and Methods

Thirty normal pediatric (from patients aged 0–10 years) contrast material–enhanced, de-identified abdominal CT scans obtained from July 1, 2012, through July 1, 2016, were retrospectively collected from the clinical database. The study was exempt from institutional review board approval. Zero to three simulated, low-contrast liver lesions (≤6 mm) were digitally inserted by using software, and noise was added to simulate reductions in volume CT dose index (representing radiation dose estimation) of 25% and 50%. Pediatric, abdominal, and resident radiologists (three of each) reviewed 90 data sets in three sessions using an online interface, marking each lesion location and rating confidence (scale, 0–100). Statistical analysis was performed by using software.

Results

Mixed-effects models revealed a significant decrease in detection sensitivity as radiation dose decreased (P < .001). The mean confidence of the full-dose and 25% dose reduction examinations was significantly higher than that of the 50% dose reduction examinations (P = .011 and .012, respectively) but not different from one another (P = .866). Dose was not a significant predictor of time to complete each case, and subspecialty was not a significant predictor of sensitivity or false-positive results.

Conclusion

Sensitivity for lesion detection significantly decreased as dose decreased; however, confidence did not change between the full-dose and 25% reduced-dose scans. This suggests that readers are unaware of this decrease in performance, which should be accounted for in clinical dose reduction efforts.

Keywords: Abdomen/GI, CT, Liver, Observer Performance, Pediatrics, Perception Image

© RSNA, 2019

Summary

A simulation paradigm and user interface for dose reduction research allowed assessment of performance, confidence, and other behavioral metrics that should be accounted for when clinical CT protocols are created and revised.

Key Points

  • ■ This simulation paradigm for dose reduction research revealed a discrepancy between sensitivity and confidence as dose decreased, suggesting that readers may not be aware of their decreased performance.

  • ■ Decreasing CT dose did not affect reader specificity or number of clicks, perhaps because of perceived decreased discrimination.

  • ■ Reader subspecialty training did not affect sensitivity or specificity, but it did affect confidence and time to complete each case.

Introduction

Use of CT in the pediatric population has substantially increased over the past 3 decades (1,2), although this trend has leveled in the past decade (3). Because of this high utilization, children’s increased radiation sensitivity, and longer life expectancies (1), much attention has been placed on radiation safety. European diagnostic reference ranges (4), the Image Gently Campaign (5,6), and the ALARA (as low as reasonably achievable) principle aim to reduce radiation exposure, although variability persists for similar examinations (7,8).

CT depiction of small, low-contrast liver lesions is critical in patients with known malignancies and those who are immunosuppressed (9). Although most lesions are benign (9,10), a low-attenuation hepatic lesion can change the follow-up imaging timeline, necessitate further workup, or otherwise modify management. Reduced accuracy in identifying these lesions caused by attempts to reduce CT dose could adversely affect patient outcomes.

Clinical studies looking at the effect of dose reduction on lesion detection have faced obstacles. Retrospective studies are limited by the inability to control CT technique, including differences in contrast enhancement, which may affect lesion visibility independent of dose settings. Retrospective investigations are also limited by the rarity of a single or few hepatic lesions and unclear truth regarding lesion presence. Prospective investigations are also limited by different phases of enhancement, knowledge of a lesion’s presence, requirement for institutional review board approval for investigations entailing additional radiation exposure, and the risk of a nondiagnostic examination.

The development of noise addition paradigms and lesion insertion tools have allowed for investigations that use one set of patient scans to create multiple simulated data sets without additional radiation exposure (11). A validated lesion insertion software tool found that observers could not readily distinguish real liver lesions from simulated anthropomorphic liver lesions (12).

Although the primary outcome measure for dose reduction investigations is reader accuracy (11,13), other factors play into the clinical applications. For example, confidence levels and examination review times may relate to fatigue but do not necessarily affect performance. Adjusting CT parameters without accounting for these fatigue factors could have unintended consequences relating to productivity and burnout.

The purpose of our investigation was to use noise addition and lesion creation software to create and validate a systematic observer performance paradigm for simulated liver lesions in pediatric CT. We hypothesized that decreasing dose would be detrimental to sensitivity and the platform would help us gain insight into some aforementioned fatigue factors.

Materials and Methods

Study Cohort

Thirty normal pediatric contrast material–enhanced abdominal CT scans from 30 different patients obtained between July 1, 2012, and July 1, 2016, were retrospectively collected from the clinical database. The cohort was divided equally among three age groups: younger than 1 year, 1–5 years, and 6–10 years. All scans were obtained in the portal venous phase on the same CT scanner (SOMATOM Definition Flash; Siemens, Erlangen, Germany) with use of size-based protocols. Each scan was manually checked for liver lesions and de-identified for further image manipulation. This study was declared exempt by the institutional review board.

Lesion Insertion

For each scan, 20 possible liver lesion locations were manually identified to avoid artificial overlap with vasculature or other structures. These were evenly subdivided by location in one of the following categories: perivascular, periportal, greater than 1 cm from the liver edge (central, nonperivascular), and less than 1 cm from the liver edge (peripheral, nonperivascular). The perivascular lesions were more peripheral and adjacent to the hepatic veins or distal portal vein branches, whereas the periportal lesions were closer to the porta hepatis and adjacent to the main, right, or left portal veins. By randomly selecting from these locations, zero to three liver lesions were inserted into the CT scans to create hybrid scans with use of Lesion Tool software (Duke Lesion Tool; Carl E. Ravin Advanced Imaging Laboratories, Durham, NC) (12). Ten scans had one lesion, 10 had two lesions, five had three lesions, and five had no lesions. All lesions had anthropomorphic shapes, were 4–6 mm in size, and measured 25–45 HU less than the normal liver (Fig 1).

Figure 1:

Figure 1: Screenshot of the user interface with a sample lesion in the left hepatic lobe, periportal region.

Noise Addition

Noise was added to these image sets by using software to simulate volume CT dose index reductions of 25% and 50% less than the original scan. Hereafter, volume CT dose index reduction will be referred to as dose reduction, recognizing that volume CT dose index is not the true patient dose. The noise addition software first estimated the baseline mean noise in each image section by using a method from Christianson et al (14) and then superimposed randomly generated zero-mean noise according to the target dose reduction level. This randomly generated noise was synthesized to have the proper correlations (ie, noise texture) according to phantom-based measurements of the CT reconstruction’s noise power spectrum. In addition, noise “streakiness” and inhomogeneity were accounted for by incorporating the known optics of the CT system (eg, source-to-detector distance) and variable attenuation properties along different rays through the patient. This software was validated as equivalent in magnitude and texture using both adult CT scans and phantom data scanned at different dose levels (Fig 2). Upon completion, CT series corresponding to three dose levels (full dose, 25% reduction, and 50% reduction) per patient were available (90 total).

Figure 2:

Figure 2: CT image data of a phantom showing full dose, real 67% reduced dose, and simulated 67% reduced dose. The reduced-dose examination results look very similar, and the noise power spectra (NPS) for the two examinations are nearly identical in the bottom graph.

Lesion Detection Reader Study

Three pediatric radiologists (J.T.D., G.R.S., C.M.M.; experience: 7–20 years, mean, 11.7 years), three abdominal radiologists (B.C.A., D.M., B.N.P.; experience: 10–12 years; mean, 10.7 years), and three 4th-year radiology residents (A.D., R.G., V.P.) reviewed all 90 CT series in three sessions spaced at least 2 weeks apart. The 4th-year residents were intended to be the closest representation to general radiologists. Each reader was instructed to review the entire axial abdomen CT, mark the location(s) of hepatic lesion(s), and rate his or her confidence of a lesion’s presence on a scale of 0–100 (100 = certain, 0 = suspicious but very uncertain). Coronal and sagittal reformations were not provided. The 2-week interval between sessions was intended to reduce recall bias.

All reader sessions were performed by using a web-based interface in a single diagnostic reading room to ensure uniformity. The interface was designed to simulate a picture archiving and communication system with basic tools, such as window and level, zoom, and pan (Fig 1). Readers were asked to review cases at the same speed with which they would in a diagnostic reading room. The order in which the 90 cases appeared was randomized, with the stipulation that different dose levels for a single patient would not appear more than once in a single reader session. The software recorded the time to complete each case.

Readers were surveyed after completion (https://www.qualtrics.com, version 12.2015; Qualtrics, Provo, Utah) regarding the interface and their behavior during the study.

Data Analysis

A click was defined as any mouse click on a perceived lesion, including true-positives and false-positives. Clicks within the lesion and its immediate vicinity (25-pixel buffer) were considered true-positives, whereas clicks outside of this buffer were classified as false-positives. Free-response receiver operating characteristic (FROC) curves were generated by calculating the sensitivity as a function of the mean number of false-positives per case at a particular confidence threshold. FROC curves are analogous to receiver operating characteristic curves for free response tasks, and the curve is generated by changing the confidence threshold.

Statistical analysis was performed by creating a series of linear mixed effects models in R Statistical Software (version 1.1.442) using the lmer function within the lme4 package (R Foundation; https://www.r-project.org). The response (dependent) variables for each model were sensitivity, specificity, number of clicks, confidence, and mean time to complete a case. Dose reduction (0%, 25%, and 50%) and subspecialty (abdominal, pediatric, and resident) were included as fixed effects in all models. Lesion location (perivascular, periportal, central nonperivascular, and peripheral nonperivascular) was included as a fixed effect in the sensitivity model. Reader (readers 1–9) was included in all models as a random intercept term to reflect individual differences in readers and to allow us to extrapolate our conclusions to other reader groups. The distribution was assumed to be normal. Second-degree interaction terms were not significant in any of the models and therefore were excluded from the analyses. Significance was defined as P < .05.

Results

Overall lesion detection sensitivity was 39.6% (577 of 1458) among all nine readers and all dose levels. A hierarchical mixed-effects model was created by using dose reduction, subspecialty, and lesion location as fixed effects and reader as a random effect to predict sensitivity. This revealed that sensitivity significantly decreased across all three dose levels (0% vs 25%: β = 0.163 [standard error of the estimate {SEE}, 0.033], P < .0001; 0% vs 50%: β = 0.294 [SEE, 0.033], P < .0001; 25% vs 50%: β = 0.131 [SEE, 0.033], P < .0001) (Fig 3). Subspecialty was not a significant predictor of sensitivity (P = .326) (Fig 3). Lesion location was a significant predictor of sensitivity in that periportal and perivascular lesions had lower detection sensitivities than nonperivascular lesions greater than 1 cm from the liver edge (periportal: β = −0.227 [SEE, 0.034], P < .0001; perivascular: β = −0.243 [SEE, 0.039], P < .0001). Sensitivity did not differ between the nonperivascular lesions less than 1 cm and greater than 1 cm from the edge of the liver (β = −0.021 [SEE, 0.039], P = .591).

Figure 3:

Figure 3: Sensitivity of the readers across all three dose levels. Each vertical bar represents a reader, and the color of the vertical bar denotes the subspecialty. The sensitivity decreases as dose decreases, with no significant difference between the subspecialties.

Mean confidence per case was 52.3, based on a scale of 0–100 (100 = certain, 0 = suspicious but very uncertain). The mixed-effects model incorporating dose, subspecialty, and reader demonstrated that confidence significantly decreased as dose decreased, specifically between the 0% and 50% dose reduction levels (β = 8.964 [SEE, 2.239], P <.0001) and between the 25% and the 50% dose reduction levels (β = 8.964 [SEE, 2.409], P =.0002). There was no difference between the 0% and 25% dose reduction levels (β = −2.52 × 10−4 [SEE, 2.12], P >.99) (Fig 4). The model also demonstrated that the pediatric radiologist confidence level was significantly less than that of the abdominal imagers (β = −21.556 [SEE, 7.938], P =.035) and the residents (β = −23.5 [SEE, 7.91], P =.025) (Fig 4). Confidence did not differ between the abdominal imagers and the residents (β = −1.97 [SEE, 7.95], P =.813).

Figure 4:

Figure 4: Confidence of the readers across all three dose levels. Each vertical bar represents a reader, and the color of the vertical bar denotes the subspecialty. The confidence generally decreases as dose decreases, but confidence did not significantly differ between the 0% dose reduction and the 25% dose reduction. The pediatric subspecialists were significantly less confident than the other subspecialists.

Specificity is traditionally defined as the true-negative rate. However, readers in our study were clicking only on perceived lesions; we therefore defined specificity as the number of false-positives per case. A mixed-effects model incorporating dose, subspecialty, and reader demonstrated that specificity significantly decreased as dose decreased, specifically between the 0% and 50% dose reduction levels (β = 0.411 [SEE, 0.085], P < .0001) and between the 0% and the 25% dose reduction levels (β = 0.367 [SEE, 0.085], P < .0001). There was no difference between the 25% and 50% dose reduction levels (β = 0.044 [SEE, 0.085], P = .601). The model also demonstrated no difference in specificity between the different subspecialties (P = .292) (Fig 5).

Figure 5:

Figure 5: Specificity of the readers across all three dose levels. Each vertical bar represents a reader, and the color of the vertical bar denotes the subspecialty. The specificity decreases as dose decreases, with no significant difference between the subspecialties.

The mean number of mouse clicks was also evaluated to see whether the change in false-positives could be explained by differences in the number of clicks. A mixed-effects model incorporating dose, subspecialty, and reader demonstrated that the number of clicks significantly decreased as dose decreased, most notably between the 0% and 50% dose reduction levels (β = 0.882 [SEE, 0.106], P < .0001) and the 0% and the 25% dose reduction levels (β = 0.663 [SEE, 0.106], P < .0001). There was also a small but statistically significant difference between the 25% and 50% dose reduction levels (β = 0.219 [SEE, 0.106], P =.039).

Time to complete each case was recorded by using the web interface. Three outliers were excluded from this analysis because they were drastically different from the other data, and we surmised that the readers had been pulled away from the task at hand. A gamma regression model was used to account for the heterogeneity in the data, incorporating dose, subspecialty, and reader. This revealed no difference in time spent per case across all three dose levels (P = .1). There was a statistically significant difference in the amount of time spent depending on subspecialty. The abdominal imagers spent significantly less time than did the pediatric imagers (β = −0.663 [SEE, 0.225], P =.003) and the residents (β = −0.654 [SEE, 0.225], P = .004) (Fig 6). Time spent did not differ between the pediatric imagers and the residents (β = 0.009 [SEE, 0.225], P = .968).

Figure 6:

Figure 6: Mean time (in milliseconds) spent per case across all three dose levels. Each vertical bar represents a reader, and the color of the vertical bar denotes the subspecialty. The mean review time remained consistent across all three doses, with the abdominal imagers taking less time than the other subspecialists.

The FROC curve analysis averaging all readers demonstrated decreased sensitivity and specificity (false-positives) as dose decreased, which supports the results of the mixed-effects model. It also demonstrates variability in sensitivity and specificity across different readers at the same dose levels as well as between the same reader at different dose levels (Fig 7).

Figure 7:

Figure 7: Free-response receiver operating characteristic curves for all readers (columns) across all doses (rows). The last column and bottom row represent pooled data. This pooled analysis demonstrated decreased sensitivity and specificity (false-positives [FP]) as dose decreased, which supports the mixed-effects model results. It also demonstrates variability in sensitivity and specificity across different readers at the same dose levels as well as between the same reader at different dose levels.

Reader survey results indicated that 89% (eight of nine) thought that the images were similar in quality to diagnostic images and that the interface tools were adequate for the task they were asked to perform. All readers (nine of nine) thought that the interface was an efficient and effective method of recording lesion location and was easy to use. Two-thirds of the readers (six of nine) reported spending an equal amount of time per case compared with the reading room, whereas the other 33% (three of nine) reported spending more time than usual. One-third (three of nine) of readers reported interpreting these scans with equal sensitivity for liver lesions compared with the reading room environment, whereas 56% (five of nine) reported higher sensitivity.

Discussion

Our simulation paradigm demonstrated decreased sensitivity, specificity, confidence, and number of clicks with decreasing dose. Subspecialty did not influence sensitivity, specificity, or number of clicks, but it did affect confidence and timing. Lesion location was also a significant predictor of sensitivity.

Each result is discussed below, but at the core of these findings is the simulation tool. This paradigm can be used in dose reduction research in many contexts, with variation in organ system as well as lesion characteristics, such as size, heterogeneity, and attenuation. It is more realistic than the signal-known-exactly task used in many observer studies, in which the lesion characteristics and location are known a priori (1517). The interface allowed electronic recording of all data used for standard detection metrics as well as reader confidence, time spent per case, and number of clicks. These additional behavioral metrics are not typically assessed with reader studies and revealed unexpected results.

Our study demonstrated a significant decrease in sensitivity as dose decreased. Although not surprising, this finding validates the study design in that lesions were at the threshold of detectability. We also demonstrated lesion location as a significant predictor of sensitivity, with periportal and perivascular lesions being harder to detect. This observation is analogous to data on lung nodule detection, in which perihilar and vessel-attached nodules are harder to see than peripherally located nodules (18).

Reader confidence overall decreased with decreasing dose; however, this was not a linear relationship. There were significant differences between the 0% and 50% dose reduction levels and between the 25% and 50% dose reduction levels, but confidence did not differ between the 0% and 25% dose reductions. We previously described a significant difference in sensitivity between all dose levels, suggesting that readers may not be aware of their decreased performance between these two higher doses. This implies that further attempts to decrease CT dose could be detrimental to performance without physician awareness, ultimately affecting patient care.

One unexpected result was that pediatric radiologists were less confident than the other subspecialties. This was counterintuitive because they tend to read lower-dose CT acquisitions regularly, so one might expect this group to be more confident. This discrepancy could be due to this group of readers. Alternatively, this could be explained by the fact that liver lesions in pediatric patients are much less common than in adults (19), and the overall volume of our pediatric division is less than that of the abdominal division. An important application is the effect that low confidence has on fatigue, especially in an era of high physician burnout. Prior research has shown that higher intolerance for uncertainty is correlated with higher rates of burnout (20); by decreasing confidence with lower-dose scans, we are perhaps putting more radiologists at risk for burnout.

We also showed decreasing false-positives (specificity) and number of clicks with decreasing dose. This was counterintuitive because we had anticipated that added noise would result in more false-positives. Perhaps readers put forth less effort for the lower-dose studies as a result of perceived decreased discrimination, essentially changing their assessment threshold because noise was prohibitively high.

Another surprising result was that readers spent the same amount of time per case regardless of dose. There are a few possible explanations. First, readers may habitually spend the same amount of time evaluating each liver. As dose decreased and lesions took more time to find, the number of clicks and number of false-positives decreased for a set time period. Alternatively, readers may have simply found fewer lesions on the low-dose images, with each lesion taking more time to find.

Our study had several limitations. First, we asked readers to perform a task, which may not reflect actual performance in the reading room. Our survey indicated that many readers spent more time and had higher sensitivities than they normally would. Second, our full-dose CT examinations are already relatively low dose, following standard weight-based protocols. Thus, we do not have performance metrics for higher-dose examinations. Additionally, our study was performed in the academic setting, so we do not know how community practice radiologists would compare. Finally, we had a relatively small number of readers from each subspecialty. For this reason, we used a mixed-effects model with reader as a random intercept to better model the general population.

In summary, we translated a simulation paradigm and user interface for dose reduction research that allowed assessment of performance, confidence, and other behavioral metrics not typically evaluated with observer studies. These behavioral data revealed surprising results, including decreased false-positives and number of clicks with decreasing dose, low confidence of pediatric radiologists, and equal time spent per study regardless of dose. This highlights some important factors beyond performance alone that should be accounted for when clinical CT protocols are created and revised.

Disclosures of Conflicts of Interest: J.S.N. disclosed no relevant relationships. J.B.S. Activities related to the present article: disclosed no relevant relationships. Activities not related to the present article: receives royalties (<$5000/year) from a license deal between Duke University and Sun Nuclear Gammex for the design of a CT image quality phantom. Other relationships: disclosed no relevant relationships. E.S. Activities related to the present article: disclosed no relevant relationships. Activities not related to the present article: paid by Imalogix for board membership and consultancy; paid by for expert testimony (patent law); institution receives grants from Siemens and Bracco; institution receives royalties from GE, Imalogix, 12Sigma, SunNuclear, Cambridge University Press, and Wiley and Sons. Other relationships: disclosed no relevant relationships. T.R. disclosed no relevant relationships. L.N. disclosed no relevant relationships. A.E. disclosed no relevant relationships. B.Z. disclosed no relevant relationships. B.C.A. disclosed no relevant relationships. J.T.D. disclosed no relevant relationships. A.D. disclosed no relevant relationships. R.G. disclosed no relevant relationships. D.M. disclosed no relevant relationships. C.M.M. disclosed no relevant relationships. V.P. disclosed no relevant relationships. B.N.P. Activities related to the present article: disclosed no relevant relationships. Activities not related to the present article: institution receives grant from Siemens; author receives grant from GE Healthcare; author receives payment for lectures from GE Healthcare. Other relationships: disclosed no relevant relationships. G.R.S. disclosed no relevant relationships. D.P.F. disclosed no relevant relationships.

Author Contributions

Author contributions: Guarantors of integrity of entire study, J.S.N., A.E., D.P.F.; study concepts/study design or data acquisition or data analysis/interpretation, all authors; manuscript drafting or manuscript revision for important intellectual content, all authors; approval of final version of submitted manuscript, all authors; agrees to ensure any questions related to the work are appropriately resolved, all authors; literature research, J.S.N., J.B.S., E.S., L.N., C.M.M., D.P.F.; clinical studies, J.S.N., E.S., B.C.A., J.T.D., G.R.S.; experimental studies, J.S.N., J.B.S., E.S., T.R., R.G., D.P.F.; statistical analysis, J.S.N., J.B.S., T.R., L.N., A.E., B.Z.; and manuscript editing, J.S.N., J.B.S., E.S., T.R., L.N., A.E., B.C.A., J.T.D., R.G., D.M., C.M.M., V.P., B.N.P., G.R.S., D.P.F.

Authors declared no funding for this work.

References

  • 1. Miglioretti DL, Johnson E, Williams A, et al. The use of computed tomography in pediatrics and the associated radiation exposure and estimated cancer risk. JAMA Pediatr 2013;167(8):700–707.
  • 2. Broder J, Fordham LA, Warshauer DM. Increasing utilization of computed tomography in the pediatric emergency department, 2000-2006. Emerg Radiol 2007;14(4):227–232.
  • 3. Lodwick DL, Cooper JN, Kelleher KJ, Brilli R, Minneci PC, Deans KJ. Variation in utilization of computed tomography imaging at tertiary pediatric hospitals. Pediatrics 2015;136(5):e1212–e1219.
  • 4. Goske MJ, Strauss KJ, Coombs LP, et al. Diagnostic reference ranges for pediatric abdominal CT. Radiology 2013;268(1):208–218.
  • 5. Image Gently. Image Gently Development of Pediatric CT Protocols 2014. http://www.imagegently.org/Portals/6/Procedures/IG%20CT%20Protocols%20111714.pdf. Accessed December 30, 2015.
  • 6. Goske MJ, Applegate KE, Boylan J, et al. The ‘Image Gently’ campaign: increasing CT radiation dose awareness through a national education and awareness program. Pediatr Radiol 2008;38(3):265–269.
  • 7. Mileto A, Nelson RC, Larson DG, et al. Variability in radiation dose from repeat identical CT examinations: longitudinal analysis of 2851 patients undergoing 12,635 thoracoabdominal CT scans in an academic health system. AJR Am J Roentgenol 2017;208(6):1285–1296.
  • 8. Armao D, Hartman T, Shea CM, et al. A toolkit for pediatric CT dose reduction in community hospitals. J Am Coll Radiol 2016;13(11):1337–1342.e11.
  • 9. Schwartz LH, Gandras EJ, Colangelo SM, Ercolani MC, Panicek DM. Prevalence and importance of small hepatic lesions found at CT in patients with cancer. Radiology 1999;210(1):71–74.
  • 10. Jones EC, Chezmar JL, Nelson RC, Bernardino ME. The frequency and significance of small (less than or equal to 15 mm) hepatic lesions detected by CT. AJR Am J Roentgenol 1992;158(3):535–539.
  • 11. Swanick CW, Gaca AM, Hollingsworth CL, et al. Comparison of conventional and simulated reduced-tube current MDCT for evaluation of suspected appendicitis in the pediatric population. AJR Am J Roentgenol 2013;201(3):651–658.
  • 12. Solomon J, Samei E. A generic framework to simulate realistic lung, liver and renal pathologies in CT imaging. Phys Med Biol 2014;59(21):6637–6657.
  • 13. Frush DP, Slack CC, Hollingsworth CL, et al. Computer-simulated radiation dose reduction for abdominal multidetector CT of pediatric patients. AJR Am J Roentgenol 2002;179(5):1107–1113.
  • 14. Christianson O, Winslow J, Frush DP, Samei E. Automated technique to measure noise in clinical CT examinations. AJR Am J Roentgenol 2015;205(1):W93–W99.
  • 15. Xu XG, Eckerman KF, eds. Handbook of Anatomical Models for Radiation Dosimetry. Boca Raton, Fla: CRC, 2009.
  • 16. Eckstein MP, Abbey CK. Model observers for signal-known-statistically tasks (SKS). Medical Imaging 2001: Image Perception and Performance. In: Krupinski EA, Chakraborty DP, eds. Proceedings of SPIE: medical imaging 2001—image perception and performance. Vol 4324. Bellingham, Wash: International Society for Optics and Photonics, 2001.
  • 17. Park S, Clarkson E, Kupinski MA, Barrett HH. Efficiency of the human observer detecting random signals in random backgrounds. J Opt Soc Am A Opt Image Sci Vis 2005;22(1):3–16.
  • 18. Naidich DP, Rusinek H, McGuinness G, Leitman B, McCauley DI, Henschke CI. Variables affecting pulmonary nodule detection with computed tomography: evaluation with three-dimensional computer simulation. J Thorac Imaging 1993;8(4):291–299.
  • 19. Chiorean L, Cui XW, Tannapfel A, et al. Benign liver tumors in pediatric patients - review with emphasis on imaging features. World J Gastroenterol 2015;21(28):8541–8561.
  • 20. Cooke GPE, Doust JA, Steele MC. A survey of resilience, burnout, and tolerance of uncertainty in Australian general practice registrars. BMC Med Educ 2013;13(1):2.

Article History

Received: May 13 2019
Revision requested: May 29 2019
Revision received: June 18 2019
Accepted: July 5 2019
Published online: Sept 27 2019