Systematic Review Imaging| Volume 49, ISSUE 10, P1367-1378, October 2020

Accuracy and reliability of automatic three-dimensional cephalometric landmarking

  • G. Dot
    Address: Gauthier Dot, Institut de Biomecanique Humaine Georges Charpak (IBHGC), Arts et Metiers Institute of Technology, 151 bd de l’Hopital, 75013 Paris, France.
    Institut de Biomecanique Humaine Georges Charpak (IBHGC), Arts et Metiers Institute of Technology, Paris, France
    Search for articles by this author
  • F. Rafflenbeul
    Department of Dento-Facial Orthopedics, Faculty of Dental Surgery, Strasbourg University, Strasbourg, France
    Search for articles by this author
  • M. Arbotto
    Institut de Biomecanique Humaine Georges Charpak (IBHGC), Arts et Metiers Institute of Technology, Paris, France
    Search for articles by this author
  • L. Gajny
    Institut de Biomecanique Humaine Georges Charpak (IBHGC), Arts et Metiers Institute of Technology, Paris, France
    Search for articles by this author
  • P. Rouch
    Institut de Biomecanique Humaine Georges Charpak (IBHGC), Arts et Metiers Institute of Technology, Paris, France
    Search for articles by this author
  • T. Schouman
    Institut de Biomecanique Humaine Georges Charpak (IBHGC), Arts et Metiers Institute of Technology, Paris, France

    Sorbonne Universite, AP-HP, Hopital Pitie-Salpetriere, Service de Chirurgie Maxillo-Faciale, Paris, France
    Search for articles by this author
Published:March 10, 2020DOI:


      The aim of this systematic review was to assess the accuracy and reliability of automatic landmarking for cephalometric analysis of three-dimensional craniofacial images. We searched for studies that reported results of automatic landmarking and/or measurements of human head computed tomography or cone beam computed tomography scans in MEDLINE, Embase and Web of Science until March 2019. Two authors independently screened articles for eligibility. Risk of bias and applicability concerns for each included study were assessed using the QUADAS-2 tool. Eleven studies with test dataset sample sizes ranging from 18 to 77 images were included. They used knowledge-, atlas- or learning-based algorithms to landmark two to 33 points of cephalometric interest. Ten studies measured mean localization errors between manually and automatically detected landmarks. Depending on the studies and the landmarks, mean errors ranged from <0.50 mm to>5 mm. The two best-performing algorithms used a deep learning method and reported mean errors <2 mm for every landmark, approximating results of operator variability in manual landmarking. Risk of bias regarding patient selection and implementation of the reference standard were found, therefore the studies might have yielded overoptimistic results. The robustness of these algorithms needs to be more thoroughly tested in challenging clinical settings. PROSPERO registration number: CRD42019119637.

      Key words

      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to International Journal of Oral and Maxillofacial Surgery
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Leonardi R.
        • Giordano D.
        • Maiorana F.
        • Spampinato C.
        Automatic cephalometric analysis: a systematic review.
        Angle Orthod. 2008; 78: 145-151
        • Gribel B.F.
        • Gribel M.N.
        • Frazão D.C.
        • McNamara J.A.
        • Manzi F.R.
        Accuracy and reliability of craniometric measurements on lateral cephalometry and 3D measurements on CBCT scans.
        Angle Orthod. 2011; 81: 26-35
        • Olszewski R.
        • Cosnard G.
        • Macq B.
        • Mahy P.
        • Reychler H.
        3D CT-based cephalometric analysis: 3D cephalometric theoretical concept and software.
        Neuroradiology. 2006; 48: 853-862
        • Lee S.-H.
        • Kil T.-J.
        • Park K.-R.
        • Kim B.C.
        • Kim J.-G.
        • Piao Z.
        • Corre P.
        Three-dimensional architectural and structural analysis—a transition in concept and design from Delaire’s cephalometric analysis.
        Int J Oral Maxillofac Surg. 2014; 43: 1154-1160
        • Pittayapat P.
        • Limchaichana-Bolstad N.
        • Willems G.
        • Jacobs R.
        Three-dimensional cephalometric analysis in orthodontics: a systematic review.
        Orthod Craniofac Res. 2014; 17: 69-91
        • Kapila S.D.
        • Nervina J.M.
        CBCT in orthodontics: assessment of treatment outcomes and indications for its use.
        Dentomaxillofac Radiol. 2015; 4420140282
        • Swennen G.R.J.
        • Schutyser F.A.C.
        • Hausamen J.-E.
        Three-Dimensional Cephalometry: A Color Atlas and Manual.
        Springer-Verlag, Berlin Heidelberg2006
        • Smektała T.
        • Jędrzejewski M.
        • Szyndel J.
        • Sporniak-Tutak K.
        • Olszewski R.
        Experimental and clinical assessment of three-dimensional cephalometry: a systematic review.
        J Craniomaxillofac Surg. 2014; 42: 1795-1801
        • Lagravère M.O.
        • Low C.
        • Flores-Mir C.
        • Chung R.
        • Carey J.P.
        • Heo G.
        • Major P.W.
        Intraexaminer and interexaminer reliabilities of landmark identification on digitized lateral cephalograms and formatted 3-dimensional cone-beam computerized tomography images.
        Am J Orthod Dentofacial Orthop. 2010; 137: 598-604
        • Hassan B.
        • Nijkamp P.
        • Verheij H.
        • Tairie J.
        • Vink C.
        • van der Stelt P.
        • van Beek H.
        Precision of identifying cephalometric landmarks with cone beam computed tomography in vivo.
        Eur J Orthod. 2013; 35: 38-44
        • Sam A.
        • Currie K.
        • Oh H.
        • Flores-Mir C.
        • Lagravére-Vich M.
        Reliability of different three-dimensional cephalometric landmarks in cone-beam computed tomography: a systematic review.
        Angle Orthod. 2018; 89: 317-332
        • Lindner C.
        • Wang C.-W.
        • Huang C.-T.
        • Li C.-H.
        • Chang S.-W.
        • Cootes T.F.
        Fully automatic system for accurate localisation and analysis of cephalometric landmarks in lateral cephalograms.
        Sci Rep. 2016; 6: 33581
        • Gupta A.
        • Kharbanda O.P.
        • Sardana V.
        • Balachandran R.
        • Sardana H.K.
        A knowledge-based algorithm for automatic detection of cephalometric landmarks on CBCT images.
        Int J Comput Assist Radiol Surg. 2015; 10: 1737-1752
        • Zhang J.
        • Liu M.
        • Wang L.
        • Chen S.
        • Yuan P.
        • Li J.
        • Shen S.G.-F.
        • Tang Z.
        • Chen K.-C.
        • Xia J.J.
        • Shen D.
        Joint craniomaxillofacial bone segmentation and landmark digitization by context-guided fully convolutional networks.
        in: Medical Image Computing and Computer-Assisted Intervention ? MICCAI 2017. MICCAI 2017. Lecture Notes in Computer Science. vol 10434. Springer, Cham2017
        • Whiting P.F.
        QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies.
        Ann Intern Med. 2011; 155: 529
        • Reitsma J.B.
        • Rutjes A.W.S.
        • Whiting P.
        • Vlassov V.V.
        • Leeflang M.M.G.
        • Deeks J.J.
        Chapter 9: Assessing methodological quality.
        in: Deeks J.J. Bossuyt P.M. Gatsonis C. Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy, Version 1.0.0. The Cochrane Collaboration, 2009
        • Shahidi S.
        • Bahrampour E.
        • Soltanimehr E.
        • Zamani A.
        • Oshagh M.
        • Moattari M.
        • Mehdizadeh A.
        The accuracy of a designed software for automated localization of craniofacial landmarks on CBCT images.
        BMC Med Imaging. 2014; 14: 32
        • Zhang J.
        • Gao Y.
        • Wang L.
        • Tang Z.
        • Xia J.J.
        • Shen D.
        Automatic craniomaxillofacial landmark digitization via segmentation-guided partially-joint regression forest model and multiscale statistical features.
        IEEE Trans Biomed Eng. 2016; 63: 1820-1829
        • Codari M.
        • Caffini M.
        • Tartaglia G.M.
        • Sforza C.
        • Baselli G.
        Computer-aided cephalometric landmark annotation for CBCT data.
        Int J Comput Assist Radiol Surg. 2017; 12: 113-121
        • de Jong M.A.
        • Gül A.
        • de Gijt J.P.
        • Koudstaal M.J.
        • Kayser M.
        • Wolvius E.B.
        • Böhringer S.
        Automated human skull landmarking with 2D Gabor wavelets.
        Phys Med Biol. 2018; 63105011
        • Montúfar J.
        • Romero M.
        • Scougall-Vilchis R.J.
        Hybrid approach for automatic cephalometric landmark annotation on cone-beam computed tomography volumes.
        Am J Orthod Dentofac Orthop. 2018; 154: 140-150
        • Neelapu B.C.
        • Kharbanda O.P.
        • Sardana V.
        • Gupta A.
        • Vasamsetti S.
        • Balachandran R.
        • Sardana H.K.
        Automatic localization of three-dimensional cephalometric landmarks on CBCT images by extracting symmetry features of the skull.
        Dentommaxillofacial Radiol. 2018; 4720170054
        • Torosdagli N.
        • Liberton D.K.
        • Verma P.
        • Sincan M.
        • Lee J.S.
        • Bagci U.
        Deep geodesic learning for segmentation and anatomical landmarking.
        IEEE Trans Med Imaging. 2019; 38: 919-931
        • O’Neil A.Q.
        • Kascenas A.
        • Henry J.
        • Wyeth D.
        • Shepherd M.
        • Beveridge E.
        • Clunie L.
        • Sansom C.
        • Šeduikytė E.
        • Muir K.
        • Poole I.
        Attaining human-level performance with atlas location autocontext for anatomical landmark detection in 3d CT data.
        in: Leal-Taixé L. Roth S. Computer Vision – ECCV 2018 Workshops. Springer International Publishing, 2019: 470-484
        • Gupta A.
        • Kharbanda O.P.
        • Sardana V.
        • Balachandran R.
        • Sardana H.K.
        Accuracy of 3D cephalometric measurements based on an automatic knowledge-based landmark detection algorithm.
        Int J Comput Assist Radiol Surg. 2016; 11: 1297-1309
        • Fryback D.G.
        • Thornbury J.R.
        The efficacy of diagnostic imaging.
        Med Decis Mak Int J Soc Med Decis Mak. 1991; 11: 88-94
        • Vandaele R.
        • Aceto J.
        • Muller M.
        • Péronnet F.
        • Debat V.
        • Wang C.-W.
        • Huang C.-T.
        • Jodogne S.
        • Martinive P.
        • Geurts P.
        • Marée R.
        Landmark detection in 2D bioimages for geometric morphometrics: a multi-resolution tree-based approach.
        Sci Rep. 2018; 8
        • Wang C.-W.
        • Huang C.-T.
        • Lee J.-H.
        • Li C.-H.
        • Chang S.-W.
        • Siao M.-J.
        • Lai T.M.
        • Ibragimov B.
        • Vrtovec T.
        • Ronneberger O.
        • Fischer P.
        • Cootes T.F.
        • Lindner C.
        A benchmark for comparison of dental radiography analysis algorithms.
        Med Image Anal. 2016; 31: 63-76
        • SEDENTEXCT project
        Cone Beam CT for dental and maxillofacial radiology (evidence based guidelines).
        European Commission, 2012