Registration of Deformed Tissue: A GNN-VAE Approach with Data Assimilation for Sim-to-Real Transfer

In image-guided surgery, deformation of soft tissues can
cause substantial errors in targeting internal targets,
since deformation can affect the translation of
preoperative image-based surgical plans during surgery.
Having a realistic tissue deformation simulator could
enhance the accuracy of internal targets localization by
giving an accurate estimation of the deformation applied to
a preoperative model of the organ. A key challenge is to
address the sim-to-real gap between the simulator and the
actual intraoperative behaviour of the tissue. The
sim-to-real transfer challenge is addressed by formulating
the problem as a probabilistic inference over a
low-dimensional representation of deformed objects. The
proposed method utilizes a generative variational
autoencoder structure based on graph neural networks
(GNN-VAE) to generate a probabilistic low-dimensional
representation of the outputs of a physics-based simulator.
To match simulation data to real data, the resultant
low-dimensional distribution (i.e., prior distribution) is
updated iteratively using an Ensemble Smoother with
Multiple Data Assimilation (ES-MDA). The advantages of the
proposed method are 1) it only uses simulation data for
training the GNN-VAE, and no retraining of GNN-VAE is
required intraoperatively, 2) it does not require
estimating the mechanical properties of the tissue it is
simulating, and 3) is able to work with any physics based
simulator. The proposed framework was verified both in
experimental and simulation studies and showed it can
reduce the registration error in tissue deformation.

See our lab website for more details and a copy of the paper: http://www.ece.ualberta.ca/~tbs (go to Publications).