Real action pose collection 01 model scan

As the tool is always partly occluded by the hand, when in use, taking hand tracking into account could be beneficial to estimate the tool pose. The spatial localization of the tool combined with a co-registration of a preoperative plan or intra-operative medical imaging to the patient anatomy enables surgical guidance to improve the outcome of the intervention and reduce the radiation exposure for both the patient and the surgical staff. Instrument pose estimation is an essential part of computer aided surgery and is deployed in state-of-the-art surgical navigation systems, as well as augmented reality systems, through optical tracking to localize a surgical instrument in the 3D space of the operating theatre. Monocular RGB video is still the most common optical system used in todayвЂ™s operating rooms and is employed for surgical education, performance enhancement, and error analysis. As a consequence, there are no publicly available clinical datasets yet. However, regulations and the risk of patient compromise make it challenging to collect sufficient amounts of training data to develop robust and generalizable methods. Visual 3D pose recognition of surgical tools, the patient anatomy, but also of the surgical staff in video data is becoming increasingly important in clinical research. Our realistic synthetic data generation pipeline may contribute to overcome the data bottleneck in the surgical domain and can easily be transferred to other medical applications. We present three baseline models for RGB based object and object/hand pose estimation based on RGB frames. To the best of our knowledge, we propose the first synthetic and real data generation pipelines to generate hand and object pose labels for open surgery. The best performing baseline achieves an average tool 3D vertex error of 16.7 mm on synthetic data as well as 13.8 mm on real data which is comparable to the state-of-the art in RGB-based hand/object pose estimation. We evaluate three baseline models on the proposed datasets. We furthermore present three state-of-the-art RGB-based pose estimation baselines. Subsequently, we propose a pipeline to capture and label real data with hand and object pose ground truth in an experimental setup to gather high-quality real data. We developed a rendering pipeline to create inexpensive and realistic synthetic data for model pretraining.

In this work, we present a data generation framework, dataset and baseline methods to facilitate further research in the direction of markerless hand and instrument pose estimation in realistic surgical scenarios. Tracking of tools and surgical activity is becoming more and more important in the context of computer assisted surgery.

Real action pose collection 01 model scan