Smart microscopy, computer vision, and controlling cells with light -- my PhD thesis in 3 minutes

Work conducted at the University of Bern (Institute of Cell Biology) in the laboratory of Prof. Olivier Pertz.

Thesis abstract:
Biological signaling is inherently dynamic, spatially organized, and context-sensitive. To understand how cells encode and respond to signals, it is essential to move beyond passive observation. This thesis explores the idea that mechanistic insight can be gained by combining real-time measurement, adaptive perturbation, and spatial control in live cells. To this end, it presents a suite of tools for real-time feedback microscopy and demonstrates how these tools enable a new class of adaptive, high-resolution, and high-throughput experiments.

At the center of this work is a modular smart microscopy platform that integrates live imaging, real-time image analysis, and optogenetic stimulation within a closed-loop system. This real-time feedback control microscopy (RTM) setup enables the microscope to autonomously deliver stimuli in response to dynamic features such as biosensor activity, cell shape, or subcellular structures. In contrast to traditional methods, RTM allows for conditional perturbations with high spatial and temporal resolution, significantly enhancing experimental precision and throughput.

A key component of this system is rapid and adaptable image segmentation. To address variability in cell types, imaging modalities, and experimental conditions, we developed Convpaint, an interactive pixel classifier that leverages pretrained deep neural networks to extract image features. Originally designed to support RTM, Convpaint has proven broadly useful across a variety of segmentation tasks. Its ability to adapt quickly without retraining makes it ideal for fast, user-guided segmentation in dynamic settings.

In parallel, the physical variability of cellular environments, such as morphology changes induced by external conditions, posed challenges for long-term imaging and precise stimulation. To address this, we developed a microfabrication technique that enables rapid prototyping of microstructured environments using inexpensive materials and standard lab equipment, without the need for cleanroom facilities. By repurposing existing microscope setups, it provides a cost-effective alternative to commercial photolithography and allows labs to create custom-patterned substrates for a variety of cell biology applications in house.

These tools were applied to investigate spatial signaling dynamics in the ERK and Rho GTPase pathways: In the MAPK/ERK system, closed-loop optogenetics was used to study how local receptor tyrosine kinase (RTK) activation leads to downstream ERK activity. The results revealed excitable dynamics with spatial thresholds that depend on cell geometry, suggesting that MAPK/ERK functions as a spatially excitable system rather than a simple global integrator.
In the Rho GTPase system, real-time detection and stimulation of cytoskeletal structures were used to examine the feedback between contractility and adhesion. The RhoGAP DLC1 was found to regulate RhoA activity and focal adhesion disassembly in a mechanically responsive manner.

Beyond these specific studies, the thesis proposes a broader vision for smart microscopy: open, modular platforms that combine sensing, computation, and actuation. In collaboration with partners from academia and industry, we developed a community roadmap that outlines strategies for standardization and interoperability. Future directions include the \textbf{development of natural language interfaces and autonomous experimental design} to further increase the accessibility and scalability of smart microscopy. Together, the tools and concepts presented in this thesis make adaptive experimentation a practical and accessible approach for live-cell imaging. By integrating real-time sensing, computation, and actuation within a modular framework, this work demonstrates how smart microscopy can uncover previously hidden mechanisms in subcellular signaling. Reframing the microscope as an active participant that responds to cellular dynamics, rather than a passive observer, enables precise, automated, and mechanistically insightful experiments in cell biology.