Robust deep learning optical autofocus system applied to automated multiwell plate single molecule localization microscopy. (13th August 2021)

• Record Type: Journal Article
• Authors: Lightley, Jonathan; Görlitz, Frederik; Kumar, Sunil; Kalita, Ranjan; Kolbeinsson, Arinbjorn; Garcia, Edwin; Alexandrov, Yuriy; Bousgouni, Vicky; Wysoczanski, Riccardo; Barnes, Peter; Donnelly, Louise; Bakal, Chris; Dunsby, Christopher; Neil, Mark A.A.; Flaxman, Seth; French, Paul M.W.
• Is Part Of: Journal of microscopy. Volume 288:Part 2(2022)
• Page Start: 130
• Page End: 141
• Publication Date: 2021-08-13
• Subjects: Optical Autofocus -- Automated imaging -- High content analysis -- super‐resolved microscopy -- STORM -- SMLM; Microscopy -- Periodicals; 502.82
• DOI: 10.1111/jmi.13020
• Languages: English
• ISSNs: 0022-2720

Abstract

Abstract: We presenta robust, long‐range optical autofocus system for microscopy utilizing machine learning. This can be useful for experiments with long image data acquisition times that may be impacted by defocusing resulting from drift of components, for example due to changes in temperature or mechanical drift. It is also useful for automated slide scanning or multiwell plate imaging where the sample(s) to be imaged may not be in the same horizontal plane throughout the image data acquisition. To address the impact of (thermal or mechanical) fluctuations over time in the optical autofocus system itself, we utilize a convolutional neural network (CNN) that is trained over multiple days to account for such fluctuations. To address the trade‐off between axial precision and range of the autofocus, we implement orthogonal optical readouts with separate CNN training data, thereby achieving an accuracy well within the 600 nm depth of field of our 1.3 numerical aperture objective lens over a defocus range of up to approximately +/–100 μm. We characterize the performance of this autofocus system and demonstrate its application to automated multiwell plate single molecule localization microscopy. Lay Description: Many microscopy experiments involve extended imaging of samples over timescales from minutes to days, during which the microscope can 'drift' out of focus. When imaging at high magnification, the depth of field is of the order of one micron and so the imaging system should keep the sample in the focal plane of the microscope objective lens to this precision. Unfortunately, temperature changes in the laboratory can cause thermal expansion of microscope components that can move the focal plane by more than a micron and such changes can occur on a timescale of minutes. This is a particular issue for super‐resolved microscopy experiments using single molecule localization microscopy (SMLM) techniques, for which 1000s of images are acquired, and for automated imaging of multiple samples in multiwell plates. It is possible to maintain the sample in the focal plane focus position by either automatically moving the sample or adjusting the imaging system, for example by moving the objective lens. This is called 'autofocus' and is frequently achieved by reflecting a light beam from the microscope coverslip and measuring its position of beam profile as a function of defocus of the microscope. The correcting adjustment is then usually calculated analytically but there is recent interest in using machine learning techniques to determine the required focussing adjustment. Here, we present a system that uses a neural network to determine the required defocus correcting adjustment from camera images of a laser beam that is reflected from the coverslip. Unfortunately, this approach will only work when the microscope is in the same condition as it was when the neural network was trained – and this can be compromised by the same drift of the optical system that causes the defocus needing to be corrected. We show, however, that by training a neural network over an extended period, for example 10 days, this approach can 'learn' about the optical system drifts and provide the required autofocus function. We also show that an optical system utilizing a rectangular slit can make two measurements of the defocus simultaneously, with one measurement being optimized for high accuracy over a limited range (±10 μm) near focus and the other providing lower accuracy but over a much longer range (±100 μm). This robust autofocus system is suitable for automated super‐resolved microscopy of arrays of samples in a multiwell plate using SMLM, for which an experiment routinely lasts more than 5 h.