PlankFormer: Robust Plankton Instance Segmentation via MAE-Pretrained Vision Transformers and Pseudo Community Image Generation
Abstract
Plankton monitoring is essential for assessing aquatic ecosystems but is limited by the labor-intensive nature of manual microscopic analysis. Automating the segmentation of plankton from crowded images is crucial, however, it faces two major challenges: (i) the scarcity of pixel-level annotated datasets and (ii) the difficulty of distinguishing plankton from debris and overlapping individuals using conventional CNN-based methods. To address these issues, we propose PlankFormer, a novel framework for plankton instance segmentation. First, to overcome the data shortage, we introduce a method to generate labeled Pseudo Community Images (PCI) by synthesizing individual plankton images onto diverse backgrounds, including those created by generative models. Second, we propose a segmentation model utilizing a Vision Transformer (ViT) backbone with a Mask2Former decoder. To robustly capture the global structural features of plankton against occlusion and debris, we employ a Masked Autoencoder (MAE) for self-supervised pretraining on unlabeled individual images. Experimental results on realworld datasets demonstrate that our method significantly outperforms conventional methods, such as Mask R-CNN, particularly in challenging environments with high debris density. We demonstrate that our synthetic training strategy and MAE-based architecture enable highprecision segmentation with requiring less manual annotations for individual plankton images.
Pseudo Community Images (PCI)
We propose a method to automatically generate labeled “Pseudo Community Images (PCI)” by synthesizing a small number of labeled individual plankton images onto background images. In our method, PCI and their corresponding ground truth labels are automatically generated by compositing pixel-level labeled individual images onto various backgrounds. During synthesis, diverse plankton variations are reproduced by applying random flipping, rotation, and resizing to the individual images. Furthermore, to improve the domain diversity of PCI, we use not only real background regions extracted from actual community images but also images generated by generative models.
Figure 1: Overview of the Pseudo Community Image (PCI) generation process.
PlankFormer
Capturing global image features is essential to separate and detect plankton in community images where debris and overlapping individuals are significant. We propose PlankFormer, a plankton detection method based on global feature representation using an instance segmentation model that adopts a Vision transfirmer (ViT) as the encoder. A key feature of the proposed method is the introduction of pre-training for the model encoder using a large-scale set of individual images. Specifically, we employ Masked Autoencoder (MAE), which can learn structural and shape features from unlabeled data, as the pre-training method. This enables high generalization performance even when the segmentation model is fine-tuned using only PCI synthesized from a small number of labeled individual images.
Figure 2: Overview of the plankton image segmentation method PlankFormer.
Evaluation
In this experiment, to demonstrate the effectiveness of the proposed method for plankton detection, we compare detection accuracy with conventional instance segmentation methods: Mask R-CNN and Mask2Former. To verify the effectiveness of pre-training using MAE in the proposed method, we conduct comparative experiments under the following conditions: (i) no pre-training, (ii) pre-training using the UrFound framework, (iii) pre-training using the MoCo framework, and (iv) pre-training via an individual image classification task (Family-level classification). Note that when pre-training with individual plankton images is not performed, weights pre-trained on ImageNet are used as the initialization. In this experiment, to evaluate model performance under different environmental conditions, we employed two evaluation datasets: “Biwako” with many small individuals and heavy debris, and “Tsuruike” with larger individuals and minimal debris.
Table 1: Experimental results on the “Biwako” dataset. Bold indicates the best results, and underlined indicates the second-best results.
Table 2: Experimental results on the “Tsuruike” dataset. Bold indicates the best results, and underlined indicates the second-best results.
Figure 3: Example of segmentation results (zoomed-in views). Red dashed circles indicate false positives and false negatives.
BibTeX
@article{Miyazaki2026PlankFormerRP,
title={"PlankFormer: Robust Plankton Instance Segmentation via MAE-Pretrained Vision Transformers and Pseudo Community Image Generation"},
author={"Miyazaki, Masaharu and Otake, Yurie and Ito, Koichi and Makino, Wwataru, and Urabe, Jotaro and Aoki, Takafumi"},
journal={"2026 28th International Conference on Pattern Recognition (ICPR)"},
year={"2026"},
month={aug}
}