Amodal Segmentation through Out-of-Task and Out-of-Distribution with a Bayesian Model#
Authors: Yihong Sun, Adam Kortylewski, Alan Yuille
Affiliations: Johns Hopkins University
CVPR, 2022
Links: project
Summary#
Amodal segmentation aims to segment object boundaries which are occluded and hence invisible. The authors formulate amodal segmentation as an out-of-task and out-of-distribution generalization problem. The model is trained from non-occluded images using bounding box annotations and class labels only, but is applied to generalize out-of-task to object segmentation and to generalize out-of-distribution to segment occluded objects. The proposed approach outperfoms alternate methods with the same supervision by a wide margin, and even outperforms supervised methods when the occlusion is large.
Key Ideas#
The model taks as input a feature map \(\bar{F} = \psi(I, \eta)\) where \(I\) is the input image and \(\zeta\) are the neural network weights. Let the features within a given bounding box \(\mathcal{D}\) be \(F = \{ f_a: a \in \mathcal{D} \}\).
A Bayesian model for amodal segmentation. The authors introduce a latent variable \(w_a \in \{0, 1\}\) to indicate foreground/background which are learnt without additional supervision.
The foreground and background model \(P_a(f_a \mid y, m), B_a(f_a \mid y, m)\) are mixtures of von Mises Fisher distributions.
Shape modeling. The authors also introduce shape priors \(P(\overrightarrow{w} \mid y, m) = \prod_{a \in \mathcal{D} P_a(w_a \mid y, m)\), a learned 2D spatial map conditioned on the object category \(y\) and the class mixture \(m\). Finally this gives a generative model of this data:
Technical Details#
Notes#
References#
[1] Y. Sun, A. Kortylewski, A. Yuille. “Amodal segmentation through out-of-task and out-of-distribution with a bayesian model.”. In CVPR, 2021.