MARS-CLIP: Multi-Resolution and Attention Refined Zero-Shot Image Segmentation

CLIP is trained to align an entire image with a caption, so its image encoder is optimized for global representations rather than pixel-level detail. This creates two obstacles for dense prediction. First, insufficient spatial resolution: the encoder takes a fixed input size, so feature maps are of a fixed low resolution and existing methods such as MaskCLIP and SCLIP often fail to detect tiny objects or delineate complex boundaries. Second, loss of spatial structure: semantic abstraction in deep ViT layers dilutes the original shape information, and the spatial layouts preserved in intermediate layers, along with low-level cues such as edges and colors, are left unused.

MARS-CLIP attacks both problems at once with two strategies: a multi-resolution feature extraction module that fuses local fine-grained features with global context to circumvent the input resolution limit, and a structure-aware attention refinement mechanism that injects spatial and color biases into the final self-attention block to restore object boundaries. Unlike methods that borrow representations from external foundation models such as DINO or Stable Diffusion, our approach relies only on CLIP's own internal features, so it keeps the computational cost of running a single model.

Step 1 — Feature extraction with multi-resolution images. Given an input image \(I \in \mathbb{R}^{3 \times H \times W}\), we first apply reflective padding so that its dimensions become multiples of the crop size \(S_{\mathrm{crop}}\). The padded image is divided into an \(N \times M\) grid of local regions, and each region is resized to the encoder input resolution \(K \times K\). Because \(S_{\mathrm{crop}} < K\), this amounts to zooming into local regions, which is what lets us extract fine-grained detail. The resulting local feature maps are recombined according to their original spatial arrangement into a single high-resolution map \(F_{\mathrm{local}}\). Since processing regions independently discards global context, we also resize the whole image and encode it into \(F_{\mathrm{global}}\). The two are then integrated as

where \(\mathrm{Up}(\cdot)\) is bilinear upsampling that aligns the spatial resolutions and \(\alpha\) balances global against local information. We use \(S_{\mathrm{crop}} = 112\) and \(\alpha = 0.8\), selected by the ablation study below.

Step 2 — Structure-aware attention refinement. The final Transformer block is replaced by a refined attention block that injects spatial and color biases, described in the next subsection.

Step 3 — Zero-shot segmentation. Each class name of the target dataset is inserted into the 80 predefined ImageNet prompt templates, and the text feature \(F^{\mathrm{text}}_k\) for class \(k\) is the average of the resulting embeddings. We compute the cosine similarity between \(F^{\mathrm{text}}_k\) and the image feature at every pixel position, upsample the similarity map to the original image size, and assign to each pixel the class with the maximum score.

The self-attention maps in the final layer of CLIP's ViT encoder group semantics well but leave object boundaries spatially ambiguous. Following SCLIP, we use \(K K^{\top}\) similarity in place of the standard \(Q K^{\top}\), which resolves much of the spatial feature entanglement — yet boundary inconsistencies persist in complex scenes. We therefore add a bias term \(B_{\mathrm{total}} = B_{\mathrm{int}} + B_{\mathrm{color}}\) to the similarity matrix before the softmax.

The internal bias \(B_{\mathrm{int}}\) comes from the attention maps of an intermediate layer that still strongly preserves the spatial layout of the input. We use the 8th layer, which was the most stable choice on average; scores peak around it and decline in deeper layers, consistent with spatial layout being lost as feature abstraction progresses.

The external bias \(B_{\mathrm{color}}\) supplies precise boundary detail that carries no semantic discriminability of its own. We convert the input from sRGB to the CIELAB color space, downsample it to patch-level color features \(c_i \in \mathbb{R}^{3}\), and define the color affinity between patches \(i\) and \(j\) with a Gaussian kernel:

with bandwidth \(\sigma = 30.0\). The color bias is not applied to the class token, which has no spatial position. Finally, following the insight of ClearCLIP, we remove the residual connection and the feed-forward network from the final Transformer block. This keeps the original features from being dominated by the residual component and maximizes the effect of the refined attention.