MetaPer

Personalized Retrieval Results

Language Queries	Top-5 Personalized Retrievals
a man is riding <Casey's boosted board> and wearing white t-shirt and gray shorts
<Zak's dog Kona> is playing with a black and white dog on the grass
<Zak's dog Coffee> is lying down in front of a man and three women
<Casey's friend Marlan> is standing on the 2nd floor with a woman
a woman is wearing <Alex's hat> on the grass with black fence behind

Figure 1. Given a video where a user-specific instance, e.g., "My dog Biscuit" is mentioned, our method automatically learns a representation for the user-specific instance in the VLM's text input space.

TL;DR

The problem: VLMs support category-level queries but struggle with personalized searches for specific object instances (e.g., "My dog Biscuit") in video.
Our approach: Meta-personalization learns how to learn novel word embeddings at test time, representing each instance as a combination of shared category features and instance-specific weights.
The payoff: Fully automatic training from video transcripts, no human supervision. 15% relative improvement over state of the art on DeepFashion2, plus a new This-Is-My benchmark.

Abstract

Large-scale vision-language models (VLM) have shown impressive results for language-guided search applications. While these models allow category-level queries, they currently struggle with personalized searches for moments in a video where a specific object instance such as "My dog Biscuit" appears. We present the following three contributions to address this problem. First, we describe a method to meta-personalize a pre-trained VLM, learning how to learn to personalize a VLM at test time to search in video. Our method extends the VLM's token vocabulary by learning novel word embeddings specific to each instance. To capture only instance-specific features, we represent each instance embedding as a combination of shared and learned global category features. Second, we propose to learn such personalization without explicit human supervision. Our approach automatically identifies moments of named visual instances in video using transcripts and vision-language similarity in the VLM's embedding space. Finally, we introduce This-Is-My, a personal video instance retrieval benchmark. We evaluate our approach on This-Is-My and DeepFashion2 and show that we obtain a 15% relative improvement over the state of the art on the latter dataset.

Key Findings

Meta-Personalization Works

Learning how to personalize at meta-training time enables effective test-time adaptation to novel user-specific instances, achieving 15% relative improvement over the state of the art.

No Human Supervision Needed

Named instances are automatically mined from video transcripts using possessive patterns and vision-language similarity, eliminating the need for manual annotation.

Category + Instance Decomposition

Representing instance tokens as $w = Cz$ (shared category features $C$ combined with instance weights $z$) disentangles category-level and instance-level information.

Our Meta-Personalized VLM

Personalized Vision-Language Model Overview: Our model enhances a pre-trained VLM (blue) with personalized instance tokens $w = Cz$ (orange), combining global category features $C$ (green) with instance-specific weights $z \in Z$ (yellow). Our personalized instance retrieval approach consists of three stages. First, we train global category features $C_D$ using automatically collected named personal instances from videos, a process known as Meta-Personalization (left). In the second step (middle), we adapt the meta-personalized category features $C_D$ during test-time and learn novel instance weights $z \in Z_P$ to represent user-specific instances, expressed as $w = C_P z$. Finally (right), we utilize the (frozen) personalized instance tokens $w$ in natural language queries during retrieval.

Personalized VLM overview with three stages: meta-personalization, test-time adaptation, and retrieval

Figure 2. Personalized VLM overview. Left: meta-personalization training. Middle: test-time adaptation. Right: personalized retrieval.

Model Overview

Our model $\mathcal{M}_{C,z}$ extends CLIP's language input vocabulary with $n_w$ novel instance-specific tokens $w_i^y = C_l z_i^y$, which we model as a linear combination of meta-personalized category features $C_l$ with weights $z_i^y$.

Figure 3. Model overview. Instance-specific tokens are generated as a linear combination of meta-personalized category features with instance weights.

Automatic Mining

Step 1: finds named instances via string-matching of possessive patterns in video transcripts. Step 2: filters non-visual instances using text-to-visual relevance between the instance name and the neighboring shots. Step 3: retrieves additional shots with high visual similarity to the instance reference shot.

Figure 4. Automatic mining pipeline. Step 1: possessive pattern matching. Step 2: text-to-visual filtering. Step 3: visual similarity retrieval.

This-Is-My Dataset

Examples from This-Is-My { Meta-Personalization (top) vs Test-time personalization (bottom-left) vs Query-time (bottom-right) } datasets. In the Query-time dataset (bottom-right), we design a challenging video instance retrieval task.

Figure 5. Examples from the This-Is-My dataset. Top: meta-personalization. Bottom-left: test-time personalization. Bottom-right: query-time retrieval.

BibTeX

@inproceedings{yeh2023meta,
  title={Meta-Personalizing Vision-Language Models To Find Named Instances in Video},
  author={Yeh, Chun-Hsiao and Russell, Bryan and Sivic, Josef and Heilbron, Fabian Caba and Jenni, Simon},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={19123--19132},
  year={2023}
}

Meta-Personalizing Vision-Language Models to Find Named Instances in Video