Meme search
Describe a situation in words. The search finds a blank meme template that fits it and puts your words into the text boxes.
Model: CLIP ViT-B/16, fine-tuned · Data: MemeCap (6,416 memes; 2,000 templates in this demo) · Runs in your browser
Try it
The first visit downloads the text encoder (127 MB) and the template index (5 MB). Your browser keeps them for the next visit. Your text stays on your computer.
What the project does
Most meme search tools read the text that is printed on the meme. A blank template has no text, so we must find it from the picture only. This project teaches CLIP to connect a situation ("my code compiled after 6 hours") with the picture of the right template.
- Remove the text from every meme. This gives a blank template and the positions of the text boxes.
- Fine-tune CLIP on pairs of (blank template, description of the meme).
- Compute one vector for each template, one time, and save the vectors.
- For a search: compute one vector for your text and find the nearest template vectors.
- Write your text into the text boxes of the template.
Architecture
CLIP in one paragraph
CLIP has two encoders: one for images and one for text. Each encoder gives a vector with 512 numbers. During training, CLIP learns to make the vector of an image and the vector of its caption point in the same direction. After training, you can compare any text with any image: a large dot product means a good match. We use the ViT-B/16 version from OpenAI.
The search index is a flat file
There is no vector database. The index is one file with one row of 512 numbers for each template. The browser computes the dot product of your text vector with every row. That takes a few milliseconds. A vector database becomes useful at about 100,000 rows or more.
The web demo uses 2,000 of the 6,416 templates. First we remove near-duplicate reposts. Then we group the
remaining templates into 2,000 clusters with k-means on their CLIP vectors, and we keep the template nearest to the center of each cluster.
This keeps every kind of meme, with fewer files to download. The code is in web/make_web_subset.py.
Some templates are almost the same image (reposts). If two results have a cosine similarity above 0.95, we show only one.
Filling the text boxes
The OCR step saved the position of each text box, its line height and its text color. The page uses this data to draw your text in the correct place.
- One text box. If your full text fits, we use it. If not, we remove short common words ("the", "a", "my"). Then we remove the least important words until the text fits.
- Word importance. We remove one word at a time and compute the match score again. If the score drops a lot, the word is important. This is a "leave-one-out" test. The chips under each result show the most important words.
- Two or more text boxes. Your text goes into the largest box. The most important keywords go into the other boxes. You can edit every box.
Training
Step 1: make blank templates
- Find the text. Florence-2 (a small vision-language model) reads the text and gives a box for each line.
- Make a mask. The text boxes become a black-and-white mask.
- Paint over the text. LaMa (an inpainting model) fills the masked area with a background that matches the picture.
This step takes about 30 minutes on one A100 GPU for all 6,416 memes (6 processes in parallel).
Step 2: fine-tune CLIP
MemeCap gives three captions for each meme:
| Caption type | Example |
|---|---|
| meaning | "The poster feels happy because the code works." |
| title | The title of the Reddit post. |
| literal | "A man is smiling and holding a cup." |
Each training example is one blank template and one random caption of the three types. The loss is the standard CLIP contrastive loss: in a batch of 128 pairs, each template must match its own caption and not the other 127.
| Setting | Value |
|---|---|
| Start weights | OpenAI CLIP ViT-B/16 (open_clip) |
| Trained parts | All weights, image and text encoders |
| Optimizer | AdamW, learning rate 1e-5, weight decay 0.1, 5% warm-up |
| Batch / epochs | 128 pairs, up to 10 epochs |
| Split | 5,323 train, 500 validation (picks the checkpoint), 559 test |
python remove_text.py --n 0 --shard 0/6 # x6 shards, blank templates + text boxes python train_clip.py --epochs 10 --out runs/clip_b16 python web/export_web.py # text encoder -> ONNX, template vectors, thumbnails
Results
Text-to-image Recall@10 on the 559 test memes: how often the correct template is in the top 10.
| meaning | title | literal | |
|---|---|---|---|
| Zero-shot CLIP, original memes (text visible) | 80.3 | 36.9 | 62.9 |
| Zero-shot CLIP, text removed | 25.2 | 17.0 | 80.5 |
| Fine-tuned CLIP, text removed | 34.8 | 17.5 | 93.2 |
- CLIP reads text. Most of the zero-shot "meaning" score comes from the printed text. Without the text, the score drops from 80 to 25.
- Removing text helps visual search. For "literal" captions, the score goes up from 63 to 80. The text was noise for these queries.
- Fine-tuning helps. "meaning" goes from 25 to 35, and "literal" goes to 93. The best checkpoint is at epoch 3. After that, the model overfits (there are only about 5,300 training images).
- There is a limit. Many "meaning" captions describe the joke in the text. That joke is gone from the blank template, so no model can find it from pixels only.
Running it in the browser
- We export only the text encoder to ONNX. The image encoder is not necessary in the browser.
- We convert the weights to fp16 (127 MB). On our test queries, the fp16 vectors have a cosine similarity of 1.0000 with fp32 and give the same top 6 templates.
- int8 (64 MB) changes the vectors too much (cosine about 0.96), so we do not use it.
- transformers.js runs the tokenizer and the model.
Papers and links
- Learning Transferable Visual Models From Natural Language Supervision (CLIP) (Radford et al., 2021). The model and the contrastive loss.
- MemeCap: A Dataset for Captioning and Interpreting Memes (Hwang and Shwartz, 2023). The memes and the three caption types.
- Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks (Xiao et al., 2023). OCR with boxes.
- Resolution-robust Large Mask Inpainting with Fourier Convolutions (LaMa) (Suvorov et al., 2021). Removes the text.
- OpenCLIP. The training code for CLIP.