S2 pretrained model of InternVideo2 does not work well for Zero-Shot Video-Text Retrieval

直接跑demo/demo.ipynb, 模型选用https://huggingface.co/OpenGVLab/InternVideo2-Stage2_1B-224p-f4/blob/main/InternVideo2-stage2_1b-224p-f4.pt 发现效果不太理想。
首先需要修改两个地方才能正确加载模型：
1、demo/demo.ipynb 中在setup_internvideo2(config)前面加上一句 config['pretrained_path'] = model_pth
2、demo/utils.py 第82和84行改成is_pretrain=True
修改后demo中提供的视频和10个句子的相似度分数（不经过softmax）为：

可以发现分数最高者并不是正确的描述，同时十个句子得分都比较接近。

我也遇到了类似的问题

提供的示例是个极端难的样例。这些句子是通过从视频中提取的关键元素（例如狗、人类、雪、玩耍）由GPT生成，来描述视频内容的，所以它们本身很多意思都很接近。我们打算利用这个案例研究来展示我们对进一步深入理解运动描述中的微妙区别的兴趣。

正常的视频检索，看看试试我们模型在主流基准上的测试或者试一些正常的例子可以感受下它的效果。

The provided example poses a considerable challenge for video understanding models due to its high level of complexity. These sentences have been generated by GPT, utilizing key elements such as dogs, humans, snow, and play that were extracted from the video in order to accurately capture the essence of the footage. We intend to leverage this case study to demonstrate our commitment to the deep understanding of subtle and detailed motion descriptions in the future.

Furthermore, we encourage you to evaluate our performance in typical video retrieval cases, as it consistently ranks among the top performers across mainstream benchmarks.

重复运行demo代码，发现结果每次都不一样。模型内部有随机化流程，还是和seed有关？
下面是连续两次跑的结果

在导入包之后加上下面的代码，如果直接运行能够保证结果一样。但是模型内部应该还有随机化流程，如果同样的seed在运行一次加载权重，结果就会变。
seed = 4491734
print("Seed:", seed)

np.random.seed(seed)
torch.manual_seed(seed)
if torch.cuda.is_available():
torch.cuda.manual_seed_all(seed)

@SHYuanBest

@SHYuanBest @1093842024 @Wenju-Huang 之前的demo加载权重有些问题，实际上没加载对预训练权重，现已修复，模型内部是没有随机化流程的，多次运行的轻微差异可能是由于Pytorch的计算误差。

@SHYuanBest

我上面是修复模型加载问题后的结果，只不过是没有经softmax做归一化，加上softmax后结果和您的还是不一样，请问你测试的是https://huggingface.co/OpenGVLab/InternVideo2-Stage2_1B-224p-f4/blob/main/InternVideo2-stage2_1b-224p-f4.pt 这个模型吗。

@SHYuanBest

是的，请问是得到的分数排序不一样吗，我理解只要排序一样，分数有细微差异也可以接受，这个问题其实我们很早就发现了，可能是代码隐式的包含了一些并行加速模块之类的，导致每次计算有一些随机波动，但是不影响最终排序结果

@SHYuanBest

顺序也不一样，我得到的结果是这样的

@SHYuanBest

Yes, the results are very relevant to the setting of the random number seed.

Can you release the 6B model? Although it may not improve much.

@SHYuanBest

如果在加载完模型后更换多个随机数种子，结果还是会发生变化吗，如果会的话，说明确实是计算误差，反之可能是因为权重没加载对导致有权重是随机初始化的。

@SHYuanBest

在导入包之后加上下面的代码，如果直接运行能够保证结果一样。但是模型内部应该还有随机化流程，如果同样的seed在运行一次加载权重，结果就会变。
seed = 4491734
print("Seed:", seed)

np.random.seed(seed)
torch.manual_seed(seed)
if torch.cuda.is_available():
torch.cuda.manual_seed_all(seed)

@shepnerd

@shepnerd @Wenju-Huang @1093842024 @SHYuanBest I find this bug is because we forget add model.eval() in demo, we need to enable it to turn off drop path, now this bug was fixed.

@shepnerd

I changed the text_candidates to 700 list of kinetics class names but no playing with dog or play snow likewise is picked for example.mp4 . Could you help me. thx :P

@SHYuanBest

我用最新代码也遇到了同样的问题，顺序不对，utils中的model是经过eval的

@SHYuanBest

stage2这个版本不支持中文，要支持中文需要用clip版本模型

@leexinhao

@leexinhao
Hello, i have same problem. I got different question when run inference twice. I have use the lastet code.
I notice that there two place to set model weights in config file internvideo2_stage2_config.py .
First place is visionencoder:

Second place is pretrained_path:

I don't know how to set model weights in this two places? All use https://huggingface.co/OpenGVLab/InternVideo2-Stage2_1B-224p-f4/blob/main/InternVideo2-stage2_1b-224p-f4.pt?

If not set Second place config item pretrained_path, Below moudle params will be random.

I think this is the reason why above friends met.

So, please tell me how to correctly set model weights in first place and second place.

Thanks for you reply.

@leexinhao

@leexinhao
Another question is I want to retrieval video by chinese. Above, you mentioned should use clip, but why the clip weights(https://huggingface.co/OpenGVLab/InternVideo2-CLIP-1B-224p-f8/tree/main) is so small.

Thanks for you reply.

@SHYuanBest

请问一下啥叫权重没有加载对呀？

@SHYuanBest

你好请问你解决了吗？我跑出来的顺序和prob的数值几乎和你一摸一样，该如何解决呢？

@leexinhao

Actually, you only need to set second pretrained_path, which covers all model weights of vision encoder and text encoder.

@SHYuanBest

就是加载权重的位置有问题，应该是参考demo.ipynb里面的路径加载，不需要修改config

@leexinhao

Because we use lora to train llm, and we only train the last layer of InternVideo2-stage2 to get the InternVideo2-CLIP, so you need load the pretrained weight of InternVideo2-stage2 and the llm weights as we mention in README

I am facing the same issue. I have tried setting the seed as well but no luck. I am using this weight file InternVideo2-stage2_1b-224p-f4.pt


text: A man in a gray hat and coat walks through the snowy yard, carefully navigating around the trees. ~ prob: 0.5865
text: A pet dog excitedly runs through the snowy yard, chasing a toy thrown by its owner. ~ prob: 0.1185
text: A person stands on the snowy floor, pushing a sled loaded with blankets, preparing for a fun-filled ride. ~ prob: 0.0954
text: A man in a gray coat walks through the snowy landscape, pulling a sleigh loaded with toys. ~ prob: 0.0674
text: A person bundled up in a blanket walks through the snowy landscape, enjoying the serene winter scenery. ~ prob: 0.0446
text: A playful dog and its owner wrestle in the snowy yard, chasing each other with joyous abandon. ~ prob: 0.0284
text: A person in a blue jacket walks their pet on a leash, enjoying a peaceful winter walk among the trees. ~ prob: 0.0189
text: A man in a gray sweater plays fetch with his dog in the snowy yard, throwing a toy and watching it run. ~ prob: 0.0188
text: A playful dog slides down a snowy hill, wagging its tail with delight. ~ prob: 0.0128
text: A person dressed in a blue jacket shovels the snow-covered pavement outside their house. ~ prob: 0.0086

我也遇到了同样的问题，每次结果不一样

所以最后这个问题修复了么

如果想复现论文的video-retrieval的数值，可以跑通作者提供的代码，数值和论文几乎对的上。

I had the same issue of getting different results each time. I was able to solve it by

setting model.eval() in demo code
setting correct model path in internvideo2_stage2_config.py in two places: model.vision_encoder.pretrained and pretrained_path

@leexinhao

thanks for your guide. I solved the issue

But it's quite confusing to config model path for twice(model.vision_encoder.pretrained and pretrained_path).
@leexinhao maybe it's better to merge the two config

When I test the model with a video showing a person riding a bicycle and several text descriptions, such as
['someone running in a campus walkway.', 'a van or vehicle driving in the campus pedestrian area.', 'someone is riding a bicycle.'],
the retrieval results seem poor—the correct sentence gets a lower similarity score. I’ve been confused about this issue for several days.
This is one frame of the video.

Could anyone please help me figure it out? Or could you share the WeChat group QR code?

Navigation Menu

Repository navigation

S2 pretrained model of InternVideo2 does not work well for Zero-Shot Video-Text Retrieval #107

2 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

S2 pretrained model of InternVideo2 does not work well for Zero-Shot Video-Text Retrieval #107

Description

Activity

SHYuanBest commented on Apr 22, 2024

shepnerd commented on Apr 22, 2024

1093842024 commented on Apr 22, 2024

SHYuanBest commented on Apr 23, 2024

leexinhao commented on Apr 23, 2024

Wenju-Huang commented on Apr 25, 2024

leexinhao commented on Apr 26, 2024

Wenju-Huang commented on Apr 28, 2024

SHYuanBest commented on Apr 28, 2024

SHYuanBest commented on Apr 28, 2024

leexinhao commented on Apr 28, 2024

SHYuanBest commented on Apr 29, 2024

leexinhao commented on Apr 29, 2024

Major1994 commented on May 23, 2024

normanShan0917 commented on Oct 15, 2024

2 remaining items

leexinhao commented on Oct 30, 2024

payne4handsome commented on Nov 8, 2024

payne4handsome commented on Nov 8, 2024

Volibear1234 commented on Nov 18, 2024

Volibear1234 commented on Nov 18, 2024

leexinhao commented on Nov 19, 2024

leexinhao commented on Nov 19, 2024

leexinhao commented on Nov 19, 2024

rishabh-akridata commented on Feb 18, 2025

mazhengyu8282 commented on Apr 8, 2025

mazhengyu8282 commented on Apr 8, 2025

Volibear1234 commented on Apr 8, 2025

prajwalgatti commented on Jul 22, 2025

MrWH123 commented on Aug 2, 2025

shyern commented on Nov 9, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions