OSError: Can't load tokenizer for 'StreamVLN_Video_qwen_1_5_r2r_rxr_envdrop_scalevln_v1_3'

Hi, thank you for your great work!

Issue: Missing Tokenizer Files in Downloaded Model

Problem Description
When running the evaluation script with the locally downloaded model, I encounter tokenizer loading errors. The model directory appears to be missing necessary tokenizer files.

Error Details

OSError: Can't load tokenizer for 'StreamVLN_Video_qwen_1_5_r2r_rxr_envdrop_scalevln_v1_3'. 
If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. 
Otherwise, make sure 'StreamVLN_Video_qwen_1_5_r2r_rxr_envdrop_scalevln_v1_3' is the correct path to a directory containing all relevant files for a LlamaTokenizerFast tokenizer.

The detailed information is as follows:

(streamvln) wjw@mat721-T640:~/why/StreamVLN$ sh ./scripts/streamvln_eval_offline_fixed.sh 
CHECKPOINT: StreamVLN_Video_qwen_1_5_r2r_rxr_envdrop_scalevln_v1_3
Running in FULL OFFLINE mode
Checking model files...
[2025-11-13 20:35:13,608] torch.distributed.run: [WARNING] 
[2025-11-13 20:35:13,608] torch.distributed.run: [WARNING] *****************************************
[2025-11-13 20:35:13,608] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
[2025-11-13 20:35:13,608] torch.distributed.run: [WARNING] *****************************************
Gym has been unmaintained since 2022 and does not support NumPy 2.0 amongst other critical functionality.
Please upgrade to Gymnasium, the maintained drop-in replacement of Gym, or contact the authors of your software and request that they upgrade.
See the migration guide at https://gymnasium.farama.org/introduction/migration_guide/ for additional information.
Gym has been unmaintained since 2022 and does not support NumPy 2.0 amongst other critical functionality.
Please upgrade to Gymnasium, the maintained drop-in replacement of Gym, or contact the authors of your software and request that they upgrade.
See the migration guide at https://gymnasium.farama.org/introduction/migration_guide/ for additional information.
Please install petrel_client to Client.
Please install petrel_client to Client.
| distributed init (rank 0): env://, gpu 0
| distributed init (rank 1): env://, gpu 1
Traceback (most recent call last):
  File "/home/wjw/why/StreamVLN/streamvln/streamvln_eval.py", line 584, in <module>
    eval()
  File "/home/wjw/why/StreamVLN/streamvln/streamvln_eval.py", line 519, in eval
    tokenizer = transformers.AutoTokenizer.from_pretrained(args.model_path,
  File "/home/wjw/miniconda3/envs/streamvln/lib/python3.9/site-packages/transformers/models/auto/tokenization_auto.py", line 926, in from_pretrained
    return tokenizer_class_fast.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "/home/wjw/miniconda3/envs/streamvln/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 2200, in from_pretrained
    raise EnvironmentError(
OSError: Can't load tokenizer for 'StreamVLN_Video_qwen_1_5_r2r_rxr_envdrop_scalevln_v1_3'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'StreamVLN_Video_qwen_1_5_r2r_rxr_envdrop_scalevln_v1_3' is the correct path to a directory containing all relevant files for a LlamaTokenizerFast tokenizer.
Traceback (most recent call last):
  File "/home/wjw/why/StreamVLN/streamvln/streamvln_eval.py", line 584, in <module>
    eval()
  File "/home/wjw/why/StreamVLN/streamvln/streamvln_eval.py", line 519, in eval
    tokenizer = transformers.AutoTokenizer.from_pretrained(args.model_path,
  File "/home/wjw/miniconda3/envs/streamvln/lib/python3.9/site-packages/transformers/models/auto/tokenization_auto.py", line 926, in from_pretrained
    return tokenizer_class_fast.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "/home/wjw/miniconda3/envs/streamvln/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 2200, in from_pretrained
    raise EnvironmentError(
OSError: Can't load tokenizer for 'StreamVLN_Video_qwen_1_5_r2r_rxr_envdrop_scalevln_v1_3'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'StreamVLN_Video_qwen_1_5_r2r_rxr_envdrop_scalevln_v1_3' is the correct path to a directory containing all relevant files for a LlamaTokenizerFast tokenizer.
[2025-11-13 20:35:23,639] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 1218707) of binary: /home/wjw/miniconda3/envs/streamvln/bin/python3.9
Traceback (most recent call last):
  File "/home/wjw/miniconda3/envs/streamvln/bin/torchrun", line 7, in <module>
    sys.exit(main())
  File "/home/wjw/miniconda3/envs/streamvln/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "/home/wjw/miniconda3/envs/streamvln/lib/python3.9/site-packages/torch/distributed/run.py", line 806, in main
    run(args)
  File "/home/wjw/miniconda3/envs/streamvln/lib/python3.9/site-packages/torch/distributed/run.py", line 797, in run
    elastic_launch(
  File "/home/wjw/miniconda3/envs/streamvln/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/wjw/miniconda3/envs/streamvln/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
streamvln/streamvln_eval.py FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2025-11-13_20:35:23
  host      : mat721-T640
  rank      : 1 (local_rank: 1)
  exitcode  : 1 (pid: 1218708)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2025-11-13_20:35:23
  host      : mat721-T640
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 1218707)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
(streamvln) wjw@mat721-T640:~/why/StreamVLN$

Thanks a lot! I've solved this error.

The tokenizer error occurred because the model checkpoint doesn't include tokenizer files.

I resolved this by downloading the tokenizer files from the Real-World Deployment model checkpoint.

Is the tokenizer for these two models the same?

多谢！我已经解决了这个错误。

分词器错误的原因是模型检查点不包含分词器文件。

我通过下载真实世界部署模型检查点的分词器文件解决了这个问题。

Navigation Menu

Repository navigation

OSError: Can't load tokenizer for 'StreamVLN_Video_qwen_1_5_r2r_rxr_envdrop_scalevln_v1_3' #64

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

OSError: Can't load tokenizer for 'StreamVLN_Video_qwen_1_5_r2r_rxr_envdrop_scalevln_v1_3' #64

Description

Activity

sheepwhy commented on Nov 13, 2025

sheepwhy commented on Nov 14, 2025

SereneNan commented on Jan 3, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions