VLM Classifier sample command/code not working

[rhsu-nuro]

Trying to run the very first command for the latest released VLM fine-tuning feature from tinker_cookbook/recipes/vlm_classifier:

python -m tinker_cookbook.recipes.vlm_classifier.train \
    experiment_dir=./vlm_classifier \
    wandb_project=vlm-classifier \
    dataset=caltech101 \
    renderer_name=qwen3_vl \
    model_name=Qwen/Qwen3-VL-30B-A3B-Instruct

Got me:

tinker.lib.retry_handler:222 [ERROR] Request failed with non-retryable error: BadRequestError: Error code: 400 - {'detail': 'Expected 63 tokens, got 70 from image'}

More terminal output:

tinker.lib.retry_handler:222 [ERROR] Request failed with non-retryable error: BadRequestError: Error code: 400 - {'detail': 'Expected 63 tokens, got 70 from image'}
tinker.lib.retry_handler:222 [ERROR] Request failed with non-retryable error: BadRequestError: Error code: 400 - {'detail': 'Expected 40 tokens, got 77 from image'}
tinker.lib.retry_handler:222 [ERROR] Request failed with non-retryable error: BadRequestError: Error code: 400 - {'detail': 'Expected 63 tokens, got 70 from image'}
tinker.lib.telemetry:204 [INFO] Exception logged for session ID: 0788e796-c9e5-5f41-b718-329641ca5a58
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/rhsu/bates-mcp/tinker-cookbook/tinker_cookbook/recipes/vlm_classifier/train.py", line 156, in <module>
    chz.nested_entrypoint(run_experiment)
  File "/opt/homebrew/lib/python3.11/site-packages/chz/blueprint/_entrypoint.py", line 39, in inner
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/chz/blueprint/_entrypoint.py", line 109, in nested_entrypoint
    return main(value)
           ^^^^^^^^^^^
  File "/Users/rhsu/bates-mcp/tinker-cookbook/tinker_cookbook/recipes/vlm_classifier/train.py", line 152, in run_experiment
    asyncio.run(train.main(config))
  File "/opt/homebrew/Cellar/python@3.11/3.11.14_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.14_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.14_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/Users/rhsu/bates-mcp/tinker-cookbook/tinker_cookbook/utils/trace.py", line 332, in async_wrapper
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rhsu/bates-mcp/tinker-cookbook/tinker_cookbook/supervised/train.py", line 349, in main
    submitted_batch = await submit_batch(epoch_idx, batch_idx)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rhsu/bates-mcp/tinker-cookbook/tinker_cookbook/utils/trace.py", line 332, in async_wrapper
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rhsu/bates-mcp/tinker-cookbook/tinker_cookbook/supervised/train.py", line 265, in submit_batch
    eval_metrics = await run_evals(evaluators, training_client, step)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rhsu/bates-mcp/tinker-cookbook/tinker_cookbook/utils/trace.py", line 332, in async_wrapper
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rhsu/bates-mcp/tinker-cookbook/tinker_cookbook/supervised/train.py", line 140, in run_evals
    eval_metrics = await run_evaluator(evaluator)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rhsu/bates-mcp/tinker-cookbook/tinker_cookbook/utils/trace.py", line 332, in async_wrapper
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rhsu/bates-mcp/tinker-cookbook/tinker_cookbook/supervised/train.py", line 135, in run_evaluator
    return await evaluator(sampling_client)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rhsu/bates-mcp/tinker-cookbook/tinker_cookbook/recipes/vlm_classifier/eval.py", line 214, in __call__
    outputs = await asyncio.gather(*async_tasks)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rhsu/bates-mcp/tinker-cookbook/tinker_cookbook/recipes/vlm_classifier/eval.py", line 196, in bounded_generate_output
    return await self.generate_output(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rhsu/bates-mcp/tinker-cookbook/tinker_cookbook/recipes/vlm_classifier/eval.py", line 144, in generate_output
    r: types.SampleResponse = await sampling_client.sample_async(
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/tinker/lib/public_interfaces/sampling_client.py", line 249, in sample_async
    return await AwaitableConcurrentFuture(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/tinker/lib/public_interfaces/api_future.py", line 132, in result_async
    return await asyncio.wrap_future(self._future)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/tinker/lib/telemetry.py", line 384, in _awrapper
    return await cast(Callable[..., Awaitable[R]], func)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/tinker/lib/public_interfaces/sampling_client.py", line 235, in _sample_async_with_retries
    return await self.retry_handler.execute(_sample_async)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/tinker/lib/retry_handler.py", line 136, in execute
    result = await self._execute_with_retry(func, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/tinker/lib/retry_handler.py", line 188, in _execute_with_retry
    result = await func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/tinker/lib/public_interfaces/sampling_client.py", line 225, in _sample_async
    return await self._sample_async_impl(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/tinker/lib/public_interfaces/sampling_client.py", line 168, in _sample_async_impl
    untyped_future = await self.holder.execute_with_retries(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/tinker/lib/internal_client_holder.py", line 364, in execute_with_retries
    raise e
  File "/opt/homebrew/lib/python3.11/site-packages/tinker/lib/internal_client_holder.py", line 325, in execute_with_retries
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/tinker/lib/public_interfaces/sampling_client.py", line 148, in _send_asample_request
    raise e
  File "/opt/homebrew/lib/python3.11/site-packages/tinker/lib/public_interfaces/sampling_client.py", line 140, in _send_asample_request
    return await client.sampling.asample(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/tinker/resources/sampling.py", line 57, in asample
    return await self._post(
           ^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/tinker/_base_client.py", line 1232, in post
    return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/tinker/_base_client.py", line 1033, in request
    raise self._make_status_error_from_response(err.response) from None
tinker.BadRequestError: Error code: 400 - {'detail': 'Expected 60 tokens, got 78 from image'}

[rhsu-nuro]

I also tried with Qwen/Qwen3-VL-235B-A22B-Instruct, but same issue.

[rhsu-nuro]

oh my, I think yeah it's problem with the caltech dataset. flowers102 dataset confirm to actually work rather.

[Kun-Xiang]

oh my, I think yeah it's problem with the caltech dataset. flowers102 dataset confirm to actually work rather.

hi, have you resolved the Caltech dataset issue? Is it due to image size or other reasons? I've already resized the images before input, but it didn't work. My images are in a PIL image list format, and it works fine in other RL frameworks, e.g. VeRL.

[danobi]

Hi, we're looking into this. As a workaround, you're allowed to drop the expected_tokens parameter in PublicImageChunk - those are simply advisory.