[rollout, vllm] fix: make LoRA with async vLLM work properly

[listar2000]

What does this PR do?

Add concise overview of what this PR aims to achieve or accomplish. Reference related GitHub issues and PRs that help with the review.

The previous #3639 addressed the crashing issues in update_weights of vLLMAsyncRollout. However, experiments (see Tests below) reveal an implicit off-policy issue: the rollout generation still uses the base model instead of the updated LoRA model, resulting in degraded performance. We traced this to a bug in vllm_async_server.vLLMHttpServerBase causing a mismatch between LoRA updates and rollout generation. Specifically:

• In vLLMAsyncRollout, update_weights correctly updates LoRA weights from the FSDP actor to the rollout AsyncLLM engine. However, the updated adapter is assigned a random lora_name and lora_int_id (generated from time.ns()), which are not stored—making them hard to reuse.
  

  verl/verl/workers/rollout/vllm_rollout/vllm_rollout_spmd.py

  Lines 595 to 604 in f209c6f

  	if peft_config and base_sync_done:
  	lora_int_id = int(time.time_ns() % 0x7FFFFFFF)
  	lora_reqest = TensorLoRARequest(
  	lora_name=f"{lora_int_id}",
  	lora_int_id=lora_int_id,
  	lora_path="simon_lora_path",
  	peft_config=asdict(peft_config),
  	lora_tensors=dict(weights),
  	)
  	self.inference_engine.worker.add_lora(lora_reqest)

• During rollout generation, the newly added LoRA adapter is never used due to two issues:

  1. The vllm_config used to create AsyncLLM lacks a LoRAConfig (e.g., max_lora_rank), so AsyncLLM is not prepared for LoRA-based generation requests.
     See

     verl/verl/workers/rollout/vllm_rollout/vllm_async_server.py

     Lines 299 to 304 in f209c6f

     	engine_client = AsyncLLM.from_vllm_config(
     	vllm_config=vllm_config,
     	usage_context=usage_context,
     	disable_log_requests=engine_args.disable_log_requests,
     	disable_log_stats=engine_args.disable_log_stats,
     	)

  2. When calling generate in vLLMHttpServerBase, the request to self.engine (the AsyncLLM instance) omits any LoRARequest, meaning generation always uses the base model. See

     verl/verl/workers/rollout/vllm_rollout/vllm_async_server.py

     Line 360 in f209c6f

     generator = self.engine.generate(prompt=prompt, sampling_params=sampling_params, request_id=request_id)

Proposed Fixes in this PR

• Standardize and persist VLLM_LORA_INT_ID and VLLM_LORA_NAME across the training process to consistently locate and apply updated LoRA weights.
• Inject LoRAConfig during AsyncLLM initialization and ensure vLLMHttpServerBase passes a proper LoRARequest (identified via VLLM_LORA_NAME) during rollout generation.
• Add utility methods to automatically validate and set max_lora_rank in vLLM from config.actor_rollout_ref.model.lora_rank, addressing issues like Unable to Create Loras of Rank 1 #3696

Remarks

Special thanks to @sanxing-chen for inspiring this fix with his prior patches. Also his PR #3765 -- while also tackling an issue hurting LoRA performance -- seems to be orthogonal to the issues addressed here.

Checklist Before Starting

• Search for similar PRs. Paste at least one query link here: [rollout,vllm] fix: Add LoRA Loading to Async vLLM #3639 fix: gradient vanish in FusedLinearForPPO #3765

• Format the PR title as [{modules}] {type}: {description} (This will be checked by the CI)

  • {modules} include fsdp, megatron, sglang, vllm, rollout, trainer, ci, training_utils, recipe, hardware, deployment, ray, worker, single_controller, misc, perf, model, algo, env, tool, ckpt, doc, data
  • If this PR involves multiple modules, separate them with ,, e.g., [megatron, fsdp, doc]
  • {type} ∈ {feat, fix, refactor, chore, test}
  • If this PR breaks any API, prepend [BREAKING] to the title.
  • Example: [BREAKING][fsdp, megatron] feat: dynamic batching

Test

For changes that cannot be tested by CI (e.g., algorithm implementation, new model support), validate with experiments and include results such as training curves or evaluation metrics.

Controlled experiments based on examples/grpo_trainer/run_qwen2_5-3b_gsm8k_grpo_lora.sh (see adapted script) clearly demonstrate both the issue and the effectiveness of the fix.

See the full W&B training log.
Summary:

• sync-lora-32 — baseline (synchronous mode).
• async-lora-32-before-fix — async LoRA on main branch, showing degraded performance.
• async-lora-32-no-remove — ablation variant with fixes applied but without removing old LoRA adapters between updates (showing the importance of removal).
• async-lora-32-after-fix — full fix applied, achieving expected improvement.

API and Usage Example

Demonstrate how the API changes if any, and provide usage example(s) if possible.

# Add code snippet or script demonstrating how to use this

Design & Code Changes

Demonstrate the high-level design if this PR is complex, and list the specific changes.

Checklist Before Submitting

Important

Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review.

• Read the Contribute Guide.
• Apply pre-commit checks: pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always
• Add / Update the documentation. Not Applicable
• Add unit or end-to-end test(s) to the CI workflow to cover all the code. If not feasible, explain why: This PR can hardly be covered by regular CI. I instead run concrete experiments with GSM8K dataset.
• Once your PR is ready for CI, send a message in the ci-request channel in the verl Slack workspace. (If not accessible, please try the Feishu group (飞书群).)

[gemini-code-assist]

Code Review

This pull request effectively resolves a critical issue with LoRA adapters in async vLLM rollouts, where the base model was incorrectly used instead of the updated LoRA model. The fixes, including standardizing LoRA identifiers, ensuring correct engine initialization, and passing LoRARequest during generation, are well-implemented and directly address the root causes. The introduction of constant identifiers and the explicit removal of old adapters are solid improvements. The new utility for calculating max_lora_rank also enhances configuration robustness. I have one suggestion to improve the robustness of this new utility function.