Skip to content

codeshll不支持shared吗 #58

@lomoonmoonbird

Description

@lomoonmoonbird

通过TGI托管的模型,启动方式为:
BNB_CUDA_VERSION=122 CUDA_VISIBLE_DEVICES=0,1 text-generation-launcher --model-id /data/llms/codeshell-7b-chat --tokenizer-config-path /data/llms/codeshell-7b-chat/tokenizer_config.json --sharded true --trust-remote-code --port=8080

CUDA_VISIBLE_DEVICES=0,1 和 --sharded true 设置后报错:
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions