rocm/vllm-dev repository overview

⁠vLLM-dev

vLLM-dev is a weekly dev stream toolkit and library for large language model (LLM) inference and serving. These containers are a quick way to run or try vLLM on AMD Instinct™ accelerators. For building or extensions for vLLM-dev, please see the Build Dockerfiles⁠.

Running Containers

docker run -it --rm \
    --ipc=host \
    --network=host \
    --privileged \
    --cap-add=CAP_SYS_ADMIN \
    --device=/dev/kfd \
    --device=/dev/dri \
    --device=/dev/mem \
    --group-add render \
    --cap-add=SYS_PTRACE \
    --security-opt seccomp=unconfined \
    rocm/vllm-dev:main

Documentation

Please see here⁠ for full documentation on getting started, setting up the environment, and more.