goeranh/rocm-v620-setup
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
No description
1 commit 1 branch 0 tags 76 KiB
Find a file
Exact
goeranh 2e36703dba initial commit
2026-06-24 08:08:49 +00:00
ansible initial commit 2026-06-24 08:08:49 +00:00
docs initial commit 2026-06-24 08:08:49 +00:00
ansible.cfg initial commit 2026-06-24 08:08:49 +00:00
README.md initial commit 2026-06-24 08:08:49 +00:00

README.md

ROCm + Radeon PRO V620 — LLM Inference Server Setup

Fully automated Ansible provisioning for AMD Radeon PRO V620 (32GB GDDR6, gfx1030/RDNA2) with ROCm 7.2.4 and the complete LLM inference stack.

Hardware Summary — Radeon PRO V620

Spec Value
GPU AMD Radeon PRO V620 (Navi 21 XT)
Architecture RDNA 2
LLVM Target gfx1030
VRAM 32 GB GDDR6 (ECC)
Memory Bandwidth 512 GB/s
Stream Processors 4608
FP16 (Half) 40.55 TFLOPS
FP32 (Single) 20.28 TFLOPS
TBP 300W (2× 8-pin PCIe)
PCIe 4.0 ×16
Display Outputs None (compute only)

ROCm Support: ✅ Officially supported — Ubuntu 24.04.4 / 22.04.5 only
Ollama Support: ✅ Officially listed as supported

What This Sets Up

Component Version Notes
OS Ubuntu 24.04 (Noble) Only officially supported OS for V620
Kernel HWE (6.8+) With pci=realloc=off amdgpu.gpu_recovery=1 amdgpu.mcbp=0
ROCm 7.2.4 Production stable, gfx1030/RDNA2 support
Ollama Latest ROCm backend, systemd service
llama.cpp Built from source HIP/ROCm backend, gfx1030 optimized
PyTorch 2.7+ ROCm wheels from pytorch.org
Python ML transformers, accelerate, huggingface-hub In isolated venv

Quick Start

# 1. Clone this repo
git clone <repo-url> rocm-v620-setup
cd rocm-v620-setup

# 2. Set your target host
export V620_HOST=192.168.1.100
export V620_USER=ubuntu

# 3. Run the full bootstrap (requires passwordless sudo or --ask-become-pass)
ansible-playbook playbooks/site.yml -i inventory/hosts.yml

# Or run stages individually:
ansible-playbook playbooks/site.yml --tags prep     # OS tuning + GRUB
# ── REBOOT ──
ansible-playbook playbooks/site.yml --tags rocm     # ROCm 7.2.4 install
# ── REBOOT ──
ansible-playbook playbooks/site.yml --tags llm-stack # Ollama + llama.cpp + PyTorch

Project Structure

rocm-v620-setup/
├── README.md                          # This file
├── ansible.cfg                        # Ansible configuration
├── ansible/
│   ├── inventory/
│   │   └── hosts.yml                  # Target host definitions
│   ├── playbooks/
│   │   ├── site.yml                   # Main playbook (all stages)
│   │   ├── rocm-install.yml           # ROCm-only playbook
│   │   └── llm-tools.yml             # LLM tools-only playbook
│   └── roles/
│       ├── system-prep/               # OS tuning, GRUB, kernel modules
│       ├── rocm/                      # amdgpu-dkms + ROCm 7.2.4
│       └── llm-stack/                 # Ollama, llama.cpp, PyTorch
└── docs/
    ├── installation-guide.md          # Full manual walkthrough
    ├── bios-configuration.md          # BIOS settings reference
    ├── troubleshooting.md             # Common errors and fixes
    ├── performance-tuning.md          # Optimization for LLM inference
    └── hardware-buying-guide.md       # Motherboards, PSUs, cooling, full BOM

Prerequisites

Hardware

  • AMD Radeon PRO V620 GPU (properly seated, 2× 8-pin PCIe power connected)
  • Motherboard with PCIe 4.0 ×16 (or PCIe 3.0 ×16 — see BIOS notes)
  • 32GB+ system RAM (for 32GB VRAM GPU, recommend 64GB for headroom)
  • SSD with 100GB+ free space

BIOS Settings (Must-Do Before Anything Else)

  1. Above 4G Decoding → Enabled (critical!)
  2. PCIe Speed → Force to maximum (Gen3 or Gen4, not Auto)
  3. IOMMU → Enabled
  4. SR-IOV Support → Enabled (optional, for GPU virtualization)
  5. ACS Enable → Enabled
  6. Secure Boot → Disabled (can re-enable after setup)
  7. Resizable BAR / Smart Access Memory → Enabled

Without "Above 4G Decoding" and forced PCIe speed, the V620 will fail with amdgpu: Fatal error during GPU init.

Software

  • Ansible 2.15+ on your control machine
  • SSH access to the target server
  • Ubuntu 24.04 (Noble) or 22.04 (Jammy) installed on target

Post-Installation — Run Your First LLM

# ─── Ollama ─────────────────────────────────────────
# Pull a model (choose based on VRAM)
ollama pull llama3.2:3b          # ~2GB, fast
ollama pull llama3.1:8b          # ~5GB
ollama pull qwen2.5:14b          # ~9GB  (fits easily in 32GB)
ollama pull mistral-small:22b    # ~13GB (good balance)
ollama pull qwen2.5:32b          # ~20GB (max for single model)

# Chat
ollama run qwen2.5:14b

# ─── llama.cpp ──────────────────────────────────────
# Download a GGUF model
huggingface-cli download bartowski/Qwen2.5-14B-Instruct-GGUF \
  Qwen2.5-14B-Instruct-Q4_K_M.gguf --local-dir ./models

# Benchmark
llama-bench -m ./models/Qwen2.5-14B-Instruct-Q4_K_M.gguf -ngl 99 -fa 1

# Serve (OpenAI-compatible API)
llama-server -m ./models/Qwen2.5-14B-Instruct-Q4_K_M.gguf \
  -ngl 99 -fa 1 --port 8080 --host 0.0.0.0

# ─── vLLM (if installed) ──────────────────────────
source /opt/ml-venv/bin/activate
vllm serve Qwen/Qwen2.5-14B-Instruct --dtype half

Performance Expectations (V620, 32GB)

Model Quant VRAM Used Est. tok/s Usable Context
Llama 3.2 3B Q4_K_M ~2.5 GB ~100+ 32K
Llama 3.1 8B Q4_K_M ~5.5 GB ~55-65 32K
Qwen 2.5 14B Q4_K_M ~9 GB ~35-45 32K
Mistral Small 22B Q4_K_M ~14 GB ~20-25 16K
Qwen 2.5 32B Q4_K_M ~20 GB ~12-15 8K

Estimates based on 512 GB/s memory bandwidth. Use llama-bench for precise numbers. V620 has 32GB — a 20GB model file leaves ~12GB for KV cache at large contexts.

References