llama-cpp

Guide for llama.cpp, the C/C++ LLM inference framework by ggml-org. Covers the C API (llama.h), GGUF format, quantization (Q4_K_M, Q8_0, IQ4_XS), CMake builds, GPU backends (CUDA, Vulkan, Metal, ROCm), HTTP server with OpenAI-compatible API, embeddings, grammar constraints, function calling, LoRA, speculative decoding, multimodal, and UE5 integration. Use when: llama.cpp, GGUF models, local LLM inference, llama.h, llama-server, quantizing, ggml, building/linking llama.cpp, GPU acceleration, llama.cpp embeddings, grammar/JSON output, llama.cpp in Unreal Engine, llama_* API functions, GGUF format, converting HuggingFace to GGUF, or comparing with vLLM/Ollama/TensorRT-LLM.

适合你，如果需要在本地或边缘设备上高效运行LLM推理。