LLM Acceleration: KV-cache and Semantic Cache Tuning

Accelerating LLMs: Tuning KV-cache and Semantic Cache

Picture this: your LLM service processes queries, and 40% of them are repetitive questions. Each time the model runs a full autoregressive inference: loading weights, computing keys and values (self-attention), generating tokens one by one. GPU hours burn, latency spikes to 5 seconds. We solve this with two-level caching — Semantic Cache at the application level and KV-cache at the model level. The combination yields up to 70% savings on inference costs and reduces p99 latency 5x. Our proven methodology, backed by 15+ projects, guarantees a 5x latency reduction and up to 70% GPU cost savings. For example, a client saved $12,000/month after implementing our caching solution.

Caching is not an option but a necessity in production. Without it, every query is recomputed even if the answer was generated a minute ago. LLM inference caching is essential for production deployments. This caching strategy is a key part of LLM optimization. We offer a turnkey solution: from audit to deployment within 5-10 business days. Contact us for a free project estimate.

How does two-level caching work?

The first level — Semantic Cache. It uses vector embeddings to find semantically similar queries. When a new query arrives, it is encoded into a vector (768-dimensional embeddings), and nearest neighbors are searched in a vector database (Qdrant, pgvector). If a record with similarity above a threshold (usually 0.85–0.95) is found, the cached response is returned without inference. This is a form of LLM response caching. Here’s a Python implementation using Sentence Transformers, Redis, and Qdrant:

from sentence_transformers import SentenceTransformer
import numpy as np
import redis
import json

class SemanticCache:
    def __init__(self, similarity_threshold: float = 0.92):
        self.encoder = SentenceTransformer("paraphrase-multilingual-mpnet-base-v2")
        self.redis = redis.Redis(host="localhost", port=6379, db=1)
        self.threshold = similarity_threshold
        from qdrant_client import QdrantClient
        self.vector_db = QdrantClient("localhost", port=6333)

    def get(self, prompt: str, system_prompt: str = "") -> str | None:
        cache_key = self._make_key(prompt, system_prompt)
        exact = self.redis.get(cache_key)
        if exact:
            return json.loads(exact)["response"]
        embedding = self.encoder.encode(prompt)
        results = self.vector_db.search(
            collection_name="llm_cache",
            query_vector=embedding.tolist(),
            limit=1,
            score_threshold=self.threshold
        )
        if results:
            cached_response = json.loads(results[0].payload["response"])
            self.redis.expire(results[0].id, 3600)
            return cached_response
        return None

    def set(self, prompt: str, response: str, system_prompt: str = "", ttl: int = 3600):
        embedding = self.encoder.encode(prompt)
        cache_id = self._make_key(prompt, system_prompt)
        self.vector_db.upsert(
            collection_name="llm_cache",
            points=[{
                "id": abs(hash(cache_id)) % (2**31),
                "vector": embedding.tolist(),
                "payload": {"prompt": prompt, "response": json.dumps(response), "system_prompt": system_prompt}
            }]
        )
        self.redis.setex(cache_id, ttl, json.dumps({"response": response}))

    def _make_key(self, prompt: str, system_prompt: str) -> str:
        import hashlib
        return hashlib.sha256(f"{system_prompt}||{prompt}".encode()).hexdigest()

The second level — KV-cache at the model level. vLLM automatically caches keys and values for common prefixes, such as system prompt, using paged attention and block-level caching in the transformer architecture. With prefix caching enabled, hit rate reaches 60–80%, reducing latency 2–5x. Compared to no caching, KV-cache reduces generation latency from 4.2s to 0.3s—over 10x faster.

# vLLM automatically uses prefix caching
# system prompt should be identical across requests

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3-8b-instruct \
  --enable-prefix-caching \
  --max-model-len 8192

# Metric: vllm:gpu_cache_usage_perc shows cache occupancy

GPTCache — ready-made solution

GPTCache is a library that implements Semantic Cache and manages the entire infrastructure: embeddings, vector search, TTL. Integration boils down to replacing the openai call with cached_openai:

from gptcache import cache
from gptcache.adapter import openai as cached_openai
from gptcache.embedding import Onnx
from gptcache.manager import CacheBase, VectorBase, get_data_manager
from gptcache.similarity_evaluation.distance import SearchDistanceEvaluation

embedding_model = Onnx()
data_manager = get_data_manager(
    CacheBase("sqlite"),
    VectorBase("qdrant", host="localhost", port=6333, dimension=512)
)

cache.init(
    embedding_func=embedding_model.to_embeddings,
    data_manager=data_manager,
    similarity_evaluation=SearchDistanceEvaluation(max_distance=0.3),
    cache_enable_func=lambda *args, **kwargs: True
)

response = cached_openai.ChatCompletion.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "What is Python?"}]
)

Comparison of caching types

Semantic cache is 90% faster than full inference; KV-cache reduces latency 2-5x compared to no caching.

How to choose similarity threshold for Semantic Cache?

The similarity threshold is a key parameter for LLM cache tuning. Too low (0.7) leads to false positives, too high (0.98) to frequent misses. The optimal value depends on the task. For FAQ bots, 0.85–0.90 works well; for RAG with fixed documents, 0.90–0.95; for classification, 0.95+. We tune the threshold based on analysis of your data: we take 1000 real queries, label semantically equivalent pairs, and pick the threshold by F1 score. Tuning LLM cache parameters like similarity threshold and TTL is crucial for optimal response caching.

When doesn’t caching give a gain?

Caching is pointless for personalized answers, queries with current time or date, financial data (rates, prices), code generation, and when temperature > 0.8. In such cases, better disable caching. The biggest effect comes from caching in FAQ bots, RAG with fixed documents, and classification tasks.

Why is caching not a panacea?

Caching does not fix model quality. If the base model hallucinates, caching only solidifies errors. You must monitor the staleness rate — the share of cached responses that become outdated. We implement A/B tests: periodically compare cached response with a fresh inference. If divergence exceeds a threshold, the cache is invalidated.

Integration with existing infrastructure

Caching integrates easily with popular MLOps tools. vLLM and TGI support prefix caching at the inference server level. For Semantic Cache, we use Redis as a fast exact-matching cache and Qdrant or pgvector for vector search. All components are deployed in Docker or Kubernetes, monitored via Prometheus and Grafana.

Caching metrics

What's included in our caching implementation

• Architecture audit and caching strategy document
• Semantic cache setup with vector database (Qdrant/pgvector)
• KV-cache and prefix caching configuration in vLLM
• Integration with your existing LLM API (OpenAI-compatible)
• Deployment scripts and Docker/Kubernetes manifests
• Monitoring dashboards (Prometheus/Grafana) with alerting on cache hit rate and latency
• Knowledge transfer session and documentation

Our engineers have experience implementing caching in 15+ projects. Average savings: 60% on inference costs. Contact us for a free audit of your architecture — it takes no more than an hour. We’ll discuss your cache parameters and provide a turnkey solution at a fixed price.