Memory-efficient Inference On Long Documents Via Compact Kv-cache Merging

Official ICLR 2024 implementation of HOMER, a training-free hierarchical KV-cache merging method that extends pre-trained LLM context limits (e.g. Llama-2) with lower memory. Ships patched LlamaForCausalLM, plus passkey-retrieval and PG19 perplexity scripts.

AI Frameworks | Python · long-context · kv-cache