Direct Semantic Communication Between Large Language Models

View a PDF of the paper titled Cache-to-Cache: Direct Semantic Communication Between Large Language Models, by Tianyu Fu and 6 different authors

View PDF
HTML (experimental)

Abstract:Multi-LLM techniques harness the complementary strengths of various Large Language Models, attaining efficiency and effectivity beneficial properties that aren’t attainable by a single mannequin. In current designs, LLMs talk by textual content, forcing inner representations to be remodeled into output token sequences. This course of each loses wealthy semantic data and incurs token-by-token technology latency. Motivated by these limitations, we ask: Can LLMs talk past textual content? Oracle experiments present that enriching the KV-Cache semantics can enhance response high quality with out growing cache dimension, supporting KV-Cache as an efficient medium for inter-model communication. Thus, we suggest Cache-to-Cache (C2C), a brand new paradigm for direct semantic communication between LLMs. C2C makes use of a neural community to challenge and fuse the supply mannequin’s KV-cache with that of the goal mannequin to allow direct semantic switch. A learnable gating mechanism selects the goal layers that profit from cache communication. Compared with textual content communication, C2C makes use of the deep, specialised semantics from each fashions, whereas avoiding specific intermediate textual content technology. Experiments present that C2C achieves 6.4-14.2% larger common accuracy than particular person fashions. It additional outperforms the textual content communication paradigm by roughly 3.1-5.4%, whereas delivering a median 2.5x speedup in latency. Our code is on the market at this https URL.