MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool.

Mon, 01 Jan 2024 00:00:00 +0100

Cunchen Hu, Heyang Huang, Junhao Hu, Jiang Xu, Xusheng Chen, Tao Xie, Chenxi Wang, Sa Wang, Yungang Bao, Ninghui Sun, Yizhou Shan:
MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool. CoRR abs/2406.17565 (2024)