Prerequisites
Feature Description
Currently the server slot's KV-cache is available only if the slot that served a request is now idle. But there are many situations when the slot is in use and another request with the same prompt prefix is received. For example, if a few users work with a knowledge base that is relatively big (hundreds of thousands of tokens) then when the first user starts the processing of the knowledge base's text, which could take minutes, or even tens of minutes, the other users can not reuse the processed prompt's kv-cache and must reprocess all the text entirely, spending minutes or even tens of minutes. But it is even worse if we add the generation stage, it again could take minutes or even tens of minutes. During the generation the slot is still used, then its kv-cache is still unavailable for reuse. In total it could take tens of minutes until the kv-cache of the first slot will be released and become available for reuse. Then, during an half of hour, for example, every new user should start his own prompt processing task which is put into a queue and waits until the other users will get their prompts processed, and it is beside of the time it takes to process the user's prompt again. So, in total it could take more than an hour of absolutely wasted user and GPU time.
The feature request is about to increase the efficiency of the same prompt processing. For some model architectures the current implementation of llama.cpp already saves a state of kv-cache per each predefined number of processed tokens. Then for new requests for the processing of the same text it is enough just to include in the longest prefix search the saved states. If such state is found it could be copied into a new slot and there will be no additional processing and related inefficiency. For the architectures that do not require the state saving it is possible to introduce such practice of saving intermediary state, unified with the architectures where the state is already saved. As a result the first user will always process the large prompt "for all", meaning every next user could start the work with the knowledge base without waiting for the reprocesing of the whole large text and, additionally, not wasting time of other users who now must wait until GPU will finish the completely wasted reprocessing effort.
And if we remember that there could be a lot of scenarios when the same prefix is required by many users/agents then the need for such optimization becomes even more demanding.
Motivation
Significantly (many times in some cases) decrease the waiting time of the users that work with the same text.
Possible Implementation
No response
Prerequisites
Feature Description
Currently the server slot's KV-cache is available only if the slot that served a request is now idle. But there are many situations when the slot is in use and another request with the same prompt prefix is received. For example, if a few users work with a knowledge base that is relatively big (hundreds of thousands of tokens) then when the first user starts the processing of the knowledge base's text, which could take minutes, or even tens of minutes, the other users can not reuse the processed prompt's kv-cache and must reprocess all the text entirely, spending minutes or even tens of minutes. But it is even worse if we add the generation stage, it again could take minutes or even tens of minutes. During the generation the slot is still used, then its kv-cache is still unavailable for reuse. In total it could take tens of minutes until the kv-cache of the first slot will be released and become available for reuse. Then, during an half of hour, for example, every new user should start his own prompt processing task which is put into a queue and waits until the other users will get their prompts processed, and it is beside of the time it takes to process the user's prompt again. So, in total it could take more than an hour of absolutely wasted user and GPU time.
The feature request is about to increase the efficiency of the same prompt processing. For some model architectures the current implementation of llama.cpp already saves a state of kv-cache per each predefined number of processed tokens. Then for new requests for the processing of the same text it is enough just to include in the longest prefix search the saved states. If such state is found it could be copied into a new slot and there will be no additional processing and related inefficiency. For the architectures that do not require the state saving it is possible to introduce such practice of saving intermediary state, unified with the architectures where the state is already saved. As a result the first user will always process the large prompt "for all", meaning every next user could start the work with the knowledge base without waiting for the reprocesing of the whole large text and, additionally, not wasting time of other users who now must wait until GPU will finish the completely wasted reprocessing effort.
And if we remember that there could be a lot of scenarios when the same prefix is required by many users/agents then the need for such optimization becomes even more demanding.
Motivation
Significantly (many times in some cases) decrease the waiting time of the users that work with the same text.
Possible Implementation
No response