Skip to main content

​
Functions

​
FUNC legacy_cache_smash 

legacy_cache_smash(a: LegacyCache, b: LegacyCache) -> LegacyCache
Concatenates two LegacyCache Ks and Vs along the time axis.

​
FUNC merge_dynamic_caches 

merge_dynamic_caches(caches: Iterable[DynamicCache]) -> DynamicCache
Merges two DynamicCache Ks and Vs along the time axis.

​
FUNC tokens_to_legacy_cache 

tokens_to_legacy_cache(model, device: str, tokens_or_cache: BatchEncoding | DynamicCache) -> Iterable[LegacyCache]
Prefills and returns Ks and Vs as a LegacyCache.