Skip to main content

​
Classes

​
CLASS Cache

A Cache for storing model state (e.g., kv cache).
Methods:

​
FUNC put 

put(self, key: str, value: Any)
Inserts into the cache. May result in eviction of other cached values.

​
FUNC get 

get(self, key: str) -> Any | None
Retrieves a value from the cache. Returns None if the id has no cached value. May impact which cache values are evicted.

​
FUNC current_size 

current_size(self) -> int
Returns the number of things currently in the cache. Mostly useful for debugging.

​
CLASS SimpleLRUCache

A simple LRU cache.
Methods:

​
FUNC current_size 

current_size(self)
Just return the size of the key set. This isn’t necessarily safe.

​
FUNC get 

get(self, key: str) -> Any | None
Gets a value from the cache.

​
FUNC put 

put(self, key: str, value: Any)
Put a value into the cache.