Is this page helpful?

core.inference.async_stream#

Module Contents#

Classes#

AsyncStream

Class for encapsulating an asynchronous stream of request outputs.

Data#

STOP_ITERATION

T

API#

core.inference.async_stream.STOP_ITERATION#

‘Exception(…)’

core.inference.async_stream.T#

‘TypeVar(…)’

class core.inference.async_stream.AsyncStream(
request_id: int,
cancel: Callable[[], None],
loop: Optional[asyncio.AbstractEventLoop] = None,
)#

Bases: typing.Generic[core.inference.async_stream.T]

Class for encapsulating an asynchronous stream of request outputs.

Adopted from https://github.com/vllm-project/vllm/blob/eb881ed006ca458b052905e33f0d16dbb428063a/vllm/v1/engine/async_stream.py # pylint: disable=line-too-long

Initialization

property request_id: int#

The request associated with this stream.

put(
item: Union[core.inference.async_stream.T, Exception],
) → None#

Adds a new value to the stream

finish(
exception: Optional[Union[BaseException, Type[BaseException]]] = None,
) → None#

Completes the stream by adding a sentinel value

property finished: bool#

Whether the stream has finished

__aiter__()#
async __anext__() → core.inference.async_stream.T#
async aclose() → None#

Cancel the request if the stream is still active.

async generator() → AsyncGenerator[core.inference.async_stream.T, None]#

Creates an AsyncGenerator over the stream queue

static _is_raisable(value: Any)#