Skip to content

Rework map_async to handle failures better#495

Merged
martindurant merged 5 commits intopython-streamz:masterfrom
dwskoog:map_async_improvements
Jan 21, 2026
Merged

Rework map_async to handle failures better#495
martindurant merged 5 commits intopython-streamz:masterfrom
dwskoog:map_async_improvements

Conversation

@dwskoog
Copy link
Contributor

@dwskoog dwskoog commented Jan 16, 2026

map stops the flow of items in the stream when the function raises but map_async is outside of the direct line of return so it fails weirdly during an exception. To address that, I added the idea about stopping the stream or not. This way, if the stream does not deliberately invoke stop during an exception, the stream continues to process inputs after an exception.

Since the map_async now conceives of stopping or not, I added a boolean in the node state to control the loop inside the worker task.

In the case of an exception during mapping, map_async will now release the references held on the metadata for the offending input.

I added an example that shows off the failure modes of map and map_async that plainly demonstrates that exceptions can leave the stream in a weird state.

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants