Writing a stateful Stream from scratch

Home » Writing a stateful Stream from scratch

Writing a stateful Stream from scratch


You need to query a web API that returns paged data and throttles your requests? Would you appreciate an abstraction layer that separates getting the data from handling it? Let’s implement a Stream step by step, learning the current limitations of the language and how to avoid them.


It is very common to interact with Web APIs that expose paged data and, at the same time, use some algorithm like the leaky bucket in order to limit the amount of requests per user. Nonetheless, it is not trivial at all to handle these situations in a clean way, and the ideal solution is to separate the logic behind requesting the paged data from the processing of the results.
During the workshop we are going to learn how to implement a Stream in order to build an abstraction layer able to perform the right queries with the right timing. In details, we will learn about:
– expressing a Stream as a state machine;
– how Pin works in practice;
– the current limitations of the type system and how to avoid them;
– when to use async and when to avoid it (and why).
During the event we will not have time to write everything, instead you will be given some basic code to work with, in order to focus on the implementation of the Stream. Moreover, the entire workflow will be divided in steps and a repository will be available with the code for each step.


– Brief introduction of the issues when a non-trivial Stream has to be implemented.
– Clone of the basic structure to start working on the implementation without worrying on details not relevant for us.
– Incremental implementation of a working Stream, using naive approaches.
– Removal of an allocation for an !Unpin structure with the help of pin-project.
– Further removal of an allocation, taking advantage of careful usage of async in a non-trivial context.
– Final thoughts of future possibilities in the short and long term.


– How the Stream trait works.
– What is Pin, why it is important, what is Unpin and how to handle both concepts.
– The concept of pin projection and how to taking advantage of it.
– How to take advantage of async and how to overcome cryptic lifetime errors in non-trivial contexts.
– How to build an extremely reliable asynchronous abstraction with a very small overhead.


– A laptop;
– The Rust toolchain
– Your favorite editor
– A browser and git.

Edoardo Morandi


Full-stack freelance developer and ex-bioinformatician. Rustacean since 2017, I love learning about nerdy tech things, programming languages in particular. I like to keep fit, but I also enjoy cooking bread and pizza.

Expert on: Rust,

Share on

Follow us on