A Tokio service drops inbound TCP messages or loses channel events when you wrap network reads in a tokio::select! loop. The loop runs fine under light traffic, but under load, data vanishes from your stream without an explicit drop or error.
This happens because tokio::select! drops the losing branches at every iteration, and many common async functions are not cancellation-safe.
The symptom: dropped bytes in select loops
Consider a framed reader that reads length-prefixed frames from a socket while listening for a shutdown signal.
rustuse tokio::io::{AsyncReadExt, Result};
use tokio::net::TcpStream;
use tokio::sync::oneshot;
pub async fn run_loop(mut stream: TcpStream, mut shutdown: oneshot::Receiver<()>) -> Result<()> {
loop {
tokio::select! {
_ = &mut shutdown => {
println!("Shutdown requested");
break Ok(());
}
res = read_frame(&mut stream) => {
let frame = res?;
process_frame(frame).await;
}
}
}
}
async fn read_frame(stream: &mut TcpStream) -> Result<Vec<u8>> {
let mut len_buf = [0u8; 4];
stream.read_exact(&mut len_buf).await?;
let len = u32::from_be_bytes(len_buf) as usize;
let mut payload = vec![0u8; len];
stream.read_exact(&mut payload).await?;
Ok(payload)
}
async fn process_frame(_frame: Vec<u8>) {}
Under production conditions, this eventually returns an error or corrupts data:
textError: Custom { kind: UnexpectedEof, error: "failed to fill whole buffer" }
Sometimes it fails with invalid payloads instead of UnexpectedEof. The upstream client sent valid data, but your server interpreted payload bytes as the start of the next packet.
Why select cancellation burns your state
The tokio::select! macro polls all branches concurrently on the current task thread. When one branch finishes, all other branches that were pending are dropped immediately. Dropping a future means calling its Drop implementation and abandoning its internal state.
In the example above, read_frame has two distinct .await points: reading the 4-byte header and reading the payload body.
Suppose a frame arrives. stream.read_exact(&mut len_buf) completes, reading the 4-byte length. The runtime immediately polls stream.read_exact(&mut payload). That read is pending because the rest of the bytes are still in the OS socket buffer or in transit.
While waiting for payload bytes, another branch completes. It could be your shutdown channel, or a timeout branch like tokio::time::sleep, or a heartbeat branch. The macro drops the read_frame future.
The 4 bytes read from the stream are gone forever. They were stored in len_buf, which lived inside the dropped stack frame of read_frame. The underlying OS socket pointer advanced by 4 bytes. When the loop restarts and calls read_frame(&mut stream) again, it reads the payload bytes and treats them as the header of a new frame. The entire stream protocol is now misaligned.
A function is cancellation-safe if dropping its future before completion does not cause side effects, lose buffered progress, or corrupt future attempts. AsyncReadExt::read_exact is explicitly not cancellation-safe.
The fix: stateful buffers and split operations
To make framing safe inside a select! branch, keep the buffer outside the branch future, or use Tokio’s framing abstractions that retain an internal buffer across polls.
Solution 1: Use BytesMut with AsyncReadExt::read
Tokio’s raw read operation is cancellation-safe. It either reads some number of bytes into your buffer and returns, or reads nothing before yielding. If cancelled while pending, no data was pulled from the socket kernel buffer yet.
rustuse bytes::{Buf, BytesMut};
use tokio::io::{AsyncReadExt, Result};
use tokio::net::TcpStream;
use tokio::sync::oneshot;
pub async fn run_loop(mut stream: TcpStream, mut shutdown: oneshot::Receiver<()>) -> Result<()> {
let mut buffer = BytesMut::with_capacity(4096);
loop {
tokio::select! {
_ = &mut shutdown => {
break Ok(());
}
res = stream.read_buf(&mut buffer) => {
let n = res?;
if n == 0 {
if buffer.is_empty() {
return Ok(());
} else {
return Err(std::io::Error::new(
std::io::ErrorKind::UnexpectedEof,
"connection closed with incomplete frame",
));
}
}
while let Some(frame) = parse_frame(&mut buffer)? {
process_frame(frame).await;
}
}
}
}
}
fn parse_frame(buf: &mut BytesMut) -> Result<Option<Vec<u8>>> {
if buf.len() < 4 {
return Ok(None);
}
let len = u32::from_be_bytes(buf[..4].try_into().unwrap()) as usize;
if buf.len() < 4 + len {
// Buffer does not have full frame yet. Wait for more reads.
return Ok(None);
}
buf.advance(4);
let payload = buf.split_to(len).to_vec();
Ok(Some(payload))
}
async fn process_frame(_frame: Vec<u8>) {}
Because buffer lives outside the tokio::select! block, cancellation cannot destroy partially received chunks. If stream.read_buf(&mut buffer) resolves, bytes are safely inside buffer. If it gets cancelled mid-read, no bytes were extracted from the kernel yet.
Solution 2: Tokio FramedRead
If writing manual buffer checks is tedious, Tokio’s tokio-util crate handles framing via the Decoder trait.
Add tokio-util to Cargo.toml:
toml[dependencies]
bytes = "1.5"
tokio = { version = "1.35", features = ["full"] }
tokio-util = { version = "0.7", features = ["codec"] }
Implement Decoder and wrap the stream. FramedRead::next() is cancellation-safe because the frame decoder retains internal buffer state inside the FramedRead struct:
rustuse bytes::{Buf, BytesMut};
use tokio::net::TcpStream;
use tokio::sync::oneshot;
use tokio_stream::StreamExt;
use tokio_util::codec::{Decoder, FramedRead};
struct LengthPrefixedCodec;
impl Decoder for LengthPrefixedCodec {
type Item = Vec<u8>;
type Error = std::io::Error;
fn decode(&mut self, src: &mut BytesMut) -> Result<Option<Self::Item>, Self::Error> {
if src.len() < 4 {
return Ok(None);
}
let len = u32::from_be_bytes(src[..4].try_into().unwrap()) as usize;
if src.len() < 4 + len {
src.reserve(4 + len - src.len());
return Ok(None);
}
src.advance(4);
Ok(Some(src.split_to(len).to_vec()))
}
}
pub async fn run_loop_codec(stream: TcpStream, mut shutdown: oneshot::Receiver<()>) -> std::io::Result<()> {
let mut framed = FramedRead::new(stream, LengthPrefixedCodec);
loop {
tokio::select! {
_ = &mut shutdown => {
break Ok(());
}
frame_opt = framed.next() => {
match frame_opt {
Some(Ok(frame)) => process_frame(frame).await,
Some(Err(err)) => return Err(err),
None => break Ok(()),
}
}
}
}
}
async fn process_frame(_frame: Vec<u8>) {}
Edge Case 1: tokio::sync::mpsc::Sender::send is not cancellation-safe
Socket streams are not the only hazard. Tokio’s channel send operations have cancellation traps.
tokio::sync::mpsc::Receiver::recv is safe to cancel. If recv is cancelled before receiving an item, the item remains safely in the channel buffer.
tokio::sync::mpsc::Sender::send is not safe to cancel if you rely on delivery guarantees.
rust// DANGEROUS: If cancelled after acquiring a permit but before putting
// the item into the queue, the permit is spent and the payload is dropped.
tokio::select! {
res = tx.send(item) => {
if res.is_err() {
eprintln!("Receiver dropped");
}
}
_ = tokio::time::sleep(tokio::time::Duration::from_millis(50)) => {
eprintln!("Send timed out");
}
}
If the timeout triggers while send(item) is waiting for channel capacity, item is dropped and lost.
To cancel safely without losing the item, split permit acquisition from sending:
rustuse tokio::sync::mpsc::Sender;
use tokio::time::{sleep, Duration};
pub async fn safe_send_with_timeout<T>(tx: &Sender<T>, item: T) -> Result<(), T> {
let permit = tokio::select! {
res = tx.reserve() => match res {
Ok(permit) => permit,
Err(_) => return Err(item),
},
_ = sleep(Duration::from_millis(50)) => {
return Err(item);
}
};
// reserve() was the only cancellation point.
// send_reserve is synchronous and infallible once permit exists.
permit.send(item);
Ok(())
}
tx.reserve() does not consume your data. If cancelled, it only relinquishes its reservation. Your data stays in the local variable and can be retried or logged.
Edge Case 2: MutexGuard across await points in branches
A subtle freeze happens when combining locks with select!. If you acquire a standard std::sync::Mutex or parking_lot::Mutex across an .await boundary, you trigger undefined behavior or a rustc error. If you use tokio::sync::Mutex, it compiles, but cancellation can create lock starvation.
rustuse std::sync::Arc;
use tokio::sync::Mutex;
use tokio::time::{sleep, Duration};
struct SharedState {
counter: u64,
}
async fn worker(state: Arc<Mutex<SharedState>>) {
loop {
tokio::select! {
mut guard = state.lock() => {
// If this inner .await takes too long or cancels,
// the lock remains held until the future resolves.
slow_async_write(&mut guard).await;
}
_ = sleep(Duration::from_millis(100)) => {
println!("Heartbeat or backoff trigger");
}
}
}
}
async fn slow_async_write(_state: &mut tokio::sync::MutexGuard<'_, SharedState>) {
sleep(Duration::from_millis(200)).await;
}
Here, state.lock() acquires the lock. Then slow_async_write(&mut guard).await begins executing. But wait: sleep(100) is also racing inside the same select! block on the next tick.
Keep locks strictly synchronous. If you need data from a mutex across network calls, clone the data out of the critical section and drop the guard immediately before awaiting.
rustasync fn better_worker(state: Arc<std::sync::Mutex<SharedState>>) {
loop {
// Read or update state synchronously, release immediately
let current_count = {
let guard = state.lock().unwrap();
guard.counter
};
tokio::select! {
res = send_over_network(current_count) => {
if let Ok(()) = res {
let mut guard = state.lock().unwrap();
guard.counter += 1;
}
}
_ = sleep(Duration::from_millis(100)) => {
println!("Network slow, retrying");
}
}
}
}
async fn send_over_network(_val: u64) -> Result<(), ()> {
sleep(Duration::from_millis(50)).await;
Ok(())
}
Verifying cancellation safety
You cannot detect cancellation safety bugs through unit tests that feed complete strings into mock readers. Those tests run synchronously on the local reactor and will not interrupt midway through a payload read.
Write an integration test that chunks data byte-by-byte with artificially delayed yields:
rust#[cfg(test)]
mod tests {
use super::*;
use std::pin::Pin;
use std::task::{Context, Poll};
use tokio::io::{AsyncRead, ReadBuf};
struct FragmentedReader {
data: Vec<u8>,
pos: usize,
}
impl AsyncRead for FragmentedReader {
fn poll_read(
mut self: Pin<&mut Self>,
_cx: &mut Context<'_>,
buf: &mut ReadBuf<'_>,
) -> Poll<std::io::Result<()>> {
if self.pos >= self.data.len() {
return Poll::Ready(Ok(()));
}
// Emit only 1 byte at a time to force partial reads
buf.put_slice(&[self.data[self.pos]]);
self.pos += 1;
Poll::Ready(Ok(()))
}
}
#[tokio::test]
async fn test_cancellation_safety_with_chunks() {
let (tx, rx) = tokio::sync::oneshot::channel();
let payload = vec![0x00, 0x00, 0x00, 0x02, 0xAA, 0xBB];
let mut reader = FragmentedReader { data: payload, pos: 0 };
let mut buffer = BytesMut::new();
// Simulate an early cancellation triggered by a timer
let mut shutdown_fut = rx;
// Loop step 1: Read part of the buffer
let mut chunk_buf = [0u8; 2];
tokio::select! {
_ = &mut shutdown_fut => {}
_ = reader.read(&mut chunk_buf) => {
buffer.extend_from_slice(&chunk_buf);
}
}
// Simulate a tick where shutdown doesn't happen, continuing the read
tokio::select! {
_ = &mut shutdown_fut => {}
res = reader.read_to_end(&mut buffer.to_vec()) => {
// If framing was done safely outside the branch,
// data remains consistent.
assert!(res.is_ok());
}
}
}
}
Run this with the Tokio console or under thread sanitizer to catch deadlocks and torn states:
bashRUSTFLAGS="-Zsanitizer=thread" cargo test --target x86_64-unknown-linux-gnu
The Tokio documentation also explicitly lists the cancellation safety of every method on types like AsyncReadExt, AsyncWriteExt, and mpsc. Check the “Cancellation safety” section in the docs for every method called directly inside a select! branch. If it does not explicitly say “This method is cancellation safe”, keep its execution state outside the macro.