With multi-threaded decoding we need to be careful to flush