\documentclass{article}
\renewcommand{\c}[1]{\texttt{#1}}
\begin{document}

There are two sources of video gaps:

\begin{enumerate}
\item Discontiguous outputs from \c{FFmpegDecoder} --- it makes sense
  to fix these when the frame after the gap is seen, as that's the
  first time we know about the gap.  For example, it emits frame~1
  then frame~3; when we see 3 we should emit 2 to fill the gap.
\item Periods where there is no video content --- these could be long,
  so you can't just wait for the frame after the gap.
\end{enumerate}

Two solutions suggest themselves for the period of no video content:

\begin{enumerate}
\item Create `black' \c{Piece}s to act as dummy content and emit black
  frames as required.
\item Fix it in \c{Player::pass()}.
\end{enumerate}

Dummy pieces feels like a nice solution but quite wordy as you need a
hierarchy of \c{Piece}s with virtual functions and so on.

If we can trust \c{Decoder::position()} we know the earliest time that
a decoder will emit data when it is next \c{pass()}ed.  If this is
more than one frame since the last video frame emitted we know we need
to emit a black frame.  This much seems straightforward.

Things appear to get harder with seeking.  There are two paths here:

\begin{enumerate}
\item Seeking into the middle of some video content.
\item Seeking into some empty space.
\end{enumerate}

and also differences between accurate and inaccurate seek.

Let's take them case-by-case:

\begin{enumerate}
  \item \emph{Accurate seek into content} --- we should not fill
    anything since the correct data will be generated, in time, by
    \c{pass()}.
  \item \emph{Accurate seek into space} --- we should fill up to the
    earliest decoder position.
  \item \emph{Inaccurate seek into content} --- we should not fill
    anything since \c{pass()} will generate something at some
    unknown time.
  \item \emph{Inaccurate seek into space} --- we should fill up to
    the earliest decoder position from the seek time.
\end{enumerate}

\end{document}