Deep Audio Redundancy (DRED) Extension for the Opus Codec

2.2.1. Encoder architecture

Every 20 ms, the encoder takes in a pair of 20-dimensional acoustic feature vectors as input and produces one initial state (IS) and one latent vector. Each latent vector encodes 40 ms (their information overlaps), so only half the latent vectors need to be transmitted. Although an encoder is provided for reference, the encoder architecture is not normative. Each redundancy packet contains the latest initial state, along with latent vectors ordered from the latest (the one aligned with the initial state) to the earliest one the encoder includes. Each conponent of the IS and latent vectors are quantized and then entropy-coded following a Laplace distribution. The same procedure is used for both the latent vectors and the initial state (we will describe the process for a latent variable). The quantized index X is obtained by scaling the i'th latent variable z_i by a scaling factor s_{i,q} that depends on both i and on the quantizer q. We then apply a "dead-zone" function zeta(z) = z - d*tanh(z / (d + epsilon)), where d also depends on i and q, and epsilon=0.1. The result is then rounded to the nearest integer: X = round(zeta(s_{i,q}*z_i)). The Laplace distribution used for entropy coding is parameterized with a probability that the value is zero (p0), as well as a decay factor r (0 < r < 1). Both p0 and r depend on i and q. The probability p(X) for a coefficient is given by:¶

                          /
                          | p0               ,   if X = 0
                          |
                   P(X) = <             |X|
                          | (1 - p0) * r     ,   if X != 0
                          | ---------------
                          \   2 * (1 - r)

2.2.2. Decoder architecture

Unlike the encoder, the decoder is normative. The decoder uses the same Laplace distribution above to decode the symbols and then scales them back by 1/s_{i,q}. The initial state is used as input to initialize the decoder's gated recurrent units (GRUs). The latent vectors are used one at a time as input the DNN decoder, which produces 4 vectors of 20 acoustic features for each input latent vector.¶

The decoder is mostly structured as a DenseNet network, with 5 sets of alternating GRU and convolutional layers. Let gru1..gru5 denote the 5 GRUs, conv1..conv5 denote the 5 convolutional layers, hidden_init/gru_init/dense1/output/cdense* denote fully-connected layers, glu1..glu5 denote gated linear units (GLUs), and cat() denote tensor concatenation. All GRU layers have 64 outputs (number of neurons) and all convolutional layers and cdense* layers have 32 outputs. Despite using a functional notation, both the GRU and convolutional layers have an internal state when used one latent vector at a time. The fully-connected layers all have different sizes. Unless otherwise noted, the GRUs, convolutional and fully-connected layers all use tanh output activations and the GRUs use sigmoid as gate activation. GLUs are defined as:¶

                   L(y) = sigmoid(W*y)*y

where y is the input and W is a square matrix of the same dimensions as y. The decoder starts with the 50-dimensional initial state vector IS. The IS is used to compute the GRU initialization vector V using both hidden_init and gru_init:¶

                   V = gru_init(hidden_init(IS))

where hidden_init has 50 inputs and 128 output, and D2 has 128 inputs and 320 (5*64) outputs. The components of V are split (sequentially) into the V1..V5 initialization vectors (original state before the decoding process) for GRUs gru1..gru5. Let Z be a 25-dimensional vector constructed from the decoded 24-dimensional latent vector for a particular 40-ms chunk, to which we append the value of Q0/8-1, where Q0 is the initial quantizer (see below). From there, the DenseNet structure can be expressed as:¶

                   t1 = dense1(Z)
                   t2 = cat(t1, conv1(cdense1(t1)))
                   t3 = cat(t2, glu1(gru1(t2)))
                   t4 = cat(t3, conv2(cdense2(t3)))
                   t5 = cat(t4, glu2(gru2(t4)))
                   t6 = cat(t5, conv3(cdense3(t5)))
                   t7 = cat(t6, glu3(gru3(t6)))
                   t8 = cat(t7, conv4(cdense4(t7)))
                   t9 = cat(t8, glu4(gru4(t8)))
                   t10 = cat(t9, conv5(cdense5(t9)))
                   t11 = cat(t10, glu5(gru5(t10)))
                   x = output(t11)

where t1..tN are temporary vectors and "output" is the only layer to have a linear output activation, with 80 output neurons (4*20). The dimensionality of t1..t11 (and corresponding GRU/convolutional input size) can be inferred from the concatenation operations. The output vector x is split (sequentially) into 4 feature vectors of 20 dimensions each that can be sent to the vocoder is packets are lost.¶

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                           Header ID                           |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                            Version                            |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                              Type                             |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                              Size                             |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                           Block size                          |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                         Name (44 bytes)                       |
   :                               ...                             :
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                       Data (N x 64 bytes)                     |
   :                               ...                             :
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

2.2.3. Statistical data

We define 16 different quantization settings, ranging from q=0 (higher bitrate) to q=15 (lower bitrate). For each quantizer and for each latent variable or initial state coefficient, we have a normative scale (s), decay (r), and p0 value. Note that the dead-zone parameters d are not normative.¶

2.2.4. Vocoder

A vocoder is needed to turn the acoustic features into actual speech to fill in the audio for any missing packets. Although the decoder is not normative, certain properties are needed for DRED to function adequately. First, the vocoder SHOULD be able to start synthesizing speech by continuing an existing waveform, reducing the artifacts caused at the beginning of a lost packet. If such property cannot be achieved, then the implementation SHOULD at least make an attempt to synchronize the phase of the synthesized speech with the last received speech, and attempt some form of blending, e.g. by splicing the signals in the LPC residual domain.¶

A second important property of the vocoder is to not rely on more than one feature vector of look-ahead. To synthesize speech between time t-10ms and t, the vocoder SHOULD NOT rely on acoustic features centered beyond t+5ms (i.e. covering t-5ms to t+15ms). The vocoder MAY use more look-ahead when it is available, but there are cases (e.g. last lost packet) where the amount of acoustic feature vectors will be limited. For frames sizes less than 20 ms, the decoder SHOULD be prelated to deal with having less than one feature vector of look-ahead.¶