Latent Codecs¶
Latent space encoding and decoding modules.
Overview¶
Latent codecs handle the encoding and decoding of latent representations in compression models. They combine entropy models with quantization and context modeling.
Bases: LatentCodec
Reconstructs groups of channels using previously decoded groups.
Context model from [Minnen2020] and [He2022]. Also known as a "channel-conditional" (CC) entropy model.
See :py:class:~tinify.models.sensetime.Elic2022Official
for example usage.
[Minnen2020]: "Channel-wise Autoregressive Entropy Models for
Learned Image Compression" <https://arxiv.org/abs/2007.08739>_, by
David Minnen, and Saurabh Singh, ICIP 2020.
[He2022]: "ELIC: Efficient Learned Image Compression with
Unevenly Grouped Space-Channel Contextual Adaptive Coding"
<https://arxiv.org/abs/2203.10886>_, by Dailan He, Ziming Yang,
Weikun Peng, Rui Ma, Hongwei Qin, and Yan Wang, CVPR 2022.
CheckerboardLatentCodec(latent_codec, entropy_parameters, context_prediction, anchor_parity='even', forward_method='twopass', **kwargs)
Bases: LatentCodec
Reconstructs latent using 2-pass context model with checkerboard anchors.
Checkerboard context model introduced in [He2021].
See :py:class:~tinify.models.sensetime.Cheng2020AnchorCheckerboard
for example usage.
forward_method="onepass"is fastest, but does not use quantization based on the intermediate means. Uses noise to model quantization.forward_method="twopass"is slightly slower, but accurately quantizes via STE based on the intermediate means. Uses the same operations as Chandelier2023.forward_method="twopass_faster"uses slightly fewer redundant operations.
[He2021]: "Checkerboard Context Model for Efficient Learned Image
Compression" <https://arxiv.org/abs/2103.15306>_, by Dailan He,
Yaoyan Zheng, Baocheng Sun, Yan Wang, and Hongwei Qin, CVPR 2021.
https://github.com/VincentChandelier/ELiC-ReImplemetation`_, by Vincent Chandelier, 2023.
.. warning:: This implementation assumes that entropy_parameters
is a pointwise function, e.g., a composition of 1x1 convs and
pointwise nonlinearities.
.. code-block:: none
0. Input:
□ □ □ □
□ □ □ □
□ □ □ □
1. Decode anchors:
◌ □ ◌ □
□ ◌ □ ◌
◌ □ ◌ □
2. Decode non-anchors:
■ ◌ ■ ◌
◌ ■ ◌ ■
■ ◌ ■ ◌
3. End result:
■ ■ ■ ■
■ ■ ■ ■
■ ■ ■ ■
LEGEND:
■ decoded
◌ currently decoding
□ empty
Bases: LatentCodec
Entropy bottleneck codec.
Factorized prior "entropy bottleneck" introduced in
"Variational Image Compression with a Scale Hyperprior"
<https://arxiv.org/abs/1802.01436>_,
by J. Balle, D. Minnen, S. Singh, S.J. Hwang, and N. Johnston,
International Conference on Learning Representations (ICLR), 2018.
.. code-block:: none
┌───┐ y_hat
y ──►──┤ Q ├───►───····───►─── y_hat
└───┘ EB
Bases: LatentCodec
Entropy bottleneck codec with surrounding h_a and h_s transforms.
Gain-controlled side branch for hyperprior introduced in
"Asymmetric Gained Deep Image Compression With Continuous Rate Adaptation"
<https://arxiv.org/abs/2003.02012>_, by Ze Cui, Jing Wang,
Shangyin Gao, Bo Bai, Tiansheng Guo, and Yihui Feng, CVPR, 2021.
.. note:: GainHyperLatentCodec should be used inside
GainHyperpriorLatentCodec to construct a full hyperprior.
.. code-block:: none
gain gain_inv
│ │
▼ ▼
┌───┐ z │ ┌───┐ z_hat z_hat │ ┌───┐
y ──►──┤h_a├──►──×──►──┤ Q ├───►───····───►────×────►──┤h_s├──►── params
└───┘ └───┘ EB └───┘
Bases: LatentCodec
Hyperprior codec constructed from latent codec for y that
compresses y using params from hyper branch.
Gain-controlled hyperprior introduced in
"Asymmetric Gained Deep Image Compression With Continuous Rate Adaptation"
<https://arxiv.org/abs/2003.02012>_, by Ze Cui, Jing Wang,
Shangyin Gao, Bo Bai, Tiansheng Guo, and Yihui Feng, CVPR, 2021.
.. code-block:: none
z_gain z_gain_inv
│ │
▼ ▼
┌┴────────┴┐
┌──►──┤ lc_hyper ├──►─┐
│ └──────────┘ │
│ │
│ y_gain ▼ params y_gain_inv
│ │ │ │
│ ▼ │ ▼
│ │ ┌──┴───┐ │
y ──┴────►───×───►─────┤ lc_y ├────►─────×─────►── y_hat
└──────┘
By default, the following codec is constructed:
.. code-block:: none
z_gain z_gain_inv
│ │
▼ ▼
┌───┐ z │ z_g ┌───┐ z_hat z_hat │ ┌───┐
┌─►──┤h_a├──►──×──►──┤ Q ├───►───····───►────×────►──┤h_s├──┐
│ └───┘ └───┘ EB └───┘ │
│ │
│ ┌──────────────◄─────────────┘
│ │ params
│ ┌──┴──┐
│ y_gain │ EP │ y_gain_inv
│ │ └──┬──┘ │
│ ▼ │ ▼
│ │ ┌───┐ ▼ │
y ──┴───►───×───►───┤ Q ├────►────····───►─────×─────►── y_hat
└───┘ GC
Common configurations of latent codecs include
- entropy bottleneck
hyper(default) and gaussian conditionaly(default) - entropy bottleneck
hyper(default) and autoregressivey
GaussianConditionalLatentCodec(scale_table=None, gaussian_conditional=None, entropy_parameters=None, quantizer='noise', chunks=('scales', 'means'), **kwargs)
Bases: LatentCodec
Gaussian conditional for compressing latent y using ctx_params.
Probability model for Gaussian of (scales, means).
Gaussian conditonal entropy model introduced in
"Variational Image Compression with a Scale Hyperprior"
<https://arxiv.org/abs/1802.01436>_,
by J. Balle, D. Minnen, S. Singh, S.J. Hwang, and N. Johnston,
International Conference on Learning Representations (ICLR), 2018.
.. note:: Unlike the original paper, which models only the scale (i.e. "width") of the Gaussian, this implementation models both the scale and the mean (i.e. "center") of the Gaussian.
.. code-block:: none
ctx_params
│
▼
│
┌──┴──┐
│ EP │
└──┬──┘
│
┌───┐ y_hat ▼
y ──►──┤ Q ├────►────····──►── y_hat
└───┘ GC
Bases: LatentCodec
Entropy bottleneck codec with surrounding h_a and h_s transforms.
"Hyper" side-information branch introduced in
"Variational Image Compression with a Scale Hyperprior"
<https://arxiv.org/abs/1802.01436>_,
by J. Balle, D. Minnen, S. Singh, S.J. Hwang, and N. Johnston,
International Conference on Learning Representations (ICLR), 2018.
.. note:: HyperLatentCodec should be used inside
HyperpriorLatentCodec to construct a full hyperprior.
.. code-block:: none
┌───┐ z ┌───┐ z_hat z_hat ┌───┐
y ──►──┤h_a├──►──┤ Q ├───►───····───►───┤h_s├──►── params
└───┘ └───┘ EB └───┘
Bases: LatentCodec
Hyperprior codec constructed from latent codec for y that
compresses y using params from hyper branch.
Hyperprior entropy modeling introduced in
"Variational Image Compression with a Scale Hyperprior"
<https://arxiv.org/abs/1802.01436>_,
by J. Balle, D. Minnen, S. Singh, S.J. Hwang, and N. Johnston,
International Conference on Learning Representations (ICLR), 2018.
.. code-block:: none
┌──────────┐
┌─►──┤ lc_hyper ├──►─┐
│ └──────────┘ │
│ ▼ params
│ │
│ ┌──┴───┐
y ──┴───────►─────────┤ lc_y ├───►── y_hat
└──────┘
By default, the following codec is constructed:
.. code-block:: none
┌───┐ z ┌───┐ z_hat z_hat ┌───┐
┌─►──┤h_a├──►──┤ Q ├───►───····───►───┤h_s├──►─┐
│ └───┘ └───┘ EB └───┘ │
│ │
│ ┌──────────────◄────────────┘
│ │ params
│ ┌──┴──┐
│ │ EP │
│ └──┬──┘
│ │
│ ┌───┐ y_hat ▼
y ──┴─►─┤ Q ├────►────····────►── y_hat
└───┘ GC
Common configurations of latent codecs include
- entropy bottleneck
hyper(default) and gaussian conditionaly(default) - entropy bottleneck
hyper(default) and autoregressivey
Bases: LatentCodec
Autoregression in raster-scan order with local decoded context.
PixelCNN context model introduced in
"Pixel Recurrent Neural Networks"
<http://arxiv.org/abs/1601.06759>_,
by Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu,
International Conference on Machine Learning (ICML), 2016.
First applied to learned image compression in
"Joint Autoregressive and Hierarchical Priors for Learned Image
Compression" <https://arxiv.org/abs/1809.02736>_,
by D. Minnen, J. Balle, and G.D. Toderici,
Adv. in Neural Information Processing Systems 31 (NeurIPS 2018).
.. code-block:: none
ctx_params
│
▼
│ ┌───◄───┐
┌─┴─┴─┐ ┌──┴──┐
│ EP │ │ CP │
└──┬──┘ └──┬──┘
│ │
│ ▲
┌───┐ y_hat ▼ │
y ──►──┤ Q ├────►────····───►──┴──►── y_hat
└───┘ GC