# Design of Efficient Perspective Affine Motion Estimation/Compensation for Versatile Video Coding (VVC) Standard

^{1}

^{2}

^{3}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Affine Motion Estimation/Compensation in VVC

#### 2.1. 4-Parameter and 6-Parameter Affine Model

#### 2.2. Affine Inter Mode and Merge Mode

- CPMVPs extrapolated from the CPMVs of the spatial neighbour blocks
- CPMVPs constructed using the translational MVs of the spatial neighbour blocks
- CPMVPs generated by duplicating each of the HEVC AMVP candidates

## 3. Proposed Perspective Affine Motion Estimation/Compensation

#### 3.1. Perspective Motion Model for Motion Estimation

#### 3.2. Perspective Affine Motion Compensation

## 4. Experimental Results

## 5. Conclusions

## Author Contributions

## Funding

## Conflicts of Interest

## References

- Draft ITU-T Recommendation and final draft international standard of joint video specification (ITU-T Rec. H. 264| ISO/IEC 14496-10 AVC). In Proceedings of the 5th Meeting, Joint Video Team (JVT) of ISO/IEC MPEG and ITU-T VCEG, Geneva, Switzerland, 9–17 October 2002.
- Sze, V.; Budagavi, M.; Sullivan, G.J. High efficiency video coding (HEVC). In Integrated Circuit and Systems, Algorithms and Architectures; Springer: Berlin, Germany, 2014; pp. 1–375. [Google Scholar]
- Bross, B.; Chen, J.; Liu, S. Versatile Video Coding (Draft 2); JVET-K1001; JVET: Ljubljana, Slovenia, 2018. [Google Scholar]
- Chen, J.; Alshina, E. Algorithm Description for Versatile Video Coding and Test Model 1 (VTM 1); JVET-J1002; JVET: San Diego, CA, USA, 2018. [Google Scholar]
- Chen, J.; Ye, Y.; Kim, S. Algorithm Description for Versatile Video Coding and Test Model 2 (VTM 2); JVET-K1002; JVET: Ljubljana, Slovenia, 2018. [Google Scholar]
- Seferidis, V.; Ghanbari, M. General approach to block-matching motion estimation. Opt. Eng.
**1993**, 32, 1464–1474. [Google Scholar] [CrossRef][Green Version] - Lee, O.; Wang, Y. Motion compensated prediction using nodal based deformable block matching. J. Vis. Commun. Image Represent.
**1995**, 6, 26–34. [Google Scholar] [CrossRef] - Cheung, H.K.; Siu, W.C. Local affine motion prediction for H.264 without extra overhead. In Proceedings of the IEEE International Symposium on Circuits and Systems (ISCAS), Paris, France, 30 May–2 June 2010; pp. 1555–1558. [Google Scholar]
- Kim, H.-S.; Lee, J.-H.; Kim, C.-K.; Kim, B.-G. Zoom motion estimation using block-based fast local area scaling. IEEE Trans. Circuits Syst. Video Technol.
**2012**, 22, 1280–1291. [Google Scholar] [CrossRef] - Narroschke, M.; Swoboda, R. Extending HEVC by an affine motion model. In Proceedings of the 2013 Picture Coding Symposium (PCS), San Jose, CA, USA, 8–11 December 2013; pp. 321–324. [Google Scholar]
- Huang, H.; Woods, J.W.; Zhao, Y.; Bai, H. Affine SKIP and DIRECT modes for efficient video coding. In Proceedings of the Visual Communications and Image Processing (VCIP), San Diego, CA, USA, 27–30 November 2012; pp. 1–6. [Google Scholar]
- Heithausen, C.; Vorwerk, J.H. Motion compensation with higher order motion models for HEVC. In Proceedings of the 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brisbane, QLD, Australia, 19–24 April 2015; pp. 1438–1442. [Google Scholar]
- Chen, H.; Liang, F.; Lin, S. Affine SKIP and MERGE modes for video coding. In Proceedings of the 2015 IEEE 17th International Workshop on Multimedia Signal Processing (MMSP), Xiamen, China, 19–21 October 2015; pp. 1–5. [Google Scholar]
- Heithausen, C.; Bläser, M.; Wien, M.; Ohm, J.R. Improved higher order motion compensation in HEVC with block-to-block translational shift compensation. In Proceedings of the 2016 IEEE International Conference on Image Processing (ICIP), Phoenix, AZ, USA, 25–28 September 2016; pp. 2008–2012. [Google Scholar]
- Sullivan, G.J.; Ohm, J.R.; Han, W.J.; Wiegand, T. Overview of the high efficiency video coding (HEVC) standard. IEEE Trans. Circuits Syst. Video Technol.
**2012**, 22, 1649–1668. [Google Scholar] [CrossRef] - Heithausen, C.; Meyer, M.; Bläser, M.; Ohm, J.R. Temporal prediction of motion parameters with interchangeable motion model. In Proceedings of the 2017 Data Compression Conference (DCC), Snowbird, UT, USA, 4–7 April 2017; pp. 400–409. [Google Scholar]
- Li, L.; Li, H.; Lv, Z.; Yang, H. An affine motion compensation framework for High Efficiency Video Coding. In Proceedings of the IEEE International Symposium on Circuits and Systems (ISCAS), Lisbon, Portugal, 24–27 May 2015; pp. 525–528. [Google Scholar]
- Li, L.; Li, H.; Liu, D.; Li, Z.; Yang, H.; Lin, S.; Chen, H.; Wu, F. An Efficient Four-Parameter Affine Motion Model for Video Coding. IEEE Trans. Circuits Syst. Video Technol.
**2018**, 28, 1934–1948. [Google Scholar] [CrossRef] - Lin, S.; Chen, H.; Zhang, H.; Maxim, S.; Yang, H.; Zhou, J. Affine Transform Prediction for Next Generation Video Coding; ITU-T SG16 Doc. COM16–C1016; Huawei Technologies: Shenzhen, China, 2015. [Google Scholar]
- Chen, J.; Alshina, E.; Sullivan, G.J.; Ohm, J.R.; Boyce, J. Algorithm Description of Joint Exploration Test Model 1; JVET-A1001; JVET: Geneva, Switzerland, 2015. [Google Scholar]
- Choi, Y.J.; Kim, J.H.; Lee, J.H.; Kim, B.G. Performance Analysis of Future Video Coding (FVC) Standard Technology. J. Multimed. Inf. Syst.
**2017**, 4, 73–78. [Google Scholar] - Zhang, K.; Chen, Y.W.; Zhang, L.; Chien, W.J.; Karczewicz, M. An Improved Framework of Affine Motion Compensation in Video Coding. IEEE Trans. Image Process.
**2019**, 28, 1456–1469. [Google Scholar] [CrossRef] [PubMed] - Lichtenauer, J.F.; Sirmacek, B. A semi-automatic procedure for texturing of laser scanning point clouds with google streetview images. Available online: https://repository.tudelft.nl/islandora/object/uuid%3A8bb4d40b-0950-471f-b774-7f74449fe26e (accessed on 16 September 2019).
- Versatile Video Coding (VVC) Test Model 2.0 (VTM 2.0). Available online: https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM.git (accessed on 16 September 2018).
- Bossen, F.; Boyce, J.; Suehring, K.; Li, X.; Seregin, V. JVET Common Test Conditions and Software Reference Configurations for SDR Video; JVET-K1010; JVET: Ljubljana, Slovenia, 2018. [Google Scholar]
- Bjøntegaard, G. Calculation of Average PSNR Differences between RDcurves; ITU-T SG.16 Q.6, Document VCEG-M33; ITU-T VCEG: Austin, TX, USA, 2001. [Google Scholar]
- Hanhart, P.; Boyce, J.; Choi, K. JVET Common Test Conditions and Evaluation Procedures for 360 Video; JVET-K1012: Ljubljana, Slovenia, 2018. [Google Scholar]
- Yoon, Y.U.; Kim, H.H.; Lee, Y.J.; Kim, J.G. Methods of padding inactive regions for rotated sphere projection of 360 video. In Proceedings of the 2019 Joint International Workshop on Advanced Image Technology (IWAIT) and International Forum on Medical Imaging in Asia (IFMIA), Singapore, 6–9 January 2019. [Google Scholar]
- Ma, S.; Lin, Y.; Zhu, C.; Zheng, J.; Yu, L.; Wang, X. Improved segment-wise DC coding for HEVC intra prediction of depth maps. In Proceedings of the Signal and Information Processing Association Annual Summit and Conference (APSIPA), Siem Reap, Cambodia, 9–12 December 2014. [Google Scholar]
- Zhao, J.; Kim, S.H.; Li, G.; Xu, X.; Li, X.; Liu, S. CE2: History Based Affine Motion Candidate (Test 2.2.3); JVET-M0125; JVET: Marrakech, Morocco, 2019. [Google Scholar]
- Galpin, F.; Robert, A.; Le Léannec, F.; Poirier, T. CE2.2.7: Affine Temporal Constructed Candidates; JVET-M0256; JVET: Marrakech, Morocco, 2019. [Google Scholar]
- Zhang, K.; Zhang, L.; Liu, H.; Xu, J.; Wang, Y.; Zhao, P.; Hong, D. CE2-Related: History-Based Affine Merge Candidates; JVET-M0266; JVET: Marrakech, Morocco, 2019. [Google Scholar]

**Figure 1.**Example of a non-affine transformation [23].

**Figure 2.**Affine motion vector control points: (

**a**) 4-parameter motion model, (

**b**) 6-parameter motion model.

**Figure 6.**The motion models: (

**a**) 6-parameter affine model with three CPs, (

**b**) perspective model with four CPs.

Class | Sequence | Resolution | RA | LDP | ||||
---|---|---|---|---|---|---|---|---|

Y | U | V | Y | U | V | |||

A | Campfire | 3840 × 2160 | −0.09% | −0.02% | 0.06% | - | - | - |

CatRobot1 | 3840 × 2160 | −0.10% | 0.41% | 0.33% | - | - | - | |

B | RitualDance | 1920 × 1080 | −0.04% | 0.15% | 0.22% | −0.06% | 0.55% | 0.01% |

BasketballDrive | 1920 × 1080 | −0.14% | −0.05% | 0.44% | −0.06% | 0.28% | 0.47% | |

BQTerrace | 1920 × 1080 | −0.08% | −0.11% | −0.03% | −0.08% | 0.38% | −0.16% | |

C | BasketballDrill | 832 × 480 | −0.09% | −0.12% | −0.16% | −0.02% | 0.04% | −0.15% |

PartyScene | 832 × 480 | −0.06% | −0.21% | −0.04% | 0.01% | 0.46% | 0.24% | |

D | BQSquare | 416 × 240 | 0.03% | 0.45% | −0.04% | −0.03% | −0.42% | −1.87% |

RaceHorses | 416 × 240 | 0.07% | −0.76% | 0.09% | −0.25% | 0.21% | 0.49% | |

E | FourPeople | 1280 × 720 | - | - | - | −0.16% | −0.58% | 0.15% |

KristenAndSara | 1280 × 720 | - | - | - | 0.07% | −0.22% | 0.03% | |

F | BasketballDrillText | 832 × 480 | −0.01% | −0.18% | 0.25% | −0.07% | −0.57% | −0.28% |

SlideEditing | 1280 × 720 | −0.07% | −0.03% | −0.03% | −0.28% | −0.22% | −0.47% | |

SlideShow | 1280 × 720 | −0.30% | 1.29% | 0.19% | −0.45% | −1.39% | 0.08% | |

Avg. | −0.07% | 0.07% | 0.11% | −0.12% | −0.12% | −0.12% |

**Table 2.**BD-Rate (%) performance of the proposed algorithm for ERP format sequences compared to VTM 2.0 Baseline.

Sequence | Resolution | LDP | ||
---|---|---|---|---|

Y | U | V | ||

Broadway | 6144 × 3072 | −0.15% | 0.03% | −0.19% |

Freefall | 6144 × 3072 | −0.13% | 0.12% | 0.35% |

BranCastle2 | 6144 × 3072 | −0.08% | −0.11% | −0.28% |

Balboa | 6144 × 3072 | −0.10% | −0.06% | 0.17% |

Avg. | −0.12% | −0.01% | 0.01% |

© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Choi, Y.-J.; Jun, D.-S.; Cheong, W.-S.; Kim, B.-G. Design of Efficient Perspective Affine Motion Estimation/Compensation for Versatile Video Coding (VVC) Standard. *Electronics* **2019**, *8*, 993.
https://doi.org/10.3390/electronics8090993

**AMA Style**

Choi Y-J, Jun D-S, Cheong W-S, Kim B-G. Design of Efficient Perspective Affine Motion Estimation/Compensation for Versatile Video Coding (VVC) Standard. *Electronics*. 2019; 8(9):993.
https://doi.org/10.3390/electronics8090993

**Chicago/Turabian Style**

Choi, Young-Ju, Dong-San Jun, Won-Sik Cheong, and Byung-Gyu Kim. 2019. "Design of Efficient Perspective Affine Motion Estimation/Compensation for Versatile Video Coding (VVC) Standard" *Electronics* 8, no. 9: 993.
https://doi.org/10.3390/electronics8090993