3dtv: Processing and Transmission of 3D Video Signals

3DTV introduces a number of new challenges compared to existing television production and broadcast scenarios. This book introduces the key concepts associated with 3DTV, examining all major aspects of the 3DTV chain from capture to display. It provides readers with a systems–level appreciation of 3DTV systems, allowing an understanding of the fundamental principles behind each part of the chain. Key features: Provides a systematic introduction and description of 3DTV key techniques, which build up the whole 3DTV system from capture to the consumer viewing at home Details the workflow within the 3DTV delivery chain, including 3D media capture, processing, coding, transmission and rendering Offers a state–of–the–art review of key techniques in 3DTV – discusses recent advances and outlines the future of the 3DTV services by addressing the areas that need further progress Addresses the fast–moving field of 3D displays  

Preface ix Acknowledgements xi List of Abbreviations xiii 1 Introduction 1 1.1 History of 3D Video 2 1.1.1 3D in the Nineteenth Century 3 1.1.2 Early Twentieth–Century Developments 4 1.1.3 The 1950s ‘Golden’ Period 6 1.1.4 The 1980s Revival and the Arrival of IMAX 8 1.1.5 The Twenty–first–Century Revival 12 1.1.6 Auto–Stereoscopic 13 1.1.7 3D Television Broadcasts 14 1.2 3D Video Formats 17 1.2.1 Frame Compatible and Service Compatible Stereoscopic Video 17 1.2.2 Colour–Plus–Depth 20 1.2.3 Multi–View Video 22 1.2.4 Multi–View Plus Depth Video 23 1.2.5 Layered Depth Video 23 1.3 3D Video Application Scenarios 25 1.3.1 3DTV Broadcast Systems 25 1.3.2 Mobile 3DTV 26 1.3.3 3D Video on Demand 27 1.3.4 3D Immersive Video–Conferencing 28 1.3.5 Remote Applications 29 1.4 Motivation 29 1.5 Overview of the Book 30 References 31 2 Capture and Processing 34 2.1 3D Scene Representation Formats and Techniques 34 2.2 3D Video Capturing Techniques 36 2.2.1 Camera Technologies 37 2.2.2 Stereoscopic Video Capture 39 2.2.3 Multi–View Video Capture 45 2.2.4 Integral Imaging Capture 50 2.3 3D Video Processing 52 2.3.1 Rectification and Colour Correction 54 2.3.2 Extraction of Range Images 56 References 59 3 Compression 61 3.1 Video Coding Principles 61 3.2 Overview of Traditional Video Coding Standards 64 3.2.1 Overview of MPEG–4 Part 10/H.264 AVC Standard 65 3.2.2 High Efficiency Video Coding (HEVC) 68 3.3 3D Video Coding 71 3.3.1 Stereoscopic Video Coding 71 3.3.2 Multi–View Video Coding 73 3.3.3 Coding of Multi–View Plus Depth 78 3.4 Recent Trends in 3D Video Coding 84 3.4.1 3D Video with AVC–Based Coding Technology 86 3.4.2 3D Video with HEVC–Based Coding Technology 86 References 87 4 Transmission 91 4.1 Challenges of 3D Video Transmission 91 4.2 Error Resilience and Concealment Techniques 91 4.2.1 Background 92 4.2.2 Error Resilience Tools 93 4.2.3 Forward Error Correction (FEC) 97 4.3 3D Video Transmission: Example Scenarios 98 4.3.1 3D Video Broadcast over DVB–T 98 4.3.2 3D Video Streaming over IP Networks 102 4.3.3 3D Video Transmission over Mobile Broadband 105 4.4 Conclusion 121 References 121 5 Rendering, Adaptation and 3D Displays 123 5.1 Why Rendering? 123 5.2 3D Video Rendering 124 5.3 3D Video Adaptation 135 5.3.1 Importance of the Depth Map in Adaptation 135 5.3.2 Context Adaptation 136 5.3.3 3D Video Adaptation for Mobile Terminals 136 5.3.4 Multi–View Video Adaptation 138 5.4 3D Display Technologies 140 5.4.1 Anaglyphic Stereoscopic Displays 141 5.4.2 Passive Stereoscopic Displays 142 5.4.3 Active Stereoscopic Displays 143 5.4.4 Auto–Stereoscopic Displays 144 5.4.5 Light–Field Display 146 References 148 6 Quality Assessment 150 6.1 2D Video Quality Metrics 150 6.1.1 Peak–Signal–to–Noise–Ratio (PSNR) 151 6.1.2 Structural Similarity Index (SSIM) 151 6.1.3 Video Quality Metric (VQM) 151 6.2 3D Video Quality 152 6.2.1 Image Quality 153 6.2.2 Visual Perception of Depth 153 6.3 3D Video Quality Evaluation Methods 159 6.3.1 Subjective and Objective Quality Measurements 162 6.3.2 Effects of Colour Texture Video and Depth Maps on Perceptual Quality 167 6.4 Modelling the Perceptual Attributes of 3D Video 168 6.4.1 Modelling the Image Quality of 3D Video 169 6.4.2 Modelling the Depth Quality of 3D Video 170 6.4.3 Compound 3D Video Quality Model 179 6.4.4 Application of the Proposed Quality Models 182 6.4.5 Context Dependency of Visual Experience 183 6.4.6 3D–Specific Technical Properties that Affect the Viewing Experience 184 6.5 Conclusion 185 References 186 7 Conclusions and the Future of 3DTV 188 7.1 Chapter Summary 188 7.1.1 Chapter 1: Introduction 188 7.1.2 Chapter 2: Capture and Processing 189 7.1.3 Chapter 3: Compression 189 7.1.4 Chapter 4: Transmission 190 7.1.5 Chapter 5: Rendering and 3D Displays 190 7.1.6 Chapter 6: Quality Assessment 191 7.2 The Future of 3DTV 191 7.2.1 Understanding of Human 3D Perception 191 7.2.2 Display Technologies 192 7.2.3 Production Approaches and Technologies 193 7.2.4 Compression Algorithms 195 7.2.5 Looking Further Ahead 196 Appendix A Test Video Sequences 197 A.1 2D Video Test Sequences 197 A.2 3D Test Video Sequences 198 Appendix B Introduction to the Experiment and Questionnaire 200 B.1 Introduction to the Experiment 200 B.2 Questionnaire 203 Index 205

Anil Fernando , CVSSP, University of Surrey, UK Stewart T. Worrall , Ericsson Television, UK Erhan Ekmekcioglu , CVSSP, University of Surrey, UK