Multimedia Information Extraction: Advances in Video, Audio, and Imagery Analysis for Search, Data Mining, Surveillance and Authoring

The definitive guide to the state of the art of multimediainformation extraction

Government analysts, think tank researchers, managers at topwebsites basically everyone is searching for the bestways to access and exploit the vast amounts of multimedia data madeavailable over large networks every day. Written by aninternational team of experts, Multimedia InformationExtraction provides a detailed road map to how that′s done.

The first book to address not only multimedia retrieval but alsoinformation extraction from and across media, it offers diverseperspectives on how this emerging technology can help meet thegrowing demand in industry and government for stock media access,media preservation, broadcast news retrieval, identity management,video surveillance, and more.

Including a Foreword by Professor Alan Smeaton, foundingcoordinator of the international TRECVid, Multimedia InformationExtraction covers:

The fundamental issues in processing and multimedia sourceextraction The history and state of the art of multimedia informationextraction Image and video extraction, with tools ranging from visualfeature localization to social redundancy Affect extraction in audio and imagery, from paralinguisticinformation retrieval to affect–based indexing Multimedia annotation and authoring

An inspiring, much–needed resource for researchers anddevelopers in government, industry, and academia, this book alsooffers guidance on using the material in the core curriculum of ACMSIGCHI, ACM/IEEE Computer Science, and ACM/IEEE InformationTechnology.

FOREWORD ix

Alan F. Smeaton

PREFACE xiii
Mark T. Maybury

ACKNOWLEDGMENTS xv

CONTRIBUTORS xvii

1 INTRODUCTION 1
Mark T. Maybury

2 MULTIMEDIA INFORMATION EXTRACTION: HISTORY AND STATE OF THEART 13
Mark T. Maybury

SECTION 1 IMAGE EXTRACTION 41

3 VISUAL FEATURE LOCALIZATION FOR DETECTING UNIQUE OBJECTS INIMAGES 45
Madirakshi Das, Alexander C. Loui, and Andrew C. Blose

4 ENTROPY–BASED ANALYSIS OF VISUAL AND GEOLOCATION CONCEPTS INIMAGES 63
Keiji Yanai, Hidetoshi Kawakubo, and Kobus Barnard

5 THE MEANING OF 3D SHAPE AND SOME TECHNIQUES TO EXTRACT IT81
Sven Havemann, Torsten Ullrich, and Dieter W. Fellner

6 A DATA–DRIVEN MEANINGFUL REPRESENTATION OF EMOTIONAL FACIALEXPRESSIONS 99
Nicolas Stoiber, Gaspard Breton, and Renaud Seguier

SECTION 2 VIDEO EXTRACTION 113

7 VISUAL SEMANTICS FOR REDUCING FALSE POSITIVES IN VIDEO SEARCH119
Rohini K. Srihari and Adrian Novischi

8 AUTOMATED ANALYSIS OF IDEOLOGICAL BIAS IN VIDEO 129
Wei–Hao Lin and Alexander G. Hauptmann

9 MULTIMEDIA INFORMATION EXTRACTION IN A LIVE MULTILINGUAL NEWSMONITORING SYSTEM 145
David D. Palmer, Marc B. Reichman, and Noah White

10 SEMANTIC MULTIMEDIA EXTRACTION USING AUDIO AND VIDEO159
Evelyne Tzoukermann, Geetu Ambwani, Amit Bagga, Leslie Chipman,Anthony R. Davis, Ryan Farrell, David Houghton, Oliver Jojic, JanNeumann, Robert Rubinoff, Bageshree Shevade, and HongzhongZhou

11 ANALYSIS OF MULTIMODAL NATURAL LANGUAGE CONTENT IN BROADCASTVIDEO 175
Prem Natarajan, Ehry MacRostie, Rohit Prasad, and JonathanWatson

12 WEB–BASED MULTIMEDIA INFORMATION EXTRACTION BASED ON SOCIALREDUNDANCY 185
Jose San Pedro, Stefan Siersdorfer, Vaiva Kalnikaite, and SteveWhittaker

13 INFORMATION FUSION AND ANOMALY DETECTION WITH UNCALIBRATEDCAMERAS IN VIDEO SURVEILLANCE 201
Erhan Baki Ermis, Venkatesh Saligrama, and Pierre–MarcJodoin

SECTION 3 AUDIO, GRAPHICS, AND BEHAVIOR EXTRACTION217

14 AUTOMATIC DETECTION, INDEXING, AND RETRIEVAL OF MULTIPLEATTRIBUTES FROM CROSS–LINGUAL MULTIMEDIA DATA 221
Qian Hu, Fred J. Goodman, Stanley M. Boykin, Randall K. Fish,Warren R. Greiff, Stephen R. Jones, and Stephen R. Moore

15 INFORMATION GRAPHICS IN MULTIMODAL DOCUMENTS 235
Sandra Carberry, Stephanie Elzer, Richard Burns, Peng Wu, DanielChester, and Seniz Demir

16 EXTRACTING INFORMATION FROM HUMAN BEHAVIOR 253
Fabio Pianesi, Bruno Lepri, Nadia Mana, Alessandro Cappelletti,and Massimo Zancanaro

SECTION 4 AFFECT EXTRACTION FROM AUDIO AND IMAGERY269

17 RETRIEVAL OF PARALINGUISTIC INFORMATION IN BROADCASTS273
Björn Schuller, Martin Wöllmer, Florian Eyben, andGerhard Rigoll

18 AUDIENCE REACTIONS FOR INFORMATION EXTRACTION ABOUTPERSUASIVE LANGUAGE IN POLITICAL COMMUNICATION 289
Marco Guerini, Carlo Strapparava, and Oliviero Stock

19 THE NEED FOR AFFECTIVE METADATA IN CONTENT–BASED RECOMMENDERSYSTEMS FOR IMAGES 305
Marko TkalÈiÈ, Jurij TasiÈ, and AndrejKo ir

20 AFFECT–BASED INDEXING FOR MULTIMEDIA DATA 321
Gareth J. F. Jones and Ching Hau Chan

SECTION 5 MULTIMEDIA ANNOTATION AND AUTHORING 347

21 MULTIMEDIA ANNOTATION, QUERYING, AND ANALYSIS IN ANVIL351
Michael Kipp

22 TOWARD FORMALIZATION OF DISPLAY GRAMMAR FOR INTERACTIVE MEDIAPRODUCTION WITH MULTIMEDIA INFORMATION EXTRACTION 369
Robin Bargar

23 MEDIA AUTHORING WITH ONTOLOGICAL REASONING: USE CASE FORMULTIMEDIA INFORMATION EXTRACTION 385
Insook Choi

24 ANNOTATING SIGNIFICANT RELATIONS ON MULTIMEDIA WEB DOCUMENTS401
Matusala Addisu, Danilo Avola, Paola Bianchi, Paolo Bottoni,Stefano Levialdi, and Emanuele Panizzi

ABBREVIATIONS AND ACRONYMS 419

REFERENCES 425

INDEX 461

MARK T. MAYBURY, PhD, is an Executive Director at MITRE, a federally funded research and development center. In 2010, Dr. Maybury took a leave of absence from MITRE when appointed to the role of Chief Scientist of the United States Air Force. He is a former member of the board of directors of the Object Management Group and the Air Force Scientific Advisory Board. An IEEE Fellow, Dr. Maybury is a member of the ACM Intelligent User Interface Steering Committee and has served on the Advanced Visual Interfaces Program Committee for over ten years. He holds several patents and has edited, coedited, or coauthored a number of books on information retrieval and related topics.