Content Analytics - distill content into visual and statistical representations

What are Content Analytics?

Content analytics is the general term for algorithms that break down content into smaller visual or statistical representations. As opposed to content features, which are intermediate descriptions of content for subsequent machine-based processing, analytics often numerically characterize specific events or objects at a specific time in human-understandable descriptions.  

Example Analytic Services

Aggregated Demographic Statistics

Coupling face detection and image classification modules, one can generate events that characterize specific faces present in a scene. In the example to the right, an advertisement was analyzed for facial demographics, which could then be anonymously combined across several instances (locations) and multiple capture instances (time).

This capability allows facilitates learning about their customers and visitors to a location with wholly passive analysis of opt-in content. While the models can be retrained for specific data, they provide a coarse distribution of properties that could be applied to any type of object (faces, clothing, etc.) that may offer interesting complements to existing telemetry data from other sources.  

 

Traffic flow and dwell analaysis

When describing a peron's experience in a location, marketing managers are often interested in learning about where people walk, linger, and inspect products. In the synthesized content clip on the left, the system automatically determines a background image and observes moving objects (blobs) within the clip. The object locations and movement (both speed and direction) are pooled into small time intervals and saved as events for subsequent analysis. To avoid a full 3D registration of a location (eg, a maticulous, manually specified floorplan), the system assists end user interpretations of what was important about a location with dwell and path heat-map (red is more frequent) overlays.

As a complentary use, non-intrusive, distributed monitoring for elderly individuals could be accomplished by observing walking and lingering patterns and alerting a care provider if an atypical event (e.g. a fall, a missing person during head count, or erratic walking patterns) were detected by the analytics system.  

 

Safety within the Home (Retrieval and Inspection)

In this use case, events created by human motion are aggregated into a timeline-based browser. Much like navigating an online retail website, users can quickly narrow their browsing criterion with a few clicks and selections to reduce the number of events included in their search. In the example below an interative dashboard lets users quickly narrow their criterion to find the desired events.

 

    • A specific time of day (between 1pm and 6pm) is chosen, which corresponds to times that children may be home from school.
    • Only the camera facing the pool is selected for viewing.
    • The system automatically redraws to show only the desired events on the timeline.
    • A duration range (20-50 seconds) is set for events where children linger near the pool.
    • The system indicates that events matching this search occur mainly on the weekend.
    • The timeline is again redrawn to show the few remaining events.
    • The parent can have peace of mind verifying that while children are around the pool, they are always accompanied by an adult!
     
 
 
 

 

Architecture

How can one use content analytics services?

One type of analytic feature might simply describe a region of moving pixels. While the region may also be a specific type of object (a person, a tree, an animal, etc.) the analytical output could describe the speed of movement and the angle of movement in the scene. Combining several of these outputs, a derived analytic might also describe the overall path length of the object over multiple time instances -- like the path of a person walking through a room. Generally a Content Analysis Engine will output and store several different types of analytics for a piece of content so that interesting applications like facted retreival and aggregate analsysis can be performed at a later date. Systems that produce analytics in a real-time fashion can also be used to create enhanced media experiences or summaries of topical social media trends.

Analysis Data Flow

Events: Analytics can have diverse properties (frequency, magnitude, area, etc.) but they must always correspond to one or more points in time within a piece of content. This encapsulation is called an event and mandatory attributes of an event from the CAE are start, duration, region of interest (ROI), event type and unique identifier. For reference within the datastore, each event also has a unique 64-byte id (GUID), the name of its capture stream, and a wall-clock reference for capture time. While not required, most events also include a frame reference that is stored which is exclusively stored on disk.

Modules: Similar to the generic framework for events, individual processing modules also follow a general template. They buffer events as inputs that are validated against a pre-determined type mask, asynchronously process events from the buffer and deliver new events (with unique GUIDs) downstream to other connected modules. To synchronize events across modules, the stream name and wall-clock persist through processing.

Profiles: Profiles define the "receipe" or structural event flow for module creation. Profiles may include one or more of the same module types, connect different modules to output (sink) objects, and in future versions may also have embedded module specific configurations.

Archival and Retrieval: The output of analysis always comes from a "sink" module. If provided with datastore credentials, event data will be stored in a database and filesystem. Otherwise, event metadata will be stored along with any video or image content that the analysis has created.  

 

Where can I access these functionalities?

These functionalities exist in the AT&T CAE℠ but their deployment into the Visual API is still ongoing. If you are interested in one of these technologies, please ask us about how to get it delivered to you more quickly!

Multimedia (videos, demos, interviews)
null    VisualAPI_Analytics_Sampler (0k)


Project Members

Eric Zavesky

David Gibbon

Lee Begeja

Zhu Liu

Bernard Renger

Related Projects

Project Space

AT&T Application Resource Optimizer (ARO) - For energy-efficient apps

Assistive Technology

CHI Scan (Computer Human Interaction Scan)

CoCITe – Coordinating Changes in Text

Connecting Your World

Darkstar

Daytona

E4SS - ECharts for SIP Servlets

Scalable Ad Hoc Wireless Geocast

AT&T 3D Lab

Graphviz System for Network Visualization

Information Visualization Research - Prototypes and Systems

Swift - Visualization of Communication Services at Scale

Smart Grid

Speech Mashup

Omni Channel Analytics

Speech translation

StratoSIP: SIP at a Very High Level

Telehealth

Content Augmenting Media (CAM)

Content-Based Copy Detection

Content Acquisition Processing, Monitoring, and Forensics for AT&T Services (CONSENT)

MIRACLE and the Content Analysis Engine (CAE)

Social TV - View and Contribute to Public Opinions about Your Content Live

Visual API - Visual Intelligence for your Applications

Enhanced Indexing and Representation with Vision-Based Biometrics

Visual Semantics for Intuitive Mid-Level Representations

eClips - Personalized Content Clip Retrieval and Delivery

iMIRACLE - Content Retrieval on Mobile Devices with Speech

AT&T WATSON (SM) Speech Technologies

Wireless Demand Forecasting, Network Capacity Analysis, and Performance Optimization