\documentclass[10pt,twocolumn,letterpaper]{article}

\usepackage{cvpr}
\usepackage{times}
\usepackage{epsfig}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{subfigure}
\usepackage{bm}
\usepackage{multirow}
\usepackage{color}
\newcommand{\feifeix}[1]{{\color{red}{\bf\sf [Fei-Fei: #1]}}}
% Include other packages here, before hyperref.

% If you comment hyperref and then uncomment it, you should delete
% egpaper.aux before re-running latex.  (Or just hit 'q' on the first latex
% run, let it finish, and you should be clear).
\usepackage[pagebackref=true,breaklinks=true,letterpaper=true,colorlinks,bookmarks=false]{hyperref}


% \cvprfinalcopy % *** Uncomment this line for the final submission

\def\cvprPaperID{255} % *** Enter the CVPR Paper ID here
\def\httilde{\mbox{\tt\raisebox{-.5ex}{\symbol{126}}}}

% Pages are numbered in submission mode, and unnumbered in camera-ready
\ifcvprfinal\pagestyle{empty}\fi
\begin{document}

%%%%%%%%% TITLE
%\title{Verbs and the Interaction with Nouns and Human Poses:\\ Constructing a Semantically Meaningful Activity Space}
\title{Recognition in Activity Space by Connecting Verbs, Nouns, and Human Poses}

\author{Bangpeng Yao \hspace*{7mm} Aditya Khosla \hspace*{7mm} Li Fei-Fei\\
Computer Science Department, Stanford University, Stanford, CA\\
{\tt\small {\{bangpeng,aditya86,feifeili\}@cs.stanford.edu}}
% For a paper whose authors are all at the same institution,
% omit the following lines up until the closing ``}''.
% Additional authors and addresses can be added with ``\and'',
% just like the second author.
% To save space, use either the email address or home page, not both
}

\maketitle
% \thispagestyle{empty}

%%%%%%%%% ABSTRACT
\begin{abstract}
  While detection and classification of nouns (in most cases objects) has been enjoying great progress in the past ten years, there is relatively little understanding in computer vision of human activities using verbs and the nouns (i.e. objects) associated with them. In order to represent a semantically meaningful space of human activities, we observe the necessity of using visually rich human poses to link the meanings of verbs and nouns. Given training data, we first learn a dictionary of many different poses, which we call ``atomic poses'', and construct a joint model for the verbs, nouns and atomic poses. The result is a set of learned connectivity among these three concepts, characterizing the space of different human activities. Using this connectivity graph, we can 1) build meaningful taxonomies of verbs and nouns in the activity space, 2) cluster and categorize different types of activities based on the verb-pose-noun interaction, and 3) recognize different components of the activity (object detection, pose estimation). Our model outperforms the state-of-the-art algorithms in a number of tasks using the HOI Sports~\cite{Gupta09} and PASCAL action~\cite{PASCAL10} datasets.
\end{abstract}

%%%%%%%%% BODY TEXT
\section{Introduction}\label{sec:intro}

Human activities are often described using verbs or phrases of verb and nouns, such as when the person is ``walking'' or ``playing basketball''. While computer vision research has made great progress in recognizing nouns~\cite{PASCAL10,Felzenszwalb10}, scaling up to tens of thousands of object classes~\cite{Torralba08,Deng09}), recognizing verbs that describe human activities is still a wide open area of research.

Psychologists have shown that infants learn nouns much earlier than verbs~\cite{Gentner82}. One hypothesis is that most nouns refer to existing entities, often easily accessible to humans. The verbs, on the other hand, tend to describe the concepts that are more abstract and diffuse~\cite{Gentner81}. In this paper, we aim to jointly model verbs and nouns and explore their relationship, which provides a novel representation of human activities.

\begin{figure}[t]
\centering
\includegraphics[width=0.95\linewidth,height=0.75\linewidth]{./figure/pull/pull.eps}
\caption{We represent a human action as a triangulation between a verb, a noun, and a human pose. Our method explicitly learns semantically meaningful connections between sets of verbs, nouns, and human poses. We also deal with the cases where multiple nouns or no noun is included.}\label{fig:pull}
\end{figure}

An illustration of our ideas is shown in Fig.\ref{fig:pull}. The key insight is that {\em verbs} and {\em nouns} are often impoverished linguistic symbols for characterizing human activities. For example, the phrase ``serve a volleyball'' can render different images depending on whether the serve is overhand or underhand. There are also linguistic ambiguities between verbs and nouns where one verb corresponds to different nouns and vice versa; for example, one can serve both a volleyball or a tennis ball, as shown in Fig.\ref{fig:pull}. To better characterize human activities we introduce a third dimension that encodes rich visual information: {\em human poses}, which serve as a visual link between the verbs and the nouns. Possible layouts of body parts are discretized into a set of representative poses that distinguish the ambiguities of verb-noun combinations; for example, ``talking on the phone'' has a large variety of visual appearances depending on the person's pose. We obtain a dictionary of spatial layouts of body parts which we call {\em atomic poses} (see Fig.\ref{fig:atomic_poses} for examples).

Building upon this intuition, we represent human activities as interactions of verbs, nouns, and poses. We observe that connecting these three concepts establishes a semantically meaningful activity space, where recognition of specific components (e.g. objects, or human poses) or clustering of different types of activities are a natural outcome. Our work is related to a handful of recent work on recognizing human-object interaction (HOI) activities in still images~\cite{Yao10,Desai10,Prest10,Gupta09}. While our work is synergistic, it does not assume the existence of objects in every activity (e.g. walking) and can accommodate multiple objects (e.g. serving tennis ball with a racket), as shown in Fig.\ref{fig:pull}. Furthermore, we emphasize three goals of this paper that are beyond object detection, pose estimation, and activity classification in the previous HOI work:\vspace*{-3pt}
\begin{itemize}
 \item {\em Constructing} an activity space of verbs, nouns, and human poses, and explicitly learning a semantically meaningful association between them (Sec.\ref{sec:taxonomy}).\vspace*{-5pt}
 \item {\em Measuring} distance and {\em building} taxonomies of verbs and nouns\footnote{We use {\bf nouns} and {\bf objects} interchangeably, if not explicitly stated otherwise since most nouns involved in human activities are objects.} that consider both the semantic meaning and the image features in human activities (Sec.\ref{sec:taxonomy}).\vspace*{-5pt}
 \item {\em Clustering} images in the activity space, where images of similar activities are naturally closer to each other due to the interactions of verbs, poses and objects (Sec.\ref{sec:clustering}).
\end{itemize}

\vspace*{-3pt}

The rest of the paper is organized as follows. We introduce related work in Sec.\ref{sec:related}, and elaborate on our approach for learning the verb-noun-pose associations and for constructing the activity space in Sec.\ref{sec:model}. In Sec.\ref{sec:experiment} we represent experimental results.


\begin{figure}[t]
 \begin{center}
  \subfigure[Examples of atomic poses obtained from the sports dataset~\cite{Gupta09}.]{
  \includegraphics[width=0.995\linewidth]{./figure/atomic_pose/atomic_pose.eps}
  \label{fig:pose_sports}
  }
  \subfigure[Examples of atomic poses obtained from the PASCAL dataset~\cite{PASCAL10}.]{
  \includegraphics[width=0.995\linewidth]{./figure/atomic_pose/atomic_pose2.eps}
  \label{fig:pose_pascal}
  }
  \caption{We show four atomic poses on each data set; notice the similarity of human poses in each atomic pose. This figure also shows that similar poses might represent different verbs and convey different semantic meaning. For example, the last atomic pose of (a) corresponds to three different activities (volleyball smashing, cricket bowling, and tennis serving).}\label{fig:atomic_poses}
 \end{center}
\end{figure}


\section{Related Work}\label{sec:related}

Activity recognition in still images is mostly treated as an image classification problem, without a detailed understanding of the different components in the image (see~\cite{Laptev10} for a thorough review). Recently, contextual information has been used to achieve better human activity recognition performance, such as human faces~\cite{Jie10}, scene background~\cite{Ikizler-Cinbis10}, objects, and human poses~\cite{Gupta09,Yao10}. Activity recognition is more widely studied in videos, where spatial temporal information is used for identifying classes of motions~\cite{Bobick01}. However, this is outside of the scope of this paper since our goal is to map the activity space using poses, verbs, and nouns modeled from still images.

There has been work on relating language to pictures, mostly using nouns~\cite{Duygulu06}, adjectives~\cite{Lampert09,Farhadi09} and propositions~\cite{Gupta09}. Little has been done to model verbs and to represent human activity space using verbs and nouns.

In order to effectively model verbs and nouns, our method builds upon state-of-the-art object detection~\cite{Felzenszwalb10,Desai09}, pose estimation~\cite{Sapp10}, and image classification~\cite{Lazebnik06} approaches. In particular, our model draws inspiration from the mutual context model~\cite{Yao10}. However, we focus on learning a universal representation of verbs, poses and nouns that enables the construction of an activity space and verb taxonomy, in addition to the recognition tasks in~\cite{Yao10}. Therefore unlike~\cite{Yao10}, our learning objective does not optimize for a classification task.

We propose using the atomic poses as an intermedia of the verb-noun interactions. The set of atomic poses can be thought of a dictionary of human poses. We use a semi-supervised method to obtain the atomic poses, where only annotations of human body parts are used. This is in contrast to the unsupervised approach~\cite{Wang06} and the fully supervised approach where annotations of both body parts and action classes are used~\cite{Yao10}. Our atomic poses are discovered in a way similar to poselets~\cite{Bourdev09,Bourdev10,Yang10}, which are detectors for specific body parts. In contrast, the atomic poses carry higher-level information by forming the vocabulary for the whole human body.


\section{Modeling the Activity Space by Joint Representation of Verbs, Nouns, and Poses}\label{sec:model}

To represent a human activity as a verb-noun-pose interaction, we construct a three dimensional activity space based on a set of verbs ${\mathcal V}$ (such as ``serve'' and ``ride''), a set of nouns or objects ${\mathcal O}$ (such as ``volleyball'' and ``guitar''), and a set of atomic poses ${\mathcal H}$ that form a dictionary of human body layouts (see Fig.\ref{fig:atomic_poses} for examples of atomic poses and Sec.\ref{sec:pose} for how to obtain them). The space is constructed in the way such that two images are close to each other if the activities depicted in them are semantically related (e.g. similar poses, same objects, or related verbs).

In order to construct this space, we first propose a model (Sec.\ref{sec:model_rep}) to characterize the relationships between verbs, nouns, and atomic poses. Our model learns a universal connectivity (Sec.\ref{sec:connectivity}) between them, where stronger interactions are depicted by large weights in the connection potential (e.g. ``ride'' and ``bike'' are expected to have a strong connection). We then use a probabilistic interpretation of this connectivity to build a verb taxonomy and a noun taxonomy of human activities (Sec.\ref{sec:taxonomy}). Combining the taxonomies with the set of atomic poses, we build the three dimensional verb-noun-pose space of human activities (Sec.\ref{sec:taxonomy}).

\subsection{Model Details}\label{sec:model_rep}

Given an image $I$ with the annotations of verb $V\in {\mathcal V}$, bounding boxes of nouns $O\in {\mathcal O}$ and body parts in the human pose $H\in {\mathcal H}$, our model learns the connectivity between verbs, nouns, and atomic poses that reflects the strength of the interactions between them. We further make the connectivity conditioned on the image evidence, so that concepts that are harder to recognize will play a less important role in the interaction. Our model is represented as

{\small\vspace*{-10pt}\begin{align}
& \Psi(V,O,H,I) = \underbrace{\phi_1(V,O,H)}_{\mathrm{v-n-p \hspace*{2pt} comp.}} + \underbrace{\phi_2(V,I)}_{\mathrm{verb}} \label{equ:model}\\ & \hspace*{20pt} + \underbrace{\phi_3(O,I)}_{\mathrm{noun}} + \underbrace{\phi_4(H,I)}_{\mathrm{pose}} + \underbrace{\phi_5(O,H)}_{\mathrm{spatial \hspace*{2pt} rlat.}}\notag
\end{align}}where $\phi_1$ models the compatibility between $V$, $O$, and $H$; $\phi_{2-4}$ models the image evidence using the state-of-the-art image classification~\cite{Lazebnik06}, object detection~\cite{Felzenszwalb10}, and pose estimation approaches~\cite{Andriluka09}; $\phi_5$ considers the spatial relationship between the objects and body parts. We now go through each term in Eq.\ref{equ:model}. Please refer to the supplementary document for more details.

\vspace*{2mm}

\noindent {\bf Compatibility between verb, noun, and pose.} \hspace*{1mm} The compatibility between $V$, $O$, and $H$ reflects the strength of interaction between them. $\phi_1(V,O,H)$ is parameterized as

{\small\vspace*{-10pt}\begin{align}
& \phi_1(V,O,H)\label{equ:label}\\ & = \sum\limits_{i=1}^{N_h} \sum\limits_{m=1}^M \sum\limits_{j=1}^{N_o} \sum\limits_{k=1}^{N_v} {\bf 1}_{(H=h_i)} \cdot {\bf 1}_{(O^m=o_j)} \cdot {\bf 1}_{(V=v_k)} \cdot \zeta_{i,j,k} \notag
\end{align}}where $N_h$ is the the total number of atomic poses (see Sec.\ref{sec:pose}) and $h_i$ is the $i$-th atomic pose in ${\mathcal H}$ (similarly for $N_o$, $o_j$, $N_v$, and $v_k$). $M$ is the number of object bounding boxes within the image, and $O^m$ is the object label of the $m$-th box.

\vspace*{2mm}

\noindent {\bf Modeling verbs.} \hspace*{1mm} $\phi_2(V,I)$ is parameterized by training a verb classifier based on the image region in the bounding box of the human. We have

{\small\vspace*{-10pt} \begin{align}
\phi_2(V,I) = \sum\limits_{k=1}^{N_v} {\bf 1}_{(V=v_k)} \cdot \eta_k^T \cdot s(I)
\end{align}}where $s(I)$ is the output of an SVM classifier~\cite{Lazebnik06}.

\vspace*{2mm}

\noindent {\bf Modeling nouns.} \hspace*{1mm} Inspired by~\cite{Desai09}, we model the nouns (objects) in this image using the object detection score in each object bounding box and the spatial relationships between all these boxes. Denoting the detection score of the $m$-th bounding box as $g(x^m)$, $\phi_3(O,I)$ is parameterized as

{\small\vspace*{-10pt} \begin{align}
& \phi_3(O,I) = \sum\limits_{m=1}^M \sum\limits_{j=1}^{N_o} {\bf 1}_{(O^m=o_j)}\cdot \gamma_j^T\cdot g(x^m) \\ & + \sum\limits_{m=1}^M \sum\limits_{m^{\prime}=1}^M \sum\limits_{j=1}^{N_o} \sum\limits_{j^{\prime}=1}^{N_o} {\bf 1}_{(O^m=o_j)} \cdot {\bf 1}_{(O^{m^{\prime}}=o_{j^{\prime}})} \cdot \gamma_{j,j^{\prime}}^T \cdot b(x^m,x^{m^{\prime}}) \notag
\end{align}}where $b(x^m,x^{m^{\prime}})$ is a bin function as in~\cite{Desai09} that encodes the relative geometric configurations of object bounding boxes $x^m$ and $x^{m^{\prime}}$ using a grid representation.

\begin{figure*}[!t]
\centering
   \includegraphics[width=0.82\linewidth]{./figure/links/links.eps}
   \caption{The learned connectivity map of verbs (top row), poses (middle row), and nouns (bottom row) using the sports~\cite{Gupta09} data set. Thicker lines indicate stronger connections and thinner lines indicate weaker connections. We did not show the connections between verbs and nouns to avoid clutter; we also ignore connections that are too weak.}
\label{fig:parameters}
\end{figure*}

\vspace*{2mm}

\noindent {\bf Modeling poses.} \hspace*{1mm} $\phi_4(H,I)$ models the atomic pose that $H$ belongs to and the likelihood of observing this image given the atomic pose. We have

{\small\vspace*{-10pt} \begin{align}
& \phi_4(H,I) \\ & = \sum\limits_{i=1}^{N_h} \sum\limits_{l=1}^L {\bf 1}_{(H=h_i)} \cdot \left( \alpha_{i,l}^T\cdot p({\bf x}_{H}^l|{\bf x}_{h_i}^l) + \beta_{i,l}^T\cdot f^l(I) \right) \notag
\end{align}}where $p({\bf x}_H^l|{\bf x}_{h_i}^l)$ is the Gaussian likelihood of observing ${\bf x}_H^l$, the joint of the $l$-th part in $H$, given the standard location of the $l$-th part in atomic pose $h_i$. $f^l(I)$ is the output of a detector~\cite{Felzenszwalb10} for the $l$-th body part in this image.

\vspace*{2mm}

\noindent {\bf Spatial relationships between objects and body parts.} We achieve a better modeling of objects and human body parts by considering their spatial relationships. $\phi_5(H,O)$ is parameterized as

{\small\vspace*{-10pt}\begin{align}
& \phi_5(H,O) \\ & = \sum\limits_{m=1}^M \sum\limits_{i=1}^{N_h} \sum\limits_{j=1}^{N_o} \sum\limits_{l=1}^L {\bf 1}_{(H=h_i)} \cdot {\bf 1}_{(O^m=o_j)} \cdot \lambda_{i,j,l}^T \cdot b(x^m,{\bf x}_H^l)\notag
\end{align}}where $b(x^m,x_H^l)$ denotes the spatial relationship between the $m$-th object bounding box and the $l$-th body part in $H$. We again use the bin function of~\cite{Desai09}.


\subsection{Constructing the Activity Space}

Now that we have introduced the basic model formulation, we show how verbs, nouns and poses can be put together for constructing the activity space. Our goal is to use this representation for discovering taxonomies and patterns of activities, as well as for performing recognition task related to the image contents.

\subsubsection{Obtaining the Atomic Poses}\label{sec:pose}

We discuss here in details on how to obtain the atomic poses, a set of the possible layouts of human body parts, using a clustering method. Given the training images, We first align the annotations of each image so that the torsos of all the humans have the same position and size, and constrain the range of variations of both position and angle to $[-1,1]$. If there is a missing body part due to occlusion, we fill in the annotation with the average annotation values for that particular part. We then use hierarchical clustering with the max linkage measure to obtain a set of clusters. Each cluster represents an atomic pose. Given two images $i$ and $j$, their distance is measured by

{\small\vspace*{-10pt} \begin{align}
d(i,j) = \sum\limits_{l=1}^L {\bf w}^T \cdot |{\bf x}_i^l - {\bf x}_j^l|\label{equ:pose_dist}
\end{align}}where ${\bf x}_i^l$ denotes the position and orientation of the $l$-th body part in image $i$, ${\bf w}$ is a weight vector, and $L$ is the number of body parts. The weight for location and orientation are set to $0.15$ and $0.1$ respectively to account for the fact that empirically the variation range of orientation is larger than that of location. We obtain 12 atomic poses on the sports~\cite{Gupta09} and the PASCAL dataset~\cite{PASCAL10} respectively. Examples of the obtained atomic poses are shown in Fig.\ref{fig:atomic_poses}.

\begin{figure}[b]
 \begin{center}
  \subfigure[]{
  \includegraphics[width=0.43\linewidth]{./figure/samples/sample_ours.eps}
  \label{fig:sample_ours}
  }
  \hspace*{0.04\linewidth}
  \subfigure[]{
  \includegraphics[width=0.43\linewidth]{./figure/samples/sample_uniform.eps}
  \label{fig:sample_uniform}
  }
  \caption{Triplets of verb-pose-noun generated from sampling the learned model (i.e. the connectivity map illustrated in Fig.\ref{fig:parameters}): (a) examples sampled from our learned distribution; (b) examples sampled from a uniform distribution.}\label{fig:sports_taxonomy}
 \end{center}
\end{figure}

\subsubsection{Learning and Analyzing the Connectivity}\label{sec:connectivity}

Our model (Eq.\ref{equ:label}) is a standard CRF with no hidden variables. We use a belief propagation method~\cite{Pearl88} with Gaussian priors to learn the model parameters. All object detectors and body part detectors are trained using the deformable parts model~\cite{Felzenszwalb10}, and the verb classifier is trained using the spatial pyramid method. A constant 1 is appended to each feature vector so that the model can learn biases between different classes.

While $\phi_{2-4}$ estimate the parameters for recognizing verbs, nouns, and human poses individually, $\phi_1$ learns the strength of compatibility between them. For example, given the observation of an object, we can use $\phi_1$ to predict the verb describing the action and estimate the corresponding pose. Fig.\ref{fig:parameters} visualizes the connectivity that is learned from the sports data set~\cite{Gupta09}. Each connection is obtained by marginalizing $\zeta$ in Eq.\ref{equ:label} with respect to the other concept. For instance, the strength of the connection between the verb $v_k$ and noun $o_j$ is estimated by $\sum_{i=1}^{N_h}\exp(\zeta_{i,j,k})$.

Fig.\ref{fig:parameters} shows that our method can successfully learn meaningful verb-noun-pose interactions, such as the connection between ``forehand'' and the fourth atomic pose which is a reasonable gesture for the activity of ``tennis forehand'', the noun ``volleyball'' and the last atomic pose, etc. To further analyze the quality of the connectivity, we take a probabilistic interpretation of our model and use Gibbs sampling to sample two triples of verbs, poses, and nouns. In comparison we also sample from a uniform distribution where all verbs, nouns, and poses are connected with equal weights. Fig.\ref{fig:parameters} shows that we can sample semantically meaningful triples of verbs, nouns, and poses using our learned weights but not the uniform ones.

\subsubsection{Constructing the Taxonomies and Activity Space}\label{sec:taxonomy}

Our learned verb-noun-pose connectivity offers a way to generate a {\bf verb taxonomy} of human activities. Using a probabilistic interpretation of this connectivity, we represent each verb as a joint distribution of nouns and atomic poses. We can now measure the ``distance'' between each pair of verbs, from which we build the verb taxonomy. Given a verb $v_k$, the likelihood of observing pose $h_i$ and noun $o_j$ is estimated as:

{\small\vspace*{-10pt}\begin{align}
p(H=h_i,O=o_j|V=v_k) = \frac{1}{Z(v_k)} \exp(\zeta_{i,j,k})
\end{align}}where {\small $Z(v_k) = \sum_{i=1}^{N_h}\sum_{j=1}^{N_o}\exp(\zeta_{i,j,k})$}. Then the ``distance'' between two verbs $v_k$ and $v_{k^{\prime}}$ is estimated as

{\small\vspace*{-10pt}\begin{align}
D_v(v_k,v_{k^{\prime}}) = D_{\mathrm{KL}}\left(p(H,O|v_k)\|p(H,O|v_{k^{\prime}})\right)\label{equ:kl} \\
\hspace*{22mm} + D_{\mathrm{KL}}\left(p(H,O|v_{k^{\prime}})\|p(H,O|v_k)\right) \notag
\end{align}}where $D_{\mathrm{KL}}(p\|p^{\prime})$ is the K-L divergence between $p$ and $p^{\prime}$.

\begin{figure}[t]
 \begin{center}
  \includegraphics[width=0.85\linewidth]{./figure/taxonomy/taxonomy.eps}
  \caption{Comparison of verb taxonomies obtained from: (a) our method on the sports dataset~\cite{Gupta09}; (b) only using pose distribution on the sports dataset; (c) our method on the PASCAL dataset~\cite{PASCAL10}; (d) only using pose distribution on the PASCAL dataset.}\label{fig:taxonomy}
 \end{center}
\end{figure}

With this distance measure, we use hierarchical clustering with the min linkage measure to build the verb taxonomy. The verb taxonomies that are estimated from the sports dataset~\cite{Gupta09} and the PASCAL action dataset~\cite{PASCAL10} are shown in Fig.\ref{fig:taxonomy}{\color{red}(a)} and Fig.\ref{fig:taxonomy}{\color{red}(c)} respectively. Comparing with Fig.\ref{fig:taxonomy}{\color{red}(b)} and Fig.\ref{fig:taxonomy}{\color{red}(d)}, where the taxonomies are built by only using the distribution of atomic poses in each activity, our taxonomies better captures the semantic meaning of different verbs. For example, in Fig.\ref{fig:taxonomy}{\color{red}(a)}, ``forehand'' and ``serve'' are neighbors due to their interaction with the same set of nouns (``tennis racket'' and ``tennis ball''). They are also close to ``smash'' because the human pose of ``smash'' is similar to the pose of ``serve''. Similarly, two related verbs ``walk'' and ``run'' in the PASCAL dataset whose corresponding activities do not explicitly involve any objects are placed together by our taxonomy in Fig.\ref{fig:taxonomy}{\color{red}(c)} but not by the taxonomy relying only on human poses in Fig.\ref{fig:taxonomy}{\color{red}(d)}.

In order to construct the {\bf activity space}, we also define distance measures and construct taxonomies for nouns and poses in a way that the images that are close together correspond to related nouns or have similar poses. For nouns, we adopt the same way as that in verbs by considering a distribution $p(H=h_i,V=v_k|O=o_j)$ for each object $o_j$. Considering that human poses lie in a more continuous visual feature space, we use a more direct way to obtain the pose taxonomy by continuing the clustering procedure described in Sec.\ref{sec:pose}. The distance between two atomic poses are computed as in Eq.\ref{equ:pose_dist}.

By using the taxonomies of verbs, nouns, and atomic poses, we can construct a 3D space of human activities. This space jointly considers the three components that are critical for human activity recognition. In this space, the distance measure for the verbs and the nouns considers both semantic meaning (as captured by their interaction with the other two components) and the underlying visual features (verb classification, human pose estimation, and object detection). An example of the activity space is shown in Fig.\ref{fig:sports_space}. We give more analysis and show how to apply this space in recognition tasks in Sec.\ref{sec:clustering}.


\section{Recognition in the Activity Space}\label{sec:experiment}

\subsection{Data Sets and Experiment Settings}

We carry out experiments on Gupta's sports dataset~\cite{Gupta09} and the PASCAL action dataset~\cite{PASCAL10}. Verbs and nouns in different activity classes on each data set are summarized in Table~\ref{tbl:sports_data} and Table~\ref{tbl:pascal_data}. We observe that the two data sets cover many different types of activities, such as ones where the human is interacting with multiple objects or with no object at all. Because our approach involves pose estimation, for the PASCAL dataset we only use the images where at least the upper body parts of the human are visible. Some atomic poses on the two data sets are shown in Fig.\ref{fig:atomic_poses}.

We have a pre-processing step that trains an upper-body detector~\cite{Felzenszwalb10} on each data set. The detectors work almost perfectly in our setting, because the image background of the sports data set is relatively clean and there are bounding boxes of the humans available in the PASCAL action dataset. We normalize the images based on the size of the detection boxes so that we do not need to search over scales for pose estimation. We use the pictorial structure code from~\cite{Sapp10} in our experiments for pose estimation.

\begin{table}[t]
\small{
\begin{center}
\begin{tabular}{|c|c|c|c|}
\hline {\bf Activity} & \hspace*{-10pt} & {\bf Verb} & {\bf Nouns} \\\hline
cricket batting & \hspace*{-10pt} & {\em bat} & {\em cricket bat, ball, {\rm and} stump} \\\hline
cricket bowling & \hspace*{-10pt} & {\em bowl} & {\em cricket ball {\rm and} stump} \\\hline
croquet shot & \hspace*{-10pt} & {\em shoot} & {\em croquet mallet, ball, {\rm and} hoop} \\\hline
tennis forehand & \hspace*{-10pt} & {\em forehand} & {\em tennis racket {\rm and} ball} \\\hline
tennis serve & \hspace*{-10pt} & {\em serve} & {\em tennis racket {\rm and} ball} \\\hline
volleyball smash & \hspace*{-10pt} & {\em smash} & {\em volleyball {\rm and} net} \\\hline
\end{tabular}
\end{center}
}
\caption{Verbs and nouns in each activity class of the sports data~\cite{Gupta09}. Every verb interacts with more than one object, and some objects correspond to more than one verb. For example, the objects in ``forehand'' and ``serve'' are exactly the same.}\label{tbl:sports_data}
\end{table}

\begin{table}[t]
\small{
\begin{center}
\begin{tabular}{|c|c|c|c|}
\hline {\bf Activity} & \hspace*{-10pt} & {\bf Verb} & {\bf Nouns} \\\hline
phoning & \hspace*{-10pt} & {\em phone} & {\em telephone {\rm or} cell phone} \\\hline
\multirow{2}*{playing instrument} & \hspace*{-10pt} & \multirow{2}*{\em play} & {\em violin, guitar, cello, trumpet, } \\ & \hspace*{-10pt} & & {\em flute, saxophone, {\rm or} piano} \\\hline
reading & \hspace*{-10pt} & {\em read} & {\em book {\rm or} newspaper} \\\hline
riding bike & \hspace*{-10pt} & {\em ride} & {\em bike {\rm or} motorbike} \\\hline
riding horse & \hspace*{-10pt} & {\em ride} & {\em horse} \\\hline
running & \hspace*{-10pt} & {\em run} & NULL \\\hline
taking photo & \hspace*{-10pt} & {\em take} & {\em camera} \\\hline
using computer & \hspace*{-10pt} & {\em use} & {\em keyboard, monitor, {\rm or} laptop} \\\hline
walking & \hspace*{-10pt} & {\em walk} & NULL \\\hline
\end{tabular}
\end{center}
}
\caption{Verbs and nouns in each activity class of the PASCAL action data~\cite{PASCAL10}. ``NULL'' means no noun is involved. The same verb might correspond to different activities (``riding bike'' and ``riding horse''). Also, one verb might interact with different objects in different images, even though they represent the same activity such as playing musical instrument. Some verbs do not correspond to any nouns (e.g. ``run'' and ``walk'').}\label{tbl:pascal_data}
\end{table}


\subsection{Pose Estimation, Object Detection, and Activity Classification}
\label{sec:inference}
The model in Eq.\ref{equ:model} jointly models the verbs, the nouns (objects), and the human poses in an image. We use this model for conventional recognition tasks including pose estimation, object detection, and activity classification. Inference on Eq.\ref{equ:model} gives us the outputs of pose estimation, object detection, and verb classification. Activity classification results on the sports data set can be directly obtained from verb classification, because every verb only corresponds to one activity on this dataset. On the PASCAL dataset, if the verb of an image is ``ride'', we further use horse and bike/motorbike detection results to classify this image as ``riding bike'' or ``riding horse''. Given the initial object and body part detection results, and the SVM outputs of verb classification of a new image, we iteratively perform the following three steps for inference until a local maximum of $\Psi(V,O,H,I)$ is reached. We only give a very brief outline of our inference method due to space limitations. Please refer to the supplementary document for more details.

\vspace*{-2mm}

\begin{enumerate}
  \item {\bf Updating the layout of human body parts.} \hspace*{1mm} From the current inference result, we compute the marginal distribution of the human pose $H$ over all atomic poses. We use this distribution to refine the priors of the joint locations, and then apply the Pictorial Structure (PS) method~\cite{Andriluka09} to update the layout of human body parts using the new priors. \vspace*{-2mm}
  \item {\bf Updating the nouns.} \hspace*{1mm} With the current pose estimation result and the marginal distribution over atomic poses and activity classes, we use greedy forward search~\cite{Desai09} to update the object detection results. \vspace*{-2mm}
  \item {\bf Updating the verb and atomic pose labels.} \hspace*{1mm} Based on the current pose estimation and object detection results, we optimize $\Psi(V,O,H,I)$ by enumerating all possible combinations of $H$ and $V$ labels.
\end{enumerate}

\vspace*{-2mm}

\begin{table}[b]
\small{
\begin{center}
\begin{tabular}{|c|c|c|cc|cc|cc|cc|}
\hline \multicolumn{11}{|l|}{\bf Pose Estimation} \\\hline
Method & H & T & \multicolumn{2}{|c|}{UA} & \multicolumn{2}{|c|}{LA} & \multicolumn{2}{|c|}{UL} & \multicolumn{2}{|c|}{LL} \\\hline
\hspace*{-2pt}{\footnotesize mutual context~\cite{Yao10}}\hspace*{-2pt} & \hspace*{-1pt}58\hspace*{-1pt} & \hspace*{-1pt}66\hspace*{-1pt} & \hspace*{-1pt}44\hspace*{-4pt} & \hspace*{-4pt}40\hspace*{-1pt} & \hspace*{-1pt}27\hspace*{-4pt} & \hspace*{-4pt}29\hspace*{-1pt} & \hspace*{-1pt}43\hspace*{-4pt} & \hspace*{-4pt}39\hspace*{-1pt} & \hspace*{-1pt}44\hspace*{-4pt} & \hspace*{-4pt}34\hspace*{-1pt} \\\hline PS baseline~\cite{Andriluka09} & \hspace*{-1pt}73\hspace*{-1pt} & \hspace*{-1pt}71\hspace*{-1pt} & \hspace*{-1pt}46\hspace*{-4pt} & \hspace*{-4pt}42\hspace*{-1pt} & \hspace*{-1pt}37\hspace*{-4pt} & \hspace*{-4pt}38\hspace*{-1pt} & \hspace*{-1pt}60\hspace*{-4pt} & \hspace*{-4pt}65\hspace*{-1pt} & \hspace*{-1pt}61\hspace*{-4pt} & \hspace*{-4pt}73\hspace*{-1pt}\\\hline
Our method & \hspace*{-1pt}{\bf 78}\hspace*{-1pt} & \hspace*{-1pt}{\bf 82}\hspace*{-1pt} & \hspace*{-1pt}{\bf 56}\hspace*{-4pt} & \hspace*{-4pt}{\bf 47}\hspace*{-1pt} & \hspace*{-1pt}{\bf 39}\hspace*{-4pt} & \hspace*{-4pt}{\bf 41}\hspace*{-1pt} & \hspace*{-1pt}{\bf 62}\hspace*{-4pt} & \hspace*{-4pt}{\bf 67}\hspace*{-1pt} & \hspace*{-1pt}{\bf 65}\hspace*{-4pt} & \hspace*{-4pt}{\bf 81}\hspace*{-1pt} \\\hline
\end{tabular}
\begin{tabular}{|c|c|c|c|}
\hline \multicolumn{4}{|l|}{\bf Object Detection} \\\hline \hspace*{3pt}Method\hspace*{3pt} & \hspace*{3pt}DPM~\cite{Felzenszwalb10}\hspace*{3pt} & \hspace*{2pt}Multi-objects~\cite{Desai09}\hspace*{2pt} & \hspace*{2pt}Our method\hspace*{2pt} \\\hline mAP & 41.6\% & 42.9\% & {\bf 48.3\%} \\\hline
\end{tabular}
\begin{tabular}{|c|c|c|c|c|c|}
\hline \multicolumn{6}{|l|}{\bf Activity Classification} \\\hline Method & \cite{Gupta09} & \cite{Prest10} & \cite{Yao10} & SPM~\cite{Lazebnik06} & Our method \\\hline \hspace*{1pt}Accuracy\hspace*{1pt} & 79\% & 83\% & 83\% & {\bf 100\%} & {\bf 100\%} \\\hline
\end{tabular}
\end{center}
}
\caption{Pose estimation, object detection, and activity classification results on the sports data. The body part abbreviations are: H - head, T - torso, UA - upper arms, LA - lower arms, UL - upper legs, LL - lower legs. For pose estimation, detection accuracy of each body parts are reported (omitting ``\%'' due to space limitation). For object detection, we use the mean average precision over all the object classes. And for activity classification,  average classification accuracy over all activities are reported. We bold the best performance in each experiment.}\label{tbl:sport_exp}
\end{table}

\begin{table}[t]
\small{
\begin{center}
\begin{tabular}{|c|c|c|cc|cc|cc|cc|}
\hline \multicolumn{11}{|l|}{\bf Pose Estimation} \\\hline
Method & H & T & \multicolumn{2}{|c|}{UA} & \multicolumn{2}{|c|}{LA} & \multicolumn{2}{|c|}{UL} & \multicolumn{2}{|c|}{LL} \\\hline
PS baseline~\cite{Andriluka09} & \hspace*{-1pt}49\hspace*{-1pt} & \hspace*{-1pt}55\hspace*{-1pt} & \hspace*{-1pt}37\hspace*{-4pt} & \hspace*{-4pt}35\hspace*{-1pt} & \hspace*{-1pt}27\hspace*{-4pt} & \hspace*{-4pt}30\hspace*{-1pt} & \hspace*{-1pt}44\hspace*{-4pt} & \hspace*{-4pt}42\hspace*{-1pt} & \hspace*{-1pt}41\hspace*{-4pt} & \hspace*{-4pt}44\hspace*{-1pt}\\\hline
Our method & \hspace*{-1pt}{\bf 56}\hspace*{-1pt} & \hspace*{-1pt}{\bf 60}\hspace*{-1pt} & \hspace*{-1pt}{\bf 39}\hspace*{-4pt} & \hspace*{-4pt}{\bf 41}\hspace*{-1pt} & \hspace*{-1pt}{\bf 32}\hspace*{-4pt} & \hspace*{-4pt}{\bf 32}\hspace*{-1pt} & \hspace*{-1pt}{\bf 46}\hspace*{-4pt} & \hspace*{-4pt}{\bf 50}\hspace*{-1pt} & \hspace*{-1pt}{\bf 47}\hspace*{-4pt} & \hspace*{-4pt}{\bf 46}\hspace*{-1pt} \\\hline
\end{tabular}
\begin{tabular}{|c|c|c|c|}
\hline \multicolumn{4}{|l|}{\bf Object Detection} \\\hline \hspace*{3pt}Method\hspace*{3pt} & \hspace*{3pt}DPM~\cite{Felzenszwalb10}\hspace*{3pt} & \hspace*{1pt}Multi-objects~\cite{Desai09}\hspace*{1pt} & \hspace*{1pt}Our method\hspace*{1pt} \\\hline mAP & 13.5\% & 13.7\% & {\bf 16.9\%} \\\hline
\end{tabular}
\begin{tabular}{|c|c|c|}
\hline \multicolumn{3}{|l|}{\bf Activity Classification} \\\hline \hspace*{18pt}Method\hspace*{18pt} & \hspace*{14pt}SPM~\cite{Lazebnik06}\hspace*{14pt} & \hspace*{13pt}Our method\hspace*{13pt} \\\hline Accuracy & 67.3\% & {\bf 69.2\%} \\\hline
\end{tabular}
\end{center}
}
\caption{Pose estimation, object detection, and activity classification result on the PASCAL data. Please refer to the caption of Table~\ref{tbl:sport_exp} for interpretations of the results.}\label{tbl:pascal_exp}
\end{table}

The results of pose estimation, object detection, and activity classification on the sports and PASCAL data sets are shown in Tables~\ref{tbl:sport_exp} and \ref{tbl:pascal_exp} respectively. Our method outperforms the baselines by considering the interactions between the verbs, nouns, and poses. In Table~\ref{tbl:sport_exp}, even the PS baseline of~\cite{Andriluka09} can do better pose estimation than the mutual context model, probably because the body part detectors that we use are more discriminative, and because we use an upper-body detector to avoid searching over different scales. The activity classification results in Table~\ref{tbl:sport_exp} are surprising. Both spatial pyramid matching (SPM)~\cite{Lazebnik06} and our method can achieve $100\%$ accuracy. Note that in our method, the verb classification is based on the outputs of an SPM classifier. Finally, we can see that the performance on the sports data set is generally better than that on PASCAL. This is because the resolutions of many images in PASCAL is very low, and many of the human body parts are occluded.

%Qualitative results on the sports data set are in Fig.\ref{fig:sports_space}.

\subsection{Image Clustering in the Activity Space}\label{sec:clustering}

One of our main contributions of the paper is the construction of an activity space by jointly modeling verbs, poses and nouns. Given an activity space, we can perform many new activity recognition tasks besides the conventional activity classification; for example, the distance measures defined for verbs, nouns, and poses (Sec.\ref{sec:taxonomy}) make it possible to evaluate the distance between every two images considering the semantic information of human activities. With this distance measure, we can discover meaningful activity clusters, perform activity retrieval and activity annotation, etc. In this paper, we offer preliminary results on the first application - image clustering, which has many potential applications in digital image management and image search.

Fig.\ref{fig:sports_space} shows the activity space constructed from the sports dataset. (Please refer to the supplementary document for the activity space on the PASCAL dataset.) 
%We now show an activity space constructed by our algorithm along with some experiment results, by using the approach in Sec.\ref{sec:taxonomy} and the Gupta's Sports dataset. 
To perform image clustering, we first denote the inference results of verbs, nouns, and human poses for an image as $(v,o,h)$. Given two images $(v,o,h)$ and $(v^{\prime},o^{\prime},h^{\prime})$, their distance in the activity space is measured by $D_v(v,v^{\prime}) + D_o(o,o^{\prime}) + D_h(h,h^{\prime})$, where $D_v(v,v^{\prime})$ and $D_o(o,o^{\prime})$ are the K-L divergence as in Eq.\ref{equ:kl}, $D_h(h,h^{\prime})$ is the distance between the layouts of body parts in the two atomic poses as in Eq.\ref{equ:pose_dist}. With this distance measure, an assortment of clustering algorithms can be applied. For simplicity, we choose a simple k-means clustering method. The results are shown in Fig.\ref{fig:sports_space}. We observe that the clusters obtained accurately capture the semantic meaning of human activities. In most cases images in the same cluster belong to the same activity class. Some of the activities such as cricket bowling correspond to more than one cluster. The reason is that there are large human variations in these images. But within each cluster, the images still have very similar human poses, as well as the same or related verbs and nouns.

\begin{figure*}[!t]
\centering
   \includegraphics[width=0.88\linewidth]{./figure/space/space.eps}
   \caption{The 3D activity space on the sports dataset~\cite{Gupta09} and the inference results on the testing images in this space. Each test image is indicated by a colored marker, where the colors and shapes indicate the ground truth labels. The axes correspond to verbs, nouns, and atomic poses, where we also show the corresponding taxonomies for each of these concepts. The coordinates of the test images in this space are obtained by first inferring the verb, noun, and human pose labels for each image (see Sec.\ref{sec:inference}), and then projecting the image into the 3D space. We only show the highest scoring object detection result. The distance measure in this space is defined in Sec.\ref{sec:taxonomy}. Yellow ellipses indicate the k-means clustering boundaries.}
\label{fig:sports_space}
\end{figure*}

\section{Discussion and Conclusion}

In this paper, we construct a three dimensional activity space by modeling the verb-noun-pose interactions. This project is just the first step to understanding the semantic meanings of human activities in terms of the relationships between verbs, nouns, and human poses, where there are many more interesting things to explore. Using the verb-noun-pose representations of human activities, our future work is to study the activity space on larger-scale problems where there will be more ambiguities between the three concepts, which makes the construction of verb and noun taxonomies and distance measures more challenging.


{\footnotesize
\bibliographystyle{ieee}
\bibliography{egbib}
}

\end{document}














