\subsection{\texttt{selectionrule.cpp}}
\lstinputlisting[language=C++]{../image-restoration/selectionrule.cpp}
+\subsection{\texttt{neighborhood.hpp}}
+\lstinputlisting[language=C++]{../image-restoration/neighborhood.hpp}
+
\subsection{\texttt{image.hpp}}
\lstinputlisting[language=C++]{../image-restoration/image.hpp}
\subsection{\texttt{image.cpp}}
with capacity function $c$, we call it a network, and write $G = (V, E,
c)$.
-There are two special nodes in the network, the source $s$ and the sink
-$t$. Contrary to other nodes, which can neither produce nor receive
-excess flow, the source can produce and the sink can receive an
+There are two special vertices in the network, the source $s$ and the
+sink $t$. Contrary to other vertices, which can neither produce nor
+receive excess flow, the source can produce and the sink can receive an
unlimited amount of flow. The most basic problem in flow network theory
is the question of how much flow it is possible to send through the
network from the source to the sink.
\subsection{Network construction}
\label{sec:network_construction}
+
+\begin{figure}[t]
+ \input{fig/norm_subgraph}
+\end{figure}
+
We will construct a network in such a way that if a variable
$u^\lambda_x$ ends up on the source side of the cut we set $u^\lambda_x
= 0$, and if it ends up on the sink side we set $u^\lambda_x = 1$, as in
the output image close to the original image, and the regularization
term coming from our aim to minimize the total variation.
-\subsubsection{Fidelity term}
-The fidelity term of our energy function in \eqref{eq:total_energy}
-simplifies to
-\begin{align}
- E_\lambda^x(0) &=
- N_x(\lambda + 1) -
- N_x(\lambda)
- \label{eq:fidelity_energy0} \\
- E_\lambda^x(1) &= 0
- \label{eq:fidelity_energy1}
-\end{align}
-where $E_\lambda^x(0)$ might be positive or negative depending on
-$\lambda$ and the pixel value $v_x$.
-
-\begin{figure}
-\input{fig/norm_subgraph}
-\end{figure}
-
-Figure \ref{fig:norm_subgraph} shows how networks can be constructed to
-represent this part of the total energy. The construction differs
-depending on whether $E_\lambda^x(0)$ is positive or negative. Table
-\ref{tab:fid_energy} shows how the cuts correspond to the values of
-$u^\lambda_x$ and we can easily verify that the constructed network
-actually represents the fidelity term in the energy function.
-
-\begin{table}
+\begin{table}[b]
\centering
\caption{Each row represents one of the two possible values of
$u^\lambda_x \in \{0,1\}$. The energy $E_\lambda^x(u^\lambda_x)$
\label{tab:fid_energy}
\end{table}
+\subsubsection{Fidelity term}
+
+\begin{figure}[t]
+ \input{fig/neigh_subgraph}
+\end{figure}
+
+The fidelity term of our energy function in \eqref{eq:total_energy}
+simplifies to
+\begin{align}
+ E_\lambda^x(0) &=
+ N_x(\lambda + 1) -
+ N_x(\lambda)
+ \label{eq:fidelity_energy0} \\
+ E_\lambda^x(1) &= 0
+ \label{eq:fidelity_energy1}
+\end{align}
+where $E_\lambda^x(0)$ might be positive or negative depending on
+$\lambda$ and the pixel value $v_x$.
+
+Figure \ref{fig:norm_subgraph} shows how networks can be constructed to
+represent this part of the total energy. The construction differs
+depending on whether $E_\lambda^x(0)$ is positive or negative. Table
+\ref{tab:fid_energy} shows how the cuts correspond to the values of
+$u^\lambda_x$ and we can easily verify that the constructed network
+actually represents the fidelity term in the energy function.
+
+
\subsubsection{Regularization term}
For our neighboring relation in \eqref{eq:total_energy} of the form
\begin{equation}
energies in \eqref{eq:neigh_energies} are especially simple, the
construction and presentation is simplified.
-\begin{figure}
- \input{fig/neigh_subgraph}
-\end{figure}
Figure \ref{fig:neigh_subgraph} shows two different ways of how a network
can be constructed to represent the regularization term. See Table
\ref{tab:neigh_energy} for an overview of how the two values of
$u^\lambda_x$ corresponds to cuts in the network.
-\begin{table}
+\begin{table}[t]
\centering
\caption{An overview of the possible configurations of the variables
in the term $E^{x,y}(u^\lambda_x, u^\lambda_y)$. For each
\label{tab:neigh_energy}
\end{table}
-\begin{figure}
- \input{fig/big_network}
-\end{figure}
-
Figure \ref{fig:big_network} shows a visualization of how the final
network might look with all its edges. The source will have a lot of
outgoing edges, one for each pixel, while the sink has one incoming edge
connected to the source, the sink, and their neighboring pixels, so
their edge degree is much smaller than for $s$ and $t$.
+\begin{figure}[bh]
+ \input{fig/big_network}
+\end{figure}
+
\centering
\begin{tikzpicture}[scale=1.3]
- \node[vertex] (s) at (.6, 0) {s};
- \node[vertex] (t) at (.6, -6) {t};
+ \node[vertex] (s) at (.6, -1) {s};
+ \node[vertex] (t) at (.6, -5) {t};
\begin{scope}[
yshift=-100,every node/.append style={
\section{Maximum flow approach}
-\fixme{Must be clear earlier that we are considering one level at the
-time.}
In the previous section we have seen how finding the minimum cut of
carefully constructed network can give us the thresholded image
\end{algorithm}
There are different ways to find augmenting paths, and a common choice
-is to do a breadth-first search from the source node until the sink node
-is found, as this will yield the shortest possible augmenting path. This
-version of the algorithm is called Edmonds-Karp and has a running time
-of $O(\abs{V}\abs{E}^2)$. See for example \cite{cormen2009introduction}
+is to do a breadth-first search from the source until the sink is found,
+as this will yield the shortest possible augmenting path. This version
+of the algorithm is called Edmonds-Karp and has a running time of
+$O(\abs{V}\abs{E}^2)$. See for example \cite{cormen2009introduction}
for a description of the breadth-first search, and a formal proof of the
running time of the Edmonds-Karp algorithm.
\section{Introduction}
-Ever since the invention of the camera people have been improving
-the way we record images for future use, and with the advent of
-computers, the use of images is no longer restricted to capturing a
-moment. Tasks previously reserved for the human brain, like recognizing
-textures, detecting edges and inferring shapes and motion, can now be
-performed algorithmically. The background of these methods span several
-fields, including statistics and analysis for the mathematical
-foundation, and computer science for the implementation and performance
+In everyday life cameras are used to capture a moment and save it for
+eternity, but imaging technology technology can be used in many other
+contexts, including medical and astronomical applications.
+With the advent of computers, tasks previously reserved for the human
+brain, like recognizing textures, detecting edges and inferring shapes
+and motion, can now be performed algorithmically. The background of
+these methods span several fields, including psychology and biology for
+the study of human vision, statistics and analysis for the mathematical
+foundations, and computer science for the implementation and performance
analysis.
-Imagining the things we can do to an image, it is possible to roughly
-spread it out along a line spanning from the raw capturing of light
-coming into the camera, a purely physical and engineering problem, to
-the computer vision methods in the other end, semantically interpreting
-the scene, recognizing objects, and their position and movement. In
-between we have algorithms working on the captured image, without
-implying too much about what the image contains. This includes but is
-not limited to image restoration, segmentation, filtering and edge
-detection.
-
-In the process of capturing the image with our engineered apparatus,
+It is possible to roughly spread the tasks of image processing out along
+a line spanning from the raw capturing of light coming into the camera,
+a purely physical problem, to the computer vision
+methods in the other end, semantically interpreting the scene,
+recognizing objects, their position and their movement. In between we
+have algorithms working on the captured image, without implying too much
+about what the image contains. These methods are often categorized as
+\emph{early vision} and includes but is not limited to image
+restoration, segmentation, filtering and edge detection.
+
+In the process of capturing the image with our physical apparatus,
there is always some noise included. Some might come from the physical
nature of how light travels from the scene to the objective, while some
might result from inaccuracies in the construction of the capturing
apparatus. These noise-inducing processes can be studied, and modeled
mathematically, and the process of image restoration looks at how one
can remove some -- or optimally all -- of the noise in the captured
-image. These methods often takes into account how we would expect a
+image. These methods often take into account how we would expect a
``normal'' image to look in the capturing conditions, and also what
-kinds of noise we expect to be a part of the captured image.
+kinds of noise we expect to be a part of the image.
In the next section we will see a small overview of some of the most
-popular methods in image restoration, but in the rest of the paper we
-will focus of the method of total variation. One of the strong points of
-this method is its ability to preserve edges, instead of smoothing over
-them. The problem is formulated as a minimization of an energy function.
-We will discretize this energy function, and find a global minimum by
-formulating it as a series of minimum cut problems from graph theory.
-
-The problem formulation and algorithmical solution is the main focus of
-this project, but we will also see some image restoration results in the
-end, showing how the method performs for different kinds of noise, and
-different input parameters.
+popular methods in image restoration, but in the rest of the report we
+will focus on a total variation based method. One of the strong points
+of these methods are their ability to preserve edges, instead of
+smoothing over them. Different approaches to total variation image
+restoration exist, but we will formulate it as an energy minimization
+problem.
+
+As our input images are digital images, the first step will be to
+discretize the energy function. Next comes an important part of this
+project which is to reformulate the minimization problem as a series of
+minimum cut problems from graph theory. This means we have to carefully
+construct graphs and verify that finding the minimum cuts actually
+yield a global minimizer of the original energy function.
+
+A few different algorithms for finding these minumum cuts are
+considered, but we will mainly focus on the push-relabel algorithm and
+exploit its ability to re-use some of the results across the separate
+subproblems.
+
+Towards the end we will look at some image restoration results showing
+how the method performs for different kinds of noise, and for different
+input parameters.
the vertex.
\end{description}
-\fixme{Vertex vs.\ node.}
-
As in most of the cited push-relabel literature, we define $N =
\abs{V}$, and for all vertices $u \in V$ we define the excess
\begin{equation}
\begin{algorithm}
\begin{algorithmic}
\Function{Relabel}{$u$}
-% \If{$u$ is only node at its height}
+% \If{$u$ is only vertex at its height}
% \Call{Gap}{$u$}
% \Else
\If{there is a $v \in V$ such that $(u, v) \in E_f$}
satisfied before the procedure was started.
\subsection{Putting it all together}
-These basic procedures are then applied to active nodes and admissible
+These basic procedures are then applied to active vertices and admissible
edges until we obtain our minimum cut. We will see later that when there
-are no more active nodes, we can extract the minimum cut from the
+are no more active vertices, we can extract the minimum cut from the
network.
In the first phase of the algorithm we initialize a valid preflow and
\subsection{Vertex selection rules}
Until now we have stated that the discharge procedure is run on active
-nodes until there are no more active nodes left. The choice of the order
-in which to discharge these active vertices remain, and multiple
+vertices until there are no more active vertices left. The choice of the
+order in which to discharge these active vertices remain, and multiple
possibilities exist.
The First In, First Out (FIFO) approach is to always maintain a queue of
\end{proof}
When running the push-relabel algorithm with the gap heuristic, we can
-be sure that there will never be a node $u$ with label $d(u) = N-1$ at
+be sure that there will never be a vertex $u$ with label $d(u) = N-1$ at
the end of the algorithm, i.e.\ we know that there will always be a gap
at label $N-1$. This can be seen using the same reasoning as in Theorem
\ref{thm:cut_identification}, because if there was a vertex with label
are the only ones depending on $\lambda$, the following is true for
\emph{decreasing} values of $\lambda$
\begin{description}
- \item[Edges from $s$ to $u$]
+ \item[Edges from $s$ to $u$\textmd{:}]
As seen in Figure \ref{fig:norm_subgraph_neg} the capacity of
these edges will increase monotonically with decreasing
$\lambda$ parameter.
- \item[Edges from $u$ to $v$]
+ \item[Edges from $u$ to $v$\textmd{:}]
These edges have no $\lambda$-dependence and will remain
unchanged.
- \item[Edges from $v$ to $t$]
+ \item[Edges from $v$ to $t$\textmd{:}]
As seen in Figure \ref{fig:norm_subgraph_pos} the capacity of
these edges will decrease monotonically with decreasing
$\lambda$ parameter.
two different kinds of edges, and this is done in the following way to
keep the capacity, preflow and labeling constraints satisfied.
\begin{description}
- \item[Edges from $s$ to $u$]
+ \item[Edges from $s$ to $u$\textmd{:}]
The capacity $c(s, u)$ is increased, and the flow is set to be
equal to the capacity $f(s, u) = c(s, u)$. The vertex $u$ might
have an increased excess $e(u)$, which might in turn make it
active.
- \item[Edges from $v$ to $t$]
+ \item[Edges from $v$ to $t$\textmd{:}]
The capacity $c(v, t)$ is decreased. If it is decreased to a
value below the current flow value, we set $f(v, t) = c(v, t)$
which will decrease the excess of the sink $t$, and increase the
artificially added noise, and try to remove the noise to obtain an
output image as close to the original as possible.
-The most obvious approach is to just look at the two images and see how
-much alike they are, something that makes sense especially if the output
-is made for the eye to see. One can also consider the \emph{method
-noise} which is the difference between the noisy image and the de-noised
-version. If noise is independent of the original image, the method noise
-would optimally not contain too many features from the original image,
-since it is only the noise we want to remove.
-
-Since our digitized representation of the images we are restricted to
-pixel values in $\{0, \ldots, 255\}$, we show the method noise as an
-image with pixel values $127 + \alpha (v - u)$. The scaling $\alpha$ can
-be tuned depending on the magnitude of difference $v - u$.
-
-Figure \ref{fig:noisy_lena} shows the Lena Söderberg portrait
-which has been used as a test image in digital imaging countless times.
-Gaussian and Laplace noise has been added. Since the pixel values in
-these gray scale images are limited to the range $\{0, \ldots, 255\}$,
-we have to truncate the pixel value if the addition of noise makes it
-exit the interval.
-
\begin{figure}
\centering
\begin{subfigure}[t]{0.30\textwidth}
\label{fig:gaussian_restored}
\end{figure}
+The most obvious approach is to just look at the two images and see how
+much alike they are, something that makes sense especially if the output
+is made for the eye to see. One can also consider the \emph{method
+noise} which is the difference between the noisy image and the de-noised
+version. If noise is independent of the original image, the method noise
+would optimally not contain too many features from the original image,
+since it is only the noise we want to remove.
+
+Since our digitized representation of the images we are restricted to
+pixel values in $\{0, \ldots, 255\}$, we show the method noise as an
+image with pixel values $127 + \alpha (v - u)$. The scaling $\alpha$ can
+be tuned depending on the magnitude of difference $v - u$.
+
+Figure \ref{fig:noisy_lena} shows the Lena Söderberg portrait
+which has been used as a test image in digital imaging countless times.
+Gaussian and Laplace noise has been added. Since the pixel values in
+these gray scale images are limited to the range $\{0, \ldots, 255\}$,
+we have to truncate the pixel value if the addition of noise makes it
+exit the interval.
+
\begin{figure}
\centering
\begin{subfigure}[t]{0.30\textwidth}
\centering
+\vfill
+
\LARGE{Total variation based image restoration using graph cuts} \\[2cm]
\large{Bjørn Rustad} \\[1cm]
-\today \\[2cm]
+\today \\[4cm]
%\includegraphics[width=0.5\textwidth]{lena512.png}
\centering
-\begin{tikzpicture}[scale=1.3]
+\begin{tikzpicture}[scale=1.5]
\node[vertex] (s) at (.6, 0) {s};
\node[vertex] (t) at (.6, -6) {t};
\begin{scope}[
yshift=-100,every node/.append style={
- yscale=.3,yslant=.3,xslant=-0.6},yscale=.3,yslant=.3,xslant=-0.6
+ yscale=.4,yslant=.4,xslant=-0.6},yscale=.4,yslant=.4,xslant=-0.6
]
\node[scale=.3] at (1.5,1.5) {\includegraphics{lena512.png}};
\foreach \x in {0,...,3} {
projects / prosjektoppgave / commitdiff