URL = {http://dx.doi.org/10.1007/978-1-4612-1015-3},\r
}\r
\r
+@PhdThesis{kolmogorovphd,\r
+ TITLE={{G}raph {B}ased {A}lgorithms for {S}cene {R}econstruction from {T}wo or {M}ore {V}iews},\r
+ AUTHOR={Kolmogorov, Vladimir},\r
+ SCHOOL={Cornell University},\r
+ MONTH={September},\r
+ YEAR={2003}\r
+}\r
+\r
@misc{github,\r
AUTHOR={Bjørn Rustad},\r
TITLE={\cpp{} implementation},\r
\chapter{Discussion and conclusion}
+\label{chap:conclusion}
In this thesis we have seen how the total variation restoration method
can be extended using an anisotropy tensor, and how this fits into the
discretization and graph cut framework used in my project work
\cite{project}. The anisotropy tensor was introduced in hopes of
-reducing the amount of regularization across edges in the image. It was
-constructed based on the structure tensor with three parameters exposed
-for controlling the method.
+reducing the amount of regularization applied across edges in the image. It was
+constructed based on the structure tensor which contains information
+about edge direction and steepness, and three parameters were exposed
+for controlling the anisotropy.
-The continuous functional we initally wanted to minimize was then
+The continuous functional we initially wanted to minimize was then
transformed using an anisotropic coarea formula and an anisotropic
Cauchy--Crofton formula, both described in detail. The functional was
discretized, and the discretization was shown---under some
As in my project work~\cite{project} the discrete functional was
minimized using maximum flow algorithms to obtain successive minimum
-graph cuts. The description of the push-relabel algorithm was included from
-my project work for completeness, and a description of the
-Boykov--Kolmogorov algorithm was also given, as it is taylored for these
-kinds of graph cut applications.
+graph cuts. A description of the push-relabel algorithm was given as it
+is known to have
+good performance for general graphs. Further the
+Boykov--Kolmogorov algorithm was described, as it is tailored for the
+kinds of graphs appearing in imaging applications.
Effort was put into giving a complete overview of the method, describing
each part needed to go from the initial continuous problem, to the
Further work is possible in the study of the continuous problem, its
well-foundedness, and also the anisotropic coarea and Cauchy--Crofton
-formulas, which can be studied on a measure-theoretic foundation.
-\fixme{tensor construction?}
+formulas, which can be studied on a measure-theoretic foundation. The
+construction of the tensor offers choices that can be explored further.
+
Regarding the discretization, the choice of neighborhood stencil also
allows for further discussion, as approximation error can be traded for
algorithm performance. Opening for non-uniform stencils, where the
stencil choice depends on the level of detail in the neighborhood is
-also a possiblity.
+a possibility.
In the end we looked at how the different parameters affect the
performance of the algorithm. We will not give any unified conclusion as
variation method it is based on, or other methods. In different
applications the input images have different properties, and one might
also have different hopes for, and restrictions on the output image. But
-we have seen that the anisotropy can help reduce the contrast loss.
+we have seen how the anisotropy tensor affects the restoration, and that
+it can have positive effects on contrast loss as well as method noise.
%Because we have spread our attention across all parts of the restoration
%method, it is also possible to go deeper into the theory behind all of
\chapter{Continuous formulation}
+\label{chap:continuous}
In the previous chapter we saw that there are many different approaches to
the image restoration problem, all with their own strengths and
regular total variation found in \eqref{eq:first_min_presentation}.
When reducing the regular total variation, we will also try to reduce
the variation over known edges in the image. This can lead to unwanted
-contrast loss, expecially in fine details.
+contrast loss, especially in fine details.
By controlling $A(x)$ such that the contribution of $\nabla u(x)$ is
reduced across known edges, we hope to retain the regularization
properties of the original method while reducing some of the negative
\begin{equation}
F(u) = \int_\Omega (u - f)^2 + \beta \, \TVA(u)
\end{equation}
-to have a \emph{well-posed} problem are lower semicontinuity
+to have a \emph{well-posed} problem are lower semi-continuity
and coercivity for existence, and convexity for uniqueness. We restrict
ourself to $L^2(\Omega)$ which makes sense with our fidelity term,
assuming that $f \in L^2(\Omega)$ initially.
\subsection{Lower semi-continuity}
-The lower semicontinuity is the most tricky part, and this is where we
-will take some shortcuts. Lower semicontinuity for a functional $F$ at a
+The lower semi-continuity is the most tricky part, and this is where we
+will take some shortcuts. Lower semi-continuity for a functional $F$ at a
point $u$ means that at points $u_\epsilon$ close to $u$, the functional
takes values either close to or above $F(u)$. More specifically, for
every sequence $u_k$ converging to $u$, we have $F(u) \leq \liminf_k
\end{equation}
and thus our functional is sequentially weakly lower semi-continuous.
-The usual ways of going from coercivity and lower semicontinuity to
+The usual ways of going from coercivity and lower semi-continuity to
existence do not work in infinite dimensions. But with sequential
coercivity and sequential lower semi-continuity in the weak topology we
can conclude that we have existence from \cite[Theorem
% \min_{u \in L^2(\Omega)} F_n(u).
% \end{equation}
% Assume that each of the functionals $F_n$ is weakly sequentially
-% lower-semicontinuous, has a unique minimizer $u_n$, and in addition that they are
+% lower semi-continuous, has a unique minimizer $u_n$, and in addition that they are
% uniformly coercive in the sense that for any sequence $v_n \in
% L^2(\Omega)$
% \begin{equation}
% Assume that the convergence in \eqref{eq:un_u_weak} does not hold,
% and denote the weakly convergent subsequence by $u_{n_j}
% \rightharpoonup \bar{u} \neq u^*$. By the weak sequential lower
-% semicontinuity we have
+% semi-continuity we have
% \begin{equation}
% \begin{aligned}
% F(\bar{u}) &\leq \liminf_{j \to \infty} F\left(u_{n_j}\right) \\
\eqref{eq:first_anisotropic_functional} through the anisotropic coarea
formula and the Cauchy--Crofton formula was motivated by the
discrete formulation which will be described here. After discretizing
-the functional, we will se how a graph cut approach can be used to find
+the functional, we will see how a graph cut approach can be used to find
a global minimizer in polynomial time.
\section{Discretization}
stop the sum at $L-2$ since the level set $\{ u > L - 1\}$ is empty and
has zero perimeter.
-\subsubsection{Discrete Riemannian Cauchy--Crofton formula}
+\subsubsection{Discrete anisotropic Cauchy--Crofton formula}
\begin{figure}
\input{fig/line_disc}
\begin{tabular}{ccccc}
\hline
$u^\lambda_x$ & $F_\lambda^x(u^\lambda_x)$ & Min.\ cut $(S, T)$
- & Graph \subref{fig:norm_subgraph_pos} cut cap. & Graph
- \subref{fig:norm_subgraph_neg} cut cap. \\
+ & Graph (\subref{fig:norm_subgraph_pos}) cut cap. & Graph
+ (\subref{fig:norm_subgraph_neg}) cut cap. \\
\hline
$0$ & $0$ & $(\{s\}, \{u^\lambda_x, t\})$ & $0$ &
$-F_\lambda^x(1)$ \\
$(u^\lambda_x, u^\lambda_y)$
& $F^{x,y}(u^\lambda_x, u^\lambda_y)$
& Min.\ cut $(S, T)$
- & Alt.\ \subref{fig:neigh_subgraph_alt1} cut cap. & Alt.\
- \subref{fig:neigh_subgraph_alt2} cut cap. \\
+ & Alt.\ (\subref{fig:neigh_subgraph_alt1}) cut cap. & Alt.\
+ (\subref{fig:neigh_subgraph_alt2}) cut cap. \\
\hline
$(0, 0)$ & $0$ & $(\{s\}, \{u^\lambda_x, u^\lambda_y, t\})$
& $w_{xy}$ & $0$ \\
},
'finger': {
- 'orig': 'finger.pgm',
- 'q': [200],
+ 'orig': 'nfinger.pgm',
+ 'q': [0],
'p': [2],
's': [3],
'r': [10, 20],
'g': [80]
},
+ #time ./sobel -p 2 -n 16 -r 8 -s 3 -b 3000 -g 1100000000000
+ 'lena_method': {
+ 'orig': 'nlena.pgm',
+ 'q': [0],
+ 'p': [2],
+ 's': [3],
+ 'r': [8],
+ 'n': [16],
+ 'b': [3000, 3191],
+ 'g': [110, 1000000000000]
+ },
+
'grad': {
'orig': 'grad.pgm',
'q': [40],
pngs.extend(rlist)
print
# Method noise
- mf = methodRule(f, nf, rf, 127, 30)
+ mf = methodRule(f, nf, rf, 127, 10)
mfiles.append(mf)
print
Image restoration methods are concerned with trying to remove noise in
images. This noise can result from the physical nature of light
-traveling to your sensor, dust on your lens, or many other sources.
+traveling to your sensor, dust on your lens, and many other sources.
Therefore
-numerous different approaches exist, each having their own
+numerous different approaches to denoising exist, each having their own
strengths and weaknesses. Before going into
detail on the main focus of this thesis---the anisotropic total
-variation method---we will look into some other popular methods.
+variation method---we will look into some other popular methods in
+Chapter~\ref{chap:methods}.
In my project work \cite{project}, I described a total variation based
image restoration method, using a graph cut framework for the numerical
-solution.
-As the name suggests, total variation is a measure of how much variation
-there is in an image. Thus these methods are often trying to
-reduce the total variation while still staying ``close'' to the original
-image, in some sense. The main strength of the total variation
+solution. The method consisted of trying to reduce the total variation,
+while still
+staying ``close'' to the original
+image. The main strength of the total variation
method is
-its ability to recover edges. The total variation does not favor smooth gradients
-over edges like many other methods. However, it may introduce edges from
+its ability to recover edges, as the total variation does not favor smooth gradients
+over edges like many other methods. However, the method may introduce edges from
noise, and it may also reduce the contrast of the image.
In this thesis we extend the total variation method by introducing an anisotropy
-tensor into the total variation norm. This makes the norm
-directionally dependant, such that we can control the regularization
-applied based on position and direction. The main idea is then to reduce
-the regularization across edges in the image.
-The continuous problem is transformed through an
+tensor into the total variation, thus making it directionally dependent.
+This means we can control the regularization
+applied to the image based on position and direction. The main idea is then to reduce
+the regularization across edges in the image. Information about the
+edges are extracted using the structure tensor.
+The continuous problem is then transformed through an
anisotropic coarea formula and an anisotropic Cauchy--Crofton formula to
-facilitate the discretization.
+facilitate the discretization. The description of the continuous
+functional and its transformation can be found in
+Chapter~\ref{chap:continuous}.
-The integrals of the continuous formulation are discretized into sums,
-and we make
-sure that the discrete formulation is consistent with the continuous
-one.
+The integrals of the continuous formulation are discretized into sums in
+Chapter~\ref{chap:discrete},
+and we make sure that the discrete formulation is consistent with the
+continuous one.
A provable optimal solution to the discrete problem is then found using
a graph cut algorithm. For each level in the image, a graph is
constructed, and a minimum cut is found. The restored image can then
For finding the minimum cuts, two algorithms are presented. The
push-relabel algorithm is considered to be the fastest and most
versatile for general graphs, while the Boykov--Kolmogorov algorithm is
-specially taylored for the type of graphs we find in these kinds of
-imaging applications.
+specially tailored for the type of graphs we find in these kinds of
+imaging applications. These are presented in Chapter~\ref{chap:maxflow}.
In the end we inspect restoration results to see how the method
performs. The construction of the tensor exposes three parameters, two
\pgfsetlayers{background,main,foreground}
\usepackage{color}
-%\usepackage[margin=0.8cm]{caption}
-\usepackage{caption}
+\usepackage[margin=0.6cm,labelfont=bf]{caption}
+%\usepackage{caption}
%\usepackage[subrefformat=parens,labelformat=parens]{subcaption}
\usepackage{subcaption}
\usepackage{xfrac}
\section*{Abstract}
-There are numerous methods for image denoising and restoration. In this
-thesis we consider a particular kind of edge-enhancing method based on
-total variation. A directional dependence is introduced through an
-anisotropy tensor based on the noisy image. Using a coarea formula and a
-Cauchy--Crofton style formula the continuous problem is transformed to obtain a discretizable
-problem. A solution is found by constructing graphs where the minimum
-cuts correspond to minimizers of the discrete problem. Parameters
-exposed allows us to control the amount of regularization applied across
-edges, and some experiments are presented to show how the anisotropy and
-its parameters affect the method.
+%There are numerous methods for image denoising and restoration.
+In this
+thesis we consider a particular kind of edge-enhancing image restoration method based on
+total variation. A directional dependence is introduced to the total
+variation method through an
+anisotropy tensor. The tensor controls the regularization applied based on
+position and direction, and is
+constructed using edge information extracted from the noisy image.
+Through a coarea formula and a
+Cauchy--Crofton style formula the continuous problem is transformed to
+obtain a discretizable minimization problem. A solution is found by constructing
+graphs where the minimum cuts correspond to minimizers of the discrete
+functional. Parameters exposed allows us to control the amount of
+regularization applied across edges, and experiments are presented
+to show how the anisotropy and its parameters affect the method.
%There are numerous methods for image denoising and restoration. In this
Mathematical Sciences for invaluable help and discussion throughout my
work with my project and this thesis.
+Thanks to Kine for helpful life coaching.
+
Bjørn Rustad, \today.
\vspace*{\fill}
\chapter{Maximum flow approach}
+\label{chap:maxflow}
In the previous section we have seen how finding the minimum cut of
carefully constructed graph can give us the thresholded image
which is derived from the original graph $G$ and contains the edges
along which it is possible to send additional flow. This means that
$E_f$ contains the edges $(u, v)$ from $E$ where $f(u,v) < c(u,v)$. But
-that is not all; an important realisation is that it is also possible to
+that is not all; an important realization is that it is also possible to
push flow \emph{back} along an edge where the flow is already positive.
-In other words, sending flow from $v$ to $u$ by cancelling some or all
+In other words, sending flow from $v$ to $u$ by canceling some or all
of the flow that is already going from $u$ to $v$.
Thus the capacity function $c_f$ of our residual graph becomes
Boykov and Kolmogorov present a variant of the augmenting path algorithm
in \cite{boykov2004experimental}, specialized for the kinds of graphs
-occuring in graphical applications. It will be presented in
+occurring in graphical applications. It will be presented in
Section~\ref{sec:bk}
\section{Push--relabel algorithm}
\subsection{Complexity}
In their original article \cite{goldberg1988new}, Goldberg and Tarjan
-analyze the complexity of the push-relabel algoritm by considering the
+analyze the complexity of the push-relabel algorithm by considering the
maximum number of basic operations we can possibly do before the
algorithm terminates.
When no more active vertices remain, there are no more possible
augmenting paths, and the algorithms terminates with a maximum flow.
-A proof of correctness can be found in \fixme{kolmogorov-phd}.
+A proof of correctness can be found in Kolmogorov's PhD thesis
+\cite{kolmogorovphd}.
\subsection{Graph reuse}
\chapter{Methods in image restoration}
+\label{chap:methods}
\fixme{Rating: 7/10}
lessen the regularization applied across what we know, or at least are
pretty sure to be edges in the image. Before considering the anisotropy
tensor construction described earlier, we will look at how a simple
-predescribed tensor affects the regularization. Imagine a tensor which
+pre-described tensor affects the regularization. Imagine a tensor which
is a diagonal matrix $A = \operatorname{diag}(1, \epsilon)$, where
$\epsilon \ll 1$ is small. This would result in us down-weighting the
size of $\nabla u$ in the $y$-direction, and thus regularization
size of the structures we want to detect with our anisotropy tensor. In
Figure~\ref{fig:scale_comp_int} a high value for $\rho$ means that the
structure found in the inner square is almost completely ignored in
-favour of the larger, more coherent structure around it.
+favor of the larger, more coherent structure around it.
\begin{figure}
\centering
circumference of circles of different radii. Note that the circumference
approximated is that of an actual continuous circle $u : \mathbb{R}^2
\to \{0, 1\}$ and not a discrete representation. We see that the
-perimeter of the smalles circle is grossly underestimated by the larger
+perimeter of the smallest circle is grossly underestimated by the larger
neighborhoods. Thus the perimeter of one-pixel noise will be
underestimated, and in turn the contribution to the total variation by
one-pixel noise will be smaller for large neighborhoods.
respectively.
To further compare the three restored images, a single row has been
-extracted from the three images and is shown agains eachother in
+extracted from the three images and is shown against each other in
Figure~\ref{fig:contrast_plot}. Note that although the results in
Figure~\ref{fig:finger_contrast_atv2} look promising compared to
Figure~\ref{fig:finger_contrast_tv}, some details in the singularity in
can in that case be if the method noise contains a lot of the details of
the image.
-\fixme{fuck}
-
-%Figure~\ref{fig:lena_method_noise} shows the results after restoring the
-%noisy Lena picture in Figure \fixme{noisylena}, with different
-%parameters, and their method noise. We note that the method noise of the
-%regular total variation method shows and outline of Lena, which
-%indicates some detail loss. Further, we see that the method noise of the
-%anisotropically restored image, has less details.
-
-%\begin{figure}
-% \centering
-% \begin{subfigure}[t]{0.30\textwidth}
-% \centering
-% \includegraphics[width=\textwidth]{fig/lena_tv.png}
-% \end{subfigure}
-% ~
-% \begin{subfigure}[t]{0.30\textwidth}
-% \centering
-% \includegraphics[width=\textwidth]{fig/lena_atv.png}
-% \end{subfigure}
-% ~
-% \begin{subfigure}[t]{0.30\textwidth}
-% \centering
-% \includegraphics[width=\textwidth]{fig/lena_atv72.png}
-% \end{subfigure}
-%
-% \centering
-% \begin{subfigure}[t]{0.30\textwidth}
-% \centering
-% \includegraphics[width=\textwidth]{fig/lena_method_tv.png}
-% \caption{%
-% Regular total variation.
-% }
-% \end{subfigure}
-% ~
-% \begin{subfigure}[t]{0.30\textwidth}
-% \centering
-% \includegraphics[width=\textwidth]{fig/lena_method_atv.png}
-% \caption{%
-% Anisotropic total variation.
-% }
-% \end{subfigure}
-% ~
-% \begin{subfigure}[t]{0.30\textwidth}
-% \centering
-% \includegraphics[width=\textwidth]{fig/lena_method_atv72.png}
-% \caption{%
-% Size 72 neighborhood.
-% }
-% \end{subfigure}
-% \caption{%
-% Funny caption. Parameters.
-% }
-% \label{fig:lena_method_noise}
-%\end{figure}
+Figure~\ref{fig:lena_method_noise} shows the results after restoring a
+noisy Lena picture with different parameters, and their method noise. We
+see that the method noise of Figure~\ref{fig:lena_method_noise_tv} shows
+slightly more details than the anisotropic counterpart in
+Figure~\ref{fig:lena_method_noise_atv2}.
+
+\begin{figure}
+ \centering{}
+ \begin{subfigure}[t]{0.30\textwidth}
+ \centering
+ \includegraphics[trim=200 200 0 0,clip=true,width=\textwidth]{fig/factory/lena_method/r_p2_n16_b3000_g1000000000000_r8_s3.png}
+ \end{subfigure}
+ ~
+ \begin{subfigure}[t]{0.30\textwidth}
+ \centering
+ \includegraphics[trim=200 200 0 0,clip=true,width=\textwidth]{fig/factory/lena_method/r_p2_n16_b3000_g110_r8_s3.png}
+ \end{subfigure}
+ ~
+ \begin{subfigure}[t]{0.30\textwidth}
+ \centering
+ \includegraphics[trim=200 200 0 0,clip=true,width=\textwidth]{fig/factory/lena_method/r_p2_n16_b3191_g110_r8_s3.png}
+ \end{subfigure}
+
+ \begin{subfigure}[t]{0.30\textwidth}
+ \centering
+ \includegraphics[trim=200 200 0 0,clip=true,width=\textwidth]{fig/factory/lena_method/m_r_p2_n16_b3000_g1000000000000_r8_s3.png}
+ \caption{Regular total variation, $\beta = 3000$, $\norm{u -
+ f}_{L^2} = 9771$}
+ \label{fig:lena_method_noise_tv}
+ \end{subfigure}
+ ~
+ \begin{subfigure}[t]{0.30\textwidth}
+ \centering
+ \includegraphics[trim=200 200 0 0,clip=true,width=\textwidth]{fig/factory/lena_method/m_r_p2_n16_b3000_g110_r8_s3.png}
+ \caption{Anisotropic TV, $\beta = 3000$, $\omega = 110$,
+ $\norm{u-f}_{L^2} = 9556$}
+ \label{fig:lena_method_noise_atv1}
+ \end{subfigure}
+ ~
+ \begin{subfigure}[t]{0.30\textwidth}
+ \centering
+ \includegraphics[trim=200 200 0 0,clip=true,width=\textwidth]{fig/factory/lena_method/m_r_p2_n16_b3191_g110_r8_s3.png}
+ \caption{Anisotropic TV, $\beta = 3191$, $\omega = 110$,
+ $\norm{u-f}_{L^2} = 9771$}
+ \label{fig:lena_method_noise_atv2}
+ \end{subfigure}
+ \caption[Lena restored, shown with method noise]{%
+ Lena with additive Gaussian noise ($\sigma = 30$) shown restored with
+ different parameters, with corresponding method noise below. A
+ slight improvement in the amount of detail in the method noise
+ is seen from (\subref{fig:lena_method_noise_tv}) to
+ (\subref{fig:lena_method_noise_atv2}).
+ }
+ \label{fig:lena_method_noise}
+\end{figure}
projects / master / commitdiff