\documentclass{beamer}
\begin{document}

\title{Implementing Deep Learning \\ for Sentiment Analysis}
\author{Johann Mitl\"ohner}
\institute{Vienna University of Economics and Business}
\date{Mai 4, 2018}
\frame{\titlepage}

\begin{frame}
\frametitle{Sentiment Analysis}

Applying Natural Language Processing and Text Analysis techniques to identify and extract subjective information from a piece of text

\begin{itemize}
\item product reviews
\item social media
\end{itemize}

Business Applications:

\begin{itemize}
\item Listen to the voice of the customer
\item What features do they like and dislike about products
\item Determine marketing strategy
\item Identify trends
\item Respond in a timely manner
\end{itemize}

Other Applications, e.g. politics: monitor sentiment on election candidates

\end{frame}


\begin{frame}
\frametitle{Types Of Sentiment Analysis}

\begin{itemize}
\item Manual processing
\begin{itemize}
\item most accurate judge of sentiment 
\item still not 100\% accurate
\item increasingly infeasible due to prolific growth of social media data
\end{itemize}

\item{Keyword processing}
\begin{itemize}
\item assign a degree of positivity or negativity to individual words
 \\ positive: great, like, love or negative: terrible, dislike
\item overall percentage score 
\item very fast and cheap to implement
\item problems with multiple meanings and context
\end{itemize}

\item{Natural Language Processing}
\begin{itemize}
\item aims at detecting meaning and context
\item Many approaches, in the following:
\begin{itemize}
\item word embeddings
\item neural nets
\item Simple architecture i.e. Feed-forward
\end{itemize}
\end{itemize}

\end{itemize}

\end{frame}



\begin{frame}
\frametitle{SemEval-2017 Task 4}

Sentiment Analysis in Twitter

Subtask A Message Polarity Classification: positive, negative, or neutral sentiment

\begin{itemize}
\item 50.000 tweets 
\item CrowdFlower used to annotate tweets
  \begin{itemize}
  \item Each tweet annotated by at least five people  
  \item hidden  tests  for  quality  control
  \item Label accepted if three out of five annotators agreed
  \item Otherwise average   mapped to  closest  integer  value
  \end{itemize}
\item Results are available for comparison
\end{itemize}

\end{frame}


\begin{frame}
\frametitle{Sample Data}

100002521464053761      positive        

  Going to see Rise of the Planet of the Apes tonight...oh yeah.

100002606734245888      negative        

  Uh drunk and tired and just remembered I'm going to Ireland in the morning and need to pack. Ireland :(

100002627009515520      neutral 

  @Mariners Guti's catch at the wall in May against the Yankees at Safeco and Trayvon's catch are the leaders in the clubhouse.

100001422170529792      positive        

  @jordanknight Hope u have a great show tonight. Me \& the Chi/IN girls are so excited to see you in London tomorrow!

256869888373714944      negative        

  Just noticed that the music for Bittersweet Symphony is credited to Mick Jagger and Keith Richards. Find out why and you may dislike them.

\end{frame}


\begin{frame}
\frametitle{Feature Extraction}

\begin{itemize}
\item text is messy  
\item machine learning algorithms need well-defined inputs 
  \begin{itemize}  
  \item convert text to numbers 
  \item add more features, e.g. part of speech  
  \end{itemize}
\end{itemize}

Many approaches, e.g.

\begin{itemize}
\item Bag of Words
  \begin{itemize}
  \item One-Hot
  \item TF-IDF
  \end{itemize}
\item Word Embeddings
  \begin{itemize}
  \item Word2Vec
  \item GloVe
  \item pre-trained or train your own (needs very large corpus)
  \end{itemize}
\end{itemize}

\end{frame}


\begin{frame}
\frametitle{Bag of Words: One-Hot}

\begin{itemize}
\item define set of words, e.g. most frequent 
\item 1 if word present in tweet, 0 if not
% \item variation: number of occurances
\item fixed-size input vector for each tweet
\end{itemize}

100001422170529792      positive        

  @jordanknight Hope u have a great show tonight. Me \& the Chi/IN girls are so excited to see you in London tomorrow!

256869888373714944      negative        

  Just noticed that the music for Bittersweet Symphony is credited to Mick Jagger and Keith Richards. Find out why and you may dislike them.

\begin{tabular}{rrrr}
great&awesome&terrible&dislike \\
1    &0      &0       &0\\
0    &0      &0       &1\\
\end{tabular}

\begin{itemize}
\item Simple method
\item word order lost
\item still, often acceptable results
\end{itemize}

\end{frame}


\begin{frame}
\frametitle{Bag of Words: TF-IDF}

Problems with One-Hot:

\begin{itemize}
\item frequent words carry little information
\item stopwords are not flexible to the application domain
\end{itemize}

TF(t) = (Number of times term t appears in document) / (Total number of terms in document)

IDF(t) = log(Total number of documents / Number of documents with term t in it)

TFIDF(t) = TF(t) * IDF(t)

Still problems common to all Bag of Words methods:

\begin{itemize}
\item vocabulary choice is difficult
\item sparse encoding makes computation harder
\item word order ignored
\item context ignored, meaning often lost
\end{itemize}


\end{frame}


\begin{frame}
\frametitle{Word Embeddings}

Words mapped to vectors of real numbers

E.g. Glove Global Vectors for Word Representation

\begin{itemize}
\item GloVe model trains on global co-occurrence counts of words
\item preserves word similarities with vector distance
\end{itemize}

Various pre-trained word vectors available, e.g.

\begin{itemize}
\item Wikipedia + GigaWord
\item Common Crawl
\item Twitter 2B tweets, 27B tokens, 1.2M vocab, uncased, 25d, 50d, 100d, \& 200d vectors, 1.42 GB 
\end{itemize}

great   0.10751  0.15958  0.13332 0.16642 -0.032737 0.17592 ... \\
dislike 0.049184 0.19144 -0.40721 0.58064 -0.33959  0.75671 ...

phrase encoding: element-wise sum of word encodings, optionally divide by number of words

\end{frame}


\begin{frame}
\frametitle{Single-layer Feed-forward Network}

\begin{itemize}
\item input vector $\bf{x}$ 
\item correct response $\bf{y}$
\item weight matrix $\bf{W}$
\item bias $\bf{b}$
\item non-linear activation function, e.g. tanh
\item network output $\bf{o}$
\end{itemize}

$$ \bf{o} = \tanh(\bf{W} \bf{x} - \bf{b}) $$

Optimise weights to minimise error $\bf{y} - \bf{o}$ 

\begin{itemize}
\item various optimisers
\item learning rule, e.g. gradient descent
\end{itemize}

\end{frame}


\begin{frame}
\frametitle{Activation Functions}

\begin{itemize}

\item Sigmoid $\sigma(x) = 1 / (1 + e^{-x})$ 
fallen out of favour because
\begin{itemize}
\item vanishing gradient at tails, no weight update
\item output not zero-centered, problems with gradient descent
\end{itemize}

\item tanh; note that $\tanh(x) = 2 \sigma(2x) -1$
\begin{itemize}
\item also suffers from vanishing gradient, but
\item output is zero-centered, therefore
\item in practise preferred to sigmoid
\end{itemize}

\item ReLU rectified linear unit $f(x) = \max(0, x)$
\begin{itemize}
\item accelerates convergence of stochastic gradient descent compared to tanh (as much as factor 6)
\item cheap computation
\item can be fragile and never activate again
\item need careful choice of learning rate
\end{itemize}

\item Softmax

\end{itemize}

\end{frame}


\begin{frame}
\frametitle{Linear Softmax Classifier}

$\bf{y}$ correct class e.g. $(1, 0, 0, ...)$

$\bf{f} = \bf{W} \bf{x} $ scores for each class

$\bf{p}$ vector of probabilities with 

$$ p_k = \frac{e^{f_k}}{\sum_j e^{f_j}} $$

Cross-entropy loss function:

$$ L = - \bf{y} \cdot \log (\bf{p}) $$

{\small E.g. $L = -1 \times (1 \times log(0.3) + 0 \times log(0.2) + 0 \times log(0.2) + ...)$ \\
When probability for correct class goes towards 1 loss goes towards 0 since $log(1) = 0$}

$$ \frac{\delta L_i}{\delta f_k} = p_k - 1(y_i = k) $$

where $1(y_i = k)$ is 1 if $y_i = k$ and 0 otherwise. 

\end{frame}


\begin{frame}
\frametitle{Two-Layer Network}

\begin{itemize}
\item
Hidden layer $\bf{h}$, activation function tanh or ReLU
\item
Output layer class probabilities $\bf{p}$, activation function softmax 
\item Weight matrices $\bf{W}, \bf{V}$, bias $\bf{b}$
\end{itemize}

$$ \bf{h} = \tanh(\bf{W} \bf{x} - \bf{b}) $$

$$ \bf{f} = \bf{V} \bf{h} $$

$$ p_k = \frac{e^{f_k}}{\sum_j e^{f_j}} $$

\begin{itemize}
\item optimise weights and biases, e.g. gradient descent
\item
more units in hidden layer: better on {\it training data}
\item 
not generalising to unseen data, i.e. validation/test data
\end{itemize}

\end{frame}


\begin{frame}
\frametitle{Training, Validation, Test}

Three data sets:
\begin{itemize}
\item training data: optimiser works on this data to find weights (and biases)
\item validation data: 
  \begin{itemize}
  \item network responses for data not seen during training
  \item serve as measure of success for parameter tuning such as number of hidden units
  \end{itemize}
\item test data:
  \begin{itemize}
  \item measure computed only when parameter tuning is finished
  \item this result is reported
  \end{itemize}
\end{itemize}

Reasoning: performance on validation data is also result of optimisation (of parameter
and architecture choices by the developer) and therefore specific to validation data

Usual splits: 60/20/20 or 80/10/10

\end{frame}


\begin{frame}
\frametitle{Deep Learning}

Connectionism popular in early 90s, but:

\begin{itemize}
\item limited computer power
\item large training data sets not available
\item deep networks with more than two layers not feasible
\end{itemize}

Deep Learning brings renaissance to neural networks:

\begin{itemize}
\item huge data sets available, e.g. social media
\item computer power and memory increased dramatically, esp. GPU
\item advances in algorithms, e.g. stochastic gradient descent, ReLU
\item spectacular successes in image recognition
\end{itemize}

\end{frame}


\begin{frame}
\frametitle{Stochastic Gradient Descent}

Weight update uses gradient, e.g. for softmax

$$ \frac{\delta L_i}{\delta f_k} = p_k - 1(y_i = k) $$

where $1(y_i = k)$ is 1 if $y_i = k$ and 0 otherwise. 

\begin{itemize}

\item
Classic gradient descent: 
\begin{itemize}
\item whole data set used
\item expensive computation
\end{itemize}

\item
Stochastic gradient descrent: 
\begin{itemize}
\item choose small number of random data points (minibatch)
\item usually sufficient to calculate gradient and converge
\item {\bf much} faster computation
\end{itemize}

\end{itemize}

\end{frame}


\begin{frame}
\frametitle{GPU Computing}

\begin{itemize}
\item NVIDIA graphics card with CUDA support
\item speedup 5x, 10x, .. depending on HW and application
\item high-level framework like Keras: no change of code
\item data has to fit in card memory
\end{itemize}

PC Hardware:
\begin{itemize}
\item RAM rule of thumb: at least twice the GPU memory
\item Power supply 
  \begin{itemize}
  \item high-end graphics cards need several power connectors
  \item check if mainboard supplies enough connectors and total power
  \end{itemize}
\item space! high-end cards are BIG
  \begin{itemize}
  \item does the card fit in the case? check dimensions
  \item no components on mainboard in the way?
  \end{itemize}
\end{itemize}

Multiple card feasible with e.g. Tensorflow and Keras

\end{frame}


\begin{frame}
\frametitle{Choices for Graphics Card}

NVIDIA or AMD?
\begin{itemize}
\item CUDA good support with DL frameworks
\item AMD uses OpenCL, currently (2018) much less support
\end{itemize}

$\Rightarrow$ NVIDIA

Popular models with good value:

{\small
\begin{tabular}{lrrrr}
Model       & VRAM GB & Bandw. GB/sec & Cores & Price \\ \hline 
GTX 1080 Ti &      11 &              484 &  3584 & 760 \\
\multicolumn{5}{l}{High-end, best choice} \\
GTX 1070 Ti &       8 &              256 &  2432 & 500 \\
\multicolumn{5}{l}{Can still compete on Kaggle} \\
GTX 1060    &       6 &              216 &  1280 & 300 \\
\multicolumn{5}{l}{Good enough for many NLP tasks} \\
GTX 1050 Ti &       4 &              112 &   768 & 170 \\
\multicolumn{5}{l}{Budget model to get into DL} \\
\end{tabular}
}

\end{frame}


\begin{frame}
\frametitle{GPU Computing: Software}

Software for Linux: Python + DL framework

\begin{itemize}
\item optimized linear algebra library, e.g. OpenBLAS
\item Numpy 
  \begin{itemize}
  \item Python library for fast and convenient array computing
  \item if linear algebra library installed:
  \item makes use of multi-core with no change of code
  \item speed up close to number of cores
  \end{itemize}
\item Keras: Python framework for Deep Learning
\item Backend for Keras:
  \begin{itemize}
  \item Theano
  \item TensorFlow
  \end{itemize}
\item CUDA: Nvidia library for GPU computing
\end{itemize}

\end{frame}


\begin{frame}
\frametitle{Pure Python vs Framework}

\begin{itemize}
\item Programming neural networks in pure Python is possible
\item Simple two-layer feed-forward net is just a few lines
\item More complicated architectures:
  \begin{itemize}
  \item gradient computation complex
  \item pure Python infeasible
  \end{itemize}
\item Theano
  \begin{itemize}
  \item can compute gradients automatically
  \item solid choice for experiments in new architectures
  \item still too complex for application-oriented projects
  \end{itemize}
\item Keras
  \begin{itemize}
  \item high-level neural networks API 
  \item written in Python 
  \item can run on top of Theano or TensorFlow
  \item provides standard components for deep learning networks
  \item convenient support for GPU computing
  \end{itemize}
\end{itemize}

\end{frame}


\begin{frame}[fragile]
\frametitle{Keras Layers}

\begin{itemize}
\item Dense -- fully connected neural net layer
\item Dropout -- randomly set fraction of input to 0 to avoid overfitting
\item Convolution -- repeatedly applied same computation to parts of input
\item MaxPooling -- propagate only maximum of input regions
\item Recurrent layers -- RNN, GRU, LSTM, ..
\end{itemize}

{\small
\begin{verbatim}
model = Sequential()
model.add(Dense(400, activation='relu', input_shape=(200,)))
model.add(Dropout(0.2))
model.add(Dense(100, activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(num_classes, activation='softmax'))
\end{verbatim}
}

\end{frame}


\begin{frame}[fragile]
\frametitle{Keras Optimizers}

Adapt weights and biases after each minibatch to decrease loss 

\begin{itemize}
\item SGD -- stochastic gradient descent
\item RMSprop -- adapt step size to gradient
\end{itemize}

{\small
\begin{verbatim}
model.compile(loss='categorical_crossentropy',
              optimizer=RMSprop(),
              metrics=['accuracy'])
history = model.fit(x_train, y_train,
                    batch_size=batch_size,
                    epochs=epochs,
                    verbose=1,
                    validation_data=(x_val, y_val))
score = model.evaluate(x_val, y_val, verbose=0)
\end{verbatim}
}

\end{frame}


\begin{frame}
\frametitle{GPU Configuration and Timings}

\begin{itemize}
\item Linux config file \$HOME/.keras/keras.json

    "backend": "tensorflow"

Backend choices: theano, tensorflow

\item Hardware:
\begin{itemize}
\item CPU: i7-6700 Quad-Core, 8 threads \\
\item GPU: GTX 1060 6GB
\end{itemize}

\item Timings: {\small 200 dim embeddings, two hidden ReLU layers with h units each, dropout 0.2, output Softmax, minibatch 128, 30 epochs}

\begin{tabular}{rrrr}
h & CPU & CUDA & Speedup  \\ \hline
 200 &  15.3 & 12.4 & 1.2 \\
 400 &  27.0 & 12.9 & 2.0 \\
 800 &  75.4 & 16.8 & 4.4 \\
1600 & 212.8 & 35.4 & 6.0 \\
\end{tabular}

\item Switch off GPU on command line e.g. 

{\small CUDA\_VISIBLE\_DEVICES=""  python kera.py tweets.txt 1600}
 
\end{itemize}

{\small continue in sentiment.html}
\href{http://balrog.wu.ac.at/~mitloehn/graz/sentiment.html}{\beamergotobutton{Link}}

\end{frame}

\end{document}
