Text Classification using Naive Bayes Method
Naive Bayes Text Classification
Probabilistic Model of Classification
In a probabilistic classification model we want to estimate the value of \(P(cx)\) , the probability of a sample x being of class c. Naive Bayes is one such probabilistic classifier that uses Bayes’ Rule to classify samples. And Naive Bayes is “Naive” because it assumes strong independence among all the features of sample x.
Bayes Rule:
\[P(cx) = \frac{P(xc)P(c)}{P(x)}\]Text Classification using Naive Bayes classifier
Consider the task of classifying textual documents into having positive or negative sentiments. We will design the Naive Bayes classifier for this problem as follows:
Samples are text documents, and their features are the words that comprises these documents.

Each document \(d\) is a sequence of words, \(d = w_1w_2...w_n\), where \(w_i\) are the tokens of the document and \(n\) is the total number of tokens in the document \(d\).

The training dataset consists of many document, sentiment pairs, \({d_i, s_i}\)

Each document \(d_i\) is associated with a sentiment \(s_i \in \{0,1\}\), \(0\) being negative sentiment and \(1\) being positive sentiment.
Using Bayes’ Rule we have
\[p(sd) = \frac{p(ds)p(s)}{p(ds)p(s) + p(d\bar{s})p(\bar{s})}\]And from the independence assumption of features
\[p(ds) = p(w_1,w_2,..., w_ns) = p(w_1s)p(w_2s)...p(w_ns)\]Also in the IMDb reviews dataset that we are considering here have equal number of positive and negative datasets.
We have \(p(s) = 0.5\) and \(p(\bar{s})=0.5\).
This simplifies our formulation for \(p(sd)\)
\[p(sd) = \frac{p(ds)}{p(ds) + p(d\bar{s})}\]If we assign threshold of \(p_T(sd) = 0.5\) for deciding the final label, the model simplifies to,
\(y= \begin{cases} 1, & \text{if } p(ds=1) \geq p(ds=0)\\ 0, & \text{otherwise} \end{cases}\)
A measure for numerical stability
\(p(w_i)\) will be very small in magnitude, and when we take a product of such very small numbers to compute \(p(ds)\) , even double precision floating points fail to store such small numbers and becomes zero. Hence, for numerical stability, we will convert the probabilities to log probability,
\[\log p(ds) = \log p(w_1,w_2,..., w_ns) = \log p(w_1s) + \log p(w_2s) + ...+ \log p(w_ns)\]Github Repository: bsantraigi/Sentiment Analysis
# Import 'os' for preliminary tasks like directory listing etc.
import os
# Import re for regex string matching
import re
# Import nltk for word tokenization
import nltk
# Import Python's native data structures Counter and defaultdict
# Counter  maintains count of element
# defaultdict  dictionary data structure with exception handling for missing keys
from collections import Counter, defaultdict
# Import tqdm for fancy progressbars!
from tqdm import tqdm_notebook
# Import numpy for different mathematical operations on arrays / matrices
import numpy as np
# Install the nltk punkt component for tokenization
nltk.download('punkt')
Downloading the data
!wget https://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz P data/
20190714 03:04:30 https://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz
Connecting to 172.16.2.30:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 84125825 (80M) [application/xgzip]
Saving to: “data/aclImdb_v1.tar.gz”
100%[======================================>] 84,125,825 2.98M/s in 36s
20190714 03:05:08 (2.23 MB/s)  “data/aclImdb_v1.tar.gz” saved [84125825/84125825]
Extract data
Please wait for ~15s
%%time
!tar xzf data/aclImdb_v1.tar.gz C data/
CPU times: user 8.3 s, sys: 1.7 s, total: 10 s
Wall time: 5min 13s
Data Samples
 Dataset is split into two parts for training and testing
 Positive and negative samples are organized in individual folders
 Each sample document is stored in a .txt file
Let’s read in the data
data_folder = 'data/aclImdb/'
rp = os.path.join(data_folder, 'train/pos')
train_positive = [os.path.join(rp, f) for f in os.listdir(rp)]
rp = os.path.join(data_folder, 'train/neg')
train_negative = [os.path.join(rp, f) for f in os.listdir(rp)]
rp = os.path.join(data_folder, 'test/pos')
test_positive = [os.path.join(rp, f) for f in os.listdir(rp)]
rp = os.path.join(data_folder, 'test/neg')
test_negative = [os.path.join(rp, f) for f in os.listdir(rp)]
Regex for cleaning html tags
 Pattern <.*?> means “anything within two angular brackets”. The qualifier *? denotes “as few times as possible”. This makes sure we match only one html tag at a time.
re_html_cleaner = re.compile(r"<.*?>")
Limit number of samples
To quickly train a small model, consider setting n_train and n_test to some relatively small numbers e.g. 1000
. Set,
n_train = n_test = 1
to use all the samples available.
n_train = 2500
n_test = 2500
(Conditional) Unigram Counter

Calculates the distribution $$p(w s=1)\(and\)p(w s=0)$$, empirically, from training data.
# Distribution of word tokens in positive samples
positive_word_counts = Counter()
for _fname in tqdm_notebook(train_positive[:n_train], desc="Crunching +ve samples: "):
with open(_fname) as f:
text = f.read().strip()
text = re_html_cleaner.sub(" ", text)
positive_word_counts += Counter(nltk.word_tokenize(text))
# Distribution of word tokens in negative samples
negative_word_counts = Counter()
for _fname in tqdm_notebook(train_negative[:n_train], desc="Crunching ve samples: "):
with open(_fname) as f:
text = f.read().strip()
text = re_html_cleaner.sub(" ", text)
negative_word_counts += Counter(nltk.word_tokenize(text))
Unigram counts to probability distribution
\[p(ws) = \frac{N_{s,w}}{N_{s,*}} = \frac{N_{s,w}}{\sum_{w' \in W}N_{s,w'}}\]Additive Smoothing
 Note that, if some token, \(u\), unseen in training documents, occurrs in a test document, \(p(doc_{test}s)\) becomes \(0\) as \(N_{s,u}\) for that token is \(0\).
 We apply Additive Smoothing to prevent probability from going to zero.
where V is the total vocab size.
len_corpus_pos = sum(positive_word_counts.values())
len_corpus_neg = sum(negative_word_counts.values())
V_pos = len(positive_word_counts)
V_neg = len(negative_word_counts)
alpha = 0.1
log_p_vocab_pos = defaultdict(
lambda: np.log(alpha/len_corpus_pos),
{w:np.log((alpha + c)/(V_pos*alpha + len_corpus_pos)) for w,c in positive_word_counts.items()}
)
log_p_vocab_neg = defaultdict(
lambda: np.log(alpha/len_corpus_neg),
{w:np.log((alpha + c)/(V_neg*alpha + len_corpus_neg)) for w,c in negative_word_counts.items()}
)
p_data_pos = len(train_positive)/(len(train_positive) + len(train_negative))
print(f"Prob. of +ve sentiment in our dataset: {p_data_pos}")
Prob. of +ve sentiment in our dataset: 0.5
get_prob_pos(doc)
A function that accepts a document string as input, tokenizes it and computes the probability \(p(ds=1)\) and \(p(ds=0)\) . It returns 1 if \(p(ds=1) \geq p(ds=0)\) otherwise 0.
def get_prob_pos(doc):
text = doc.strip()
text = re_html_cleaner.sub(" ", text)
tokens = nltk.word_tokenize(text)
p_pos = 1
p_neg = 1
for token in tokens:
p_pos += log_p_vocab_pos[token]
p_neg += log_p_vocab_neg[token]
return 1.0*(p_pos >= p_neg) #/(p_pos+p_neg)
results = []
for _fname in tqdm_notebook(test_positive[:n_test], desc="Classifying test data: "):
with open(_fname) as f:
results.append((1, get_prob_pos(f.read())))
for _fname in tqdm_notebook(test_negative[:n_test], desc="Classifying test data: "):
with open(_fname) as f:
results.append((0, get_prob_pos(f.read())))
Performance evaluation of our model
Accuracy: Overall performance of our model, fraction of samples that were labelled correctly
Recall: Out of all +ve data samples in test set, what fraction of it was labelled correctly
Precision: How precise is the model? Out of all samples that were tagged +ve by the model, how many were actually positive.
true_pos = 0
false_pos = 0
true_neg = 0
false_neg = 0
for true_label, pred_label in results:
if true_label == 1 and pred_label == 1:
true_pos += 1
elif true_label == 1 and pred_label == 0:
false_neg += 1
elif true_label == 0 and pred_label == 1:
false_pos += 1
elif true_label == 0 and pred_label == 0:
true_neg += 1
print(f"Accuracy: {(true_pos + true_neg)/(true_pos + true_neg + false_pos + false_neg):0.4F}")
print(f"Recall: {(true_pos)/(true_pos + false_neg):0.4F}")
print(f"Precision: {(true_pos)/(true_pos + false_pos):0.4F}")
Accuracy: 0.7904
Recall: 0.7416
Precision: 0.8218