Hw4 106061212 賴傳堯¶
Read Data¶
In [1]:
import pandas as pd
df = pd.read_csv('Google_AI_published_research.csv', encoding='utf-8')
df.head()
Out[1]:
Read¶
- 用pd讀進資料，將頭5項印出來方便看內容、紀錄處理階段
In [2]:
import re
In [3]:
for i in range(len(df['title_abstract'])):
df['title_abstract'][i] =\
re.sub(re.compile('<.*?>|&([a-z0-9]+|#[0-9]{1,6}|#x[0-9a-f]{1,6});'), '', df['title_abstract'][i])
Cleaning Html Syntax¶
- 將讀進的原始檔進行一次noise cleaning，清除檔案中帶有的html syntax
- 方法: 用for迴圈迭代整份資料的每一筆文章，一次對一筆文章用re.sub()找到html的內容用框字串取代
- 目的: 獨立處理原始資料，方便之後LDA印出、觀察清理過的文章
Data Cleaning & Tokenizing¶
In [4]:
from nltk.stem.porter import PorterStemmer
from nltk.corpus import stopwords
In [5]:
stop = stopwords.words('english')
def cleaner_tokenizer(text, stopword=stop):
text = re.sub('[\W]+', ' ', text.lower())
text = re.sub(r'[0123456789]*', '', text)
text = text.split()
text = [w for w in text if w not in stopword]
return text
Cleaning¶
- 用re.sub()將數字以及標點符號清掉，只留下文字
- 因為希望只留下能有代表意義的部分，所以文字以外都在這一步清掉
- 有些compound word會用到dash符號，原本希望能保留原本的複合字(比較能完整代表其意義)，但因觀察到會產生只有dash符號的token(可能來自運算式的減符號，或是例如1998-2019這種的時間表示，數字刪掉之後剩下的)，所以還是決定將它清掉
- 用從nltk.corpus引入的stopword來清除一些很常出現但無法代表文章意義的詞
Tokenizing¶
- 在做cleaning時順便將文章轉為小寫(text.lower())
- 做stopword時需要逐字檢查，所以做split()處理
- 結果如下，每筆文章轉為單字token組成的list
In [6]:
X = pd.DataFrame(df['title_abstract'].apply(cleaner_tokenizer))
X.head()
Out[6]:
Apply Method¶
- 用apply()將df['title_abstract']套用定義好的method，最後的結果存到一個新的變數X，與原始資料區隔
Stemming & Lemmatizing¶
In [7]:
from nltk.corpus import wordnet
In [8]:
def get_wordnet_pos(word):
tag = nltk.pos_tag([word])[0][1][0].upper()
tag_dict = {"J": wordnet.ADJ,
"N": wordnet.NOUN,
"V": wordnet.VERB,
"R": wordnet.ADV}
return tag_dict.get(tag, wordnet.NOUN)
In [9]:
import nltk
from nltk.stem import WordNetLemmatizer
In [10]:
lemmatizer = WordNetLemmatizer()
X_lem = []
for row in X['title_abstract']:
X_lem.append(' '.join([lemmatizer.lemmatize(w, get_wordnet_pos(w)) for w in row]))
X_lem = pd.DataFrame(X_lem, columns=['Text_Lemmatized'])
X_lem.head()
Out[10]:
Lemmatizing¶
- Lemmatization會按照文法邏輯將文字正確轉回原型，例如可以將better轉回good，但缺點是例如evaluations只會轉回evaluation，而不是evaluate，這會使得兩個意思相近或一樣的字，對機器而言是完全不同的兩個字
- 這裡用WordNetLemmatizer()來轉換，一樣用for迴圈一次處理一筆資料，並將轉換後的結果存到X_lem；因為WordNetLemmatizer()需要提供詞性才能正確轉換，所以透過get_wordnet_pos來取得字的詞性，並轉換為WordNetLemmatizer()能接受的格式
- get_wordnet_pos: nltk.pos_tag()取得詞性，建立取得的詞性與WordNetLemmatizer()能接受的詞性格式對應的字典，回傳WordNetLemmatizer()能接受的詞性格式，default會回傳名詞(wordnet.NOUN)
In [11]:
porter = PorterStemmer()
X_stem = []
for row in X['title_abstract']:
X_stem.append(' '.join([porter.stem(word) for word in row]))
X_stem = pd.DataFrame(X_stem, columns=['Text_Stemmed'])
X_stem.head()
Out[11]:
Stemming¶
- stem的功用是將文字轉為詞幹，有些時候這些詞幹並不會是一個存在的字，例如evaluating被轉為evaluate，有時候則不會有正確的轉換，例如this被轉為thi
- 原本按照講義中的順序，先做stemming再清除stopword，就會得到thi這個token，甚至會出現再topic的top words之中，明顯對結果是不好的影響，所以改為先清除stopword，才做stemming
- 這裡用PorterStemer()來轉換，用for迴圈一次處理一筆資料，並將轉換後的結果存到X_stem
Comparison¶
- 因為stemming和lemmatization各有優缺，所以兩種分別都做，會在之後的方法中稍微比較兩者哪個較好
LDA¶
In [12]:
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation
Count Vectorize Without Setting max_df¶
In [13]:
count = CountVectorizer(stop_words='english',
max_features=10000)
lda = LatentDirichletAllocation(n_components=10, random_state=6, learning_method='batch')
X_lda = count.fit_transform(df['title_abstract'].values)
X_topics = lda.fit_transform(X_lda)
n_top_words = 5
top_words = []
feature_names = count.get_feature_names()
for idx, topic in enumerate(lda.components_):
print("Topic %d:" % (idx + 1))
top_words.append([feature_names[i] for i in topic.argsort()[:-n_top_words-1:-1]])
print(" ".join(top_words[idx]))
Comment on Obtained Top Words & Possible Topics¶
- Top Words: 這裡用的是沒經過Stemming/Lemmatization處理的文章，所以會看到同一個主題同時出現同一個字的單複數或第一、第三人稱用法，好處是可以很清楚觀察是那些字，壞處是反而會占用或擋掉其他也代表這個topic的字
- Possible Topics:
- Topic 1: optimizing algorithm
- Topic 2: image processing
- Topic 3、4: NLP
- Topic 5: ploting graph of large dataset
- Topic 6: speech recognizing
- Topic 7: vedio processing
- Topic 8: study of google research data
- Topic 9: website systems
- Topic 10: media vs privacy
- Trivial Word Problem:
- 可以看到像是data、 model、use等等的字不斷重複出現
- 主要原因是LDA假設每個字是從多項式分布中選取產生的，使用Tfidf取倒數之後產生的weight並不合哩，因此只適合用CountVectorizer，連帶就會有其缺點:無法過濾頻繁出現，卻不具代表性的字
- 這個現象再加上使用沒經過Stemming/Lemmatization處理的文章，使得會有model、models同時出現的現象，導致更難看出主題。
In [14]:
text_idx = []
for i in range(10):
topic = X_topics[:, i].argsort()
text_idx.append(topic[-1:-4:-1])
print('Topic #%d:' % (i + 1), '(text #%d)' %topic[-1])
print(df['title_abstract'][topic[-1]])#[:300],'...')
Comparison Between Top Words and Corresponding Articles¶
- Topic 1: Whole-page optimization (optimization)
- Topic 2: Online Learning of Image Similarity (image)
- Topic 3: Cardinal Contest (no match)
- Topic 4: Speech Recognition (language)
- Topic 5: Planar Group->planar graphs (graph, which however is not really the main topic)
- Topic 6: Identifying Exoplanet (neural is the matching word, yet not really the main topic)
- Topic 7: Novel modes and adaptive block scanning order (time is the matching word, yet not the main topic)
- Topic 8: User Experience Research (research)
- Topic 9: DNA-Based Approach to Synthetic Biology (system is the matching word, yet not the main topic)
- Topic 10: human brain regions (no match)
- Comment:
- 因為top words一開始就是許多trivial的字，因此實際上大多與文章主題關
- Topic 3、10幾乎沒有相符的字出現，表示這兩群的分類相對較差
- Topic 2、4看起來符合度很不錯，分得相對較佳
- 除了CountVecterizer的問題，這份Dataset內容其實本來就很雜，也導致容易發現共通點比較傾向於一些ML的常見用詞
Adding max_df Parameter¶
In [15]:
count = CountVectorizer(stop_words='english',
max_df=0.1,
max_features=10000)
lda = LatentDirichletAllocation(n_components=10, random_state=6, learning_method='batch')
X_lda = count.fit_transform(df['title_abstract'].values)
X_topics = lda.fit_transform(X_lda)
n_top_words = 5
top_words = []
feature_names = count.get_feature_names()
for idx, topic in enumerate(lda.components_):
print("Topic %d:" % (idx + 1))
top_words.append([feature_names[i] for i in topic.argsort()[:-n_top_words-1:-1]])
print(" ".join(top_words[idx]))
Tuning max_df Parameter¶
- Top Words (Tuning max_df Parameter): 同樣使用未經stem/lemmatization的資料，max_df從0.2開始測試，降到0.1之後發現原本回有的data、user等等這些比較無法代表文章卻很常出現的詞消失，取而代之的是(例如Topic 1的)3d、object，更容易判斷主題
In [16]:
text_idx = []
for i in range(10):
topic = X_topics[:, i].argsort()
text_idx.append(topic[-1:-4:-1])
print('Topic #%d:' % (i + 1), '(text #%d)' %topic[-1])
print(df['title_abstract'][topic[-1]])#[:300],'...')
Comparison¶
- Topic 1: Image Focus Quality (image)
- Topic 2: Power Management of Online Data-Intensive Services (power、service)
- Topic 3: DNA-Based Approach to Synthetic Biology (dna)
- Topic 4: Recursive Sparse Spatiotemporal Coding ('vedio' is the only match, yet not really the topic)
- Topic 5: Planar Group->planar graphs (graph, which however is not really the main topic)
- Topic 6: Syntactic and Semantic Annotation (can be related to 'text'(NLP), however no matching word)
- Topic 7: Sharing Curated Friend Groups (online、social(e.g. Google+, Twitter, and Facebook))
- Topic 8: Rational Deconstruction of Landin's SECD Machine (control)
- Topic 9: Models for Neural Spike Computation and Cognition (no match for top words)
- Topic 10: Progressive Web App Features (web)
- Comment:
- 分類的結果比前面要好的許多，幾乎每篇都和Top words蠻相符的
- 第6和第9篇都有在文章中找不到top words，但內容卻與top word類似的情況，也許是由其他未被印出的top eords貢獻的
- 因為少了許多trivial的字，比較起來相符度大幅提高，但因為輸入資料是沒經過處理，所以還是有例如Topic 7中出現的ad和ads這種只差在單複數的詞
Using Lemmatized Data¶
In [17]:
count = CountVectorizer(stop_words='english',
max_df=0.1,
max_features=10000)
lda = LatentDirichletAllocation(n_components=10, random_state=6, learning_method='batch')
X_lda = count.fit_transform(X_lem['Text_Lemmatized'])
X_topics = lda.fit_transform(X_lda)
n_top_words = 5
top_words_lem = []
feature_names = count.get_feature_names()
for idx, topic in enumerate(lda.components_):
print("Topic %d:" % (idx + 1))
top_words_lem.append([feature_names[i] for i in topic.argsort()[:-n_top_words-1:-1]])
print(" ".join(top_words_lem[idx]))
Comment on Top Words¶
- 資料經過Lemmatization處理，會發現就不會像原本那樣同個topic重複出現一樣的字，使得有不同的可能更有意義的字出現
- 但Lemmatization有另一個問題是他是根據詞性轉換，因此會有像測試過程中use、user同時出現的情況
In [18]:
text_idx_lem = []
for i in range(10):
topic = X_topics[:, i].argsort()
text_idx_lem.append(topic[:-4:-1])
print('Topic #%d:' % (i + 1), '(text #%d)' %topic[-1])
print(df['title_abstract'][topic[-1]])#[:300],'...')
Comparison¶
Topic: Main topic of text (related top words)
- Topic 1: Video Recommendations (vedio)
- Topic 2: F-ing modules (no match)
- Topic 3: Rational Deconstruction of Landin's SECD Machine (no match)
- Topic 4: Inquiry Into Election Opinion Polls (survey)
- Topic 5: Human Brain (not related)
- Topic 6: DNA-Based Approach to Synthetic Biology (program)
- Topic 7: On-Skin Interaction (interaction)
- Topic 8: Effects of Sharing Curated Friend Groups (share、social)
- Topic 9: Hand-Eye Coordination for Robotic Grasping (no match for top words)
- Topic 10: Non-Parametric Parametricity (no match)
- Comment:
- 分類的結果，變得比較難找到top word的相關性，分類效果也相較比較差
- Lemmatization並沒有比原始資料好，反而增加運算的time cost(lemmatization除了要逐字做，還要另外判斷詞性，效率低)
Using Stemmed Data¶
In [19]:
count = CountVectorizer(stop_words='english',
max_df=0.1,
max_features=10000)
lda = LatentDirichletAllocation(n_components=10, random_state=6, learning_method='batch')
X_lda = count.fit_transform(X_stem['Text_Stemmed'])
X_topics = lda.fit_transform(X_lda)
n_top_words = 5
top_words_stem = []
feature_names = count.get_feature_names()
for idx, topic in enumerate(lda.components_):
print("Topic %d:" % (idx + 1))
top_words_stem.append([feature_names[i] for i in topic.argsort()[:-n_top_words-1:-1]])
print(" ".join(top_words_stem[idx]))
Comment on Top Words¶
- Possible Topics:
- Topic 1: supervised training
- Topic 2: online survey
- Topic 3、4: media & advertisement (ad suggest...)
- Topic 5: website policies
- Topic 6: vedio processing
- Topic 7: marketing program
- Topic 8: software programming
- Topic 9: speech recognition
- Topic 10: kernel models(ML)
- Top words因為stem的關係只剩詞幹，有些比較難猜出原本的字
- 有幾個topic(例如第5、第7)的字感覺不出有甚麼關聯，比較難猜主題
In [20]:
text_idx_stem = []
for i in range(10):
topic = X_topics[:, i].argsort()
text_idx_stem.append(topic[-1:-4:-1])
print('Topic #%d:' % (i + 1), '(text #%d)' %topic[-1])
print(df['title_abstract'][topic[-1]])#[:300],'...')
Comparison¶
- Topic 1: Hierarchical Label Propagation and Discovery for Machine Generated Email (label)
- Topic 2: Analysis of Progressive Web App Features (web)
- Topic 3: Aerospace Communication Networks (commun)
- Topic 4: DNA-Based Approach to Synthetic Biology (no match)
- Topic 5: Evaluating Machine Learning Models (test)
- Topic 6: Content-based Related Video Recommendations (video、visual)
- Topic 7: Postmarket Drug Surveillance (market)
- Topic 8: Online Data-Intensive Services (service)
- Topic 9: Acoustic Modeling With Back-off N-grams (speech、word)
- Topic 10: Computational Value of Finite Range Tunneling (quantum)
- Comment:
- 分類的結果比lemmatized資料要好，且幾乎每篇都和Top words蠻相符的
- 經過stem的資料雖然只剩詞幹有些難以辨認，但比起未經處理，不會有重複的字出現在同一組top word，更容易分辨出topic，效果較好
In [21]:
match = []
print('Raw Text\n')
for n, idx in enumerate(text_idx):
m = 0
for i in idx:
for w in X['title_abstract'][i]:
if w in top_words[n]:
m+=1
print(' Topic %2d (%s): %d matches' % ((n+1), ', '.join(top_words[n]), m))
match.append(m)
print(' Total matches:', sum(match))
match = []
print('\nLemmatized Text\n')
for n, idx in enumerate(text_idx_lem):
m = 0
for i in idx:
for w in X_lem['Text_Lemmatized'][i].split():
if w in top_words_lem[n]:
m+=1
print(' Topic %2d (%s): %d matches' % ((n+1), ', '.join(top_words_lem[n]), m))
match.append(m)
print(' Total matches:', sum(match))
match = []
print('\nStemmed Text\n')
for n, idx in enumerate(text_idx_stem):
m = 0
for i in idx:
for w in X_stem['Text_Stemmed'][i].split():
if w in top_words_stem[n]:
m+=1
print(' Topic %2d (%s): %d matches' % ((n+1), ', '.join(top_words_stem[n]), m))
match.append(m)
print(' Total matches:', sum(match))
Summary for LDA¶
- Meaning wise
- 前面每個LDA的個別分析主要是以文章主題內容與top word字義為基準來分析分群的效果如何
- 就結果而言，我得到的結論是Stemming > 原始資料 > Lemmatization
- Word wise
- 另外用top words去與每個主題的頭三篇去比較，尋找完全相符的用字的字數，單純以重複的數量來分析
- Raw Text: 可以看到topic 4只有一個符合字，與前面結論"no match"相符，是分類較差的一群；總符合數=199
- Lemmatized Text: 第2、9群幾乎沒有符合，也與前面分析相符；總符合數=146
- Stemmed Text: 每群符合量都差不多，說明各群的分類效果還蠻平均的；總符合數=195
- 結果: Raw > Stemmed > Lemmatized
- 小結:
- Lemmatized是表現最差的，Raw 和 Stemmed Data表現差不多，但我認為Stemmed會更容易將類似的字併在一起，比較不會導致意義相近的字卻被當成不同字來計算，影響結果，會是比較好的選擇。以下的分群將會以Stemmed的資料為主。
- LDA的分群能夠將top words與article本身比對，人為解釋大部分看起來都還蠻相符的。
Vectorize¶
- 因為Tfidf可以濾掉在每一篇文章都很常出現(無助於區分不同cluster)的token，用TfidfVectorizer處理過的資料比CountVectorizer得到的distortion還小，所以這裡直接用TfidfVectorizer
- 分別用stemmed、lemmatized的資料轉成vector，並用array形式印出來
In [13]:
from sklearn.feature_extraction.text import TfidfVectorizer
In [14]:
tfidf1 = TfidfVectorizer(max_features=10000, max_df=0.1)
X_lem_vect = tfidf1.fit_transform(X_lem['Text_Lemmatized'])
tfidf2 = TfidfVectorizer(max_features=10000, max_df=0.1)
X_stem_vect = tfidf2.fit_transform(X_stem['Text_Stemmed'])
In [15]:
X_lem_vect = X_lem_vect.toarray()
X_stem_vect = X_stem_vect.toarray()
print('X Lemmatized:')
print(X_lem_vect)
print('X Stemmed:')
print(X_stem_vect)
K-Means++¶
In [17]:
from sklearn.cluster import KMeans
In [18]:
import pyprind
pbar = pyprind.ProgBar(18)
distortions = []
for i in range(3, 21):
km = KMeans(n_clusters=i,
n_init=10,
max_iter=500,
tol=1e-04,
random_state=0,
n_jobs=-1)
km.fit(X_stem_vect.T)
distortions.append(km.inertia_)
pbar.update()
print('# of Clusters | Distortion')
print('--------------|-----------')
for i, d in enumerate(distortions):
print(' %-2d | %.2f' % (i+3, d))
In [16]:
%matplotlib inline
import matplotlib.pyplot as plt
In [20]:
plt.plot(range(3,21), distortions, marker='o')
plt.xlabel('Number of clusters')
plt.ylabel('Distortion')
plt.xlim(3,20)
plt.grid()
plt.tight_layout()
plt.show()
Elbow Method¶
- Algorithm: K-means++
- Parameters:
- max_iter: 若設為1000，會大幅提高執行時間，但並沒有明顯得到較低的Distortion，顯示這樣的設定效率不彰，因此設定為500
- n_init: 10
- tol: 1e-4
- Result:
- 如果以大尺度來看(測試n_cluster的範圍較大，例如1~50群)，會得到一條幾乎是斜直線的圖形
- 將範圍降到3~20之後發現在7~8群之間有一段明顯下降，除此之外下降率還是幾乎差不多
- 選擇分為8群
In [18]:
from matplotlib import cm
from sklearn.metrics import silhouette_samples
import numpy as np
In [23]:
km = KMeans(n_clusters=8, n_init=10, max_iter=1000, tol=1e-04, random_state=0)
y_km = km.fit_predict(X_stem_vect.T)
cluster_labels = np.unique(y_km)
n_clusters = cluster_labels.shape[0]
silhouette_vals = silhouette_samples(X_stem_vect.T, y_km, metric='euclidean')
In [25]:
feature_stem = tfidf2.get_feature_names()
for n in range(8):
print('Cluster #%2d | ' % (n+1), end='')
val = []
index = []
for i, c in enumerate(y_km):
if c == n:
val.append(silhouette_vals[i])
index.append(i)
sort = np.argsort(-np.array(val))
print('Avg Silhouette: %6.3f | Top Words: ' %np.average(val), end='')
for j in range(5):
if j < len(index):
print(feature_stem[index[sort[j]]], end=' ')
print('')
Comment On Top Words¶
- 大部份與LDA結果不同，第4、7群有些類似，分別可能為speech recognition、media advertising相關主題
- 第6群包含大部分的字，因此Top Words看起來反而比較砸，無法直觀判斷相關性
In [26]:
y_ax_lower, y_ax_upper = 0, 0
yticks = []
for i, c in enumerate(cluster_labels):
c_silhouette_vals = silhouette_vals[y_km==c]
c_silhouette_vals.sort()
y_ax_upper += len(c_silhouette_vals)
color = cm.jet(float(i) / n_clusters)
plt.barh(range(y_ax_lower, y_ax_upper), c_silhouette_vals, height=1.0, edgecolor='none', color=color)
yticks.append((y_ax_lower+y_ax_upper) / 2.)
y_ax_lower += len(c_silhouette_vals)
silhouette_avg = np.mean(silhouette_vals)
plt.axvline(silhouette_avg, color='red')
plt.yticks(yticks, cluster_labels +1)
plt.ylabel('Cluster')
plt.xlabel('Silhouette coefficient')
plt.tight_layout()
plt.show()
K-Means++ Model Training & Result¶
- Model Parameters:
- max_iter: 選擇保守的1000次(希望盡量能讓model更好)
- n_cluster: 由elbow method結果選擇8
- Top Words & Silhouette:
- 有幾群只包含單一個字，因此sihlouette值會是0(no distance)
- 第6群是唯一平均silhouette為正值的一群，包含大部分的字
- 剩下的幾群silhouette值約都在-0.2左右，表示是分類錯誤的結果，嘗試改動參數(max_iter from 500 $\rightarrow$ 1000)，沒有太大改善
- Plot: 由寬度可以看出第6群包含了大部分的字，且是唯一一群silhouette值為正的cluster，平均值約0.6，並不算非常理想
Word Clustering: 這一部份用來training的資料都經過transpose，使得word成為instance，而文章本身則是feature。這麼做是將文字分群，類似LDA所做的事情，方便比較。但有一例外是FCM的training資料要求格式本來就恰好是其他方法的transpose，因此要用FCM做word clustering，反而不用transpose。之後會有Article Clustering，則不需要transpose，其結果是將文章依類似的主題(擁有相似的字)分群。¶
Fuzzy C-Means¶
In [27]:
import skfuzzy as fuzz
fpcs = []
print('# of Clusters | FPC ')
print('--------------|-----------')
for ncenters in range(3, 11):
cntr, u, u0, d, jm, p, fpc = fuzz.cluster.cmeans(
X_stem_vect, ncenters, 2, error=0.0001, maxiter=1000, init=None)
fpcs.append(fpc)
print(' %2d | %.3f' % (ncenters, fpc))
In [28]:
fig2, ax2 = plt.subplots()
ax2.plot(np.r_[3:11], fpcs)
ax2.set_xlabel("Number of centers")
ax2.set_ylabel("Fuzzy partition coefficient")
Out[28]:
Fuzzy Partition Coeffitient (FPC)¶
- 用來判斷Fuzzy C-Means 好壞的參數，越接近1表示越好
- 定義: $F(U;c)=\frac{1}{n}\sum^n_{i=1}\sum^c_{j=1}u_{ij}^2$
- $n$: number of pattern(words)
- $c$: number of clusters
- $u_{ij}$: membership degree of $i_{th}$ pattern to $j_{th}$ center
- 結果: 每個FPC都是c的倒數，因此只要c越大，FPC越小，但不確定原因，猜測與training data中帶有許多0有關
In [29]:
cntr, u, u0, d, jm, p, fpc = fuzz.cluster.cmeans(
X_stem_vect, 3, 2, error=0.005, maxiter=1000, init=None)
cluster_membership = np.argmax(u, axis=0)
print(u, u.shape)
print(cluster_membership, len(cluster_membership))
Model Training¶
- Parameters:
- cluster數: 依FPC結果選擇3群
- maxiter: 1000
- Result:
- u存的是每個資料屬於各群的機率，印出來觀察會看到幾乎每筆資料，屬於各群的機率都是0.3333，也就是機率相等，有分等於沒分，結果不具參考價值
- cluster_membership顯示的是最終該字會被分到哪一群，雖然看出幾乎都屬於三群中的第二群，但如前面所述，不大具有參考價值
In [ ]:
for i, c in enumerate(cluster_membership):
if c == 1:
print(feature_stem[i], end=' ')
In [ ]:
print(X_stem_vect.T.shape)
print(u.shape)
print(u.T)
In [30]:
from sklearn.metrics import silhouette_samples
silhouette_vals = silhouette_samples(X_stem_vect.T, (u.T).argmax(axis = 1), metric='euclidean')
feature_stem = tfidf2.get_feature_names()
for n in range(3):
print('Cluster #%2d | ' % (n+1), end='')
val = []
index = []
for i, c in enumerate(cluster_membership):
if c == n:
val.append(silhouette_vals[i])
index.append(i)
sort = np.argsort(-np.array(val))
print('Top Words: ', end='')
for j in range(5):
if j < len(index):
print(feature_stem[index[sort[j]]], end=' ')
print('')
Agglomerative¶
In [29]:
from sklearn.cluster import AgglomerativeClustering
from sklearn.metrics import silhouette_score
%matplotlib inline
import matplotlib.pyplot as plt
In [30]:
import pyprind
pbar = pyprind.ProgBar(8)
sil_val = []
for n in range(3, 11):
ac = AgglomerativeClustering(n_clusters=n,
affinity='euclidean',
linkage='complete')
y_ac = ac.fit_predict(X_stem_vect.T)
sil_val.append(silhouette_score(X_stem_vect.T, y_ac, metric='euclidean'))
pbar.update()
plt.plot(range(3,11), sil_val, marker='s')
plt.xlabel('Number of clusters')
plt.ylabel('Silhouette Value')
plt.xlim(3,10)
plt.grid()
plt.tight_layout()
plt.show()
Silhouette Score¶
- 和silhouette sample不一樣，score回傳的是整份的平均silhouette值，可以用來觀察分幾群的效果好不好；sample則是回傳的是個別instance的值
- 結果: 分3~4群的效果看起來差不多，而第4群silhouette值又較高一點，到第5群明顯下降，所以選擇分4群
In [32]:
ac = AgglomerativeClustering(n_clusters=4,
affinity='euclidean',
linkage='complete')
labels = ac.fit_predict(X_stem_vect.T)
print('Cluster labels %s' % labels[0:10])
In [33]:
from matplotlib import cm
from sklearn.metrics import silhouette_samples
cluster_labels = np.unique(labels)
n_clusters = cluster_labels.shape[0]
silhouette_vals = silhouette_samples(X_stem_vect.T, labels, metric='euclidean')
y_ax_lower, y_ax_upper = 0, 0
yticks = []
for i, c in enumerate(cluster_labels):
c_silhouette_vals = silhouette_vals[labels==c]
c_silhouette_vals.sort()
y_ax_upper += len(c_silhouette_vals)
color = cm.jet(float(i) / n_clusters)
plt.barh(range(y_ax_lower, y_ax_upper), c_silhouette_vals, height=1.0, edgecolor='none', color=color)
yticks.append((y_ax_lower+y_ax_upper) / 2.)
y_ax_lower += len(c_silhouette_vals)
silhouette_avg = np.mean(silhouette_vals)
plt.axvline(silhouette_avg, color='red')
plt.yticks(yticks, cluster_labels +1)
plt.ylabel('Cluster')
plt.xlabel('Silhouette coefficient')
plt.tight_layout()
plt.show()
In [34]:
silhouette_vals = silhouette_samples(X_stem_vect.T, labels, metric='euclidean')
feature_stem = tfidf2.get_feature_names()
for n in range(4):
print('Cluster #%2d | ' % (n+1), end='')
val = []
index = []
for i, c in enumerate(labels):
if c == n:
val.append(silhouette_vals[i])
index.append(i)
sort = np.argsort(-np.array(val))
print('Top Words: ', end='')
for j in range(5):
if j < len(index):
print(feature_stem[index[sort[j]]], end=' ')
print('')
Model Training & Result¶
- 分4群
- 從Silhouette Plot可以看出幾乎都分在第1群，平均值約0.8左右，是不錯的值，但其他群分布則不平均
- 第3群可以看出是speech recognition相關主題，第1群包含大部分字，top words較雜看不出關聯性，其他群則都只包含一個字。
Summary for Word Clustering¶
- LDA效果應屬最佳，其中尤其用raw data(max_df=0.1)、stemmed data
- FCM 的效果最差，無法由FPC判斷適合分幾群，且分出來每群機率都相同，沒有太大意義
- K-Means++ 與 Agglomerative結果差不多，都是有一群特別多，差在Agglomerative的silhouette值較高，相對好一點
- 因此底下Article Clustering 會選擇使用K-Means++、Agglomerative做traing
- K-Means++是K-means的較佳版本(有依據的選擇起始center，而不是random的選擇，因此本篇報告直接選擇K-Means++，沒有使用K-Means)
In [19]:
import pyprind
pbar = pyprind.ProgBar(18)
distortions = []
for i in range(3, 21):
km = KMeans(n_clusters=i,
n_init=10,
max_iter=500,
tol=1e-04,
random_state=0,
n_jobs=-1)
km.fit(X_stem_vect)
distortions.append(km.inertia_)
pbar.update()
print('# of Clusters | Distortion')
print('--------------|-----------')
for i, d in enumerate(distortions):
print(' %-2d | %.2f' % (i+3, d))
In [20]:
plt.plot(range(3,21), distortions, marker='o')
plt.xlabel('Number of clusters')
plt.ylabel('Distortion')
plt.xlim(3,20)
plt.grid()
plt.tight_layout()
plt.show()
Elbow Method¶
- 可以看到幾乎是條斜直線，沒有明顯轉折
- 在第5群有些微轉折，所以最後選擇分5群
In [21]:
km = KMeans(n_clusters=5, n_init=10, max_iter=1000, tol=1e-04, random_state=0)
y_km = km.fit_predict(X_stem_vect)
cluster_labels = np.unique(y_km)
n_clusters = cluster_labels.shape[0]
silhouette_vals = silhouette_samples(X_stem_vect, y_km, metric='euclidean')
In [44]:
for n in range(5):
print('Cluster #%2d | ' % (n+1), end='')
val = []
index = []
for i, c in enumerate(y_km):
if c == n:
val.append(silhouette_vals[i])
index.append(i)
sort = np.argsort(-np.array(val))
print('Avg Silhouette: %6.3f | Topics: ' %np.average(val), end='')
for j in range(3):
print(df['title_abstract'][index[sort[j]]].split('.')[0], end='. ')
print('')
Comment on Top Topics¶
- 第一群很明顯都與vedio有關
- 第二群都在做speech synthesis
- 第三群比較不確定共通性(maybe NLP)
- 第四群不明所以印出兩個一模一樣的題目(也許資料中有重複)
- 第五群有兩篇與Mutation有關
- 雖說silhouette值很差，但還是看得出分出來的文章間具有一定相關性。
In [28]:
y_ax_lower, y_ax_upper = 0, 0
yticks = []
for i, c in enumerate(cluster_labels):
c_silhouette_vals = silhouette_vals[y_km==c]
c_silhouette_vals.sort()
y_ax_upper += len(c_silhouette_vals)
color = cm.jet(float(i) / n_clusters)
plt.barh(range(y_ax_lower, y_ax_upper), c_silhouette_vals, height=1.0, edgecolor='none', color=color)
yticks.append((y_ax_lower+y_ax_upper) / 2.)
y_ax_lower += len(c_silhouette_vals)
silhouette_avg = np.mean(silhouette_vals)
plt.axvline(silhouette_avg, color='red')
plt.yticks(yticks, cluster_labels +1)
plt.ylabel('Cluster')
plt.xlabel('Silhouette coefficient')
plt.tight_layout()
plt.show()
Model Training & Result¶
- Silhouette值全部都超級低，甚至很多是負的
- 分群也不是很平均(不一定是壞事)，第一群特別少，第三、五群則特別多
- 總結來說印出題目還是能看得出相關性，但由silhouette值低推測，第一可能是這樣的資料並不適合這樣分群，第二可能是只有頭幾篇有相關，後面則相關性很差
Agglomerative¶
In [35]:
import pyprind
pbar = pyprind.ProgBar(8)
sil_val = []
for n in range(3, 11):
ac = AgglomerativeClustering(n_clusters=n,
affinity='euclidean',
linkage='complete')
y_ac = ac.fit_predict(X_stem_vect)
sil_val.append(silhouette_score(X_stem_vect, y_ac, metric='euclidean'))
pbar.update()
plt.plot(range(3,11), sil_val, marker='s')
plt.xlabel('Number of clusters')
plt.ylabel('Silhouette Value')
plt.xlim(3,10)
plt.grid()
plt.tight_layout()
plt.show()
Comment¶
- Silhouette分超過4群以上都是負的，只有分4群時為正，但也非常小
- 選擇分4群
In [36]:
ac = AgglomerativeClustering(n_clusters=4,
affinity='euclidean',
linkage='complete')
labels = ac.fit_predict(X_stem_vect)
print('Cluster labels %s' % labels[0:10])
In [37]:
from matplotlib import cm
from sklearn.metrics import silhouette_samples
cluster_labels = np.unique(labels)
n_clusters = cluster_labels.shape[0]
silhouette_vals = silhouette_samples(X_stem_vect, labels, metric='euclidean')
y_ax_lower, y_ax_upper = 0, 0
yticks = []
for i, c in enumerate(cluster_labels):
c_silhouette_vals = silhouette_vals[labels==c]
c_silhouette_vals.sort()
y_ax_upper += len(c_silhouette_vals)
color = cm.jet(float(i) / n_clusters)
plt.barh(range(y_ax_lower, y_ax_upper), c_silhouette_vals, height=1.0, edgecolor='none', color=color)
yticks.append((y_ax_lower+y_ax_upper) / 2.)
y_ax_lower += len(c_silhouette_vals)
silhouette_avg = np.mean(silhouette_vals)
plt.axvline(silhouette_avg, color='red')
plt.yticks(yticks, cluster_labels +1)
plt.ylabel('Cluster')
plt.xlabel('Silhouette coefficient')
plt.tight_layout()
plt.show()
In [45]:
silhouette_vals = silhouette_samples(X_stem_vect, labels, metric='euclidean')
for n in range(4):
print('Cluster #%2d | ' % (n+1), end='')
val = []
index = []
for i, c in enumerate(labels):
if c == n:
val.append(silhouette_vals[i])
index.append(i)
sort = np.argsort(-np.array(val))
print('Article: ', end='')
for j in range(3):
print(df['title_abstract'][index[sort[j]]].split('.')[0], end='. ')
print('')
Result¶
- Cluster 1: speech synthesis
- Cluster 2: quantum related topics
- Cluster 3: vedio related
- Cluster 4: (not sure what the relationship is) something to do with 'randomized'
- 與上一個結果一樣，還是可以看出明顯的相關性，應該是後面的文章的相關性很低，所以silhouette很低
Summary¶
- Preporcessing
- text cleaning: 清掉了html、標點符號、數字，也把stop words刪去
- tokenizing, then stemming、lematizing
- 在LDA中比較了 stemming、lematizing ，結論為stemming較佳
- 也有兩者並用的作法，但我主要想做兩者的比較，所以選擇平行的做
- vectorizing: CountVectorizer for LDA(因其計算方式關係只能用這個), TfidfVectorizer for the rest(可以去掉常見卻不具代表意義的詞)
- Parameter Choosing
- 觀察trivial的有無來調動max_df
- 用Silhouette、Elbow Method選擇分幾群
- 增加max_iter來達到較好的分群結果
- Algorithm Chossing/Comparing
- 作業題目中提到的五個方法中，K-Means++是K-Means的較佳版，故除了K-Means以外其他都有做
- 在Word Clustering比較結果，我認為LDA會是最佳(最適合以字分群)，K-Means++與Agglomerative次佳
- 在Article Clustering，選擇K-Means++與Agglomerative來做，兩者結果依然相近，效果都差不多。