What is text clustering?
Text clustering is to transform the original natural language text information into mathematical information, which is displayed in the form of high-dimensional spatial points. By calculating the distance between those points, those points are clustered into a cluster, and the center of the cluster is called the cluster center. A good clustering should ensure that the distance between the points in the cluster is as close as possible, but the points between clusters should be as far as possible.
What are the difficulties of text clustering?
Clustering is a kind of unsupervised learning, that is to say, we don't know how to cluster into several categories, and we can only try it out a little bit. However, sometimes machines think that these two piles of points can be regarded as two clusters, but human understanding may be one cluster, so text clustering is difficult. The understanding of machines and people is different. Generally, people who can see this blog have learned the basic clustering algorithm. Take k-means as an example. The selection of cluster center is a very random process, resulting in different clustering results every time when the k value is the same, and it is not easy to take an average, so it is difficult to evaluate the quality of clustering.
How to evaluate clustering?
http://blog.csdn.net/chixujohnny/article/details/51852633 S, who talked about everything_ I haven't tried the DBW evaluation index yet. Students who are interested can try it. In short, it should be used sooner or later.
Text clustering process
Word weighting method and textrate are useless. jieba comes with it. It looks good to the naked eye. You can try it.
The above figure shows several parts. Generally, the format of generating document vector matrix is that each row represents a document, and each column is a dimension, representing the weight of the word "document". If the word does not appear, it is 0, and thousands of file dimensions are more than 10 w Left and right (see the size of the document). With such a large dimension, the human brain thought and thought that the matrix would be extremely sparse. That is to say, in a high-dimensional space, thousands of points almost gather together. Although there is a distance between them, the distance is very small. Obviously, the clustering effect must be very poor. It has been measured, which is the same as the probability of coin tossing. So I thought of making the matrix a little denser pca Dimensionality reduction, pca It is the abbreviation of principal component analysis, which roughly means to take the direction with the largest variance in this high-dimensional vector, retain the useful part and discard the useless part after some mathematical transformation. This method is also suitable for finding the most hanging features in the classification algorithm. The specific details I saw in the book "machine learning practice" (the book of a little brother carrying a sack), I didn't speak in detail, but I understood it roughly. Why not SVD Dimensionality reduction, SVD Suitable for dense matrix, such as image matrix or recommendation system, take 80%Useful information, suitable for image compression algorithm (if you don't know deeply, please slap your face). I actually saw the concept of contour coefficient in the blog of this Beiyou student: buptguo.com/2016/05/31/learn-ml-from-scikit-learn-silhouette-analysis/ Just click in and have a look. He speaks better than me. I read some data about clustering. Those of Baidu are all used k-means Do text clustering, I just want to ask if you do school homework? Thousands of dimensional vectors k-means Do? Funny? I measured it, the effect is very poor, the accuracy of throwing the sieve. Later, I began to check the literature. One kind is called BIRCH Hierarchical clustering algorithm, the algorithm can be better solved k-means The deviation of each clustering result is too large dbscan The number of clusters can be set (of course, the threshold can also be set). The most important thing is sklearn There are ready-made library calls, and the speed is very fast. Give it a try, Billy kmeans OK, but I'm not particularly satisfied. After checking, it's still not suitable for high-dimensional space. I'm ready to check the data again.
There are several points that may need to be tried:
The threshold of clustering algorithm can be set. Try any other clustering algorithm, especially the high-dimensional clustering algorithm.
Assign the weight with textrate to see how the effect is
code:
# coding:utf-8
# 2.0 use jieba for word segmentation, completely abandon the inefficient NLPIR, and assign the weight with textrank algorithm (the measured textrank effect is better)
# 2.1 make tfidf with gensim
# 2.2 sklearn does tfidf and kmeans
# 2.3 change kmeans to BIRCH and use traditional tfidf
import logging
import time
import os
import jieba
import glob
import random
import copy
import chardet
import gensim
from gensim import corpora,similarities, models
from pprint import pprint
import jieba.analyse
from sklearn import feature_extraction
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.feature_extraction.text import CountVectorizer
import os
from sklearn.decomposition import PCA
# logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)
start = time.clock()
print '#----------------------------------------#'
print '# #'
print '# Loading corpus #'
print '# #'
print '#----------------------------------------#\n'
def PreprocessDoc(root):
allDirPath = [] # The left and right folder paths under the corpus dataset folder, string,[1:] is the required path
fileNumList = []
def processDirectory(args, dirname, filenames, fileNum=0):
allDirPath.append(dirname)
for filename in filenames:
fileNum += 1
fileNumList.append(fileNum)
os.path.walk(root, processDirectory, None)
totalFileNum = sum(fileNumList)
print 'The total number of documents is: ' + str(totalFileNum)
return allDirPath
print '#----------------------------------------#'
print '# #'
print '# Synthetic corpus document #'
print '# #'
print '#----------------------------------------#\n'
# Each document has one line, and the first word is the category of the document
def SaveDoc(allDirPath, docPath, stopWords):
print 'Start synthesizing corpus documents:'
category = 1 # Category of document
f = open(docPath,'w') # Put all the text together in this document
for dirParh in allDirPath[1:]:
for filePath in glob.glob(dirParh + '/*.txt'):
data = open(filePath, 'r').read()
texts = DeleteStopWords(data, stopWords)
line = '' # Reduce these words to one line. The first position is the document category, separated by spaces
for word in texts:
if word.encode('utf-8') == '\n' or word.encode('utf-8') == 'nbsp' or word.encode('utf-8') == '\r\n':
continue
line += word.encode('utf-8')
line += ' '
f.write(line + '\n') # Write this line into the file
category += 1 # After scanning a folder, category + 1
return 0 # Generate document without return value
print '#----------------------------------------#'
print '# #'
print '# participle+De stop word #'
print '# #'
print '#----------------------------------------#\n'
def DeleteStopWords(data, stopWords):
wordList = []
# Divide the words first
cutWords = jieba.cut(data)
for item in cutWords:
if item.encode('utf-8') not in stopWords: # The word segmentation code shall be consistent with the stop word code
wordList.append(item)
return wordList
print '#----------------------------------------#'
print '# #'
print '# tf-idf #'
print '# #'
print '#----------------------------------------#\n'
def TFIDF(docPath):
print 'start tfidf:'
corpus = [] # Document corpus
# Read the corpus, and one line of corpus is a document
lines = open(docPath,'r').readlines()
for line in lines:
corpus.append(line.strip()) # There are no spaces before and after strip(), but the middle space is still reserved
# Convert the words in the text into word frequency matrix, and the matrix element a[i][j] represents the word frequency of j words under class I text
vectorizer = CountVectorizer()
# This class will count the tfidf weight of each word
transformer = TfidfTransformer()
# First fit_transform is the second fit to calculate TF IDF_ Transform is to convert text into word frequency matrix
tfidf = transformer.fit_transform(vectorizer.fit_transform(corpus))
# Get all words in the word bag model
word = vectorizer.get_feature_names()
# The TF IDF matrix is extracted, and the element w[i][j] represents the TF IDF weight of j words in class I text
weight = tfidf.toarray()
print weight
# # Output all words
# result = open(docPath, 'w')
# for j in range(len(word)):
# result.write(word[j].encode('utf-8') + ' ')
# result.write('\r\n\r\n')
#
# # Output all weights
# for i in range(len(weight)):
# for j in range(len(word)):
# result.write(str(weight[i][j]) + ' ')
# result.write('\r\n\r\n')
#
# result.close()
return weight
print '#----------------------------------------#'
print '# #'
print '# PCA #'
print '# #'
print '#----------------------------------------#\n'
def PCA(weight, dimension):
from sklearn.decomposition import PCA
print 'Original dimension: ', len(weight[0])
print 'Start dimensionality reduction:'
pca = PCA(n_components=dimension) # Initialize PCA
X = pca.fit_transform(weight) # Return dimension reduced data
print 'Dimensionality after dimensionality reduction: ', len(X[0])
print X
return X
print '#----------------------------------------#'
print '# #'
print '# k-means #'
print '# #'
print '#----------------------------------------#\n'
def kmeans(X, k): # X=weight
from sklearn.cluster import KMeans
print 'Start clustering:'
clusterer = KMeans(n_clusters=k, init='k-means++') # Set clustering model
# X = clusterer.fit(weight) # According to the text vector fit
# print X
# print clf.cluster_centers_
# Cluster to which each sample belongs
y = clusterer.fit_predict(X) # Throw the weight matrix into fit and output label
print y
# i = 1
# while i <= len(y):
# i += 1
# It is used to evaluate whether the number of clusters is appropriate. The better the distance is, the number of clusters at the critical point is selected
# print clf.inertia_
return y
print '#----------------------------------------#'
print '# #'
print '# BIRCH #'
print '# #'
print '#----------------------------------------#\n'
def birch(X, k): # Lattice to be clustered, number of clusters
from sklearn.cluster import Birch
print 'Start clustering:'
clusterer = Birch(n_clusters=k)
y = clusterer.fit_predict(X)
print 'Output clustering results:'
print y
return y
print '#----------------------------------------#'
print '# #'
print '# Contour coefficient #'
print '# #'
print '#----------------------------------------#\n'
def Silhouette(X, y):
from sklearn.metrics import silhouette_samples, silhouette_score
print 'Calculate contour coefficient:'
silhouette_avg = silhouette_score(X, y) # Average contour coefficient
sample_silhouette_values = silhouette_samples(X, y) # Contour coefficient of each point
pprint(silhouette_avg)
return silhouette_avg, sample_silhouette_values
print '#----------------------------------------#'
print '# #'
print '# Draw a picture #'
print '# #'
print '#----------------------------------------#\n'
def Draw(silhouette_avg, sample_silhouette_values, y, k):
import matplotlib.pyplot as plt
import matplotlib.cm as cm
import numpy as np
# Create a subplot with 1-row and 2-column
fig, ax1 = plt.subplots(1)
fig.set_size_inches(18, 7)
# First subplot contour coefficient point
# The range is [- 1, 1]
ax1.set_xlim([-0.2, 0.5])
# The following (k + 1) * 10 is to show these points more clearly
ax1.set_ylim([0, len(X) + (k + 1) * 10])
y_lower = 10
for i in range(k): # Traverse these clusters respectively
ith_cluster_silhouette_values = sample_silhouette_values[y == i]
ith_cluster_silhouette_values.sort()
size_cluster_i = ith_cluster_silhouette_values.shape[0]
y_upper = y_lower + size_cluster_i
color = cm.spectral(float(i)/k) # Make a color
ax1.fill_betweenx(np.arange(y_lower, y_upper),
0,
ith_cluster_silhouette_values,
facecolor=color,
edgecolor=color,
alpha=0.7) # I don't know what this coefficient does
# Add the cluster category number to the contour coefficient point
ax1.text(-0.05, y_lower + 0.5 * size_cluster_i, str(i))
# Calculate the y of the next point_ Lower Y-axis position
y_lower = y_upper + 10
# Make a vertical dotted line in the figure
ax1.axvline(x=silhouette_avg, color='red', linestyle="--")
plt.show()
if __name__ == "__main__":
root = '/Users/John/Desktop/test'
stopWords = open('/Users/John/Documents/NLPStudy/stopwords-utf8', 'r').read()
docPath = '/Users/John/Desktop/test/doc.txt'
k = 3
allDirPath = PreprocessDoc(root)
SaveDoc(allDirPath, docPath, stopWords)
weight = TFIDF(docPath)
X = PCA(weight, dimension=800) # Dimensionality reduction of original weight data
# y = kmeans(X, k) # y = class label after clustering
y = birch(X, k)
silhouette_avg, sample_silhouette_values = Silhouette(X, y) # Contour coefficient
Draw(silhouette_avg, sample_silhouette_values, y, k)
end = time.clock()
print 'Running time: ' + str(end - start)
reference material:
Text clustering tutorial