Beginners Guide to Topic Modelling in Python

One of the primary uses of natural language processing is identifying and analyzing popular topics from huge volumes of text automatically. A few examples of huge text could be content from online media, client audits of hotels and restaurants, motion pictures, and so on. Other examples of massive text streams include client criticisms, reports, customer complaints, and so forth (Prabhakaran, 2018). Being aware of what individuals are discussing and understanding their issues and opinions is profoundly important to businesses, political campaigns, and the like. What’s more, it’s truly difficult to manually peruse such enormous volumes of data and gather the points.

The analytics industry is primarily concerned with getting the “relevant information” within the collection of data (Shivam Bansal, 2016). An increasing amount of data excessively, for the most part, unstructured, has made it difficult to get important and relevant information. In any case, innovation has built up some amazing methods that can be utilized to mine and analyze the data and get the precise data that we are searching for. 

Introduction to Topic Modelling

Topic modeling is a sophisticated natural language processing approach that allows us to extract relevant subjects from a collection of text documents. It assists us in uncovering hidden themes and patterns in data, giving significant insights for a variety of applications. By understanding the basics of topic modeling, beginners can dive into this fascinating field and unlock its potential.

In topic modeling, we aim to discover latent topics present in the text corpus without any prior knowledge of the topics themselves. One popular algorithm used for topic modeling is Latent Dirichlet Allocation (LDA). LDA assumes that each document consists of a mixture of topics, and each topic is a probability distribution over words.

Resources and Tools for Topic Modelling in Python

When it comes to implementing Python topic modeling, a variety of resources and tools are at your disposal that may help simplify the process and improve your productivity. Here are some prominent sites and tools to consider:

• Gensim: Gensim is a popular Python library for topic modeling, text similarity analysis, and other natural language processing tasks. It provides an easy-to-use interface for implementing topic modeling algorithms like LDA and LSI (Latent Semantic Indexing). Gensim also offers functionalities for pre-processing text data, creating document-term matrices, and visualizing topic models.
• NLTK (Natural Language Toolkit): NLTK is a comprehensive Python package that provides a wide range of natural language processing capabilities and resources. It features tokenization, stemming, part-of-speech tagging, and other modules. NLTK also provides corpora and datasets that can be used for training and evaluating topic models.
• Scikit-learn: Although primarily known for machine learning tasks, Scikit-learn also offers useful utilities for topic modeling. It provides efficient implementations of NMF (Non-negative Matrix Factorization) and LDA algorithms. Scikit-learn integrates seamlessly with other data processing and analysis libraries, making it a versatile choice for topic modeling in Python.
• pyLDAvis: pyLDAvis is a Python library that enables interactive visualization and interpretation of topic models. It allows you to explore the topics, their associated keywords, and the document-topic distributions intuitively and interactively. pyLDAvis can help you gain insights into the coherence and relevance of topics in your model.
• Topic Modelling Courses and Tutorials: Online learning platforms like Coursera, Udemy, and DataCamp offer courses specifically dedicated to topic modeling in Python. These courses cover the fundamentals of topic modeling algorithms, implementation techniques, and best practices. They often include hands-on exercises and real-world examples to reinforce your understanding.
• Research Papers and Publications: Keep up to speed on the most recent research papers and publications in the field of topic modeling. Cutting-edge research on this subject is published in journals such as the Journal of Machine Learning Research (JMLR), Computational Linguistics, and Transactions of the Association for Computational Linguistics (TACL). Reading research papers can provide valuable insights into advanced topic-modeling techniques and novel approaches.
• Online Communities and Forums: Engage with online communities and forums dedicated to natural language processing and topic modeling. Websites like Stack Overflow, Reddit, and Data Science Stack Exchange have active communities where you can ask questions, seek guidance, and learn from experienced practitioners.

By leveraging these resources and tools, you can enhance your topic modeling skills and streamline your implementation process in Python. Remember to explore the documentation, examples, and tutorials provided by these resources to maximize your understanding and proficiency in topic modeling.

Topic modelling is one such method in the text mining field

. Topic modelling is a process where the topics existing in a text object are automatically identified, and the hidden patterns and trends displayed by a text corpus are derived, eventually improving the decision-making capability (Shivam Bansal, 2016).

The major difference between topic modelling and rule-based text mining is that rule-based text mining is a supervised approach that utilizes ordinary expressions or word reference-based keyword finding strategies, whereas topic modelling is a technique that is unsupervised and used for identifying and observing the collection of words (also known as topics) in large groups of texts (Shivam Bansal, 2016). It is an unaided methodology utilized for finding and noticing the pack of words (called “subjects”) in huge groups of texts (Shashank Kapadia, 2019). Topics can be characterized as “recurring instances of co-existing terms in a corpus.” 

Check out our data science courses to upskill yourself.

Topic models are exceptionally helpful for record clustering, putting together huge squares of textual information, extracting information from unstructured data, and feature selection (Shashank Kapadia, 2019). Topic models can be considered as a type of statistical language model that is used to identify the underlying structure from a collection of texts. It can be thought of as a task of:

Dimensionality Reduction

It can be simply defined as a process by which the dimensions of an existing feature set are reduced (Raj, 2019). In simple words, we can say that dimensionality reduction means reducing the number of columns from a dataset. By using dimensionality reduction for topic modelling, we can represent a text in a topic space rather than its feature space.

Feature space representation: {Word_x: count(Word_x, A)}, A is the text

Topic space representation: {Topic_x: Weight(Topic_x, A)}

Unsupervised Learning

In unsupervised learning, there are no labels, and the algorithm finds hidden patterns and can be compared to clustering (Sanatan Mishra, 2017). The number of topics is an output parameter, just like the number of clusters in clustering. In topic modelling, clusters of words are built rather than clusters of texts. Therefore, a text is a combination of all the topics identified and defined by the weight associated with each topic (Shashank Kapadia, 2019).

Tagging

Text tagging refers to automatically or manually adding labels or comment to different parts of unstructured data during the process of preparing data for analysis (McKenzie, 2015). In topic modelling, abstract topics appear in a group of documents that correctly express the information present in them (Shashank Kapadia, 2019).

Topic modelling includes counting words and gathering comparable word patterns to derive topics inside unstructured information. Suppose you’re a product organization, and you need to understand what clients are saying about specific highlights of your item. Rather than going through stacks of feedback for several hours, trying to conclude which writings are discussing subjects of your interest, you could examine them with a topic modelling algorithm (Pascual, 2019). By identifying patterns like word recurrence and distance between words, a topic model groups input that is analogous and words and expressions that show up frequently. With this data, we can rapidly reason what each set of texts are discussing. However, it should be noted that this methodology is unsupervised, implying that no training is required.

Let’s consider an example. The New York Times are utilizing topic models for supporting their client–article proposal engines. Professionals in different fields are utilizing topic models in recruitment ventures where they mean to separate inert features of sets of expectations for a particular job and guide them to the right applicants (Shivam Bansal, 2016). They are being utilized to arrange huge datasets of messages, client audits, and user’s social media profiles.

Another common example of topic modelling is grouping an enormous number of paper articles that have a place with a similar classification or simply a cluster of reports that have a similar subject (Malik, n.d.). It is essential to consider here that it is incredibly hard to assess the performance of topic modelling since there are no correct answers. It relies on the client to discover comparable characteristics between the archives of one group and appoint it a suitable name or topic.

Several algorithms can be used to perform topic modelling like Latent Semantic Analysis, Probabilistic Latent Semantic Analysis, Pachinko Allocation Model, and Latent Dirichlet Allocation.

Latent Semantic Analysis

Latent Semantic Analysis (LSA) is perhaps the most widely used topic modelling technique. It depends on the distributional hypothesis, which states that it is possible to get the semantics of words by assessing the context of the appearance of words. According to this theory, the semantics of two words will be comparative in the event that they will, in general, happen in comparable contexts. 

All things considered, LSA calculates how frequently a word appears in a text document – and the entire corpus – and accepts that comparable documents will contain around the similar distribution of word frequencies for specific words. For this situation, syntactic data (for example, word order) and semantic data (for example, the assortment of implications of a word given) are disregarded, and each document is considered as a bag of words. 

The standard method for calculating word frequencies is tf-idf. This technique calculates frequencies by contemplating not just how frequently words occur in a given text but also considers how frequently those words appear together in the corpus of documents. Words with a higher recurrence in the full corpus are preferable to represent a document over words that are less frequent, paying little heed to how frequently they show up in a single document. Accordingly, tf-idf representations are far superior to those that only consider word frequencies at the document level. 

On the basis of tf-idf frequencies, we can make a document-term matrix that shows the tf-idf values associated with every term in a document under consideration. Rows for each document in the corpus and columns for each term considered are present in the Document-term matrix (Pascual, 2019). This is illustrated in the image below.

Singular Value Decomposition can be used to represent the document-term matrix as a product of 3 matrices (USV). The U matrix is called the Document topic matrix, and the V matrix is called the Term topic matrix (Pascual, 2019). The S matrix is a diagonal matrix as we use Linear Algebra here. The LSV algorithm will then take into consideration a single value that is taken from the main diagonal of the S matrix as a potential topic extracted from the documents.

In the event that we keep the biggest t values along with the first t columns in U and the first t rows of V, we can get the t more frequently occurring topics in the Document-term matrix (Pascual, 2019). The resultant matrix is called truncated SVD because it doesn’t store all the singular values taken from the original matrix and, to use it for LSA, we need to set the estimation of t as a hyperparameter.

The quality and performance of the topic assignment task for each document and the quality of the terms relegated to every topic can be evaluated through various methods by analyzing the vectors that constitute the U and V matrices.

Latent Dirichlet Allocation (LDA)

LDA is one of the most popular methods for topic modelling and is frequently used for the same. LDA is a generative probabilistic model that accepts every point as a blend over a hidden arrangement of words and each document as a combination of a set of topic probabilities (Shashank Kapadia, 2019). LDA assumes that the topics present in each document are used to create words on the basis of their probability distribution. For a given set of documents, the Latent Dirichlet Allocation algorithm performs backtracking, and hence it tries to identify the topics that can produce those documents (Shivam Bansal, 2016). The algorithm uses matrix factorization. A group of documents is represented in the form of a document-term matrix.

The image given below represents a set of N documents D1, D2,.., Dn and associated vocabulary size of M words W1, W2,…, Wn. A word’s frequency count Wj in Document Di is given by the value of i,j cell in the matrix (Shivam Bansal, 2016).

The LDA algorithm decomposes the document term matrix into lower-dimensional matrices M1 and M2, where M1 is a document topic matrix, and M2 is a topic term matrix (Shivam Bansal, 2016).

The main aim of LDA is to improve the topic word distribution and document topic distribution with the help of sampling techniques which improves the two matrices. These two matrices shown in the above diagram already give the topic word and document topic distribution. Each word “w” for each document, “d” is iterated over, and it attempts to adjust the current topic (Shivam Bansal, 2016). A new topic “k” is allocated to the word “w” with a likelihood P which is a result of 2 probabilities p1 and p2.

For each topic in the document, the algorithm calculates two probabilities p1 and p2.

p1 = p (topic t / document d), and it represents the word proportions in document d that have been currently assigned to topic d

p2 = p (word w / topic t), and it represents the proportion of allocations to a topic t among all the documents that come from the word w

Whenever a new topic is considered or encountered, the recent topic-word assignment is updated with a probability and the product of p1 and p2. The model believes that all other word-topic assignments are correct except the current word (Shivam Bansal, 2016). It is the probability that word w is generated by topic t. This may be the reason to adjust the current word’s topic with the new probability. After a number of repetitions, a constant state is reached in which the distribution of document topic and document term is acceptable. This is known as the convergence point of the LDA algorithm.

There are two parameters in the LDA algorithm. One is the Alpha parameter, and the other is the Beta parameter.

Alpha Parameter: It denotes the document topic density, also known as Dirichlet prior concentration (Shashank Kapadia, 2019). When the alpha is high, there are more topics in the document, which in turn results in a more precise topic distribution in each document.

Beta Parameter: It denotes the topic word density (Shashank Kapadia, 2019). When the beta is high, it is considered that more words make up the topic, and it results in a more precise word distribution in each topic.

The Implementation of LDA

The following steps are to be performed for implementing the LDA algorithm:

1. Getting the data: Data can be in any form like a csv file or a text file. 
2. Data pre-processing and Data cleaning: In this step, data is cleaned by removing unnecessary data. It involves removing punctuations, stop-words and normalizing the corpus.
3. Exploratory Data Analysis: We can create a word cloud to visualize the most common words. Other pre-processing can also be done.  
4. Preparing the data for LDA analysis: Textual data is converted into a format that can be fed as input to the LDA model. It involves tokenizing, and the tokenized objects are converted into a corpus and dictionary.
5. Training the LDA model: In order to keep things simple, all the parameters must be set to default except the number of topics and run the model.
6. Analysis of the model results: After the model has been trained, we can visualize the topics for interpretation. 

If you’d want to dive deeper into working with Python, especially for data science, upGrad brings you the Executive PGP in Data Science. This program is designed for mid-level IT professionals, software engineers looking to explore Data Science, non-tech analysts, early career professionals, etc. Our structured curriculum and extensive support ensure our students reach their full potential without difficulties.  

Example

The following figures illustrate an example of LDA implementation based on the publications included in the NeurIPS conference (Shashank Kapadia, 2019).

Step 1:

Step 2:

Step 3: 

Step 4: