NLP: Text Processing In Data Science Projects
NLP: Text Processing In Data Science Projects

Learn The Data Science Techniques To Process Text To Use For NLP Projects In Python

Once we have gathered the text, the next stage is about cleaning and consolidating the text. It is important to ensure the text is standardised, the noise is removed so that efficient analysis can be performed on the text to derive meaningful insights.

It’s important to note that the cleaning and processing of text is highly dependent on the nature of the NLP project. As an instance for your project, numbers might be important.

This article aims to explain the steps we can perform to clean the text for NLP projects.

Photo by NordWood Themes on Unsplash

Article Aim

I will explain following key techniques:

Convert Text To Lowercase Tokenise Paragraphs To Sentences Tokenise Sentences To Words Remove Numbers Remove Punctuation Remove Stop words Remove Whitespaces

I will demonstrate how we can achieve the goal by using the NLTK library in Python and the regular expressions. We can install NLTK library using:

pip install nltk

The first task is to normalise the paragraphs of text.

1. Convert Text To Lowercase

The key concept here is to reduce the number of words, in particular if they are same. We can change the casing of the words to ensure every word is in lowercase.

As an instance, “Article” and “article” can be represented as “article”. We can use the lowercase() function of Python to change the casing of the text.

text = 'This is an NLP article of FinTechExplained' lower_case_text = lowercase(text)

print(lower_case_text) #This will print:

#this is an nlp article of fintechexplained

The text we extract from the sources such as documents, are usually represented as groups of sentences (paragraphs).

The next task is to break the paragraphs into sentences.

2. Tokenise Paragraphs To Sentences

I highly recommend the NLTK library in Python to perform tokenisation. We can use PunktSentenceTokenize. It is a pre-trained model of the NLTK library that can perform sentence-level tokenising by determining punctuation and character marking the end of sentence.

import nltk

from nltk.tokenize import sent_tokenize text = 'FinTechExplained aims to explain how text processing works. Once we have gathered the text, the next stage is about cleaning and consolidating the text. It is important to ensure the text is standardised and the noise is removed so that efficient analysis can be performed on the text to derive meaningful insights.' list = sent_tokenize(text)

print(list) #----output---- [

'FinTechExplained aims to explain how text processing works.',

'Once we have gathered the text, the next stage is about cleaning and consolidating the text.',

'It is important to ensure the text is standardised and the noise is removed so that efficient analysis can be performed on the text to derive meaningful insights.'

]

We can see from the code snippet above that the paragraph has been tokenised into sentences.

NLTK supports punctuation and sentence endings for 17 European languages.

Once we have a list of sentences, we need to break the sentences into words.

3. Tokenise Sentences To Words

We can use the TreebankWordTokenizer class of the NLTK library in Python to tokenise the sentences into words.

from nltk.tokenize import TreebankWordTokenizer

tokenizer = TreebankWordTokenizer() text = 'FinTechExplained aims to explain how text processing works. Once we have gathered the text, the next stage is about cleaning and consolidating the text. It is important to ensure the text is standardised and the noise is removed so that efficient analysis can be performed on the text to derive meaningful insights.' print(tokenizer.tokenize(text))

#This will tokenise the sentences into words 'FinTechExplained', 'aims', 'to', 'explain', 'how', 'text', 'processing', 'works', '.', 'Once', 'we', 'have', 'gathered', 'the', 'text', ',' 'the', 'next', 'stage', 'is', 'about', 'cleaning', 'and', 'consolidating', 'the', 'text', '.', 'It', 'is', 'important', 'to', 'ensure', 'the', 'text', 'is', 'standardised', 'and', 'the', 'noise', 'is', 'removed', 'so', 'that', 'efficient', 'analysis', 'can', 'be', 'performed', 'on', 'the', 'text', 'to', 'derive', 'meaningful', 'insights', '.'

Photo by Clément H on Unsplash

If you want to read on how to gather the text from various sources for your NLP project then read:

The next task is to remove the numbers of the text

4. Remove Numbers

This is highly dependent on the project. One of the common tasks is to remove the numbers from the text as numbers are not usually important to text analytics.

We can use the Regular Expression to achieve the goal:

import re

result = re.sub(r'\d+', '', '909FinTechExplained9876')

print(result) # 'FinTechExplained'

Next remove all of the punctuation

5. Remove Punctuation

We also need to remove the punctuation from the text.

import string punctuation = string.punctuation

words = ['You','Are','Reading','FinTechExplained', '!', 'NLP', '.'] clean_words = [w for w in words if w not in punctuation] #it will return

clean_words = ['You','Are','Reading','FinTechExplained', 'NLP']

We can then do a “ “.join(clean_words) to return a clean sentence (if required).

Remove the noise that stop words are adding

6. Remove Stop words

There are many stop words in English as an instance “a”, “an”, “the”, “and”, “but”, “if”, “or”, “because” are some of the common English stop words.

We can remove the stop words by using the NLTK library:

from nltk.corpus import stopwords text = 'FinTechExplained is an important publication'

words = nltk.word_tokenize(text)

stopwords = stopwords.words('english') clean = [w for w in words if w not in stopwords] print(clean)

#This will return an array where stopwords have been removed.

#'FinTechExplained' , 'important', 'publication'

Finally remove the whitespaces

7. Remove Whitespaces

Lastly, I want to demonstrate how we can remove whitespaces such as space, tab, carriage return, line feeds.

We can use the split() along with join() function of the Python programming language. It returns a list of the words in the string.

’’.join(FinTechExplained Is A Publication.

This is about NLP’.split()) #This will return

'FinTechExplained Is A Publication. This is about NLP'

Photo by John Schnobrich on Unsplash

Summary

This article explained how we can clean the text once we have gathered the text for our NLP project.

It explained following key techniques:

Convert Text To Lowercase Tokenise Paragraphs To Sentences Tokenise Sentences To Words Remove Numbers Remove Punctuation Remove Stop words Remove Whitespaces

Hope it helps.
['Farhad Malik']
['NLP', 'Technology', 'Programming', 'Fintech', 'Data Science']