NLP Tutorial: Learn Natural Language from Scratch (2026)
Natural language processing lets computers understand, interpret, and generate human language. I fell in love with NLP when I built a simple sentiment analyzer that could tell whether movie reviews were positive or negative. It was crude by today's standards but it worked. By 2026, NLP has been transformed by transformers and large language models, but the foundational skills of text processing, tokenization, and linguistic analysis remain essential. Understanding these fundamentals makes you a better practitioner even when using modern LLMs.
Text Preprocessing and Cleaning
Text data is messy. It contains punctuation, capitalization, URLs, emojis, and formatting that confuses statistical models. Preprocessing normalizes text into a clean, consistent format. Common steps include converting to lowercase, removing punctuation, stripping whitespace, expanding contractions, and handling special characters. Regular expressions are your primary tool for text cleaning.
I learned the hard way that preprocessing decisions have outsized impact on downstream tasks. Removing punctuation helps bag-of-words models but can hurt sentiment analysis where 'not good' and 'not good!' have different intensities. Always test preprocessing choices on your specific task rather than blindly applying a standard pipeline.
import re
def clean_text(text):
text = text.lower()
text = re.sub(r'<[^>]+>', '', text) # remove HTML
text = re.sub(r'\s+', ' ', text).strip()
return text
sample = 'Hello!!! Check out my site http://example.com'
print(clean_text(sample))Tokenization and Word Segmentation
Tokenization splits text into words, phrases, or symbols called tokens. NLTK provides several tokenizers: word_tokenize for English, sent_tokenize for sentences, and TweetTokenizer for social media text. Languages like Chinese and Japanese are challenging because words are not separated by spaces. SpaCy offers more sophisticated tokenization with linguistic features attached to each token.
I once built a multilingual NLP pipeline and discovered that NLTK's default tokenizer fails on text with contractions like 'don't' or possessives like 'John's'. The TreebankWordTokenizer handles these better. For production systems, I use SpaCy's tokenizer which provides lemmas, part-of-speech tags, and dependency parse as part of the token object.
from nltk.tokenize import word_tokenize, sent_tokenize
text = "Dr. Smith isn't coming to the U.S. today."
words = word_tokenize(text)
sentences = sent_tokenize(text)
print(f'Words: {words}')Part-of-Speech Tagging and Lemmatization
Part-of-speech tagging assigns grammatical categories like noun, verb, adjective to each word. NLTK uses the Penn Treebank tagset with 36 tags. POS tagging is essential for lemmatization, which reduces words to their base dictionary form. 'Running', 'ran', and 'runs' all reduce to 'run'. Lemmatization is more sophisticated than stemming because it considers the word's meaning and context.
The combination of POS tagging and lemmatization dramatically reduces the vocabulary size while preserving meaning. I reduced a 50,000-word corpus to 15,000 unique lemmas, which made my topic models both faster and more interpretable. WordNetLemmatizer requires the POS tag to work correctly, always pass the tag.
from nltk import pos_tag
from nltk.stem import WordNetLemmatizer
from nltk.corpus import wordnet
tagged = pos_tag(word_tokenize('The dogs are running quickly'))
lemmatizer = WordNetLemmatizer()
lemmas = [lemmatizer.lemmatize(word, pos='v') for word, tag in tagged]Named Entity Recognition and Information Extraction
Named Entity Recognition identifies and classifies named entities in text: people, organizations, locations, dates, monetary values, and more. NER is the foundation of information extraction systems. NLTK's ne_chunk uses a pre-trained classifier. Modern approaches using SpaCy or Hugging Face transformers achieve higher accuracy with contextual embeddings.
I built a news aggregator that extracts entities from articles and links them to Wikidata entries. The NER pipeline identified companies, people, and locations, then a resolution step connected mentions of 'Apple' the company versus 'apple' the fruit. SpaCy's entity linking integration with knowledge bases made this much more accurate than NLTK's basic NER.
from nltk import ne_chunk
from nltk.tag import pos_tag
sentence = 'Barack Obama was born in Hawaii and worked at Harvard.'
tagged = pos_tag(word_tokenize(sentence))
entities = ne_chunk(tagged)
for entity in entities:
if hasattr(entity, 'label'):
print(f'{entity.label()}: {" ".join(c[0] for c in entity)}')Sentiment Analysis and Opinion Mining
Sentiment analysis determines whether text expresses positive, negative, or neutral sentiment. NLTK's VADER sentiment analyzer is specifically tuned for social media and works well without training. For custom domains, you can train classifiers using bag-of-words features with NLTK's NaiveBayesClassifier. Modern approaches use transformer models fine-tuned on sentiment datasets.
I used VADER to analyze customer feedback for a restaurant chain. The beauty of VADER is that it handles emoticons, slang, and capitalization intensity like 'GOOD!!!' correctly out of the box. For deeper analysis, I built a custom classifier using NLTK's movie_reviews dataset as a starting point, then fine-tuned on restaurant-specific reviews. The domain-specific model significantly outperformed the generic one.
from nltk.sentiment import SentimentIntensityAnalyzer
sia = SentimentIntensityAnalyzer()
review = 'The food was amazing but the service was terrible!'
scores = sia.polarity_scores(review)
print(f'Compound: {scores["compound"]:.2f}, Negative: {scores["neg"]:.2f}, Positive: {scores["pos"]:.2f}')Topic Modeling with Latent Dirichlet Allocation
Topic modeling discovers latent themes in large text collections without supervision. Latent Dirichlet Allocation assumes each document is a mixture of topics and each topic is a distribution over words. NLTK provides an LDA implementation, but Gensim's implementation is more scalable for large corpora. The number of topics K must be specified in advance.
I ran LDA on 100,000 customer support tickets to identify common issues. The topics revealed clusters: billing problems, technical bugs, feature requests, and account management. The interpretable topics enabled the support team to route tickets automatically and identify systemic issues. The key to good topic models is preprocessing: removing stop words, filtering rare words, and using bigrams to capture phrases like 'credit card' as single tokens.
from nltk.corpus import stopwords
from gensim import corpora, models
stop_words = set(stopwords.words('english'))
texts = [[w for w in doc if w not in stop_words] for doc in documents]
dictionary = corpora.Dictionary(texts)
corpus = [dictionary.doc2bow(text) for text in texts]
lda = models.LdaModel(corpus, num_topics=5, id2word=dictionary)
for topic in lda.print_topics():
print(topic)Frequently Asked Questions
What is the difference between NLTK and SpaCy?
NLTK is educational and comprehensive with many algorithms and corpora. SpaCy is designed for production use with faster performance and better default models. NLTK is great for learning, SpaCy is better for building applications.
Do I still need traditional NLP with modern LLMs?
Yes. LLMs are powerful but expensive and overkill for many tasks. Traditional NLP techniques like regex, POS tagging, and NER are faster, cheaper, and more reliable for well-defined tasks. Use traditional NLP for preprocessing and simple tasks, LLMs for understanding and generation.
How do I handle multiple languages in NLP?
NLTK supports tokenization and stop words for many languages. For multilingual NLP, SpaCy offers trained pipelines for 20+ languages. Universal dependencies provide cross-lingual annotation standards. For LLMs, multilingual models like GPT-4o work across many languages.
What is the difference between stemming and lemmatization?
Stemming chops off affixes to get the root form (running -> runn). Lemmatization uses vocabulary and morphological analysis to return the dictionary form (running -> run). Lemmatization is more accurate but slower. Use stemming for search indexing and lemmatization for analysis.
Originally published on Ayodhyyya. Last updated June 1, 2026.