Category: Programming
-
Understanding Corpus Tools: An Introduction
A trip through the linguistic isn’t complete without stumbling upon the term “corpus.” As we delve deeper into language studies and natural language processing, the significance of a corpus and the tools to manage them become increasingly evident. So, let’s break down the concept of a corpus tool, its utilities, and why it’s an essential…
-
Apache OpenNLP – Tokenization
Tokenization is a process of segmenting strings into smaller parts called tokens(say sub-strings). Usually, these tokens are words, numbers, or punctuation marks. There’re three types of tokenizers available in OpenNLP. TokenizerME – A maximum entropy tokenizer, detects token boundaries based on probability model WhitespaceTokenizer – A whitespace tokenizer, non whitespace sequences are identified as tokens…
-
NLP – Natural language processing
From voice-activated assistants like Siri and Alexa to chatbots on customer service websites, there’s a hidden technology working behind the scenes, enabling machines to understand and generate human language. This technology, known as Natural Language Processing (NLP), has revolutionized the way we interact with machines. Dive in with us as we explore the realm of…
-
English Lemmatization: Simplifying Words in NLP
Language, in all its complexity, offers multiple ways to express similar concepts. We have “running”, “ran”, and “runner” — all derivatives of the simple verb “run”. In Natural Language Processing (NLP), navigating these variations efficiently is crucial. Enter lemmatization, a technique that helps simplify and standardize words. Let’s dive into the concept of English lemmatization…