CVE-2021-43854

NLTK < 3.6.5: ReDoS in PunktSentenceTokenizer, sent_tokenize, word_tokenize

• CVSS 7.5
• CWE-400 Uncontrolled Resource Consumption
• Input Validation and Sanitization
• Local

NLTK (Natural Language Toolkit) is a suite of open source Python modules, data sets, and tutorials supporting research and development in Natural Language Processing. Versions prior to 3.6.5 are vulnerable to regular expression denial of service (ReDoS) attacks. The vulnerability is present in PunktSentenceTokenizer, sent_tokenize and word_tokenize. Any users of this class, or these two functions, are vulnerable to the ReDoS attack. In short, a specifically crafted long input to any of these vulnerable functions will cause them to take a significant amount of execution time. If your program relies on any of the vulnerable functions for tokenizing unpredictable user input, then we would strongly recommend upgrading to a version of NLTK without the vulnerability. For users unable to upgrade the execution time can be bounded by limiting the maximum length of an input to any of the vulnerable functions. Our recommendation is to implement such a limit.

CVE

CVE-2021-43854

CWE

CWE-400 Uncontrolled Resource Consumption

CVSS base score

7.5

Published

2021-12-23

OWASP

A06 Vulnerable and Outdated Components

Orthogonal defect classification

Timing/Serialization

Code defect classification

Timing Issues

Category

Input Validation and Sanitization

Subcategory

Insecure Parsing or Deserialization

Accessibility scope

Local

Impact

Denial of Service (DoS)

Fixed by upgrading

Yes

Solution

Update NLTK to version 3.6.5 or higher.

import nltk

text_input = "A" * 1000000

sentences = nltk.tokenize.sent_tokenize(text_input)
words = nltk.tokenize.word_tokenize(text_input)

print(sentences)
print(words)

import nltk

def safe_tokenize(text, max_length=1000):
    if len(text) > max_length:
        raise ValueError(f"Input text exceeds the maximum allowed length of {max_length}.")
    return nltk.tokenize.sent_tokenize(text), nltk.tokenize.word_tokenize(text)

try:
    text_input = "A" * 1000000
    sentences, words = safe_tokenize(text_input)
    print(sentences)
    print(words)
except ValueError as e:
    print(e)

@misc{vaitp:cve202143854,
  title        = {{NLTK < 3.6.5: ReDoS in PunktSentenceTokenizer, sent_tokenize, word_tokenize}},
  author       = {Bogaerts, Fr\'ed\'eric and Ivaki, Naghmeh and Fonseca, Jos\'e},
  year         = {2021},
  note         = {VAITP Python Vulnerability Dataset, entry CVE-2021-43854},
  howpublished = {\url{https://netpack.pt/vaitp/vulnerability/CVE-2021-43854/}}
}