NatML
Search…
ITokenizer
interface NatSuite.MLX.Tokenizers.ITokenizer
A tokenizer is responsible for pre-processing plain text for input to natural language processing models. All tokenizers in NatML implement this interface.
This interface is part of the NatMLX extension library.

Tokenizing Text

1
/// <summary>
2
/// Tokenize a piece of text into tokens.
3
/// </summary>
4
/// <param name="text">Input text.</param>
5
/// <returns>Array of tokens.</returns>
6
string[] Tokenize (string text);
Copied!
This method tokenizes an input text into tokens that are recognizable by an NLP model.
Last modified 2mo ago
Copy link