A Tokenizer is a TokenStream whose input is a Reader. More...

#include <Tokenizer.h>

Inheritance diagram for Lucene::Tokenizer:

Public Member Functions
virtual	~Tokenizer ()
virtual String	getClassName ()
boost::shared_ptr< Tokenizer >	shared_from_this ()
virtual void	close ()
	By default, closes the input Reader.
virtual int32_t	correctOffset (int32_t currentOff)
	Return the corrected offset. If `input` is a `CharStream` subclass this method calls `CharStream#correctOffset`, else returns currentOff.
virtual void	reset (const ReaderPtr &input)
	Reset the tokenizer to a new reader. Typically, an analyzer (in its reusableTokenStream method) will use this to re-use a previously created tokenizer.
virtual void	reset ()
	Resets this stream to the beginning. This is an optional operation, so subclasses may or may not implement this method. `reset()` is not needed for the standard indexing process. However, if the tokens of a TokenStream are intended to be consumed more than once, it is necessary to implement `reset()`. Note that if your TokenStream caches tokens and feeds them back again after a reset, it is imperative that you clone the tokens when you store them away (on the first pass) as well as when you return them (on future passes after `reset()`).
Public Member Functions inherited from Lucene::TokenStream
virtual	~TokenStream ()
boost::shared_ptr< TokenStream >	shared_from_this ()
virtual bool	incrementToken ()=0
	Consumers (ie., `IndexWriter`) use this method to advance the stream to the next token. Implementing classes must implement this method and update the appropriate `Attribute`s with the attributes of the next token.
virtual void	end ()
	This method is called by the consumer after the last token has been consumed, after `incrementToken()` returned false (using the new TokenStream API). Streams implementing the old API should upgrade to use this feature.
Public Member Functions inherited from Lucene::AttributeSource
	AttributeSource ()
	An AttributeSource using the default attribute factory `DefaultAttributeFactory`.
	AttributeSource (const AttributeSourcePtr &input)
	An AttributeSource that uses the same attributes as the supplied one.
	AttributeSource (const AttributeFactoryPtr &factory)
	An AttributeSource using the supplied `AttributeFactory` for creating new `Attribute` instances.
virtual	~AttributeSource ()
boost::shared_ptr< AttributeSource >	shared_from_this ()
AttributeFactoryPtr	getAttributeFactory ()
	returns the used AttributeFactory.
template<class ATTR>
boost::shared_ptr< ATTR >	addAttribute ()
	This method first checks if an instance of that class is already in this AttributeSource and returns it. Otherwise a new instance is created, added to this AttributeSource and returned.
void	addAttribute (const String &className, const AttributePtr &attrImpl)
	Adds a custom Attribute instance.
bool	hasAttributes ()
	Returns true if this AttributeSource has any attributes.
template<class ATTR>
bool	hasAttribute ()
	Returns true, if this AttributeSource contains the passed-in Attribute.
template<class ATTR>
boost::shared_ptr< ATTR >	getAttribute ()
	Returns the instance of the passed in Attribute contained in this AttributeSource.
void	clearAttributes ()
	Resets all Attributes in this AttributeSource by calling `AttributeImpl#clear()` on each Attribute implementation.
AttributeSourceStatePtr	captureState ()
	Captures the state of all Attributes. The return value can be passed to `restoreState` to restore the state of this or another AttributeSource.
void	restoreState (const AttributeSourceStatePtr &state)
	Restores this state by copying the values of all attribute implementations that this state contains into the attributes implementations of the targetStream. The targetStream must contain a corresponding instance for each argument contained in this state (eg. it is not possible to restore the state of an AttributeSource containing a TermAttribute into a AttributeSource using a Token instance as implementation).
virtual int32_t	hashCode ()
	Return hash code for this object.
virtual bool	equals (const LuceneObjectPtr &other)
	Return whether two objects are equal.
virtual String	toString ()
	Returns a string representation of the object.
AttributeSourcePtr	cloneAttributes ()
	Performs a clone of all `AttributeImpl` instances returned in a new AttributeSource instance. This method can be used to eg. create another TokenStream with exactly the same attributes (using `AttributeSource(AttributeSource)`).
Collection< AttributePtr >	getAttributes ()
	Return a vector of attributes based on currentState.
Public Member Functions inherited from Lucene::LuceneObject
virtual	~LuceneObject ()
virtual void	initialize ()
	Called directly after instantiation to create objects that depend on this object being fully constructed.
virtual LuceneObjectPtr	clone (const LuceneObjectPtr &other=LuceneObjectPtr())
	Return clone of this object.
virtual int32_t	compareTo (const LuceneObjectPtr &other)
	Compare two objects.
Public Member Functions inherited from Lucene::LuceneSync
virtual	~LuceneSync ()
virtual SynchronizePtr	getSync ()
	Return this object synchronize lock.
virtual LuceneSignalPtr	getSignal ()
	Return this object signal.
virtual void	lock (int32_t timeout=0)
	Lock this object using an optional timeout.
virtual void	unlock ()
	Unlock this object.
virtual bool	holdsLock ()
	Returns true if this object is currently locked by current thread.
virtual void	wait (int32_t timeout=0)
	Wait for signal using an optional timeout.
virtual void	notifyAll ()
	Notify all threads waiting for signal.

Static Public Member Functions
static String	_getClassName ()
Static Public Member Functions inherited from Lucene::TokenStream
static String	_getClassName ()
Static Public Member Functions inherited from Lucene::AttributeSource
static String	_getClassName ()

Protected Member Functions
	Tokenizer ()
	Construct a tokenizer with null input.
	Tokenizer (const ReaderPtr &input)
	Construct a token stream processing the given input.
	Tokenizer (const AttributeFactoryPtr &factory)
	Construct a tokenizer with null input using the given AttributeFactory.
	Tokenizer (const AttributeFactoryPtr &factory, const ReaderPtr &input)
	Construct a token stream processing the given input using the given AttributeFactory.
	Tokenizer (const AttributeSourcePtr &source)
	Construct a token stream processing the given input using the given AttributeSource.
	Tokenizer (const AttributeSourcePtr &source, const ReaderPtr &input)
	Construct a token stream processing the given input using the given AttributeSource.
Protected Member Functions inherited from Lucene::TokenStream
	TokenStream ()
	A TokenStream using the default attribute factory.
	TokenStream (const AttributeSourcePtr &input)
	A TokenStream that uses the same attributes as the supplied one.
	TokenStream (const AttributeFactoryPtr &factory)
	A TokenStream using the supplied AttributeFactory for creating new `Attribute` instances.
Protected Member Functions inherited from Lucene::AttributeSource
AttributePtr	getAttribute (const String &className)
	The caller must pass in a className value. This method checks if an instance of that class is already in this AttributeSource and returns it.
bool	hasAttribute (const String &className)
	Returns true, if this AttributeSource contains the passed-in Attribute.
void	computeCurrentState ()
Protected Member Functions inherited from Lucene::LuceneObject
	LuceneObject ()

Protected Attributes
ReaderPtr	input
	The text source for this Tokenizer.
CharStreamPtr	charStream
Protected Attributes inherited from Lucene::AttributeSource
AttributeFactoryPtr	factory
MapStringAttribute	attributes
AttributeSourceStatePtr	currentState
Protected Attributes inherited from Lucene::LuceneSync
SynchronizePtr	objectLock
LuceneSignalPtr	objectSignal

Detailed Description

A Tokenizer is a TokenStream whose input is a Reader.

This is an abstract class; subclasses must override incrementToken()

Note: Subclasses overriding incrementToken() must call AttributeSource#clearAttributes() before setting attributes.

Constructor & Destructor Documentation

◆ Tokenizer() [1/6]

Lucene::Tokenizer::Tokenizer ( )

protected

Construct a tokenizer with null input.

Referenced by _getClassName().

◆ Tokenizer() [2/6]

Lucene::Tokenizer::Tokenizer ( const ReaderPtr & input )

protected

Construct a token stream processing the given input.

References input.

◆ Tokenizer() [3/6]

Lucene::Tokenizer::Tokenizer ( const AttributeFactoryPtr & factory )

protected

Construct a tokenizer with null input using the given AttributeFactory.

References Lucene::AttributeSource::factory.

◆ Tokenizer() [4/6]

Lucene::Tokenizer::Tokenizer	(	const AttributeFactoryPtr &	factory,
		const ReaderPtr &	input )

protected

Construct a token stream processing the given input using the given AttributeFactory.

References Lucene::AttributeSource::factory, and input.

◆ Tokenizer() [5/6]

Lucene::Tokenizer::Tokenizer ( const AttributeSourcePtr & source )

protected

Construct a token stream processing the given input using the given AttributeSource.

◆ Tokenizer() [6/6]

Lucene::Tokenizer::Tokenizer	(	const AttributeSourcePtr &	source,
		const ReaderPtr &	input )

protected

Construct a token stream processing the given input using the given AttributeSource.

References input.

◆ ~Tokenizer()

virtual Lucene::Tokenizer::~Tokenizer ( )

virtual

Member Function Documentation

◆ _getClassName()

String Lucene::Tokenizer::_getClassName ( )

inlinestatic

References Tokenizer().

◆ close()

virtual void Lucene::Tokenizer::close ( )

virtual

By default, closes the input Reader.

Reimplemented from Lucene::TokenStream.

◆ correctOffset()

virtual int32_t Lucene::Tokenizer::correctOffset ( int32_t currentOff )

virtual

Return the corrected offset. If input is a CharStream subclass this method calls CharStream#correctOffset, else returns currentOff.

Parameters

currentOff offset as seen in the output

Returns: corrected offset based on the input

See also: CharStream::correctOffset

References Lucene::TokenStream::reset().

◆ getClassName()

virtual String Lucene::Tokenizer::getClassName ( )

inlinevirtual

Reimplemented from Lucene::TokenStream.

Reimplemented in Lucene::CharTokenizer, Lucene::KeywordTokenizer, Lucene::LetterTokenizer, Lucene::LowerCaseTokenizer, Lucene::StandardTokenizer, and Lucene::WhitespaceTokenizer.

◆ reset() [1/2]

virtual void Lucene::TokenStream::reset ( )

virtual

Resets this stream to the beginning. This is an optional operation, so subclasses may or may not implement this method. reset() is not needed for the standard indexing process. However, if the tokens of a TokenStream are intended to be consumed more than once, it is necessary to implement reset(). Note that if your TokenStream caches tokens and feeds them back again after a reset, it is imperative that you clone the tokens when you store them away (on the first pass) as well as when you return them (on future passes after reset()).

Reimplemented from Lucene::TokenStream.

◆ reset() [2/2]

virtual void Lucene::Tokenizer::reset ( const ReaderPtr & input )

virtual

Reset the tokenizer to a new reader. Typically, an analyzer (in its reusableTokenStream method) will use this to re-use a previously created tokenizer.

Reimplemented in Lucene::CharTokenizer, and Lucene::StandardTokenizer.

References input.

◆ shared_from_this()

boost::shared_ptr< Tokenizer > Lucene::Tokenizer::shared_from_this ( )

inline

Field Documentation

◆ charStream

CharStreamPtr Lucene::Tokenizer::charStream

protected

◆ input

ReaderPtr Lucene::Tokenizer::input

protected

The text source for this Tokenizer.

Referenced by Lucene::CharTokenizer::CharTokenizer(), Lucene::CharTokenizer::CharTokenizer(), Lucene::CharTokenizer::CharTokenizer(), Lucene::StandardTokenizer::init(), Lucene::KeywordTokenizer::KeywordTokenizer(), Lucene::KeywordTokenizer::KeywordTokenizer(), Lucene::KeywordTokenizer::KeywordTokenizer(), Lucene::KeywordTokenizer::KeywordTokenizer(), Lucene::LetterTokenizer::LetterTokenizer(), Lucene::LetterTokenizer::LetterTokenizer(), Lucene::LetterTokenizer::LetterTokenizer(), Lucene::LowerCaseTokenizer::LowerCaseTokenizer(), Lucene::LowerCaseTokenizer::LowerCaseTokenizer(), Lucene::LowerCaseTokenizer::LowerCaseTokenizer(), Lucene::CharTokenizer::reset(), Lucene::StandardTokenizer::reset(), reset(), Lucene::StandardTokenizer::StandardTokenizer(), Lucene::StandardTokenizer::StandardTokenizer(), Lucene::StandardTokenizer::StandardTokenizer(), Tokenizer(), Tokenizer(), Tokenizer(), Lucene::WhitespaceTokenizer::WhitespaceTokenizer(), Lucene::WhitespaceTokenizer::WhitespaceTokenizer(), and Lucene::WhitespaceTokenizer::WhitespaceTokenizer().

The documentation for this class was generated from the following file:

Tokenizer.h

Public Member Functions

Static Public Member Functions

Protected Member Functions

Protected Attributes

Detailed Description

Constructor & Destructor Documentation

◆ Tokenizer() [1/6]

◆ Tokenizer() [2/6]

◆ Tokenizer() [3/6]

◆ Tokenizer() [4/6]

◆ Tokenizer() [5/6]

◆ Tokenizer() [6/6]

◆ ~Tokenizer()

Member Function Documentation

◆ _getClassName()

◆ close()

◆ correctOffset()

◆ getClassName()

◆ reset() [1/2]

◆ reset() [2/2]

◆ shared_from_this()

Field Documentation

◆ charStream

◆ input