pp::Tokenizer Class Reference

#include <Tokenizer.h>

Inherits pp::Lexer.

Classes
struct	Context

Public Member Functions
	Tokenizer (Diagnostics *diagnostics)
	~Tokenizer ()
bool	init (size_t count, const char *const string[], const int length[])
virtual void	lex (Token *token)
void	setFileNumber (int file)
void	setLineNumber (int line)
Public Member Functions inherited from pp::Lexer
virtual	~Lexer ()

Static Public Attributes
static const std::size_t	kMaxTokenLength = 256

Detailed Description

Definition at line 19 of file Tokenizer.h.

Constructor & Destructor Documentation

pp::Tokenizer::Tokenizer

(

Diagnostics *

diagnostics

)

Definition at line 2297 of file Tokenizer.cpp.

References pp::Tokenizer::Context::diagnostics.

pp::Tokenizer::~Tokenizer

(

)

Definition at line 2302 of file Tokenizer.cpp.

Member Function Documentation

bool pp::Tokenizer::init	(	size_t	count,
		const char *const	string[],
		const int	length[]
	)

Definition at line 2307 of file Tokenizer.cpp.

References pp::Tokenizer::Context::input.

void pp::Tokenizer::lex ( Token *  token )

virtual

Implements pp::Lexer.

Definition at line 2327 of file Tokenizer.cpp.

References pp::Tokenizer::Context::diagnostics, pp::Token::flags, kMaxTokenLength, pp::Tokenizer::Context::leadingSpace, pp::Tokenizer::Context::lineStart, pp::Token::location, pplex(), pp::Diagnostics::report(), pp::Token::setAtStartOfLine(), pp::Token::setHasLeadingSpace(), pp::Token::text, pp::Diagnostics::TOKEN_TOO_LONG, and pp::Token::type.

Referenced by pp::DirectiveParser::lex().

void pp::Tokenizer::setFileNumber

(

int

file

)

Definition at line 2315 of file Tokenizer.cpp.

References ppset_column().

void pp::Tokenizer::setLineNumber

(

int

line

)

Definition at line 2322 of file Tokenizer.cpp.

References ppset_lineno().

Member Data Documentation

const size_t pp::Tokenizer::kMaxTokenLength = 256

static

Definition at line 35 of file Tokenizer.h.

Referenced by lex().

---

The documentation for this class was generated from the following files:

• zenilib/jni/external/angle/src/compiler/preprocessor/Tokenizer.h
• zenilib/jni/external/angle/src/compiler/preprocessor/Tokenizer.cpp