zenilib: pp::Tokenizer Class Reference

zenilib 0.5.3.0

- Main Page
- Modules
- Namespaces
- Classes
- Files

- Class List
- Class Index
- Class Hierarchy
- Class Members

All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Properties Friends Macros Groups Pages
Classes | Public Member Functions | Static Public Attributes | List of all members
pp::Tokenizer Class Reference

`#include <Tokenizer.h>`

Inherits pp::Lexer.

## Classes

struct | Context

## Public Member Functions

| Tokenizer (Diagnostics *diagnostics)
| ~Tokenizer ()
bool | init (size_t count, const char *const string[], const int length[])
virtual void | lex (Token *token)
void | setFileNumber (int file)
void | setLineNumber (int line)
Public Member Functions inherited from pp::Lexer
virtual | ~Lexer ()

## Static Public Attributes

static const std::size_t | kMaxTokenLength = 256

## Detailed Description

Definition at line 19 of file Tokenizer.h.

## Constructor & Destructor Documentation

pp::Tokenizer::Tokenizer | ( | Diagnostics * | diagnostics | ) |

Definition at line 2297 of file Tokenizer.cpp.

References pp::Tokenizer::Context::diagnostics.

pp::Tokenizer::~Tokenizer | ( |  | ) |

Definition at line 2302 of file Tokenizer.cpp.

## Member Function Documentation

bool pp::Tokenizer::init | ( | size_t | count,
|  | const char *const | string[],
|  | const int | length[]
| ) |  |

Definition at line 2307 of file Tokenizer.cpp.

References pp::Tokenizer::Context::input.

void pp::Tokenizer::lex | ( | Token * | token | ) |

virtual

Implements pp::Lexer.

Definition at line 2327 of file Tokenizer.cpp.

References pp::Tokenizer::Context::diagnostics, pp::Token::flags, kMaxTokenLength, pp::Tokenizer::Context::leadingSpace, pp::Tokenizer::Context::lineStart, pp::Token::location, pplex(), pp::Diagnostics::report(), pp::Token::setAtStartOfLine(), pp::Token::setHasLeadingSpace(), pp::Token::text, pp::Diagnostics::TOKEN_TOO_LONG, and pp::Token::type.

Referenced by pp::DirectiveParser::lex().

void pp::Tokenizer::setFileNumber | ( | int | file | ) |

Definition at line 2315 of file Tokenizer.cpp.

References ppset_column().

void pp::Tokenizer::setLineNumber | ( | int | line | ) |

Definition at line 2322 of file Tokenizer.cpp.

References ppset_lineno().

## Member Data Documentation

const size_t pp::Tokenizer::kMaxTokenLength = 256

static

Definition at line 35 of file Tokenizer.h.

Referenced by lex().

The documentation for this class was generated from the following files:

- zenilib/jni/external/angle/src/compiler/preprocessor/Tokenizer.h
- zenilib/jni/external/angle/src/compiler/preprocessor/Tokenizer.cpp