GncDummyTokenizer Class Reference

Inheritance diagram for GncDummyTokenizer:

Public Member Functions
	GncDummyTokenizer (const GncDummyTokenizer &)=default
GncDummyTokenizer &	operator= (const GncDummyTokenizer &)=default
	GncDummyTokenizer (GncDummyTokenizer &&)=default
GncDummyTokenizer &	operator= (GncDummyTokenizer &&)=default
int	tokenize () override
Public Member Functions inherited from GncTokenizer
	GncTokenizer (const GncTokenizer &)=default
GncTokenizer &	operator= (const GncTokenizer &)=default
	GncTokenizer (GncTokenizer &&)=default
GncTokenizer &	operator= (GncTokenizer &&)=default
virtual void	load_file (const std::string &path)
const std::string &	current_file ()
void	encoding (const std::string &encoding)
const std::string &	encoding ()
const std::vector< StrVec > &	get_tokens ()

Additional Inherited Members
Protected Attributes inherited from GncTokenizer
std::string	m_utf8_contents
std::vector< StrVec >	m_tokenized_contents

Detailed Description

Definition at line 46 of file gnc-tokenizer-dummy.hpp.

Member Function Documentation

tokenize()

int GncDummyTokenizer::tokenize ( )

overridevirtual

Implements GncTokenizer.

Definition at line 38 of file gnc-tokenizer-dummy.cpp.

{
    StrVec vec;
    std::string line;
 
    m_tokenized_contents.clear();
    std::istringstream in_stream(m_utf8_contents);
 
    while (std::getline (in_stream, line))
    {
        vec.push_back (line);
        m_tokenized_contents.push_back(vec);
 
        line.clear();
        vec.clear();
    }
 
    return 0;
}

---

The documentation for this class was generated from the following files:

• gnc-tokenizer-dummy.hpp
• gnc-tokenizer-dummy.cpp