GncPriceImport Class Reference

The actual PriceImport class It's intended to use in the following sequence of actions: More...

#include <gnc-import-price.hpp>

Public Member Functions
	GncPriceImport (GncImpFileFormat format=GncImpFileFormat::UNKNOWN)
	Constructor for GncPriceImport.
	~GncPriceImport ()
	Destructor for GncPriceImport.
void	file_format (GncImpFileFormat format)
	Sets the file format for the file to import, which may cause the file to be reloaded as well if the previously set file format was different and a filename was already set.
GncImpFileFormat	file_format ()
void	over_write (bool over)
bool	over_write ()
void	from_commodity (gnc_commodity *from_commodity)
	Sets a from commodity.
gnc_commodity *	from_commodity ()
void	to_currency (gnc_commodity *to_currency)
	Sets a to currency.
gnc_commodity *	to_currency ()
void	currency_format (int currency_format)
int	currency_format ()
void	date_format (int date_format)
int	date_format ()
void	encoding (const std::string &encoding)
	Converts raw file data using a new encoding.
std::string	encoding ()
void	update_skipped_lines (std::optional< uint32_t > start, std::optional< uint32_t > end, std::optional< bool > alt, std::optional< bool > errors)
uint32_t	skip_start_lines ()
uint32_t	skip_end_lines ()
bool	skip_alt_lines ()
bool	skip_err_lines ()
void	separators (std::string separators)
std::string	separators ()
void	enable_escape (bool enable)
bool	enable_escape ()
void	settings (const CsvPriceImpSettings &settings)
bool	save_settings ()
void	settings_name (std::string name)
std::string	settings_name ()
void	load_file (const std::string &filename)
	Loads a file into a GncPriceImport.
void	tokenize (bool guessColTypes)
	Splits a file into cells.
std::string	verify ()
void	create_prices ()
	This function will attempt to convert all tokenized lines into prices using the column types the user has set.
bool	check_for_column_type (GncPricePropType type)
void	set_column_type_price (uint32_t position, GncPricePropType type, bool force=false)
std::vector< GncPricePropType >	column_types_price ()

Data Fields
std::unique_ptr< GncTokenizer >	m_tokenizer
	Will handle file loading/encoding conversion/splitting into fields.
std::vector< parse_line_t >	m_parsed_lines
	source file parsed into a two-dimensional array of strings.
int	m_prices_added
int	m_prices_duplicated
int	m_prices_replaced

Detailed Description

The actual PriceImport class It's intended to use in the following sequence of actions:

• set a file format
• load a file
• optionally convert it's encoding
• parse the file into lines, which in turn are split up in to columns the result of this step can be queried from tokenizer
• the user should now map the columns to types, which is stored in column_types
• last step is convert the mapped columns into a list of prices to add

Definition at line 82 of file gnc-import-price.hpp.

Constructor & Destructor Documentation

GncPriceImport()

GncPriceImport::GncPriceImport

(

GncImpFileFormat

format = GncImpFileFormat::UNKNOWN

)

Constructor for GncPriceImport.

Definition at line 65 of file gnc-import-price.cpp.

{
    /* All of the data pointers are initially NULL. This is so that, if
     * gnc_csv_parse_data_free is called before all of the data is
     * initialized, only the data that needs to be freed is freed. */
    m_skip_errors = false;
    file_format(m_settings.m_file_format = format);
}

~GncPriceImport()

GncPriceImport::~GncPriceImport

(

)

Destructor for GncPriceImport.

Definition at line 76 of file gnc-import-price.cpp.

{
}

Member Function Documentation

check_for_column_type()

bool GncPriceImport::check_for_column_type

(

GncPricePropType

type

)

Definition at line 674 of file gnc-import-price.cpp.

{
    return (std::find (m_settings.m_column_types_price.begin(),
                       m_settings.m_column_types_price.end(), type)
                        != m_settings.m_column_types_price.end());
}

column_types_price()

std::vector< GncPricePropType > GncPriceImport::column_types_price

(

)

Definition at line 794 of file gnc-import-price.cpp.

{
    return m_settings.m_column_types_price;
}

create_prices()

void GncPriceImport::create_prices

(

)

This function will attempt to convert all tokenized lines into prices using the column types the user has set.

Creates a list of prices from parsed data.

The parsed data will first be validated. If any errors are found in lines that are marked for processing (ie not marked to skip) this function will throw an error.

Parameters

skip_errors

true skip over lines with errors

Exceptions

throws

std::invalid_argument if data validation or processing fails.

Definition at line 646 of file gnc-import-price.cpp.

{
    /* Start with verifying the current data. */
    auto verify_result = verify();
    if (!verify_result.empty())
        throw std::invalid_argument (verify_result);
 
    m_prices_added = 0;
    m_prices_duplicated = 0;
    m_prices_replaced = 0;
 
    /* Iterate over all parsed lines */
    for (auto parsed_lines_it = m_parsed_lines.begin();
            parsed_lines_it != m_parsed_lines.end();
            ++parsed_lines_it)
    {
        /* Skip current line if the user specified so */
        if ((std::get<PL_SKIP>(*parsed_lines_it)))
            continue;
 
        /* Should not throw anymore, otherwise verify needs revision */
        create_price (parsed_lines_it);
    }
    PINFO("Number of lines is %d, added %d, duplicated %d, replaced %d",
         (int)m_parsed_lines.size(), m_prices_added, m_prices_duplicated, m_prices_replaced);
}

currency_format() [1/2]

int GncPriceImport::currency_format

(

)

Definition at line 217 of file gnc-import-price.cpp.

{ return m_settings.m_currency_format; }

currency_format() [2/2]

void GncPriceImport::currency_format

(

int

currency_format

)

Definition at line 209 of file gnc-import-price.cpp.

{
    m_settings.m_currency_format = currency_format;
 
    /* Reparse all currency related columns */
    std::vector<GncPricePropType> commodities = { GncPricePropType::AMOUNT };
    reset_formatted_column (commodities);
}

date_format() [1/2]

int GncPriceImport::date_format

(

)

Definition at line 227 of file gnc-import-price.cpp.

{ return m_settings.m_date_format; }

date_format() [2/2]

void GncPriceImport::date_format

(

int

date_format

)

Definition at line 219 of file gnc-import-price.cpp.

{
    m_settings.m_date_format = date_format;
 
    /* Reparse all date related columns */
    std::vector<GncPricePropType> dates = { GncPricePropType::DATE };
    reset_formatted_column (dates);
}

enable_escape() [1/2]

bool GncPriceImport::enable_escape

(

)

Definition at line 301 of file gnc-import-price.cpp.

{ return m_settings.m_enable_escape; }

enable_escape() [2/2]

void GncPriceImport::enable_escape

(

bool

enable

)

Definition at line 293 of file gnc-import-price.cpp.

{
    if (file_format() != GncImpFileFormat::CSV)
        return;
    m_settings.m_enable_escape = enable;
    auto csvtok = dynamic_cast<GncCsvTokenizer*>(m_tokenizer.get());
    csvtok->set_enable_escape (enable);
}

encoding() [1/2]

std::string GncPriceImport::encoding

(

)

Definition at line 251 of file gnc-import-price.cpp.

{ return m_settings.m_encoding; }

encoding() [2/2]

void GncPriceImport::encoding

(

const std::string &

encoding

)

Converts raw file data using a new encoding.

This function must be called after load_file only if load_file guessed the wrong encoding.

Parameters

encoding

Encoding that data should be translated using

Definition at line 234 of file gnc-import-price.cpp.

{
    // TODO investigate if we can catch conversion errors and report them
    if (m_tokenizer)
    {
        m_tokenizer->encoding(encoding); // May throw
        try
        {
            tokenize(false);
        }
        catch (...)
        { };
    }
 
    m_settings.m_encoding = encoding;
}

file_format() [1/2]

GncImpFileFormat GncPriceImport::file_format

(

)

Definition at line 131 of file gnc-import-price.cpp.

{
    return m_settings.m_file_format;
}

file_format() [2/2]

void GncPriceImport::file_format

(

GncImpFileFormat

format

)

Sets the file format for the file to import, which may cause the file to be reloaded as well if the previously set file format was different and a filename was already set.

Parameters

format

the new format to set

Exceptions

std::ifstream::failure

if file reloading fails

Definition at line 87 of file gnc-import-price.cpp.

{
    if (m_tokenizer && m_settings.m_file_format == format)
        return;
 
    auto new_encoding = std::string("UTF-8");
    auto new_imp_file = std::string();
 
    // Recover common settings from old tokenizer
    if (m_tokenizer)
    {
        new_encoding = m_tokenizer->encoding();
        new_imp_file = m_tokenizer->current_file();
        if (file_format() == GncImpFileFormat::FIXED_WIDTH)
        {
            auto fwtok = dynamic_cast<GncFwTokenizer*>(m_tokenizer.get());
            if (!fwtok->get_columns().empty())
                m_settings.m_column_widths = fwtok->get_columns();
        }
    }
 
    m_settings.m_file_format = format;
    m_tokenizer = gnc_tokenizer_factory(m_settings.m_file_format);
 
    // Set up new tokenizer with common settings
    // recovered from old tokenizer
    m_tokenizer->encoding(new_encoding);
    load_file(new_imp_file);
 
    // Restore potentially previously set separators or column_widths
    if (file_format() == GncImpFileFormat::CSV)
    {
        if (!m_settings.m_separators.empty())
            separators (m_settings.m_separators);
        enable_escape (m_settings.m_enable_escape);
    }
    else if ((file_format() == GncImpFileFormat::FIXED_WIDTH)
        && !m_settings.m_column_widths.empty())
    {
        auto fwtok = dynamic_cast<GncFwTokenizer*>(m_tokenizer.get());
        fwtok->columns (m_settings.m_column_widths);
    }
}

from_commodity() [1/2]

gnc_commodity * GncPriceImport::from_commodity

(

)

Definition at line 171 of file gnc-import-price.cpp.

{ return m_settings.m_from_commodity; }

from_commodity() [2/2]

void GncPriceImport::from_commodity

(

gnc_commodity *

from_commodity

)

Sets a from commodity.

This is the commodity all import data relates to. When a from commodity is set, there can't be any from columns selected in the import data.

Parameters

from_commodity

pointer to a commodity or NULL.

Definition at line 147 of file gnc-import-price.cpp.

{
    m_settings.m_from_commodity = from_commodity;
    if (m_settings.m_from_commodity)
    {
        auto col_type_sym = std::find (m_settings.m_column_types_price.begin(),
                m_settings.m_column_types_price.end(), GncPricePropType::FROM_SYMBOL);
 
        if (col_type_sym != m_settings.m_column_types_price.end())
            set_column_type_price (col_type_sym -m_settings.m_column_types_price.begin(),
                            GncPricePropType::NONE);
 
        auto col_type_name = std::find (m_settings.m_column_types_price.begin(),
                m_settings.m_column_types_price.end(), GncPricePropType::FROM_NAMESPACE);
 
        if (col_type_name != m_settings.m_column_types_price.end())
            set_column_type_price (col_type_name -m_settings.m_column_types_price.begin(),
                            GncPricePropType::NONE);
 
        // force a refresh of the to_currency if the from_commodity is changed
        std::vector<GncPricePropType> commodities = { GncPricePropType::TO_CURRENCY };
        reset_formatted_column (commodities);
    }
}

load_file()

void GncPriceImport::load_file

(

const std::string &

filename

)

Loads a file into a GncPriceImport.

This is the first function that must be called after creating a new GncPriceImport. As long as this function didn't run successfully, the importer can't proceed.

Parameters

filename

Name of the file that should be opened

Exceptions

may	throw std::ifstream::failure on any io error

Definition at line 361 of file gnc-import-price.cpp.

{
    /* Get the raw data first and handle an error if one occurs. */
    try
    {
        m_tokenizer->load_file (filename);
        return;
    }
    catch (std::ifstream::failure& ios_err)
    {
        // Just log the error and pass it on the call stack for proper handling
        PWARN ("Error: %s", ios_err.what());
        throw;
    }
}

over_write() [1/2]

bool GncPriceImport::over_write

(

)

Definition at line 140 of file gnc-import-price.cpp.

{ return m_over_write; }

over_write() [2/2]

void GncPriceImport::over_write

(

bool

over

)

Definition at line 136 of file gnc-import-price.cpp.

{
    m_over_write = over;
}

save_settings()

bool GncPriceImport::save_settings

(

)

Definition at line 335 of file gnc-import-price.cpp.

{
    if (preset_is_reserved_name (m_settings.m_name))
        return true;
 
    /* separators are already copied to m_settings in the separators
     * function above. However this is not the case for the column
     * widths in fw mode, so do this now.
     */
    if (file_format() == GncImpFileFormat::FIXED_WIDTH)
    {
        auto fwtok = dynamic_cast<GncFwTokenizer*>(m_tokenizer.get());
        m_settings.m_column_widths = fwtok->get_columns();
    }
    return m_settings.save();
}

separators() [1/2]

std::string GncPriceImport::separators

(

)

Definition at line 291 of file gnc-import-price.cpp.

{ return m_settings.m_separators; }

separators() [2/2]

void GncPriceImport::separators

(

std::string

separators

)

Definition at line 281 of file gnc-import-price.cpp.

{
    if (file_format() != GncImpFileFormat::CSV)
        return;
 
    m_settings.m_separators = separators;
    auto csvtok = dynamic_cast<GncCsvTokenizer*>(m_tokenizer.get());
    csvtok->set_separators (separators);
 
}

set_column_type_price()

void GncPriceImport::set_column_type_price	(	uint32_t	position,
		GncPricePropType	type,
		bool	force = false
	)

Definition at line 731 of file gnc-import-price.cpp.

{
    if (position >= m_settings.m_column_types_price.size())
        return;
 
    auto old_type = m_settings.m_column_types_price[position];
    if ((type == old_type) && !force)
        return; /* Nothing to do */
 
    // Column types should be unique, so remove any previous occurrence of the new type
    std::replace(m_settings.m_column_types_price.begin(), m_settings.m_column_types_price.end(),
            type, GncPricePropType::NONE);
 
    m_settings.m_column_types_price.at (position) = type;
 
    // If the user has set a 'from namespace' column, we can't have a 'commodity from' selected
    if (type == GncPricePropType::FROM_NAMESPACE)
        from_commodity (nullptr);
 
    // If the user has set a 'from symbol' column, we can't have a 'commodity from' selected
    if (type == GncPricePropType::FROM_SYMBOL)
        from_commodity (nullptr);
 
    // If the user has set a 'currency to' column, we can't have a 'currency to' selected
    if (type == GncPricePropType::TO_CURRENCY)
        to_currency (nullptr);
 
    /* Update the preparsed data */
    for (auto parsed_lines_it = m_parsed_lines.begin();
            parsed_lines_it != m_parsed_lines.end();
            ++parsed_lines_it)
    {
        /* Reset date and currency formats for each price props object
         * to ensure column updates use the most recent one
         */
        std::get<PL_PREPRICE>(*parsed_lines_it)->set_date_format (m_settings.m_date_format);
        std::get<PL_PREPRICE>(*parsed_lines_it)->set_currency_format (m_settings.m_currency_format);
 
        uint32_t row = parsed_lines_it - m_parsed_lines.begin();
 
        /* If the column type actually changed, first reset the property
         * represented by the old column type
         */
        if (old_type != type)
        {
            auto old_col = std::get<PL_INPUT>(*parsed_lines_it).size(); // Deliberately out of bounds to trigger a reset!
            if ((old_type > GncPricePropType::NONE)
                    && (old_type <= GncPricePropType::PRICE_PROPS))
                update_price_props (row, old_col, old_type);
        }
        /* Then set the property represented by the new column type */
        if ((type > GncPricePropType::NONE)
                && (type <= GncPricePropType::PRICE_PROPS))
            update_price_props (row, position, type);
 
        /* Report errors if there are any */
        auto price_errors = std::get<PL_PREPRICE>(*parsed_lines_it)->errors();
        std::get<PL_ERROR>(*parsed_lines_it) =
                price_errors +
                (price_errors.empty() ? std::string() : "\n");
    }
}

settings()

void GncPriceImport::settings

(

const CsvPriceImpSettings &

settings

)

Definition at line 303 of file gnc-import-price.cpp.

{
    /* First apply file format as this may recreate the tokenizer */
    file_format (settings.m_file_format);
    /* Only then apply the other settings */
    m_settings = settings;
    from_commodity (m_settings.m_from_commodity);
    to_currency (m_settings.m_to_currency);
    encoding (m_settings.m_encoding);
 
    if (file_format() == GncImpFileFormat::CSV)
        separators (m_settings.m_separators);
    else if (file_format() == GncImpFileFormat::FIXED_WIDTH)
    {
        auto fwtok = dynamic_cast<GncFwTokenizer*>(m_tokenizer.get());
        fwtok->columns (m_settings.m_column_widths);
    }
    try
    {
        tokenize(false);
    }
    catch (...)
    { };
 
    /* Tokenizing will clear column types, reset them here
     * based on the loaded settings.
     */
    std::copy_n (settings.m_column_types_price.begin(),
            std::min (m_settings.m_column_types_price.size(), settings.m_column_types_price.size()),
            m_settings.m_column_types_price.begin());
}

settings_name() [1/2]

std::string GncPriceImport::settings_name

(

)

Definition at line 353 of file gnc-import-price.cpp.

{ return m_settings.m_name; }

settings_name() [2/2]

void GncPriceImport::settings_name

(

std::string

name

)

Definition at line 352 of file gnc-import-price.cpp.

{ m_settings.m_name = name; }

skip_alt_lines()

bool GncPriceImport::skip_alt_lines

(

)

Definition at line 278 of file gnc-import-price.cpp.

{ return m_settings.m_skip_alt_lines; }

skip_end_lines()

uint32_t GncPriceImport::skip_end_lines

(

)

Definition at line 277 of file gnc-import-price.cpp.

{ return m_settings.m_skip_end_lines; }

skip_err_lines()

bool GncPriceImport::skip_err_lines

(

)

Definition at line 279 of file gnc-import-price.cpp.

{ return m_skip_errors; }

skip_start_lines()

uint32_t GncPriceImport::skip_start_lines

(

)

Definition at line 276 of file gnc-import-price.cpp.

{ return m_settings.m_skip_start_lines; }

to_currency() [1/2]

gnc_commodity * GncPriceImport::to_currency

(

)

Definition at line 196 of file gnc-import-price.cpp.

{ return m_settings.m_to_currency; }

to_currency() [2/2]

void GncPriceImport::to_currency

(

gnc_commodity *

to_currency

)

Sets a to currency.

This is the to currency all import data relates to. When a to currency is set, there can't be any to currency columns selected in the import data.

Parameters

to_currency

pointer to a commodity or NULL.

Definition at line 178 of file gnc-import-price.cpp.

{
    m_settings.m_to_currency = to_currency;
    if (m_settings.m_to_currency)
    {
        auto col_type_currency = std::find (m_settings.m_column_types_price.begin(),
                m_settings.m_column_types_price.end(), GncPricePropType::TO_CURRENCY);
 
        if (col_type_currency != m_settings.m_column_types_price.end())
            set_column_type_price (col_type_currency -m_settings.m_column_types_price.begin(),
                            GncPricePropType::NONE);
 
        // force a refresh of the from_commodity if the to_currency is changed
        // either namespace or symbol will be sufice
        std::vector<GncPricePropType> commodities = { GncPricePropType::FROM_SYMBOL };
        reset_formatted_column (commodities);
    }
}

tokenize()

void GncPriceImport::tokenize

(

bool

guessColTypes

)

Splits a file into cells.

This requires having an encoding that works (see GncPriceImport::convert_encoding). Tokenizing related options should be set to the user's selections before calling this function. Notes: - this function must be called with guessColTypes set to true once before calling it with guessColTypes set to false.

• if guessColTypes is true, all the column types will be set GncPricePropType::NONE right now as real guessing isn't implemented yet

  Parameters

  guessColTypes

  true to guess what the types of columns are based on the cell contents

  Exceptions

  std::range_error

  if tokenizing failed

Definition at line 388 of file gnc-import-price.cpp.

{
    if (!m_tokenizer)
        return;
 
    uint32_t max_cols = 0;
    m_tokenizer->tokenize();
    m_parsed_lines.clear();
    for (auto tokenized_line : m_tokenizer->get_tokens())
    {
        auto length = tokenized_line.size();
        if (length > 0)
            m_parsed_lines.push_back (std::make_tuple (tokenized_line, std::string(),
                    std::make_shared<GncImportPrice>(date_format(), currency_format()),
                    false));
        if (length > max_cols)
            max_cols = length;
    }
 
    /* If it failed, generate an error. */
    if (m_parsed_lines.size() == 0)
    {
        throw (std::range_error ("Tokenizing failed."));
        return;
    }
 
    m_settings.m_column_types_price.resize(max_cols, GncPricePropType::NONE);
 
    /* Force reinterpretation of already set columns and/or base_account */
    for (uint32_t i = 0; i < m_settings.m_column_types_price.size(); i++)
        set_column_type_price (i, m_settings.m_column_types_price[i], true);
 
    if (guessColTypes)
    {
        /* Guess column_types based
         * on the contents of each column. */
        /* TODO Make it actually guess. */
    }
}

update_skipped_lines()

void GncPriceImport::update_skipped_lines	(	std::optional< uint32_t >	start,
		std::optional< uint32_t >	end,
		std::optional< bool >	alt,
		std::optional< bool >	errors
	)

Definition at line 253 of file gnc-import-price.cpp.

{
    if (start)
        m_settings.m_skip_start_lines = *start;
    if (end)
        m_settings.m_skip_end_lines = *end;
    if (alt)
        m_settings.m_skip_alt_lines = *alt;
    if (errors)
        m_skip_errors = *errors;
 
    for (uint32_t i = 0; i < m_parsed_lines.size(); i++)
    {
        std::get<PL_SKIP>(m_parsed_lines[i]) =
            ((i < skip_start_lines()) ||             // start rows to skip
             (i >= m_parsed_lines.size() - skip_end_lines()) ||          // end rows to skip
             (((i - skip_start_lines()) % 2 == 1) && // skip every second row...
                  skip_alt_lines()) ||                   // ...if requested
             (m_skip_errors && !std::get<PL_ERROR>(m_parsed_lines[i]).empty())); // skip lines with errors
    }
}

verify()

std::string GncPriceImport::verify

(

)

Definition at line 505 of file gnc-import-price.cpp.

{
    auto newline = std::string();
    auto error_msg = ErrorListPrice();
 
    /* Check if the import file did actually contain any information */
    if (m_parsed_lines.size() == 0)
    {
        error_msg.add_error(_("No valid data found in the selected file. It may be empty or the selected encoding is wrong."));
        return error_msg.str();
    }
 
    /* Check if at least one line is selected for importing */
    auto skip_alt_offset = m_settings.m_skip_alt_lines ? 1 : 0;
    if (m_settings.m_skip_start_lines + m_settings.m_skip_end_lines + skip_alt_offset >= m_parsed_lines.size())
    {
        error_msg.add_error(_("No lines are selected for importing. Please reduce the number of lines to skip."));
        return error_msg.str();
    }
 
    verify_column_selections (error_msg);
 
    update_skipped_lines (std::nullopt, std::nullopt, std::nullopt, std::nullopt);
 
    auto have_line_errors = false;
    for (auto line : m_parsed_lines)
    {
        if (!std::get<PL_SKIP>(line) && !std::get<PL_ERROR>(line).empty())
        {
            have_line_errors = true;
            break;
        }
    }
 
    if (have_line_errors)
        error_msg.add_error( _("Not all fields could be parsed. Please correct the issues reported for each line or adjust the lines to skip."));
 
    return error_msg.str();
}

Field Documentation

m_parsed_lines

std::vector<parse_line_t> GncPriceImport::m_parsed_lines

source file parsed into a two-dimensional array of strings.

Per line also holds possible error messages and objects with extracted price properties.

Definition at line 145 of file gnc-import-price.hpp.

m_prices_added

int GncPriceImport::m_prices_added

Definition at line 148 of file gnc-import-price.hpp.

m_prices_duplicated

int GncPriceImport::m_prices_duplicated

Definition at line 149 of file gnc-import-price.hpp.

m_prices_replaced

int GncPriceImport::m_prices_replaced

Definition at line 150 of file gnc-import-price.hpp.

m_tokenizer

std::unique_ptr<GncTokenizer> GncPriceImport::m_tokenizer

Will handle file loading/encoding conversion/splitting into fields.

Definition at line 144 of file gnc-import-price.hpp.

---

The documentation for this class was generated from the following files:

• gnc-import-price.hpp
• gnc-import-price.cpp