Development Notes

General Notes on Development

dev.py

A mirror of main.py has been provided to help with development. This script will allow you to set logging options as well logging types. Script options are:

• -h or --help - show help message and exit
• -l or --log-level - set the logging level. Options include:
  • DEBUG (default) - show all messages
  • INFO - show info messages and above
  • WARNING - show warning messages and above
  • ERROR - show error messages and above
  • CRITICAL - show critical messages and above
• -t or --log-type - set the logging type. Options include:
  • console (default) - log to console - can be used with logdy to view logs in a more readable format, or another log viewer.
  • file - log to thr file debug.log.

Example:

 python dev.py -l DEBUG -t console

linting

pylint is a tool to help with linting. It can be used to check the code for errors and style issues. This can be run using the following command:

pylint src/

autopep8 is a tool to help with formatting. It can be used to format the code to conform to PEP 8 style guide. This can be run using the following command:

autopep8 --in-place --aggressive --aggressive --recursive src/

Testing

pytest is a tool to help with testing. It can be used to run the tests and check the code for errors. This can be run using the following command:

pytest

Data Validation

pydantic is a tool to help with data validation. It can be used to validate the data and check for errors. This can be run using the following command:

Extending Functionality

Both the exporters and connctors allow for you to easily extend the functionality of the system. This can be done using the guide before for each and referencing the code in the already in place.

logging

Logging should be included in all classes to provide error tracing in the event that there is an issue. Logging can be easily included by adding rh following to the start of your class:

import logging
logger = logging.getLogger(__name__)

This will pass all logging information to the main method and allow it to be passed to with the rest of the logging information.

Actions

New actions can be added by creating a class in the actions directory.

Naming convention.

• The file name must be snake_case and the class name must be PascalCase.
• The class name must be the same as the file name.

Init

conf, folio_connection, trans_active

Exporters

New exporters can be added by creating a class in the exporters directory.

Naming convention.

• The file name must be snake_case and the class name must be PascalCase.
• The class name must be the same as the file name.

Init

• The class must have an __init__ method that takes the following parameters:
  • conf - This is the config file that is used to control the system, processed from the YAML file for this action. This is a dictionary object.
  • template_processor - This is the template processor that is used to process the templates. This is the TemplateProcessor that has been passed the template data and template configuration. This class can be used in your code to generate a new rendered template by passing it the the configuration for that template; a string will be returned.

Required methods

ship_it

• The class must have a ship_it method that takes no parameters:

This function can call any other methods that is needed to process the provided template, connect to a remote clients and send the information. This is the method that will be called by the job processor to send the information.

Connectors

New connectors can be added by creating a class in the connectors directory.

Naming convention.

• The file name must be snake_case and the class name must be PascalCase.
• The class name must be the same as the file name.

Init

• The class must have an __init__ method that takes the following parameters:
  • env_key - This is the start of the env variable holding you connection secrets.

There is no pattern to the what env variables are required and if needed one of them can be used to import the secrets of another set of secrets; See share_point_connector.py for an example of this.

Required methods

write_rows

• The class must have a write_rows method that takes the following parameters:
  • data - The data to be written to the connector. This is a list of dictionaries.

The data will be formatted based on the YAML configuration file. In the case of INLINE formatting the data will be a flat dictionary object such as:

[
    {
        "name": "John Doe",
        "age": 30,
        "city": "New York"
    },
    {
        "name": "Jane Doe",
        "age": 25,
        "city": "Los Angeles"
    }
]

Using a template will allow users to create a more complex data structure. It is your responsibility to ensure that the data is in the correct format for the connector and if needed breaking the data into smaller pieces to be sent to the connector.

update_rows

• The class must have a update_rows method that takes the following parameters:
  • data - The data to be updated in the connector. This is a list of dictionaries.
  • filter_string - The name of the field that the data should be filtered on as set in the YAML file.

The data will be formatted based on the YAML configuration file. In the case of INLINE formatting the data will be a flat dictionary object such as:

[
    {
        "name": "John Doe",
        "age": 30,
        "city": "New York"
    },
    {
        "name": "Jane Doe",
        "age": 25,
        "city": "Los Angeles"
    }
]

Using a template will allow users to create a more complex data structure. It is your responsibility to ensure that the data is in the correct format for the connector and if needed breaking the data into smaller pieces to be sent to the connector.

delete_rows

• The class must have a delete_rows method that takes the following parameters:
  • data - The data to be deleted from the connector. This is a list of dictionaries.
  • filter_string - The name of the field that the data should be filtered on as set in the YAML file. The data will be formatted based on the YAML configuration file. In the case of INLINE formatting the data will be a flat dictionary object such as:

[
    {
        "name": "John Doe",
        "age": 30,
        "city": "New York"
    },
    {
        "name": "Jane Doe",
        "age": 25,
        "city": "Los Angeles"
    }
]

Using a template will allow users to create a more complex data structure. It is your responsibility to ensure that the data is in the correct format for the connector and if needed breaking the data into smaller pieces to be sent to the connector.