Home

Welcome to the python_folio_to_bursar_transfer wiki!

Introduction

Data Processing Order

All processing jobs are defined in the Configuration Jobs file(s), and can include sub files for each step of the process.

Data is processed in the following order:

1. job.yaml file is processed Configuration Jobs
   • This lists all the jobs and when they should be ran. Only jobs that are scheduled to run will be processed.
2. <<JOB_NAME>>.yaml file is processed Configuration Job Profile
   • This file contains the settings for the job.
3. Charge Data
   • Pull Raw data rom FOLIO (internal)
   • Include User Data (internal)
   • Include Material Data (internal)
   • Run Formatters Configuration Reformatting
   • Run Merges Configuration Merging Data
   • Run Filters Configuration Filters
   • Generate Summary data (internal)
4. Credit Data
   • Pull Raw data rom FOLIO (internal)
   • Include User Data (internal)
   • Include Material Data (internal)
   • Run Formatters Configuration Reformatting
   • Run Merges Configuration Merging Data
   • Run Filters Configuration Filters
   • Generate Summary data (internal)
5. Run the processing actions Configuration Actions
6. Run the export actions Configuration Exporting
7. Run the external connections Configuration Connecters

Config Files

FOLIO

• Configuration FOLIO This document covers basic information for FOLIO that is required to be be in place prior to running these scripts.

.env Fields

• Configuration env

YAML Job Files

The jobs.yaml is required to be in the config directory and is the main file that is used to control the system. This files is used to schedule what jobs will be ran by the system.

• Configuration Jobs

Config files are called by the jobs.yaml file based on the job settings. Each config file can be configured as follows:

• Configuration Job Profile
  • Configuration Filters: Defines the filters that are used against the FOLIO data.
  • Configuration Formatters: Defines how the raw data should be reformated.
  • Configuration Merging Data: Defines hwo data should be merged, and allows for the inclusion of external data.
  • Configuration Exporting: This section covers how to export the data that is returned from FOLIO. This is done by using the export section in the config file.
  • Configuration Connecters: This section covers how to connect to the FOLIO API. This is done by using the connecters section in the config file.
  • Configuration Actions: This section covers how to process the data that is returned from FOLIO. This is done by using the processors section in the config file.

Templates

• Templates Templates used by this system use the handlebars templating engine to process the data. This allows for the data to be formatted in a way that is useful to the end user. Some default templates have been include in the repo as an example of how they can be written.

Deployment

• Deploying Docker This section covers how to deploy the code to a server and run it as a cron job.
• Deploying Lambda This section covers how to deploy the code to AWS Lambda and run it as a cron job.
• Deploying Local This section covers how to run the code locally and test it.

Utilities

Data Samples

• Utilities Data Sample

Utility Apps

There has been a number of small utility apps written to help with config testing, setup and pre data processing. More information can be found here:

• Utilities App

Logging

Logging has been added to the system to help with debugging and tracking the progress of the system. information on how to use the logging system can be found here:

• Utilities Logging

Integrations

The following integrations are packed with this system.

• Integrations Slack
• Integrations AWS
• Integrations Microsoft 365
• Integrations AirTable

Know issues:

Know Limitations

Development

General notes to aid in development if you wish to update or extend this system. Development Notes