add openssl3 to el8 - #176
Open
adivinho wants to merge 13 commits into
Open
Conversation
…t 1) (Percona-Lab#162) https://perconadev.atlassian.net/browse/PBS-39 'binsrv::storage_config' extended with new optional field 'encryption' of class 'binsrv::encryption_config' that if present indicates that binlog data files in the storage should be encrypted and holds encryption configuration parameters (currently only 'format' and 'keyring_uri'). Added config validation functions for 'binsrv::encryption_config' and 'binsrv::storage_config'. Added new abstract class 'binsrv::basic_keyring' that provides basic interface for working with encryption keys (currently only 'get_key()' method). Added new model classes (based on 'util::nv_tuple') for representing individual encryption keys and their collections: 'binsrv::keyring_record' and 'binsrv::keyring_record_collection'. Added concrete implementation of the 'binsrv::basic_keyring' interface called 'binsrv::file_keyring' that uses 'binsrv::keyring_record_collection' underneath. Added new factory class 'binsrv::keyring_factory' that helps with constructing concrete implementation of the 'binsrv::basic_keyring' interface based on the provided URI (the value taken from the '<storage.encryption.keyring_uri>' configuration parameter). Main application extended with analyzing optional encryption configuration section, printing its primary parameters to the log file, creating an instance of concrete 'binsrv::basic_keyring' interface using 'binsrv::keyring_factory' class, and printing loaded keyring statistics (number of keys and key/algorithm pairs). Introduced 'utils::hex_value' class that holds a byte blob that can be serialized to / from a hex string. This class is supposed to be used in configuration files to store binary data. Similarly to 'native_file_operations_helpers.hpp' introduced new 'file_operations_helpers.hpp' header file in the 'util' namespace which provides simple functions for reading / writing file content (using 'std::ifstream' / 'std::ofstream') into 'std::string'. 'binsrv::filesystem_storage_backend', 'binsrv::main_config', and `binsrv::keyring_record_collection` refactored to use these helper functions. Sample configuration file ('main_config.json') extended with the '<storage.encryption>' section. Added a sample file representing file keyring data in JSON format ('file_keyring_data.json'). Updated 'README.md' file with new configuration section description.
…t 2) (Percona-Lab#164) https://perconadev.atlassian.net/browse/PBS-39 'binsrv::encryption_config' class extended with two new fields: - 'kek_id', representing Key-Encrypting Key identifier in the keyring, that should be used to encrypt individual binlog file keys, - 'cipher', representing the encryption cipher that should be used for encrypting binlog data files. Storage metadata ('binsrv::storage_metadata') that is created during initial storage initialization now also holds info about storage encryption in the new optional 'encryption' field. The value of this field in the storage metadata is now also validated during storage initialization (compared with the '<storage.encryption.format>' field specified in the configuration file). Main application now also prints info about active KEK (key-encrypting key) to the log file. 'algorithm' field in the 'binsrv::keyring_record' renamed to 'cipher'. Sample keyring data file updated correspondingly. Sample configuration file extended with the new fields. README.md extended with new configuration parameters description.
…t 3) (Percona-Lab#165) https://perconadev.atlassian.net/browse/PBS-39 'binsrv::binlog_file_metadata' class extended with additional optional field 'encryption' of class 'binsrv::binlog_file_encryption_metadata' that consists of two fields: - 'file_key_envelope' holding info about file key encryption, - 'file_data_envelope' holding info about binlog data file encryption. Added new 'binsrv::file_key_envelope' class with the following fields: - 'kek_id' - the identifier of the key from the keyring that was used as KEK for file key encryption operation, - 'iv_hex' - the initialization vector that was used for this file key encryption operation (optional, should be set for every mode apart from 'XXX-ECB'), - 'data_hex' - the result of the key-encryption operation, - 'tag_hex' - the AEAD tag calculated as the result of this file key encryption operation (optional, is set only in 'XXX_GCM' encryption modes). Added new 'binsrv::file_data_envelope' class with the following fields: - 'cipher' - the name of the encryption cipher used for binlog file data encryption operation (e.g. 'AES-256-CTR'), - 'iv_hex' - the initialization vector that was used for this binlog file data encryption operation (always present as file data encryption operation requires a streaming cipher like 'XXX-CTR' or 'XXX-GCM'), - 'tag_hex' - the AEAD tag calculated as the result of this binlog file data encryption operation (optional, is set only in 'XXX-GCM' encryption modes). 'binsrv::basic_keyring' interface extended with one more method 'contains()' which checks if a key with the specified identifier present in the keyring. 'binsrv::storage::binlog_record' internal class extended with additional optional 'encryption' field of class 'binsrv::storage::binlog_encryption_record' which holds info from the envelopes above in the form more suitable for encryption / decryption operations (unhexed). If encryption is enabled in the main configuration file each JSON binlog metadata file will now include additional 'encryption' subelement. 'save_binlog_metadata()' / 'load_binlog_metadata()' methods in the 'binsrv::storage' class extended with converting encryption envelopes from / to internal 'binsrv::storage::binlog_encryption_record' class instances. As an intermediate step added fake file key / iv generation functions, file key encryption and file key encryption tag calculation functions, fake file data encryption and file data encryption tag calculation functionality to the 'binsrv::storage::generate_binlog_encryption_record()' method. At the next step these functions should be replaced with proper OpenSSL calls.
…t 4) (Percona-Lab#166) https://perconadev.atlassian.net/browse/PBS-39 JSON responses generated by the 'list' / 'search_by_timestamp' / 'search_by_gtid_set' / 'purge_binlogs' extended with the optional 'encryption' subobject (of the same structure as in binlog metadata files). 'binsrv::file_key_envelope' renamed/moved to 'binsrv::models::file_key_envelope_record'. 'binsrv::file_data_envelope' renamed/moved to 'binsrv::models::file_data_envelope_record'. 'binsrv::binlog_file_encryption_metadata' renamed/moved to 'binsrv::models::binlog_file_encryption_record'. 'binsrv::models::binlog_file_record' extended with a new optional field 'encryption' of class 'binsrv::models::binlog_file_encryption_record'. Added "record to model" / "model to record" conversion functions for the 'binsrv::storage::binlog_encryption_record' structure. Updated README.md with the new 'encryption' subobject in JSON responses of the 'list' / 'search_by_timestamp' / 'search_by_gtid_set' / 'purge_binlogs' commands.
…t 5) (Percona-Lab#167) https://perconadev.atlassian.net/browse/PBS-39 Added new static library 'opensslpp' which is intended to include c++ wrappers for OpenSSL cryptographic primitives. The initial version includes: * 'opensslpp::crypto_rng' class that provides random cryptographically-strong byte sequences generation (based on 'RAND_butes()' API call). * 'opensslpp::cipher_context' class that provides a convenient way of encrypting / decrypting data with various algorithms (e.g. 'AES'), key lengths (e.g 128, / 192 / 256 bit), and in various modes (e.g. 'ECB', 'CBC', 'CTR', 'GCM'). It is based on 'EVP_CIPHER_CTX'. * 'opensslpp::core_error' exception class that automatically extracts error message from the OpenSSL subsystem (via the 'ERR_get_error()' call) Currently we support only the following encryption modes: * 'ECB' - does not need IV, does not generate / verify tag, operates on blocks only (16 bytes) * 'CBC' - needs IV, does not generate / verify tag, operates on blocks only (16 bytes) * 'CTR' needs IV, does not generate / verify tag, operates on byte sequences of any size. * 'GCM' needs IV, generates / verifies tag, operates on byte sequences of any size. For both 'ECB' and 'CBC' modes we deliberately disable padding. Added tho new unit tests: * 'crypto_rnd_test.cpp' (BOOST_TEST_MODULE CryptoRndTests) * 'cipher_context_test.cpp' (BOOST_TEST_MODULE CipherContextTests)
…t 6) (Percona-Lab#168) https://perconadev.atlassian.net/browse/PBS-39 'binsrv::storage::generate_binlog_encryption_record()' method reworked with proper 'opensslpp' cryptography primitives. Removed all remainders of fake key generation / encryption. Implemented proper binlog data encryption before writing: instead of calls to 'backend_->write_data_to_stream()' we now have a new method 'storage::write_data_to_stream()' that decides whether encryption is needed or not. Currently we stick to the strategy of never holding file keys in memory in plaintext. Instead, 'binsrv::storage::storage::binlog_encryption_record' holds them in encrypted form and we temporarily restore them on each write. This may cause some additional CPU time spent on decryption but it is expected to be insignificant in comparison to IO operations. 'opensslpp::cipher_context' extended with one new method 'extract_updated_iv()' that helps to identify the value of the updated IV (original IV + block counter) after some calls to the 'update()' method have been made. This function helps with resuming streaming encryption (for CTR modes, for instance). 'opensslpp::cipher_context' extended with one more static method 'create_with_offset()' that helps to create a context that would have internal state identical as if it had already processed 'offset' bytes. Again, this function helps with streaming operations. This commits also adds a list of TODO items that suggest to add more diagnostics for key / key combinations validation. Added new 'CipherContextUpdatedIVCTR' boost test case to the 'cipher_context.cpp' (BOOST_TEST_MODULE CipherContextTests) module that checks that 'opensslpp::cipher_context::create_with_offset()' method produces expected results.
…t 7) (Percona-Lab#170) https://perconadev.atlassian.net/browse/PBS-39 Added more diagnostics for keyring data, encryption configuration parameter and their combinations - We now check that 'cipher' parameter in the keyring record is known to OpenSSL and is of 'ECB', 'CBC', 'CRT', or 'GCM' mode. - We now check that the actual length of the 'data_hex' matches the one derived from the 'cipher' (i.g., 256 bit / 32 bytes / 64 hexadecimal characters for 'AES-256-CTR'). - We now check the uniqueness of the key IDs ('id' field) amongst keyring records. - We also make sure that random file keys (of length that corresponds to the '<storage.encryption.cipher>' configuration parameter) can be encrypted with the active KEK ('<storage.encryption.kek_id>' configuration parameter). For instance, if active data cipher is 'AES-192-CRT' (key length 24 bytes), then the active KEK cannot be of 'ECB' or 'CBC' mode as these ciphers can only encrypt data of length that is a multiple of the block size (16 bytes). 'opensslpp::cipher_context' class extended with: - 'get_mode()' (both static and non-static versions) that returns the cipher mode ('ECB', 'CBC', 'CRT', or 'GCM'). - 'is_cipher_name_known()' (static) - returns true if this cipher name is known to OpenSSL. - 'is_mode_supported()' (static) - returns true if the mode is one of the 'ECB', 'CBC', 'CRT', or 'GCM'. - 'is_cipher_name_supported()' (static) - returns true if this cipher name is known to OpenSSL and its mode is supported. 'cipher_context_test.cpp' (BOOST_TEST_MODULE CipherContextTests) unit test extended with the following test cases. - 'CipherContextGetModeStatic' / 'CipherContextGetMode' that check for the behavior of the 'opensslpp::cipher_context::get_mode()' methods (both static and non-static). - 'CipherContextCTRResume' that checks for "resume streaming" operation for 'XXX-NNN-CTR' ciphers - creating a new encryption context, updating it with some data, finalizing, creating a new context with offset, updating it with more data, and finalizing. The result of this operation should be identical to simple encryption of the concatenated data blocks from the first and the second updates.
…t 8) (Percona-Lab#171) https://perconadev.atlassian.net/browse/PBS-39 'keyring_url' configuration parameter moved from the '<storage.encryption>' section into the new top-level '<keyring>' section and renamed to 'uri'. This change allows to disable encryption for new files created in non-empty storage (that already has some binlog files encrypted) and at the same time specify keyring configuration parameters (needed to access already existing encrypted files). Before this change, this simply wasn't possible. Relaxed the validation rule that 'encryption_format' of the storage saved in the storage metadata file must always match to the value of the '<storage.encryption.format>' configuration parameter. Instead, we now allow to freely change between encrypted / unencrypted configuration options provided that they are using the same 'format'. At the moment, the last part of this statement is always true as the only currently supported format value is 'generic', but this rule exists for forward compatibility when, for instance, we will have 'mysql_native'. Updated storage initialization logic - in case when the storage was not empty and had no encrypted binlog files (its storage metadata file 'metadata.json' did not have the 'encryption' parameter) and we start PBS with encryption enabled (non-empty '<storage.encryption>' section), we now also update the storage metadata file (its 'encryption' key will be set to 'generic'). Added additional validation of the binlog metadata files - we now also make sure that if encryption is enabled for a certain binlog file record, then the storage must have its keyring object properly configured and this keyring must have the KEK ID specified in the record. Added new 'binsrv::encryption_config' struct based on 'util::nv_tuple' that now keeps information required to configure keyring (currently only 'uri' of the 'file://' scheme). Removed 'keyring_uri' field from the 'binsrv::encryption_config' struct. Added new optional section 'keyring' of type 'binstv::encryption_config'to the 'binsrv::main_config' class. Added new 'binlog_streaming.encryption' MTR test case that tests the case when individual binlog data files within the same storage have different encryption settings (some encrypted, some not, some with one cipher, some with another). Added new 'generate_keyring_data_file.inc' MTR include file that helps to generate a keyring data file with a predefined set of keys with different ciphers. Added new 'delete_keyring_data_file.inc' MTR include file that helps to delete the keyring data file. Extracted 'generate_binsrv_config.inc' MTR include file from the existing 'set_up_binsrv_environment.inc' which allows to regenerate the PBS configuration file with different parameters without need to call 'tear_down_binsrv_environment.inc'. Packaging specs updated to version '0.4.0'. 'main_config.json' sample configuration file updated to reflect 'keyring_uri' parameter rename / move. Updated 'README.md' to reflect configuration file changes.
…lization (Percona-Lab#174) https://perconadev.atlassian.net/browse/PBS-21 Implemented "automatic recovery" logic for storage initialization. In cases when PBS process was not gracefully shut down, we may observe the following artifacts in the storage: 1. There could be temporary files (e.g. 'binlog.000001.tmp') not properly deleted. 2. The actual size of the latest binlog data file may not match the 'size' field written to the corresponding metadata file. PBS now properly handles both situations and performs automatic recovery (returning the storage to its consistent state): 1. We write a 'warning' message to the log file for each '.tmp' file found and in case of 'streaming' operation mode, we also simply delete these files (for other modes we just ignore them). 2. For size mismatch we check if this mismatch happened in the latest (currently active) binlog file and simply truncate it to the size specified in the metadata file. We emit a warning to the log file for this case as well. 'binsrv::basic_storage_backend' interface extended with a new method 'resize_object()' that is supposed to set the new object size (in bytes). This method implemented in both 'binsrv::filesystem_storage_backend' and 'binsrv::s3_storage_backend' concrete classes. In this version the 'resize()' method for the S3 storage backend is implemented via downloading object content to a temporary file, truncating this temporary file, and then re-uploading it. However, this can be improved with Multipart Uploads - added a corresponding 'TODO:' item. 'binsrv::storage' class constructor now also accepts the 'logger' parameter that is saved in a new class member and used for generating additional log records (e.g. during the storage 'recovery'). Filtering for temporary files in the 'binsrv::filesystem_storage_backend' is no longer performed inside the 'do_list_objects()' method. Instead we now have 'binsrv::storage::remove_temporary_objects()' method with more advanced logic at the storage level that works for all concrete backend storage implementations. Added new 'binlog_streaming.recover' MTR test case that simulates various storage inconsistencies and checks the 'recovery' logic.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.