Skip to content

Replace legacy Base64 implementation with libbase64 - #116

Open
Belonit wants to merge 2 commits into
OpenTS-Developers:mainfrom
Belonit:replace-base64-codec
Open

Replace legacy Base64 implementation with libbase64#116
Belonit wants to merge 2 commits into
OpenTS-Developers:mainfrom
Belonit:replace-base64-codec

Conversation

@Belonit

@Belonit Belonit commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

The custom Base64 codec is legacy code used only for INI UUBlock serialization.
This replaces it with SIMD-accelerated libbase64 while keeping the existing UUBlock format compatible.
The UUBlock coverage is part of the existing IniContract test, not a separate test target.

@Belonit Belonit changed the title Replace base64 codec Replace legacy Base64 implementation with libbase64 Sep 2, 2026
@Belonit
Belonit marked this pull request as draft September 2, 2026 22:07
Comment thread code/ini.cpp
int outcount = b64pipe.Put(buffer, length);
total += outcount;
int filtered_count = 0;
for (int offset = 0; offset < length; offset++) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is all this, why is it needed?
The input should already be base64, I dont think the original implementation deviated?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The original decoder was different here: Base64_Decode explicitly ignored every character outside the Base64 alphabet and stopped at the first "=" character. Get_UUBlock reached it via Base64Pipe, so the existing permissive decoding behavior for old INI data is preserved.

I agree that this is more code than is required for the normal path.
filtering exists only for compatibility with the old decoder.

If we do not consider support for damaged/non-standard UUBlocks important, then the code in this place can be greatly simplified

@tomsons26 tomsons26 Sep 4, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should be moved to its own function that just validates the input and returns a log error if its invalid.
I don't see a logical reason to try to work around a bad input, bad input is bad input, the user needs to fix it.

Comment thread code/ini.cpp Outdated
@tomsons26

Copy link
Copy Markdown
Contributor

Is there a benchmark driven reason for this?
Original code was simple and compact, unless the perf gain is massive from this library i don't see a point in doing changes for change sake

@ZivDero

ZivDero commented Sep 3, 2026

Copy link
Copy Markdown
Member

Can we keep the pipe/straw classes? They're idiomatic here. Replace the actual base64 impl instead?

@Belonit

Belonit commented Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

Is there a benchmark driven reason for this?
Original code was simple and compact, unless the perf gain is massive from this library i don't see a point in doing changes for change sake

UUBlock encoding and decoding are not per-frame hot paths, so I do not think a benchmark is necessary. libbase64 should still be significantly faster for large blocks, and I can measure that if useful.

The primary motivation was maintenance: the bespoke codec plus Base64Pipe and Base64Straw were only used for UUBlock handling. libbase64 replaces that isolated implementation with a tested library while preserving the stored format

@Belonit

Belonit commented Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

Can we keep the pipe/straw classes? They're idiomatic here. Replace the actual base64 impl instead?

We can, but these are not general-purpose adapters in the current tree: Base64Pipe and Base64Straw are only used by the two UUBlock functions.

They also buffer a single Base64 quantum (PBuffer[3] / CBuffer[4]). Replacing only Base64_Encode and Base64_Decode would call libbase64 on 3 or 4 bytes at a time, which removes any meaningful SIMD benefit. To use libbase64 effectively, the adapters would need larger batching buffers and become a larger rewrite than the direct UUBlock implementation.

Keeping them is reasonable if preserving the local Pipe/Straw shape is more important than removing the unused abstraction. Otherwise, I think calling the library directly is the simpler result.

@ZivDero

ZivDero commented Sep 3, 2026

Copy link
Copy Markdown
Member

Can we keep the pipe/straw classes? They're idiomatic here. Replace the actual base64 impl instead?

We can, but these are not general-purpose adapters in the current tree: Base64Pipe and Base64Straw are only used by the two UUBlock functions.

They also buffer a single Base64 quantum (PBuffer[3] / CBuffer[4]). Replacing only Base64_Encode and Base64_Decode would call libbase64 on 3 or 4 bytes at a time, which removes any meaningful SIMD benefit. To use libbase64 effectively, the adapters would need larger batching buffers and become a larger rewrite than the direct UUBlock implementation.

Keeping them is reasonable if preserving the local Pipe/Straw shape is more important than removing the unused abstraction. Otherwise, I think calling the library directly is the simpler result.

Would it not be possible to rework the pipes to solve the buffering?

Keep the existing UUBlock INI format while enabling SIMD-accelerated encoding and decoding
@Belonit
Belonit force-pushed the replace-base64-codec branch from e99bf3c to 1d97c20 Compare September 3, 2026 12:25
@Belonit

Belonit commented Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

Can we keep the pipe/straw classes? They're idiomatic here. Replace the actual base64 impl instead?

We can, but these are not general-purpose adapters in the current tree: Base64Pipe and Base64Straw are only used by the two UUBlock functions.
They also buffer a single Base64 quantum (PBuffer[3] / CBuffer[4]). Replacing only Base64_Encode and Base64_Decode would call libbase64 on 3 or 4 bytes at a time, which removes any meaningful SIMD benefit. To use libbase64 effectively, the adapters would need larger batching buffers and become a larger rewrite than the direct UUBlock implementation.
Keeping them is reasonable if preserving the local Pipe/Straw shape is more important than removing the unused abstraction. Otherwise, I think calling the library directly is the simpler result.

Would it not be possible to rework the pipes to solve the buffering?

We can rework the adapters around libbase64’s streaming API, but I do not think they are needed here.
Base64Pipe and Base64Straw are only used by INIClass::Put_UUBlock and Get_UUBlock. They do not provide a reusable pipeline boundary or a public API, so retaining and expanding them would add an abstraction with no current consumer beyond these two functions.
Calling libbase64 directly keeps the UUBlock code smaller and makes the dependency explicit while preserving the stored format. I would prefer that unless there is a concrete planned use for Base64 Pipe/Straw outside UUBlock.

@Belonit
Belonit marked this pull request as ready for review September 3, 2026 12:34
@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown

Development builds of 1d97c20:

The links work without a GitHub account. Artifacts expire after 90 days, and this comment follows the latest successful build.

@Krisztiaan Krisztiaan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#113 overlaps with this btw

Comment thread code/ini.cpp
int copy_count = std::min(len - total, static_cast<int>(decoded_count));
std::memcpy(static_cast<char *>(block) + total, decoded, copy_count);
total += copy_count;
if (finished || total == len) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This stops the entire section at the first padding marker. Base64Pipe decoded each four-character group independently and continued into later entries, entries containing TQ== followed by TWFu change from MMan to M. Preserve that accepted input unless we want to break compat.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants