Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 24 additions & 1 deletion TODO.md
Original file line number Diff line number Diff line change
Expand Up @@ -1166,7 +1166,8 @@ only reproducible input.
opaque callee that returns its strict `arguments` costs 340 ns per call against
Node's 6.6 (`call-arguments-escape`). Dense storage needs define, delete,
freeze, and own-key order to understand it, and a move to the table once an
index gets non-default attributes.
index gets non-default attributes. In V8's EarleyBoyer, creating arguments
objects takes 19% of the main thread's samples.

- [ ] Deterministically release nonescaping RegExp and ICU handles at compiled scope
end once ownership analysis proves their lifetime. Retain finalization for every
Expand All @@ -1184,6 +1185,28 @@ contracts or investigates costs still visible after the string follow-ups.

## Focused performance work

- [ ] Build fewer primordials at startup. A closed-world `console.log` program
starts in 2.1 ms against Porffor's 1.3 and a 1.2 ms spawn floor, and
`mal_intrinsics_init` takes about 0.7 ms of that. The cost is spread over the
builtin installs at 10 to 50 µs each, so only lazy installs for families a
program never reaches, or a prebuilt heap image, would remove most of it.
- [ ] Search UTF-16 subjects with regress start predicates. regress 0.11.1 skips its
start-position prefilter whenever the `utf16` feature is on, so a pattern that
can only start at `<` tries a backtracking match at every code unit. Upstream
master searches UTF-16 input with unit predicates, which takes
`/<script|<style|<link/` over a 35 KB UTF-16 subject from 309 to 25 ms per 1000
searches, but it is unreleased. Porffor's `regex` benchmark stays
backtracking-bound either way: its first pattern takes 890 ms in regress alone,
against Porffor's 432 and Bun's 145. Its tag-name run
`[^\/\0>\x20\t\r\n\f]*` is followed by `[^>]*`, which accepts every unit the
first run gives back, so treating such a run as possessive would remove most
of that work.
- [ ] Admit array literals whose constant keys LICM hoisted into a later loop's
preheader. The native initialization region resolves keys only from the
function's entry chain or the instructions just before the allocation. A
13-element literal in a second loop that shares keys with an earlier literal
therefore falls back to `createArray` and one `defineProperty` per element:
4M of them take 270 ms against Bun's 21.
- [ ] Keep medium flat strings' units in their own cell. A string past the 16
Latin-1 or 8 UTF-16 inline units allocates a separate owned buffer that a
finalizer frees, so building `${order.region}/${order.category}` costs 27.7 ns
Expand Down
23 changes: 14 additions & 9 deletions runtime/secret_buffer_test_main.c
Original file line number Diff line number Diff line change
Expand Up @@ -54,15 +54,19 @@ static void secret_observation_reset(void) {
memset(&g_observed, 0, sizeof(g_observed));
}

/* An ordinary store this small would live in its cell and never reach the release
* observer, so the ordinary control uses one past the inline limit. */
#define PLAIN_LENGTH (MAL_ARRAY_BUFFER_INLINE_MAX_BYTES + SECRET_LENGTH)

static MalArrayBufferObject *secret_buffer_new(MalVm *vm, bool sensitive) {
MalObject *prototype =
mal_value_to_object(vm->intrinsics[MAL_INTRINSIC_ARRAY_BUFFER_PROTOTYPE]);
u32 length = sensitive ? SECRET_LENGTH : PLAIN_LENGTH;
MalArrayBufferObject *buffer = sensitive
? mal_array_buffer_object_new_sensitive(&vm->heap, prototype, SECRET_LENGTH)
: mal_array_buffer_object_new(
&vm->heap, prototype, SECRET_LENGTH, SECRET_LENGTH, false, false);
? mal_array_buffer_object_new_sensitive(&vm->heap, prototype, length)
: mal_array_buffer_object_new(&vm->heap, prototype, length, length, false, false);
if (buffer == nullptr || buffer->data == nullptr) return nullptr;
memset(buffer->data, SECRET_BYTE, SECRET_LENGTH);
memset(buffer->data, SECRET_BYTE, length);
g_observed.target = (uptr) buffer;
return buffer;
}
Expand Down Expand Up @@ -128,8 +132,9 @@ static bool a_shrunken_store_is_scrubbed_to_its_capacity(MalVm *vm) {
mal_value_to_object(vm->intrinsics[MAL_INTRINSIC_ARRAY_BUFFER_PROTOTYPE]);
MalArrayBufferObject *buffer = mal_array_buffer_object_new(
&vm->heap, prototype, SECRET_LENGTH, SECRET_LENGTH, true, false);
if (buffer->data == nullptr) return false;
buffer->sensitive = true;
if (buffer->data == nullptr || !mal_array_buffer_object_set_sensitive(&vm->heap, buffer, true)) {
return false;
}
memset(buffer->data, SECRET_BYTE, SECRET_LENGTH);
g_observed.target = (uptr) buffer;
bool shrunk = mal_array_buffer_object_resize(buffer, 8);
Expand Down Expand Up @@ -215,11 +220,11 @@ static NativeStore native_store(MalVm *vm, bool adopted, u32 length, u32 capacit
MalArrayBufferObject *buffer = adopted
? mal_array_buffer_object_adopt(&vm->heap, prototype, malloc(capacity), capacity, true)
: mal_array_buffer_object_new(&vm->heap, prototype, length, capacity, resizable, false);
buffer->sensitive = true;
if (!mal_array_buffer_object_set_sensitive(&vm->heap, buffer, true)) abort();
if (buffer->data != nullptr) memset(buffer->data, SECRET_BYTE, buffer->allocation_capacity);
if (resizable && !mal_array_buffer_object_resize(buffer, 8)) abort();
MalArrayBufferStoreLease *lease = mal_array_buffer_object_retain_store(buffer);
MalArrayBufferStoreLease *other = mal_array_buffer_object_retain_store(buffer);
MalArrayBufferStoreLease *lease = mal_array_buffer_object_retain_store(&vm->heap, buffer);
MalArrayBufferStoreLease *other = mal_array_buffer_object_retain_store(&vm->heap, buffer);
if (lease == nullptr || other == nullptr) abort();
mal_array_buffer_store_release(other);
return (NativeStore) {lease, buffer->data, buffer->allocation_capacity, (uptr) buffer, false};
Expand Down
94 changes: 74 additions & 20 deletions runtime/src/array_buffer_object.c
Original file line number Diff line number Diff line change
Expand Up @@ -29,9 +29,33 @@ static void release_store_data(byte *data, u32 capacity, const MalArrayBufferRel
mal_gc_process_release(capacity);
}

MalArrayBufferStoreLease *mal_array_buffer_object_retain_store(MalArrayBufferObject *buffer) {
bool mal_array_buffer_object_move_out_of_cell(MalHeap *heap, MalArrayBufferObject *buffer) {
u32 capacity = buffer->allocation_capacity;
byte *data = malloc(capacity);
if (data == nullptr) return false;
memcpy(data, buffer->inline_bytes, capacity);
buffer->data = data;
mal_heap_cell_needs_finalizer(&buffer->object.header);
mal_heap_array_buffer_acquire(heap, capacity);
mal_gc_process_charge(capacity);
mal_profile_native_allocation(heap, capacity, MAL_PROFILE_ALLOCATION_FAMILY_BUFFER);
return true;
}

bool mal_array_buffer_object_set_sensitive(
MalHeap *heap, MalArrayBufferObject *buffer, bool sensitive
) {
if (sensitive && !mal_array_buffer_object_externalize(heap, buffer)) return false;
buffer->sensitive = sensitive;
return true;
}

MalArrayBufferStoreLease *mal_array_buffer_object_retain_store(
MalHeap *heap, MalArrayBufferObject *buffer
) {
if (buffer->shared || buffer->detached
|| (buffer->data == nullptr && buffer->allocation_capacity > 0)) return nullptr;
|| (buffer->data == nullptr && buffer->allocation_capacity > 0)
|| !mal_array_buffer_object_externalize(heap, buffer)) return nullptr;
MalArrayBufferStoreLease *lease = buffer->store_lease;
if (lease == nullptr) {
lease = malloc(sizeof(*lease));
Expand Down Expand Up @@ -68,14 +92,15 @@ struct MalArrayBufferPin {
u32 references;
};

MalArrayBufferPin *mal_array_buffer_object_retain_pin(MalArrayBufferObject *buffer) {
if (buffer->pins == UINT32_MAX) return nullptr;
MalArrayBufferPin *mal_array_buffer_object_retain_pin(MalHeap *heap, MalArrayBufferObject *buffer) {
if (buffer->pins == UINT32_MAX || !mal_array_buffer_object_externalize(heap, buffer)) return nullptr;
MalArrayBufferPin *pin = buffer->retained_pin;
if (pin == nullptr) {
pin = malloc(sizeof(*pin));
if (pin == nullptr) return nullptr;
*pin = (MalArrayBufferPin) {.buffer = buffer};
buffer->retained_pin = pin;
mal_heap_cell_needs_finalizer(&buffer->object.header);
}
pin->references++;
buffer->pins++;
Expand All @@ -98,23 +123,35 @@ static MalArrayBufferObject *mal_array_buffer_object_new_impl(
u32 max_byte_length,
bool resizable,
bool shared,
bool initialize
bool initialize,
bool sensitive
) {
MalArrayBufferObject *buffer = mal_heap_alloc(heap, sizeof(MalArrayBufferObject), MAL_HEAP_ARRAY_BUFFER_OBJECT);
mal_object_init(heap, &buffer->object, MAL_HEAP_ARRAY_BUFFER_OBJECT, prototype);

// Resizable buffers reserve the maximum so the backing store never moves.
u32 capacity = resizable ? max_byte_length : byte_length;
// A small fixed store lives in the cell, which allocation already zeroed, so
// the sweep reclaims it with the cell instead of finalizing and freeing it.
bool store_inline = !resizable && !shared && !sensitive && capacity > 0
&& capacity <= MAL_ARRAY_BUFFER_INLINE_MAX_BYTES;
MalArrayBufferObject *buffer = mal_heap_alloc(
heap, sizeof(MalArrayBufferObject) + (store_inline ? capacity : 0),
MAL_HEAP_ARRAY_BUFFER_OBJECT);
mal_object_init(heap, &buffer->object, MAL_HEAP_ARRAY_BUFFER_OBJECT, prototype);

buffer->data = nullptr;
buffer->shared_memory = nullptr;
if (store_inline) {
buffer->data = buffer->inline_bytes;
} else {
mal_heap_cell_needs_finalizer(&buffer->object.header);
}
if (shared) {
// Even a zero-length SharedArrayBuffer needs a backing: its identity is
// what clones and wait/notify keys share across isolates.
buffer->shared_memory = mal_shared_memory_new(byte_length, max_byte_length, resizable);
if (buffer->shared_memory != nullptr) {
buffer->data = mal_shared_memory_data(buffer->shared_memory);
}
} else if (capacity > 0) {
} else if (capacity > 0 && !store_inline) {
// Only currently exposed bytes require zero semantics. Reserving a much
// larger resizable capacity must not eagerly dirty every future page;
// resize zeroes each newly exposed range before publishing its length.
Expand All @@ -127,7 +164,7 @@ static MalArrayBufferObject *mal_array_buffer_object_new_impl(
buffer->data = calloc(capacity, 1);
}
}
if (buffer->data != nullptr && !shared) {
if (buffer->data != nullptr && !shared && !store_inline) {
mal_heap_array_buffer_acquire(heap, capacity);
mal_gc_process_charge(capacity);
mal_profile_native_allocation(
Expand All @@ -140,7 +177,7 @@ static MalArrayBufferObject *mal_array_buffer_object_new_impl(
buffer->detached = false;
buffer->shared = shared;
buffer->immutable = false;
buffer->sensitive = false;
buffer->sensitive = sensitive;
buffer->pins = 0;
buffer->retained_pin = nullptr;
buffer->store_lease = nullptr;
Expand All @@ -157,16 +194,16 @@ MalArrayBufferObject *mal_array_buffer_object_new(
bool shared
) {
return mal_array_buffer_object_new_impl(
heap, prototype, byte_length, max_byte_length, resizable, shared, true);
heap, prototype, byte_length, max_byte_length, resizable, shared, true, false);
}

MalArrayBufferObject *mal_array_buffer_object_new_uninitialized(
MalHeap *heap, MalObject *prototype,
u32 byte_length, u32 max_byte_length,
bool resizable, bool shared) {
bool resizable, bool shared, bool sensitive) {
return mal_array_buffer_object_new_impl(
heap, prototype, byte_length, max_byte_length,
resizable, shared, false);
resizable, shared, false, sensitive);
}

MalArrayBufferObject *mal_array_buffer_object_move_store(
Expand All @@ -179,14 +216,28 @@ MalArrayBufferObject *mal_array_buffer_object_move_store(
(resizable && byte_length > max_byte_length)) {
abort();
}
if (mal_array_buffer_object_store_is_inline(source)) {
// Inline bytes die with the source cell, so they are copied rather than moved.
// An inline store is fixed and never sensitive, and the copy fits inline too.
MalArrayBufferObject *result = mal_array_buffer_object_new_impl(
heap, prototype, byte_length, max_byte_length, resizable, false, true, false);
u32 kept = byte_length < source->byte_length ? byte_length : source->byte_length;
memcpy(result->data, source->data, kept);
result->immutable = immutable;
source->data = nullptr;
source->byte_length = 0;
source->allocation_capacity = 0;
source->detached = true;
return result;
}
if (byte_length > source->byte_length) {
memset(source->data + source->byte_length, 0,
byte_length - source->byte_length);
}
// The process charge travels with the allocation: the empty header charged
// nothing, and the detached source no longer frees it.
MalArrayBufferObject *result = mal_array_buffer_object_new_impl(
heap, prototype, 0, 0, false, false, false);
heap, prototype, 0, 0, false, false, false, false);
result->data = source->data;
result->store_lease = mal_array_buffer_object_take_store_lease(source);
result->byte_length = byte_length;
Expand All @@ -207,10 +258,8 @@ MalArrayBufferObject *mal_array_buffer_object_move_store(
MalArrayBufferObject *mal_array_buffer_object_new_sensitive(
MalHeap *heap, MalObject *prototype, u32 byte_length
) {
MalArrayBufferObject *buffer =
mal_array_buffer_object_new(heap, prototype, byte_length, byte_length, false, false);
buffer->sensitive = true;
return buffer;
return mal_array_buffer_object_new_impl(
heap, prototype, byte_length, byte_length, false, false, true, true);
}

MalArrayBufferObject *mal_array_buffer_object_adopt(
Expand All @@ -236,7 +285,7 @@ MalArrayBufferObject *mal_array_buffer_object_adopt(
MalArrayBufferObject *mal_array_buffer_object_wrap_shared(
MalHeap *heap, MalObject *prototype, MalSharedMemory *memory) {
MalArrayBufferObject *buffer = mal_array_buffer_object_new_impl(
heap, prototype, 0, 0, false, false, false);
heap, prototype, 0, 0, false, false, false, false);
mal_shared_memory_retain(memory);
buffer->shared_memory = memory;
buffer->data = mal_shared_memory_data(memory);
Expand Down Expand Up @@ -280,6 +329,11 @@ void mal_array_buffer_object_release_store(MalHeap *heap, MalArrayBufferObject *
buffer->allocation_capacity = 0;
return;
}
if (mal_array_buffer_object_store_is_inline(buffer)) {
buffer->data = nullptr;
buffer->allocation_capacity = 0;
return;
}
MalArrayBufferStoreLease *lease = mal_array_buffer_object_take_store_lease(buffer);
if (buffer->data != nullptr) mal_heap_array_buffer_release(heap, buffer->allocation_capacity);
if (lease != nullptr) mal_array_buffer_store_release(lease);
Expand Down
53 changes: 45 additions & 8 deletions runtime/src/array_buffer_object.h
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,14 @@ typedef struct MalArrayBufferStoreLease MalArrayBufferStoreLease;
/**
* ArrayBuffer / SharedArrayBuffer backing store.
*
* `data` is a native allocation. A store lease can retain its bytes independently
* of the GC wrapper; ordinary stores are released by detach or the GC sweep.
* An ordinary store charges its allocation_capacity to the process budget
* `data` is a native allocation, or the cell's own `inline_bytes` for a small
* fixed-length store. An inline store dies with its cell, so such a buffer owns
* nothing for the sweep to finalize; it moves to a native allocation before its
* bytes can outlive the cell (a pin, a store lease, a transfer) or must be scrubbed
* (sensitive). The heap never moves cells, so inline bytes stay put while the
* buffer lives. A store lease can retain native bytes independently of the GC
* wrapper; ordinary stores are released by detach or the GC sweep.
* A native store charges its allocation_capacity to the process budget
* (gc_process.h) once; the charge moves with the allocation (move_store,
* structured-clone transfer) and is released only where it is freed. Shared
* backings are charged by shared_memory.c instead.
Expand Down Expand Up @@ -52,14 +57,46 @@ typedef struct MalArrayBufferObject {
u32 pins;
MalArrayBufferPin *retained_pin;
MalArrayBufferStoreLease *store_lease;
alignas(16) byte inline_bytes[];
} MalArrayBufferObject;

/** A VM-thread pin that can be retired after its buffer wrapper has been swept. */
MalArrayBufferPin *mal_array_buffer_object_retain_pin(MalArrayBufferObject *buffer);
/** Largest fixed-length ordinary store kept inside its cell. */
#define MAL_ARRAY_BUFFER_INLINE_MAX_BYTES 256u

static inline bool mal_array_buffer_object_store_is_inline(const MalArrayBufferObject *buffer) {
return buffer->data == buffer->inline_bytes;
}

/** Whether a dead buffer holds storage outside its cell that its finalizer must release. */
static inline bool mal_array_buffer_object_owns_storage(const MalArrayBufferObject *buffer) {
return (buffer->data != nullptr && !mal_array_buffer_object_store_is_inline(buffer))
|| buffer->shared_memory != nullptr || buffer->retained_pin != nullptr
|| buffer->store_lease != nullptr;
}

bool mal_array_buffer_object_move_out_of_cell(MalHeap *heap, MalArrayBufferObject *buffer);

/** Move an inline store to a native allocation; false when that allocation fails. */
static inline bool mal_array_buffer_object_externalize(
MalHeap *heap, MalArrayBufferObject *buffer
) {
return !mal_array_buffer_object_store_is_inline(buffer)
|| mal_array_buffer_object_move_out_of_cell(heap, buffer);
}

/** Apply the scrub-before-release contract, moving an inline store out of its cell
* first so the sweep scrubs it; false when that allocation fails. */
bool mal_array_buffer_object_set_sensitive(
MalHeap *heap, MalArrayBufferObject *buffer, bool sensitive);

/** A VM-thread pin that can be retired after its buffer wrapper has been swept.
* Pinned bytes never live in the cell, since C may hold them past the wrapper. */
MalArrayBufferPin *mal_array_buffer_object_retain_pin(MalHeap *heap, MalArrayBufferObject *buffer);
void mal_array_buffer_object_release_pin(MalArrayBufferPin *pin);

/** Retain ordinary backing bytes independently of their wrapper. Does not pin it. */
MalArrayBufferStoreLease *mal_array_buffer_object_retain_store(MalArrayBufferObject *buffer);
MalArrayBufferStoreLease *mal_array_buffer_object_retain_store(
MalHeap *heap, MalArrayBufferObject *buffer);
/** Retire a native owner on any thread; the last owner scrubs and frees the store. */
void mal_array_buffer_store_release(MalArrayBufferStoreLease *lease);
/** Move the lease's wrapper reference alongside a raw store ownership transfer. */
Expand All @@ -84,12 +121,12 @@ MalArrayBufferObject *mal_array_buffer_object_new(
* This is only for leaf native constructors that overwrite the complete visible
* range before the object can become observable. Hidden resizable capacity is
* still zeroed by resize immediately before exposure. Prefer the initialized
* constructor otherwise.
* constructor otherwise. `sensitive` selects the scrub-before-release contract.
*/
MalArrayBufferObject *mal_array_buffer_object_new_uninitialized(
MalHeap *heap, MalObject *prototype,
u32 byte_length, u32 max_byte_length,
bool resizable, bool shared);
bool resizable, bool shared, bool sensitive);

/**
* Move an existing store into a fresh ArrayBuffer header without copying its
Expand Down
2 changes: 1 addition & 1 deletion runtime/src/array_object.h
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ static inline void mal_array_object_card(MalArrayObject *array, u32 index, MalVa
}
mal_gc_array_card_slow(owner, index, value);
}
#define MAL_ARRAY_INLINE_CAPACITY_MAX 8u
#define MAL_ARRAY_INLINE_CAPACITY_MAX 14u

static_assert(sizeof(MalArrayObject) <= 64, "MalArrayObject outgrew its 64-byte size class");

Expand Down
Loading
Loading