Fix excessive memory use with large external-level projects

[GameGuyy]

Fixes #1172

Large projects with external levels were crashing the renderer because Electron's V8 heap is capped around 4GB (pointer compression), and every level was kept in memory 3 times: the live data, a parsed LevelJson object, and the pretty-printed JSON string. With the crasher project from the issue this goes over the cap before loading even finishes.

Changes:

• For external-level projects, the level JSON cache no longer keeps the string/object in RAM. It just points at the level's own .ldtkl file on disk, and reads it back only when actually needed.
• Saving doesn't wipe and rewrite the whole levels folder anymore. Unchanged levels are skipped entirely, renamed levels get their file copied, and only dirty levels are re-serialized — one at a time, so the peak memory during a save is a single level. Orphaned .ldtkl files are cleaned up after the save succeeds.
• The main project JSON is built with layer instances skipped instead of serializing everything and stripping fields afterwards.
• Crash backups go through the same one-level-at-a-time path.

Single-file projects keep the old in-memory cache, nothing changes for them.

Tested with the crasher project from the issue (55 levels, ~276MB): loads and saves fine now, renderer stays under 1GB where it used to hit 4GB and die. Also checked: saving without edits touches only the main .ldtk file, editing one level rewrites only that level's file, level rename/delete, save as to another folder, 0.9.3 project migration, and close/reopen round-trips (multi-world sample included).

[GameGuyy]

Just wanted to bump this. It fixes a renderer OOM crash on large external-level projects (#1172). Tested against the crasher project from that issue: renderer memory stays under 1GB now, where it previously blew past the ~4GB V8 heap cap and crashed before load even finished. Happy to address any review feedback.

[deepnight]

Hi!

Thanks for that amazing PR, it will definitely help on large projects.

Two issues I may have spotted though:

1. Disk changes can silently conflict with the loaded project. I mean, the saver checks whether the cached file exists, then skips writing if source and destination match. But it never checks whether that file changed since loading. This could lead to very unexpected results if some versioning environment for example changes a file, and LDtk doesn't notice, keeping out of sync data in the main project file. My reco: store a file fingerprint with the cache and detect conflicts before reusing it

2. Multiline strings may lose newlines through forceSingleFile. Disk-cache reads return parsed JSON without the escaping previously performed by crawlObjectRec. The existing serializer then replaces actual newlines with spaces. I haven't tested the issue, but that something I had in the past with other similar reworks. Could you please double check this?

[GameGuyy]

Thanks for the review, both were real issues.

1. External changes: the cache now stores a fingerprint (mtime + size) of the level file when it's loaded or saved. Every time the cache is reused (skip, copy or read), the fingerprint is checked first. If the file changed or disappeared outside LDtk, the cache is dropped with a warning in the log and the level gets re-serialized from the loaded data on the next save, same as the old behavior.

2. Newlines: confirmed, reading the cached file skipped the crawlObjectRec escaping, so JsonPretty turned newlines into spaces in single-file output (last-chance save, for example). It now gets the same escaping as the in-memory cache.

Tested both in the app: a multi-line text field survives a single-file save and a reload, editing or deleting a .ldtkl outside LDtk gets detected and rewritten on save, and a save with no edits still doesn't touch any level file. The crasher project from #1172 still sits around 530MB of JS heap after loading and saving.

[deepnight]

I merged the change in dev-1.5.4 branch, I'll be testing it a bit locally before pushing it live. Thanks a LOT for this very important rework 🧡

[deepnight]

Coming back after a few days of field testing:

• The saving time now seems slower, while project loading is now faster 🤷
• The InMemory enum has a serialized json String that is actually never used. Maybe this would help with the previous saving issue
• I got a few "permission errors" while saving since the change. Not a big deal, as "retry" works just fine, but it seems that the file writing is some times blocked by windows.

The 1.5.4 branch is up to date with my latest fixes. Any idea for these? ☝️

[GameGuyy]

Thanks a lot for taking the time to test it!

Here's what I think is happening:

1. Permission errors: this is most likely from my change. Levels are now written to a temp file that is then renamed over the real one, and Windows sometimes blocks that rename for a moment when another program (antivirus, indexer, cloud sync) has the file open. That would explain why retry works.
2. Slower saving: it could be the same temp-file-and-rename step, or every level being rewritten on each save. If you can share a save log with your new timings, and say whether the project uses separate level files, I can tell which one it is.
3. Unused InMemory string: you're right, nothing reads it, so removing it makes sense. It should make single-file projects a bit faster, but it doesn't change how separate level files are saved.

Would you like me to send a fix for these on top of dev-1.5.4, or would you prefer to handle it your way? Happy either way.

[deepnight]

The latest version on 1.5.4 now has timestamps in the log file, which should make tracking down the culprit a bit easier 🤔
I will see if I can spot anything obvious there.