Solution: Using LittleFS on SD Cards with Large Log Files

I am not sure if this is the right place to throw this out there, but here goes. And apologies if this is reinventing someone else's wheel.

My system creates smallish numbers (<50) of large log files (1 to 200 megabytes) on SD Cards of typically 32-64G size, because that is what is cheap these days. I have run into all the same issues that people have discovered under these circumstances. I have worked out a solution that works really well for me. If nothing else, the solution might have broader application for other readers. Or who knows, maybe get incorporated into LittleFS in some fashion.

I start by dividing the SD card into chunks/partitions. This is done by dividing the entire SD Card's block count by a constant. I use 1024, but it could be anything. For a 64G card the chunks end up being about 64M. LittleFS is assigned chunk 0, the first one on the card. This helps with one problem right away: LittleFS never has to manage or search the full 64G address space: it only ever manages a much smaller 64M space.

For normal non-log files, LittleFS gets used directly.

For log files, I built a thin LogStore layer on top of LittleFS. The remaining 1023 chunks on the SD card are managed by the new LogStore layer. Tracking the busy/free state the 1024 chunks can be done with only 128 bytes of RAM. This free list is built at boot time, and stays in RAM so that chunk-allocation searches are very fast.

When a logfile gets opened at the LogStore level, it allocates a chunk from the RAM-resident freelist. The chunks are gigantic at 64M bytes, but it lines up OK with my expected use cases. The LogStore could manage up to 1023 log files (way more than the 50 I need), Most of my log files would fit in a single 64M chunk with a bunch of wastage, and some might take a couple of chunks.

To describe a log file, all I need is a small amount of metadata: current file length, and the order of the chunks that have been allocated to the log. The key to everything is this: all I use LittleFS for is to store that tiny amount of metadata.

You can probably see where this is going:

• The actual log writes are trivially fast: they go directly to SD block offsets within their chunk. I write data in 4K chunks, but it could be more or less. The ONLY time a write would be delayed is if it requires allocating a new 64M chunk. And that allocation requires a search of a 128-byte RAM data struct, so it is essentially zero-cost compared to anything that requires reading the SD card.
• After performing the log write directly into the chunk space, the log file's metadata object gets updated with the new log length and where the next log write will begin
• The metadata object is written back to LittleFS as a complete rewrite of the metadata file

LittleFS gets used as a reliable mechanism to commit/journal the LogStore writes to the chunk space. If the system crashes anywhere between writing the log data to the chunk and re-writing the metadata back to LittleFS, all that happens is that the latest data written to the chunk gets lost. I use LittleFS's guarantees that such things will not corrupt the LittleFS file system. And that means that the metadata files and the data they point at are covered by the same guarantee.

Of course, there are some extra costs. When the system boots, the LogStore has to scan every .meta file to reconstruct the chunk free list. My system gets this done in tens of milliseconds, so it is not expensive, and it is a one-time cost at boot time. And just like LittleFS, reconstructing a freelist instead of maintaining it means that there is nothing to get out of sync.

Another cost is that a LogStore log write takes about twice as long: writing the log data to a chunk offset, then writing the new metadata back to LittleFS. But the huge benefit is that this write cost is far more consistent than the cost of writing data to a LittleFS system that contains a lot of data. The LittleFS file system only stores a small number of very small files, which totally plays to its strengths.

Before implementing my LogStore, my original system needed to devote 96K of its RAM for buffering incoming log data for dealing with those situations where LittleFS would go on walkabout looking for a free block. Under worst-case conditions, that corresponded to about 7 seconds of incoming data before the 96K RAM buffer would overflow. The tough part for me was that once LittleFS had more than about 250 megabytes of data in its filesystem, a free-block search would exceed 7 seconds. That meant I was getting less than 0.4% utilization of the card's storage space before I would lose data.

With the LogStore approach, I have dropped the RAM buffer down to 16K, and honestly, I could drop it to 8K without fear of an incoming log buffer overflow even with an SD card that is 99% full of logfile data.

With regards to SD Card wear, I am not too worried. LittleFS can manage wear leveling inside its partition. If I really cared, I could add one more metadata file in LittleFS that defined a single number from 1 to 1023: the starting place to look for the next chunk to allocate in the RAM-based chunk freelist. Each chunk allocation would rewrite this value so that the chunk usage would get spread more evenly across the card space. But I really think that is an optimization more than a requirement.

To recap, LittleFS gets used for what it does best: reliable writes of small files within a small file system. The LogStore leverages that capability to reliably manage my extremely large data sets.

I hope someone finds this useful. And who knows, maybe it might be an interesting meta-feature to add to LittleFS. Basically, legitimize large log files, but keep their data out of direct management by LittleFS.