Support for two LFS regions and on-ESP LFS building

New feature

Additional support for two LFS regions, and the ability to both save and load (update) LFS regions on ESP.

Justification

Most committers and many developers would seem to want this.

Highlights

This enhancement will be for Lua 5.3 only as this builds upon the groundwork that I've already laid in the Lua53 implementation.

• Nonetheless for technical reasons this will be a breaking change in terms of compiled Lua file formats. The old formats will not be supported so any image and compiled LC files will need recompiling. However, we will detect and throw an "old version" error if an old format file is used.
• LFS and LC file formats will be the same, albeit slightly different to the existing formats (as discussed below).
• lua_dump() can now take an array argument as well as a function argument at ToS. If an array is specified then the dump stream will contain all functions in the array.
• An extra node.dumpfile() function essentially does the same as string.dump() but writing the dump stream direct to file without needing to assemble and store the compiled content in RAM. (This scales much better.) Both of these functions also support the "array of functions" argument type.
• lua_load() (and derived functions) now support the multi-function format, and in which case return a keyed array instead of a single function. The array is of the form {"name" = nameFunc, ...}
• loadfile() and node.LFS.reload() can take an array of filenames as well as a single file, in which case the set of files will be loaded into RAM or LFS. In the case of loadfile this will return the {"name" = nameFunc, ...} array if multiple files or multi-function files are loaded. In the case of the LFS reload, this array is not returned but is used as LFS region's index ROTable after restart.
• Note that node.LFS.reload() can only process compiled files and will error on any source files.
• An RCR option is used to determine whether LFS1, LFS2, LFS1 + LFS2, or LFS2 + LFS1 is used for string and function resolution. Hence a single LFS can be used; either can be used in flip-flop mode; or they can be chained in a System / App configuration.
• When used in a chained configuration there is an inherent parent / child precedence implied. (Let's use the terms sysLFS and appLFS for convenience. In an LFS1 + LFS2 configuration:
  • The sysLFS is loaded into LFS1 as a parent LFS.
  • The appLFS is loaded into LFS2 as a "chained" LFS and this depends on sysLFS. The appLFS can be reloaded without invalidating the sysLFS.
  • ~~The appLFS will inherit any strings already in sysLFS ROstrt and hence these TStrings will not be duplicated in appLFS. The ROstrtfor the appLFS also includes all (short) TStrings in sysLFS and hence the sysLFS ROstrt isn't used. The G(L) global state points to this new appLFS ROstrt. From a string lookup perspective there is no difference between a single and a double LFS configuration. (Not true. I do need two ROstrt indexes because overflow chaining depends on the table size and I can't redo these chains in the sysLFS when loading the appLFS.)
  • The index ROTable for the appLFS has a methamethod __index entry pointing to the sysLFS index ROTable. Hence resolution across the two ROtables is "free" using standard Lua RTS array access.
  • Reloading the sysLFS will invalidate the appLFS and so it will need reloading as well
• Change the RCR option does the expected integrity checks on any loaded LFS image and invalidates them when necessary. So for example if both LFS1 and LFS2 contain a parent LFS then the option can be toggled at will. If it is changed to LFS1 + LFS2 then the LFS2 will be invalidated. Etc.
• The node.LFS.list property has already been updated to a function which takes an optional argument: 'parent' lists only those functions in the parent LFS and likewise for 'child'; 'system' and 'application' are synonyms for these options. Omitting the argument list all functions in both LFSs if configured as a parent / child pair.

Example usecases

-- Create a (child) LFS based on all LC files in SPIFFS
do
  local f,a = files.list('%.lc$'),{}
  for k in pairs(f) do a[#a+1] = k end
  node.LFS.reload(a)
end

-- Create a new (child) LFS replacing one specfic function from SPIFFS
do
  local v = node.LFS.list('application')
  local a = loadfile {'mysub.lua'} -- use array form to return an array
  for _,n in ipairs(v) do
    if not a[n] then a[n] = node.LFS.get(n) end
  end
  v = {}
  for n,f in pairs(a) do v[#v+1] = f end
  node.dumpfile(v, 'lfs.img')
  node.LFS.reload('lfs.img')
end

How the NodeMCU binary format differs from standard Lua 5.3

In general terms the Lua RTS dump function determinately traverses a Proto hierarchy converting all fields to a stream of binary tokens and this stream is the compiled file format. The load executes an "undump" which does the inverse traverse recreating the Proto hierarchies. This much is the same. But as to why the differences:

• File size and RAM usage is a lot more important to us so for example:

  • The default line encoding for traceback on the last code example is 192 bytes in the standard RTS but we use a run length encoding that takes 12 byte.
  • The standard tokenisation is aimed at simplicity so an integer is represented by 1+4 byte stream. We use a multibyte stream so in our case the constant 1 takes a single byte.

• I have also reordered the dump walk so that compiled code on reloading can be written sequentially to the LFS region using spi_flash_write() operations.

• When loading one or more Proto hierarchies in a file into LFS we need to add any TString constants that are not already in LFS. I do this by maintaining in RAM a copy of what will become the ROstrt for the LFS. This is a lookup that allows fast resolution against TString clashes, but in case of a clash I still need to compare the new TString against the copy in LFS to differentiate between a true match and a hash duplication. This uses ICACHE resolved access, and so I need to flush the cache to ensure cache coherence. I would rather do this once per file rather than once per proto.

• Hence the dump process collects the array of TStrings used in the dump and appends this as a string vector at the end of the file. Any inline TString references use an index into this vector.

• The NodeMCU file format includes a fixed header which includes a file CRC and the offset of this TString vector. The undumper fseeks to the TString vector and processes this first before fseeking back to the start of the file to process the Protos. This avoids the need for two passes during dump.

• The CRC-32 is at a fixed offset from the start of the file and can be used as an image ID, and node.LFS.verify(image) will check that the dump format is the current version and return this checksum. (Special request from @HHHartmann.) It can optionally checksum the image. The checksum is probably worth doing before reloading LFS

Technical Issues

Cache coherence.

I currently do a botch to flush the ICACHE, and that is to read a sequential 32Kb address window in flash. @jmattsson: Q: do you know a better way?