High memory usage when saving workbooks written with StreamWriter
Description
The StreamWriter keeps memory usage flat while writing rows by spilling worksheet XML to a temporary file every 16 MB. However, that benefit is lost when the workbook is saved: Save, SaveAs, and Write all route through File.WriteTo, which calls WriteToBuffer and assembles the entire compressed zip archive in an in-memory bytes.Buffer before writing a byte to the destination. Peak memory therefore scales with the output file size (plus bytes.Buffer growth doubling), which defeats the purpose of the streaming writer for large workbooks (1 million rows and more).
Two smaller related problems in the same path:
writeToZipcopies the stream temp file into the archive withio.Copy's default 32 KB buffer; on a 1M-row file these smallpreadsyscalls account for ~46% of total CPU time in profiles.- The
tempFilesloop usesreadBytes, which loads each on-disk part fully into memory and also stores it intof.Pkgpermanently.
Steps to reproduce the issue
- Write 1,000,000 rows (10 cells each) with
NewStreamWriter/SetRow/Flush— memory stays flat (~22 MB heap) as expected. - Call
SaveAsand observe peak memory while saving.
package main
import (
"fmt"
"runtime"
"strconv"
"github.com/xuri/excelize/v2"
)
func heapMB() uint64 {
var m runtime.MemStats
runtime.GC()
runtime.ReadMemStats(&m)
return m.HeapAlloc >> 20
}
func main() {
f := excelize.NewFile()
defer f.Close()
sw, err := f.NewStreamWriter("Sheet1")
if err != nil {
fmt.Println(err)
return
}
row := make([]interface{}, 10)
for r := 1; r <= 1000000; r++ {
for c := 0; c < 10; c++ {
if c%2 == 0 {
row[c] = r + c
} else {
row[c] = "value-" + strconv.Itoa(r+c)
}
}
cell, _ := excelize.CoordinatesToCellName(1, r)
if err := sw.SetRow(cell, row); err != nil {
fmt.Println(err)
return
}
}
if err := sw.Flush(); err != nil {
fmt.Println(err)
return
}
fmt.Println("heap after write:", heapMB(), "MB") // ~22 MB, streaming works
if err := f.SaveAs("Book1.xlsx"); err != nil {
fmt.Println(err)
return
}
fmt.Println("heap after save:", heapMB(), "MB") // memory scaled with file size
}Describe the results you received
Peak memory during save grows linearly with the output file size (peak heap sampled every 10 ms by a background goroutine reading runtime.MemStats; rows above the 1,048,576 worksheet limit split across multiple sheets):
| Rows | Output size | Peak heap | Peak process memory |
|---|---|---|---|
| 1,000 | 0.3 MB | 3 MB | 12 MB |
| 1,000,000 | 41 MB | 133 MB | 183 MB |
| 10,000,000 (10 sheets) | 419 MB | 1,098 MB | 1,431 MB |
For workbooks whose archive approaches gigabytes, saving requires a multiple of the archive size in RAM, even though all row data was carefully streamed to disk during writing.
Describe the results you expected
Saving a workbook written with the stream writer should use constant memory regardless of the output size — the archive entries already exist on disk (stream temp files) or as in-memory parts, so they can be streamed into the zip archive and to the destination writer directly. Full in-memory buffering should only be necessary where it is genuinely required, such as password encryption, which needs the complete archive bytes.
Go version
go version go1.25.11 darwin/arm6
Excelize version or commit ID
v2.11.0
Environment
macOS 14.1 (23B2073), Apple M3 Pro (arm64, 11 cores), 18 GB RAM, SSD storagValidations
- Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
- The provided reproduction is a minimal reproducible example of the bug.
Source: qax-os/excelize