Improve available memory estimation
Which package is the feature request for? If unsure which one to select, leave blank
@crawlee/utils
Feature
Using this simple resource monitor
import { getMemoryInfo } from '@crawlee/utils';
// Print memory
const memoryInfo = await getMemoryInfo();
const toMB = (bytes) => Math.round(bytes / 1024 / 1024);
console.log('Available memory', {
totalMB: toMB(memoryInfo.totalBytes),
freeMB: toMB(memoryInfo.freeBytes),
usedMB: toMB(memoryInfo.usedBytes),
mainProcessMB: toMB(memoryInfo.mainProcessBytes),
childProcessesMB: toMB(memoryInfo.childProcessesBytes),
});
// Allocate ~200 MB
const ALLOCATION_MB = 200;
console.log(`Trying to allocate ${ALLOCATION_MB} MB of memory`);
const ballast = Buffer.alloc(ALLOCATION_MB * 1024 * 1024).fill(1);
console.log(`Allocated ${toMB(ballast.byteLength)} MB of memory`);
// Print memory again
const memoryInfoAfter = await getMemoryInfo();
console.log('Memory after allocation', {
totalMB: toMB(memoryInfoAfter.totalBytes),
freeMB: toMB(memoryInfoAfter.freeBytes),
usedMB: toMB(memoryInfoAfter.usedBytes),
mainProcessMB: toMB(memoryInfoAfter.mainProcessBytes),
childProcessesMB: toMB(memoryInfoAfter.childProcessesBytes),
});And running it with cgroups-imposed constraints, we see it fails to detect the limits. Limits are imposed, which are proven by crash.
1 - Unrestricted run - we see that it consumes around 275 MB (default mem consumption + 200MB ballast )
.../actors/jscrawlee$ systemd-run --user --scope --quiet --no-ask-password -p MemorySwapMax=0 pnpm start
> [email protected] start .../actors/jscrawlee
> node src/main.js
Available memory {
totalMB: 31365,
freeMB: 17076,
usedMB: 14289,
mainProcessMB: 74,
childProcessesMB: 0
}
Trying to allocate 200 MB of memory
Allocated 200 MB of memory
Memory after allocation {
totalMB: 31365,
freeMB: 16863,
usedMB: 14502,
mainProcessMB: 275,
childProcessesMB: 0
}2 - Restricted run to 190 MB - we see that it is killed when it tries to allocate an additional 200MB ballast
.../actors/jscrawlee$ systemd-run --user --scope --quiet --no-ask-password -p MemoryMax=190M -p MemorySwapMax=0 pnpm start
> [email protected] start .../actors/jscrawlee
> node src/main.js
Available memory {
totalMB: 31365,
freeMB: 16952,
usedMB: 14413,
mainProcessMB: 72,
childProcessesMB: 0
}
Trying to allocate 200 MB of memory
Killed3 - Restricted run to 300 MB - we see that it manages to allocate memory for the ballast 200 MB
.../actors/jscrawlee$ systemd-run --user --scope --quiet --no-ask-password -p MemoryMax=300M -p MemorySwapMax=0 pnpm start
> [email protected] start .../actors/jscrawlee
> node src/main.js
Available memory {
totalMB: 31365,
freeMB: 17041,
usedMB: 14324,
mainProcessMB: 71,
childProcessesMB: 0
}
Trying to allocate 200 MB of memory
Allocated 200 MB of memory
Memory after allocation {
totalMB: 31365,
freeMB: 16832,
usedMB: 14533,
mainProcessMB: 271,
childProcessesMB: 0
}getMemoryInfo fails to detect the memory limits.
Also seems like Docker container started with these arguments will failt to detect correct memory -m 300m --cgroupns=host
Motivation
Improved memory estimation investigated in Python tooling
Ideal solution or implementation, and any additional constraints
Resource reporting is more robust and working correctly in more environments
Alternative solutions or implementations
No response
Other context
No response
Source: apify/crawlee