Sockets/file descriptors are left out open unless the body is consumed
Sockets/file descriptors are left out open unless the body is consumed. This seems to happen in both Linux and MacOS.
Reproduction
I wrote a resourceExists util function to see if the given URL exists or not. Bear in mind that I'd normally use HEAD method for this but in some of the URLs I have to test that method is not implemented so I have to use a regular GET request.
Now given this code in an opensockettest.js file:
import fetch from 'node-fetch';
const urls = [
'https://i.dailymail.co.uk/i/pix/2016/06/25/02/35A4954F00000578-0-image-a-13_1466819022937.jpg',
'https://i.dailymail.co.uk/i/pix/2016/02/05/17/30EAB51200000578-0-image-m-5_1454695002729.jpg',
'https://i.dailymail.co.uk/i/pix/2016/09/02/14/37D5E61C00000578-0-image-a-8_1472824415895.jpg',
'https://i.dailymail.co.uk/i/pix/2017/10/24/19/45A4723600000578-0-image-a-2_1508868932466.jpg',
'https://i.dailymail.co.uk/i/pix/2015/10/25/01/2DBFBEEA00000578-0-image-m-11_1445735767824.jpg',
'https://i.dailymail.co.uk/i/newpix/2018/09/12/21/50189AEE00000578-0-image-a-16_1536785559682.jpg',
'https://i.dailymail.co.uk/i/newpix/2018/06/14/11/4D38883100000578-0-image-a-1_1528971515562.jpg',
'https://i.dailymail.co.uk/i/newpix/2018/09/10/15/4FFCB43B00000578-0-image-a-1_1536591220607.jpg',
'https://i.dailymail.co.uk/i/newpix/2018/05/08/05/4BF69E0000000578-0-image-m-103_1525754036269.jpg',
'https://i.dailymail.co.uk/1/2018/06/14/11/2018/06/14/10/video-3293220-219_636x358.jpg',
'https://i.dailymail.co.uk/1s/2022/09/28/00/video-62865615-982_636x358.jpg',
'https://i.dailymail.co.uk/1s/2022/10/05/16/63142239-0-image-a-7_1664982798552.jpg',
'https://i.dailymail.co.uk/1s/2022/10/05/15/63140431-0-image-m-34_1664980059636.jpg',
'https://i.dailymail.co.uk/1s/2022/10/05/12/63134711-0-image-m-25_1664969549665.jpg',
'https://i.dailymail.co.uk/1s/2022/10/05/10/63131471-0-image-a-7_1664963489500.jpg',
'https://i.dailymail.co.uk/1s/2022/09/30/17/video-62976713-405_636x358.jpg',
'https://i.dailymail.co.uk/1s/2022/09/30/22/video-62986031-384_636x358.jpg',
'https://i.dailymail.co.uk/1s/2022/10/01/07/62994677-0-image-m-4_1664607016900.jpg',
'https://i.dailymail.co.uk/1s/2022/10/07/22/video-63243097-225_636x358.jpg',
'https://i.dailymail.co.uk/1s/2022/09/25/14/62780539-0-image-a-19_1664113310966.jpg',
'https://i.dailymail.co.uk/1s/2022/09/25/13/video-62779795-794_636x358.jpg',
'https://i.dailymail.co.uk/1s/2022/11/11/21/64461065-0-image-m-4_1668203772838.jpg',
'https://i.dailymail.co.uk/1s/2022/11/14/04/64515861-0-image-a-13_1668398628989.jpg',
'https://i.dailymail.co.uk/1s/2022/10/25/21/video-63844227-187_636x358.jpg',
'https://i.dailymail.co.uk/1s/2022/10/25/17/video-63836581-805_636x358.jpg'
];
const resourceExists = async (url) => {
try {
const response = await fetch(url);
return response.ok;
} catch (_) {
return false;
}
};
for (const url of urls) {
await resourceExists(url);
}
console.log('Done');
// give some time to look for open file descriptors
await new Promise((resolve) => setTimeout(resolve, 60000));
Then I run it and look for open descriptors and I see a few after Done has been printed out.
> node opensockettest.js
> ps aux | grep opensockettest
diego.manilla 49305 0.0 0.0 33740844 944 s017 S+ 12:18pm 0:00.00 grep opensocket
diego.manilla 49284 0.0 0.1 46050880 47636 s015 S+ 12:18pm 0:00.23 node opensockettest.js
> lsof -p 49284 | grep TCP | wc -l
5Expected behavior
No file descriptors/socket should be left dangling even if the body is not used/consumed. At the moment to get around this I have to use an abort controller since the Response class doesn't have any sort of close or destroy method (edit: it seems that Node introduced ReadableStream.cancel() in v16.5 so maybe doing response.body.cancel() is an option, but I haven't tested it; edit2: it doesn't seem to work). This way no sockets seem to be left open.
const resourceExists = async (url) => {
const controller = new AbortController();
try {
const response = await fetch(url, {signal: controller.signal});
return response.ok;
} catch (_) {
return false;
} finally {
controller.abort();
}
};If this is expected behaviour maybe it could be mentioned in the docs, I couldn't find anything.
Screenshots
N/A
Your Environment
| software | version |
|---|---|
| node-fetch | 3.3.0 |
| node | 16.11.1 |
| npm | 8.0.0 |
| Operating System | MacOS Monterey 12.6.1 |
Additional context
N/A.
Source: node-fetch/node-fetch