The problem appears
A few weeks ago, the infrastructure team here at Frameable received a report from one of our colleagues that made us feel like the floor had dropped out from under us. Our internal monitoring system – codename ‘Canary’ – had been silently failing for days!
Canary is a small, simple NodeJs app that once a minute runs a series of si…
I feel like node’s async model makes it really easy to cause a bug like this, and really difficult to track it down.
It was left to the OS to catch the leak, because the program was written in such a way that it was able to run a gazillion of these tasks concurrently.
But why should we have to close the socket manually, and why only when a certain number of bytes has been received? Based on the documentation here it certainly seems like it should automatically destroy its file descriptor when the server is done sending data.
My suspicion is, the client here never reads all the data that’s being sent by the server, so the server is there waiting to finish sending the page and on the client nobody’s reading it, so the connection never closes because the request isn’t completely finished. The TLS callbacks are probably still working, just not making any progress on the actual data stream. So it sits there for a really long time, until something times out, if anything times out there.
That’d make sense if the check stops reading after the HTTP headers. It reads the chunks it needs and then just stops processing because it doesn’t need the rest of the data, and the TLS wrapper is probably preventing node from collecting the socket itself. The TLS stream happily processes TLS layer pings and keeps trying to push data out but nobody’s reading the Stream anymore. Node can’t release the underlying TCP connection because the TLS wrapper is still handling it, and it can’t collect the wrapper because the TCP connection is still piped into it.
It probably works with small payloads because it’s small enough to be read in one chunk, so even if the TLS wrapper isn’t fully read, the underlying TCP connection is, so the server gets to close the connection, the fd is closed, and node is able to collect the TCP connection object and then the wrapper is fully orphaned and can also be collected.
I always feel a little paranoid when I explicitly close transactions, connections, and files (for quick running scripts, the OS will close the file when my process exits and for long running applications, the garbage collector will close it when the object leaves the scope). Then I read a blog post like this an remember that it is always better to explicitly free resources when I am done with them.
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !programming@programming.dev
Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!
Cross posting is strongly encouraged in the instance. If you feel your post or another person’s post makes sense in another community cross post into it.
Hope you enjoy the instance!
Rules
Rules
Follow the programming.dev instance rules
Keep content related to programming in some way
If you’re posting long videos try to add in some form of tldr for those who don’t want to watch videos
Magic word being watchdogalert
I feel like node’s async model makes it really easy to cause a bug like this, and really difficult to track it down.
It was left to the OS to catch the leak, because the program was written in such a way that it was able to run a gazillion of these tasks concurrently.
My suspicion is, the client here never reads all the data that’s being sent by the server, so the server is there waiting to finish sending the page and on the client nobody’s reading it, so the connection never closes because the request isn’t completely finished. The TLS callbacks are probably still working, just not making any progress on the actual data stream. So it sits there for a really long time, until something times out, if anything times out there.
That’d make sense if the check stops reading after the HTTP headers. It reads the chunks it needs and then just stops processing because it doesn’t need the rest of the data, and the TLS wrapper is probably preventing node from collecting the socket itself. The TLS stream happily processes TLS layer pings and keeps trying to push data out but nobody’s reading the Stream anymore. Node can’t release the underlying TCP connection because the TLS wrapper is still handling it, and it can’t collect the wrapper because the TCP connection is still piped into it.
It probably works with small payloads because it’s small enough to be read in one chunk, so even if the TLS wrapper isn’t fully read, the underlying TCP connection is, so the server gets to close the connection, the fd is closed, and node is able to collect the TCP connection object and then the wrapper is fully orphaned and can also be collected.
This was my immediate suspicion as well, as soon as I read that it would leak for a GET but not a HEAD.
I always feel a little paranoid when I explicitly close transactions, connections, and files (for quick running scripts, the OS will close the file when my process exits and for long running applications, the garbage collector will close it when the object leaves the scope). Then I read a blog post like this an remember that it is always better to explicitly free resources when I am done with them.