Why in busboy I am making use of stream for uploading files
// accept POST request on the homepage
app.post('/', function (req, res) {
var busboy = new Busboy({ headers: req.headers });
busboy.on('file', function(fieldname, file, filename, encoding, mimetype) {
var saveTo = path.join('.', filename);
console.log('Uploading: ' + saveTo);
file.pipe(fs.createWriteStream(saveTo));
});
busboy.on('finish', function() {
console.log('Upload complete');
res.writeHead(200, { 'Connection': 'close' });
res.end("That's all folks!");
});
return req.pipe(busboy);
});
Handling file uploads in Node.js may seem at first glance like a simple task. Modules like multer allow you to easily handle file uploads as part of an express route. This works great for small files that don’t leave a significant memory footprint. But what happens when you upload very large files and load them into memory?
Imagine you want to download very large file and upload it to a destination. In this example, we’ll use the request module to both handle the download and the upload as a stream.
First, we need to create a read stream when we download the file. Request automatically does this when you do a GET:
const request = require('request');
const readStream = request('https://dummydomain.com/some-very-large-file');
Once we have a read stream, we can simply pipe it into a POST request that will in turn upload the file to our destination:
readStream.pipe(request.post('https://dummydomain.com/some-destination'));
Rather than loading the large file into memory, this sends chunks of data to the destination as soon as they’re downloaded, leaving a small memory footprint. Streams are great for handling chunks of data at a time and reducing memory use. In some cases, the expected memory usage of handling some kinds of data may not require the use of streams, but when memory usage is expected to be a bottleneck, then streams are extremely valuable for optimizing performance.
A node.js module for parsing incoming HTML form data. Its used to upload files. Busboy is a Writable stream and its an alternative for multer. A writable stream is an abstraction for a destination to which data can be written. An example of that is the fs.createWriteStream method.
On busboy ‘file’ event you get parameter named ‘file’ and this is a stream so you can pipe it.
busboy.on('file', function(fieldname, file, filename, encoding, mimetype) => {
file.pipe(streamToSQS)
})
An example of file upload with busboy and express
// accept POST request on the homepage
app.post('/', function (req, res) {
var busboy = new Busboy({ headers: req.headers });
busboy.on('file', function(fieldname, file, filename, encoding, mimetype) {
var saveTo = path.join('.', filename);
console.log('Uploading: ' + saveTo);
file.pipe(fs.createWriteStream(saveTo));
});
busboy.on('finish', function() {
console.log('Upload complete');
res.writeHead(200, { 'Connection': 'close' });
res.end("That's all folks!");
});
return req.pipe(busboy);
});
The function fs.createWriteStream() creates a writable stream in a very simple manner. After a call to fs.createWriteStream with the filepath, you have a writeable stream to work with.
Comparison with multer
Some developers opine that Multer is easier because it abstracts away some of the details of Busboy.
Difference between busboy and connect-busboy
https://stackoverflow.com/questions/39439922/difference-between-busboy-and-connect-busboy Connect is a middleware layer for building servers in Node.js. It was originally the basis for the Express web framework.
What middleware here really means is essentially an array of functions that conform to an interface which get called on each request in the order they are defined.
connect-busboy wraps the busboy library into a connect compatible middleware. You can see in the source it really just returns a function.
If you’re using express you might want to take a look at express-busboy which uses connect-busboy under the hood and has recent updates.
What exactly are streams?*
Streams are collections of data — just like arrays or strings. The difference is that streams might not be available all at once, and they don’t have to fit in memory. This makes streams really powerful when working with large amounts of data, or data that’s coming from an external source one chunk at a time.
However, streams are not only about working with big data. They also give us the power of composability in our code. Just like we can compose powerful linux commands by piping other smaller Linux commands, we can do exactly the same in Node with streams.
Here’s the magic line about pipe() and stream that you need to remember:
readableSrc.pipe(writableDest)
In this simple line, we’re piping the output of a readable stream — the source of data, as the input of a writable stream — the destination.
Using fs.createWriteStream()
When handling particularly large files, or files that come in chunks, say from a network connection, using streams is preferable to writing files in one go via the above methods that write entire files.
Streams write small amounts of data at a time. While this has the disadvantage of being slower because data is transferred in chunks, it has advantages for RAM performance. Since the whole file is not loaded in memory all at once, RAM usage is lower.
To write to a file using streams, you need to create a new writable stream. You can then write data to the stream at intervals, all at once, or according to data availability from a server or other process, then close the stream for good once all the data packets have been written.
const fs = require('fs');
const writeStream = fs.createWriteStream('secret.txt');
// write some data with a base64 encoding
writeStream.write('aef35ghhjdk74hja83ksnfjk888sfsf', 'base64');
// the finish event is emitted when all data has been flushed from the stream
writeStream.on('finish', () => {
console.log('wrote all data to file')
})
writeStream.end();
We created a writable stream, then wrote some data to the stream. We have included a log statement when the “finish” event is emitted, letting us know that all data has been flushed to the underlying system. In this case, that means all data has been written to the file system.
https://carlosrymer.com/using-node-streams-to-handle-large-file-uploads-24c1a0141b9c
