Node.js Streams: Process Large Files Without Memory Issues

Why Streams

A 1GB file loaded into memory crashes small servers. Streams process in chunks with constant ~64KB memory.

File Pipeline

import { createReadStream, createWriteStream } from 'fs'
import { createGzip } from 'zlib'
import { pipeline } from 'stream/promises'

// Compress large file — constant memory usage
await pipeline(createReadStream('large.csv'), createGzip(), createWriteStream('large.csv.gz'))

Transform Stream

import { Transform } from 'stream'

class UpperCase extends Transform {
  _transform(chunk, _enc, cb) {
    this.push(chunk.toString().toUpperCase())
    cb()
  }
}

HTTP Streaming

app.get('/export', async (req, res) => {
  res.setHeader('Content-Type', 'application/json')
  res.write('[')
  let first = true
  for await (const doc of db.collection('records').find().stream()) {
    if (!first) res.write(',')
    res.write(JSON.stringify(doc))
    first = false
  }
  res.write(']')
  res.end()
})

Server-Sent Events

app.get('/events', (req, res) => {
  res.setHeader('Content-Type', 'text/event-stream')
  const id = setInterval(() => {
    res.write('data: ' + JSON.stringify({ time: Date.now() }) + '

')
  }, 1000)
  req.on('close', () => clearInterval(id))
})

-> Encode stream data with the Base64 Converter.