# Node Streams
I'm Marco Rogers (@polotek).
I work at Yammer.
I'm a javascript guy.
## What are they?
Callbacks
fs.readFile('/path/to/file.txt', function(err, data) {
// do something with your data
});
## What are they?
Event Emitter
fileHandle.on('data', function(data) {
// do something with your data
});
## What are They?
Streams are the way node tells you about incremental data.
Streams are the way node encourages you to stop procrastinating.
// uploading
var data = '';
req.on('data', function(chunk) {
data += chunk;
});
req.on('end', function() {
fs.writeFile('./files/test.json', data);
res.end();
});
## Why not?
Buffering into memory is inefficient. That's how the other guys do it.
Streams are meant to be composed together into a "pipe" of continuous data.
## Streams API
The Streams API is a set of patterns and conventions for dealing with streaming data in node.
## Streams API
// uploading
var file = fs.createWriteStream('./files/test.json');
req.pipe(file);
## Why?
* Keep things moving.
* Lower memory usage.
* Higher throughput.
* Handle backpressure.
What the hell is backpressure?
(ask me later)
## Base classes
Getting the details of streaming right is hard.
So you don't have to.
## Base classes
Node provides base implementations that make it easy for you to achieve common patterns.
* Readable/Writable
* Duplex
* Transform
* Passthrough
## Duplex
A stream that is both readable and writable.
## Transform
A duplex stream that processes incoming data and pushes out transformed data.
## Passthrough
A transform stream that doesn't transform anything. Mostly for testing, but...
## Passthrough
uploadedFile.pipe(s3Stream);
uploadedFile.pipe(tmpFileStream);
What happens when s3 craps out?
## Passthrough
uploadedFile.pipe(errorHandler).pipe(s3Stream);
uploadedFile.pipe(tmpFile);
Handle s3 problem, but you still get the backup on disk.
## Why?
Node streams are how you bring modularity to streaming data.
## Questions?
* Strings vs. Buffers? (yes)
* Better error handling? (domains?)
* Doing async work before piping? (fixed)
* Why doesn't pause() actually pause? (fixed)
* Piping to multiple destinations?
* When not to use streams? (???)
We need to get better at this. Help us.
## Final thought?
Try to think in streams, until you can't.
## Thanks
Marco Rogers
github: [@polotek](https://github.com/polotek)
twitter: [@polotek](https://twitter.com/polotek)