Node.js Streams vs l-RAM: Kifach t-jéri l-files l-kbar bla ma t-khssr r-rass

Kamlin traat lina. Khassk t-jéri wahed l-file CSV kbir awla logs d production, k-t-mchi t-dir fs.readFile(). F l-Mac M1 ddyalk b 32GB d RAM, kolchi kiy-douz nqi. Walakin mlli k-t-deployi f wahed l-instance sghira f AWS, kiy-t-terteq lik l-film: JavaScript heap out of memory.

F l-mantiq d GhiKankhewer, ma k-n-hllouch had l-mochkil b-ano n-chriw RAM ktr. K-n-hllouh b Streams.

"Stel" vs "Tyou"

T-khayyel m3aya bghiti t-khwi wahed l-piscine d l-ma.

• Tariqa l-3adyia (readFile): Kat-qlleb 3la wahed l-stel kbir qdd dik l-piscine bach t-hzzha kamla f deqqa whda. Ma ghadi t-lqahch, o ghadi t-hrreg d-hrk (l-RAM kiy-t-bloka).
• Tariqa d Streams: Kat-khddem "Tyou" (Pipe). Ma kiy-hmmkch ch-hal kbir dik l-piscine; kiy-hmmk ghir ch-hal d l-ma kiy-douz f l-tyou f kolla tanyia.

L-Tkhwira l-m9awda: stream.pipe()

F blast ma t-hzz l-file kaml f l-mémoire, k-n-khdmou 3lih trf b trf (chunks).

L-Code (Simplified):

Kifach t-compressi file kbir o t-loho f S3 bla sda3:

const fs = require('fs');
const zlib = require('zlib');

// Create a read stream (L-Piscine)
const readStream = fs.createReadStream('./massive-log.txt');

// Transform stream (L-Filter)
const gzip = zlib.createGzip();

// Write stream (Fin ghadi l-ma)
const writeStream = fs.createWriteStream('./massive-log.txt.gz');

// Connect l-tyouat
readStream
  .pipe(gzip)
  .pipe(writeStream)
  .on('finish', () => console.log('Sali l-film bla ma y-t-fga3 l-server!'));

3lach hada move d "Senior"?

Mlli k-t-khdem b Streams, l-consommation d l-RAM k-t-bqa t-abta. Swyya l-file fih 10MB awla 10GB, Node.js ghadi y-khdem ghir b 50MB d RAM t-qriban.

Hadi hiya l-tkhwira dyal t-weffir l-flouss. T-qder t-khddem pipelines d l-data f instance t3.micro rkhisa, f l-wqt li l-ghachi d "Enterprise" kiy-khllsou $200 f ch-hr f instance kbira ghir hit l-code ddyalhom fih l-t-qba.

L-Khoulassa: Ma t-khssrch l-flouss f l-matériel mlli y-qder l-code y-hll l-mochkil. Khddem l-tyou.