Why loading the whole file is not a durable design
A reporting or data-transfer file can be much larger than the memory budget of its process. JSONL gives each record its own line, but read chunks do not necessarily end at a record boundary or even at a character boundary. Reading everything first defeats streaming; parsing every chunk as a separate JSON document rejects valid input. Define accepted records, maximum line size and failure behavior before writing the transformation.
The example connects file reading, record transformation and output writing through pipeline. The producer must respect the consumer instead of accumulating an unlimited list of writes. A highWaterMark setting is a buffering threshold, not an absolute limit on total application memory. The parser also bounds unfinished lines. It retains a read chunk plus a limited remainder rather than the entire file, which makes the capacity contract easier to inspect.
Record boundaries, errors and output publication
To preserve Persian text, bytes are assembled up to a newline before strict UTF-8 decoding. A character split between chunks remains intact. Blank lines are ignored, while malformed JSON and invalid encoded bytes produce explicit failures. This example accepts only an integer id and a string name. Change that schema deliberately for your own data instead of assuming that syntactically valid JSON is automatically an acceptable business record.
Output is first written to an exclusively created temporary name. After pipeline completes, a same-directory hard link publishes the completed file, then the temporary name is removed. An existing destination causes failure without replacing its contents. The filesystem must support hard links. This publication rule does not, by itself, promise power-loss durability; a separate synchronization and storage policy is required when that guarantee matters to the application.
Tests split a Persian character across chunks and exercise an oversized line, malformed JSON and invalid UTF-8. Another case preserves an existing destination and verifies that a pre-aborted operation leaves no temporary output. AbortSignal reaches pipeline, which coordinates its stages. For a network-backed source, propagate the signal to the actual request too. A visible cancel control is insufficient when the underlying work continues independently.
Complete files, execution and tests
Node.js 22+, a filesystem supporting hard links and built-in Node.js modules only. Save the four complete files below together using the stated filenames; no code is replaced by ellipses.
1. Parse records without breaking characters
File: jsonl-transform.mjs. Save this complete file without removing lines.
import { TextDecoder } from 'node:util';
export async function* parseJsonLines(source, { maxLineBytes = 4096, signal } = {}) {
if (!Number.isInteger(maxLineBytes) || maxLineBytes < 1) throw new RangeError('Invalid limit');
const decoder = new TextDecoder('utf-8', { fatal: true });
let pending = Buffer.alloc(0);
function parse(line) {
if (line.length > maxLineBytes) throw new RangeError('Line exceeds byte limit');
const text = decoder.decode(line).trim();
return text ? JSON.parse(text) : undefined;
}
for await (const chunk of source) {
signal?.throwIfAborted();
pending = Buffer.concat([pending, Buffer.from(chunk)]);
let end;
while ((end = pending.indexOf(10)) !== -1) {
const value = parse(pending.subarray(0, end));
pending = pending.subarray(end + 1);
if (value !== undefined) yield value;
}
if (pending.length > maxLineBytes) throw new RangeError('Line exceeds byte limit');
}
signal?.throwIfAborted();
if (pending.length) {
const value = parse(pending);
if (value !== undefined) yield value;
}
}The parser bounds lines in bytes and decodes UTF-8 strictly. highWaterMark is not a total-memory guarantee; the example file source uses sixteen-kilobyte chunks.
2. Pipeline and exclusive output publication
File: export-jsonl.mjs. Save this complete file without removing lines.
import { createReadStream, createWriteStream } from 'node:fs';
import { link, unlink } from 'node:fs/promises';
import { randomUUID } from 'node:crypto';
import { pipeline } from 'node:stream/promises';
import { parseJsonLines } from './jsonl-transform.mjs';
export async function exportJsonl(input, output, { signal, maxLineBytes = 4096 } = {}) {
const temporary = `${output}.pending-${randomUUID()}`;
try {
await pipeline(
createReadStream(input, { highWaterMark: 16 * 1024 }),
async function* (source, { signal: pipelineSignal }) {
for await (const row of parseJsonLines(source, { maxLineBytes, signal: pipelineSignal })) {
if (!row || !Number.isInteger(row.id) || typeof row.name !== 'string') {
throw new TypeError('Each row needs an integer id and a string name');
}
yield JSON.stringify({ id: row.id, name: row.name.trim() }) + '\n';
}
},
createWriteStream(temporary, { flags: 'wx' }),
{ signal },
);
// Same-directory hard link publishes a completed file without overwriting.
await link(temporary, output);
} finally {
await unlink(temporary).catch(error => {
if (error.code !== 'ENOENT') throw error;
});
}
}Existing output is preserved. A same-directory hard link publishes the completed file and the temporary name is removed. Filesystems without this feature need another explicit publication policy.
3. Text, failure and cancellation tests
File: test-jsonl.mjs. Save this complete file without removing lines.
import test from 'node:test';
import assert from 'node:assert/strict';
import { Readable } from 'node:stream';
import { mkdtemp, writeFile, readFile, readdir, unlink, rmdir } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { parseJsonLines } from './jsonl-transform.mjs';
import { exportJsonl } from './export-jsonl.mjs';
async function collect(chunks, options) {
const rows = [];
for await (const row of parseJsonLines(Readable.from(chunks), options)) rows.push(row);
return rows;
}
test('UTF-8 characters may span two chunks', async () => {
const bytes = Buffer.from('{"id":1,"name":"لیان"}\n');
assert.deepEqual(await collect([bytes.subarray(0, 18), bytes.subarray(18)]),
[{ id: 1, name: 'لیان' }]);
});
test('oversized lines, malformed JSON and invalid UTF-8 fail', async () => {
await assert.rejects(collect([Buffer.from('x'.repeat(20))], { maxLineBytes: 8 }), RangeError);
await assert.rejects(collect([Buffer.from('{bad}\n')]), SyntaxError);
await assert.rejects(collect([Buffer.from([0xff, 10])]), TypeError);
});
test('completed output is exclusive; cancellation leaves no temporary output', async () => {
const dir = await mkdtemp(join(tmpdir(), 'liyan-jsonl-test-'));
try {
const input = join(dir, 'input.jsonl'), output = join(dir, 'output.jsonl');
await writeFile(input, '{"id":7,"name":" Liyan "}\n');
await exportJsonl(input, output);
const result = '{"id":7,"name":"Liyan"}\n';
assert.equal(await readFile(output, 'utf8'), result);
await assert.rejects(exportJsonl(input, output), { code: 'EEXIST' });
assert.equal(await readFile(output, 'utf8'), result);
const controller = new AbortController(); controller.abort();
await assert.rejects(exportJsonl(input, join(dir, 'cancelled.jsonl'), { signal: controller.signal }));
assert.deepEqual((await readdir(dir)).sort(), ['input.jsonl', 'output.jsonl']);
} finally {
for (const name of await readdir(dir)) await unlink(join(dir, name));
await rmdir(dir); // Non-recursive: unexpected subdirectories are never removed.
}
});Run node --test test-jsonl.mjs. Tests cover character boundaries, line size, invalid data, existing output and temporary-name cleanup after cancellation.
4. A complete demonstration input
File: demo-jsonl.mjs. Save this complete file without removing lines.
import { mkdtemp, writeFile, readFile } from 'node:fs/promises';
import { join } from 'node:path';
import { tmpdir } from 'node:os';
import { exportJsonl } from './export-jsonl.mjs';
const dir = await mkdtemp(join(tmpdir(), 'liyan-jsonl-demo-'));
const input = join(dir, 'input.jsonl'), output = join(dir, 'output.jsonl');
await writeFile(input, '{"id":1,"name":" Liyan "}\n{"id":2,"name":" Data "}\n', { flag: 'wx' });
await exportJsonl(input, output);
console.log(await readFile(output, 'utf8'));
console.log('Inspect the input and output in:', dir);Run node demo-jsonl.mjs to print two normalized records. The final message identifies the generated directory so input and output files can be inspected.
Running tests and measuring real input
Save the four files together and run node --test test-jsonl.mjs. The demonstration creates a small inspectable input and prints two normalized records along with their directory. For real files, measure record count, elapsed time, peak memory and cancellation behavior. The four-kilobyte line limit is an example contract rather than a universal recommendation. Keep the boundary tests when adjusting chunk sizes or replacing the filesystem source with a network stream.
Implementation checklist
- Pass split-character and oversized-record tests.
- Choose a record limit from real data.
- Check hard-link support and existing-output behavior.
- Propagate cancellation into every real stage.
References
Editorial guidance: Liyan Knowledge.
