Node read stream hangs on specific file, base64 encoding

Version: 7.9.0/8.9.3/9.7.1
Platform: Windows/Linux/MacOS

I have following code working for every file except the one that keeps hanging without emitting end or error events (I tried other stream events too).

const fs = require('fs');

const rs = fs.createReadStream(filePath, {
    encoding: 'base64',
});

rs.on('data', () => {
    console.log('data');
});
rs.on('end', () => {
    console.log('end');
});
rs.on('error', e => {
    console.log('error', e);
});

If I move read point with start option to 1 instead of 0 it works properly. Same if highWaterMark is set to value other than default. It doesn't really help as it seems it can fail with other "corrupted" file.

It seems like Node bug, but maybe there's something I'm missing here.

Here's file to recreate the issue:
http://s3.eu-west-1.amazonaws.com/jjapitest/file

Here's interactive demo of the issue:
https://repl.it/repls/AnimatedDisguisedNumerator

Fwiw, this seems to work on master but not on the other release lines, so maybe it’s just about figuring out which change fixed this.

It will fail If highWaterMark is file.length - 1, and encoding base64, at least on v8.x.

In the following snippet, not even data is emitted.

const fs = require('fs');
const invalid = '/tmp/invalid';
fs.writeFileSync(invalid, '1234');

function readFileBase64(filePath) {
  const rs = fs.createReadStream(filePath, {
      encoding: 'base64',
      highWaterMark: 2 // with 3 data is called, but end no
  });

  rs.on('data', (data) => {
      console.log(filePath, 'data', Buffer.from(data, 'base64').toString());
      // Nothing for hwm == 2
      // 123 for hwm == 3
      // 123 & 4 for hwm == 4 == file.length
  });
  rs.on('end', () => {
      console.log(filePath, 'end');
  });
  rs.on('error', e => {
      console.log(filePath, 'error', e);
  });
}

readFileBase64(invalid);

This works fine on v10.1.0

I'm willing to track down the changes on master to fix this issue.

@addaleax

@addaleax I found the issue, and already solved it for v8.x. I haven't got the time to go through all the code in _stream_readable.js, so if someone with more expertise in the subject care to take a look at my code, to make sure nothing else would break, even though all tests are passing, that would be great.

cc @nodejs/streams

I fear this might not be fixable in Node 6, 8, 9 without backporting a semver-major change: #18994

Are we planning to backport?

@antsmartian

@antsmartian I would not recommend it, as it could actually break things.

This is fixed in Node 10, I'm closing this.

@mcollina

Thanks for the update @mcollina

Navigation Menu

Node read stream hangs on specific file, base64 encoding #19862

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Node read stream hangs on specific file, base64 encoding #19862

Description

Activity

addaleax commented on Apr 7, 2018

marcosc90 commented on Jun 1, 2018

marcosc90 commented on Jun 2, 2018

mcollina commented on Jun 2, 2018

mcollina commented on Jun 2, 2018

antsmartian commented on Oct 11, 2018

mcollina commented on Oct 11, 2018

mcollina commented on Oct 11, 2018

antsmartian commented on Oct 11, 2018

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions