Goodbye, Node.js Buffer

Thanks for doing this!

I had to convert a ReadableSteam to base64 recently, and I was shocked at how much boilerplate it required in 2023:

    if (isReadableStream<Uint8Array>(value)) {
      const chunks = [];

      for await (const chunk of value) {
        chunks.push(chunk);
      }

      if (chunks[0].byteLength) {
        const length = chunks.reduce(
          (agg, next) => agg + next.length, 0
        );

        // Make the same species TypedArray that ReadableStream gave us.
        value = new (chunks[0].constructor as Uint8ArrayConstructor)(length);

        for (let i = 0, offset = 0; i < chunks.length; i++) {
          value.set(chunks[i], offset);
          offset += chunks[i].length;
        }
      } else {
        throw new Error(`Unrecognized readable stream type: ${ chunks[0].constructor.name }`);
      }
    }

    return  `data:application/octet-stream;base64,${
      btoa(
        (value as Uint8Array).reduce(
          (result, charCode) => result + String.fromCharCode(charCode),
          ''
        )
      )
    }`;

It's really because we have some wonderful collaboration between vendors now, and they don't want to have to come up with their own APIs.

https://wintercg.org/

The accusation of malice goes a bit far, but I agree it's a problem whether it's intentional or not. Bun takes a slapdash approach to shipping features and APIs, which means things get out fast but also rubs me the wrong way re: long-term implications and vision

> can't help feeling that it could be intentional vendor lock-in.

it's always vendor lock-in. can never trust companies, mate!

It's still OSS, not proprietary APIs. Feel free to fork it in the future, once they need to make their money back.

If it's faster than node and the API to do what I need is better (Bun.FileSystemRouter comes to mind), I don't see why not to switch over.

Heck, I would accept even less retro-compatibility to get rid of transpiling, that completely oscene political mess they did with modules and treating TS as a 2nd class citizen.

The amount of time I'm still hit by ESM vs CJS is insane. Truly a Python 2vs3 moment for the node.js community.

- Blob: Immutable raw data container with a size and MIME type, not directly readable.

- File: Like a Blob, but with additional file-specific properties (e.g., filename).

- ArrayBuffer: Fixed-length raw binary data in JavaScript, not directly accessible.

- Uint8Array: Interface for reading/writing binary data in ArrayBuffer, showing them as 8-bit unsigned integers.

- Buffer: Readable/writable raw binary data container in Node.js (subclass of Uint8Array)

Typed arrays are essential for web apps that use WebGL and WebGPU. Being able to send this type of data to run computations on the GPU can give you 1000x speed up.

You can see it in action on this WebGL fluid simulator[0] by PavelDoGreat.

[0] https://github.com/PavelDoGreat/WebGL-Fluid-Simulation

> I can't think of many use cases in JS land were Uint8Array, Uint16Array, Uint32Array, Int8Array would be absolutely necessary.

Buffer is a subclass of Uint8Array.

Sized numerics are useful even in high level, loosely and dynamically typed languages. Two examples are binary (de)serialization and OpenGL. In Python, libraries for those generally use sized numerics.

High level languages will still want to do low level things

Buffer is a utility. It combines several abstractions that are common in nature into one highly useful class. I quite prefer using it as well.

This is what I feel the node people get right over the ES people, the ES ideology is so abstracted and pushes everything out into small utility classes so you have to create a handful of different objects with odd combinations of methods to get one useful conversion done.

The ES people also seem to have no love of the CLI or for type types of debugging and testing done there. As a result, I almost always choose the node created abstractions over the ES specified ones.

> For WebAssembly? Surely a plain array of numbers can be used in most cases.

So a list of floats? No! Let's not use floats everywhere just because JavaScript does.

Consider a web server serving multiple ongoing requests from different users (with separate permissions). Each of them uses a buffer at some point (either a Buffer or a Uint8Array)

If the buffer doesn't have any cross-contamination with global state, there's no way one user could access another's data (because it's behind an object reference that never comes into the scope of the request logic for the other user). But if it did, and a malicious user found some other kind of vulnerability, they could potentially access data across-scopes

From a cursory read, it matters because all Buffer.from(...) calls use a shared buffer, and buffer over-reads are a much more common vulnerability than easy access to arbitrary memory.

Not a security expert etc.

People are working on bringing Base64/Hex conversion to JavaScript: https://github.com/tc39/proposal-arraybuffer-base64

I also provide a package to make the transition easier: https://github.com/sindresorhus/uint8array-extras (Feel free to copy-paste the code if you don't want another dependency)

You can use ‘atob’ and ‘btoa’ functions for some of that.

Polyfill Buffer.

> // Required as `btoa` and `atob` don't properly support Unicode: https://developer.mozilla.org/en-US/docs/Glossary/Base64#the...

As discussed in another thread, `subarray()`[0] fills the same purpose.

[0] https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...

`Buffer.slice` is deprecated: https://nodejs.org/api/buffer.html#bufslicestart-end

`Uint8Array.prototype.isPrototypeOf` and `instanceof Uint8Array` do not work across realms (frames, Node.js VM, etc).

Feel free to copy-paste the function to your own code base if you don't want the dependency:

``` const objectToString = Object.prototype.toString;

export function isUint8Array(value) { return value && objectToString.call(value) === '[object Uint8Array]'; } ```

I added a github issue to my project where this matters literally this morning. I'd love to talk to someone dealing with the same stuff. I have baked dataframe -> JS serialization utils so many times over the past decade. I never want to do much in JS, just simple selection and filtering. But you end up needing both sides of the serialization to do it right.

nodejs-polars is node-specific and uses native FFI. polars can be compiled to Wasm but doesn't yet have a js API out of the box.

As for the fastest way to serialize data to Pandas data to the browser, you should use Parquet; it's the fastest to write on the Python side and read on the JS side, while also being compressed. See https://github.com/kylebarron/parquet-wasm (full disclosure, I wrote this)