server

Documentation for using streamListDiff in a NodeJS environment.

IMPORT

import { streamListDiff } from "@donedeal0/superdiff/server";

---

FORMAT

Input

In a server environment, Readable refers to NodeJS streams, and FilePath refers to the path of a file (e.g., ./list.json). Examples are provided in the #usage section below.

 prevList: Readable | FilePath | Record<string, unknown>[],
 nextList: Readable | FilePath | Record<string, unknown>[],
 referenceKey: keyof Record<string, unknown>,
 options: {
  showOnly?: ("added" | "deleted" | "moved" | "updated" | "equal")[], // [] by default
  chunksSize?: number, // 0 by default
  considerMoveAsUpdate?: boolean; // false by default
  useWorker?: boolean; // true by default
  showWarnings?: boolean; // true by default
}

• prevList: the original object list.

• nextList: the new object list.

• referenceKey: a key common to all objects in your lists (e.g. id).

• options

  • chunksSize the number of object diffs returned by each streamed chunk. (e.g. 0 = 1 object diff per chunk, 10 = 10 object diffs per chunk).

  • showOnly gives you the option to return only the values whose status you are interested in (e.g. ["added", "equal"]).

  • considerMoveAsUpdate: if set to true a moved value will be considered as updated.

  • useWorker: if set to true, the diff will be run in a worker for maximum performance. Only recommended for large lists (e.g. +100,000 items).

  • showWarnings: if set to true, potential warnings will be displayed in the console.

Using Readable streams may impact workers' performance since they need to be converted to arrays. Consider using arrays or files for optimal performance. Alternatively, you can turn the useWorker option off.

Output

The objects diff are grouped into arrays - called chunks - and are consumed thanks to an event listener. You have access to 3 events:

• data: to be notified when a new chunk of object diffs is available.

• finish: to be notified when the stream is finished.

• error: to be notified if an error occurs during the stream.

interface StreamListener<T> {
  on(event: "data", listener: (chunk: StreamListDiff<T>[]) => void);
  on(event: "finish", listener: () => void);
  on(event: "error", listener: (error: Error) => void);
}

type StreamListDiff<T extends Record<string, unknown>> = {
  value: T | null;
  index: number | null;
  previousValue: T | null;
  previousIndex: number | null;
  status: "added" | "deleted" | "moved" | "updated" | "equal";
};

---

USAGE

Input

Array

const diff = streamListDiff(
      [ 
        { id: 1, name: "Item 1" },  
        { id: 2, name: "Item 2" },
        { id: 3, name: "Item 3" } 
      ],
      [
        { id: 0, name: "Item 0" }, 
        { id: 2, name: "Item 2" },
        { id: 3, name: "Item Three" },
      ],
      "id", 
      { chunksSize: 2 }
    );

Stream

const previousStream = Readable.from(previousArray, { objectMode: true });
    
const nextStream = Readable.from(nextArray, { objectMode: true });
    
const diff = streamListDiff(previousStream, nextStream, "id", { chunksSize: 2 });

File

const previousFile = path.resolve(__dirname, "./previousArrayFile.json"); 
                              
const nextFile = path.resolve(__dirname, "./previousArrayFile.json");
                              
const diff = streamListDiff(previousFile, nextFile, "id", { chunksSize: 10 });
 

Example

const diff = streamListDiff(
      [ 
-       { id: 1, name: "Item 1" },  
        { id: 2, name: "Item 2" },
        { id: 3, name: "Item 3" } 
      ],
      [
+       { id: 0, name: "Item 0" }, 
        { id: 2, name: "Item 2" },
+       { id: 3, name: "Item Three" },
      ],
      "id", 
      { chunksSize: 2 }
    );

Output

diff.on("data", (chunk) => {
      // first chunk received (2 object diffs)
      [
+       {
+         value: { id: 0, name: "Item 0" },
+         index: 0,
+         previousValue: null,
+         previousIndex: null,
+         status: "added"
+       },
-       {
-         value: null,
-         index: null,
-         previousValue: { id: 1, name: "Item 1" },
-         previousIndex: 0,
-         status: "deleted"
-       }
      ]
    // second chunk received (2 object diffs)
      [
        {
          value: { id: 2, name: "Item 2" },
          index: 1,
          previousValue: { id: 2, name: "Item 2" },
          previousIndex: 1,
          status: "equal"
        },
+       {
+         value: { id: 3, name: "Item Three" },
+         index: 2,
+         previousValue: { id: 3, name: "Item 3" },
+         previousIndex: 2,
+         status: "updated"
+       },
     ]
});

diff.on("finish", () => console.log("Your data has been processed. The full diff is available."))
diff.on("error", (err) => console.log(err))