Marklogic 8 - Stream Large Result Set To A File - Javascript - Node.js Client Api

Let's say I have a query that is going to return a very large response. Possibly thousands of records and possibly gigabytes of data. Normally in the UI, we just show a single page

Solution 1:

If you want the document descriptors, you can open an object stream as in the following example:

https://github.com/marklogic/node-client-api/blob/develop/examples/query-builder.js#L38

If you only want the content of the documents, you can use a chunked stream as shown in the following example (the same approach can be used for a query):

https://github.com/marklogic/node-client-api/blob/develop/examples/read-stream.js#L27

The general approach would be as follows:

• open the destination file as a write stream

https://nodejs.org/api/fs.html#fs_fs_createwritestream_path_options

• query for the first page of documents, piping the read stream for the documents to the write stream for the file, taking care to set the end option to false:

https://nodejs.org/api/stream.html#stream_readable_pipe_destination_options

• loop on reading documents, incrementing the start page by the page length until finished reading

• call end() on the write stream to close the file

https://nodejs.org/api/stream.html#stream_writable_end_chunk_encoding_callback

Hoping that helps