Skip to main content

Read Many Records

The code sample below shows how to iterate over the list of pages and get the required number of records that is larger than a page size using search API.

note

The following code uses fetch API that is available starting from Node.js version 18. In case you are using the older version or the runtime without fetch support, the implementation can be changed in the request function.

In order to read the records from a RestAPI, first you have to create a set of credentials and assign them in the constant LOCATION in the code sample below.

By default, this code requests a RestAPI with no search criteria meaning eventually all the records will be returned. If you want to send any search criteria to read specific records, please change the value of the FIND_PAYLOAD constant. For the possible values please refer to the search API documentation.

To run the script using Node.js, assuming you saved the source code in the file named read_many.js, please run the following command in the terminal:

node read_many.js

Since there might be a huge number of records in the vault, there is a simple logger in the script that prints the number of records read and the total number of records in stderr. Even though it is not an error, we are using stderr rather than stdout to allow the results to be written in the valid JSON file. If this behavior is unwanted, there is a feature flag called TRACK_PROGRESS. In order to write results to a JSON file, the script can be run like this:

node read_many.js > results.json

Also, please note, that the following code snippet is simplified to keep the example small. There is limited error handling, results are collected in memory and if you have the huge number of records in the vault you might want to append them to a file on every iteration.

read_many.js
/**
 * @typedef {object} LocationDefinition
 * @property {string} url - Actual URL of the RestAPI
 * @property {string} authProviderURL - URL of OAuth2 provider
 * @property {string} envId - Target environment ID
 * @property {string} country - Two letter country code
 * @property {string} clientId - Created on InCountry portal
 * @property {string} clientSecret - Created on InCountry portal
 * @property {string} [audience] - Space separated list of URLs of appropriate resources in a RestAPI.
 * Like API and Storage. In case missing, it will be generated using a common template
 */

/**
 * @typedef {object} FindPayload
 * @property {Object.<string, any>} filter - A set of search criteria
 * @property {Object.<string, any>} options - A list of options for the find endpoint

/**
 * RestAPI to search
 * To a create new service and get the credentials please refer to the documentation
 * https://docs.incountry.com/portal/managing-services/#creating-a-new-rest-api-service
 *
 * @constant
 * @type {LocationDefinition}
 */
const LOCATION = {
  url: "https://<incountry RestAPI hostname>/api/records/find",
  authProviderURL: "https://<selected auth provider hostname>/oauth2/token",
  envId: "<environment id>",
  country: "<two letter country code>",
  clientId: "<client identifier>",
  clientSecret: "<password>",
};

/**
 * The number of records to read
 * Zero means read all of them
 *
 * @constant
 * @type {number}
 */
const RECORDS_TO_READ = 1000;

/**
 * Feature flag to toggle progress messages in stderr
 *
 * @constant
 * @type {boolean}
 */
const TRACK_PROGRESS = true;

/**
 * Find payload
 * Consists of filters and options. For the examples and use cases please refer to the documentation
 * https://docs.incountry.com/data-residency-as-a-service/search-rest-api/
 *
 * @constant
 * @type {FindPayload}
 */
const FIND_PAYLOAD = {
  filter: {},
  options: { offset: 0 },
};

(async function (location, payload, limit) {
  const results = [];

  // In case we're looking for the number of records less than a page size
  if (limit && limit < 100) {
    payload.options.limit = limit;
  }

  const { data, meta } = await search(location, payload);

  const pageSize = meta.limit;
  const total = limit || meta.total;

  if (!total || total < pageSize) {
    console.log(JSON.stringify(data));
    process.exit(0);
  }

  results.push(...data);
  logProgress(pageSize, total);

  while (results.length < total) {
    payload.options.offset += pageSize;

    // In case the page size is not multiple to the number of records
    // For the last request we have to set the limit
    if (limit && limit - results.length < pageSize) {
      payload.options.limit = limit - results.length;
    }
    const { data, meta } = await search(location, payload);
    results.push(...data);
    logProgress(results.length, total);
  }

  console.log(JSON.stringify(results));
})(LOCATION, FIND_PAYLOAD, RECORDS_TO_READ);

// Below this line, there are internal functions to execute find requests. In order to alter the results set
// please modify FIND_PAYLOAD object above
// --------------------------------------------------------------------------------------------------------------------

/**
 * Search a given RestAPI
 *
 * @param {LocationDefinition} location
 * @param {FindPayload} payload - An object that will be passed as a payload to the find endpoint
 * @returns {Promise.<FindResults>}
 */
async function search(
  { url, authProviderURL, country, envId, clientId, clientSecret, audience },
  payload,
) {
  const accessToken = await authenticate(
    authProviderURL,
    envId,
    clientId,
    clientSecret,
    audience || generateAudience(country),
  );
  return readRecords(url, envId, accessToken, { ...payload, country });
}

/**
 * Get access token for a given location
 *
 * For more information please refer to:
 * https://docs.incountry.com/data-residency-as-a-service/authentication/
 *
 * @param {string} url - OAuth2 server URL
 * @param {string} envId - Target environment ID
 * @param {string} clientId - Client identifier created on InCountry portal
 * @param {string} clientSecret - Password created on InCountry portal
 * @param {string} audience - A list of URLs of appropriate resources in a RestAPI. Like API and Storage
 * @returns {Promise.<string>} Opaque access token
 */
async function authenticate(url, envId, clientId, clientSecret, audience) {
  const { access_token: accessToken } = await request(
    url,
    {
      grant_type: "client_credentials",
      response_type: "token",
      scope: envId,
      audience: audience,
    },
    {
      "Content-Type": "application/x-www-form-urlencoded;charset=UTF-8",
      Authorization:
        "Basic " +
        Buffer.from(clientId + ":" + clientSecret).toString("base64"),
    },
  );
  if (!accessToken) {
    console.debug(
      `Unable to get an access token from ${url} using`,
      clientId,
      clientSecret,
      audience,
    );
    throw new Error(
      `Unable to get an access token from ${url}. Please check your url and credentials.`,
    );
  }
  return accessToken;
}

/**
 * @typedef {Object} FindResults
 * @property {Array.<Object.<string, any>>} data
 * @property {{count: number, limit: number, offset: number, total: number}} meta
 */

/**
 * Read an actual list of records from the RestAPI
 * For more information and examples please refer to the documentation
 * https://docs.incountry.com/data-residency-as-a-service/search-rest-api/
 *
 * @param {string} url - Actual URL of the RestAPI
 * @param {string} envId - Environment ID
 * @param {string} accessToken - Opaque access token
 * @param {FindPayload} payload - Search criteria and options
 * @returns {Promise.<FindResults>}
 */
async function readRecords(url, envId, accessToken, payload) {
  const response = await request(url, payload, {
    Authorization: `Bearer ${accessToken}`,
    "X-Env-Id": envId,
  });
  if (!("data" in response)) {
    console.debug(
      "Unable to execute find request",
      url,
      envId,
      accessToken,
      payload,
      response,
    );
    throw new Error("Unable to execute find request to", url);
  }
  return response;
}

/**
 * Send HTTP POST request using fetch API
 * For this example we will use only POST requests
 *
 * @param {string} url
 * @param {Object.<string, any>} payload - An object that will be serialized and sent to the target endpoint
 * @param {Object.<string, string|number|boolean} - A list of HTTP headers
 * @returns {Promise.<Object.<string, any>>}
 */
async function request(url, payload, headers) {
  payload = headers["Content-Type"]?.startsWith(
    "application/x-www-form-urlencoded",
  )
    ? new URLSearchParams(payload)
    : JSON.stringify(payload);

  const response = await fetch(url, {
    mode: "cors",
    method: "POST",
    cache: "no-cache",
    headers: new Headers({
      "Content-Type": "application/json",
      ...headers,
    }),
    body: payload,
  });
  if (response.status !== 200) {
    console.debug(
      "Fail to execute an HTTP request",
      url,
      payload,
      headers,
      response,
    );
    throw new Error(
      `Unable to execute request to ${url}. Please check the payload and the headers.`,
    );
  }
  return response.json();
}

/**
 * Generate audience using our generic template
 *
 * @param {string} country - Two letter country code
 * @returns {string}
 */
function generateAudience(country) {
  return `https://${country}-restapi-mt-01.api.incountry.io https://${country}-mt-01.api.incountry.io`;
}

/**
 * Helper function to log the progress
 * It writes information to stderr to keep stdout pipeable to the valid JSON file
 *
 * @param {number} current
 * @param {number} total
 */
function logProgress(current, total) {
  TRACK_PROGRESS && console.error("Read", current, "records out of", total);
}