InCountry Node.js SDK

InCountry Storage SDK

Incountry Storage SDK is available at https://github.com/incountry/sdk-nodejs

Installation

The SDK is available via NPM:

npm install incountry --save

Countries List

To get the full list of supported countries and their codes, please follow this link.

Quickstart guide

To access your data in InCountry Platform by using NodeJS SDK, you need to create an instance of the Storage class using the createStorage async factory method. You can retrieve the oauth.clientId, oauth.clientSecret and environmentId variables from your dashboard on InCountry Portal.

const { createStorage } = require('incountry');
const storage = await createStorage({
  environmentId: '<environment_id>',
  oauth: {
    clientId: '<client_id>',
    clientSecret: '<client_secret>',
  },
  getSecrets: () => '<encryption_secret>',
});

Storage Configuration

Below you can find a full list of possible configuration options for creating a Storage instance.

type StorageOptions = {
  apiKey?: string;          // Required when using API key authorization, or as the INC_API_KEY environment variable
  environmentId?: string;   // Required to be passed in, or as the INC_ENVIRONMENT_ID environment variable

  oauth?: {
    clientId?: string;      // Required when using oAuth authorization, can be also set through the INC_CLIENT_ID environment variable
    clientSecret?: string;  // Required when using oAuth authorization, can be also set through INC_CLIENT_SECRET environment variable
    authEndpoints?: {       // Custom endpoints regional map to use for fetching oAuth tokens
      default: string;
      [key: string]: string;
    };
  };

  endpoint?: string;        // Defines API URL
  encrypt?: boolean;        // If false, encryption is not used. Defaults to true.

  logger?: Logger;
  getSecrets?: Function;    // Used to fetch an encryption secret
  normalizeKeys?: boolean;
  countriesCache?: CountriesCache;
  hashSearchKeys?: boolean; // Set to false to enable partial match search among record's text fields `key1, key2, ..., key10`. Defaults to true.

  /**
   * Defines API base hostname part to use.
   * If set, all requests will be sent to https://${country}${endpointMask} host instead of the default
   * one (https://${country}-mt-01.api.incountry.io)
   */
  endpointMask?: string;

  /**
   * If your PoPAPI configuration relies on a custom PoPAPI server (rather than the default one)
   * use the `countriesEndpoint` option to specify the endpoint responsible for fetching the list of supported countries.
   */
  countriesEndpoint?: string;

  httpOptions?: {
    timeout?: NonNegativeInt; // Timeout in milliseconds.
  };
};

async function createStorage(
  options: StorageOptions,
  customEncryptionConfigs?: CustomEncryptionConfig[]
): Promise<Storage> {
  /* ... */
}

Below you can find API Key authorization usage example:

const { createStorage } = require('incountry');
const storage = await createStorage({
  apiKey: '<api_key>',
  environmentId: '<environment_id>',
  getSecrets: () => '<encryption_secret>',
});

oAuth options configuration

The SDK allows to precisely configure oAuth authorization endpoints (if needed). Use this option only if your plan configuration requires so.

Below you can find the example of how to create a storage instance with custom oAuth endpoints:

const { Storage } = require('incountry');
const storage = new Storage({
  environmentId: '<environment_id>',
  getSecrets: () => '<encryption_secret>',
  oauth: {
    clientId: '<client_id>',
    clientSecret: '<client_secret>',
    authEndpoints: {
      "default": "<default_auth_endpoint>",
      "emea": "<auth_endpoint_for_emea_region>",
      "apac": "<auth_endpoint_for_apac_region>",
      "amer": "<auth_endpoint_for_amer_region>",
    },
  },
});

Encryption key/secret

The getSecrets storage config option allows you to pass function which will be used to fetch an encryption key(s) or secret. This function should return either a string representing your secret or an object (so-called SecretsData) or a Promise which is resolved to such string or object:

type SecretOrKey = {
  secret: string;
  version: NonNegativeInt;
  isKey?: boolean;
  isForCustomEncryption?: boolean;
};

type SecretsData = {
  currentVersion: NonNegativeInt;
  secrets: Array<SecretOrKey>;
};

/// SecretsData example
{
  secrets: [
    {
      secret: 'aaa',
      version: 0
    },
    {
      secret: 'base64...IHN0cmluZw==', // Should be a base64-encoded key (32 byte key)
      version: 1,
      isKey: true
    },
    {
      secret: 'ccc',
      version: 2,
      isForCustomEncryption: true
    }
  ],
  currentVersion: 1
};

Note: even though SDK uses PBKDF2 to generate a cryptographically strong encryption key, you must ensure that you provide a secret/password which follows the modern security best practices and standards.

The SecretsData object allows you to specify multiple keys/secrets which the SDK will use for decryption based on the version of the key or secret used for encryption. Meanwhile SDK will encrypt data only by using a key (or secret) which matches the currentVersion parameter provided in the SecretsData object.

This enables the flexibility required to support Key Rotation policies when secrets (or keys) must be changed with time. The SDK will encrypt data by using the current secret (or key) while maintaining the ability to decrypt data records that were encrypted with old secrets (or keys). The SDK also provides a method for data migration which allows you to re-encrypt data with the newest secret (or key). For details please see the migrate method.

The SDK allows you to use custom encryption keys, instead of secrets. Please note that a user-defined encryption key should be a base64-encoded 32-bytes-long key as required by AES-256 cryptographic algorithm.

Below you can find several examples of how you can use the getSecrets storage config option:

// Synchronous
const getSecretsSync = () => 'longAndStrongPassword';
const storage = await createStorage({
  ...,
  getSecrets: getSecretsSync,
});

// Asynchronous
const getSecretsAsync = async () => {
  const secretsData = await getSecretsDataFromSomewhere();
  return secretsData;
};
const storage = await createStorage({
  ...,
  getSecrets: getSecretsAsync,
});

// Using promises syntax
const getSecretsPromise = () =>
  new Promise(resolve => {
    getSecretsDataFromSomewhere(secretsData => {
      resolve(secretsData);
    });
  });
const storage = await createStorage({
  ...,
  getSecrets: getSecretsPromise,
});

Logging

By default, the SDK outputs logs into console in JSON format. You can override this behavior by passing the logger object as a Storage constructor parameter. The logger object must correspond to the following structure:

// Custom logger must implement `write` method

const customLogger = {
  write: (logLevel: LogLevel, message: string, meta?: {}): void => {}
};

const storage = await createStorage({
  apiKey: '',
  environmentId: '',
  getSecrets: () => '',
  logger: customLogger
});

Skipping Storage validation

You can create an instance of the Storage class and run all asynchronous checks by yourself (or skip them at your own risk!).

const { Storage } = require('incountry');
const storage = new Storage({...});

await storage.validate();

The validate method fetches the secret using getSecrets and validates it. If custom encryption configurations were provided they would also be checked with all the matching secrets.

Usage

Writing data to Storage

Use the write method to create/replace a record (by recordKey).

List of available record fields

v3.0.0 release introduced a series of new fields available for data storage. Below you can find the full list of all the fields available for storage in InCountry Platform along with their types and storage methods. Each field is either encrypted, hashed or stored as follows:

String fields, hashed

recordKey
profileKey
serviceKey1
serviceKey2

String fields, hashed if Storage options "hashSearchKeys" is set to true (by default it is)

key1
key2
key3
key4
key5
key6
key7
key8
key9
key10

String fields, encrypted

body
precommitBody

Int fields, plain

rangeKey1
rangeKey2
rangeKey3
rangeKey4
rangeKey5
rangeKey6
rangeKey7
rangeKey8
rangeKey9
rangeKey10

type StorageRecordData = {
  recordKey: string;
  profileKey?: string | null;
  key1?: string | null;  // If `hashSearchKeys` is set to `false` key1 has length limit 256
  key2?: string | null;  // If `hashSearchKeys` is set to `false` key2 has length limit 256
  key3?: string | null;  // If `hashSearchKeys` is set to `false` key3 has length limit 256
  key4?: string | null;  // If `hashSearchKeys` is set to `false` key4 has length limit 256
  key5?: string | null;  // If `hashSearchKeys` is set to `false` key5 has length limit 256
  key6?: string | null;  // If `hashSearchKeys` is set to `false` key6 has length limit 256
  key7?: string | null;  // If `hashSearchKeys` is set to `false` key7 has length limit 256
  key8?: string | null;  // If `hashSearchKeys` is set to `false` key8 has length limit 256
  key9?: string | null;  // If `hashSearchKeys` is set to `false` key9 has length limit 256
  key10?: string | null; // If `hashSearchKeys` is set to `false` key10 has length limit 256
  serviceKey1?: string | null;
  serviceKey2?: string | null;
  body?: string | null;
  precommitBody?: string | null;
  rangeKey1?: t.Int | null;
  rangeKey2?: t.Int | null;
  rangeKey3?: t.Int | null;
  rangeKey4?: t.Int | null;
  rangeKey5?: t.Int | null;
  rangeKey6?: t.Int | null;
  rangeKey7?: t.Int | null;
  rangeKey8?: t.Int | null;
  rangeKey9?: t.Int | null;
  rangeKey10?: t.Int | null;
};

type WriteResult = {
  record: StorageRecordData;
};

async write(
  countryCode: string,
  recordData: StorageRecordData,
  requestOptions: RequestOptions = {},
): Promise<WriteResult> {
  /* ... */
}

Below you can find the example of how to use the write method.

const recordData = {
  recordKey: '<key>',
  body: '<body>',
  profileKey: '<profile_key>',
  rangeKey1: 0,
  key2: '<key2>',
  key3: '<key3>'
}

const writeResult = await storage.write(countryCode, recordData);

Batches

You can use the batchWrite method to create/replace multiple records at once.

type BatchWriteResult = {
  records: Array<StorageRecordData>;
};

async batchWrite(
  countryCode: string,
  records: Array<StorageRecordData>,
  requestOptions: RequestOptions = {},
): Promise<BatchWriteResult> {
  /* ... */
}

Example of usage:

batchResult = await storage.batchWrite(countryCode, recordDataArr);

Reading stored data

You can read the stored data records by its recordKey by using the read method. It accepts an object with the two fields - country and recordKey. It returns a Promise which is resolved to { record } or is rejected if there are no records for the passed recordKey.

Date fields

You can use the createdAt and updatedAt fields to access date-related information about records. The createdAt field stores the date when the record was initially created in the target country. The updatedAt field stores the date of the latest write operation for the given recordKey.

type StorageRecord = {
  recordKey: string;
  body: string | null;
  profileKey: string | null;
  precommitBody: string | null;
  key1?: string | null;  // If `hashSearchKeys` is set to `false` key1 has length limit 256
  key2?: string | null;  // If `hashSearchKeys` is set to `false` key2 has length limit 256
  key3?: string | null;  // If `hashSearchKeys` is set to `false` key3 has length limit 256
  key4?: string | null;  // If `hashSearchKeys` is set to `false` key4 has length limit 256
  key5?: string | null;  // If `hashSearchKeys` is set to `false` key5 has length limit 256
  key6?: string | null;  // If `hashSearchKeys` is set to `false` key6 has length limit 256
  key7?: string | null;  // If `hashSearchKeys` is set to `false` key7 has length limit 256
  key8?: string | null;  // If `hashSearchKeys` is set to `false` key8 has length limit 256
  key9?: string | null;  // If `hashSearchKeys` is set to `false` key9 has length limit 256
  key10?: string | null; // If `hashSearchKeys` is set to `false` key10 has length limit 256
  serviceKey1: string | null;
  serviceKey2: string | null;
  rangeKey1: t.Int | null;
  rangeKey2: t.Int | null;
  rangeKey3: t.Int | null;
  rangeKey4: t.Int | null;
  rangeKey5: t.Int | null;
  rangeKey6: t.Int | null;
  rangeKey7: t.Int | null;
  rangeKey8: t.Int | null;
  rangeKey9: t.Int | null;
  rangeKey10: t.Int | null;
  createdAt: Date;
  updatedAt: Date;
  attachments: StorageRecordAttachment[];
}

type ReadResult = {
  record: StorageRecord;
};

async read(
  countryCode: string,
  recordKey: string,
  requestOptions: RequestOptions = {},
): Promise<ReadResult> {
  /* ... */
}

Example of usage:

const readResult = await storage.read(countryCode, recordKey);

Find records

You can look up for data records either by using exact match search operators or partial text match operator in almost any combinations.

Exact match search

The following exact match search criteria are available:

• single value:

  // Matches all records where record.key1 = 'abc' AND record.rangeKey = 1
  { key1: 'abc', rangeKey1: 1 }
  

• multiple values as an array:

  // Matches all records where (record.key2 = 'def' OR record.key2 = 'jkl') AND (record.rangeKey = 1 OR record.rangeKey = 2)
  { key2: ['def', 'jkl'], rangeKey1: [1, 2] }
  

• a logical NOT operator for String fields and version:

  // Matches all records where record.key3 <> 'abc'
  { key3: { $not: 'abc' } }
  
  // Matches all records where record.key3 <> 'abc' AND record.key3 <> 'def'
  { key3: { $not: ['abc', 'def'] } }
  
  // Matches all records where record.version <> 1
  { version: { $not: 1 }}
  

• comparison operators for Int fields:

  // Matches all records where record.rangeKey >= 5 AND record.rangeKey <= 100
  { rangeKey1: { $gte: 5, $lte: 100 } }
  

Partial text match search

You can also look up for data records by partial match using the searchKeys operator which performs partial match search (similar to the LIKE SQL operator, without special characters) within records text fields key1, key2, ..., key10.

// Matches all records where record.key1 LIKE 'abc' OR record.key2 LIKE 'abc' OR ... OR record.key10 LIKE 'abc'
{ searchKeys: 'abc' }

Please note: The searchKeys operator cannot be used in combination with any of key1, key2, ..., key10 keys and works only in combination with the non-hashing Storage mode (hashSearchKeys param for Storage).

// Matches all records where (record.key1 LIKE 'abc' OR record.key2 LIKE 'abc' OR ... OR record.key10 LIKE 'abc') AND (record.rangeKey = 1 OR record.rangeKey = 2)
{ searchKeys: 'abc', rangeKey1: [1, 2] }

// Causes validation error (StorageClientError)
{ searchKeys: 'abc', key1: 'def' }

Search options

The options parameter defines the limit - number of records that are returned, and the offset- the starting index used for record pagination.

Note: The SDK returns 100 records at most.

type FilterStringValue = string | string[];
type FilterStringQuery = FilterStringValue | { $not?: FilterStringValue };

type FilterNumberValue = number | number[];
type FilterNumberQuery =
  FilterNumberValue |
  {
    $not?: FilterNumberValue;
    $gt?: number;
    $gte?: number;
    $lt?: number;
    $lte?: number;
  };

type FindFilter = Record<string, FilterStringQuery | FilterNumberQuery>;

type FindOptions = {
  limit?: number;
  offset?: number;
};

type FindResult = {
  meta: {
    total: number;
    count: number;
    limit: number;
    offset: number;
  };
  records: Array<StorageRecord>;
  errors?: Array<{ error: StorageCryptoError; rawData: ApiRecord }>;
};

async find(
  countryCode: string,
  filter: FindFilter = {},
  options: FindOptions = {},
  requestOptions: RequestOptions = {},
): Promise<FindResult> {
  /* ... */
}

Example of usage

const filter = {
  key1: 'abc',
  key2: ['def', 'jkl'],
  profileKey: 'test2',
  rangeKey1: { $gte: 5, $lte: 100 },
  rangeKey2: { $not: [0, 1] },
}

const options = {
  limit: 100,
  offset: 0,
};

const findResult = await storage.find(countryCode, filter, options);

The returned findResult object looks like the following:

{
  records: [{/* StorageRecord */}],
  errors: [],
  meta: {
    limit: 100,
    offset: 0,
    total: 24
  }
}

Error handling

You may encounter a situation when the find method receives records that cannot be decrypted. For example, this may happen once the encryption key has been changed while the found data was encrypted with the older version of that key. In such cases data returned by the find() method will be as follows:

{
  records: [/* successfully decrypted records */],
  errors: [/* errors */],
  meta: {/* ... */}
}: FindResult

Find one record matching a filter

If you need to find only one of the records matching a specific filter, you can use the findOne method. If a record is not found, it returns null.

type FindOneResult = {
  record: StorageRecord | null;
};

async findOne(
  countryCode: string,
  filter: FindFilter = {},
  options: FindOptions = {},
  requestOptions: RequestOptions = {},
): Promise<FindOneResult> {
  /* ... */
}

Example of usage:

const findOneResult = await storage.findOne(countryCode, filter);

Delete records

You can use the delete method to delete a record from InCountry Platform. It is possible by using the recordKey field only.

type DeleteResult = {
  success: true;
};

async delete(
  countryCode: string,
  recordKey: string,
  requestOptions: RequestOptions = {},
): Promise<DeleteResult> {
  /* ... */
}

Example of usage:

const deleteResult = await storage.delete(countryCode, recordKey);

Attaching files to a record

InCountry Storage allows you to attach files to the previously created records. Attachments' meta information is available through the attachments field of StorageRecord object.

type StorageRecord = {
  /* ... other fields ...  */
  attachments: StorageRecordAttachment[];
}

type StorageRecordAttachment = {
  fileId: string;
  fileName: string;
  hash: string;
  mimeType: string;
  size: number;
  createdAt: Date;
  updatedAt: Date;
  downloadLink: string;
}

Adding attachments

The addAttachment method allows you to add or replace attachments. File data can be provided either as Readable stream, Buffer or string with a path to the file in the file system.

type AttachmentData = {
  file: Readable | Buffer | string;
  fileName: string;
}

async addAttachment(
  countryCode: string,
  recordKey: string,
  { file, fileName }: AttachmentData,
  upsert = false,
  requestOptions: RequestOptions = {},
): Promise<StorageRecordAttachment> {
  /* ... */
}

Example of usage:

// using file path
await storage.addAttachment(COUNTRY, recordData.recordKey, { file: '../file' });

// using data Stream
import * as fs from 'fs';

const file = fs.createReadStream('./LICENSE');
await storage.addAttachment(COUNTRY, recordData.recordKey, { file });

Deleting attachments

The deleteAttachment method allows you to delete attachment using its fileId.

deleteAttachment(
  countryCode: string,
  recordKey: string,
  fileId: string,
  requestOptions: RequestOptions = {},
): Promise<unknown> {
  /* ... */
}

Example of usage:

await storage.deleteAttachment(COUNTRY, recordData.recordKey, attachmentMeta.fileId);

Downloading attachments

The getAttachmentFile method allows you to download attachment contents. It returns object with readable stream and filename.

async getAttachmentFile(
  countryCode: string,
  recordKey: string,
  fileId: string,
  requestOptions: RequestOptions = {},
): Promise<GetAttachmentFileResult> {
  /* ... */
}

Example of usage:

import * as fs from 'fs';

const { attachmentData } = await storage.getAttachmentFile(COUNTRY, recordData.recordKey, attachmentMeta.fileId);

const { file, fileName } = attachmentData;
const writeStream = fs.createWriteStream(`./${fileName}`);
file.pipe(writeStream);

Working with attachment meta info

The getAttachmentMeta method allows you to retrieve attachment's metadata using its fileId.

async getAttachmentMeta(
  countryCode: string,
  recordKey: string,
  fileId: string,
  requestOptions: RequestOptions = {},
): Promise<StorageRecordAttachment> {
  /* ... */
}

Example of usage:

const meta: StorageRecordAttachment = await storage.getAttachmentMeta(COUNTRY, recordData.recordKey, attachmentMeta.fileId);

The updateAttachmentMeta method allows you to update attachment's metadata (MIME type and file name).

type AttachmentWritableMeta = {
  fileName?: string;
  mimeType?: string;
};

async updateAttachmentMeta(
  countryCode: string,
  recordKey: string,
  fileId: string,
  fileMeta: AttachmentWritableMeta,
  requestOptions: RequestOptions = {},
): Promise<StorageRecordAttachment> {
    /* ... */
}

Example of usage:

await storage.updateAttachmentMeta(COUNTRY, data.recordKey, attachmentMeta.fileId, { fileName: 'new name!' });

Data Migration and Key Rotation support

Using getSecrets storage config options that provides SecretsData object enables key rotation and data migration support.

SDK introduces migrate method which allows you to re-encrypt data encrypted with old versions of the secret. It returns an object which contains some information about the migration - the amount of records migrated (migrated) and the amount of records left to migrate (totalLeft) (which basically means the amount of records with version different from currentVersion provided by SecretsData).

For a detailed example of a migration script please see examples/migration.js

type MigrateResult = {
  meta: {
    migrated: number;
    totalLeft: number;
  };
};

async migrate(
  countryCode: string,
  limit = FIND_LIMIT,
  findFilter: FindFilter = {},
  requestOptions: RequestOptions = {},
): Promise<MigrateResult> {
  /* ... */
}

Example of usage:

const migrateResult = await storage.migrate(countryCode);

Error Handling

InCountry Node SDK throws following Exceptions:

• StorageClientError - used for various input validation errors. Can be thrown by all public methods.

• StorageServerError - thrown if SDK failed to communicate with InCountry servers or if server response validation failed.

• StorageCryptoError - thrown during encryption/decryption procedures (both default and custom). This may be a sign of malformed/corrupt data or a wrong encryption key provided to the SDK.

• StorageError - general exception. Inherited by all other exceptions

We suggest gracefully handling all the possible exceptions:

try {
  // use InCountry Storage instance here
} catch(e) {
  if (e instanceof StorageClientError) {
    // some input validation error
  } else if (e instanceof StorageServerError) {
    // some server error
  } else if (e instanceof StorageCryptoError) {
    // some encryption error
  } else {
    // ...
  }
}

Custom encryption

SDK supports the ability to provide custom encryption/decryption methods if you decide to use your own algorithm instead of the default one.

createStorage(options, customEncConfigs) allows you to pass an array of custom encryption configurations with the following schema, which enables custom encryption:

{
  encrypt: (text: string, secret: string, secretVersion: string) => Promise<string> | string,
  decrypt: (encryptedText: string, secret: string, secretVersion: string) => Promise<string> | string,
  isCurrent: boolean, // Optional but at most one in array should be isCurrent: true
  version: string
}

They should accept raw data to encrypt/decrypt, key data and key version received from SecretKeyAccessor. The resulted encrypted/decrypted data should be a string.

version attribute is used to differ one custom encryption from another and from the default encryption as well. This way SDK will be able to successfully decrypt any old data if encryption changes with time.

isCurrent attribute allows to specify one of the custom encryption configurations to use for encryption. Only one configuration can be set as isCurrent: true.

Here's an example of how you can set up SDK to use custom encryption (using XXTEA encryption algorithm):

const xxtea = require('xxtea');
const encrypt = (text, secret) => xxtea.encrypt(text, secret);
const decrypt = (encryptedText, secret) => xxtea.decrypt(encryptedText, secret);

const config = {
  encrypt,
  decrypt,
  isCurrent: true,
  version: 'current',
};

const getSecretsCallback = () => {
  return {
    secrets: [
      {
        secret: 'longAndStrongPassword',
        version: 1,
        isForCustomEncryption: true
      }
    ],
    currentVersion: 1,
  };
}

const options = {
  apiKey: 'API_KEY',
  environmentId: 'ENVIRONMENT_ID',
  getSecrets: getSecretsCallback,
}};

const storage = await createStorage(options, [config]);

await storage.write('us', { recordKey: '<key>', body: '<body>' });

Testing Locally

1. In terminal run npm test for unit tests
2. In terminal run npm run integrations to run integration tests