elasticsearch-dump

elasticdump

Tools for moving and saving indices from Elasticsearch and OpenSearch

---

Version Warnings!

• Version 1.0.0 of Elasticdump changes the format of the files created by the dump. Files created with version 0.x.x of this tool are likely not to work with versions going forward. To learn more about the breaking changes, vist the release notes for version 1.0.0. If you recive an "out of memory" error, this is probably or most likely the cause.
• Version 2.0.0 of Elasticdump removes the bulk options. These options were buggy, and differ between versions of Elasticsearch. If you need to export multiple indexes, look for the multielasticdump section of the tool.
• Version 2.1.0 of Elasticdump moves from using scan/scroll (ES 1.x) to just scroll (ES 2.x). This is a backwards-compatible change within Elasticsearch, but performance may suffer on Elasticsearch versions prior to 2.x.
• Version 3.0.0 of Elasticdump has the default queries updated to only work for ElasticSearch version 5+. The tool may be compatible with earlier versions of Elasticsearch, but our version detection method may not work for all ES cluster topologies
• Version 5.0.0 of Elasticdump contains a breaking change for the s3 transport. s3Bucket and s3RecordKey params are no longer supported please use s3urls instead
• Version 6.1.0 and higher of Elasticdump contains a change to the upload/dump process. This change allows for overlapping promise processing. The benefit of which is improved performance due increased parallel processing, but a side-effect exists where-by records (data-set) aren't processed in a sequential order (the ordering is no longer guaranteed)
• Version 6.67.0 and higher of Elasticdump will quit if the node.js version does not match the minimum requirement needed (v10.0.0)
• Version 6.76.0 and higher of Elasticdump added support for OpenSearch (forked from Elasticsearch 7.10.2)

Installing

(local)

npm install elasticdump
./bin/elasticdump

(global)

npm install elasticdump -g
elasticdump

Use

Standard Install

Elasticdump works by sending an input to an output. Both can be either an elasticsearch URL or a File.

Elasticsearch/OpenSearch:

• format: {protocol}://{host}:{port}/{index}
• example: http://127.0.0.1:9200/my_index

File:

• format: {FilePath}
• example: /Users/evantahler/Desktop/dump.json

Stdio:

• format: stdin / stdout
• format: $

You can then do things like:

…

Non-Standard Install

If Elasticsearch/OpenSearch is not being served from the root directory the --input-index and --output-index are required. If they are not provided, the additional sub-directories will be parsed for index and type.

Elasticsearch/OpenSearch:

• format: {protocol}://{host}:{port}/{sub}/{directory...}
• example: http://127.0.0.1:9200/api/search

# Copy a single index from a elasticsearch:
elasticdump \
  --input=http://es.com:9200/api/search \
  --input-index=my_index \
  --output=http://es.com:9200/api/search \
  --output-index=my_index \
  --type=mapping

# Copy a single type:
elasticdump \
  --input=http://es.com:9200/api/search \
  --input-index=my_index/my_type \
  --output=http://es.com:9200/api/search \
  --output-index=my_index \
  --type=mapping

Docker install

If you prefer using docker to use elasticdump, you can download this project from docker hub:

docker pull elasticdump/elasticsearch-dump

Then you can use it just by :

• using docker run --rm -ti elasticdump/elasticsearch-dump
• you'll need to mount your file storage dir -v <your dumps dir>:<your mount point> to your docker container

Example:

…

If you need to run using localhost as your ES host:

docker run --net=host --rm -ti elasticdump/elasticsearch-dump \
  --input=http://staging.es.com:9200/my_index \
  --output=http://localhost:9200/my_index \
  --type=data

Dump Format

The file format generated by this tool is line-delimited JSON files. The dump file itself is not valid JSON, but each line is. We do this so that dumpfiles can be streamed and appended without worrying about whole-file parser integrity.

For example, if you wanted to parse every line, you could do:

while read LINE; do jsonlint-py "${LINE}" ; done < dump.data.json

Options

…

Elasticsearch's Scroll API

Elasticsearch provides a scroll API to fetch all documents of an index starting from (and keeping) a consistent snapshot in time, which we use under the hood. This method is safe to use for large exports since it will maintain the result set in cache for the given period of time.

NOTE: only works for --output

Bypassing self-sign certificate errors

Set the environment NODE_TLS_REJECT_UNAUTHORIZED=0 before running elasticdump

# An alternative method of passing environment variables before execution
# NB : This only works with linux shells
NODE_TLS_REJECT_UNAUTHORIZED=0 elasticdump --input="https://localhost:9200" --output myfile

MultiElasticDump

This package also ships with a second binary, multielasticdump. This is a wrapper for the normal elasticdump binary, which provides a limited option set, but will run elasticdump in parallel across many indexes at once. It runs a process which forks into n (default your running host's # of CPUs) subprocesses running elasticdump.

The limited option set includes:

• parallel: os.cpus(),
• match: '^.*$',
• matchType: alias,
• order: 'asc',
• input: null,
• output: null,
• scrollTime: '10m',
• timeout: null,
• limit: 100,
• offset: 0,
• size: -1,
• direction: dump,
• ignoreType: ``
• includeType: ``
• prefix: '''
• suffix: ''
• interval: 1000
• searchBody: null
• transform: null
• support-big-int: false
• big-int-fields: ``
• ignoreChildError: false

If the --direction is dump, which is the default, --input MUST be a URL for the base location of an ElasticSearch server (i.e. http://localhost:9200) and --output MUST be a directory. Each index that does match will have a data, mapping, and analyzer file created.

For loading files that you have dumped from multi-elasticsearch, --direction should be set to load, --input MUST be a directory of a multielasticsearch dump and --output MUST be a Elasticsearch server URL.

--parallel is how many forks should be run simultaneously and --match is used to filter which indexes should be dumped/loaded (regex).

--ignoreType allows a type to be ignored from the dump/load. Six options are supported. data,mapping,analyzer,alias,settings,template. Multi-type support is available, when used each type must be comma(,)-separated and interval allows control over the interval for spawning a dump/load for a new index. For small indices this can be set to 0 to reduce delays and optimize performance i.e analyzer,alias types are ignored by default

--includeType allows a type to be included in the dump/load. Six options are supported - data,mapping,analyzer,alias,settings,template.

--ignoreChildError allows multi-elasticdump to continue if a child throws an error.

--matchType allows multi-elasticdump to fetch indices from the specified elasticsearch endpoint. Two options are supported - alias,datastream

New options, --suffix allows you to add a suffix to the index name being created e.g. es6-${index} and --prefix allows you to add a prefix to the index name e.g. ${index}-backup-2018-03-13. --order accepts asc or desc and allows the indexes/aliases to be sorted before processing is performed

Usage Examples

# backup ES indices & all their type to the es_backup folder
multielasticdump \
  --direction=dump \
  --match='^.*$' \
  --input=http://production.es.com:9200 \
  --output=/tmp/es_backup

# Only backup ES indices ending with a prefix of `-index` (match regex). 
# Only the indices data will be backed up. All other types are ignored.
# NB: analyzer & alias types are ignored by default
multielasticdump \
  --direction=dump \
  --match='^.*-index$'\
  --input=http://production.es.com:9200 \
  --ignoreType='mapping,settings,template' \
  --output=/tmp/es_backup

Custom Matcher - MultiElasticDump

The default matcher for multielasticdump is regex based and won't handle complex conditions or business rules. Support was added for defining custom matchers.

When specifying the customMatcher option, prefix the value with @ (a curl convention) to load matcher file

multielasticdump \
 --direction=dump \
 --customMatcher='@./customMatcher/backup.js'\
 --input=http://production.es.com:9200 \
 --ignoreType='mapping,settings,template' \
 --output=/tmp/es_backup

The only requirement for this module is that it must define a test function.

module.exports = {
  test: function (indexName) {
    // determine if the index should be loaded/dumped.
    // must return a boolen(true/false)
  }
};

Module Transform

When specifying the transform option, prefix the value with @ (a curl convention) to load the top-level function which is called with the document and the parsed arguments to the module.

Uses a pseudo-URL format to specify arguments to the module as follows. Given:

elasticdump --transform='@./transforms/my-transform?param1=value&param2=another-value'

with a module at ./transforms/my-transform.js with the following:

module.exports = function(doc, options) {
  // do something to doc
};

will load module ./transforms/my-transform.js, and execute the function with doc and options = {"param1": "value", "param2": "another-value"}.

An example transform for anonymizing data on-the-fly can be found in the transforms folder.

searchBody Template

When specifying the searchBodyTemplate option, prefix the value with @ (a curl convention) to load the top-level function which is called with the document and the parsed arguments to the module.

Uses a pseudo-URL format to specify arguments to the module as follows. Given:

elasticdump --searchBodyTemplate='@./templates/my-template?param1=value&param2=another-value'

with a module at ./transforms/my-transform.js with the following:

module.exports = function(doc, options) {
  // result must be added to doc.searchBody
  doc.searchBody = {}
};

will load module ./templates/my-template.js', and execute the function with docandoptions={"param1": "value", "param2": "another-value"}`.

An example template for modifying dates using a simple templating engine is available in the templates folder.

How Elasticdump handles Nested Data in CSV

Elasticdump is capable of reading/writing nested data, but in an _opinionated way. This is to reduce complexity while parsing/saving CSVs The format flattens all nesting to a single level (an example of this is shown below)

{
   "elasticdump":{
      "version":"6.51.0",
      "formats":[
         "json",
         "csv"
      ]
   },
   "contributors":[
      {
         "name":"ferron",
         "id":3
      }
   ],
   "year":112
}

Output format

{
  "elasticdump":"{\"version\":\"6.51.0\",\"formats\":[\"json\",\"csv\"]}",
  "contributors":"{\"contributors\":[{\"name\":\"ferron\",\"id\":3}]}",
  "year":2020
}

Notice that the data is flattened to 1 level. Object keys are used for headers and values as row data. This might not work with existing nested data formats, but that's the format that was chosen for elasticdump because of its simplicity. This detection is disabled by default, to enable use the --csvHandleNestedData flag

N