百科.dev
全部条目在线工具AI 编程趋势榜开源项目技术资讯提交条目
登录
< 返回工具列表
D

data-anonymization

> 编程语言
开源

如果要使用生产数据进行测试,数据匿名化可以帮助您。

469 stars0 点赞1 次浏览
访问官网GitHub

工具介绍

如果要使用生产数据进行测试,数据匿名化可以帮助您。

Data::Anonymization

Afraid of using production data due to privacy issues? Data Anonymization is a tool that helps you build anonymized production data dumps which you can use for performance testing, security testing, debugging and development.

Java/Kotlin version

Java/Kotlin version of tool supporting RDBMS databases is available with similar easy to use DSL.

  • Kotlin/Java Data Anonymization Tool
  • Kotlin Maven Sample Project
  • Java Maven Sample Project

Getting started

Install gem using:

$ gem install data-anonymization

Install required database adapter library for active record:

$ gem install sqlite3

Create ruby program using data-anonymization DSL as following my_dsl.rb:

…

Run using:

$ ruby my_dsl.rb

Liked it? please share

Examples

SQLite database

  1. Whitelist
  2. Blacklist

MongoDB

  1. Whitelist
  2. Blacklist

Postgresql database having composite primary key

  1. Whitelist
  2. Blacklist

Changelog

0.9.0 (Jan 29, 2025)

  1. Fixed issue with connection pool

0.9.0 (Dec 24, 2024)

  1. Upgraded to rails 8.0

0.8.10 (Mar 20, 2024)

  1. Upgraded to rails 7.1

0.8.7 (Jan 14, 2022)

  1. Upgraded to rails 7.x

0.8.5 (May 28, 2020)

  1. Upgraded to rails 6.x

0.8.1 (Aug 19, 2017)

  1. Multi-threading support added by stanislav-tyutin using Pull Request.
  2. Fixed to work with Ruby 2.4.x, issue with Integer data type

0.8.0 (Oct 31, 2016)

  1. Upgraded to rails 5.x

0.7.4 (Oct 29, 2016)

  1. Continue to work on rails 4.x. Minor changes based on feedback.

0.8.0.rc1 (Sep 5, 2016)

  1. Upgraded to rails 5.0, please report any issue or use case not working.

0.7.3 (Feb 5, 2016)

  1. Fixed issue with batchsize. Thanks to Jan Raasch for sending pull request.

0.7.2 (Sep 26, 2015)

  1. Upgraded MongoDB to latest gem version 2.1.0 and tested with MongoDB 3.x version.
  2. Upgraded gems to latest version
  3. Adding limit functionality - Merge pull request #27 from yanismydj/master

0.7.1 (Jun 13, 2015)

  1. Fixed issues with empty array data for MongoDB
  2. Added feature to skip and continue records during anaonymisation, this is useful to apply different strategies for different types of records.

0.7.0 (Mar 9, 2015)

  1. Removed downcase from field name since it was causing issues with upper case field names. So now for databsae where case matters field name case should be maintained.
  2. Upgraded gems to latest version

0.6.7 (Jan 17, 2015)

  1. Upgraded gems to latest version including activerecord to 4.2. Please try it out and provide feedback.

0.6.6 (Oct 31, 2014)

  1. Upgraded gems to latest version.

0.6.5 (Jun 02, 2014)

  1. Upgraded most of the gems to latest version. major change is rails activerecord gem to latest versions 4.1.1, please provide feedback.

0.6.0 (Dec 09, 2013)

  1. Upgraded rails activerecord gem to latest versions 4.0.2, please provide feedback.

0.5.5 (Dec 4, 2013)

  1. Upgraded gems to latest versions

0.5.2 (Jan 29, 2013)

  1. Fixed issue #17
  2. Upgraded Thor dependency to latest version

0.5.2 (Jan 20, 2013)

  1. Upgraded all gem to latest and greatest including Rails activerecord and activesupport.

0.5.1 (Oct 26, 2012)

  1. Minor fixes release, no major functionality or feature added.

Please see the Github 0.5.1 milestone page for more details on changes/fixes in release 0.5.1

0.5.0 (Sep 28, 2012)

Major changes:

  1. MongoDB support
  2. Command line utility to generate whitelist DSL for RDBMS & MongoDB (reduces pain for writing whitelist dsl)
  3. Added support for reporting fields missing mapping in case of whitelist
  4. Errors reported at the end of process. Job doesn't fail for a single error, it fails it more than 100 records failed during anonymization.

Please see the Github 0.5.0 milestone page for more details on changes/fixes in release 0.5.0

0.3.0 (Sep 4, 2012)

Major changes:

  1. Added support for Parallel table execution
  2. Change in default String strategy from LoremIpsum to RandomString based on end user feedback.
  3. Fixed issue with table column name 'type' as this is default name for STI in activerecord.

Please see the Github 0.3.0 milestone page for more details on changes/fixes in release 0.3.0

0.2.0 (August 16, 2012)

  1. Added the progress bar using 'powerbar' gem. Which also shows the ETA for each table.
  2. Added More strategies
  3. Fixed default anonymization strategies for boolean and integer values
  4. Added support for composite primary key

0.1.2 (August 14, 2012)

  1. First initial release

Roadmap

MVP done. Fix defects and support queries, suggestions, enhancements logged in Github issues :-)

Share feedback

Please use Github issues to share feedback, feature suggestions and report issues.

What is data anonymization?

For almost all projects there is a need for production data dump in order to run performance tests, rehearse production releases and debug production issues. However, getting production data and using it is not feasible due to multiple reasons, primary being privacy concerns for user data. And thus the need for data anonymization. This tool helps you to get anonymized production data dump using either Blacklist or Whitelist strategies.

Read more about data anonymization here

Anonymization Strategies

Blacklist

This approach essentially leaves all fields unchanged with the exception of those specified by the user, which are scrambled/anonymized (hence the name blacklist). For Blacklist create a copy of prod database and chooses the fields to be anonymized like e.g. username, password, email, name, geo location etc. based on user specification. Most of the fields have different rules e.g. password should be set to same value for all users, email needs to be valid.

The problem with this approach is that when new fields are added they will not be anonymized by default. Human error in omitting users personal data could be damaging.

ruby
database 'DatabaseName' do
  strategy DataAnon::Strategy::Blacklist
  source_db :adapter => 'sqlite3', :database => 'sample-data/chinook-empty.sqlite'
  ...
end

Whitelist

This approach, by default scrambles/anonymizes all fields except a list of fields which are allowed to copied as is. Hence the name whitelist. By default all data needs to be anonymized. So from production database data is sanitized record by record and inserted as anonymized data into destination database. Source database needs to be readonly. All fields would be anonymized using default anonymization strategy which is based on the datatype, unless a special anonymization strategy is specified. For instance special strategies could be used for emails, passwords, usernames etc. A whitelisted field implies that it's okay to copy the data as is and anonymization isn't required. This way any new field will be anonymized by default and if we need them as is, add it to the whitelist explicitly. This prevents any human error and protects sensitive information.

ruby
database 'DatabaseName' do
  strategy DataAnon::Strategy::Whitelist
  source_db :adapter => 'sqlite3', :database => 'sample-data/chinook.sqlite'
  destination_db :adapter => 'sqlite3', :database => 'sample-data/chinook-empty.sqlite'
  ...
end

Read more about blacklist and whitelist here

Tips

  1. In Whitelist approach make source database connection READONLY.
  2. Change default field strategies to avoid using same strategy again and again in your DSL.
  3. To run anonymization in parallel at Table level, provided no FK constraint on tables use DataAnon::Parallel::Table strategy
  4. For large table to load them in batches from table set 'batch_size' and it will use RoR's batch mode processing. Checkout example on how to use batch processing.
  5. Make sure to give proper case for fields and table names.
  6. Use skip and continue to apply different strategies for records.
  7. Use 'limit' to limit the number of rows that will be imported in whitelist
  8. RDBMS databases utilizing schemas can be specified via schema_search_path: source_db { ... schema_search_path: 'public,my_special_schema' }

DSL Generation

We provide a command line tool to generate whitelist scripts for RDBMS and NoSQL databases. The user needs to supply the connection details to the database and a script is generated by analyzing the schema. Below are examples of how to use the tool to generate the scripts for RDBMS and NoSQL datastores

When you install the data-anonymization tool, the datanon command become available on the terminal. If you type datanon --help and execute you should see the below

Tasks:

datanon generate_mongo_dsl -d, --database=DATABASE -h, --host=HOST                        # Generates a base anonymization script(whitelist strategy) for a Mongo DB using the database schema
datanon generate_rdbms_dsl -a, --adapter=ADAPTER -d, --database=DATABASE -h, --host=HOST  # Generates a base anonymization script(whitelist strategy) for a RDBMS database using the database schema
datanon help [TASK]                                                                       # Describe available tasks or one specific task

RDBMS whitelist generation

The gem uses ActiveRecord(AR) abstraction to connect to relational databases. You can generate a whitelist script in seconds for any relational database supported by Active Record. To do so use the following command

datanon generate_rdbms_dsl [options]

The options available are :

  1. adapter(-a) : The activerecord adapter to use to connect to the database (eg. mysql2, postgresql)
  2. host(-h) : DB host name or IP address
  3. database(-d) : The name of the database to generate the whitelist script for
  4. username(-u) : Username for DB authentication
  5. password(-w) : Password for DB authentication
  6. port(-p) : The port the database service is running on. Default port provided by AR will be used if nothing is specififed.

The adapter, host and database options are mandatory. The others are optional.

A few examples of the command is shown below

datanon generate_rdbms_dsl -a mysql2 -h db.host.com -p 3306 -d production_db -u root -w password

datanon generate_rdbms_dsl -a p

Issues· 34 开放

查看全部 Issues在 GitHub 打开

暂无开放 Issues,或尚未同步最近议题。

> 标签

Ruby

暂无评论,来聊聊你的看法吧

> 工具信息

发布日期2026年8月1日
最后更新2026年9月17日
分类编程语言
定价开源

> 相关工具

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言