如果要使用生产数据进行测试,数据匿名化可以帮助您。
Afraid of using production data due to privacy issues? Data Anonymization is a tool that helps you build anonymized production data dumps which you can use for performance testing, security testing, debugging and development.
Java/Kotlin version of tool supporting RDBMS databases is available with similar easy to use DSL.
Install gem using:
$ gem install data-anonymizationInstall required database adapter library for active record:
$ gem install sqlite3Create ruby program using data-anonymization DSL as following my_dsl.rb:
…Run using:
$ ruby my_dsl.rbLiked it? please share
SQLite database
MongoDB
Postgresql database having composite primary key
Please see the Github 0.5.1 milestone page for more details on changes/fixes in release 0.5.1
Major changes:
Please see the Github 0.5.0 milestone page for more details on changes/fixes in release 0.5.0
Major changes:
Please see the Github 0.3.0 milestone page for more details on changes/fixes in release 0.3.0
MVP done. Fix defects and support queries, suggestions, enhancements logged in Github issues :-)
Please use Github issues to share feedback, feature suggestions and report issues.
For almost all projects there is a need for production data dump in order to run performance tests, rehearse production releases and debug production issues. However, getting production data and using it is not feasible due to multiple reasons, primary being privacy concerns for user data. And thus the need for data anonymization. This tool helps you to get anonymized production data dump using either Blacklist or Whitelist strategies.
Read more about data anonymization here
This approach essentially leaves all fields unchanged with the exception of those specified by the user, which are scrambled/anonymized (hence the name blacklist).
For Blacklist create a copy of prod database and chooses the fields to be anonymized like e.g. username, password, email, name, geo location etc. based on user specification. Most of the fields have different rules e.g. password should be set to same value for all users, email needs to be valid.
The problem with this approach is that when new fields are added they will not be anonymized by default. Human error in omitting users personal data could be damaging.
database 'DatabaseName' do
strategy DataAnon::Strategy::Blacklist
source_db :adapter => 'sqlite3', :database => 'sample-data/chinook-empty.sqlite'
...
endThis approach, by default scrambles/anonymizes all fields except a list of fields which are allowed to copied as is. Hence the name whitelist. By default all data needs to be anonymized. So from production database data is sanitized record by record and inserted as anonymized data into destination database. Source database needs to be readonly. All fields would be anonymized using default anonymization strategy which is based on the datatype, unless a special anonymization strategy is specified. For instance special strategies could be used for emails, passwords, usernames etc. A whitelisted field implies that it's okay to copy the data as is and anonymization isn't required. This way any new field will be anonymized by default and if we need them as is, add it to the whitelist explicitly. This prevents any human error and protects sensitive information.
database 'DatabaseName' do
strategy DataAnon::Strategy::Whitelist
source_db :adapter => 'sqlite3', :database => 'sample-data/chinook.sqlite'
destination_db :adapter => 'sqlite3', :database => 'sample-data/chinook-empty.sqlite'
...
endRead more about blacklist and whitelist here
schema_search_path: source_db { ... schema_search_path: 'public,my_special_schema' }We provide a command line tool to generate whitelist scripts for RDBMS and NoSQL databases. The user needs to supply the connection details to the database and a script is generated by analyzing the schema. Below are examples of how to use the tool to generate the scripts for RDBMS and NoSQL datastores
When you install the data-anonymization tool, the datanon command become available on the terminal. If you type datanon --help and execute you should see the below
Tasks:
datanon generate_mongo_dsl -d, --database=DATABASE -h, --host=HOST # Generates a base anonymization script(whitelist strategy) for a Mongo DB using the database schema
datanon generate_rdbms_dsl -a, --adapter=ADAPTER -d, --database=DATABASE -h, --host=HOST # Generates a base anonymization script(whitelist strategy) for a RDBMS database using the database schema
datanon help [TASK] # Describe available tasks or one specific task
The gem uses ActiveRecord(AR) abstraction to connect to relational databases. You can generate a whitelist script in seconds for any relational database supported by Active Record. To do so use the following command
datanon generate_rdbms_dsl [options]
The options available are :
The adapter, host and database options are mandatory. The others are optional.
A few examples of the command is shown below
datanon generate_rdbms_dsl -a mysql2 -h db.host.com -p 3306 -d production_db -u root -w password
datanon generate_rdbms_dsl -a p暂无开放 Issues,或尚未同步最近议题。