Cookbook

What is this Book?    How to Contribute    YouTube    Twitter    Amazon Shop

## If You Like This Book & Need More Help Check out my Data Engineering Academy at LearnDataEngineering.com trusted by almost 2,000 students! **Visit learndataengineering.com:** [Click Here](https://learndataengineering.com) - Learn Data Engineering with our online Academy - Perfect for becoming a Data Engineer or add Data Engineering to your skillset - Proven process based on years of experience and hundreds of hours of personal coaching - Over 30 prepared courses on the most important techniques, fundamental tools and platforms plus our - Associate Data Engineer Certification - Academy Discord server with over 1,000 members ## Support This Book For Free! - **Amazon:** [Click Here](https://www.amazon.com/shop/plumbersofdatascience) buy whatever you like from Amazon using this link* (Also check out my complete podcast gear and books) ## Here's what's new: Find the change log with all recent updates here: [SEE UPDATES](sections/10-Updates.md) ## Building a Portfolio Project? Use my project README template to document it like a professional would: [Get the template](Code%20Examples/template-readme.md) # Contents: - [Introduction](sections/01-Introduction.md) - [Basic Engineering Skills](sections/02-BasicSkills.md) - [Advanced Engineering Skills](sections/03-AdvancedSkills.md) - [Free Hands On Courses / Tutorials](sections/04-HandsOnCourse.md)‚ - [Case Studies](sections/05-CaseStudies.md) - [Best Practices Cloud Platforms](sections/06-BestPracticesCloud.md) - [130+ Data Sources for Portfolio Projects](sections/07-DataSources.md) - [Project README Template](Code%20Examples/template-readme.md) - [1001 Interview Questions](sections/08-InterviewQuestions.md) - [Recommended Books, Courses, and Podcasts](sections/09-BooksAndCourses.md) - [Updates](sections/10-Updates.md) - [How To Contribute](#how-to-contribute) - [Support What You Like](#support) - [Important Links](#important-links) # Full Table Of Contents: ## Introduction - [What is this Cookbook](sections/01-Introduction.md#what-is-this-cookbook) - [Data Engineers](sections/01-Introduction.md#data-engineers) - [My Data Science Platform Blueprint](sections/01-Introduction.md#my-data-science-platform-blueprint) - [Connect](sections/01-Introduction.md#connect) - [Buffer](sections/01-Introduction.md#buffer) - [Processing Framework](sections/01-Introduction.md#processing-framework) - [Store](sections/01-Introduction.md#store) - [Visualize](sections/01-Introduction.md#visualize) - [Who Companies Need](sections/01-Introduction.md#who-companies-need) - [How to Learn Data Engineering](sections/01-Introduction.md#how-to-learn-data-engineering) - [Andreas on the Super Data Science Podcast](sections/01-Introduction.md#Interview-with-Andreas-on-the-Super-Data-Science-Podcast) - [Building Blocks to Learn Data Engineering](sections/01-Introduction.md#building-blocks-to-learn-data-engineering) - [Roadmap for Beginners](sections/01-Introduction.md#roadmap-for-beginners) - [Roadmap for Data Analysts](sections/01-Introduction.md#roadmap-for-data-analysts) - [Roadmap for Data Scientists](sections/01-Introduction.md#roadmap-for-data-scientists) - [Roadmap for Software Engineers](sections/01-Introduction.md#roadmap-for-software-engineers) - [Data Engineers Skills Matrix](sections/01-Introduction.md#data-engineers-skills-matrix) - [How to Become a Senior Data Engineer](sections/01-Introduction.md#how-to-become-a-senior-data-engineer) ## Basic Engineering Skills - [Learn To Code](sections/02-BasicSkills.md#learn-to-code) - [Get Familiar With Git](sections/02-BasicSkills.md#get-familiar-with-git) - [Agile Development](sections/02-BasicSkills.md#agile-development) - [Why is agile so important?](sections/02-BasicSkills.md#Why-is-agile-so-important) - [Agile rules I learned over the years](sections/02-BasicSkills.md#agile-rules-i-learned-over-the-years) - [Agile Frameworks](sections/02-BasicSkills.md#agile-frameworks) - [Scrum](sections/02-BasicSkills.md#scrum) - [OKR](sections/02-BasicSkills.md#okr) - [Software Engineering Culture](sections/02-BasicSkills.md#software-engineering-culture) - [Learn how a Computer Works](sections/02-BasicSkills.md#learn-how-a-computer-works) - [Data Network Transmission](sections/02-BasicSkills.md#data-network-transmission) - [Security and Privacy](sections/02-BasicSkills.md#security-and-privacy) - [SSL Public and Private Key Certificates](sections/02-BasicSkills.md#ssl-public-and-private-key-Certificates) - [JSON Web Tokens](sections/02-BasicSkills.md#json-web-tokens) - [GDPR regulations](sections/02-BasicSkills.md#gdpr-regulations) - [Linux](sections/02-BasicSkills.md#linux) - [OS Basics](sections/02-BasicSkills.md#os-basics) - [Shell scripting](sections/02-BasicSkills.md#shell-scripting) - [Cron Jobs](sections/02-BasicSkills.md#cron-jobs) - [Packet Management](sections/02-BasicSkills.md#packet-management) - [Docker](sections/02-BasicSkills.md#docker) - [What is Docker and How it Works](sections/02-BasicSkills.md#what-is-docker-and-what-do-you-use-it-for) - [Don't Mess Up Your System](sections/02-BasicSkills.md#dont-mess-up-your-system) - [Preconfigured Images](sections/02-BasicSkills.md#preconfigured-images) - [Take it With You](sections/02-BasicSkills.md#take-it-with-you) - [Kubernetes Container Deployment](sections/02-BasicSkills.md#kubernetes-container-deployment) - [How to Create Start and Stop a Container](sections/02-BasicSkills.md#how-to-create-start-stop-a-container) - [Docker Micro Services](sections/02-BasicSkills.md#docker-micro-services) - [Kubernetes](sections/02-BasicSkills.md#kubernetes) - [Why and How To Do Docker Container Orchestration](sections/02-BasicSkills.md#why-and-how-to-do-docker-container-orchestration) - [Userful Docker Commands](sections/02-BasicSkills.md#useful-docker-commands) - [The Cloud](sections/02-BasicSkills.md#the-cloud) - [IaaS vs PaaS vs SaaS](sections/02-BasicSkills.md#iaas-vs-paas-vs-saas) - [AWS Azure IBM Google IBM](sections/02-BasicSkills.md#aws-azure-ibm-google) - [Cloud vs On-Premises](sections/02-BasicSkills.md#cloud-vs-on-premises) - [Security](sections/02-BasicSkills.md#security) - [Hybrid Clouds](sections/02-BasicSkills.md#hybrid-clouds) - [Security Zone Design](sections/02-BasicSkills.md#security-zone-design) - [How to secure a multi layered application](sections/02-BasicSkills.md#how-to-secure-a-multi-layered-application) - [Cluster security with Kerberos](sections/02-BasicSkills.md#cluster-security-with-kerberos) ## Advanced Engineering Skills - [Data Science Platform](sections/03-AdvancedSkills.md#data-science-platform) - [Why a Good Data Platform Is Important](sections/03-AdvancedSkills.md#why-a-good-data-platform-is-important) - [Big Data vs Data Science and Analytics](sections/03-AdvancedSkills.md#Big-Data-vs-Data-Science-and-Analytics) - [The 4 Vs of Big Data](sections/03-AdvancedSkills.md#the-4-vs-of-big-data) - [Why Big Data](sections/03-AdvancedSkills.md#why-big-data) - [Planning is Everything](sections/03-AdvancedSkills.md#planning-is-everything) - [The Problem with ETL](sections/03-AdvancedSkills.md#the-problem-with-etl) - [Scaling Up](sections/03-AdvancedSkills.md#scaling-up) - [Scaling Out](sections/03-AdvancedSkills.md#scaling-out) - [When not to Do Big Data](sections/03-AdvancedSkills.md#please-dont-go-big-data) - [81 Platform & Pipeline Design Questions](sections/03-AdvancedSkills.md#81-platform-and-pipeline-design-questions) - [Data Source Questions](sections/03-AdvancedSkills.md#data-source-questions) - [Goals and Destination Questions](sections/03-AdvancedSkills.md#goals-and-destination-questions) - [Connect](sections/03-AdvancedSkills.md#connect) - [REST APIs](sections/03-AdvancedSkills.md#rest-apis) - [API Design](sections/03-AdvancedSkills.md#api-design) - [Implemenation Frameworks](sections/03-AdvancedSkills.md#implementation-frameworks) - [Security](sections/03-AdvancedSkills.md#security) - [Apache Nifi](sections/03-AdvancedSkills.md#apache-nifi) - [Logstash](sections/03-AdvancedSkills.md#logstash) - [Buffer](sections/03-AdvancedSkills.md#buffer) - [Apache Kafka](sections/03-AdvancedSkills.md#apache-kafka) - [Why a Message Queue Tool?](sections/03-AdvancedSkills.md#why-a-message-queue-tool) - [Kafka Architecture](sections/03-AdvancedSkills.md#kafka-architecture) - [Kafka Topics](sections/03-AdvancedSkills.md#what-are-topics) - [Kafka and Zookeeper](sections/03-AdvancedSkills.md#what-does-zookeeper-have-to-do-with-kafka) - [How to Produce and Consume Messages](sections/03-AdvancedSkills.md#how-to-produce-and-consume-messages) - [Kafka Commands](sections/03-AdvancedSkills.md#kafka-commands) - [Apache Redis Pub-Sub](sections/03-AdvancedSkills.md#redis-pub-sub) - [AWS Kinesis](sections/03-AdvancedSkills.md#apache-kafka) - [Google Cloud PubSub](sections/03-AdvancedSkills.md#google-cloud-pubsub) - [Processing Frameworks](sections/03-AdvancedSkills.md#processing-frameworks) - [Lambda and Kappa Architecture](sections/03-AdvancedSkills.md#lambda-and-kappa-architecture) - [Batch Processing](sections/03-AdvancedSkills.md#batch-processing) - [Stream Processing](sections/03-AdvancedSkills.md#stream-processing) - [Three Methods of Streaming](sections/03-AdvancedSkills.md#three-methods-of-streaming) - [At Least Once](sections/03-AdvancedSkills.md#at-least-once) - [At Most Once](sections/03-AdvancedSkills.md#at-most-once) - [Exactly Once](sections/03-AdvancedSkills.md#exactly-once) - [Check The Tools](sections/03-AdvancedSkills.md#check-the-tools) - [Should You do Stream or Batch Processing](sections/03-AdvancedSkills.md#should-you-do-stream-or-batch-processing) - [Is ETL still relevant for Analytics?](sections/03-AdvancedSkills.md#is-etl-still-relevant-for-analytics) - [MapReduce](sections/03-AdvancedSkills.md#mapreduce) - [How Does MapReduce Work](sections/03-AdvancedSkills.md#How-does-mapreduce-work) - [MapReduce](sections/03-AdvancedSkills.md#mapreduce) - [MapReduce Example](sections/03-AdvancedSkills.md#example) - [MapReduce Limitations](sections/03-AdvancedSkills.md#What-is-the-limitation-of-mapreduce) - [Apache Spark](sections/03-AdvancedSkills.md#apache-spark) - [What is the Difference to MapReduce?](sections/03-AdvancedSkills.md#what-is-the-difference-to-MapReduce) - [How Spark Fits to Hadoop](sections/03-AdvancedSkills.md#how-does-spark-fit-to-hadoop) - [Spark vs Hadoop](sections/03-AdvancedSkills.md#wheres-the-difference) - [Spark and Hadoop a Perfect Fit](sections/03-AdvancedSkills.md#spark-and-hadoop-is-a-perfect-fit) - [Spark on YARn](sections/03-AdvancedSkills.md#spark-on-yarn) - [My Simple Rule of Thumb](sections/03-AdvancedSkills.md#my-simple-rule-of-thumb) - [Available Languages](sections/03-AdvancedSkills.md#available-languages) - [Spark Driver Executor and SparkContext](sections/03-AdvancedSkills.md#how-spark-works-driver-executor-sparkcontext) - [Spark Batch vs Stream processing](sections/03-AdvancedSkills.md#spark-batch-vs-stream-processing) - [How Spark uses Data From Hadoop](sections/03-AdvancedSkills.md#How-does-spark-use-data-from-hadoop) - [What are RDDs and How to Use Them](sections/03-AdvancedSkills.md#what-are-rdds-and-how-to-use-them) - [SparkSQL How and Why to Use It](sections/03-AdvancedSkills.md#available-languages) - [What are Dataframes and How to Use Them](sections/03-AdvancedSkills.md#what-are-dataframes-how-to-use-th