Data Engineer Powered by TEKsystems

Curriculum

Hands-on training along with career development and guest speakers

Linux, Version Control with Github

Intro to Linux

• Basic commands & Linux environment etc.
• Sudo (super user do)
• How to use Nano and VI editors
• How to use regular expressions: Grep, ^, [], ”, Siignals: Kill -l, trap signals: trap (for clean up temp files or to ignore signals)
• Console log, files, file paths & directories, mkdir, touch, echo, man, variables, setting permissions with chmod, pipe, grep, sort and
• Secure shell with SSH

 

Intro to Github + Version Control

• Git Basics, Setting up Git, Github
• Working directory, staging area, local repo (Head), remote repo, Origin(Master/Main)
• Git Commands – Git add, git commit, git push, git fetch, git merge, git pull, git status
• Git branch, git fork, git config, .gitignore, git remote, git checkout

SQL

Introduction to SQL

• Atomicity, Normalization, How to transform Logical into Physical Model, Data Models, Dimension Table and fact table, Schema, Data modeling approaches, Database Terminology, DDL, DML, Constrains, Data Integrity, Filtering Data, SQL Data Integrity, SQL Joins, Provided with a schema representation of a database which they have to code and make every relationship between table illustrated in the schema.
• How to extract aggregate data using predefined function, how to write their own custom function to automate some of the actions necessary when selecting, deleting, updating, adding to a table, the importance of transaction and how it is used.

Overview to Database Security

SQL with Postgres

Intro to PostgreSQL

• Installation, Schema, Transactions
• Constrains, Data Integrity, Filtering Data, Managing Tables Import & Export Data
• PostgreSQL Administration
• Roles & Privileges
• Backup & Restore Databases

Intro to Data Warehousing + ETL

Data Warehousing and Data Mart

• Data Mart concepts and Architecture, Data lake and the Data Mart types and Data pipeline
• Data Warehouse from Operational Systems, Define the Operational Data Store and Analytical Store
• Data Warehouse using Inmons’s Approach and Kimball’s Approach, Data vault modeling
• Data warehousing principles and architectures to enable business intelligence and the ETL solution (Extraction, Transportation-Transformation, and loading) and the differences between transaction and analysis workloads. Also they will understand the limitation scope (functional area) of a Data Mart.

Intro to Big Data + ETL

• Big data characteristics, 7 Vs of Big Data (Volume, Velocity, Variety, Veracity, Variability, Value.),- Big Data Life Cycle (Ingesting, Persisting, Computing and Analyzing, and Visualizing).
• Clustered Computing (Resource Pooling, High Availability, Easy Scalability
• How to recognize Structured (Defined Data) Vs Unstructured (Something else) with: Characteristics, Location, Generation, Typical Applications and Samples.
• ETL Process

Data Warehouse

• Database vs Data Warehouse, Data Warehouse Architecture, Data Warehouse Project Lifecycle, Key Characteristics of a Data Warehouse, Data Warehouse Development, Inmons’s Approach and Kimball’s Approach and Data lake

Data Mart Styles

• Why use Data Mart?,Designing Data Marts, Operational Data Stores, OLTP and OLAP, OLTP vs OLAP

Python and Database integration with python

Python & Database Integration

• Install Python IDE
• Types in Python, modules, data type, variables, conditionals, operators and operator precedence, loops, etc
• Functions, passing function arguments
• Create a class, construct an object, and knowledge on class methods, object data
• Raise, throw and exception and handle exceptions appropriately.
• Module and create module
• The set, tuple, array, list, dictionary
• Have good understanding of different data structure and operations on those data structure and list comprehensions
• Have ability to use various string methods and file handling – FILE I/O methods built in Python –
• Have deeper understanding of Lambda function
• Have ability to use statement, indentation and comment in Python
• Have ability to use namespaces and scope in Python
• Lambda functions
• Iterators
• Closures
• Decorators
• Processing JSON Data using Python for data pipeline
• Processing XLS Data using Python for data pipeline
• Processing Relational Databases using Python for Data Pipeline
• To write Python program to interact with database and perform basic database operation.
• Data wrangling using python for Data Pipeline

Data Analytics with Python

Intro to Statistics and Measure of Variability

• Introduction to data analysis and visualization, Conceptualizing Data Analysis as a Process, Types of Data Analysis and Chart Types
• Jupyter notebook and Anaconda

Data Analysis

• NumPy – Ndarray Object, numpy.arange, Array Slicing, Joining Array and etc
• Pandas – Data Structure, Dataframe slicing, Selecting, Extracting, Imputation
• Convert a dictionary to Pandas Dataframe
• Reading data from CSVs ,Json, using Pandas
• Python -Data aggregation using Pandas and Numpy libraries.
• Add a new columns to existing Dataframe in Pandas

Data Visualization

• Matplotlib – Pyplot and Plotting and Cover at least 5 to 6 Charts
• Processing Structure and Unstructured data
• Measuring central tendency
• To apply industry best coding practices & tools

Overview to Spark and Spark SQL

Apache Spark

• Fundamentals of Apache Spark, Role of Driver in Spark Architecture
• Set up a full environment for developing Spark applications, Spark Shell and PySpark
• Fundamentals of RDD, Loading data into RDD, Transformations, and Actions
• Fundamentals of DataFrame and Spark SQL
• Spark SQL performance optimization, Libraries of Spark SQL
• Manipulating the Data with Spark SQL, Loading data into DataFrame, Basic operation with PySpark

Tableau

What is Data Analytics, Types of Data Analysis, Chart Types:

• Learn to create an executive-level data analysis through visual analytics dashboards using charts, dates, table calculations, mapping, groups, sets, animations, and cover only available features in trial version

Overview of Tableau

• Connecting to data from difference sources
• Organizing your data
• Simplifying, grouping and sorting data
• Slicing your data by date
• Using multiple measures in a view
• Showing the relationship between numerical values
• Mapping data geographicallyViewing specific values
• Making your views available
• Data filtering
• Data Visualization
• Chart ( Bar Chart, Line Chart, Pie Chart scatter plot ,Highlighting, Bullet Chart, gantt chart, heat map, )
• Graphics
• Overview of Dashboard

Intro to DevOps

Intro to DevOps

• DevOps lifecycle
• DevOps Work flow
• Who is a DevOps Engineer?
• Overview to CI and CD, What are the Popular tools for DevOps?

Container and Dockers

• Intro to Containers and Virtualization
• Container vs Virtual Machine (VM)
• Introduction to Docker:? What is Docker?
• Docker Architecture: Client, Containers, Images, Registry, Daemon, Host
• Docker for Deployment and Set up your Docker Environment
• Docker operations: start, stop, commit, rm, rmi, ps, images, exec
• Vocab: Images, Containers, Docker Daemon, Docker Client, Docker Hub, Nginx, Dockerfile