sr data engineer technical lead resume example

Project: Metazoo-Metadata-driven data ingestion and ETL (OCT 2022- MAR 2023)

Client: Advent Health

Role: Cloud Data Engineer

Description: This project is migrating different on-prem data sources (Oracle, MySQL, Salesforce, etc.) to azure cloud/snowflake. Building automated metadata-driven framework and pipelines using azure data factory, creating a datalake in ADLS, and loading data to Snowflake for further reporting and analytics.

Environment: Azure, Salesforce, SQL server, snowflake, python, Azure data factory, Azure DevOps

Key Responsibilities:

• Built Metazoo automation framework for salesforce metadata generation.
• Automated source/salesforce schema extraction, schema processing, and job generation using python-based framework which can map salesforce data to snowflake.
• Built parameterized ADF pipelines from extracted metadata as input parameters and ingested data into Azure data lake storage.
• Ingested extracted parquet data into snowflake tables and created views on the top of it for further analysis.
• Implemented different load strategies full/initial load, incremental load, and Type2 while loading data into snowflake.
• Replicated on-prem nifi data pipeline in the cloud using azure data factory.
• Test end-to-end ADF data pipeline and data validation for ingested data.
• Document end-to-end process, and performance analysis on confluence.

Project : BDP(Business Data intelligent platform) (FEB 2022- OCT 2022)

Client : MyFitnessPal

Role : Sr. Data Engineer

Project Description : MyFitnessPal is one of the best weight loss apps and fitness apps, helping nearly 1 million members reach their nutrition and fitness goals every year. This project is migrating their application data to snowflake data warehouse for their BI needs as well as implementing ETL & Data Warehousing using Snowflake and orchestrates & automates complete end-to-end flow using Airflow jobs.

Environment : Aws s3, AWS Managed Apache Airflow, Snowflake

Key Responsibilities:

• Create and manage data pipelines using MWAA airflow dags to load data from aws s3 to snowflake.
• Created ETL jobs using snowflake to copy raw data into the landing schema of snowflake.
• Implemented delta/incremental load with type 2, overwrite and append load strategies from landing/raw layer to staging layer.
• Transformed and performed data curation & cleansing on raw variant data into suitable structured format using snowflake scripts.
• Used snowflake streams to identify inserts, updates and deletes operations on raw data.
• Created parameterized dags for different environments (PROD, DEV & QA) to orchestrate and schedule complete end-to-end ETL process.
• Developed a custom log snowflake operator in Airflow for logging, debugging and auditing of Airflow jobs.
• Worked closely with different stakeholders, BA, solution architect, QA as well as BI team to achieve project goals and meet project timelines.
• Worked on process flow, lineage and different Sop documentation.

Project : Batch - Ingestion & On prem to Cloud data migration ( MAR 2021- JAN 2022)

Client : HSBC

Role : Sr. Data Engineer

Project Description : This project is data migration from on-prem to google cloud and implementing data ingestion strategies from GCP storage to big query using Airflow as orchestration tool

Environment : Python, GCP, GCS, Google Big query, Airflow, Juniper

Key Responsibilities:

• Migration of source files with different file formats (.csv, .cobol, fixed width,.avro) from on prem servers to Google cloud storage using Juniper data migration tool.
• Created juniper feeds for transferring files from on-prem virtual machine to gcs buckets.
• Developed parameterized python scripts to perform data conversion, audit process, reconciliation of data before loading it into a big query table.
• Wrote Cobol parser in python to read fixed width files and to load into target big query tables.
• Replaced existing Control-M orchestration to Airflow.
• Created Airflow dags to orchestrate complete end-to-end ingestion process and scheduling.
• Performed data validations and unit testing using python.
• created interdependent dags in Airflow using triggerdagrun operator and task sensors in airflow.
• Created SOP documents for complete end-to-end ingestion process using confluence.

Client: Exeliq Consulting Inc. / Trustmark

Project: Cloud Governance

Description: The “Cloud Governance” tool streamlines the overall governance of the client-side cloud environment

after migrating to the cloud. The tool ensures that the cloud environment ensures ease of compliance, enhanced security, optimum utilization of resources,cost optimization and standardization of

Processes for seamless scaling of the environment.

Environment: Azure platform, Python, Azure Databricks, Azure AD, Azure Storage, Python, GitHub

Github Link: https://github.com/hitesh09p/TSPL/tree/master/cloudgovernance-master

Key Responsibilities:

• Built and setup End-to-end Cloud governance framework for Client cloud environment
• customized cloud governance as per the client needs
• Created python framework with specific local and global industry compliance standards.
• Optimized workloads and resource allocations for Significant cost optimization.
• Studied and tested insightful reports and recommendations for a continuous cloud cost & resource optimization process.
• Automated centralized cloud monitoring which enabled Audit, Security, and Compliance of the cloud platform.
• Created and managed role-based access control for enhanced security compliance, granular level security and policy management using python framework.
• Built cloud resource and cost monitoring customized dashboards using Tableau.