Over 5 years of IT experience as a Developer, Designer & QA Test Engineer with cross-platform integration experience using Hadoop Ecosystem, Java and Software Functional Testing. Hands on experience in installing, configuring and using Hadoop Ecosystem - HDFS, MapReduce, Pig, Hive, Oozie, Flume, HBase, Spark, Sqoop, Flume and Oozie. Strong understanding of various Hadoop services, MapReduce and YARN architecture. Responsible for writing Map Reduce programs. Experienced in importing-exporting data into HDFS using SQOOP Experience loading data to Hive partitions and creating buckets in Hive. Developed Map Reduce jobs to automate transfer the data from HBase. Expertise in analysis using PIG, HIVE and MapReduce. Experienced in developing UDFs for Hive, PIG using Java. Strong understanding of NoSQL databases like HBase, MongoDB & Cassandra. Scheduling all Hadoop/hive/Sqoop/HBase jobs using Oozie Experience in setting cluster in Amazon EC2 & S3 including the automation of setting & extending the clusters in AWS Amazon cloud. Good understanding of Scrum methodologies, Test Driven Development and continuous integration. Major strengths are familiarity with multiple software systems, ability to learn quickly new technologies, adapt to new environments, self-motivated, team player, focused adaptive and quick learner with excellent interpersonal, technical and communication skills. Experience in defining detailed application software test plans, including organization, participant, schedule, test and application coverage scope. Experience in gathering and defining functional and user interface requirements for software applications. Experience in real time analytics with Apache Spark (RDD, Data Frames and Streaming API). Used Spark Data Frames API over Cloudera platform to perform analytics on Hive data. Experience in integrating Hadoop with Kafka. Expertise in uploading Click stream data from Kafka to HDFS. Expert in utilizing Kafka for messaging and publishing subscribe messaging system
Tools and IDE : Anaconda, PyCharm, Jupiter Eclipse, IntelliJ
Work History
10/2019 to Current
Hadoop DeveloperCognizant Technology Solutions – Berkeley, CA,
Installed and configured Hive in Hadoop cluster and help business users/application teams fine tune their HIVE QL for optimizing performance and efficient use of resources in cluster.
Conduct performance tuning of Hadoop Cluster and map reduce jobs. Also, real-time applications with best practices to fix design flaws.
Implemented Oozie workflow for ETL Process for critical data feeds across platform.
Configured Ethernet bonding for all Nodes to double network bandwidth.
Implementing Kerberos Security Authentication protocol for existing cluster.
Built high availability for major production cluster and designed automatic failover control using Zookeeper Failover Controller (ZKFC) and Quorum Journal nodes.
Worked on analyzing Hadoop clusters and different big data analytic tools including Sqoop, Pig, and Hive.
Created POC on Hortonworks and suggested best practice in terms HDP, HDF platform
Experience in understanding security requirements for Hadoop and integrating with Kerberos authentication infrastructure- KDC server setup.
Management and support of Hadoop Services including HDFS, Impala, and SPARK.
Installing, Upgrading and Managing Hadoop Cluster on Cloudera
Worked on Sequence files, RC files, Map side joins, bucketing, partitioning for Hive performance enhancement and storage improvement.
Sqoop jobs and Hive queries were created for data ingestion from relational databases to analyze historical data.
Involved in Hive/SQL queries performing spark transformations using Spark RDDs and Python.
Created Serverless data ingestion pipeline on AWS using lambda functions.
Configured Spark Streaming to receive real time data from Apache Kafka and store stream data to DynamoDB using Scala.
Developed Apache Applications by using Scala, Python and Implemented Apache Spark data processing module to handle data from various RDBMS and Streaming sources.
Experience in developing and scheduling various Spark Streaming / batch Jobs using python and Scala.
Achieved high-throughput, scalable, fault-tolerant stream processing of live data streams using Apache Spark Streaming
Involved in using various Python libraries with spark in order to create data frames and store them to Hive.
Experience in working with Elastic MapReduce (EMR) and setting up environments on amazon AWS EC2 instances.
Knowledge on handling Hive queries using SQL that integrates with Spark environment.
Executed Hadoop jobs on AWS EMR using programs, stored in S3 Buckets.
Knowledge on creating user defined functions (UDF's) in Hive.
Worked with different File Formats like c, Avro, parquet for HIVE querying and processing based on business logic.
Implemented Hive UDF's to implement business logic and Responsible for performing extensive data validation using Hive.
Involved in developing code and generated various data frames based on business requirement and created temporary tables in hive.
Utilized AWS CloudWatch to monitor performance environment instances for operational and performance metrics during load testing. Scripting Hadoop package installation and configuration to support fully automated deployments.
Involved in chef-infra maintenance including backup/security fix on Chef Server.
Deployed application updates using Jenkins. Installed, configured, and managed Jenkins
Triggering SIT environment build of client remotely through Jenkins.
Deployed and configured Git repositories with branching, forks, tagging, and notifications.
Experienced and proficient deploying and administering GitHub
Deploy builds to production and work with teams to identify and troubleshoot any issues.
Worked on MongoDB database concepts such as locking, transactions, indexes, Shading, replication, schema design.
Consulted with operations team on deploying, migrating data, monitoring, analyzing, and tuning MongoDB applications.
Viewing selected issues of web interface using SonarQube.
Developed fully functional login page for company's user facing website with complete UI and validations.
Installed, Configured and utilized AppDynamics (Tremendous Performance Management Tool) in whole JBoss Environment (Prod and Non-Prod).
Responsible for upgradation of SonarQube using upgrade center.
Worked on analyzing Hadoop cluster and different big data analytic tools including Pig, Hive and Sqoop
Created POC on Hortonworks and suggested best practice in terms HDP, HDF platform
Experience in understanding security requirements for Hadoop and integrating with Kerberos authentication infrastructure- KDC server setup, managing
Management and support of Hadoop Services including HDFS, Hive, Impala, and SPARK
Installing, Upgrading and Managing Hadoop Cluster on Cloudera
Troubleshooting many clouds related issues such as Data Node down, Network failure, login issues and data block missing
Worked as Hadoop Admin and responsible for taking care of everything related to clusters total of 100 nodes ranges from POC (Proof-of-Concept) to PROD clusters on Cloudera (CDH 5.5.2) distribution
Responsible for Cluster maintenance, Monitoring, commissioning and decommissioning Data nodes, Troubleshooting, Manage and review data backups, Manage & review log files
Day to day responsibilities includes solving developer issues, deployments moving code from one environment to other environment, providing access to new users and providing instant solutions to reduce impact and documenting same and preventing future issues
Collaborating with application teams to install operating system and Hadoop updates, patches, version upgrades
Strong experience and knowledge of real time data analytics using Spark Streaming, Kafka and Flume
Migrated Flume with Spark for real time data and developed Spark Streaming Application with java to consume data from Kafka and push them into Hive
Configured Kafka for efficiently collecting, aggregating and moving large amounts of click stream data from many different sources to HDFS
Monitored workload, job performance and capacity planning using Cloudera Manager
Involved in Analyzing system failures, identifying root causes, and recommended course of actions
Interacting with Cloudera support and log issues in Cloudera portal and fixing them as per recommendations
Imported logs from web servers with Flume to ingest data into HDFS
Using Flume and Spool directory loading data from local system to HDFS
Retrieved data from HDFS into relational databases with Sqoop
Parsed cleansed and mined useful and meaningful data in HDFS using Map-Reduce for further analysis Fine tuning hive jobs for optimized performance
Scripting Hadoop package installation and configuration to support fully automated deployments
Involved in chef-infra maintenance including backup/security fix on Chef Server
Deployed application updates using Jenkins
Installed, configured, and managed Jenkins
Triggering SIT environment build of client remotely through Jenkins
Deployed and configured Git repositories with branching, forks, tagging, and notifications
Experienced and proficient deploying and administering GitHub
Deploy builds to production and work with teams to identify and troubleshoot any issues
Worked on MongoDB database concepts such as locking, transactions, indexes, Shading, replication, schema design
Consulted with operations team on deploying, migrating data, monitoring, analyzing, and tuning MongoDB applications
Viewing selected issues of web interface using SonarQube
Developed fully functional login page for company's user facing website with complete UI and validations
Installed, Configured and utilized AppDynamics (Tremendous Performance Management Tool) in whole JBoss Environment (Prod and Non-Prod)
Reviewed OpenShift PaaS product architecture and suggested improvement features after conducting research on Competitor’s products
Migrated data source passwords to encrypted passwords using Vault tool in all JBoss application servers
Participated in Migration undergoing from JBoss 4 to Web logic or JBoss 4 to JBoss 6 and its respective POC
Responsible for upgradation of SonarQube using upgrade center
Installed and configured Hive in Hadoop cluster and help business users/application teams fine tune their HIVE QL for optimizing performance and efficient use of resources in cluster
Conduct performance tuning of Hadoop Cluster and map reduce jobs
Also, real-time applications with best practices to fix design flaws
Implemented Oozie workflow for ETL Process for critical data feeds across platform
Configured Ethernet bonding for all Nodes to double network bandwidth
Implementing Kerberos Security Authentication protocol for existing cluster
Built high availability for major production cluster and designed automatic failover control using Zookeeper Failover Controller (ZKFC) and Quorum Journal nodes
Any information uploaded, such as a resume, or input by the user is owned solely by the user, not LiveCareer. For further information, please visit our Terms of Use.
Resume Overview
School Attended
College of Staten Island of The City University of New York