Hadoop Admin Resume Example

• Working on Hadoop Hortonworks (HDP 2.6.0.2.2) distribution which managed services viz. HDFS, MapReduce2, Hive, Pig, Hbase, Sqoop, Flume, Spark, Ambari Metrics, Zookeeper, Falcon and oozie etc. for 4 cluster ranges from LAB, DEV, QA to PROD. 
• Monitor Hadoop cluster connectivity and security on Ambari monitoring system. 
• Led the installation, configuration and deployment of product soft wares on new edge nodes that connect and contact Hadoop cluster for data acquisition.
• Responsible for Cluster maintenance, Monitoring, commissioning and decommissioning Data nodes, Troubleshooting, Manage and review data backups, Manage & review log files. 
• Day to day responsibilities includes solving developer issues, deployments moving code from one environment to other environment, providing access to new users and providing instant solutions to reduce the impact and documenting the same and preventing future issues. 
• Collaborating with application teams to install operating system and Hadoop updates, patches, version upgrades. 
• Involved in Analyzing system failures, identifying root causes, and recommended course of actions. 
• Interacting with HDP support and log the issues in portal and fixing them as per the recommendations.   
• Imported logs from web servers with Flume to ingest the data into HDFS. 
• Using Flume and Spool directory loading the data from local system to hdfs 
• Retrieved data from HDFS into relational databases with Sqoop. Parsed cleansed and mined useful and meaningful data in HDFS using Map-Reduce for further analysis   
• Fine tuning hive jobs for optimized performance.   
• Implemented custom interceptors for flume to filter data and defined channel selectors to multiplex the data into different sinks. 
• Partitioned and queried the data in Hive for further analysis by the BI team. 
• Extending the functionality of Hive and Pig with custom UDF s and UDAF's. 
• Involved in extracting the data from various sources into Hadoop HDFS for processing. 
• Worked on analyzing Hadoop cluster and different big data analytic tools including Pig, Hbase database and Sqoop. 
• Creating and deploying a corresponding Solr Cloud collection. 
• Creating collections and configurations, Register a Lily HBase Indexer configuration with the Lily HBase Indexer Service. 
• Configuring, Managing permissions for the users in hue. 
• Commissioned and Decommissioned nodes on Hadoop cluster on Red hat LINUX. 
• Involved in loading data from LINUX file system to HDFS. 
• Creating and managing the Cron jobs. 
• Worked on tuning the performance Pig queries. 
• Worked with application teams to install operating system, Hadoop updates, patches, version upgrades as required. 
• Experience in configuring the Storm in loading the data from MYSQL to HBASE using jms 
• Responsible to manage data coming from different sources.   
• Involved in loading data from UNIX file system to HDFS. 
• Integrated Kerberos into Hadoop to make cluster more strong and secure from unauthorized users 
• Experience in managing and reviewing Hadoop log files. 
• Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the BI team. 
• Installed Oozie workflow engine to run multiple Hive and pig jobs. 
• Analyzed large amounts of data sets to determine optimal way to aggregate and report on it. 
• Supported in setting up QA environment and updating configurations for implementing scripts with Pig and Sqoop. 

ENVIRONMENT: Hadoop HDFS, MapReduce, HIVE, PIG, FLUME, OOZIE, Sqoop, Eclipse, Hortonworks, Ambari, Redhat, MYSQL

• Hands on Installation and configuration of Hortonworks Data Platform HDP 2.3.4.
• Worked on installing production cluster, commissioning & decommissioning of Data Nodes, Name Node recovery, capacity planning, and slots configuration
• Worked on Hadoop Administration, responsibilities include software installation, configuration, software upgrades, backup and recovery, cluster setup, cluster performance and monitoring on daily basis, maintaining cluster up and run on healthy.
• Implemented the security requirements for Hadoop and integrate with Kerberos authentication and authorization infrastructure.
• Designed, developed and implemented connectivity products that allow efficient exchange of data between the core database engine and the Hadoop ecosystem.
• Involved in defining job flows using Oozie for scheduling jobs to manage apache Hadoop jobs.
• Implemented Name Node High Availability on the Hadoop cluster to overcome single point of failure.  
• Worked on YARN capacity scheduler by creating queues to allocate resource guarantee to specific groups.  
• Worked on importing and exporting data from Oracle database into HDFS and HIVE using Sqoop.  
• Monitored and analyzed of the Map Reduce job executions on cluster at task level.  
• Extensively involved in Cluster Capacity planning, Hardware planning, Performance Tuning of theHadoop Cluster.  
• Wrote automation scripts and setting up crontab jobs to maintain cluster stability and healthy.
• Installed Ambari on an already existing Hadoop cluster.  
• Implemented Rack Awareness for data locality optimization.  
• Optimized and tuned the Hadoop environments to meet performance requirements.  
• Hand-On experience with AWS cloud with EC2, S3.  
• Collaborating with offshore team.  
• Ability to document existing processes and recommend improvements.
• Shares knowledge and assists another team member as needed.  
• Assist with maintenance and troubleshooting of scheduled processes.
• Participated in development of system test plans and acceptance criteria.
• Collaborate with offshore developers in order to monitor ETL jobs and troubleshoot steps. 

Environment: Hortonworks HDP2.3x, Ambari, Oozie 4.2, Sqoop1.4.6, Mapreduce2, Ambari, Sql Developer, Teradata, SSH, Eclipse, Jdk 1.7, CDH 3.x, 4.X, 5.X, Cloudera Manager 4&5, Ganglia, Tableau, Shell Scripting, Oozie, Pig, Hive, Flume, Kafka, Impala, Oozie, CentOS

• Involved in extracting customer's big data from various data sources into Hadoop HDFS.
• This included data from click stream data (omniture), databases and also logs data from servers.
• Developed data pipeline using Pig and Hive from Oracle, Omniture data sources.
• These pipelines had customized UDF'S to extend the ETL functionality.
• Used Sqoop to efficiently transfer data between databases and HDFS Developed Map Reduce programs to cleanse the data in HDFS obtained from heterogeneous data sources to make it suitable for ingestion into Hive schema for analysis.
• The Hive tables created as per requirement were internal or external tables defined with appropriate static and dynamic partitions, intended for efficiency.
• Implemented partitioning, bucketing in Hive for better organization of the data.
• Developed UDFs in Pig and Hive Used Oozie workflow engine to manage interdependent Hadoop jobs and to automate several types of Hadoop jobs such as Java map-reduce, hive and Sqoop as well as system specific jobs.
• Involved in End-to-End implementation of ETL logic.
• Worked with BI teams in generating the reports on Tableau
• Installed and configured various components of Cloudera Hadoop ecosystem
• Involved in Performance tuning of Hadoop clusters and Hadoop Map Reduce Programs
• Upgraded Hadoop Versions using Cloudera Manager in Dev and QA clusters.

Environment: JDK1.7, Redhat Linux, HDFS, Mahout, Map-Reduce, Hive, Pig, Sqoop, Flume, Zookeeper, Oozie, DB2, Hbase and Pentaho.

• Designed Object Model using UML.
• Developed use cases using UML according to business requirements.
• Developed project using Agile methodology and used continuous integration as for submitting changes.
• Developed the presentation components like Order management and Profile management in JSP as part of Spring framework implementing the MVC design pattern.
• Used JavaScript for creating the form Components in the UI Screens and validations.
• Implemented the DAO pattern for populating the Customer information in Customer DB and the billing details in Bills DB and developed the persistence layer using Hibernate.
• Designed and implemented Servlets, JSP, Struts for integration with provisioning and billing systems.
• Responsible for designing the front-end GUI using JSP, HTML and JavaScript for validation.

Environment: Windows 2000, UNIX, HTML, XML, Java 1.4, J2EE, hibernate 3.0, spring 2.0, Java Beans, JSP, UML, JDBC, Oracle 10g, SQL.