Big Data Hadoop Developer - Appodis

Be a part of our engineering team that is responsible for creating cutting-edge products and experiences by applying your expertise in software development to real-world situations.

Capture requirements, Sprint Planning & Execution: 10%

- Capture business and application requirements, brainstorm requirements.
- Document functional requirements and operations (calculations, technical details, data manipulation and processing etc.).
- Sprint-planning:
- Participate in meeting held on first day of each sprint involving Scrum Master, Product Owner, and Team, where Product Owner presents the set of features to be achieved in the sprint.
- Work estimates are reviewed to see the team has sufficient time to complete all the requested features in the sprint and team commits to the sprint.
- The lower priority features added back into the product backlog, until the workload for the sprint is enough to obtain the team’s commitment.

Analysis, Design, Maintenance, Performance Tuning, Documentation and Testing 10%

- Create design documents for defining process, Quality and Information analyzer jobs
- Analysis and study of existing systems.
- Identify low performance jobs and rewrite/redesign to improve performance.
- Identify the underperforming queries and rewrite.
- Create appropriate indexes for the relevant columns of the tables in database.
- Documentation of test cases and workflow instructions
- Unit test of ADF Jobs for extraction, transformation and loading of client data.

Technical Design Documentation and Data Migration 10%

- Create low level design documents for Parallel jobs, batch jobs and Sequence jobs
- Handle database server backups appropriately
- Export / Import wizard to move the data between different servers – Development, QA, and Production.
- Execute parallel jobs, batch jobs or sequencers for data migration.
- Extract client data available in Azure Blob storage and different sources like Flat files, MS SQL server and Oracle legacy data systems and map the data into target business warehouse.
- Monitor errors and warnings using Azure event manager and MS SQL Server job agents
- Track Slowly Changing Dimensions (SCD) using Surrogate Keys.
- Implementation of Naming Conventions, Parameterizing the variables from Global Environment to stage level.
- Extraction and transformation of data using Parallel Extender for parallel processing.
- Work with various stages of Parallel Extender like Sequential file, Dataset, Lookup, Transformer, Merge, Aggregator, Row generator, Surrogate key generator and many more to design jobs and load the data in to Fact and Dimension tables.
- Design and create User-Defined SQL queries for extracting the data from source systems.
- Work on Error Handling in Azure Data Factory (ADF)
- Creating data extraction packages using Azure Functions and Azure Data Bricks (ADB)
- Implement data quality and control using Azure Key vaults (AKV)and Azure LogicApps
- Develop automated and scheduled load processes using PowerShell scripting.
- Visualizing data for management using reporting tool Microsoft PowerBI
- Monitoring subscriptions, gateways and row level security using PowerBI service

Organize, Manage Development 10%

- Code deployment to Development, QA and Production environments.
- Share the functional requirements document and work on test plan development.
- Create unit test cases for different application functions and SQL procedures.
- Provide inputs on the test data preparation phase
- Review of Test cases with the client team and provide a sign-off
- Perform integration and smoke test in development before deploying application to QA.
- Support QA team with testing till code is deployed to production.

Onsite-Offshore coordination 60%

- Knowledge transfer from the client to the team – Obtain the necessary technical knowledge and share, train the team.
- Communicate timelines, progress, delays of the work assigned on daily basis from the team to the client and vice versa.
- Conduct review meetings with the concerned development and support teams on daily assignments, management of Development and testing tasks.

Deployment and maintenance of the product: 20%

Work with the systems engineering team to plan and deploy new Hadoop environments and expand.
Work in AWS environment to develop and deploy Custom Hadoop Applications.
Design and deploy Hadoop cluster and different Big Data analytic tools.
Develop Spark code using Scala and Spark-SQL/Streaming for faster testing and processing of data. Configure deployed and maintained multi-node Dev and Test Kafka Clusters.
Develop multiple POCs using PySpark and deploy them on the Yarn cluster, and compare the performance of Spark, with Hive and SQL/Teradata.

Qualifications

Position Qualifications:

Minimum Bachelor’s degree in computer science or closely related field.
Experience identifying technical solutions for business problems – identifying the benefits and risks of the solutions and providing recommendations
Strong experience with latest technologies as mentioned above

We offer a competitive total rewards package including base salary determined based on the role, experience, skill set, and location. For those in eligible roles, discretionary incentive compensation which may be awarded in recognition of individual achievements and contributions. We also offer a range of benefits and programs to meet employee needs, based on eligibility. These benefits include comprehensive health care coverage, on-site health and wellness centers, a retirement savings plan, backup childcare, tuition reimbursement, mental health support, financial coaching and more. Additional details about total compensation and benefits will be provided during the hiring process.