Piramal - Data Engineer/Senior Data Engineer

2 - 10 Years

Vadodara/Baroda

Posted 3 years ago

#Machine Learning #Data Architect #SQL #IT Product Development #Artificial Intelligence #Technical Architect

Industry - IT

Category - IT & Systems

Skills - Data Engineer, SPARK, AZURE, KAFKA

Job Type - Permanent

Job Description - Purpose of the Job

- The Data Engineer extracts data and implements machine learning models into production by utilizing state of the art tools/algorithms and methodologies following DevOps and a test-driven development process. Data Engineering is the key role for bridging the gap between insight and actions. Data Engineers work in close collaborations with the data scientists and guide them to focus not only on model performance but also delivery stability, reproducibility and scalability of a software product.

- The Data Engineer believes in a non-hierarchical culture of collaboration, transparency, safety, and trust. Working with a focus on value creation, growth and serving customers with full ownership and accountability. Delivering exceptional customer and business results

Responsibilities

- Collaborate with Data Scientists to test and scale new algorithms through pilots and later industrialize the solutions at scale to the comprehensive fashion network of the Group

- Influence, build and maintain the large-scale data infrastructure required for the AI projects, and integrate with external IT infrastructure/service to provide an e2e solution

- Leverage an understanding of software architecture and software design patterns to write scalable, maintainable, well-designed and future-proof code

- Design, develop and maintain the framework for analytical pipeline

- Develop common components to address pain points in machine learning project, like model lifecycle management, feature store and data quality evaluation

- Provide input and help implement framework and tools to improve data quality

- Work in cross-functional agile teams of highly skilled software/machine learning engineers, data scientists, designers, product managers and others to build the AI ecosystem within the Group

- Deliver on time, demonstrating strong commitment to deliver on the team mission and agreed backlog

Industry

Any (prefer - Manufacturing, Logistics); willingness to learn manufacturing systems (OT systems and data stores)

Education

Bachelors or masters in Computer Science or Engineering, BCA / MCA / B.Sc / M.Sc

Work experience

- 2+ years of industry exposure in Data Engineering projects

Skills & Competencies required

- Ability to transform proof of concept machine learning models into scalable solutions

- Well versed with data handling techniques - Data cleansing and feature scaling

- Ability to apply software development best practice into machine learning project, including unit test, DevOps integration, release management, test driven development, etc.

- Ability to automate development process of machine learning project by leverage state of art tools and technology like container, continuous integration and delivery, orchestration tools etc.

- Familiar with at least one of major cloud solutions, preferably Azure. Able to recommend and select right services available in cloud to address technical problem

- Good understanding and experience with software design principle and design pattern

- Good understanding of Agile and scrum methodology, keep team focus on deliver business value

Functional Knowledge

- DevOps and continuous integration

- Agile wow and test-driven development

- Distributed computing

- Linux

- Spark and Hadoop eco system

- Software development best practices

- Python/Scala/Java

- Docker, Kubernetes

- Experience with Azure

- Data structure and algorithm