JOB DESCRIPTION

Data Lead

Job Title: Data Lead

Location: Gurugram, India

Experience: 15 - 20 years

Function: Data Engineering, Data Architecture & Analytics

People Manager:

Yes - team leadership and coaching responsibilities

Purpose of the Role:

The Data Lead will own the design, development, governance and delivery of the organisation's data platform and data engineering capabilities. The role will lead data architecture and engineering on AWS and Databricks, ensuring that data is ingested, transformed, secured and made available for reporting and business use in a reliable, scalable and cost-effective manner. The Data Lead will work closely with business stakeholders, BI and technology teams to translate requirements into technical solutions, establish engineering standards, and take end-to-end ownership of delivery.

Key Responsibilities:

1. Data Architecture & Modelling:

- Define and maintain the data platform architecture, including lakehouse/data lake design and integration patterns.

- Design and implement Medallion Architecture (Bronze, Silver and Gold) and appropriate data layering and naming standards.

- Develop scalable data models, including star schemas, facts and dimensions, using established modelling practices such as Kimball.

- Document architecture, data flows and technical decisions using tools such as draw.io or Lucidchart.

2. Data Engineering & Platform Delivery:

- Lead the design and implementation of batch and incremental data pipelines across APIs, files and databases.

- Oversee data ingestion, transformation, cleansing, normalisation, scheduling and orchestration.

- Guide hands-on engineering using Databricks, Apache Spark/PySpark, SQL and Python.

- Use Databricks Workflows, Delta Live Tables and Auto Loader where appropriate; troubleshoot and tune pipelines for reliability and performance.

- Own end-to-end delivery from requirements and solution design through development, testing, deployment and production support.

3. AWS Cloud Platform:

- Design and manage data storage, access and security configurations on AWS.

- Use AWS S3, IAM, Glue, CloudWatch and Secrets Manager to support secure, observable and maintainable data services.

- Monitor platform performance, availability and cost, and identify opportunities for optimisation.

- Apply infrastructure automation practices where appropriate; Lambda and Terraform experience would be beneficial.

4. Data Governance, Quality & Security:

- Establish data cataloguing, data dictionary, ownership, stewardship and lineage practices, using Databricks Unity Catalog.

- Define and implement data quality checks, validation rules, exception handling and alerting to identify data issues early.

- Implement appropriate access controls, PII masking, and row- and column-level security.

- Maintain awareness of relevant privacy, regulatory and conduct requirements, including GDPR, FCA and Consumer Duty, where applicable.

5. BI, Reporting & Business Partnership:

- Work with BI and reporting teams to define trusted reporting data models and support effective dashboard design.

- Enable business users and analysts to access consistent, governed and well-documented data through Databricks SQL and related capabilities.

- Engage stakeholders to understand business needs, clarify requirements, explain technical options and translate priorities into delivery plans.

6. Engineering Standards, DevOps & Delivery:

- Establish standards for source control, code reviews, testing, release management and automated deployments.

- Support CI/CD and Databricks Asset Bundles, using Git/GitHub and other agreed delivery tooling.

- Promote Agile ways of working, maintain delivery visibility and manage priorities, dependencies, risks and issues.

- Maintain appropriate technical documentation and work tracking using tools such as Jira and Confluence.

7. People Leadership & Stakeholder Management:

- Lead, coach and develop data engineers and other team members, encouraging knowledge sharing and consistent engineering practices.

- Plan team capacity, prioritise work and align delivery commitments with business needs.

- Collaborate with business, technology, data and analytics stakeholders to resolve issues and support informed decision-making.

- Take accountability for delivery outcomes, quality, timelines and continuous improvement.

Critical Skills Required:

- Data architecture: lakehouse/data lake design, Medallion Architecture, data integration and dimensional modelling.

- Data engineering: Databricks, Apache Spark/PySpark, SQL, Python, batch and incremental processing, orchestration and performance tuning.

- Databricks capabilities: Delta Lake, Databricks Workflows, Delta Live Tables, Auto Loader and Unity Catalog.

- AWS data services: S3, IAM, Glue, CloudWatch and Secrets Manager.

- Data governance and quality: catalogues, dictionaries, lineage, ownership, validation, monitoring and access controls.

- Security and privacy: PII masking, row/column-level security and awareness of GDPR, FCA and Consumer Duty requirements where applicable.

- BI and analytics enablement: reporting data models, dashboard best practices and Databricks SQL.

- DevOps: Git/GitHub, code reviews, CI/CD, automated deployment and Databricks Asset Bundles.

- Leadership: team management, stakeholder engagement, requirements analysis, planning, prioritisation and end-to-end delivery ownership.

Experience & Knowledge Required:

- 15 - 20 years of relevant experience across data engineering, data architecture, data platforms or related technology functions.

- Demonstrated experience designing and delivering enterprise data platforms and production-grade data pipelines.

- Experience working with Databricks and AWS data services in a cloud-based environment.

- Experience setting technical standards and overseeing data quality, governance, security and platform performance.

- Experience leading teams and delivering complex data initiatives from requirement definition to production.

- Ability to communicate technical concepts clearly and work effectively with business and technology stakeholders.

- Experience with Agile delivery practices, source control and CI/CD processes.

Desirable Skills & Experience:

- Experience with AWS Lambda and Terraform.

- Experience implementing Databricks Asset Bundles and automated deployment patterns.

- Experience supporting regulated or privacy-sensitive data environments, including financial services, adviser or client data.

- Relevant cloud, data engineering or Databricks certifications.

Qualifications:

Bachelor's degree in Computer Science, Information Technology, Engineering or a related discipline, or equivalent practical experience. Relevant postgraduate qualifications or professional certifications are an advantage.

Key Leadership Behaviours:

- Strong ownership and accountability for end-to-end project delivery.

- Clear communication and effective stakeholder management.

- Structured problem-solving, sound technical judgement and attention to data quality.

- Ability to coach team members, build collaborative working relationships and manage competing priorities.

- A continuous-improvement mindset with a focus on reliability, security, scalability and cost efficiency.

Role Expectations:

The Data Lead is expected to follow organisational policies for information security, data protection and risk management; identify and escalate data or platform risks; and ensure that solutions are appropriately documented, tested, governed and maintained.