Senior Data Engineer – AI & Analytics

We are looking for a Senior AI Data Analytics Engineer with strong Data Engineering expertise, analytical thinking, and AI enablement capabilities to build scalable data solutions that power analytics, dashboards, recommendations, and AI-driven use cases.

The role involves designing and evolving data products within a modern Azure + Databricks Lakehouse architecture, enabling business insights and AI solutions through curated, consumption-ready datasets. The ideal candidate will own the end-to-end data lifecycle and work closely with business, product, and engineering teams to deliver scalable and maintainable solutions.

Qualification

• BE / B.Tech / MCA / ME / M.Tech
• 8+ years of experience in Data Engineering / Analytics Engineering

Key Responsibilities

• Lead data quality and governance initiatives, including root-cause analysis, remediation of data inconsistencies, reliability improvements, and exploratory data analysis (EDA)
• Collaborate with business and analytics stakeholders to define ownership, standardized definitions, and calculation methodologies for KPIs and core business metrics
• Establish consistent sources of truth across systems and ensure the accuracy, consistency, and trustworthiness of datasets, metrics, and analytics outputs

• Design, develop, and optimize scalable data pipelines using Databricks and Azure data services
• Integrate internal and external data sources and build reusable, modular data components
• Develop curated datasets and data products for analytics, dashboards, recommendations, and AI applications
• Design batch and streaming solutions; optimize Spark workloads, Delta tables, and low-latency data processing
• Prepare and structure datasets for AI Agents / GenAI and support Azure AI Foundry integration patterns
• Implement AI-driven workflows to automate data analysis, reporting, insight generation, and identification of business risks, inefficiencies, and performance gaps
• Design semantic and metadata-driven datasets and enable downstream AI and BI consumption
• Ensure dashboards, reports, insights, and recommendations are actionable, aligned with business priorities, and support measurable outcomes
• Support Qlik / BI performance optimization and consistent consumption of governed business metrics
• Implement testing, CI/CD, version control, and engineering best practices using Azure DevOps
• Participate in agile delivery including planning, estimation, releases, and cross-functional collaboration

Required Skills

Data Engineering & Platform

• Spark 3.x (DataFrames, SQL, Batch & Structured Streaming)
• Databricks (Workflows, SQL Warehouses, DLT, Unity Catalog, Auto Loader, Pipelines)
• Azure Data Services and Lakehouse / Medallion Architecture
• Parquet / Delta, partitioning, compaction, and performance optimization

Programming & Analytics

• Strong Python and SQL (Spark SQL, TSQL, HiveQL)
• Data quality, EDA, KPI-driven analytical modeling
• Understanding of statistical concepts and data readiness for analytics/recommendation use cases
• Experience building reusable, analytics-ready, and AI-ready datasets

Enterprise Data Governance & Metric Management

• Establishing and enforcing enterprise data governance, data quality standards, and consistent sources of truth across systems
• Defining ownership, standardized definitions, and calculation methodologies for core business metrics
• Leading data quality initiatives to identify and remediate inconsistencies across data sources and pipelines, ensuring accurate, consistent, and trusted analytics outputs

Business Partnership & Outcome Accountability

• Act as the bridge between the engineering team and the analytics/product consumers.
• Partnering with business and analytics stakeholders to align data, reporting, dashboards, and AI solutions with business priorities and measurable outcomes
• Using AI-driven workflows to automate data analysis, reporting, insight generation, and identification of risks, inefficiencies, and performance gaps
• Translating analytical findings into actionable, data-driven recommendations and risk mitigation strategies

AI, BI & Delivery

• Azure AI Foundry integration and AI/Agent data preparation
• Experience supporting Qlik / Power BI / Tableau workloads
• Testing frameworks (pytest, Great Expectations, Acceptance Testing)
• CI/CD with Azure DevOps and YAML pipelines
• Agile/Scrum development practices

Good to Know

• ADLS, Managed Identity, Azure AI Foundry
• Feature engineering concepts
• Airflow / ADF / Synapse Pipelines
• Scala or Java
• Data Catalogs (Purview, Unity Catalog, Apache Atlas)
• Healthcare domain experience (preferred)