← Back to Work Development
Enterprise Data Pipeline Architecture
Designed and delivered modern data infrastructure and ETL/ELT pipelines for multiple global clients at Accenture.
Overview
At Redkite (Accenture), I designed and delivered modern data infrastructure and pipeline architectures in cloud environments for multiple global clients. This work focused on creating scalable, maintainable data solutions using the Microsoft Azure tech stack.
Key Responsibilities
- ETL/ELT Pipeline Design: Implementation using Azure Data Factory, Synapse, and Databricks for both batch and streaming data workloads
- API Ingestion Frameworks: Built reusable frameworks for third-party and internal REST APIs, handling token refresh, pagination, dynamic schema handling, and fault tolerance
- PySpark Development: Authored clean, modular PySpark code on Azure, Databricks and Microsoft Fabric, improving performance and maintainability of data transformations on MPP systems
- Stakeholder Collaboration: Worked with client stakeholders and cross-functional teams to translate business requirements into scalable data models
Technologies Used
- Azure Data Factory
- Azure Synapse Analytics
- Databricks
- Microsoft Fabric
- PySpark / Apache Spark
- Python
- SQL
Impact
Successfully delivered data solutions for enterprise clients that enabled better decision-making through reliable, timely data access and robust data quality standards.