This role is no longer accepting applications via Rocketlist.
About the role
TL;DR
Design and optimize scalable data pipelines in Azure and Databricks.
- •We are looking for a Data Engineer to work on large-scale data solutions in Azure and Databricks.
- •Key Responsibilities Design, develop, and optimize scalable data pipelines.
- •Develop ETL/ELT processes from ingestion to data transformation and availability.
- •Work with Medallion architecture and Data Lakehouse solutions.
- •Develop solutions using PySpark, Spark SQL, SQL, and Python.
- •Integrate different sources and systems, including relational databases and APIs.
- •Requirements Solid experience as a Data Engineer in production environments.
- •Strong experience with Azure Databricks.
- •Proficiency in PySpark/Apache Spark, including processing and optimization of large volumes of data.
- •Advanced knowledge of SQL.
- •Experience with Python applied to Data Engineering.
- •Experience with Azure Data Factory (ADF) for pipeline construction and orchestration.
- •Knowledge of ETL/ELT and Medallion architecture (Bronze, Silver, and Gold).
- •Experience with Delta Lake and Data Lakehouse.
- •Experience with Git and versioning flows, preferably using Bitbucket.
- •Knowledge of CI/CD, Azure Pipelines, Azure DevOps, or similar tools.
- •Experience working with relational databases, such as SQL Server and/or Oracle.
Required skills
SparkSQLPythonGitCI/CDSQL ServerOracle
Tech stack
SparkSQLPythonGitCI/CDSQL ServerOracle