Sr. Data Engineer
Location: Powell, OH
Position type: Full time
Skills Required:
✓Design, develop, implement, and maintain scalable data ingestion, transformation, and processing applications using Python, PySpark, Spark SQL, SQL, Azure Databricks, Delta Lake, Azure Data Factory, and related technologies.
✓Develop reusable ETL/ELT frameworks, batch and incremental-processing workflows, and data integration solutions for structured and semi-structured data obtained from relational databases, REST APIs, files, and enterprise applications.
✓Develop data transformation and processing applications using SQL, PySpark, Apache Spark, and Delta Lake, including joins, aggregations, window functions, normalization, deduplication, conditional transformations, partitioning, and incremental MERGE processing.
✓Design and implement automated data-quality and validation processes to verify schemas, record counts, data types, null values, duplicates, referential integrity, business rules, source-to-target mappings, and data accuracy.
✓Design and optimize cloud-based data-processing applications by applying query optimization, partitioning, caching, repartitioning, checkpointing, file-format optimization, parallel processing, cluster configuration, and Spark performance-tuning techniques.
✓Develop and maintain cloud-based data workflows and applications, including job orchestration, workflow dependencies, authentication, secrets management, monitoring, audit logging, exception handling, failure recovery, notifications, and environment-specific configurations.
✓Integrate data-processing applications with relational databases, cloud storage, REST APIs, and enterprise systems, including the processing and standardization of XML and JSON data.
✓Participate in the software development lifecycle, including technical design, development, testing, debugging, deployment, production monitoring, troubleshooting, and enhancement of data-processing applications.
✓Analyze application and pipeline performance, identify technical issues and root causes, and implement solutions to improve scalability, reliability, maintainability, and processing efficiency.
✓Collaborate with software engineers, data architects, analysts, project managers, and business stakeholders to analyze requirements, design technical solutions, develop data-processing applications, and support implementation and production operations.
Apply: Sam@PlatinoTech.com