Senior Data Engineer

Role description

Job Title: Sr. DATA Engineer

Work Location : Tampa FL

Job Description:

• Seeking a Senior professional with 3 to 5 years of experience in Python Scala and Spark SQL to design and optimize scalable big data processing solutions within the Databricks Spark SQL Scala Python Java ecosystem
• Develop and maintain scalable data processing pipelines using Apache Spark with Python and Scala
• Utilize Spark SQL to analyze large datasets and extract meaningful insights supporting data driven decision making
• Collaborate with cross functional teams to integrate big data solutions into existing systems
• Optimize Spark jobs for performance and efficiency in distributed computing environments
• Ensure data quality and consistency across various data sources
• Stay updated with the latest advancements in Spark Scala Python and related big data technologies to continuously improve processing capabilities

Roles and Responsibilities

• Design develop and deploy complex data processing workflows leveraging Apache Spark Scala Python and SparkSQL
• Write efficient reusable and maintainable code for Spark applications
• Troubleshoot and resolve performance bottlenecks in Spark jobs and data pipelines
• Collaborate with data engineers' data scientists and stakeholders to translate business requirements into technical solutions
• Conduct code reviews and provide mentorship to junior team members on best practices and coding standards
• Participate in architectural discussions and contribute to the evolution of big data platforms
• Monitor and maintain the health availability and reliability of Spark clusters and data services Karat Interview mandate