Specialist - Data Engineering

Role description

HARD SKILLS

Sound Concepts of Large DatawarehouseData Lake Concepts ETLELT Ab Initio Apache Spark PySpark SQL Oracle HADOOP

Advanced dimensional modeling data vault and schema design for large scale Data Warehouses and Data Lakes

Deep expertise in ETLELT engineering using Ab Initio graphs plans PDL metadata driven design and migration of those patterns to Spark

Hands on PySparkSpark proficiency for batch streaming joins windowing partitioning and performance tuning on large datasets

Strong command of Hadoop ecosystem components HDFS Hive YARN OozieAirflow Ranger Atlas and securitygovernance frameworks

Oracle SQL mastery including performance tuning partitioning materialized views and implementingdecoding Virtual Private Database VPD policies

Data ingestion architecture using CDC Kafka file based ingestion and incremental load frameworks for high volume HR and financial data

Data quality engineering reconciliation frameworks validation rules audit controls lineage and automated regression testing

Cloud and lakehouse engineering on Databricks Delta Lake Unity Catalog cluster optimization job orchestration and CICD

Metadata driven pipeline design reusable transformation frameworks and parameterized job orchestration patterns

Performance engineering across platforms skew mitigation partition strategy broadcast vs shuffle decisions and storage format optimization ParquetORCDelta

SOFT SKILLS

Good Attitude

Team Player

Independent

Handon Coder Designer

Coachable

Ability to learn adapt adopt

Mandatory Karat technical interview clearance is required