KEY RESPONSIBILITIES

 

      I.          Design, develop, and maintain scalable ETL/ELT data pipelines supporting both batch and real-time processing.

    II.          Integrate data from multiple heterogeneous sources, including databases, APIs, flat files, and unstructured data (text, images, audio).

  III.          Implement and manage data cleansing processes, including deduplication, standardisation, and handling of missing or inconsistent data.

  IV.          Ensure data quality and integrity through automated validation, profiling, and monitoring mechanisms.

    V.          Execute and support data migration activities from legacy systems, including full load, incremental, and real-time data migration.

  VI.          Develop and maintain data transformation and mapping logic to produce structured, business-ready datasets (Raw, Trusted, Refined layers).

VII.          Perform data reconciliation and validation between source and target systems to ensure accuracy and completeness.

VIII.          Monitor, troubleshoot, and optimize data pipeline performance, reliability, and availability.

  IX.          Implement logging, alerting, and incident management processes for data workflows.

    X.          Support data governance initiatives, including metadata management, data lineage, and documentation.

  XI.          Collaborate with Data Architects, BI/Analytics teams, and application teams to deliver high-quality data solutions.

XII.          Maintain comprehensive technical documentation of data pipelines, transformations, and processes.

XIII.          Carry out such acts as shall be required for the proper fulfilling of duties listed above.

 

 

RESOURCE ALLOCATED (Standard as allocated to employees within Job Level)

 

 

 

QUALIFICATIONS & EXPERIENCE

• Bachelor’s degree in Computer Science, Software Engineering, Data Science, or equivalent.
• 3 to 6 years of experience in Data Engineering or related roles.
• Experience in data migration or system integration projects is a strong advantage.
• Exposure to enterprise-scale data platforms preferred.

 

TECHNICAL SKILLS

·       Strong proficiency in SQL or Python (Pandas, PySpark); knowledge of Java/Scala is an advantage.

·       Experience with ETL/ELT tools and frameworks such as Apache Airflow, dbt, Talend, or Informatica.

·       Hands-on experience with data processing frameworks (Apache Spark) and streaming technologies (Kafka, Kinesis, or Pub/Sub).

·       Familiarity with cloud data platforms:

o   AWS (S3, Glue, Redshift, EMR, DMS)

o   GCP (BigQuery, Dataflow, Dataproc, Datastream)

o   Oracle OCI (ADW, Object Storage, Data Integration, GoldenGate)

·       Strong knowledge of relational databases (Oracle, PostgreSQL, SQL Server) and data warehouses/lakes.

·       Experience with data migration and integration tools (AWS DMS, GoldenGate, APIs).

·       Understanding of data quality, validation, and governance practices

·       Experience with orchestration, monitoring, and DevOps tools (Airflow, Cloud monitoring tools, Git, CI/CD).

 

 

COMPETENCIES

• Strong engineering mindset focused on robustness and reliability
• Strong analytical and problem-solving skills
• Ability to work in complex and evolving environments
• Strong collaboration in cross-functional teams
• High level of autonomy and ownership
• Good communication skills (English required)