Data Engineer

Responsibilities for the Data Engineer include building the Datawarehouse, engaged in cleansing the data and connecting to various data sources to produce meaningful and trusted information. Ensure that processes and pipelines are automated and data distribution is done in a secured manner.

Job Description :

• Must have hands-on experience on Design Pattern, SCD implementation and good database knowledge.
• Architect and build cost efficient scalable Modern Data Platform architecture (Data Lake, DWH, AI/ML, Domain-driven Data Stores, Data Services etc.) in cloud using Azure services (or equivalent AWS services)
• Architect and build data analytics layer for BI reporting & AI/ML layer for advanced analytical insights.
• Ability to assess the existing system architecture, validate use cases, data availability and make recommendations for a best fit data platform using Azure (or AWS) Services.
• Conduct data profiling activities, discover data quality challenges, document the impacts to business and make recommendations for frameworks covering Data quality, Data Protection and Data Governance
• Design Data pipeline architecture for data ingestion (batch & real-time) and transformation processes using AWS services like Lambda, Amazon Managed Kafka, Glue, etc. (or equivalent Azure services using ADF, MSIR, Power Automate etc)
• Build and Design solution which are compliant to GDPR, Data Protection Act etc.
• Design and optimize data governance framework including the management of data, operating model, data policies and standards using Azure services (or equivalent AWS Datazone).
• Design and implement Exploratory Data Analysis (EDA) and data preparation techniques for AI/ML model generation.
• Experience or Knowledgeable with implementation of ML models for advanced analytics using Azure services (or equivalent AWS Sagemaker).
• Experience with CI/CD deployment processes, DevOps, Orchestration, etc.
• Should effectively communicate and collaborate with Leadership, Business, Technology team members
• Facilitate design and architecture workshops and drive outcomes.

Required technical & Professional expertise:

• A Degree in Information Technology/Computer Science  
• AWS (or Azure) Certification Solution Architect / Data Analytics will be a plus.
• 3+ years developing BI Solutions/Architecture with Proficient with Python, T-SQL, PL\SQL, SQL Server and Oracle Databases, SSIS, SSAS, AS400 and other Databases
• Advanced SQL skills with  Strong database-related development skills (complex SQL queries, complex store procedures and user-defined functions in SQL Server)
• Good knowledge of Google BigQuey, SharePoint, Business Objects, PowerBI, PowerPivot, PowerView, Power Query, Tableau, QlikView, MDX and DAX