Data Engineering Lead

Roles and responsibilities

• Overall 5-7 years ofexperience in data engineering and transformation onCloud
• 3+ Years of Very Strong Experience inAzure Data Engineering, Databricks
• Expertise insupporting/developing lakehouse workloads at enterpriselevel
• Experience in pyspark is required– developing and deploying the workloads to run on theSpark distributed computing
• Candidate mustpossess at least a Graduate or bachelor’s degree inComputer Science/Information Technology, Engineering(Computer/Telecommunication) orequivalent.
• Cloud deployment: PreferablyMicrosoft azure
• Experience in implementing theplatform and application monitoring using Cloud nativetools

• Team LeadershipandManagement:

  • TeamSupervision: Lead and mentor a team of dataengineers to ensure high performance, professional growth, andalignment with organizationalgoals.
  • TaskAllocation: Assign tasks and manage workloadseffectively across the team, ensuring timely delivery of projectswhile maintainingquality.
  • Collaboration:Work closely with cross-functional teams (e.g., data scientists,analysts, business stakeholders) to ensure the data infrastructuresupports their needs andobjectives.
  • Training andDevelopment: Provide guidance, training, andupskilling opportunities to junior and mid-level dataengineers.

• DataArchitectureDesign:

  • PipelineDevelopment: Design and implement scalable,reliable, and efficient data pipelines to process and transformlarge datasets from varioussources.
  • ArchitectureStrategy: Lead the design and architecture of datasystems and solutions, ensuring data is easily accessible, clean,and properly formatted foranalytics.
  • DataModeling: Work on data models, including datawarehouses, lakes, and other data storage solutions, and design theschema that supports efficient querying andreporting.

• DataIntegration andETL:

  • ETLProcesses: Lead the development of Extract,Transform, Load (ETL) processes to integrate data from multiplesources, ensuring data quality, consistency, andreliability.
  • Data QualityAssurance: Ensure that the data pipelines andprocesses meet high standards of data accuracy, consistency, andtimeliness. Establish data validation processes and error handlingmechanisms.

• DataWarehousing andStorage:

  • DataStorage Solutions: Lead efforts in selecting andimplementing data storage solutions, such as data lakes, datawarehouses, and cloud storage platforms (e.g., AmazonRedshift, GoogleBigQuery,Snowflake).
  • Scalabilityand Performance: Design and optimize data storagesystems for performance, scalability, and cost-efficiency, takinginto account future growth and businessrequirements.
  • DatabaseManagement: Oversee the management of relational andnon-relational databases, ensuring optimal performance andsecurity.

Desired candidate profile

• TechnicalExpertise:

  • Expertise indata engineering concepts, such as ETL, data warehousing, datalakes, and data pipelines.
  • Proficiency inprogramming languages such as Python,Java,Scala,SQL, and frameworks likeApache Spark or ApacheFlink for big dataprocessing.
  • Strong experience with cloudplatforms (AWS, GoogleCloud, Azure) andassociated services like S3,Redshift,BigQuery,EMR, andDatabricks.
  • Familiaritywith containerization tools (e.g.,Docker,Kubernetes) for orchestrating anddeploying datapipelines.

• Databaseand StorageKnowledge:

  • Deepknowledge of relational and NoSQL databases, includingMySQL,PostgreSQL,MongoDB,Cassandra, andHBase.
  • Familiaritywith data storage solutions, including DataLakes (e.g., AWS S3,Google Cloud Storage),Data Warehouses (e.g.,Snowflake,BigQuery), and distributed filesystems likeHDFS.

• LeadershipandManagement:

  • Experiencemanaging and leading a team of data engineers, with strongorganizational, mentoring, and communicationskills.
  • Ability to manage multiple projectssimultaneously, ensuring deadlines are met without compromisingquality.
  • Conflict resolution, team-building,and performance managementskills.

• DataModeling andArchitecture:

  • Expertisein data modeling techniques and building scalable dataarchitectures.
  • Ability to design efficient datamodels that support analytics, business intelligence (BI), andmachine learning (ML) usecases.

• ProblemSolving andOptimization:

  • Stronganalytical and problem-solving skills to optimize data pipelinesand ensure efficient data processing andstorage.
  • Experience in debugging andtroubleshooting complex dataissues.

• SoftSkills:

  • Excellentcommunication skills to interact with both technical andnon-technical stakeholders.
  • Strategic thinkingto understand business needs and align technical solutions withthose needs.
  • Ability to work in a fast-paced,collaborative environment and manage competingpriorities.

• AgileMethodologies:

  • Familiaritywith agile methodologies (e.g., Scrum,Kanban) for managing projects andensuring iterative delivery of datasolutions.

Key Skills
DatabaseAdministration,Engineering Mathematics,InformationSystem
Employment Type :Full-time
Department / Functional Area: Data Engineering
Experience: years
Gender: Male
Vacancy: 1