Data Engineering

Job Title: Middleware Engineer Position: Data Engineer Experience: 5-6yrs Category: IT Infrastructure Main location: India, Karnataka, Bangalore Employment Type: Full Time Qualification: Bachelor's degree in Computer Science or related field or higher. Roles and Responsibilities Data Engineer - 5-6 years experience. Responsibilities =============== Design, develop, and maintain data architectures, pipelines, and workflows for the collection, processing, storage, and retrieval of large volumes of structured and unstructured data from multiple sources. Collaborate with cross-functional teams to identify and prioritize data engineering requirements and to develop and deploy data-driven solutions to address business challenges. Build and maintain scalable data storage and retrieval systems (e.g., data lakes, data warehouses, databases), fault-tolerant, and high-performance data platforms on cloud infrastructure such as AWS, Azure, or Google Cloud Platform. Develop and maintain ETL workflows, data pipelines, and data transformation processes to prepare data for machine learning and AI applications. Implement and optimize distributed computing frameworks such as Hadoop, Spark, or Flink to support high-performance and scalable processing of large data sets. Build and maintain monitoring, alerting, and logging systems to ensure the availability, reliability, and performance of data pipelines and data platforms. Collaborate with Data Scientists and Machine Learning Engineers to deploy models on production environments and ensure their scalability, reliability, and accuracy. Requirements: =============== Bachelor s or master s degree in computer science, engineering, or related field. At least 5-6 years of experience in data engineering, with a strong background in machine learning, cloud computing and big data technologies. Experience with at least one major cloud platform (AWS, Azure, GCP). Proficiency in programming languages like Python, Java, and SQL. Experience with distributed computing technologies such as Hadoop, Spark, and Kafka. Familiarity with database technologies such as SQL, NoSQL, NewSQL. Experience with data warehousing and ETL tools such as Redshift, Snowflake, or Airflow. Strong problem-solving and analytical skills. Excellent communication and teamwork skills. Preferred qualification: =============== Experience with DevOps practices and tools such as Docker, Kubernetes, or Ansible, Terraform. Experience with data visualization tools such as Tableau, Superset, Power BI, or Plotly, D3.js. Experience with stream processing frameworks such as Kafka, Pulsar or Kinesis. Experience with data governance, data security, and compliance. Experience with software engineering best practices and methodologies such as Agile or Scrum. Must Have Skills =============== data engineer with expertise in machine learning, cloud computing , and big data technologies. Data Engineering Experince on multiple clouds one of them , preferably GCP data lakes, data warehouses, databases ETL workflows, data pipelines,data platforms Hadoop, Spark, or Flink Hadoop, Spark, and Kafka SQL, NoSQL, NewSQL Redshift, Snowflake, or Airflow