Senior Data Engineer

Overview:


Job Purpose





Responsible for maintaining large mortgage and real estate data sets for ICE Data and Analytics to produce data products and custom analytics tools. Responsibilities include designing, building, and maintaining the infrastructure and systems that enable the collection, storage, processing, and analysis of large volumes of ICE mortgage and real estate data.





 





Responsibilities





 





Understanding the business need, securing the right data sources, designing usable data models, and building robust & dependable data pipelines





Work with and data scientists and an application research and development team who use data and machine learning solutions to uncover valuable insights from ICE mortgage and real estate data





Design, build, and maintain robust and efficient data pipelines that collect, process, and store data from various sources.





Ensure data quality, consistency, and accuracy by building out robust data monitoring processes.





Build scalable data pipelines (SparkSQL & PySpark) leveraging Airflow scheduler/executor framework





Collaborate with cross-functional teams, including Data Scientists, Product Managers, and Software Engineers, to define data requirements, and deliver data solutions that support ICE mortgage performance and real estate data platforms.





Knowledge and Experience





 





5+ years of relevant industry experience with a BS/Masters, or 2+ years with a PhD





Experience with distributed processing technologies and frameworks, such as Hadoop, Spark, Kafka, and distributed storage systems (e.g., HDFS, S3)





Demonstrated ability to analyze large data sets to identify gaps and inconsistencies, provide data insights, and advance effective product solutions





Expertise with ETL schedulers such as Apache Airflow, Luigi, Oozie, AWS Glue or similar frameworks





Solid understanding of data warehousing concepts and hands-on experience with relational databases (e.g., PostgreSQL, MySQL) and columnar databases (e.g., Redshift, BigQuery, HBase, ClickHouse)





Excellent written and verbal communication skills





Schedule





 





This role offers work from home flexibility of one day per week. 





 





New York Base Salary Range 





  





The expected base salary for this role if located in New York City is between 139,100 - 195,700 USD.  The base salary range does not include Intercontinental Exchange’s incentive compensation.  While we provide this range as general guidance, at ICE we compensate employees based on the skillset and experience of the individual.  





 





Intercontinental Exchange, Inc. is an Equal Opportunity and Affirmative Action Employer.  All qualified applicants will receive consideration for employment without regard to , , , , , , or ancestry, , or veteran status, or other protected status.   





 





 





#LI-RS1





#LI-Hybrid