Senior Data Engineer
On behalf of our client, Procom Consultants Group is currently looking for a Senior Data Engineer.
Senior Data Engineer Job Details
- Analyze the source data, prepare data model & mapping, and develop Scala/Spark programs and related components in the work areas of Information Management ("IM") relating to claims, insurance and finance data.
- Use data pipelines to extract data from data sources which are in various formats (viz - flat files, XML, relational tables, Oracle logs), and use tools (such as StreamSets, Scala and Spark programs) to transform and store the data in big data platform (Data Lake) after data validation.
Deliverables:
1. Develop Spark and Scala pipelines for data ingestion and Data transformation as per the mapping document
2. Data Analysis as per requirements and develop data models/mapping
3. Data validations and automated data quality pipelines
4. Relevant documentation for sustainment
Senior Data Engineer Mandatory Skills
This position would typically be met through post secondary education related to computer systems programming and a few years of related experience.
Applied knowledge in Big Data platforms, ideally with exposure to
Hadoop ecosystem (HDFS, Pig, Hive, SPARK, Big SQL, NoSQL, YARN);
Solid experience in Scala
- Experience in designing efficient and robust data pipelines using Spark;
- Excellent skill in SQL development;
- Knowledge of data modeling and understanding of different data structures and their benefits and limitations under particular use cases;
- Hands-on experience with structured and unstructured databases;
- Experience with enterprise systems such as Guidewire Claim Center,
- Guidewire Policy Center would be an asset;
Senior Data Engineer Start Date
January 31, 2022
Senior Data Engineer Assignment Duration
July 29, 2022 (depending on start date)
Senior Data Engineer Working Location
Fully remote but PST hours