Principal Software Data Engineer - ID Analytics
San Diego, CA
ID Analytics is a leader in credit and fraud risk solutions with patented analytics, proven expertise and up-to-the-minute insight into consumer behavior. Founded in 2002, with a vision to think differently about how institutions solve the problem of verifying consumers’ identities, we now solve a full range of identity challenges, including assessing credit risk and improving online customer experience. Our tools help our clients protect themselves and their customers from identity risk and fraud. Our solutions are all powered by analytics, but it’s our access to data unseen by other bureaus that differentiates our solutions. ID Analytics is a Symantec company.
We are looking for an experienced Data Engineer leader to join our team. If you are interested, please follow the link below to apply via our Career Portal. We do not accept applications via e-mail.
The Principal Software Data Engineer will be part of a cross-functional scrum team responsible for support and development of ID Analytics’ data infrastructure using the latest technologies in data platforms (primarily the Hadoop ecosystem) and operating systems.
The ideal candidate will have excellent communication skills with the ability to communicate with employees at all levels of the organization, while providing technical leadership to other team members.
- Provides platform and skill set expertise in
- Pig, MapReduce and Spark to design and implement applications
- Apache Flume, Apache Sqoop and Apache Kafka to connect data ingestion pipelines from multiple, heterogeneous data sources
- Oozie and Spark scheduler schedule workflows for batch and stream based processing of data
- Architects, designs, develops, tests, performs troubleshooting, documentation and maintenance of data infrastructure to maintain the pipeline of data sources to the production environment.
- Design and coding of ancillary subsystems, utilities and scripts to support production systems.
- Design and develop automated tools and process for end to end production deployment.
- Completes documentation and procedures for installation and maintenance.
- Works on complex software and architecture problems where intangible factors and important elements of the scope of software and architecture are often unknown or extremely complex.
- Involved in broad business strategy discussions requiring technology input on data infrastructure and data strategy including discussions on driving new market opportunities with our data assets.
- Capable of selling clients in IDA’s big data technology vision.
- Establishes collaborations with peers in various disciplines across the organization to share and reuse.
- Determines the need for procedures, practices and tools and initiates efforts to acquire.
- Can be a go to person in leading and assessing efforts of a sub team or department.
- Be an active participant in defining and advancing disciplines, industry standards, and trends.
- Promotes and establishes reuse and sharing. Invents reusable tools, techniques and constructs.
- Architect toolset for assessing the quality of data using statistics based validation techniques.
- Architect component level unit tests for data infrastructure.
- Oversee integration level testing.
- Provide Second Level Production Support as needed to ensure the correct functioning of the production data systems on a 24/7 basis.
- Meet with clients in order to facilitate data ingestion.
- Other duties as assigned.
- BS degree plus 14 or more years of relevant experience or a MS degree plus 12 years of relevant experience.
- Candidate must be able to demonstrate excellent problem solving and analytical ability.
- Ten years of experience in data analytics and/or software development.
- Experience designing, building, installing, configuring, supporting, and maintaining high-volume large data systems, both streaming and batch.
- Must be highly proficient with Hadoop technologies, including Pig, Spark, Sqoop, Kafka and Oozie.
- Working knowledge of SQL.
- Strong understanding of RDBMS and NoSQL technologies.
- Other highly desirable skills/experience include:
- Database modeling, schema design and normalization
- Demonstrated working knowledge of a database system.
- Knowledge of ETL and Data Warehouse technologies.
- Knowledge of Scrum/Agile methodologies.
Interested applicants should apply via ID Analytics/Symantec’s Career Portal at the following link. We do not accept applications by e-mail.