Senior Data Engineer
About Nayya
At Nayya, we believe there’s a better way to choose benefits. A more transparent, less confusing way for employees to control their health and financial potential. Powered by billions of data points and machine learning, our benefits experience platform delivers personalized decision support and guidance during open enrollment, new employee onboarding, qualifying life events, and in the moments that matter all year round. This is one of the most stressful and challenging situations consumers face – and we see that as an opportunity to build an innovative response that can help millions of Americans possess the control and understanding they deserve.
Your Role
Work directly with the Nayya health data infrastructure team to power the claims data extraction and integration platform. In addition to maintaining and working on claims file feeds and API integrations, you will join the team as we build a high volume, batch file claims data pipeline.
The team maintains an internal facing API to power end user applications that leverage claims to enhance user experiences. You will help with maintaining that infrastructure, building new endpoints to power new use cases as well as working on an emerging, centralized data strategy for the organization. We will be working as a team to enhance data validation, governance, and access to organizational data from a single source of truth while building new infrastructure to exceed product requirements for our 2024 product goals and beyond. This is an excellent opportunity to get hands-on experience solidifying and scaling data infrastructure at a fast-growing startup.
Responsibilities
- Build and maintain our Interoperability API and claims file feed integrations.
- Help design and implement new batch processing infrastructure.
- Enhance our data enrichment service to interface with external/third party data sources.
- Help design and implement a de-identified reporting and analytics platform.
- Maintain and improve an existing Python tech stack with a focus on security and scalability for data storage and API endpoints.
- Work with the internal stakeholders to build and maintain data and pipelines for various use cases.
- Recommend monitoring and analytics tools to automate common data needs and visibility.
Requirements
- Strong Python experience.
- Strong PySpark experience.
- Experience with an orchestration tool like Airflow, Dagster, etc.
- Strong SQL skills including query performance and optimization techniques including explain plans.
- Experience building and maintaining APIs to serve data to internal stakeholders.
- Familiarity with implementing and using Apache Spark, AWS Glue, and/or EMR.
- Data pipeline development and maintenance experience.
- Familiarity with batch processing pipelines.
- Familiarity with data pipeline tuning and performance testing.
- Familiarity with building data processing infrastructure from the ground up.
- Familiarity with one or more RDBMS.
- Ability to identify tradeoffs for warehousing vs data lake infrastructure and applying solutions to the appropriate use case.
- Familiar with common pitfalls in high volume, partitioned data ingestion pipelines such as orphaned records and table locks
Nice to haves:
- Experience with Terraform.
- Experience with Apache Hudi or other data lake platforms.
- Experience with data streaming pipelines and sources.
- Experience building and maintaining APIs (Flask, FastAPI, etc).
- Experience with AWS managed services and data engineering solutions.
- Experience with other languages such as Go, Ruby, and Javascript.
The salary range for New York based candidates for this role is $140,000-$175,000. We use a location factor to adjust this range for candidates that are located outside of geographic region of our New York office. Placement within the salary band is determined based on experience.
#LI-KD1
#LI-HYBRID
#BI-Hybrid
Why Join Nayya?
- Be an early employee of a quickly growing, VC-backed start-up - grow with us!
- Have a meaningful impact on a platform that is scaling very rapidly
- Contribute to a values-based culture with an emphasis on empowerment and autonomy
- Work in a highly collaborative, cross-functional environment
- Benefits include: Competitive pay, employer-paid healthcare, stock options
Nayya is proud to be an Equal Employment Opportunity employer. We do not discriminate based upon race, religion, color, national origin, gender (including pregnancy, childbirth, or related medical conditions), sexual orientation, gender identity, gender expression, age, status as a protected veteran, status as an individual with a disability, or other applicable legally protected characteristics
