Senior Software Engineer - Spark
Space and Time
This job is no longer accepting applications
See open jobs at Space and Time.See open jobs similar to "Senior Software Engineer - Spark" Sui.About Space and Time Labs, Inc.
As a Senior Software Engineer for our Data Platform Engineering team you will join skilled Scala / Spark engineers and core database developers responsible for developing hosted cloud analytics infrastructure (Apache Spark-based), distributed SQL processing frameworks, proprietary data science platforms, and core database optimization. This team is responsible for building the automated, intelligent, and highly performant query planner and execution engines, RPC calls between data warehouse clusters, shared secondary cold storage, etc. This includes building new SQL features and customer-facing functionality, developing novel query optimization techniques for industry-leading performance, and building a database system that's highly parallel, efficient and fault-tolerant. This is a vital role reporting to exec leadership and senior engineering leadership.
Responsibilities:
Writing Scala code with tools like Apache Spark + Apache Arrow + Apache Kafka to build a hosted, multi-cluster data warehouse for Web3
Developing database optimizers, query planners, query and data routing mechanisms, cluster-to-cluster communication, and workload management techniques
Scaling up from proof of concept to “cluster scale” (and eventually hundreds of clusters with hundreds of terabytes each), in terms of both infrastructure/architecture and problem structure
Codifying best practices for future reuse in the form of accessible, reusable patterns, templates, and code bases to facilitate meta data capturing and management
Working with a team of software engineers writing new code to build a bigger, better, faster, more optimized HTAP database (using Apache Spark, Apache Arrow, Kafka, and a wealth of other open source data tools)
Interacting with exec team and senior engineering leadership to define, prioritize, and ensure smooth deployments with other operational components
Highly engaged with industry trends within analytics domain from a data acquisition processing, engineering, management perspective
Understand data and analytics use cases across Web3 / blockchains
Skills & Qualifications
5+ years experience engineering software and data platforms / enterprise-scale data warehouses, preferably with knowledge of open source Apache stack (especially Apache Spark, Apache Arrow, Apache Kafka, Apache Ignite/Geode, and others)
2+ years experience with Scala and Apache Spark (or Kafka)
Rock solid engineering fundamentals; query planning, optimizing and distributed data warehouse systems experience is preferred but not required
Nice to have: Knowledge of blockchain indexing, web3 compute paradigms, Proofs and consensus mechanisms… is a strong plus but not required
Experience with rapid development cycles in a web-based environment
Strong scripting and test automation knowledge
Nice to Have's
Bachelor’s degree in computer science or related technical field. Masters or PhD a plus.
Passionate about Web3, blockchain, decentralization, and a base understanding of how data/analytics plays into this
Notice to Third Party Recruitment/Staffing Agencies:
Space and Time Labs, Inc. does not accept unsolicited resumes or CVs from recruiters or employment agencies without a binding, fully executed, written agreement. Any written or verbal communication with any person within Space and Time Labs, Inc. does not create a binding agreement.
Space and Time Labs, Inc. will not pay a fee to any parties that do not have a valid, fully executed Agreement in place. Any unsolicited resumes or CVs, including those submitted to hiring managers or employees are deemed to be the property of Space and Time Labs, Inc. Space and Time Labs, Inc. explicitly reserves the right to hire those candidate(s) without any financial obligation to the recruiter or agency.
Parties without a fully executed and valid agreement are requested not to contact any hiring managers of the Company with recruiting inquiries or resumes. All candidate resumes must be presented to the hiring managers directly by the Company’s internal Talent Acquisition team.
Agencies interested in partnering with Space and Time Labs, Inc. may contact our Talent Acquisition team (careers@spaceandtime.io).
Our Commitment to Diversity and Inclusion:
At Space and Time, we are committed to fostering a diverse and inclusive culture where everyone can excel. We take great care to ensure that our hiring practices are inclusive and meet equal employment opportunity standards. Individuals looking for employment at Space and Time are considered without regard to age, color, disability, ethnicity, family or marital status, gender identity or expression, language, national origin, physical and mental ability, political affiliation, race, religion, sexual orientation, socio-economic status, veteran status, and other protected characteristics.
This job is no longer accepting applications
See open jobs at Space and Time.See open jobs similar to "Senior Software Engineer - Spark" Sui.