QuickNode Logo

QuickNode

Technical Operations Engineer, Validators & Rollups

Reposted 13 Days Ago
Be an Early Applicant
2 Locations
Expert/Leader
2 Locations
Expert/Leader
Manage production blockchain validators, automate processes using IaC, ensure key security, and lead incident responses to maintain high availability and performance of systems.
The summary above was generated by AI

Quicknode is a cloud-based infrastructure company that powers the blockchain ecosystem.

Our mission is to be the indispensable utility that empowers companies and innovators globally to build next-generation, Web3 enabled businesses & applications using blockchain technology. Quicknode is backed by some of the world's best investors including Tiger Global, Y Combinator, SoftBank, and the Seven Seven Six Fund. The Quicknode team has over 120 people maintaining high performance global data infrastructure for amazing customers serving billions of requests daily.

We are a global remote company with an HQ in Miami, Florida.

Location

Remote (with regional coverage for 24hr operations). Limited travel may be required for conferences or offsites, generally less than 10 days per year.

The Role

Keep validators online and un‑slashable. We run production validators across L1/L2s and a focused set of rollups. You’ll run K8s at scale, ship IaC, lead incidents, and write the tooling that makes production boring—while guarding keys, latencies, and SLOs like they matter (because they do).

What We Actually Need
  • Linux + Kubernetes: you debug real production—networking, storage, rollouts, performance.

  • IaC (Terraform/Helm/Ansible): you ship repeatable infra and zero‑touch deploys.

  • Go or Python (plus Bash): you automate noise away and build small, sharp tools/CLIs.

  • Validator operations: operate/tune validators and sentry topologies; avoid slashing; manage peers; design backups/snapshots/state‑sync; handle client diversity and version skew.

  • Key management & signing: Vault/HSM/KMS; Web3Signer/TMKMS/Horcrux; slashing‑protection DBs; rotation and break‑glass procedures.

  • Observability that matters: SLOs/error budgets; Prometheus/Grafana; alerts that only page when users hurt.

  • Networking & systems depth: DNS/TLS/LB, P2P specifics (ports/NAT/conn limits), kernel/IO tuning, capacity & cost modeling that survives contact with reality.

  • Rollups (targeted scope): deploy/operate OP Stack/Arbitrum/Polygon CDK components (sequencer/batcher/indexer), wire DA endpoints (Celestia/Avail/EigenDA/AnyTrust), upgrade safely.

What You’ll Do
  • Run validators: deploy, upgrade, and tune EVM/Cosmos/Solana validators; manage client diversity (e.g., Prysm/Lighthouse/Teku + Geth/Nethermind/Erigon); integrate MEV‑Boost/relays where relevant; keep latency & attestation/commit rates high.

  • Protect keys & stake: implement secure remote signing, slashing protection, and audited change windows; test restores and failovers regularly.

  • Automate everything: IaC modules, golden images, CI/CD, snapshot pipelines, state‑sync validation, replay/load harnesses.

  • Operate K8s at scale: safe rollouts, HPA tuning, storage choices that won’t bite you; multi‑region capacity and budgets.

  • Lead incidents (SEV0–2): isolate fast, roll forward/back, publish crisp RCAs, and ship the change that prevents a replay.

  • Limited rollups support: stand up/upgrade sequencers & batchers, wire DA, manage contracts and params, and keep L1/L2 bridges healthy.

  • Build signal, not noise: SLOs/error budgets, useful dashboards, alerts that only page when users hurt.

  • Code where it counts: write/extend tools (snapshots, replay/load, state sync checks); occasionally patch client bugs that bite production and upstream when it’s worth it.

Why This Role Stands Out
  • Impact: your work lights up production; chain launches, reliability wins, performance gains.

  • Pace > Ceremony: tight reviews, practical standards, minimal meetings, async‑first.

  • Growth: own big surfaces; learn protocol internals while leveling up distributed systems chops.

  • Remote‑first: follow‑the‑sun coverage; humane on‑call.

  • Compensation & benefits: region‑aligned, bonus-eligible and shared early; no bait‑and‑switch.

The Bar: Signals We Care About
  • Production validator ownership: you’ve run at least one of:

    • Ethereum/EVM: consensus + execution clients in prod (e.g., Prysm/Lighthouse/Teku + Geth/Erigon/Nethermind/Besu), MEV‑Boost/relays, slashing‑protection DB hygiene.

    • Cosmos/CometBFT: validators + sentries, TMKMS/Horcrux remote signing, upgrades/param changes, governance tooling.

    • Solana: Agave/Jito, QUIC tuning, accounts/runtime tuning, high‑throughput storage/I/O decisions.

  • Key security in practice: Vault/HSM/KMS, rotation/backup/restore tested; principle of least privilege enforced.

  • SLO thinking with results: before/after on alert noise, latency, attestation/commit rates, missed slots, MTTR.

  • Tooling & code impact: Go/Python utilities (snapshot/restore, state‑sync verification, health checks), plus IaC modules you’d reuse.

  • K8s & systems trade‑offs: requests/limits/QoS that match workloads; storage & kernel tuning; P2P conn limits/peer selection; change windows/canaries/rollbacks that are real.

  • Targeted rollups experience: shipped/operated OP Stack/Arbitrum/Polygon CDK components and integrated DA endpoints safely.

Process

Steps may vary slightly by region/seniority; we keep it lean.

  1. 30‑min intro + context with Talent Acquisition

  2. 60‑min technical deep dive with the Hiring Manager (incidents you led, K8s/IaC trade‑offs, tooling you built)

  3. 60‑min hands‑on with Team Members (pair on a small plan/code review or targeted tool fix)

  4. 30-min meet a Founder

New Hire Onboarding: Week 1-12

Week 1–2: ship an IaC change + a validator dashboard/alert revamp someone actually uses; verify slashing‑DB backups and restores.

Week 3–6: own 2–3 validator fleets (plus a small rollup stack), kill a flaky alert, and remove a recurring papercut with code/automation.

Week 7–12: lead one upgrade/cutover (validator client or rollup component) with clean rollback; publish an RCA and the platform change that prevents recurrence.

The Quicknode compensation philosophy includes pillars to ensure fair and unbiased compensation for all employees. To design and deliver total reward offerings that are employee-centric. To offer a competitive benefit package in all locations where we operate. To prioritize attracting and retaining the best talent globally. To maintain a high-performing and flexible way of working.

During the hiring process, we are committed to discussing compensation openly and honestly. We encourage candidates to share their salary expectations and requirements early, allowing for an individualized discussion. We know that our total rewards practices impact the lives and wellbeing of our employees. Therefore, we will never stop learning about the market, our business, your needs, and how best to achieve our goals through thoughtful and data-driven practices. If you have any questions or require further information about the compensation for this position, please don't hesitate to reach out to your Recruiter. 

We at Quicknode are an equal opportunity employer and all qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity or expression, pregnancy, age, national origin, disability status, genetic information, protected veteran status, or any other characteristic protected by law.

Top Skills

Ansible
Bash
Go
Grafana
Helm
Horcrux
Kubernetes
Linux
Prometheus
Python
Terraform
Tmkms
Web3Signer

Similar Jobs

5 Minutes Ago
Remote or Hybrid
Gurugram, Haryana, IND
Mid level
Mid level
Cloud • Information Technology • Security • Software • Cybersecurity
The Sales Development Representative engages with enterprise cybersecurity leaders, conducts research, drives personalized outreach, and schedules meetings to advance sales opportunities.
Top Skills: CybersecuritySaaSSales DevelopmentSoftware
9 Minutes Ago
Hybrid
Mumbai, Maharashtra, IND
Senior level
Senior level
Financial Services
Lead Software Engineer responsible for designing and developing software solutions, leading teams, and driving technology innovation within the Consumer & Community Banking Team.
Top Skills: AirflowAthenaAvroAWSEmrEvent BridgeGlue CatalogGlue EtlIamIcebergJavaJson-LdKinesisKmsLake FormationLambdaMskNo Sql DatabasesParquetPythonRelational DatabasesS3Secrets ManagerSpark SqlSpark StreamingStep Functions
10 Minutes Ago
Hybrid
Mumbai, Maharashtra, IND
Expert/Leader
Expert/Leader
Financial Services
Lead the compensation strategy for India GCC, managing analytics and overseeing incentive programs. Drive business goals through HR strategies, ensure compliance, and manage relationships with stakeholders.
Top Skills: Business ObjectsExcelPowerPointWeb Intelligence

What you need to know about the Delhi Tech Scene

Delhi, India's capital city, is a place where tradition and progress co-exist. While Old Delhi is known for its rich history and bustling markets, New Delhi is defined by its modern architecture. It's clear the region places a strong emphasis on preserving its cultural heritage while embracing technological advancements, particularly in artificial intelligence, which plays a central role in shaping the city's tech landscape, fueled by investments in research and development.

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account