GL
Staff Software Engineer - Databases SRE | UK | Remote
Contract type
Ongoing
Work mode
Remote
Experience
Senior · 8+ years
Job description
Staff Software Engineer role at Grafana Labs to keep the multi-tenant SaaS databases Mimir, Loki, Tempo and Pyroscope reliable at scale across AWS, GCP and Azure. Combines building production software and automation with on-call and incident response for the highest-SLA customers.
At a glance
- Employment
- Full-time
- Work mode
- Remote
- Pay
- GBP 103,958 – 124,750 / year
What you'll do
- Write production code in our services, tooling and the Mimir, Loki and Tempo codebases that makes our databases more resilient, self-healing and scalable as demand grows quickly
- Build automation and platform tooling that removes toil and scales our reliability practices across regions and cloud providers
- Author design docs, review others' designs and PRs, and shape feature design so it's scalable and operable from day one
- Define and evolve per-tenant SLOs and reliability models; use SLO burn and incident learnings to find and fix the causes of failure in the systems themselves
- Improve observability and alerting so we find problems before customers do, and so escalations are actionable rather than noisy
- Be on call for our highest-SLA customers; act as a technical escalation point for complex customer environments
- Lead customer-impacting incident response, including customer bridge calls when needed, and write blameless post-incident reviews that lead to engineering fixes
- Investigate customer escalations, follow up on incident actions, tune capacity and configuration, and turn recurring issues into permanent fixes or automation
- Mentor engineers, teach reliability practices that teams adopt early in development, and work with engineering leadership on roadmaps and product strategy
What we're looking for
- 8+ years of software engineering experience, including significant time building and owning distributed systems in production
- Strong production software skills, ideally in at least one object-oriented language; writes clean, tested code and can make targeted changes in large open-source codebases they didn't write
- Deep understanding of distributed systems: performance, scaling and failure modes
- Experience running multi-tenant systems at scale on Kubernetes in AWS, GCP or Azure, with infrastructure-as-code tools such as Terraform, Helm or Jsonnet
- Hands-on experience designing and implementing SLOs
- Experience being on call for production systems; troubleshoots calmly and methodically under pressure, and has taken part in blameless incident response and written high-quality post-incident reviews
- Staff-level technical leadership: driving cross-team projects, setting technical direction and making the engineers around you better
- Working knowledge of Linux internals, networking and cloud storage
Nice to have
- Experience with Mimir, Loki, Tempo, Pyroscope, Prometheus or other observability and time-series databases
- Open-source contributions
- Customer reliability engineering (CRE) or customer-facing escalation experience
Benefits
- Equity
- Bonus (if applicable)
- 100% Remote, Global Culture
- Scaling Organization
- Transparent Communication
- Innovation-Driven
- Open Source Roots
- Empowered Teams
- Career Growth Pathways
- Approachable Leadership
- Passionate People
- In-Person onboarding
- Global annual leave policy of 30 days per annum, with 3 days reserved for Grafana Shutdown Days
Apply
Direct apply at Grafana Labs
Opens the job posting at Grafana Labs
About Grafana Labs
The company behind Grafana Cloud, the fully managed observability platform
Grafana Labs is the company behind Grafana Cloud, the fully managed observability platform trusted by more than 10,000 organizations to ensure reliability, resolve incidents faster, and optimize telemetry at scale. Built on open source and open standards and designed for interoperability across any stack, Grafana Cloud brings AI to observability and observability to AI. We are a 100% remote company with team members across 40+ countries.
Key facts
- Observability / software
- 100% remote company with team members across 40+ countries
- Grafana Cloud trusted by more than 10,000 organizations
- Backed by leading investors including Lightspeed Venture Partners, Sequoia Capital, GIC, Coatue, J.P. Morgan, CapitalG, and Lead Edge Capital
- Global annual leave policy of 30 days per annum
Partners & customers
AnthropicBloombergNVIDIAMicrosoftSalesforce
What they value
- Default to transparency
- High bias towards action
- High trust, low ego culture that values outcomes over optics