Staff Software Engineer - OpenTelemetry Python instrumentation | UK | Remote
Job description
Staff Software Engineer role on the OpenTelemetry SDKs team, focused on the Python implementation of OpenTelemetry with an emphasis on OpenTelemetry for Generative AI. The engineer works upstream-first to develop and maintain the OpenTelemetry Python SDK and instrumentation libraries and bring those innovations into Grafana Labs products.
At a glance
- Employment
- Full-time · Employee (employment agreement)
- Work mode
- Remote · UK
- Pay
- GBP 107,000 – 134,000 / year
What you'll do
- Develop improvements to the OpenTelemetry Python SDK and instrumentation libraries, with a strong focus on AI applications and agent workflows
- Develop and maintain the OpenTelemetry Python SDK and instrumentation libraries, working with maintainers and other vendors on specifications, APIs, and implementation decisions
- Build instrumentation that connects agent runs, model calls, and tool execution, including across asynchronous code
- Contribute to evolving OpenTelemetry conventions for generative AI, with care for sensitive data and metric cardinality
- Use profiling and benchmarks to evaluate implementation approaches with the upstream community and make sound tradeoffs around runtime overhead, compatibility, and maintainability
- Teach others and promote best practices around OpenTelemetry, semantic conventions, and various technologies and frameworks within Grafana Labs and the broader OpenTelemetry community
- Support the technical direction and vision of the team, and contribute to business conversations on the OpenTelemetry strategy at Grafana Labs
- Own projects from an ambiguous problem through design, implementation, testing, documentation, and release
- Lend knowledge and skills to GTM, Product and Support teams; join customer conversations, investigate instrumentation issues, and turn learnings into fixes and practical guidance
What we're looking for
- 8+ years of experience in software development
- Strong proficiency in Python
- Experience debugging production software, language internals, libraries, and asynchronous code
- An interest in AI applications and agent systems, and curiosity about how to instrument and troubleshoot them
- Experience working in open-source communities; eager to become an active member of the OpenTelemetry community, collaborate with maintainers and users, and keep learning across languages, frameworks, and the wider observability ecosystem
- Familiarity with observability tooling such as Grafana, Prometheus, and OpenTelemetry, alongside experience with cloud-based software or systems engineering
- Understanding of production environments and ability to assess operational risks
- High agency and good communication skills
Nice to have
- Experience working on open source observability projects, ideally in OpenTelemetry or Prometheus
- Experience in Python instrumentation techniques, profiling, runtime diagnostics, or asynchronous context propagation
- Experience in a hands-on technical leadership role, setting technical direction and leading project teams
- Experience instrumenting AI frameworks or agent workflows, or working with OpenTelemetry semantic conventions for generative AI
- Experience contributing to or maintaining Open Source projects, with evidence of successful pull requests and community collaboration
- Rust experience for exploring potential Python SDK performance improvements, including Rust-backed implementations
- Prior experience building AI systems
Your team
OpenTelemetry SDKs team
Part of the Observability department; the team's mission is to make sure users succeed in instrumenting their software with OpenTelemetry, working closely with the OpenTelemetry SIGs to make OpenTelemetry SDKs and instrumentation libraries a success.
Benefits
- Restricted Stock Units (RSUs)
- 100% Remote, Global Culture
- Scaling Organization
- Transparent Communication
- Innovation-Driven
- Open Source Roots
- Empowered Teams
- Career Growth Pathways
- Approachable Leadership
- Passionate People
- In-Person onboarding
- Global annual leave policy of 30 days per annum, with 3 days reserved for Grafana Shutdown Days
Apply
Direct apply at Grafana Labs
Opens the job posting at Grafana Labs
About Grafana Labs
The company behind Grafana Cloud, the fully managed observability platform
Grafana Labs is the company behind Grafana Cloud, the fully managed observability platform trusted by more than 10,000 organizations to ensure reliability, resolve incidents faster, and optimize telemetry at scale. Built on open source and open standards and designed for interoperability across any stack, Grafana Cloud brings AI to observability and observability to AI. We are a 100% remote company with team members across 40+ countries.
Key facts
- Observability / software
- Grafana Cloud is trusted by more than 10,000 organizations
- 100% remote company with team members across 40+ countries
- Backed by leading investors including Lightspeed Venture Partners, Sequoia Capital, GIC, Coatue, J.P. Morgan, CapitalG, and Lead Edge Capital
- Built on open source and open standards
- Global annual leave policy of 30 days per annum
Partners & customers
What they value
- Transparency
- Autonomy
- Trust
- Open source roots
- High trust, low ego culture that values outcomes over optics