S
Senior Data Engineer / Databricks Expert
Stermedia
Contract type
Freelance
Work mode
Remote
Experience
Senior · Not stated
Level inferred from job title
Job description
Senior Data Engineer role on a data platform team building a global analytics environment for a pharmaceutical organization, integrating data from over 80 facilities worldwide. The role covers the full delivery lifecycle from client discussions and requirements analysis to solution design, implementation, and production deployment using Databricks and lakehouse architecture.
At a glance
- Employment
- Independent contractor
- Work mode
- Remote
- Languages
- English
- Pay
- PLN 20,000 – 30,000 / month
What you'll do
- Design, develop, and maintain data pipelines in Databricks using Python, PySpark, and SQL
- Build and maintain Delta Lake tables and transformation layers following bronze, silver, and gold architecture
- Integrate data from multiple source systems into consistent, analytics-ready datasets
- Implement incremental processing, change data capture, and schema evolution where required
- Develop and orchestrate production workflows, including dependencies, scheduling, retries, and monitoring
- Optimize Spark workloads, SQL queries, and compute usage to improve performance and manage costs
- Implement automated data quality checks, validation, and pipeline tests
- Support data access management, governance, and lineage using Unity Catalog
- Maintain reusable code, technical documentation, and Git-based development workflows
- Collaborate with business stakeholders, data engineers, analysts, and machine learning specialists to translate requirements into reliable data solutions
What we're looking for
- Strong commercial experience with Databricks, including developing and operating production data pipelines
- Advanced knowledge of Python, PySpark, and SQL
- Hands-on experience with Apache Spark, distributed data processing, and performance troubleshooting
- Practical experience with Delta Lake, including incremental loads, merge operations, and schema management
- Strong understanding of lakehouse architecture, ETL/ELT patterns, and data modeling
- Experience orchestrating and monitoring workflows in Databricks
- Practical knowledge of Unity Catalog, including permissions, data organization, and lineage
- Experience optimizing pipeline performance and compute resource usage
- Familiarity with cloud storage and services in at least one major cloud environment: Azure, AWS, or GCP
- Experience with Git, code reviews, automated testing, and CI/CD workflows
- Strong analytical and communication skills, with the ability to work directly with international stakeholders
- Good command of English
Nice to have
- Experience with DBT, including models, tests, macros, and integration with Databricks
- Experience with Structured Streaming and Databricks Auto Loader
- Familiarity with Terraform and infrastructure as code
- Experience supporting datasets and workflows used by machine learning teams
- Experience building global analytics platforms or integrating data from multiple facilities
- Familiarity with pharmaceutical, manufacturing, laboratory, or supply chain data
- Databricks certifications in data engineering
Your team
data platform team
International data engineering, analytics, and machine learning teams building a global analytics environment for a leading pharmaceutical organization.
Benefits
- Opportunities to work with modern data engineering and machine learning technologies
- An annual self-development budget
- The opportunity to contribute to a variety of interesting projects
- Internal workshops and knowledge-sharing sessions
- Support for personal branding through articles, conference talks, and leading internal workshops
- Flexible working hours
- The possibility of remote work
- A chillout room, free beverages, and team and company events
- A friendly atmosphere
- MultiSport
- LuxMed
Apply
Direct apply at Stermedia
Opens the job posting at Stermedia