<div class="show-more-less-html__markup show-more-less-html__markup--clamp-after-5 relative overflow-hidden"> <p><strong>Project Overview</strong></p><p>We are looking for a <strong>Senior Databricks Data Engineer </strong>to support our partner's existing data platform focused on monitoring, anomaly detection, and large-scale log processing.</p><p>The platform is already running in production and processes incoming data every 15 minutes through scalable Databricks pipelines. The next phase of the project focuses on improving performance, scalability, data quality, monitoring, and introducing more advanced prediction and anomaly detection capabilities.</p><p><br/></p><p>You will work in a small but highly impactful team and play a key role in further developing and optimizing the platform.</p><p><br/></p><p><strong>🤝Responsibilities</strong></p><ul><li>Develop and optimize scalable data pipelines in Databricks</li><li>Improve data quality frameworks, monitoring, and alerting mechanisms</li><li>Optimize Spark workloads, cluster configurations, and Delta Lake performance</li><li>Support implementation and improvement of anomaly detection and forecasting models</li><li>Work with structured and semi-structured log data</li><li>Improve pipeline stability, scalability, and cost efficiency</li><li>Implement and maintain CI/CD processes for Databricks environments</li><li>Collaborate closely with senior engineers and stakeholders</li><li>Support deployment and operationalization of data and ML workflows</li></ul><p><br/></p><p><strong>✅Required Skills &amp; Experience</strong></p><ul><li>Strong hands-on experience with Databricks</li><li>Solid experience with:</li><li>Spark / Spark SQL</li><li>Python</li><li>Delta Lake</li><li>Data Engineering best practices</li><li>Experience with cluster optimization, autoscaling, and performance tuning</li><li>Good understanding of data quality checks and monitoring concepts</li><li>Experience with Databricks Workflows and Databricks Asset Bundles</li><li>Experience with Unity Catalog</li><li>Familiarity with CI/CD pipelines (Azure DevOps, Jenkins or similar)</li><li>Strong troubleshooting and debugging skills in distributed environments</li><li>Experience working in cloud-based environments (preferably Azure)</li></ul><p><br/></p><p><strong>⭐Nice-to-Have</strong></p><ul><li>Experience with MLflow</li><li>Experience with anomaly detection or forecasting models</li><li>Knowledge of medallion architecture</li><li>Experience with streaming or near real-time data pipelines</li><li>Understanding of DataOps / MLOps concepts</li></ul> </div>