Note: The job is a remote job and is open to candidates in USA. Clevanoo LLC is seeking a Databricks Engineer to consult with customers and optimize Databricks environments. The role involves designing scalable ETL/ELT pipelines and integrating Databricks with various cloud platforms while ensuring data governance and compliance best practices are followed.
Responsibilities
- Consult directly with customers to design, optimize, and fine-tune Databricks environments
- Design, build, and maintain scalable ETL/ELT pipelines
- Optimize Databricks clusters, workflows, and architecture for cost, scalability, and performance
- Integrate Databricks with AWS, Azure, and GCP environments
- Implement data governance, security, compliance, and data management best practices
- Utilize Spark, Delta Lake, MLflow, and Databricks Machine Learning technologies
- Collaborate with engineers, analysts, data scientists, and business stakeholders
- Troubleshoot platform and infrastructure issues while driving continuous improvements
- Support advanced analytics, machine learning, and AI-focused customer initiatives
Skills
- Active Secret Clearance
- U.S. Citizenship
- Proven experience with Databricks in production environments
- Strong Python and/or SQL skills
- Apache Spark experience
- Experience building and maintaining ETL/ELT pipelines
- Knowledge of Data Lakehouse architecture
- Experience with AWS S3, Azure Data Lake, or GCP Storage
- Data modeling, data quality, and performance tuning experience
- CI/CD and Git experience
- Bachelor's degree preferred (experience considered in lieu of degree)
- Ability to travel as needed
- Master's Degree or PhD
- MLflow experience
- Delta Lake experience
- Unity Catalog experience
- Kafka, Kinesis, or Event Hubs
- Tableau, Power BI, or Looker
- Terraform, ARM, or CloudFormation
- FedRAMP, HIPAA, SOC2, or similar compliance experience
- DevOps/MLOps experience
- Prior consulting or customer-facing experience
- Agile project delivery experience and familiarity with professional services environments
Company Overview