Job Description
Job Title: Databricks Administrator
Location: Pittsburg, PA - Onsite
Duration:Fulltime
Term: Full Time
Qualifications:
Education:
- Bachelor's degree in Computer Science, Information Technology, Data Engineering, or a related field.
Experience:
- 3+ years of experience in administering cloud-based data platforms, with a focus on Databricks.
- Experience with big data technologies such as Apache Spark, Hadoop, and data lakes.
- Proficiency in cloud platforms like AWS, Azure, or Google Cloud, particularly in managing data and analytics services.
Skills:
- Strong understanding of Databricks architecture, including clusters, jobs, notebooks, and integrations.
- Excellent problem-solving skills and attention to detail.
- Strong communication and interpersonal skills.
- Ability to work independently and as part of a team.
- Familiarity with programming languages such as Python, Scala, or SQL.
- Certifications:
- Databricks Certified Associate/Professional Administrator or similar certification is preferred.
- Cloud certifications (e.g., AWS Certified Solutions Architect, Azure Data Engineer Associate) are a plus.
Key Responsibilities:
Platform Administration:
- Manage and monitor Databricks workspaces, clusters, jobs, and notebooks.
- Configure and optimize cluster policies, instance pools, and auto-scaling settings to ensure efficient resource usage.
- Oversee the installation, configuration, and maintenance of Databricks components and integrations.
User and Access Management:
- Manage user access and permissions, ensuring compliance with organizational security policies.
- Configure and manage Identity and Access Management (IAM) roles and policies within Databricks.
- Support users in onboarding, troubleshooting access issues, and ensuring proper usage of the platform.
Security and Compliance:
- Implement and enforce security best practices, including encryption, network security, and data access controls.
- Monitor and audit Databricks environments to ensure compliance with data governance policies and regulatory requirements.
- Work with the IT security team to ensure that Databricks is integrated with the organization's security protocols.
Performance Monitoring and Optimization:
- Monitor the performance of Databricks clusters and jobs, identifying and resolving issues related to resource usage, performance bottlenecks, and job failures.
- Optimize the performance of data pipelines, ETL processes, and machine learning workflows on Databricks.
- Implement monitoring and alerting solutions to proactively address performance and reliability issues.
Support and Troubleshooting:
- Provide technical support to users, including data engineers, data scientists, and analysts, in their use of Databricks.
- Troubleshoot and resolve issues related to cluster configurations, job execution, data access, and integration with other systems.
- Work closely with Databricks support and engineering teams to address platform issues and implement fixes or improvements.
Collaboration and Communication:
- Collaborate with data engineers, data scientists, and other stakeholders to understand their needs and ensure that the Databricks environment supports their requirements.
- Communicate platform updates, changes, and best practices to users and stakeholders.
- Participate in cross-functional projects and initiatives involving the Databricks platform.
Continuous Improvement:
- Stay updated with the latest features, updates, and best practices related to Databricks and big data technologies.
- Recommend and implement improvements to the Databricks environment to enhance performance, security, and user experience.
- Contribute to the development of documentation, training materials, and guidelines for Databricks users.
Key Skills:
Databricks, Cloud based platform, Big data technologies, Admin
Job Tags
Full time,