Big Data Engineer III

Overview: As a Big Data Engineer, you will be responsible for designing, implementing, and maintaining large-scale data processing systems. You will work closely with data scientists, analysts, and other stakeholders to develop robust data pipelines and optimize data workflows for the organization's analytical needs.

Key Responsibilities:

• Designing and implementing scalable and reliable data pipelines to ingest, process, and store large volumes of data from various sources.


• Developing and maintaining data architecture and infrastructure, including data lakes, warehouses, and streaming platforms.


• Collaborating with cross-functional teams to understand data requirements and translate them into technical solutions.


• Optimizing data workflows for performance, reliability, and cost-efficiency.


• Identifying and resolving data quality issues and ensuring data integrity and consistency.


• Implementing data security and compliance measures to protect sensitive information.


• Researching and evaluating new technologies and tools to improve data processing capabilities and efficiency.


• Providing technical guidance and support to junior team members and stakeholders.


• Documenting system architecture, design decisions, and implementation details.


• Participating in Agile development practices, including sprint planning, code reviews, and retrospectives.

• Bachelor's degree or higher in Computer Science, Engineering, or a related field.


• Proven experience as a Big Data Engineer or similar role, with a strong understanding of data processing concepts and techniques.


• Proficiency in programming languages such as Python, Java, Scala, or SQL.


• Hands-on experience with big data technologies and frameworks, such as Hadoop, Spark, Kafka, HBase, and Hive.


• Experience with cloud platforms such as AWS, Azure, or Google Cloud Platform.


• Familiarity with containerization and orchestration tools such as Docker and Kubernetes.


• Strong analytical and problem-solving skills, with the ability to troubleshoot complex data issues.


• Excellent communication and collaboration skills, with the ability to work effectively in a team environment.


• Experience with Agile methodologies and tools such as Jira or Trello.


• Knowledge of data security and compliance standards such as GDPR and HIPAA is a plus.