Senior Data Engineer (Spark)
We are the biggest datalake in our Company. Besides batch ingestion we do support near real-time data ingestion and distribution using Kafka. We plan to move to a new streaming ingest platform, using container technologies and focus more on being an event based datalake. We have build a generic, metadata driven, data pipeline on top of Nifi. You will be joining a team of experts, where you can learn, share and will have fun.
Client Details
Location: Katowice/Warszawa
Description
- Build the solutions required for optimalloading of data from a wide variety of datasources
- Create and maintain, with the team, an optimaldata pipeline architecture
- Assemble large, complex data sets that meetfunctional / non-functional requirements
- Identify, design, and implement internal process improvements
- Keep data separated and secure acrossinternational boundaries through multipledata centers
- Work with data and analytics experts to strivefor greater functionality in our data systems
- Work with stakeholders including the Data andDesign teams to assist with data-relatedtechnical issues and support their datainfrastructure needs
Profile
We are looking for you if:
- You have experience with Spark and Kafka,
- You have experience with HDFS/Hive and generally with Hadoop,
- You have advanced working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases,
- You have great development skills in Scala, Python or Java,
- You hav 5+ years of experience in a Data and/or Software Engineer role,
- You are communicative person,
- You have very good analitical skills and can solve complex problems,
- You can complete tasks and achieves results in an efficient, timely and high-quality manner.
Job Offer
You'll get extra points for:
- Experience with Nifi and groovy based processors,
- Spring/springboot based microservices creation,
- Bash and/or Ansible knowledge,
- Experience in the financial services area.
- Experience with stream-processing using Flink or Kafka,
- Experience in building and optimizing big scale data pipelines, architectures and data sets - in both batch and real-time data integration,
- Working knowledge of message queuing, stream processing, and highly scalable BigData data stores,
- Experience working in a DevOps environment - CI/CD, Azure DevOps, Test automation, Dockereld.
Please apply via the button on the right side of this job offer.
Job details
Location: Katowice
Contract type: Na czas nieokreślony
About Michael Page
Michael Page is a leading professional recruitment consultancy specialising in the recruitment of permanent, contract and temporary positions on behalf of the world's top employers.
More
Location: Katowice
Contract type: Na czas nieokreślony
About Michael Page
Michael Page is a leading professional recruitment consultancy specialising in the recruitment of permanent, contract and temporary positions on behalf of the world's top employers.
More
Follow us on:
