Senior Software Engineer (Python)

About the role

We are seeking a Senior Software Engineer with deep Python expertise to design and operate large‑scale data ingestion and search systems for a fast‑moving partner in Saudi Arabia. The role focuses on building distributed crawling pipelines, processing heterogeneous data sources and delivering reliable APIs for downstream ML/NLP and analytics workloads.

Key responsibilities

• Design and maintain high‑throughput web crawling and data extraction platforms.
• Develop robust pipelines that ingest, normalize and store data from web pages, APIs, PDFs and other documents.
• Implement data transformation logic to support classification models and search indexing.
• Build and expose APIs backed by ElasticSearch/OpenSearch.
• Collaborate with ML/NLP and infrastructure teams to integrate models into production.
• Automate workflows with Apache Airflow and deploy scalable services on Kubernetes and AWS.
• Optimize processing using Spark on EMR and ensure system reliability and scalability.

Required profile

• 5+ years of professional Python development, especially in web scraping and data pipelines.
• Strong experience with REST APIs and handling structured and unstructured data formats.
• Proven ability to work with search technologies such as ElasticSearch or OpenSearch.
• Hands‑on experience with distributed processing tools like Apache Airflow and Spark.
• Solid problem‑solving skills for anti‑scraping challenges and large‑scale data processing.

Required skills

• Python
• REST APIs
• PDF processing (Tesseract, PyMuPDF)
• ElasticSearch / OpenSearch
• Relational and NoSQL databases
• Apache Airflow
• Apache Spark (EMR)
• Kubernetes
• AWS