Overview

You will design and own the LLM-based ingestion and analysis pipeline that transforms large volumes of unstructured proprietary data into high-accuracy structured intelligence for Clearstate's AI-powered data intelligence engine.

Responsibilities

• Immerse yourself in Clearstate's data: Understand the structure of interview transcripts, import datasets, HS code mappings, and market sizing models
• Audit existing Python scripts, Google Apps Scripts, and ML experiments to understand the current state
• Define the pre-processing pipeline: Convert raw PDFs, Word documents and structured files into clean, normalised inputs suitable for LLM ingestion
• Evaluate and recommend open-source LLMs vs API-based models for the core inference workload, with total cost of ownership in mind
• Produce an architecture proposal and a proof-of-concept demonstrating achievable accuracy on a representative data sample
• Build the LLM ingestion pipeline end-to-end: Pre-processing, prompt engineering, structured output extraction, confidence scoring, exception flagging
• Implement domain-specific validation rules to catch hallucinations and data errors before they reach analysts
• Run parallel trials against the manual process to quantify accuracy and time savings
• Integrate with the data warehouse layer so outputs land in analytics-ready tables
• Instrument the pipeline: Logging, monitoring, data quality checks, exception queues for human review
• Harden the pipeline for production use and hand over maintainable code to the broader engineering team
• Explore agentic workflow patterns (multi-step reasoning, tool-calling) to further automate the analytical chain
• Identify adjacent automation opportunities across the research and data operations workflow
• Contribute to IP documentation and architecture decisions that underpin Clearstate's long-term defensibility

Qualifications

• Strong Python engineering. You write clean, testable, production-quality code
• Deep practical knowledge of large language models, including prompt engineering, temperature / sampling parameters, context management, hallucination mitigation
• Experience pre-processing unstructured data (PDFs, Word, HTML, plain text) into structured formats suitable for LLM input
• Familiarity with open-source LLM frameworks (Hugging Face, Ollama, vLLM, LangChain, or similar)
• Understanding of structured output extraction from LLM responses (JSON schema, function calling, constrained generation)
• Solid grounding in data engineering, including ETL/ELT concepts, SQL, data pipeline design
• Ability to design and run rigorous accuracy evaluations and build confidence in model outputs
• Strong scientific rigour. You validate before you ship and you quantify improvement

Nice to have

• Experience hosting and fine-tuning open-source LLMs in a cloud environment (AWS, GCP, or Azure) to manage cost at scale
• Exposure to RAG architectures for grounding LLM outputs in proprietary data
• Experience with agentic frameworks (LangGraph, AutoGen, CrewAI, or similar)
• Hands-on exposure to AI coding agents or agentic development tools such as OpenAI Codex, Claude Code, Cursor, GitHub Copilot Workspace
• Data warehouse experience: BigQuery, Snowflake, or Databricks
• ML pipeline experience, including feature engineering, model training, validation, and deployment (not just LLM - classical ML for mapping/classification tasks)
• Familiarity with distributed computing or batch processing for large-scale datasets
• Experience building exception and human-in-the-loop review workflows
• Background in a data-intensive scientific or analytical domain (physics, genomics, finance, life sciences, or similar)