Overview

AI/ML Engineer — Data, Evaluation & Model Improvement

Qualifications

• Data pipelines: Transform approved agent traces, tool outputs, expert feedback and verified outcomes into structured, versioned datasets.
• Data quality: Implement cleaning, deduplication, annotation workflows, provenance tracking and sensitive-data handling.
• Evaluation datasets: Create training, validation and held-out test sets, preventing related cases or near-duplicate examples from leaking across splits.
• Model and agent evaluation: Build repeatable evaluations for task completion, tool-use accuracy, evidence quality, false positives, latency and compute cost.
• Failure analysis: Investigate whether failures originate from the model, prompts, retrieval, tools, data or workflow design, and recommend the appropriate improvement.
• Fine-tuning: Run reproducible supervised fine-tuning experiments, including approaches such as LoRA or QLoRA where suitable.
• Retrieval improvement: Evaluate embeddings, rerankers and retrieval pipelines to improve the relevance of the information supplied to agents.
• Experiment tracking: Record dataset versions, configurations, model checkpoints, metrics and failure cases so another engineer can reproduce the results.
• Product integration: Work with the runtime engineer to integrate accepted models and improvements with compatibility tests, regression checks and rollback support.
• Feedback pipelines: Turn reviewed product failures into candidate dataset, evaluation or model improvements before promoting them into a release.
• Strong Python skills and practical experience building data processing or machine learning pipelines.
• Working knowledge of PyTorch and the Hugging Face ecosystem, or comparable tools.
• A hands-on LLM fine-tuning or post-training project covering data preparation, a baseline, held-out evaluation and failure analysis.
• Understanding of training versus inference, overfitting, generalisation, loss functions and evaluation metrics.
• Familiarity with dataset quality, annotation, deduplication and data leakage prevention.
• Understanding of LLM prompting, structured outputs, retrieval and tool use.
• Ability to work with Linux, Git, GPU environments and reproducible experiment configurations.
• Ability to explain why a change improved results—or why it should be rejected.
• Familiarity with cybersecurity, VAPT workflows, vulnerability reports or security datasets.
• Strong applied AI/ML skills are essential. Cybersecurity or VAPT experience is a plus. Our security specialists will help define domain labels, review evidence and validate security findings.
• Exposure to PEFT, TRL, experiment tracking and dataset versioning tools.
• Familiarity with private model deployment and inference serving.
• Hands-on access to SecNinjaz’s on-premises NVIDIA H200 GPU infrastructure for planned product training, fine-tuning, evaluation and inference workloads.
• An opportunity to deepen your expertise through real experiments and measurable product outcomes.
• Collaboration with cybersecurity specialists who can help turn domain expertise into high-quality datasets and evaluations.
• Ownership of the improvement process, from identifying a failure to evaluating and integrating a better solution.
• We are especially interested in how you measured improvement, prevented data leakage and investigated failures.