Note: The job is a remote job and is open to candidates in USA. Keeper Security, Inc. is a leading cybersecurity company transforming the industry with next-generation solutions. They are seeking a Senior QA Analyst to lead testing efforts for AI-driven features and machine learning models, ensuring quality and reliability across Keeper's platform.
Responsibilities
• Design and execute test plans for AI and ML-driven features, including classification, summarization, anomaly detection and autofill systems
• Develop Python-based automation and evaluation scripts to validate model performance, reliability and output quality
• Test LLM-based features for accuracy, guardrail effectiveness, hallucination risk and behavior under edge cases
• Validate data pipelines, input/output transformations and model integration points across Keeper's platform
• Conduct regression testing on ML models and AI workflows to ensure reliability across releases
• Test enterprise-scale behavior analysis features driven by data from ARAM, PAM and session recordings
• Validate security, privacy and zero-trust requirements for AI model interactions and data handling
• Collaborate with ML engineers and cross-functional teams to refine model behavior, identify defects and improve output quality
• Analyze logs, metrics and model outputs to detect anomalies, performance issues or silent failures
Skills
• 3+ years of experience in QA, test automation or evaluation of complex systems
• Hands-on experience writing Python automation or evaluation scripts for testing APIs, model outputs or data pipelines
• Understanding of machine learning concepts such as classification, summarization, embeddings, anomaly detection or behavior analysis
• Experience testing LLM-integrated features or AI-driven workflows
• Familiarity with ML tooling such as Pytest, Hugging Face libraries, Jupyter Notebooks or similar
• Experience with API testing, JSON validation and test data generation
• Ability to test nondeterministic systems and evaluate model quality with structured methodologies
• Strong analytical, troubleshooting and communication skills
• Experience testing cloud-integrated applications or systems deployed on AWS
• Bachelor's degree in Computer Science, Information Systems or related field, or equivalent practical experience
• Experience testing security-focused AI or threat analytics systems
• Familiarity with ML observability tools, vector databases or data processing frameworks
• Experience testing video, image or behavior analysis pipelines
• Knowledge of zero-trust principles and secure data handling
• Experience working with LLM providers or AI guardrail frameworks
Benefits
• Medical, Dental & Vision (inclusive of domestic partnerships)
• Employer Paid Life Insurance & Employee/Spouse/Child Supplemental life
• Voluntary Short/Long Term Disability Insurance
• 401K (Roth/Traditional)
• A generous PTO plan that celebrates your commitment and seniority (including paid Bereavement/Jury Duty, etc)
• Above market annual bonuses
Company Overview
• Keeper Security is transforming cybersecurity for millions of individuals and thousands of organizations globally. It was founded in 2011, and is headquartered in Chicago, Illinois, USA, with a workforce of 501-1000 employees. Its website is https://www.keepersecurity.com.
Company H1B Sponsorship
• Keeper Security, Inc. has a track record of offering H1B sponsorships, with 1 in 2021. Please note that this does not guarantee sponsorship for this specific role.
Apply Now
Apply Now