Senior Software Engineer II – Applied AI and Evaluations (Remote) at Smartsheet
Artificial Intelligence is reshaping the future of work, and companies are investing heavily in intelligent systems that automate tasks, improve decision-making, and increase productivity. As organizations integrate large language models (LLMs), retrieval-augmented generation (RAG), and AI-powered assistants into their products, the need for experienced software engineers who can build reliable, scalable, and high-quality AI systems continues to grow.
To support the development of its next-generation AI platform, Smartsheet is hiring a Senior Software Engineer II – Applied AI and Evaluations. This remote opportunity is ideal for experienced software engineers who are passionate about artificial intelligence, machine learning, cloud technologies, and software quality.
The role offers the opportunity to work on SmartAssist, Smartsheet's AI-powered work management platform, where you will help improve the quality, reliability, and performance of intelligent AI agents used by thousands of customers.
About Smartsheet
Smartsheet is a leading enterprise work management platform that helps organizations manage projects, automate workflows, collaborate efficiently, and improve productivity. Used by businesses of all sizes across multiple industries, the platform enables teams to organize work, track progress, and make data-driven decisions.
In recent years, Smartsheet has expanded its investment in Artificial Intelligence through its SmartAssist platform, introducing intelligent AI capabilities that automate repetitive work, generate insights, and help organizations work more efficiently. As AI becomes central to the company's product strategy, Smartsheet continues to build innovative tools that combine enterprise-grade security with cutting-edge AI technologies.
About the Position
The Senior Software Engineer II will play a key role in ensuring the quality and performance of Smartsheet's AI systems.
Unlike traditional software engineering roles, this position focuses heavily on evaluating and improving AI agents by identifying failure patterns, optimizing prompts, enhancing retrieval systems, and building robust evaluation frameworks.
The successful candidate will collaborate closely with Applied AI engineers, platform teams, and architects to create scalable systems that deliver accurate, reliable, and trustworthy AI experiences for enterprise customers.
Key Responsibilities
AI Quality Engineering
The successful candidate will help improve AI-generated responses by:
- Evaluating AI model performance.
- Identifying quality issues.
- Diagnosing failure patterns.
- Measuring response accuracy.
- Improving system reliability.
This work ensures customers receive consistent and dependable AI-powered experiences.
Prompt Engineering and Context Optimization
A major responsibility involves improving how AI models respond to user requests.
Responsibilities include:
- Designing effective prompts.
- Optimizing context engineering.
- Refining AI behavior.
- Improving response quality.
- Supporting prompt experimentation.
Strong prompt engineering skills are essential for maximizing model performance.
Retrieval-Augmented Generation (RAG)
The engineer will also contribute to improving retrieval systems by:
- Optimizing chunking strategies.
- Evaluating embedding models.
- Improving retrieval accuracy.
- Diagnosing search failures.
- Enhancing AI knowledge retrieval.
These improvements help AI systems deliver more relevant and accurate responses.
Evaluation Framework Development
The role includes building scalable AI evaluation systems.
Key activities include:
- Creating automated evaluation pipelines.
- Developing quality metrics.
- Building golden datasets.
- Implementing regression testing.
- Measuring AI performance over time.
Robust evaluation frameworks are critical for maintaining production-quality AI systems.
Cross-Functional Collaboration
The Senior Software Engineer II will work closely with:
- AI platform engineers.
- Machine learning specialists.
- Product managers.
- Software architects.
- Engineering leadership.
Collaboration across teams helps deliver innovative AI solutions while maintaining high engineering standards.
Required Qualifications
Candidates should typically possess:
- At least eight years of software engineering experience.
- Two or more years working with Large Language Models (LLMs).
- Strong Python programming skills.
- Experience with AI evaluation frameworks.
- Knowledge of RAG architectures.
- Strong analytical and problem-solving abilities.
- Excellent communication skills.
- Experience working with enterprise software systems.
Preferred Technical Skills
Applicants with the following experience may have a competitive advantage:
- Machine Learning Operations (MLOps).
- Databricks.
- MLflow.
- Kubernetes.
- Terraform.
- Infrastructure as Code.
- AI observability tools.
- Vector databases.
- Cloud platforms such as AWS or Google Cloud.
These technical capabilities support the development of scalable AI systems for enterprise environments.
Why Work at Smartsheet?
Smartsheet offers engineers the opportunity to work on some of today's most advanced enterprise AI technologies.
Employee benefits may include:
- Competitive salary.
- Comprehensive health benefits.
- Remote work flexibility.
- Paid holidays and flexible time off.
- Retirement savings support.
- Professional development programs.
- Learning resources and training opportunities.
- Paid parental leave.
- Monthly remote work stipend.
The company encourages innovation, collaboration, and continuous learning while giving engineers the freedom to solve challenging technical problems.
Career Growth Opportunities
Professionals who excel in this role can advance into positions such as:
- Staff Software Engineer.
- Principal AI Engineer.
- AI Platform Architect.
- Machine Learning Engineering Manager.
- Director of AI Engineering.
- Principal Solutions Architect.
As enterprise AI adoption continues to accelerate, professionals with expertise in LLM evaluation, AI quality engineering, and cloud-native development remain among the most sought-after technology specialists.
Why This Opportunity Stands Out
Artificial Intelligence is transforming every industry, and organizations increasingly need engineers who can ensure AI systems remain reliable, trustworthy, and production-ready.
Joining Smartsheet provides the opportunity to:
- Work on enterprise-scale AI products.
- Build production-ready AI systems.
- Solve complex engineering challenges.
- Collaborate with leading AI experts.
- Develop expertise in one of technology's fastest-growing fields.
For experienced software engineers interested in shaping the future of intelligent work management, this position offers an exceptional career opportunity.
How to Apply
Application Link:
Interested applicants should review the job requirements carefully and submit their applications as soon as possible. Remote AI engineering roles are highly competitive and often receive applications from experienced professionals worldwide.
Final Thoughts
The Senior Software Engineer II – Applied AI and Evaluations position at Smartsheet is an outstanding opportunity for experienced engineers looking to work at the cutting edge of Artificial Intelligence.
With responsibilities spanning AI evaluation, prompt engineering, cloud architecture, retrieval systems, and machine learning infrastructure, this role offers both technical challenges and significant career growth. If you're passionate about building reliable AI solutions that power the future of work, this remote opportunity is well worth exploring.
Comments
Post a Comment