Overview
Government services touch people’s lives everyday. Getting information about those services shouldn’t mean waiting on hold or figuring out complex systems. That’s why Last Call Media partnered with Massachusetts to begin an exciting journey: enhancing its chatbot, Ask MA, with advanced AI capabilities.
We’re piloting next-generation features that understand and answer questions the way people actually ask them. The pilot initially focused on the Department of Early Education and Care (EEC), helping to create a blueprint for other state agencies.
Our challenge was clear: leverage cutting-edge AI responsibly while maintaining a consistent user experience. How did we do it? By combining human-centered design principles with sophisticated cloud technology (AWS services like Lex, Bedrock, Lambda, and CloudWatch). We also pioneered an innovative approach to testing: using AI to test AI.
About the Client
We worked directly with the Executive Office of Technology Services and Security (EOTSS), which drives high-impact digital initiatives across 125+ state agencies in Massachusetts. It aims to make every online interaction with government easier, faster and more secure.
The pilot project initially focused on helping families access information on childcare financial assistance through the Department of Early Education and Care (EEC), with potential to expand similar services across other state agencies.
Goals
Service Enhancement
Although we were dealing with complex technology, our goals for service delivery were simple:
- Make it easier for families to find information about childcare assistance
- Automate routine questions to help EEC staff focus on complex cases
- Create patterns that other agencies can follow
Technical Evolution
To build a next-generation government service, we aimed to balance innovation with reliability:
- Upgrade from basic NLP-based Q&A to an advanced AI (LLM-powered) system that understands questions
- Create a secure cloud architecture that can grow with the state’s needs
- Develop new ways to test AI systems at scale
- Build foundations that other agencies can use
Challenges
System Architecture
Building government AI systems brings unique challenges:
- Design AWS architecture that handles unpredictable usage patterns
- Optimize for a balance of speed and accuracy
- Implement government-grade security for sensitive information
- Create flexible systems that other departments can adopt
- Plan for growth as more agencies join
AI Implementation
Teaching AI to handle government services requires careful planning:
- Develop comprehensive testing based on how real people ask questions
- Ensure consistent, accurate responses about complex topics
- Build appropriate guardrails for AI behavior
- Balance automation with human oversight
Content Management
Keeping government information accurate and accessible creates distinct challenges:
- Establish sustainable content synchronization between the website and chatbot
- Balance manual content optimization with automated updates
- Keep data fresh while preserving improvements to responses
Experience Optimization
Making advanced technology work for real people means we need to:
- Ensure the chatbot can understand the wide variety of ways real people ask questions
- Find effective ways to tell users about the chatbot’s capabilities and limitations
- Work within government design guidelines
Solutions
Cloud-Native AI Architecture
We moved from basic Q&A (NLP) to advanced AI LLM technology that actually understands context. Here’s what powers it:
- Natural language processing: Using AWS Lex, the chatbot understands questions even when they’re typed conversationally or with typos
- Knowledge management: Bedrock helps organize and access information from across government sources
- Constant monitoring: CloudWatch helps us catch and fix issues fast
- Custom development: We built Python-based Lambda functions to optimize how the chatbot processes and formats responses
- Scalable deployment: We used AWS services for scalability, uptime, and easy maintenance.
What makes this special?
The system uses retrieval augmented generation through AWS Lex’s custom prompt feature. In plain terms, this helps the chatbot quickly find and combine relevant information from various government sources to provide accurate answers. This approach ensures responses are grounded in verified government information rather than generated text—a crucial safeguard for providing reliable service.
Innovative Testing Framework: Using AI to test AI
We pioneered a new approach: using AI to test AI. Here’s why it matters:
- Automated regression testing using secondary LLM evaluation (a second AI system checks if responses are helpful and accurate)
- Real-world scenarios ensure the chatbot handles actual user questions
- Automatic alerts if performance drops below 80%
- Continuous monitoring through CloudWatch spots problems early
AI Testing AI: Measuring Response Quality. Our automated testing system evaluates how well the chatbot handles user frustration and stays on topic.
Human-Centered Design
We created a comprehensive testing guide based on real user patterns:
- Analyzed real user inputs, highlighting typos and informal language
- Mapped common language variations
- Considered different technical comfort levels
- Built testing data from user feedback, journey mapping, staff interviews and search term analysis
Category | Program’s Documentation | Real examples of how users have asked this question |
|---|---|---|
| Coverage | What kind of child care is covered? | “about EEC” ‘What kind of childcare is covered?” |
| Determining eligibility | What are the eligibility requirements? | “me I hopefully will be eligible for” “I don’t have any income available” |
| CCFA application process | How can I apply for CCFA? What documents do I need? | “I don’t see the apply” “Moving from out the state” “Which document” “What counts as household on the application?” |
Real questions, real language: Examples of how users naturally ask questions versus formal documentation.
We also developed a Mayflower-integrated disclaimer component that:
- Clearly identifies the system as an AI chatbot
- Informs users when features are in testing phase
- Sets clear expectations about the chatbot’s capabilities
Clear User Communication: The chatbot identifies itself and sets appropriate expectations during testing.
What makes this special?
Being transparent about AI technology builds trust with residents. By immediately letting users know they’re interacting with an AI system — and what that means for their experience — they can make informed decisions about how to get the help they need. This transparency, combined with clear pathways to human support when needed, reflects our commitment to responsible AI implementation.
Results
We’re still gathering long-term data, but early results show plenty of promise:
Technical Wins
- Successfully deployed advanced AI in government
- Achieved fast response times
- Established reliable testing systems to ensure accuracy and appropriateness
- Built architecture ready for growth
Service Improvements
- Families find information more easily
- EEC staff handle fewer basic questions
- Clear paths to human support when needed
- Other states show strong interest
Impact
This project demonstrates how sophisticated technology can make government more accessible. By combining advanced AI capabilities with thoughtful UX design, we’ve created a system that not only works well technically but truly serves residents’ needs.
The success of this pilot has positioned Massachusetts as a leader in government AI implementation, providing a blueprint for other agencies and states to follow. Most importantly, we’ve shown how cutting-edge technology can be implemented responsibly in government services, setting new standards for digital constituent service delivery.
Want to learn more? Discover how Last Call Media’s human-centered approach to technology helps government agencies build better digital experiences for their constituents. Explore our approach.
On this Page
Services
- User Experience
- Development
- Design
- DevOps