Building a Custom Orchestrator for AWS Nova Pro
The article discusses the challenges the author faced when using AWS Bedrock Agents and Action Groups with the AWS Nova Pro model for a healthcare prior authorization platform. They encountered issues with the agent framework's internal timeouts and ultimately built a custom orchestrator using the Bedrock Converse API.
Why it matters
This article provides valuable insights into the challenges of integrating large language models like AWS Nova Pro into complex enterprise applications and the importance of building custom solutions when off-the-shelf frameworks fall short.
Key Points
- 1The team wanted to use AWS Nova Pro as the foundation model for their AI assistant feature due to its scalability and lack of throttling issues.
- 2They encountered problems with the Bedrock Agents framework, where the agent would not reliably call the necessary tools and instead prefer to have a conversational response.
- 3Switching to the Nova Pro model with Bedrock Agents resulted in a 'dependencyFailedException' due to the model not generating responses fast enough within the agent framework's internal timeout.
- 4The team decided to build a custom orchestrator using the Bedrock Converse API, which allowed them to use the Nova Pro model without the issues they faced with the Agents framework.
Details
The article describes the team's experience building a healthcare prior authorization platform that includes an AI assistant feature. They initially chose to use the AWS Nova Pro model as the foundation for this feature, as it is AWS's own model and offers better scalability and fewer throttling issues compared to third-party models on Bedrock. However, they encountered problems when trying to use the Bedrock Agents and Action Groups framework to orchestrate the various tools the assistant needed to call, such as retrieving patient information, generating appeal letters, and checking eligibility. Despite numerous iterations of the agent prompt and configuration, the agent would not reliably call the necessary tools and instead prefer to have a conversational response. When they tried switching to the Nova Pro model with the Bedrock Agents, they encountered a 'dependencyFailedException' due to the model not generating responses fast enough within the agent framework's internal timeout. After researching the issue, the team decided to build a custom orchestrator using the Bedrock Converse API, which allowed them to use the Nova Pro model without the problems they faced with the Agents framework.
No comments yet
Be the first to comment