- Voice Agents - Talk to a voice agent about information stored in Box.
- Box integration - Search and knowledge retrieval from documents stored in Box.
- GitHub integration - Report issues by talking to a voice agent.

Prerequisites
Before starting, ensure you have:- Sign up and install the
autonomycommand. - A Box developer account with API credentials.
- A GitHub personal access token.
- Docker running on your machine.
Project Structure
File Structure:
Step 1: Clone the Repository
Step 2: Configure Box Credentials
Create a Box application in the Box Developer Console:- Create a new Custom App.
- Select Server Authentication (Client Credentials Grant).
- Under Configuration, note your:
- Client ID.
- Client Secret.
- Enterprise ID.
secrets.yaml:
secrets.yaml
Step 3: Configure GitHub Access
Create a GitHub Personal Access Token withrepo scope to allow issue creation.
Add the token and target repository to your secrets.yaml:
Step 4: Upload Documents to Box
The application searches documents stored in a Box folder. Use the included script to populate Box with sample documentation:- Fetches documentation from
autonomy.computer/docs/llms.txt. - Parses all markdown file URLs.
- Creates a
docsfolder in Box. - Uploads all documentation files.
Step 5: Understand the Application Code
The Main Application
The application creates a voice-enabled agent with access to a knowledge base and GitHub tools:images/main/main.py
Box Integration
The Box client handles authentication and document retrieval:images/main/box.py
GitHub Issue Tool
The GitHub tool allows the agent to create issues based on user requests:images/main/github.py
Agent Instructions
The agent has two sets of instructions - one for the primary agent and one for the voice interface:Step 6: Deploy the Application
Deploy to Autonomy Computer:autonomy.yaml defines the infrastructure:
autonomy.yaml
The
size: big setting allocates more resources for the embedding model and voice processing.Step 7: Access the Voice Interface
Once deployed, open your zone URL in a browser:Using the Application
Voice Commands
Try these voice interactions:- “What is Autonomy?” - Searches the knowledge base and responds.
- “How do I create an agent?” - Retrieves relevant documentation.
- “I found a bug, help me report it” - Creates a GitHub issue.
- “Can you file a feature request for better logging?” - Creates a GitHub issue.
API Access
You can also interact via HTTP:Refresh Knowledge Base
The knowledge base automatically refreshes every hour. To manually refresh:How It Works
Document Loading
When the application starts:- Connects to Box using CCG authentication.
- Navigates to the configured folder path (
docs). - Recursively lists all files in the folder.
- Downloads each file’s text content.
- Chunks documents and generates embeddings.
- Stores embeddings in the knowledge base.
Voice Flow
When a user speaks:- Browser captures audio via Web Audio API.
- Audio streams to the agent via WebSocket.
- Voice Activity Detection (VAD) detects speech boundaries.
- Speech is transcribed and sent to the voice agent.
- Voice agent delegates to the primary agent.
- Primary agent searches knowledge and/or creates issues.
- Response is synthesized to speech.
- Audio streams back to the browser.
Knowledge Search
When searching documents:- Query is embedded using Cohere’s embed-english-v3.
- Vector similarity search finds relevant chunks.
- Top 5 results within distance threshold (0.4) are returned.
- Agent uses retrieved context to answer.
Configuration Options
Voice Settings
Customize voice behavior inmain.py:
Knowledge Settings
Tune document search:Environment Variables
| Variable | Description |
|---|---|
BOX_CLIENT_ID | Box OAuth client ID |
BOX_CLIENT_SECRET | Box OAuth client secret |
BOX_ENTERPRISE_ID | Box enterprise ID |
BOX_FOLDER_PATH | Path to documents folder in Box |
MAX_DOCUMENTS | Limit documents loaded (0 = all) |
GITHUB_TOKEN | GitHub personal access token |
GITHUB_REPO | Target repository (owner/repo) |
Build with a coding agent
See the guide on building Autonomy apps using coding agents.Troubleshooting
Box authentication fails
Box authentication fails
Verify your credentials in
secrets.yaml. Ensure your Box app uses Server Authentication (Client Credentials Grant) and has the necessary scopes enabled.No documents loaded
No documents loaded
Check that
BOX_FOLDER_PATH matches your Box folder name. View logs with autonomy zone inlet --to logs to see which folders are found.GitHub issue creation fails
GitHub issue creation fails
Verify your GitHub token has
repo scope and the repository format is owner/repo.Voice not working
Voice not working
Ensure your browser has microphone permissions. Use Chrome or Edge for best WebSocket and Web Audio API support.

