deepu-sasidharan

Deepu is a polyglot developer, Java Champion, and OSS aficionado. He mainly works with Java, JS, Rust, and Golang. He co-leads JHipster and created the JDL Studio and KDash. He's a Staff Developer Advocate at Okta. He is also an international speaker and a published author.

Staff Developer Advocate

Deepu K Sasidharan

Learn how to build a tool-calling AI agent using LangGraph, Vercel AI SDK, and Next.js. Integrate different kinds of tools in them. Use Google services like Gmail, Calendar, and Drive as tools secured using Auth0 Token Vault.

**AI agents** and **Operators** are the next big things in software development. In a [previous post](https://auth0.com/blog/genai-tool-calling-intro/), we discussed how AI agents are taking center stage by seamlessly integrating with multiple digital tools to be more efficient. In this post, we will be building a personal assistant that can use different types of tools. Both simple unauthenticated tools and more complex authenticated tools like Gmail, Calendar, and Google Drive.

If you haven't read the previous post, I recommend you do so before continuing to better understand tool calling in AI agents and how security is currently handled.

<include
  src="LinkCard"
  title="Tool Calling in AI Agents: Empowering Intelligent Automation Securely"
  link="https://auth0.com/blog/genai-tool-calling-intro/"
  description="Discover how AI agents are taking center stage by seamlessly integrating with multiple digital tools to be more efficient. Learn why it is important to secure them."
  img="https://images.ctfassets.net/23aumh6u8s0i/4LrNhImjPcdPVpmCIrhlG9/4efed8dfb26630e6cf48b4a478a1f550/genai-tool-calling-intro.png"
/>

To keep this post focused on tool calling, we will only focus on the AI agent part and not the UI. Each step of the tutorial, including building the UI, can be found as a distinct commit in the [GitHub repository](https://github.com/auth0-samples/auth0-assistant0).

## Technology Stack

We will use a [Next.js](https://nextjs.org/) application called **Assistant 0** as the base, Assistant 0 is an AI personal assistant that consolidates your digital life by dynamically accessing multiple tools to help you stay organized and efficient.

Apart from Next.js, we mainly use the following technologies:

- [LangGraph.js](https://langchain-ai.github.io/langgraphjs/) and [LangChain's JavaScript framework](https://js.langchain.com/docs/introduction/) for building agentic workflows.
- [LangChain community tools](https://js.langchain.com/docs/integrations/tools/).
- Vercel's [AI SDK](https://github.com/vercel-labs/ai) to stream response tokens to the client and display the incoming messages.
- The [Auth0 AI SDK](https://github.com/auth0-lab/auth0-ai-js) and [Auth0 Next.js SDK](https://github.com/auth0/nextjs-auth0) to secure the application and call third-party APIs.

## Prerequisites

> **You will need the following tools and services to build this application**:
>
> - [Bun v1.2](https://bun.sh/) or [NodeJS v20](https://nodejs.org/en/download)
> - An Auth for GenAI account. [Create one](https://a0.to/ai-content).
> - An OpenAI account and API key. [Create one](https://platform.openai.com) or use any [other LLM provider supported by LangChain](https://js.langchain.com/docs/integrations/llms/).
> - A SerpAPI account and API key. [Create one](https://serpapi.com/).
> - A Google account for Gmail, Calendar, and Drive. Preferably a new one for testing.

## Getting Started

First clone [the repository](https://github.com/auth0-samples/auth0-assistant0) and install the dependencies:

```bash
git clone https://github.com/auth0-samples/auth0-assistant0.git
cd auth0-assistant0
git switch step-1 # so that we skip building the UI and get straight to the AI agent

bun install # or npm install
```

Next, you'll need to set up environment variables in your repo's `.env.local` file. Copy the `.env.example` file to `.env.local`. To start you'll just need to add your OpenAI API key.

Now, start the development server:

```bash
bun dev # or npm run dev
```

Open `http://localhost:3000` with your browser to see the result! Ask the agent something, and you'll see a streamed response.

![Assistant 0](https://images.ctfassets.net/23aumh6u8s0i/ePzgV9PLYqmwdsqdJC2VU/3507e783ee8199b6a5599742ef81fed7/assistant-0.png)

### A Peek into the Code

The application is structured as below:

- `src/app`: Contains the Next.js application routes, layout, and pages.
- `src/app/page.tsx`: Home page with chat interface.
- `src/api/chat/route.ts`: API route for handling chat requests. This is where the AI agent is defined.
- `src/components`: UI components used in the application.
- `src/lib`: Services and configurations. This is where custom tools will be defined.
- `src/utils`: Utility functions

Let's take a look at the `src/api/chat/route.ts` file where the AI agent is defined.

```js
// src/api/chat/route.ts
// ...
const AGENT_SYSTEM_TEMPLATE = `You are a personal assistant named Assistant0. 
You are a helpful assistant that can answer questions and help with tasks. 
You have access to a set of tools, use the tools as needed to answer the user's question.`;

export async function POST(req: NextRequest) {
  // ... Process the request and get messages
  // Create a new OpenAI chat model
  const llm = new ChatOpenAI({ model: 'gpt-4', temperature: 0 });
  const tools = [];
  // Use a prebuilt LangGraph agent.
  const agent = createReactAgent({
    llm,
    tools,
    messageModifier: new SystemMessage(AGENT_SYSTEM_TEMPLATE),
  });
  // Stream back all generated tokens and steps from their runs.
  const eventStream = agent.streamEvents({ messages }, { version: 'v2' });
  // Log tool calling data. Only in development mode
  const transformedStream = logToolCallsInDevelopment(eventStream);
  // Adapt the LangChain stream to Vercel AI SDK Stream
  return LangChainAdapter.toDataStreamResponse(transformedStream);
}
```

The route uses a prebuilt LangGraph agent and OpenAI chat model. No tools are defined yet; we will add them in the next section.

Here is a high-level architecture of the application with the tools we will be adding in this post.

![Assistant 0 architecture](https://images.ctfassets.net/23aumh6u8s0i/6ZTtsivmBOPUmKGWx479Xm/d8d90c3bf3f24f1c8ce99ff703a1ba7d/architecture.jpg)

Now that the base application is running, let's start building our personal assistant with tool-calling capabilities.

## Add Tools to the Assistant

If you recall the previous post, we discussed authenticated and unauthenticated tools. Let's start with an unauthenticated tool and then proceed to add more complex tools like Gmail, Calendar, and Google Drive.

### Add a Calculator Tool

We will start with a simple calculator tool from LangChain community tools to perform basic arithmetic operations.

Install the `@langchain/community` package.

```bash
bun add @langchain/community # or npm i @langchain/community
```

Update the `src/api/chat/route.ts` file with the following code.

```js
// src/api/chat/route.ts
import { Calculator } from '@langchain/community/tools/calculator';
// ...
export async function POST(req: NextRequest) {
  // ...
  const tools = [new Calculator()];
  // ...
}
```

It's that simple! Now try asking the assistant to perform some calculations. For example, `what is (2+56) x 200`. You should see the answer in the UI. If you check the console where Next.js dev server (`bun dev`) is running, you should see the tool calling data like below.

```bash
Tool calls state: {
  "call_E3Za9klJcJ5TADWiMj8XfjdB": {
    "name": "calculator",
    "args": "{\n  \"input\": \"(2+56) * 200\"\n}"
  }
}
 POST /api/chat 200 in 8382ms
```

### Add a Web Search Tool

Now let's add a web search tool to search the web for information. Let's use [SerpAPI](https://serpapi.com/) for this. It's an authenticated tool. But since this is going to be API usage and not a user impersonation, we can simply use an API key.

First, get an API key from SerpAPI and add that to your `.env.local` file.

```bash
SERPAPI_API_KEY=your-api-key
```

Now, update the `src/api/chat/route.ts` file with the following code.

```js
// src/api/chat/route.ts
import { SerpAPI } from '@langchain/community/tools/serpapi';
// ...
export async function POST(req: NextRequest) {
  // ...
  const tools = [new Calculator(), new SerpAPI()];
  // ...
}
```

Again, quite simple! Now try asking the assistant to search the web for information. For example, `What is the top 10 news today?`. You will see the response in the UI.

![Assistant 0 web search](https://images.ctfassets.net/23aumh6u8s0i/h4hWDQwBBTUiDh6gdaVET/137647c87dd51c6cc34ec0a44d6e453d/assistant-0-web-search.png)

If you check the console, you should see the tool calling data like the one below.

```bash
Tool calls state: {
  "call_e8vqAPW45vMsUbFiIF9SW5Yn": {
    "name": "search",
    "args": "{\n  \"input\": \"top 10 news today\"\n}"
  }
}
 POST /api/chat 200 in 15443ms
```

This corresponds to the `step-2` branch in case you want to double-check your code.

```bash
git switch step-2
```

### Add Gmail Tools

Now, let's get into business. I know you are here for this part, not to learn about adding calculators and web search tools. But before we can add a Gmail tool, we need to add Auth0 authentication to the application since we will be using Auth0's [Token Vault](https://auth0.com/docs/secure/tokens/token-vault) feature to get access tokens to call the Gmail API via a tool.

#### Add Auth0 Authentication

First, let's add Auth0 as the authentication provider to the application.

Follow the [Auth0 Next.js quickstart guide](https://auth0.com/ai/docs/user-authentication) to set up the application.

> If you want to create the Auth0 application manually, create a new application (Type: **Regular Web App**, Callback URL: `http://localhost:3000/auth/callback`) and get the **client ID** and **client secret**.

Update the `.env.local` file with credentials, and create the `src/lib/auth0.ts` and `src/middleware.ts` files.

If you want to skip to the next step, check out the `step-3` branch. The [full changelog for this step is on GitHub](https://github.com/auth0-samples/auth0-assistant0/commit/9a6db1ec49511afa7c361080c3f5a1854ee8efe5).

```bash
git switch step-3
```

#### Configure Auth0 for Gmail

To use Gmail API with Auth0 via a Google Social Connection, you need to do the following:

- Create Google OAuth 2.0 credentials
- Configure Gmail social connection in Auth0

You can follow the [Google Sign-in and Authorization](https://auth0.com/ai/docs/google-sign-in-and-auth) guide from Auth0 to configure this.

#### Add Gmail Search Tool

We can add Gmail tools now that we have Auth0 authentication in place. Let's start with a Gmail search tool. This tool will allow the assistant to search for emails in the user's Gmail account.

First, we need to get an access token for the Gmail API. We can get this using the [Auth0 Token Vault](https://auth0.com/docs/secure/tokens/token-vault) feature.

Update `src/lib/auth0.ts` with the following code.

```js
// src/lib/auth0.ts
// ...
export const auth0 = new Auth0Client({
  // this is required to get federated access tokens from services like Google
  authorizationParameters: {
    access_type: 'offline',
    prompt: 'consent',
  },
});

export const getGoogleAccessToken = async () => {
  const { token } = await auth0.getAccessTokenForConnection({
    connection: 'google-oauth2',
  });
  return token;
};
```

Install the `googleapis` package.

```bash
bun add googleapis # or npm i googleapis
```

Import the `GmailSearch` tool from LangChain community tools and update the `src/api/chat/route.ts` file with the following code.

```js
// src/api/chat/route.ts
import { GmailSearch } from '@langchain/community/tools/gmail';
import { getGoogleAccessToken } from '@/lib/auth0';
// ...
export async function POST(req: NextRequest) {
  // ...
  // Update the LLM model to gpt-4o-mini to support a larger context window
  const llm = new ChatOpenAI({
    model: 'gpt-4o-mini',
    temperature: 0,
  });
  // Get the access token via Auth0
  const accessToken = await getGoogleAccessToken();
  // Provide the access token to the Gmail tools
  const gmailParams = {
    credentials: { accessToken },
  };
  const tools = [new Calculator(), new SerpAPI(), new GmailSearch(gmailParams)];
  // ...
}
```

We are all set! Now try asking the assistant to search for emails. For example, `What is the latest unread email?`. You should see the response in the UI.

![Assistant 0](https://images.ctfassets.net/23aumh6u8s0i/14n2drq2woolLeJybBMWk6/3bbc575f92493dc2322f485a6b4f4ac9/assistant-0-gmail-search.png)

If you check the console, you should see the tool calling data like the one below.

```bash
Tool calls state: {
  "call_rCzcZRjueyD8p2kzTmeZMiSK": {
    "name": "search_gmail",
    "args": "{\"query\":\"is:unread\",\"maxResults\":1,\"resource\":\"messages\"}"
  }
}
 POST /api/chat 200 in 10892ms
```

#### Add Gmail Draft Tool

We will add a Gmail draft tool next. This tool will allow the assistant to draft emails on behalf of the user.

Update the `src/api/chat/route.ts` file with the following code.

```js
// src/api/chat/route.ts
import { GmailDraft } from '@langchain/community/tools/gmail';
// ...
export async function POST(req: NextRequest) {
  // ...
  const tools = [
    new Calculator(),
    new SerpAPI(),
    new GmailSearch(gmailParams),
    new GmailDraft(gmailParams),
  ];
  // ...
}
```

Now, ask the assistant to draft an email. For example, `Draft an email to John Doe about the meeting tomorrow.` You should see the response in the UI.

![Assistant 0](https://images.ctfassets.net/23aumh6u8s0i/hWz4f6iUXTacbCNoqxwLz/c16c70d786027012b4c4944639339999/assistant-0-gmail-draft.png)

If you check the console, you should see the tool calling data like the one below.

```bash
Tool calls state: {
 "call_pPAk6rFYJSSyAuYkNWFf5KYX": {
   "name": "create_gmail_draft",
   "args": "{\"message\":\"Hi John,\\n\\nI hope this message finds you well. I wanted to … [Your Name]\",\"to\":[\"john.doe@example.com\"],\"subject\":\"Meeting Reminder\",\"cc\":[],\"bcc\":[]}"
 }
}
 POST /api/chat 200 in 7594ms
```

You can find the [full changelog of these two steps on GitHub](https://github.com/auth0-samples/auth0-assistant0/commit/ae2dd28b2fd2248d1d9ca76bd77269ea75817379).

This corresponds to the `step-4` branch in case you want to double-check your code.

```bash
git switch step-4
```

## Learn more about AI Agents and Auth for GenAI

You have successfully built an AI personal assistant that can search the web, search your emails, and draft emails. In the next chapter, we will add more tools like User Info, Google Calendar, Google Drive, and Slack to the assistant.

This tutorial is part of a series of posts on tool-calling Agents with [Auth for GenAI](https://a0.to/ai-content), we learned how to add tools to an AI agent and how to secure them using Auth0.

Sign up for [Auth for GenAI in Developer Preview](https://a0.to/ai-content).

Before you go, we have some great news to share: we are working on more content and sample apps in collaboration with amazing GenAI frameworks like [LlamaIndex](https://www.llamaindex.ai/), [LangChain](https://www.langchain.com/), [CrewAI](https://www.crewai.com/), [Vercel AI](https://vercel.com/ai), and [GenKit](https://firebase.google.com/docs/genkit). [Auth for GenAI](https://auth0.com/blog/introducing-auth-for-genai-launching-identity-for-ai-agents/) is our new product to help you protect your user's information in GenAI-powered applications. Make sure to join the [Auth0 Lab Discord server](http://a0.to/auth0-lab-discord) to hear more and ask questions.


Build an AI Assistant with LangGraph, Vercel, and Next.js: Use Gmail as a Tool Securely

Discover how AI agents are taking center stage by seamlessly integrating with multiple digital tools like Gmail, Calendar, Slack, and Google Drive to be more efficient. Learn why it is important to secure them and how to do it.

**AI agents** or **Operators** are the latest trend in AI. At their core, agents are autonomous systems that not only process natural language using Large Language Models (LLMs) but also decide when and how to act by interfacing with external tools and APIs. These smart entities blend machine learning with decision-making capabilities, enabling them to perform complex tasks, interact in real-time, use the most efficient tools to perform certain tasks, and even learn from their environment—all without constant human intervention.

<AmpContent>
<amp-youtube
    data-videoid="r7wEinUG0ns"
    layout="responsive"
    width="480" height="270">
</amp-youtube>
</AmpContent>

<NonAmpContent>
<div
  className='embed-container'
  style="position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden; max-width: 100%;margin-bottom:40px;">
    <iframe
      style="position: absolute; top: 0; left: 0; width: 100%; height: 100%;"
      src='https://www.youtube.com/embed/r7wEinUG0ns'
      frameborder='0' allowfullscreen>
    </iframe>
</div>
</NonAmpContent>

## Tool Calling Agents

LLMs on their own are still susceptible to hallucinations and mistakes, but the ability to call tools opens up a whole new world of possibilities where error-prone tasks can be offloaded to a more traditional non-AI tool, like using a calculator to do math rather than relying on the LLM for it. Similarly, tool calling allows AI agents to interface with other applications and services, like Gmail, Calendar, Slack, and Google Drive, to do more for you than just answering questions.

Tool calling can be generally classified into these:

1. **Unauthenticated Tools:** These are tools that don't require authentication and are generally simpler to configure, like a calculator, a weather API, or a unit converter.
2. **Authenticated Tools:** These are tools that require authentication and are generally complex to configure, such as an API provided by another internal application or services like Gmail, Calendar, Slack, and Google Drive.

In an enterprise environment, the authenticated tools can be further classified into:

1. **First-Party Tools:** These are services that are part of the same infrastructure and require authentication, like an API provided by another internal application or a microservice.
2. **Third-Party Tools:** These are external services that require more complex authentication. They could be services that are authenticated using service accounts or API keys, like a payment API, or services that need to be authenticated by the end user, such as Gmail, Calendar, Slack, and Google Drive.

Tool calling provides many key advantages to an AI agent:

- **Autonomous Decision-Making:** AI agents analyze context and determine the next best action, reducing the need for manual oversight.
- **Optimized Task Execution:** They can connect to various services and APIs, enabling them to perform specialized tasks like data retrieval, scheduling, and communication.
- **Scalability and Adaptability:** As systems evolve, agents can easily incorporate new functionalities, adapting to changes in workflow or priorities.
- **Improved User Experience:** Their ability to tackle diverse tasks leads to more personalized and efficient interactions, making technology feel both intuitive and proactive.

## Tool Calling Agent: A Conceptual Example

Imagine an AI personal assistant that consolidates your digital life by dynamically accessing multiple tools to help you stay organized and efficient. Here’s how it could work:

1. **Gmail Integration:** The assistant regularly scans your inbox to generate concise summaries. It highlights urgent emails, categorizes conversations by importance, and even suggests drafts for quick replies.
2. **Calendar Management:** By interfacing with your calendar, it can remind you of upcoming meetings, check for scheduling conflicts, and even propose the best time slots for new appointments based on your availability.
3. **Slack Notifications:** For team communications, the assistant monitors Slack channels. It identifies key messages and creates action items, ensuring you never miss an important update from your colleagues.
4. **Google Drive Access:** Whether you need immediate access to the latest project document or a file related to a current task, the assistant retrieves pertinent documents from Google Drive on demand. It can create document summaries and even create new documents based on your instructions.

With tool-calling capabilities, the possibilities are endless. In this conceptual scenario, the AI agent embodies a digital personal secretary—one that not only processes information but also proactively collates data from connected services to provide comprehensive task management. This level of integration not only enhances efficiency but also ushers in a new era of intelligent automation, where digital assistants serve as reliable, all-in-one solutions that tailor themselves to your personal and professional needs.

## Security Challenges with Tool Calling AI Agents

Building such an assistant is not that difficult. Thanks to frameworks like [LangChain](https://www.langchain.com/), [LlamaIndex](https://www.llamaindex.ai/), and [Vercel AI](https://vercel.com/ai), you can get started quickly. The difficult part is doing it securely so that you can protect the user's data and credentials.

Many current solutions involve storing credentials and secrets in the AI agent application’s environment or letting the agent impersonate the user. Storing credentials securely in the environment might work for APIs that require API keys or service accounts. However, this is not a good idea for cases where the end users need to authenticate to the service, as it can lead to security vulnerabilities and excessive scope and access for the AI agent.

<br>

<div class="alert alert-info alert-icon">
  <i class="icon-budicon-487 icon-info"></i>
  <b>To learn more about other security concerns and vulnerabilities in AI check out this blog post on <a href="https://auth0.com/blog/identity-challenges-for-ai-powered-apps/">Identity Challenges for AI-Powered Applications</a>.</b><br> 
</div>

When you build an AI agent, you need to consider all its other security implications as well. You don't want an LLM to have unlimited access to your personal data like email and documents, and more importantly, you don't want to provide your credentials to the LLM to access these tools. Let's face it, regardless of how secure the LLM is, there is always a possibility of it getting manipulated into divulging sensitive information or doing something you don't want it to do.

## Tool Calling with the Help of Auth0

This is where Auth0 comes to the rescue. As the leading identity provider (IdP) for modern applications, our upcoming product, [Auth for GenAI](https://auth0.com/blog/auth-for-genai/), provides standardized ways built on top of OAuth and OpenID Connect to call APIs of tools on behalf of the end user from your AI agent.

Auth0 brokers a secure and controlled handshake between the AI agents and the services you want the agent to interact with on your behalf – in the form of scoped access tokens. This way, the agent and LLM do not have access to the credentials and can only call the tools with the permissions you have defined in Auth0. This also means your AI agent only needs to talk to Auth0 for authentication and not the tools directly, making integrations easier.

### Call first-party APIs on users' behalf

When your AI agent, secured with Auth0, needs to call another application also secured with Auth0, you can use standard OAuth 2.0 flows, like the [Authorization Code Flow](https://auth0.com/docs/get-started/authentication-and-authorization-flow/device-authorization-flow), to get an API token from that application with user consent. In this case, Auth0 will manage the retrieval of new access tokens using cached refresh tokens when the API token expires.

### Call third-party APIs on users' behalf

When your AI agent, secured with Auth0, wants to call external services like Gmail, Calendar, Slack, and Google Drive, Auth0 can help the agent get access tokens for the external service on behalf of the end user. In this case, Auth0 brokers the API access token from the external service to the AI agent.

![Tool calling with Federated API token exchange](https://images.ctfassets.net/23aumh6u8s0i/1gY1jvDgZHSfRloc4qVumu/d44bb7102c1e858e5ac64dea324478fe/tool-calling-with-federated-api-token-exchange.jpg)

This is made possible by Federated API token exchange, which is a way to obtain an access token from an external identity provider without the need for the user to re-authenticate every time. The end user authenticates and connects the external service once, and Auth0 intermediates the authentication and authorization process and provides the API access token to the AI agent. Auth0 takes care of storing refresh tokens and getting new access tokens when the current access token expires.

## Security Best Practices for Tool Calling AI Agents

Here is a summary of security best practices to consider when building a tool-calling AI agent:

- **Least Privilege**: Grant the minimum permissions needed (e.g., read-only access).
- **Audit Logs**: Track API calls and credential usage.
- **Proxying**: Use an IdP like Auth0 as a broker between the AI agent and the tools to protect the user's credentials.
- **Encryption**: Encrypt credentials at rest and in transit (HTTPS/TLS).
- **Token Rotation**: Regularly rotate API keys and tokens.
- **Sandboxing**: Run tools in isolated environments when possible to limit damage from leaks.

## Learn More about AI Agents and Auth for GenAI

This post offers a glimpse into the transformative potential of AI agents and highlights how secure and intelligent tool calling can empower AI agents to deliver a more connected and personalized experience. Stay tuned for a series of posts on tool-calling Agents with [Auth for GenAI](https://a0.to/ai-content), where we will build the conceptual personal assistant from the previous section in Node.js and Python, showing how to integrate with different types of services securely using your favorite AI frameworks and Auth0.

Before you go, we have some great news to share: we are working on more content and sample apps in collaboration with amazing GenAI frameworks like [LlamaIndex](https://www.llamaindex.ai/), [LangChain](https://www.langchain.com/), [CrewAI](https://www.crewai.com/), [Vercel AI](https://vercel.com/ai), and [GenKit](https://firebase.google.com/docs/genkit). [Auth for GenAI](https://auth0.com/blog/auth-for-genai/) is our upcoming product to help you protect your user's information in GenAI-powered applications. Make sure to join the [Auth0 Lab Discord server](http://a0.to/auth0-lab-discord) to hear more and ask questions.

"tool-calling"