SDK and Container Versions

Oct 13, 2025 by talynt.org 118 views

Describe the Bug

When using Vercel's AI SDK to run experiments on datasets through an experiment API, we have encountered an issue where traces don't link properly when the dataset size exceeds a certain threshold. This bug is particularly noticeable when the dataset size is increased from a small number of items (<=3) to a larger number of items (12). In such cases, about half of the traces fail to record or link properly.

To Reproduce

To reproduce this bug, we can run the code provided in the attached files. The code is an experiment API that uses the Vercel AI SDK to generate text based on a dataset. The experiment API iterates over the dataset items, creates a trace manually, and links the execution trace to the dataset item. However, when the dataset size is increased, the traces fail to link properly.

Code Snippet

The code snippet provided is the experiment API code that uses the Vercel AI SDK to generate text based on a dataset. The code is written in JavaScript and uses the Next.js framework. The code snippet is as follows:

export async function POST(request: Request) {
  try {
    const user = await currentUser()

    // Double-check authorization
    if (user?.privateMetadata?.adminAccess !== true) {
      return NextResponse.json(
        { error: 'Unauthorized access' },
        { status: 403 }
      );
    }

    const { rulesetId, datasetName, testName, promptOverride } = await request.json() as {
      rulesetId: number, 
      datasetName: string, 
      testName: string, 
      promptOverride?: string
    };

    // Get prompt from ruleset or use override
    const ruleset = await db.select().from(rulesets).where(eq(rulesets.id, rulesetId));
    const systemPromptName = promptOverride ?? ruleset?.[0]?.systemPromptName;
    if (!systemPromptName) {
      throw Error('Prompt not found');
    }
    const systemPrompt = await createSystemPrompt(systemPromptName, ruleset[0]?.courtName ?? "");

    const dataset = await langfuse.getDataset(datasetName);
 
    // iterate over the dataset items
    for (const item of dataset.items) {
      console.log(item.input)
     
      // create a trace manually in order to pass id to vercel ai sdk for later linking to the dataset run
      const trace = langfuse.trace({name: testName})
     
    
      const output = await generateText({
        model: openai('gpt-4o-mini'),
        temperature: 0.1,
        prompt: item.input as string,
        system: systemPrompt,
        maxSteps: 5,
        tools: {
          getRules: tool({
            description: `get relevant rules from the ruleset to answer questions.`,
            parameters: z.object({
              question: z.string().describe('the users question'),
            }),
            execute: async ({ question }) => findRelevantRules(question, rulesetId),
          }),
        },
        experimental_telemetry: {
          isEnabled: true,
          functionId: "rule-text",
          metadata: {
            langfuseTraceId: trace.id,
            langfuseUpdateParent: true, // Update the parent trace with execution results as the trace was created manually to enable linking
          },
        },
      });

      // link the execution trace to the dataset item and give it a run_name
      await item.link(trace, testName);
      console.log(output.response.messages[output.response.messages.length - 1]?.content)
      // optionally, evaluate the output to compare different runs more easily
      /*
      trace.score({
        name: "<score_name>",
        value: myEvalFunction(item.input, output, item.expectedOutput),
        comment: "This is a comment", // optional, useful to add reasoning
      });
      */
    }
     
    // Flush the langfuse client to ensure all data is sent to the server at the end of the experiment run
    await langfuse.flushAsync();
  
    return NextResponse.json({ success: true })
  } catch (error) {
    console.error('Error processing experiment:', error)
    return NextResponse.json(
      { error: 'Failed to process experiment' },
      { status: 500 }
    )
  }
}

SDK and Container Versions

Unfortunately, the SDK and container versions are not provided in the additional information section.

Additional Information

The additional information section is empty.

Are You Interested to Contribute a Fix for This Bug?

No, we are not interested to contribute a fix for this bug.

Conclusion

In conclusion, the bug is a critical issue that affects the proper linking of traces when running experiments on datasets using the Vercel SDK. The bug is particularly noticeable when the dataset size exceeds a certain threshold. To reproduce this bug, we can run the code provided in the attached files. The code is an experiment API that uses the Vercel AI SDK to generate text based on a dataset. The experiment API iterates over the dataset items, creates a trace manually, and links the execution trace to the dataset item. However, when the dataset size is increased, the traces fail to link properly.

Recommendations

To fix this bug, we recommend the following:

Investigate the issue: Investigate the issue to determine the root cause of the problem.
Update the SDK: Update the Vercel SDK to the latest version to ensure that the bug is fixed.
Test the code: Test the code thoroughly to ensure that the bug is fixed and that the code is working as expected.
Provide a fix: Provide a fix for the bug and make it available to the community.

Q&A

Q: What is the bug?

A: The bug is a critical issue that affects the proper linking of traces when running experiments on datasets using the Vercel SDK. The bug is particularly noticeable when the dataset size exceeds a certain threshold.

Q: What is the root cause of the bug?

A: Unfortunately, the root cause of the bug is not provided in the additional information section.

Q: How can I reproduce the bug?

A: To reproduce the bug, you can run the code provided in the attached files. The code is an experiment API that uses the Vercel AI SDK to generate text based on a dataset. The experiment API iterates over the dataset items, creates a trace manually, and links the execution trace to the dataset item. However, when the dataset size is increased, the traces fail to link properly.

Q: What are the symptoms of the bug?

A: The symptoms of the bug are as follows:

Traces don't link properly when the dataset size exceeds a certain threshold.
About half of the traces fail to record or link properly.

Q: How can I fix the bug?

A: To fix the bug, we recommend the following:

Investigate the issue: Investigate the issue to determine the root cause of the problem.
Update the SDK: Update the Vercel SDK to the latest version to ensure that the bug is fixed.
Test the code: Test the code thoroughly to ensure that the bug is fixed and that the code is working as expected.
Provide a fix: Provide a fix for the bug and make it available to the community.

Q: What are the benefits of fixing the bug?

A: The benefits of fixing the bug are as follows:

Proper linking of traces when running experiments on datasets using the Vercel SDK.
Improved accuracy and reliability of the experiment results.
Enhanced user experience and satisfaction.

Q: How can I get help with fixing the bug?

A: If you need help with fixing the bug, you can:

Contact the Vercel support team for assistance.
Reach out to the Vercel community for help and guidance.
Provide a detailed description of the bug and the steps to reproduce it.

Q: What are the next steps?

A: The next steps are to:

Investigate the issue and determine the root cause of the problem.
Update the Vercel SDK to the latest version.
Test the code thoroughly to ensure that the bug is fixed and that the code is working as expected.
Provide a fix for the bug and make it available to the community.

By following these steps, we can ensure that the bug is fixed and that the Vercel SDK is working as expected.