fix(analytics): Capture token usage and model name for Langfuse, LangSmith, and other providers (fixes #5763)

[TravisP-Greener]

Summary

Fixes #5763. Analytics integrations (Langfuse, LangSmith, Lunary, LangWatch, Arize, Phoenix, Opik) were not receiving token usage or model name for LLM generations. Only the raw text output was sent, so dashboards showed no token counts and cost tracking could not work.

This PR updates the custom analytics handler so that when an LLM run finishes, we pass structured output (including usage and model) to all providers and ensure Langfuse receives and flushes the update before the request completes.

---

Problem

• Observed: In Langfuse (and similarly in other providers), traces showed up but:
  • Token usage (prompt/completion/total) was missing.
  • Model name was missing.
  • Cost could not be calculated.
• Cause: The AnalyticHandler.onLLMEnd() method had the signature onLLMEnd(returnIds, output: string). All call sites passed only the final text (e.g. finalResponse), so the rich metadata on the LLM response (usage_metadata, response_metadata) was never sent to analytics.

---

Root cause

• LLM responses (e.g. AIMessage / AIMessageChunk) expose:
  • Usage: usage_metadata.input_tokens, output_tokens, total_tokens (or prompt_tokens, completion_tokens, total_tokens).
  • Model: response_metadata.model, model_name, or modelId.
• This metadata was available in the nodes (LLM, Agent, ConditionAgent) but was discarded before calling analyticHandlers.onLLMEnd(...), so providers only ever received a string.

---

Solution

1. Handler: accept structured output and forward usage/model

File: packages/components/src/handler.ts

• Signature: onLLMEnd(returnIds, output: string | ICommonObject) so we accept either a string (backward compatible) or an object with:
  • content (or we treat the whole value as content when it’s a string).
  • usageMetadata / usage_metadata (token counts).
  • responseMetadata / response_metadata (model name).
• Extraction: Normalize token fields (support both input_tokens/output_tokens and prompt_tokens/completion_tokens) and model (from model, model_name, or modelId).
• Per provider:
  • LangSmith: llmRun.end({ outputs: { generations, llm_output: { token_usage, model_name } } }).
  • Langfuse: generation.end({ output, model, usage: { promptTokens, completionTokens, totalTokens } }), then await langfuse.flushAsync() so the update is sent before the request ends.
  • Lunary: trackEvent('llm', 'end', { output, tokensUsage, model }).
  • LangWatch: span.end({ output, metrics, model }).
  • Arize / Phoenix / Opik: Set span attributes for llm.token_count.prompt, llm.token_count.completion, llm.token_count.total, and llm.model_name.
• Backward compatibility: If output is a string, we use it as content only and do not add usage/model (e.g. OpenAIAssistant and any other callers that only pass text keep working).

2. Call sites: pass structured output where metadata exists

• packages/components/nodes/agentflow/LLM/LLM.ts
  Pass the full output object from prepareOutputObject() (which already sets content, usageMetadata, responseMetadata) to onLLMEnd instead of finalResponse.

• packages/components/nodes/agentflow/Agent/Agent.ts
  Same: pass the full output from prepareOutputObject() to onLLMEnd.

• packages/components/nodes/agentflow/ConditionAgent/ConditionAgent.ts
  Build an analyticsOutput with content, usageMetadata (from response.usage_metadata), and responseMetadata (from response.response_metadata) and pass it to onLLMEnd.

• packages/components/nodes/agents/OpenAIAssistant/OpenAIAssistant.ts
  No change. The Assistants API does not expose token usage in the same way; these call sites continue to pass a string, which remains valid with the new signature.

3. Unit tests

File: packages/components/src/handler.test.ts

• Tests added for the extraction logic used in onLLMEnd: string vs object, token field normalization (LangChain vs OpenAI naming), model name from response_metadata.model / model_name / modelId, and missing/partial fields.
• (Execution of these tests may be affected by existing Jest/ESM setup; the logic under test is that used in the handler.)

---

Why Langfuse needed flushAsync()

The Langfuse JS SDK queues events and flushes in the background. Without an explicit flush after generation.end(), the update (output, model, usage) could still be in the queue when the HTTP response finished, so Langfuse sometimes received only the generation start, not the end. Calling await langfuse.flushAsync() after generation.end() ensures the generation update is sent before the request completes, so traces in Langfuse show token usage and model.

---

Testing

• Build: pnpm build completes with no TypeScript errors.
• Manual: Flowise run with an Agentflow using ChatOpenAI; LangSmith and Langfuse both showed token usage and model on the LLM generation; Langfuse cost/usage fields populated as expected.
• Backward compatibility: Call sites that still pass a string (e.g. OpenAIAssistant) continue to work; no changes required for them.

---

Checklist

• Fixes Analytics integration (LangSmith/Langfuse) not capturing token usage or model name from OpenAI #5763
• Backward compatible: onLLMEnd accepts both string and object
• All analytics providers that support usage/model receive it (LangSmith, Langfuse, Lunary, LangWatch, Arize, Phoenix, Opik)
• No new debug logs or console noise
• pnpm build passes

[gemini-code-assist]

Summary of Changes

Hello @TravisP-Greener, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request resolves a critical issue where analytics integrations were failing to capture essential token usage and model name information for LLM generations. By upgrading the analytics handler to accept and intelligently process structured output, and subsequently updating all relevant call sites and provider integrations, the system now ensures that platforms like Langfuse and LangSmith receive complete metadata. This enhancement significantly improves the accuracy of cost tracking and enables more detailed performance analysis for LLM operations.

Highlights

• Enhanced Analytics Handler: The AnalyticHandler.onLLMEnd method was updated to accept a structured object containing content, usage metadata, and response metadata, while maintaining backward compatibility for string inputs.
• Metadata Extraction and Normalization: Robust logic was implemented to extract and normalize token usage (supporting both input_tokens/output_tokens and prompt_tokens/completion_tokens) and model names (from model, model_name, or modelId) from the structured output.
• Provider Integration Updates: Integrations for LangSmith, Langfuse, Lunary, LangWatch, Arize, Phoenix, and Opik were modified to correctly receive and process the newly available token usage and model name data, enabling richer analytics.
• Langfuse Flush Mechanism: An explicit await langfuse.flushAsync() call was added after generation.end() for Langfuse, ensuring that usage and model updates are reliably sent before the HTTP request completes.
• Call Site Modifications: Call sites in Agent, ConditionAgent, and LLM nodes were updated to pass the full structured output object to onLLMEnd, providing comprehensive metadata to analytics providers.
• New Unit Tests: Comprehensive unit tests were introduced for the onLLMEnd metadata extraction logic, covering various input formats and field name variations to ensure correctness and reliability.

Changelog

• packages/components/nodes/agentflow/Agent/Agent.ts
  • Updated the onLLMEnd call to pass the complete output object instead of just finalResponse.
• packages/components/nodes/agentflow/ConditionAgent/ConditionAgent.ts
  • Constructed an analyticsOutput object that includes content, usageMetadata, and responseMetadata from the LLM response.
  • Modified the onLLMEnd call to pass this newly constructed analyticsOutput object.
• packages/components/nodes/agentflow/LLM/LLM.ts
  • Updated the onLLMEnd call to pass the complete output object instead of just finalResponse.
• packages/components/src/handler.test.ts
  • Added new unit tests to verify the onLLMEnd usage metadata extraction logic, covering string inputs, structured inputs with various usage and response metadata formats, and model name extraction priorities.
• packages/components/src/handler.ts
  • Modified the onLLMEnd method signature to accept string | ICommonObject for backward compatibility and structured data.
  • Implemented logic to extract outputText, usageMetadata, and modelName from the output parameter, normalizing token field names.
  • Updated the LangSmith integration to include token_usage and model_name in llm_output.
  • Enhanced the Langfuse integration to include model and usage (with promptTokens, completionTokens, totalTokens) in the generation.end payload and added await langfuse.flushAsync().
  • Modified the Lunary integration to pass tokensUsage and model in the trackEvent data.
  • Updated the LangWatch integration to include metrics (prompt/completion tokens) and model in the span.end payload.
  • Adjusted Arize, Phoenix, and Opik integrations to set llm.token_count.prompt, llm.token_count.completion, llm.token_count.total, and llm.model_name attributes on the LLM span.

Activity

• The pull request addresses and fixes issue Analytics integration (LangSmith/Langfuse) not capturing token usage or model name from OpenAI #5763.
• The author confirmed backward compatibility for onLLMEnd with both string and object inputs.
• The author verified that all supported analytics providers now receive usage and model data.
• The author ensured no new debug logs or console noise were introduced.
• The author confirmed that pnpm build passes.
• Manual testing was performed using a Flowise run with an Agentflow using ChatOpenAI, confirming that LangSmith and Langfuse displayed token usage and model information, and Langfuse cost/usage fields were populated.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[TravisP-Greener]

Hi @HenryHengZJ - Please see PR to address the missing token usage

[gemini-code-assist]

Code Review

This pull request is a great improvement for analytics integrations. It correctly modifies onLLMEnd to accept structured data, allowing token usage and model names to be passed to providers like Langfuse and LangSmith. The implementation is solid, with robust data extraction logic and comprehensive new unit tests. I have two suggestions to enhance the code's maintainability and robustness, detailed in the comments.

[TravisP-Greener]