feat(buffer-handler): add buffering support for external loggers

[acascell]

Issue number: closes #7918

Summary

This PR implements log buffering support for external libraries when using AWS Lambda Powertools Logger with buffering enabled.

Changes

1. New BufferingHandler class (aws_lambda_powertools/logging/buffer/handler.py):

   • Custom logging handler that intercepts logs from external libraries extending logging.Handler
   • Override and extends emit() method delegating to the source Logger's _add_log_record_to_buffer() function
   • Shares the same buffer cache and tracer_id as the Powertools Logger
   • Ensures external library logs are buffered and flushed together with application logs

2. Enhanced copy_config_to_registered_loggers() function (aws_lambda_powertools/logging/utils.py):

   • Added new include_buffering parameter (default: False)
   • Maintains backward compatibility - existing behavior unchanged when parameter is False

3. Updated configuration logic (-):

   • adds BufferingHandler on include_buffering flag

4. Included unit and functional tests

• tests/functional/logger/required_dependencies/test_logger_utils.py
• tests/unit/logger/required_dependencies/test_logger_buffer_handler.py

User Experience

Before this change:
When using Logger with buffering enabled, external library logs (botocore, mangum, etc.) were NOT buffered - they appeared immediately in CloudWatch, defeating the purpose of buffering:

from aws_lambda_powertools import Logger
from aws_lambda_powertools.logging import utils
from aws_lambda_powertools.logging.config import LoggerBufferConfig

# Enable buffering on Powertools Logger
buffer_config = LoggerBufferConfig(
    max_bytes=20480,
    buffer_at_verbosity="DEBUG",
    flush_on_error_log=True
)
LOGGER = Logger(service="payment", buffer_config=buffer_config)

# Configure external loggers
utils.copy_config_to_registered_loggers(source_logger=LOGGER)

# Get external logger
botocore_logger = logging.getLogger("botocore")

def handler(event, context):
    LOGGER.debug("Processing payment")              # Buffered
    LOGGER.info("Validating request")               # Buffered
    botocore_logger.info("Making API call")         # NOT buffered (appears immediately)
    botocore_logger.debug("Request headers set")    # NOT buffered (appears immediately)
    LOGGER.error("Payment failed")                  # Flushes only LOGGER's logs

Result: External library logs appeared immediately in CloudWatch, while application logs were buffered.

After this change:
External library logs are now buffered alongside application logs when include_buffering=True:

from aws_lambda_powertools import Logger
from aws_lambda_powertools.logging import utils
from aws_lambda_powertools.logging.config import LoggerBufferConfig

# Enable buffering on Powertools Logger
buffer_config = LoggerBufferConfig(
    max_bytes=20480,
    buffer_at_verbosity="DEBUG",
    flush_on_error_log=True
)
LOGGER = Logger(service="payment", buffer_config=buffer_config)

# Configure external loggers WITH buffering
utils.copy_config_to_registered_loggers(
    source_logger=LOGGER,
    include_buffering=True  # Enable buffering for external loggers
)

# Get external logger
botocore_logger = logging.getLogger("botocore")

def handler(event, context):
    LOGGER.debug("Processing payment")              # Buffered
    LOGGER.info("Validating request")               # Buffered
    botocore_logger.info("Making API call")         # Buffered
    botocore_logger.debug("Request headers set")    # Buffered
    LOGGER.error("Payment failed")                  # Flushes ALL logs (app + external)

Result: All logs (application + external libraries) are buffered together and flushed as a unit

---

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

[boring-cyborg]

Thanks a lot for your first contribution! Please check out our contributing guidelines and don't hesitate to ask whatever you need.
In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link

[dreamorosi]

Hi @acascell - thanks for the PR and sorry for the delayed review.

@leandrodamascena will most likely review this next week once he's back.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.73%. Comparing base (aa969f7) to head (53aefec).

Additional details and impacted files

@@           Coverage Diff            @@
##           develop    #7994   +/-   ##
========================================
  Coverage    96.73%   96.73%           
========================================
  Files          278      279    +1     
  Lines        13657    13675   +18     
  Branches      1086     1087    +1     
========================================
+ Hits         13211    13229   +18     
  Misses         327      327           
  Partials       119      119           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[leandrodamascena]

Hey @acascell thanks a lot for working on this! I ran some tests and I think we need to fix small changes before we merge this. I just left some comments, can you please read them and see if this make sense?

Thanks a lot.

[leandrodamascena]

I noticed the handler buffers everything regardless of level, but our Logger checks _check_minimum_buffer_log_level before deciding to buffer or emit. So if someone sets buffer_at_verbosity="DEBUG", the Logger emits INFO logs right away - but external loggers through this handler would buffer them instead. That inconsistency could cause logs to be silently lost if the buffer is never flushed.

The fix is to add the same level check:

from aws_lambda_powertools.logging.buffer.functions import _check_minimum_buffer_log_level

def emit(self, record: logging.LogRecord) -> None:
    level_name = logging.getLevelName(record.levelno)

    # If log level exceeds buffer threshold, emit directly through source logger
    if _check_minimum_buffer_log_level(self.buffer_config.buffer_at_verbosity, level_name):
        self.source_logger._logger.handle(record)
        return

    self.source_logger._add_log_record_to_buffer(
        level=record.levelno,
        msg=record.getMessage(),
        args=(),
        exc_info=record.exc_info,
        stack_info=False,
    )

[leandrodamascena]

When include_buffering=True, the external logger gets no source handlers (the return skips the handler copy loop). This is fine for buffered logs since they flush through the source logger's handler. But if you add the level check from comment 1, logs above the buffer threshold need a handler to emit through - otherwise they go nowhere.

You'd need to also attach the source handlers:

if include_buffering and buffer_config is not None:
    buffer_handler = BufferingHandler(
        buffer_cache=source_logger._buffer_cache,
        buffer_config=buffer_config,
        source_logger=source_logger,
    )
    logger.addHandler(buffer_handler)

    for source_handler in source_logger.handlers:
        logger.addHandler(source_handler)
    return

But then you'd need the BufferingHandler to suppress the record from propagating to the other handlers when it buffers - otherwise you get double logging. A cleaner approach might be to handle both paths inside emit() itself and keep the early return as-is.