es_out: support Upstream Servers with configuration overriding by mabrarov · Pull Request #7608 · fluent/fluent-bit

mabrarov · 2023-06-25T08:42:03Z

Implementation of Upstream feature for the Elasticsearch output plugin.

This pull request is based on pull request #1560 and Forward output plugin.

It was tested in a local setup with:

Fluent Bit without Upstream feature connected to a single node of Elasticsearch cluster consisting of 3 master-eligible/data and 1 coordinating nodes.

Refer to elastic-cluster directory of mabrarov/elastic-stack repository for Docker Compose project used to create target Elasticsearch cluster and Kibana.

fluent-bit.conf Fluent Bit configuration file used for the test - refer to fluent-bit-es/fluent-bit.conf and (same in YAML format) fluent-bit-es/fluent-bit.yaml in mabrarov/elastic-stack repository.

Debug log is available at flb_es.log.
Fluent Bit with Upstream feature connected to all Elasticsearch data nodes of Elasticsearch cluster consisting of 3 master-eligible/data and 1 coordinating nodes.

Refer to elastic-cluster directory of mabrarov/elastic-stack repository for Docker Compose project used to create target Elasticsearch cluster and Kibana.

fluent-bit.conf Fluent Bit configuration file used for the test - refer to fluent-bit-es-cluster/fluent-bit.conf and (same in YAML format) fluent-bit-es-cluster/fluent-bit.yaml in mabrarov/elastic-stack repository.

Debug log is available at flb_es_upstream.log.

Testing

Example configuration files for the change can be found in mabrarov/elastic-stack repository under fluent-bit-es-cluster directory.
Debug log output from testing the change - see above.
Attached Valgrind output that shows no leaks or memory corruption was found - refer to flb_run_code_analysis.log for the output of command
```
TEST_PRESET=valgrind SKIP_TESTS='flb-rt-out_td flb-it-network' ./run_code_analysis.sh
```
[N/A] Run local packaging test showing all targets (including any new ones) build.
[N/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

Documentation required for this feature - refer to es_out: support Upstream Servers with configuration overriding fluent-bit-docs#1143.

Backporting

[N/A] Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Summary by CodeRabbit

New Features
- HA-aware Elasticsearch output with per-upstream/per-node configuration and upstream-aware formatter flush-context callbacks.
Integrations
- Cloud ID parsing for Elasticsearch endpoints.
- Optional AWS SigV4 support with STS and unsigned-headers handling.
Configuration
- Consolidated ES property names, added upstream option, and corrected default Elasticsearch port to 9200.
Quality
- Ownership-aware resource handling and improved lifecycle/cleanup.
Tests
- Expanded upstream-focused tests covering formatter, flush-context, and config scenarios.

plugins/out_es/es.h

plugins/out_es/es_conf_parse.c

mabrarov · 2023-06-28T09:19:30Z

Hi reviewers,

Is it possible to approve only workflow for this pull request, so that automated checks and build can start?

Thank you.

PettitWesley · 2023-06-28T17:55:39Z

@mabrarov sure

mabrarov · 2023-06-29T04:35:15Z

Hi @PettitWesley,

It looks like all failed checks are around run-macos-unit-tests jobs and caused by the following failed unit tests:

flb-rt-in_event_test
flb-rt-out_tcp

I feel like other pull requests have the same issues, i.e. it doesn't seem that the failed checks are caused by this pull request changes.

Help of maintainers is appreciated.

Thank you.

plugins/out_es/es_conf.c

mabrarov · 2023-07-10T10:10:13Z

Hi @PettitWesley,

Is it possible to trigger automated workflow (build) for this pull request one more time? I found & fixed one issue and added tests for the new code since last build happened.

Thank you.

mabrarov · 2023-07-20T15:16:09Z

Hi dear reviewers,

Is it possible to get this pull request reviewed / accepted sooner? Is there something pending / waiting from my side to start review?

Thank you.

mabrarov · 2023-08-22T07:18:01Z

Hi @PettitWesley and @edsiper,

It feels like you are code owners for Elasticsearch output plugin. Is there something pending / waiting from my side to start review of this pull request? This new feature was requested 4 years ago and I feel it is something which multiple users of Fluent Bit (not just my team) would like to have.

Thank you.

mabrarov · 2026-02-09T20:08:45Z

Hi @cosmo0920 and @patrick-stephens,

This PR is pending for more than 2.5 years. I appreciate if you could find time and review it again. I believe I have addressed comment of @cosmo0920 which was the only concern at the time the comment was added.

My winter vacation has ended and I won't be able to spend much time on this PR till summer, especially, considering new pending PRs for the same plugin (e.g. #11422 and #11375) - reabasing such changes can be hard.

Thank you.

mabrarov · 2026-02-09T20:12:14Z

I believe that similar changes can be implemented / helpful for OpenSearch output plugin (in a dedicated PR).

…wn to parser of Upstream node configuration section are implemented, e.g. "host" and "port" Signed-off-by: Marat Abrarov <abrarov@gmail.com>

…o the test callback based on configuration of Fluent Bit and based on configuration of plugin Signed-off-by: Marat Abrarov <abrarov@gmail.com>

…llback contract Signed-off-by: Marat Abrarov <abrarov@gmail.com>

…with Upstream node configuration For Elastic cloud authentication these parameters are always taken from plugin configuration and never from Upstream node configuration: cloud_id. For AWS authentication these parameters are always taken from plugin configuration and never from Upstream node configuration: http_proxy, no_proxy, tls*. Signed-off-by: Marat Abrarov <abrarov@gmail.com>

Signed-off-by: Marat Abrarov <abrarov@gmail.com>

…o the test callback based on configuration of Fluent Bit and based on configuration of plugin Signed-off-by: Marat Abrarov <abrarov@gmail.com>

Signed-off-by: Marat Abrarov <abrarov@gmail.com>

…icate username or password used for AWS authentication Signed-off-by: Marat Abrarov <abrarov@gmail.com>

…on and retrying buffer flush Signed-off-by: Marat Abrarov <abrarov@gmail.com>

…ent creation failed Signed-off-by: Marat Abrarov <abrarov@gmail.com>

…iguration data Signed-off-by: Marat Abrarov <abrarov@gmail.com>

…tials Signed-off-by: Marat Abrarov <abrarov@gmail.com>

mabrarov requested review from PettitWesley, edsiper, fujimotos, koleini, leonardo-albertovich and niedbalski as code owners June 25, 2023 08:42

github-actions bot added the docs-required label Jun 25, 2023

mabrarov mentioned this pull request Jun 25, 2023

es_out: support Upstream Servers with configuration overriding fluent/fluent-bit-docs#1143

Open

mabrarov force-pushed the feature/out_es_upstream_support_extended branch from 812020d to ba3382a Compare June 25, 2023 09:06

mabrarov mentioned this pull request Jun 25, 2023

es_out: support Upstream Servers #1560

Open

mabrarov commented Jun 26, 2023

View reviewed changes

plugins/out_es/es.h Show resolved Hide resolved

mabrarov commented Jun 26, 2023

View reviewed changes

plugins/out_es/es_conf_parse.c Outdated Show resolved Hide resolved

mabrarov temporarily deployed to pr June 28, 2023 17:55 — with GitHub Actions Inactive

mabrarov temporarily deployed to pr June 28, 2023 18:22 — with GitHub Actions Inactive

mabrarov commented Jul 7, 2023

View reviewed changes

plugins/out_es/es_conf.c Outdated Show resolved Hide resolved

mabrarov force-pushed the feature/out_es_upstream_support_extended branch from ba3382a to b7cd81b Compare July 8, 2023 10:22

mabrarov temporarily deployed to pr July 10, 2023 12:05 — with GitHub Actions Inactive

mabrarov temporarily deployed to pr July 10, 2023 12:28 — with GitHub Actions Inactive

mabrarov force-pushed the feature/out_es_upstream_support_extended branch from b7cd81b to b81d3f7 Compare July 20, 2023 19:38

mabrarov force-pushed the feature/out_es_upstream_support_extended branch from b81d3f7 to f6431c2 Compare September 30, 2023 13:32

mabrarov force-pushed the feature/out_es_upstream_support_extended branch from 2598c57 to b4d0186 Compare February 8, 2026 15:56

mabrarov temporarily deployed to pr February 8, 2026 15:57 — with GitHub Actions Inactive

mabrarov temporarily deployed to pr February 8, 2026 16:16 — with GitHub Actions Inactive

mabrarov added 12 commits February 10, 2026 02:23

pipeline: outputs: es: basic support of Upstream, only parameters kno…

29e5c23

…wn to parser of Upstream node configuration section are implemented, e.g. "host" and "port" Signed-off-by: Marat Abrarov <abrarov@gmail.com>

engine: support of tests which require generation of context passed t…

7f8a2b2

…o the test callback based on configuration of Fluent Bit and based on configuration of plugin Signed-off-by: Marat Abrarov <abrarov@gmail.com>

pipeline: outputs: es: fixed tests after modification of formatter ca…

3f94b53

…llback contract Signed-off-by: Marat Abrarov <abrarov@gmail.com>

pipeline: outputs: es: tests for upstream node configuration properties

7200f01

Signed-off-by: Marat Abrarov <abrarov@gmail.com>

engine: support of tests which require generation of context passed t…

c4a38e7

…o the test callback based on configuration of Fluent Bit and based on configuration of plugin Signed-off-by: Marat Abrarov <abrarov@gmail.com>

pipeline: outputs: es: code deduplication

47b585f

Signed-off-by: Marat Abrarov <abrarov@gmail.com>

pipeline: outputs: es: plugin configuration failure if failed to dupl…

8e5566c

…icate username or password used for AWS authentication Signed-off-by: Marat Abrarov <abrarov@gmail.com>

pipeline: outputs: es: fix of memory leak when using AWS authenticati…

ae00dea

…on and retrying buffer flush Signed-off-by: Marat Abrarov <abrarov@gmail.com>

pipeline: outputs: es: avoid zero pointer dereferencing when HTTP cli…

d58c956

…ent creation failed Signed-off-by: Marat Abrarov <abrarov@gmail.com>

pipeline: outputs: es: custom types to track ownership of plugin conf…

eaba388

…iguration data Signed-off-by: Marat Abrarov <abrarov@gmail.com>

pipeline: outputs: es: fixed error handling when parsing cloud creden…

34be28e

…tials Signed-off-by: Marat Abrarov <abrarov@gmail.com>

mabrarov force-pushed the feature/out_es_upstream_support_extended branch from b4d0186 to 34be28e Compare February 10, 2026 01:05

mabrarov temporarily deployed to pr February 10, 2026 01:05 — with GitHub Actions Inactive

mabrarov temporarily deployed to pr February 10, 2026 01:25 — with GitHub Actions Inactive

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

es_out: support Upstream Servers with configuration overriding#7608

es_out: support Upstream Servers with configuration overriding#7608
mabrarov wants to merge 12 commits intofluent:masterfrom
mabrarov:feature/out_es_upstream_support_extended

mabrarov commented Jun 25, 2023 •

edited by coderabbitai bot

Loading

Uh oh!

Uh oh!

Uh oh!

mabrarov commented Jun 28, 2023

Uh oh!

PettitWesley commented Jun 28, 2023

Uh oh!

mabrarov commented Jun 29, 2023 •

edited

Loading

Uh oh!

Uh oh!

mabrarov commented Jul 10, 2023

Uh oh!

mabrarov commented Jul 20, 2023

Uh oh!

mabrarov commented Aug 22, 2023

Uh oh!

mabrarov commented Feb 9, 2026 •

edited

Loading

Uh oh!

mabrarov commented Feb 9, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Conversation

mabrarov commented Jun 25, 2023 • edited by coderabbitai bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Uh oh!

Uh oh!

Uh oh!

mabrarov commented Jun 28, 2023

Uh oh!

PettitWesley commented Jun 28, 2023

Uh oh!

mabrarov commented Jun 29, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

mabrarov commented Jul 10, 2023

Uh oh!

mabrarov commented Jul 20, 2023

Uh oh!

mabrarov commented Aug 22, 2023

Uh oh!

mabrarov commented Feb 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mabrarov commented Feb 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

mabrarov commented Jun 25, 2023 •

edited by coderabbitai bot

Loading

mabrarov commented Jun 29, 2023 •

edited

Loading

mabrarov commented Feb 9, 2026 •

edited

Loading

mabrarov commented Feb 9, 2026 •

edited

Loading