Feign metrics are incompatible with RestTemplate metrics when using Prometheus #1301
Description
Describe the bug
When a spring boot/cloud project has observations, metrics and prometheus enabled with Feign, the http_client_requests_active metric registered by Feign is not compatible with the one registered by notably RestTemplate (maybe others). This is a problem as soon as you have for exemplle a DiscoveryClient like EurekaDiscoveryClient that uses RestTemplates under the hood.
This results in the following warning when first calling a method of the FeignClient :
The meter (MeterId{name='http.client.requests.active', tags=[tag(clientName=com.example.demo.TestFeignClient),tag(http.method=GET),tag(http.status_code=CLIENT_ERROR),tag(http.url=/outside/test)]}) registration has failed: Prometheus requires that all meters with the same name have the same set of tag keys. There is already an existing meter named 'http_client_requests_active_seconds' containing tag keys [client_name, exception, method, outcome, status, uri]. The meter you are attempting to register has keys [clientName, http_method, http_status_code, http_url]. Note that subsequent logs will be logged at debug level.
Consequently, et more importantly, the feign metrics are not exported by prometheus.
The metrics are effectively not compatible :
RestTemplate http_client_requests_active_seconds_count :
http_client_requests_active_seconds_count{client_name="localhost",exception="none",method="GET",outcome="UNKNOWN",status="CLIENT_ERROR",uri="/test"} 0
http_client_requests_active_seconds_sum{client_name="localhost",exception="none",method="GET",outcome="UNKNOWN",status="CLIENT_ERROR",uri="/test"} 0.0
FeignClient http_client_requests_active_seconds_count :
http_client_requests_active_seconds_count{clientName="com.example.demo.TestFeignClient",http_method="GET",http_status_code="CLIENT_ERROR",http_url="/outside/test"} 0
http_client_requests_active_seconds_sum{clientName="com.example.demo.TestFeignClient",http_method="GET",http_status_code="CLIENT_ERROR",http_url="/outside/test"} 0.0
The FeignClient metric tags should probly look like this instead :
http_client_requests_active_seconds_count{client_name="com.example.demo.TestFeignClient",method="GET",status="CLIENT_ERROR",uri="/outside/test"} 0
http_client_requests_active_seconds_sum{client_name="com.example.demo.TestFeignClient",method="GET",status="CLIENT_ERROR",uri="/outside/test"} 0.0
Sample
I have a simple repro here
Just run the tests and they will fail. If you replace the restTemplateBuilder.build() by new RestTemplate() the test passes. Demonstrating that the problem effectively comes from the restTemplate being observerd.
Below the classes in the project :
@SpringBootApplication
@RestController
@EnableFeignClients
@RequiredArgsConstructor
public class TestApplication {
private final TestFeignClient testFeignClient;
@GetMapping("/test")
public String test() {
return testFeignClient.test();
}
public static void main(String[] args) {
SpringApplication.run(TestApplication.class, args);
}
}
@FeignClient(value = "TestFeignClient", name = "TestFeignClient", url = "http://localhost:9004")
public interface TestFeignClient {
@GetMapping("/outside/test")
String test();
}
@SpringBootTest(classes = TestApplication.class, webEnvironment = WebEnvironment.DEFINED_PORT, properties = {
"spring.application.name=demo",
"server.port=18080",
"management.tracing.enabled=true",
"management.prometheus.metrics.export.enabled=true",
"management.endpoints.web.exposure.include=prometheus"
})
@WireMockTest(httpPort = 9004)
class DemoTestApplicationTests {
@Autowired
private RestTemplateBuilder restTemplateBuilder;
@Test
void repro() {
WireMock.stubFor(get(urlPathEqualTo("/outside/test")) //
.willReturn(aResponse() //
.withStatus(200) //
.withHeader(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE) //
.withBody("hello world")));
RestTemplate restTemplate = restTemplateBuilder.build(); // replace by "new RestTemplate()" for test to pass
String response = restTemplate.getForObject("http://localhost:18080/test", String.class);
assertAll(
() -> assertThat(response).isEqualTo("hello world"),
() -> assertThat(logAppender.list)
.extracting(ILoggingEvent::getMessage)
.noneMatch(msg -> msg.contains("registration has failed")),
() -> assertThat(restTemplate.getForObject("http://localhost:18080/actuator/prometheus", String.class))
.contains("http_client_requests_active_seconds_count{clientName=\"com.example.demo.TestFeignClient")
);
}
// setup of log appender to capture logs
private ListAppender<ILoggingEvent> logAppender;
private Logger logger;
@BeforeEach
void setUp() {
logger = (Logger) LoggerFactory.getLogger(io.micrometer.prometheusmetrics.PrometheusMeterRegistry.class);
logAppender = new ListAppender<>();
logAppender.start();
logger.addAppender(logAppender);
}
@AfterEach
void tearDown() {
logger.detachAppender(logAppender);
}
}