Skip to content

Introduce telemetry for observability #117

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Anyitechs wants to merge 1 commit into
base: main
Choose a base branch
from
+356 −10
Open

Introduce telemetry for observability #117

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 26 additions & 4 deletions ldk-server-client/src/client.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,10 +29,10 @@ use ldk_server_protos::api::{
use ldk_server_protos::endpoints::{
BOLT11_RECEIVE_PATH, BOLT11_SEND_PATH, BOLT12_RECEIVE_PATH, BOLT12_SEND_PATH,
CLOSE_CHANNEL_PATH, CONNECT_PEER_PATH, DISCONNECT_PEER_PATH, EXPORT_PATHFINDING_SCORES_PATH,
FORCE_CLOSE_CHANNEL_PATH, GET_BALANCES_PATH, GET_NODE_INFO_PATH, GET_PAYMENT_DETAILS_PATH,
LIST_CHANNELS_PATH, LIST_FORWARDED_PAYMENTS_PATH, LIST_PAYMENTS_PATH, ONCHAIN_RECEIVE_PATH,
ONCHAIN_SEND_PATH, OPEN_CHANNEL_PATH, SIGN_MESSAGE_PATH, SPLICE_IN_PATH, SPLICE_OUT_PATH,
SPONTANEOUS_SEND_PATH, UPDATE_CHANNEL_CONFIG_PATH, VERIFY_SIGNATURE_PATH,
FORCE_CLOSE_CHANNEL_PATH, GET_BALANCES_PATH, GET_METRICS_PATH, GET_NODE_INFO_PATH,
GET_PAYMENT_DETAILS_PATH, LIST_CHANNELS_PATH, LIST_FORWARDED_PAYMENTS_PATH, LIST_PAYMENTS_PATH,
ONCHAIN_RECEIVE_PATH, ONCHAIN_SEND_PATH, OPEN_CHANNEL_PATH, SIGN_MESSAGE_PATH, SPLICE_IN_PATH,
SPLICE_OUT_PATH, SPONTANEOUS_SEND_PATH, UPDATE_CHANNEL_CONFIG_PATH, VERIFY_SIGNATURE_PATH,
};
use ldk_server_protos::error::{ErrorCode, ErrorResponse};
use prost::Message;
Expand Down Expand Up @@ -103,6 +103,28 @@ impl LdkServerClient {
self.post_request(&request, &url).await
}

/// Retrieve the node metrics in Prometheus format.
pub async fn get_metrics(&self) -> Result<String, LdkServerError> {
let url = format!("https://{}/{GET_METRICS_PATH}", self.base_url);
let response = self.client.get(&url).send().await.map_err(|e| {
Copy link
Collaborator

@benthecarman benthecarman Feb 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we refactor post_request to do be able to do GET requests and use that. That has a lot of this logic already and would be better for future use

LdkServerError::new(InternalError, format!("HTTP request failed: {}", e))
})?;

let status = response.status();
let text = response.text().await.map_err(|e| {
LdkServerError::new(InternalError, format!("Failed to read response body: {}", e))
})?;

if status.is_success() {
Ok(text)
} else {
Err(LdkServerError::new(
InternalError,
format!("Request failed with status {}: {}", status, text),
))
}
}

/// Retrieves an overview of all known balances.
/// For API contract/usage, refer to docs for [`GetBalancesRequest`] and [`GetBalancesResponse`].
pub async fn get_balances(
Expand Down
1 change: 1 addition & 0 deletions ldk-server-protos/src/endpoints.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,3 +31,4 @@ pub const SPONTANEOUS_SEND_PATH: &str = "SpontaneousSend";
pub const SIGN_MESSAGE_PATH: &str = "SignMessage";
pub const VERIFY_SIGNATURE_PATH: &str = "VerifySignature";
pub const EXPORT_PATHFINDING_SCORES_PATH: &str = "ExportPathfindingScores";
pub const GET_METRICS_PATH: &str = "metrics";
21 changes: 20 additions & 1 deletion ldk-server/src/main.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ use crate::io::persist::{
use crate::service::NodeService;
use crate::util::config::{load_config, ArgsConfig, ChainSource};
use crate::util::logger::ServerLogger;
use crate::util::metrics::{Metrics, BUILD_METRICS_INTERVAL};
use crate::util::proto_adapter::{forwarded_payment_to_proto, payment_to_proto};
use crate::util::tls::get_or_generate_tls_config;

Expand Down Expand Up @@ -256,6 +257,20 @@ fn main() {
}
};
let event_node = Arc::clone(&node);

let metrics_node = Arc::clone(&node);
let mut interval = tokio::time::interval(BUILD_METRICS_INTERVAL);
let metrics = Arc::new(Metrics::new());
let metrics_bg = Arc::clone(&metrics);
let event_metrics = Arc::clone(&metrics);

runtime.spawn(async move {
loop {
interval.tick().await;
metrics_bg.update_all_pollable_metrics(&metrics_node);
}
});

Comment on lines 267 to 273
Copy link
Collaborator

@benthecarman benthecarman Feb 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

instead of every minute updating this, we should be able to do it real time and just update it when we get a relevant event from ldk-node. ie we get a channel closed event so we update the metrics immediately to reflect that

Copy link
Contributor Author

@Anyitechs Anyitechs Feb 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I intend to maintain a hybrid approach for this, real time update for the Node events but still maintain the polling for metrics like channel/peer/payment count, balances, etc.

let rest_svc_listener = TcpListener::bind(config_file.rest_service_addr)
.await
.expect("Failed to bind listening port");
Expand Down Expand Up @@ -320,6 +335,8 @@ fn main() {
&event_node,
Arc::clone(&event_publisher),
Arc::clone(&paginated_store)).await;

event_metrics.update_total_successful_payments_count(&event_node);
},
Event::PaymentFailed {payment_id, ..} => {
let payment_id = payment_id.expect("PaymentId expected for ldk-server >=0.1");
Expand All @@ -331,6 +348,8 @@ fn main() {
&event_node,
Arc::clone(&event_publisher),
Arc::clone(&paginated_store)).await;

event_metrics.update_total_failed_payments_count(&event_node);
},
Event::PaymentClaimable {payment_id, ..} => {
if let Some(payment_details) = event_node.payment(&payment_id) {
Expand Down Expand Up @@ -415,7 +434,7 @@ fn main() {
res = rest_svc_listener.accept() => {
match res {
Ok((stream, _)) => {
let node_service = NodeService::new(Arc::clone(&node), Arc::clone(&paginated_store), api_key.clone());
let node_service = NodeService::new(Arc::clone(&node), Arc::clone(&paginated_store), api_key.clone(), Arc::clone(&metrics));
let acceptor = tls_acceptor.clone();
runtime.spawn(async move {
match acceptor.accept(stream).await {
Expand Down
24 changes: 19 additions & 5 deletions ldk-server/src/service.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,10 +21,10 @@ use ldk_node::Node;
use ldk_server_protos::endpoints::{
BOLT11_RECEIVE_PATH, BOLT11_SEND_PATH, BOLT12_RECEIVE_PATH, BOLT12_SEND_PATH,
CLOSE_CHANNEL_PATH, CONNECT_PEER_PATH, DISCONNECT_PEER_PATH, EXPORT_PATHFINDING_SCORES_PATH,
FORCE_CLOSE_CHANNEL_PATH, GET_BALANCES_PATH, GET_NODE_INFO_PATH, GET_PAYMENT_DETAILS_PATH,
LIST_CHANNELS_PATH, LIST_FORWARDED_PAYMENTS_PATH, LIST_PAYMENTS_PATH, ONCHAIN_RECEIVE_PATH,
ONCHAIN_SEND_PATH, OPEN_CHANNEL_PATH, SIGN_MESSAGE_PATH, SPLICE_IN_PATH, SPLICE_OUT_PATH,
SPONTANEOUS_SEND_PATH, UPDATE_CHANNEL_CONFIG_PATH, VERIFY_SIGNATURE_PATH,
FORCE_CLOSE_CHANNEL_PATH, GET_BALANCES_PATH, GET_METRICS_PATH, GET_NODE_INFO_PATH,
GET_PAYMENT_DETAILS_PATH, LIST_CHANNELS_PATH, LIST_FORWARDED_PAYMENTS_PATH, LIST_PAYMENTS_PATH,
ONCHAIN_RECEIVE_PATH, ONCHAIN_SEND_PATH, OPEN_CHANNEL_PATH, SIGN_MESSAGE_PATH, SPLICE_IN_PATH,
SPLICE_OUT_PATH, SPONTANEOUS_SEND_PATH, UPDATE_CHANNEL_CONFIG_PATH, VERIFY_SIGNATURE_PATH,
};
use prost::Message;

Expand Down Expand Up @@ -53,6 +53,7 @@ use crate::api::spontaneous_send::handle_spontaneous_send_request;
use crate::api::update_channel_config::handle_update_channel_config_request;
use crate::api::verify_signature::handle_verify_signature_request;
use crate::io::persist::paginated_kv_store::PaginatedKVStore;
use crate::util::metrics::Metrics;
use crate::util::proto_adapter::to_error_response;

// Maximum request body size: 10 MB
Expand All @@ -64,13 +65,15 @@ pub struct NodeService {
node: Arc<Node>,
paginated_kv_store: Arc<dyn PaginatedKVStore>,
api_key: String,
metrics: Arc<Metrics>,
}

impl NodeService {
pub(crate) fn new(
node: Arc<Node>, paginated_kv_store: Arc<dyn PaginatedKVStore>, api_key: String,
metrics: Arc<Metrics>,
) -> Self {
Self { node, paginated_kv_store, api_key }
Self { node, paginated_kv_store, api_key, metrics }
}
}

Expand Down Expand Up @@ -154,6 +157,17 @@ impl Service<Request<Incoming>> for NodeService {
type Future = Pin<Box<dyn Future<Output = Result<Self::Response, Self::Error>> + Send>>;

fn call(&self, req: Request<Incoming>) -> Self::Future {
// Handle metrics endpoint separately to bypass auth and return plain text
if req.uri().path().len() > 1 && &req.uri().path()[1..] == GET_METRICS_PATH {
let metrics = Arc::clone(&self.metrics);
return Box::pin(async move {
Ok(Response::builder()
.header("Content-Type", "text/plain")
.body(Full::new(Bytes::from(metrics.gather_metrics())))
.unwrap())
});
Comment on lines +161 to +168
Copy link
Collaborator

@benthecarman benthecarman Feb 25, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We aren't validating auth here we are short cutting before its done. I'm not sure if that'll break the typical Prometheus flow though?

}

// Extract auth params from headers (validation happens after body is read)
let auth_params = match extract_auth_params(&req) {
Ok(params) => params,
Expand Down
Loading