diff --git a/README.md b/README.md index 83780d7..9d419b4 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,13 @@ -![Build](https://github.com/central-university-dev/backend-academy-2025-spring-template/actions/workflows/build.yaml/badge.svg) +# 📌 Link Tracker -# Link Tracker +[![Spring Boot](https://img.shields.io/badge/Spring%20Boot-logo-6DB33F?logo=spring)](https://spring.io/projects/spring-boot) +[![Kafka](https://img.shields.io/badge/Apache%20Kafka-logo-000?logo=apachekafka)](https://kafka.apache.org/) +[![Hibernate](https://img.shields.io/badge/Hibernate-logo-59666C?logo=hibernate)](https://hibernate.org/) +[![Docker](https://img.shields.io/badge/Docker-logo-2496ED?logo=docker)](https://www.docker.com/) +[![Testcontainers](https://img.shields.io/badge/Testcontainers-logo-000?logo=testcontainers)](https://testcontainers.com/) +[![Swagger](https://img.shields.io/badge/Swagger-logo-85EA2D?logo=swagger)](https://swagger.io/) +[![PostgreSQL](https://img.shields.io/badge/PostgreSQL-logo-336791?logo=postgresql)](https://www.postgresql.org/) +[![Redis](https://img.shields.io/badge/Redis-logo-DC382D?logo=redis)](https://redis.io/) ---- @@ -8,9 +15,7 @@ ---- - - -Проект сделан в рамках курса Академия Бэкенда. +## 📝 Описание проекта Приложение для отслеживания обновлений контента по ссылкам. При появлении новых событий отправляется уведомление в Telegram. @@ -21,6 +26,150 @@ * Bot * Scrapper -Для работы требуется БД `PostgreSQL`. Присутствует опциональная зависимость на `Kafka`. +Для работы требуется `PostgreSQL`, `Redis`, `Kafka`. + +### 📟 Схема приложения +![Scrapper](https://github.com/user-attachments/assets/1087e9d5-518f-4650-b470-aaa62a382a9c) + + +--- + +## 🤖 Бот + +### 📌 Функционал + +Бот поддерживает следующие команды: + +- [**`/start`**](https://github.com/Delphington/LinterBot/blob/main/bot/src/main/java/backend/academy/bot/command/helper/StartCommand.java) — регистрация пользователя. +- [**`/help`**](https://github.com/Delphington/LinterBot/blob/main/bot/src/main/java/backend/academy/bot/command/helper/HelpCommand.java) — список всех доступных команд. +- [**`/track`**](https://github.com/Delphington/LinterBot/blob/main/bot/src/main/java/backend/academy/bot/command/link/TrackCommand.java) — добавление ссылки для отслеживания +- [**`/untrack`**](https://github.com/Delphington/LinterBot/blob/main/bot/src/main/java/backend/academy/bot/command/link/UntrackCommand.java) — удаление ссылки из списка отслеживаемых. +- [**`/list`**](https://github.com/Delphington/LinterBot/blob/main/bot/src/main/java/backend/academy/bot/command/link/ListCommand.java) — получение списка всех отслеживаемых ссылок. +- [**`/tag`**](https://github.com/Delphington/LinterBot/blob/main/bot/src/main/java/backend/academy/bot/command/tag/TagCommand.java) — выводит все ссылки, у которых есть определенный тег (/tag < tag >) +- [**`/taglist`**](https://github.com/Delphington/LinterBot/blob/main/bot/src/main/java/backend/academy/bot/command/tag/TagListCommand.java) — все теги, которые ввел пользователь +- [**`/untag`**](https://github.com/Delphington/LinterBot/blob/main/bot/src/main/java/backend/academy/bot/command/tag/UnTagCommand.java) — удаление тега у ссылки (/untag < tag > < link >) +- [**`/filter`**](https://github.com/Delphington/LinterBot/blob/main/bot/src/main/java/backend/academy/bot/command/filter/FilterCommand.java) — позволяет добавить фильтр +- [**`/filterlist`**](https://github.com/Delphington/LinterBot/blob/main/bot/src/main/java/backend/academy/bot/command/filter/FilterListCommand.java) — выводит все фильтры пользователя +- [**`/unfilter`**](https://github.com/Delphington/LinterBot/blob/main/bot/src/main/java/backend/academy/bot/command/filter/UnFilterCommand.java) — удаление фильтров (/unfilter < filter >) + + +### 🔄 Взаимодействие со Scrapper + +Бот общается со Scrapper API через: +- [**`ScrapperTgChatClient`**](https://github.com/Delphington/LinterBot/blob/main/bot/src/main/java/backend/academy/bot/client/chat/ScrapperTgChatClientImpl.java) — регистрация и удаление чатов. +- [**`LinkClient`**](https://github.com/Delphington/LinterBot/blob/main/bot/src/main/java/backend/academy/bot/client/link/ScrapperLinkClientImpl.java) — управление ссылками (добавление, удаление, получение списка ссылок). +- [**`TagClient`**](https://github.com/Delphington/LinterBot/blob/main/bot/src/main/java/backend/academy/bot/client/tag/ScrapperTagClientImpl.java) — получение списка тегов, получение ссылки по тегу и удаление тега у ссылки. +- [**`ScrapperFilterClient`**](https://github.com/Delphington/LinterBot/blob/main/bot/src/main/java/backend/academy/bot/client/filter/ScrapperFilterClientImpl.java) — управление фильтрами (добавление, удаление, получение списка фильтров) + +### 📩 Получение обновлений + +- Бот получает обновления о ссылках через [`UpdateController`](https://github.com/Delphington/LinterBot/blob/main/bot/src/main/java/backend/academy/bot/api/controller/UpdateController.java) по HTTP либо через [`KafkaUpdateListener`](https://github.com/Delphington/LinterBot/blob/main/bot/src/main/java/backend/academy/bot/kafka/client/KafkaLinkUpdateListener.java) по Kafka. +- Scrapper отправляет данные по OpenAPI-контракту. +- Обновления рассылаются чатам через [`NotificationService`](https://github.com/Delphington/LinterBot/blob/main/bot/src/main/java/backend/academy/bot/notification/NotificationService.java). + +### 📜 Дополнительно + +- Бот поддерживает встроенное [**меню команд**](https://github.com/Delphington/LinterBot/blob/main/bot/src/main/java/backend/academy/bot/processor/UserMessageProcessor.java) в Telegram. + +### 🧠 Кеширование Redis + +Бот кеширует ответы для следующих команд: +- /tag +- /tag +- /list + +🔄 Кеш автоматически сбрасывается в следующих случаях: +- При добавлении или удалении ссылки (/track, /untrack, /untag) + +При вызове команд бот сначала проверяет наличие ответа в кеше. Если данные найдены — используется кеш. В противном случае происходит обращение к Scrapper API, и результат сохраняется в кеш. + +--- + +## 🗂️ Scrapper + +### 📌 Функционал + +Scrapper обрабатывает запросы от бота: +- **Работа с чатами** через [`ChatController`](https://github.com/Delphington/LinterBot/blob/main/scrapper/src/main/java/backend/academy/scrapper/controller/ChatController.java). +- **Работа с ссылками** через [`LinkController`](https://github.com/Delphington/LinterBot/blob/main/scrapper/src/main/java/backend/academy/scrapper/controller/LinkController.java). +- **Работа с тегами** через [`TagController`](https://github.com/Delphington/LinterBot/blob/main/scrapper/src/main/java/backend/academy/scrapper/controller/TagController.java). +- **Работа с фильтрами** через [`FilterController`](https://github.com/Delphington/LinterBot/blob/main/scrapper/src/main/java/backend/academy/scrapper/controller/FilterController.java). + +Все контроллеры работают по OpenAPI-контракту. + +### 🔄 Получение обновлений + +- [**`LinkUpdateScheduler`**](https://github.com/Delphington/LinterBot/blob/main/scrapper/src/main/java/backend/academy/scrapper/scheduler/LinkUpdaterScheduler.java). +- **📡 Источники данных:** + - GitHub — через [`GitHubClient`](https://github.com/Delphington/LinterBot/blob/main/scrapper/src/main/java/backend/academy/scrapper/tracker/client/GitHubClient.java) + - Stack Overflow — через [`StackOverflowClient`](https://github.com/Delphington/LinterBot/blob/main/scrapper/src/main/java/backend/academy/scrapper/tracker/client/StackOverFlowClient.java) +- **⚙️ Обработка полученных данных** + 1. Запрос обновлений + - Для каждого URL запрашивается обновление через открытый API GitHub и StackOverflow + 2. Фильтрация обновлений + - Определяются подписанные пользователи (чаты), которые отслеживают данный URL. + - Для каждого пользователя применяется его список фильтров: + 3. Парсинг ответа + - Полученный JSON-ответ анализируется, извлекаются нужные значения. + 4. Проверка актуальности + - Обновление считается релевантным, если оно произошло после последнего запуска шедулера. + 5. Формирование уведомлений + - Отобранные обновления, соответствующие фильтрам, отправляются пользователям либо через HTTP, либо через Kafka. +- **⚙️ Обработка батчей и многопоточность** + - Ссылки на обновления запрашиваются партиями (batch) заданного размера. Каждый батч делится между потоками. Количество потоков настраивается через конфигурацию. + +## 📦 Хранение данных + +- `tg_chats` — таблица чатов. +- `links` — таблица ссылок. +- `tags` — таблица тегов. +- `filters` — таблица фильтров. +- `filter_list` — таблица фильтров каждого пользователя + +### 📌 Схема базы данных + +![{E4ED68AF-CD94-4964-B402-74AE70A10960}](https://github.com/user-attachments/assets/26e0773b-61db-41fb-b696-01e68d824b3a) + +💡 **Один чат может отслеживать несколько ссылок, а одна ссылка может быть отслеживаемой несколькими чатами.** +📌 **Каждая ссылка может иметь несколько тегов и фильтров в рамках одного чата.** + +## 🗂️ Метрики + +#### RED +![изображение](https://github.com/user-attachments/assets/e242b8e5-34e5-406f-9ee3-761fd25a7d87) + +#### Custom + - Количество пользовательских сообщений в секунду + - График количества активных ссылок в БД по типу (github, stackoverflow) + - p50, p95, p99 времени работы одного scrape по типу (github, stackoverflow) + +![image](https://github.com/user-attachments/assets/84ebe7bf-88e4-4563-b14b-5abc35eb3306) + +#### jvm +![image](https://github.com/user-attachments/assets/d855023e-8dda-4366-ada0-74e768264bfe) + + +--- + + +## 🚀 Запуск проекта + +1. Клонирование репозитория специальной ветки + ```bash + git clone -b feat-docker --single-branch https://github.com/Delphington/LinterBot + ``` + +2. Перейдите в директорию с docker-конфигурацией: + ```bash + cd LinterTelegramBot + ``` + +3. Нужно прокинуть специальную переменную среды + ```bash + $env:TELEGRAM_TOKEN="YOUR_TOKEN" + ``` -Для дополнительной справки: [HELP.md](./HELP.md) +4. Запустите сборку и запуск контейнеров: + ```bash + docker-compose up --build + ``` diff --git a/bot/bot.Dockerfile b/bot/bot.Dockerfile new file mode 100644 index 0000000..48cb9a9 --- /dev/null +++ b/bot/bot.Dockerfile @@ -0,0 +1,5 @@ +FROM eclipse-temurin:23-jdk-alpine +WORKDIR /app +COPY target/bot-1.0.jar /app/bot-1.0.jar +EXPOSE 8080 +CMD ["java", "-jar", "bot-1.0.jar"] diff --git a/bot/src/main/java/backend/academy/bot/api/controller/UpdateController.java b/bot/src/main/java/backend/academy/bot/api/controller/UpdateController.java index 33c2287..249d28c 100644 --- a/bot/src/main/java/backend/academy/bot/api/controller/UpdateController.java +++ b/bot/src/main/java/backend/academy/bot/api/controller/UpdateController.java @@ -2,6 +2,7 @@ import backend.academy.bot.api.dto.request.LinkUpdate; import backend.academy.bot.notification.NotificationService; +import io.micrometer.core.annotation.Timed; import io.swagger.v3.oas.annotations.Operation; import io.swagger.v3.oas.annotations.responses.ApiResponse; import io.swagger.v3.oas.annotations.responses.ApiResponses; @@ -31,6 +32,8 @@ public void update(@RequestBody @Valid LinkUpdate linkUpdate) { } @PostMapping("/public") + @ResponseStatus(HttpStatus.OK) + @Timed(value = "update.timer", histogram = true) // Явное включение гистограммы public void update() { log.info("Пришло обновление по ссылке"); } diff --git a/bot/src/main/java/backend/academy/bot/command/Command.java b/bot/src/main/java/backend/academy/bot/command/Command.java index 02b9274..8823c4e 100644 --- a/bot/src/main/java/backend/academy/bot/command/Command.java +++ b/bot/src/main/java/backend/academy/bot/command/Command.java @@ -2,6 +2,7 @@ import com.pengrad.telegrambot.model.Update; import com.pengrad.telegrambot.request.SendMessage; +import io.micrometer.core.annotation.Timed; public interface Command { @@ -9,6 +10,7 @@ public interface Command { String description(); + @Timed("helpCommandMetric") SendMessage handle(Update update); default boolean matchesCommand(Update update) { diff --git a/bot/src/main/java/backend/academy/bot/config/MetricsConfig.java b/bot/src/main/java/backend/academy/bot/config/MetricsConfig.java new file mode 100644 index 0000000..1a5c0a4 --- /dev/null +++ b/bot/src/main/java/backend/academy/bot/config/MetricsConfig.java @@ -0,0 +1,19 @@ +package backend.academy.bot.config; + +import io.micrometer.core.instrument.Counter; +import io.micrometer.core.instrument.binder.MeterBinder; +import org.springframework.context.annotation.Bean; +import org.springframework.context.annotation.Configuration; + +@Configuration +public class MetricsConfig { + + @Bean + public MeterBinder meterBinder() { + return registry -> { + Counter.builder("msg_count") + .description("Количество сообщений от пользователей") + .register(registry); + }; + } +} diff --git a/bot/src/main/java/backend/academy/bot/processor/UserMessageProcessor.java b/bot/src/main/java/backend/academy/bot/processor/UserMessageProcessor.java index 226ff39..42f0022 100644 --- a/bot/src/main/java/backend/academy/bot/processor/UserMessageProcessor.java +++ b/bot/src/main/java/backend/academy/bot/processor/UserMessageProcessor.java @@ -10,6 +10,7 @@ import com.pengrad.telegrambot.request.SendMessage; import com.pengrad.telegrambot.request.SetMyCommands; import com.pengrad.telegrambot.response.BaseResponse; +import io.micrometer.core.instrument.MeterRegistry; import java.util.List; import lombok.Getter; import lombok.RequiredArgsConstructor; @@ -25,6 +26,7 @@ public class UserMessageProcessor { private final TelegramBot telegramBot; private final List commandList; private final UserStateManager userStateManager; + private final MeterRegistry meterRegistry; public void registerCommands() { List commands = commandList.stream() @@ -42,6 +44,7 @@ public void registerCommands() { } public SendMessage process(Update update) { + meterRegistry.counter("msg_count").increment(); Long id = update.message().chat().id(); userStateManager.createUserIfNotExist(id); diff --git a/bot/src/main/resources/Metrics.md b/bot/src/main/resources/Metrics.md new file mode 100644 index 0000000..79b491d --- /dev/null +++ b/bot/src/main/resources/Metrics.md @@ -0,0 +1,22 @@ +# RED + +Rate +``` +sum by(uri, method) ( + rate(http_server_requests_seconds_count{job="Bot"}[1m]) +) +``` +Errors +``` +sum by(uri, status) ( + rate(http_server_requests_seconds_count{job="Bot", status=~"4..|5.."}[1m]) +) +``` +Duration +``` +sum by(uri) ( +rate(http_server_requests_seconds_sum{job="Bot"}[1m]) +) +``` + +# Custom diff --git a/bot/src/main/resources/application.yaml b/bot/src/main/resources/application.yaml index 5e2be7b..b9a363d 100644 --- a/bot/src/main/resources/application.yaml +++ b/bot/src/main/resources/application.yaml @@ -1,7 +1,7 @@ app: telegram-token: ${TELEGRAM_TOKEN} # env variable link: - scrapper-uri: "http://localhost:8081" + scrapper-uri: "http://scrapper:8081" webclient: timeouts: connect-timeout: 10s # 10 секунд на установку соединения @@ -30,11 +30,11 @@ spring: type: redis data: redis: - host: localhost + host: redis port: 6379 kafka: - bootstrap-servers: "localhost:29092" + bootstrap-servers: broker:9092 consumer: auto-offset-reset: earliest group-id: "consumer-group" @@ -131,13 +131,38 @@ bucket4j: refill-amount: 50 # Количество токенов для пополнения refill-seconds: 60 # Интервал пополнения в секундах (например, 60 = 1 минута) - - - -#logging: -# structured: -# format: -# file: ecs -# console: ecs -# level: -# root: INFO +management: + server: + port: 8090 # Отдельный порт для метрик + endpoints: + web: + base-path: / + path-mapping: + prometheus: metrics + exposure: + include: "*" + + distribution: + percentiles-histogram: + http.server.requests: true # Гистограммы для перцентилей + + prometheus: + metrics: + export: + enabled: true # Включить экспорт метрик для Prometheus + web: + server: + auto-time-requests: true # Должен быть здесь! + + metrics: + tags: + application: ${spring.application.name} # Добавляет тег с именем приложения + + +logging: + structured: + format: + file: ecs + console: ecs + level: + root: INFO diff --git a/bot/src/test/java/backend/academy/bot/processor/UserMessageProcessorTest.java b/bot/src/test/java/backend/academy/bot/metrics/UserMessageProcessorTest.java similarity index 52% rename from bot/src/test/java/backend/academy/bot/processor/UserMessageProcessorTest.java rename to bot/src/test/java/backend/academy/bot/metrics/UserMessageProcessorTest.java index 31ea369..f265143 100644 --- a/bot/src/test/java/backend/academy/bot/processor/UserMessageProcessorTest.java +++ b/bot/src/test/java/backend/academy/bot/metrics/UserMessageProcessorTest.java @@ -1,17 +1,20 @@ -package backend.academy.bot.processor; +package backend.academy.bot.metrics; import static org.junit.jupiter.api.Assertions.*; import static org.mockito.Mockito.*; import backend.academy.bot.command.Command; import backend.academy.bot.command.link.TrackCommand; -import backend.academy.bot.state.UserState; +import backend.academy.bot.processor.UserMessageProcessor; import backend.academy.bot.state.UserStateManager; import com.pengrad.telegrambot.TelegramBot; import com.pengrad.telegrambot.model.Chat; import com.pengrad.telegrambot.model.Message; import com.pengrad.telegrambot.model.Update; import com.pengrad.telegrambot.request.SendMessage; +import io.micrometer.core.instrument.Counter; +import io.micrometer.core.instrument.MeterRegistry; +import io.micrometer.core.instrument.simple.SimpleMeterRegistry; import java.util.List; import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.DisplayName; @@ -34,39 +37,14 @@ public class UserMessageProcessorTest { private UserStateManager userStateManager; private UserMessageProcessor userMessageProcessor; + private MeterRegistry meterRegistry; @BeforeEach void setUp() { MockitoAnnotations.openMocks(this); - userMessageProcessor = new UserMessageProcessor(telegramBot, List.of(command1, trackCommand), userStateManager); - } - - @Test - @DisplayName("Обработка сообщения: команда найдена и обработана") - void testProcess_CommandFoundAndHandled() { - Update update = createUpdateWithText("/mock"); - when(command1.matchesCommand(update)).thenReturn(true); - when(command1.handle(update)).thenReturn(new SendMessage(123L, "Mock message")); - - SendMessage result = userMessageProcessor.process(update); - verify(command1, times(1)).matchesCommand(update); - verify(command1, times(1)).handle(update); - assertEquals("Mock message", result.getParameters().get("text")); - } - - @Test - @DisplayName("Обработка сообщения: команда не найдена, состояние WAITING_URL") - void testProcess_NoCommandFound_WaitingUrlState() { - Update update = createUpdateWithText("https://github.com/example"); - when(command1.matchesCommand(update)).thenReturn(false); - when(userStateManager.getUserState(123L)).thenReturn(UserState.WAITING_URL); - when(trackCommand.handle(update)).thenReturn(new SendMessage(123L, "Track command handled")); - - SendMessage result = userMessageProcessor.process(update); - - verify(command1, times(1)).matchesCommand(update); - verify(trackCommand, times(1)).handle(update); - assertEquals("Track command handled", result.getParameters().get("text")); + meterRegistry = new SimpleMeterRegistry(); // Используем реальный MeterRegistry + userMessageProcessor = + new UserMessageProcessor(telegramBot, List.of(command1, trackCommand), userStateManager, meterRegistry); } @Test @@ -74,10 +52,14 @@ void testProcess_NoCommandFound_WaitingUrlState() { void testProcess_UserCreatedIfNotExist() { Update update = createUpdateWithText("/start"); when(command1.matchesCommand(update)).thenReturn(true); - when(command1.handle(update)).thenReturn(new SendMessage(123L, "User created")); + when(command1.handle(update)).thenReturn(new SendMessage("123", "User created")); userMessageProcessor.process(update); + // Проверяем метрику + Counter counter = meterRegistry.counter("msg_count"); + assertEquals(1, counter.count()); + verify(userStateManager, times(1)).createUserIfNotExist(123L); } @@ -89,8 +71,6 @@ private Update createUpdateWithText(String text) { when(update.message()).thenReturn(message); when(message.chat()).thenReturn(chat); when(chat.id()).thenReturn(123L); - when(message.text()).thenReturn(text); - return update; } } diff --git a/docker-compose.yaml b/docker-compose.yaml index 07dd8b9..78eba1e 100644 --- a/docker-compose.yaml +++ b/docker-compose.yaml @@ -104,14 +104,83 @@ services: networks: - kafka-net + + prometheus: + image: prom/prometheus:latest + ports: + - "9090:9090" + volumes: + - ./prometheus.yml:/etc/prometheus/prometheus.yml + networks: + - monitoring + + grafana: + image: grafana/grafana:latest + ports: + - "3000:3000" + environment: + - GF_SECURITY_ADMIN_USER=admin + - GF_SECURITY_ADMIN_PASSWORD=admin + networks: + - monitoring + + + bot: + image: delphington/bot:latest +# build: +# context: ./bot +# dockerfile: bot.Dockerfile + container_name: bot + restart: unless-stopped + ports: + - "8080:8080" # Проброс порта для доступа с хоста + - "8090:8090" # Метрики + environment: + - TELEGRAM_TOKEN=${TELEGRAM_TOKEN} + depends_on: + - broker + - redis + networks: + - backend + - kafka-net # Чтобы видеть Kafka (broker:9092) + + + # --- SCRAPPER --- + scrapper: + image: delphington/scrapper:latest +# build: +# context: ./scrapper +# dockerfile: scrapper.Dockerfile + container_name: scrapper + restart: unless-stopped + ports: + - "8081:8081" + - "8091:8091" + environment: + - GITHUB_TOKEN=${GITHUB_TOKEN} # Переменные из .env + - SO_TOKEN_KEY=${SO_TOKEN_KEY} + - SO_ACCESS_TOKEN=${SO_ACCESS_TOKEN} + depends_on: + - postgresql + - broker + - redis + networks: + - backend + - kafka-net + volumes: postgresql: redis: zookeeper: broker: + prometheus: + grafana: + networks: backend: driver: bridge kafka-net: driver: bridge + monitoring: + driver: bridge diff --git a/prometheus.yml b/prometheus.yml new file mode 100644 index 0000000..85db9aa --- /dev/null +++ b/prometheus.yml @@ -0,0 +1,38 @@ +global: + scrape_interval: 5s + external_labels: + monitor: 'codelab-monitor' + +scrape_configs: + - job_name: 'Bot' + scrape_interval: 5s + metrics_path: '/metrics' + static_configs: + - targets: ['host.docker.internal:8090'] + + - job_name: 'Scrapper' + scrape_interval: 5s + metrics_path: '/metrics' + static_configs: + - targets: ['host.docker.internal:8091'] + + + + +#scrape_configs: +# - job_name: "prometheus" # Мониторинг самого Prometheus +# static_configs: +# - targets: ["localhost:9090"] +# +# - job_name: "bot" # Сервис bot на порту 8081 +# metrics_path: "/metrics" # Если используется Spring Boot Actuator +# static_configs: +# - targets: ["localhost:8081"] +# labels: +# service: "bot" +# +# - job_name: "scrapper" # Сервис scrapper на порту 8082 (если 8081 занят) +# static_configs: +# - targets: ["localhost:8082"] +# labels: +# service: "scrapper" diff --git a/scrapper/scrapper.Dockerfile b/scrapper/scrapper.Dockerfile new file mode 100644 index 0000000..7aade4d --- /dev/null +++ b/scrapper/scrapper.Dockerfile @@ -0,0 +1,5 @@ +FROM eclipse-temurin:23-jdk-alpine +WORKDIR /app +COPY target/scrapper-1.0.jar /app/scrapper.jar +EXPOSE 8081 +CMD ["java", "-jar", "scrapper.jar"] diff --git a/scrapper/src/main/java/backend/academy/scrapper/configuration/metric/ActiveLinksMetricsConfig.java b/scrapper/src/main/java/backend/academy/scrapper/configuration/metric/ActiveLinksMetricsConfig.java new file mode 100644 index 0000000..78eda8c --- /dev/null +++ b/scrapper/src/main/java/backend/academy/scrapper/configuration/metric/ActiveLinksMetricsConfig.java @@ -0,0 +1,37 @@ +package backend.academy.scrapper.configuration.metric; + +import io.micrometer.core.instrument.Gauge; +import io.micrometer.core.instrument.MeterRegistry; +import java.util.concurrent.atomic.AtomicInteger; +import org.springframework.context.annotation.Bean; +import org.springframework.context.annotation.Configuration; + +@Configuration +public class ActiveLinksMetricsConfig { + + private final AtomicInteger githubProcessedLinks = new AtomicInteger(0); + private final AtomicInteger stackoverflowProcessedLinks = new AtomicInteger(0); + + @Bean + public AtomicInteger githubProcessedLinksCounter() { + return githubProcessedLinks; + } + + @Bean + public AtomicInteger stackoverflowProcessedLinksCounter() { + return stackoverflowProcessedLinks; + } + + @Bean + public Gauge githubProcessedLinksGauge(MeterRegistry registry, AtomicInteger githubProcessedLinksCounter) { + return Gauge.builder("scrapper.links.processed.github", githubProcessedLinksCounter::get) + .register(registry); + } + + @Bean + public Gauge stackoverflowProcessedLinksGauge( + MeterRegistry registry, AtomicInteger stackoverflowProcessedLinksCounter) { + return Gauge.builder("scrapper.links.processed.stackoverflow", stackoverflowProcessedLinksCounter::get) + .register(registry); + } +} diff --git a/scrapper/src/main/java/backend/academy/scrapper/configuration/metric/PercentileMetricsConfig.java b/scrapper/src/main/java/backend/academy/scrapper/configuration/metric/PercentileMetricsConfig.java new file mode 100644 index 0000000..127f4f8 --- /dev/null +++ b/scrapper/src/main/java/backend/academy/scrapper/configuration/metric/PercentileMetricsConfig.java @@ -0,0 +1,24 @@ +package backend.academy.scrapper.configuration.metric; + +import io.micrometer.core.instrument.MeterRegistry; +import io.micrometer.core.instrument.Timer; +import org.springframework.context.annotation.Bean; +import org.springframework.context.annotation.Configuration; + +@Configuration +public class PercentileMetricsConfig { + + @Bean + public Timer githubScrapeTimer(MeterRegistry registry) { + return Timer.builder("scrapper.scrape.time") + .description("Time taken to scrape GitHub links") + .register(registry); + } + + @Bean + public Timer stackoverflowScrapeTimer(MeterRegistry registry) { + return Timer.builder("scrapper.scrape.time") + .description("Time taken to scrape StackOverflow links") + .register(registry); + } +} diff --git a/scrapper/src/main/java/backend/academy/scrapper/scheduler/LinkUpdaterScheduler.java b/scrapper/src/main/java/backend/academy/scrapper/scheduler/LinkUpdaterScheduler.java index 852c5c7..1f9ac25 100644 --- a/scrapper/src/main/java/backend/academy/scrapper/scheduler/LinkUpdaterScheduler.java +++ b/scrapper/src/main/java/backend/academy/scrapper/scheduler/LinkUpdaterScheduler.java @@ -10,6 +10,7 @@ import java.util.concurrent.CompletableFuture; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; +import java.util.concurrent.atomic.AtomicInteger; import lombok.RequiredArgsConstructor; import lombok.extern.slf4j.Slf4j; import org.springframework.beans.factory.annotation.Value; @@ -27,6 +28,9 @@ public class LinkUpdaterScheduler { private final ExecutorService executorService = Executors.newFixedThreadPool(4); private static final int COUNT_THREAD = 4; + private final AtomicInteger githubProcessedLinksCounter; + private final AtomicInteger stackoverflowProcessedLinksCounter; + @Value("${scheduler.batch-size}") private int batchSize; @@ -34,6 +38,9 @@ public class LinkUpdaterScheduler { public void update() { log.info("Проверка обновления"); + githubProcessedLinksCounter.set(0); + stackoverflowProcessedLinksCounter.set(0); + int offset = 0; List links; diff --git a/scrapper/src/main/java/backend/academy/scrapper/tracker/update/LinkUpdateProcessor.java b/scrapper/src/main/java/backend/academy/scrapper/tracker/update/LinkUpdateProcessor.java index 1f99bfd..92b9196 100644 --- a/scrapper/src/main/java/backend/academy/scrapper/tracker/update/LinkUpdateProcessor.java +++ b/scrapper/src/main/java/backend/academy/scrapper/tracker/update/LinkUpdateProcessor.java @@ -19,11 +19,13 @@ import backend.academy.scrapper.tracker.update.exception.BadLinkRequestException; import backend.academy.scrapper.tracker.update.model.LinkUpdate; import backend.academy.scrapper.util.Utils; +import io.micrometer.core.instrument.Timer; import java.time.OffsetDateTime; import java.time.ZoneId; import java.util.ArrayList; import java.util.List; import java.util.Optional; +import java.util.concurrent.atomic.AtomicInteger; import lombok.Getter; import lombok.RequiredArgsConstructor; import lombok.extern.slf4j.Slf4j; @@ -47,14 +49,24 @@ public class LinkUpdateProcessor implements Constance { private static final String CONST_GITHUB = "github"; private static final String CONST_STACK_OVER_FLOW = "stackoverflow"; + // Для активных ссылок + private final AtomicInteger githubProcessedLinksCounter; + private final AtomicInteger stackoverflowProcessedLinksCounter; + + // Перцентиль + private final Timer githubScrapeTimer; + private final Timer stackoverflowScrapeTimer; + public void updateLink(List linkList) { updatedLinkList = new ArrayList<>(); for (LinkDto item : linkList) { String urlString = item.url().toString(); if (urlString.contains(CONST_GITHUB)) { + githubProcessedLinksCounter.incrementAndGet(); handlerUpdateGitHub(item); } else if (urlString.contains(CONST_STACK_OVER_FLOW)) { + stackoverflowProcessedLinksCounter.incrementAndGet(); handlerUpdateStackOverFlow(item); } else { throw new BadLinkRequestException( @@ -69,77 +81,79 @@ public void updateLink(List linkList) { public void handlerUpdateGitHub(LinkDto linkDto) { - if (linkDto.lastUpdated() == null) { - linkDto.lastUpdated(OffsetDateTime.now(ZoneId.systemDefault())); - Link link = linkService - .findById(linkDto.id()) - .orElseThrow(() -> new LinkNotFoundException("Ссылка с ID " + linkDto.id() + " не найдена")); - link.updatedAt(OffsetDateTime.now(ZoneId.systemDefault())); - linkService.update(link); + githubScrapeTimer.record(() -> { + if (linkDto.lastUpdated() == null) { + linkDto.lastUpdated(OffsetDateTime.now(ZoneId.systemDefault())); + Link link = linkService + .findById(linkDto.id()) + .orElseThrow(() -> new LinkNotFoundException("Ссылка с ID " + linkDto.id() + " не найдена")); + link.updatedAt(OffsetDateTime.now(ZoneId.systemDefault())); + linkService.update(link); - return; - } + return; + } - GitHubRequest gitHubRequest = - Utils.parseUrlToGithubRequest(linkDto.url().toString()); + GitHubRequest gitHubRequest = + Utils.parseUrlToGithubRequest(linkDto.url().toString()); - Optional> issuesListOptional = - gitHubClient.fetchIssue(gitHubRequest, linkDto.lastUpdated()); - Optional> pullRequestListOptional = - gitHubClient.fetchPullRequest(gitHubRequest, linkDto.lastUpdated()); + Optional> issuesListOptional = + gitHubClient.fetchIssue(gitHubRequest, linkDto.lastUpdated()); + Optional> pullRequestListOptional = + gitHubClient.fetchPullRequest(gitHubRequest, linkDto.lastUpdated()); - Optional gitHubResponseOptional = gitHubClient.getFetchDate(gitHubRequest); + Optional gitHubResponseOptional = gitHubClient.getFetchDate(gitHubRequest); - StringBuilder issueStringBuilder = new StringBuilder(); - StringBuilder pullRequestStringBuilder = new StringBuilder(); - StringBuilder repositoryStringBuilder = new StringBuilder(); + StringBuilder issueStringBuilder = new StringBuilder(); + StringBuilder pullRequestStringBuilder = new StringBuilder(); + StringBuilder repositoryStringBuilder = new StringBuilder(); - if (issuesListOptional.isPresent()) { - List issuesListTemp = - issuesListOptional.orElseThrow(() -> new IllegalStateException("Optional is Empty")); - issueStringBuilder = updateFetchIssue(linkDto, issuesListTemp); - } + if (issuesListOptional.isPresent()) { + List issuesListTemp = + issuesListOptional.orElseThrow(() -> new IllegalStateException("Optional is Empty")); + issueStringBuilder = updateFetchIssue(linkDto, issuesListTemp); + } - if (pullRequestListOptional.isPresent()) { - List pullRequestListTemp = - pullRequestListOptional.orElseThrow(() -> new IllegalStateException("Optional is Empty")); - pullRequestStringBuilder = updateFetchPullRequest(linkDto, pullRequestListTemp); - } + if (pullRequestListOptional.isPresent()) { + List pullRequestListTemp = + pullRequestListOptional.orElseThrow(() -> new IllegalStateException("Optional is Empty")); + pullRequestStringBuilder = updateFetchPullRequest(linkDto, pullRequestListTemp); + } - if (gitHubResponseOptional.isPresent()) { - GitHubResponse gitHubResponseTemp = - gitHubResponseOptional.orElseThrow(() -> new IllegalStateException("Optional is Empty")); - repositoryStringBuilder = updateFetchRepository(linkDto, gitHubResponseTemp); - } + if (gitHubResponseOptional.isPresent()) { + GitHubResponse gitHubResponseTemp = + gitHubResponseOptional.orElseThrow(() -> new IllegalStateException("Optional is Empty")); + repositoryStringBuilder = updateFetchRepository(linkDto, gitHubResponseTemp); + } - if (!issueStringBuilder.isEmpty() - || !pullRequestStringBuilder.isEmpty() - || !repositoryStringBuilder.isEmpty()) { - linkDto.lastUpdated(OffsetDateTime.now(ZoneId.systemDefault())); - - Link link = linkService - .findById(linkDto.id()) - .orElseThrow(() -> new LinkNotFoundException("ID " + linkDto.id() + "ссылка не найдена")); - link.updatedAt(OffsetDateTime.now(ZoneId.systemDefault())); - linkService.update(link); - - StringBuilder temp = new StringBuilder(); - temp.append(CONST_SPACE) - .append(CONST_NEXT_LINE) - .append(CONST_SYMBOL) - .append(" Репозиторий: "); - gitHubResponseOptional.ifPresent(gitHubResponse -> temp.append(gitHubResponse.repositoryName())); - temp.append(CONST_NEXT_LINE) - .append(pullRequestStringBuilder) - .append(CONST_NEXT_LINE) - .append(issueStringBuilder) - .append(CONST_NEXT_LINE) - .append(repositoryStringBuilder) - .append(CONST_NEXT_LINE); - - linkDto.descriptionUpdate(temp.toString()); - updatedLinkList.add(linkDto); - } + if (!issueStringBuilder.isEmpty() + || !pullRequestStringBuilder.isEmpty() + || !repositoryStringBuilder.isEmpty()) { + linkDto.lastUpdated(OffsetDateTime.now(ZoneId.systemDefault())); + + Link link = linkService + .findById(linkDto.id()) + .orElseThrow(() -> new LinkNotFoundException("ID " + linkDto.id() + "ссылка не найдена")); + link.updatedAt(OffsetDateTime.now(ZoneId.systemDefault())); + linkService.update(link); + + StringBuilder temp = new StringBuilder(); + temp.append(CONST_SPACE) + .append(CONST_NEXT_LINE) + .append(CONST_SYMBOL) + .append(" Репозиторий: "); + gitHubResponseOptional.ifPresent(gitHubResponse -> temp.append(gitHubResponse.repositoryName())); + temp.append(CONST_NEXT_LINE) + .append(pullRequestStringBuilder) + .append(CONST_NEXT_LINE) + .append(issueStringBuilder) + .append(CONST_NEXT_LINE) + .append(repositoryStringBuilder) + .append(CONST_NEXT_LINE); + + linkDto.descriptionUpdate(temp.toString()); + updatedLinkList.add(linkDto); + } + }); } public StringBuilder updateFetchRepository(LinkDto linkDto, GitHubResponse gitHubResponse) { @@ -206,70 +220,72 @@ public StringBuilder updateFetchIssue(LinkDto linkDto, List issue // Коммент https://api.stackexchange.com/2.3/questions/79486408/comments?site=stackoverflow&filter=withbody public void handlerUpdateStackOverFlow(LinkDto linkDto) { + stackoverflowScrapeTimer.record(() -> { + if (linkDto.lastUpdated() == null) { + linkDto.lastUpdated(OffsetDateTime.now(ZoneId.systemDefault())); + Link link = linkService + .findById(linkDto.id()) + .orElseThrow(() -> new LinkNotFoundException("Ссылка с ID " + linkDto.id() + " не найдена")); + link.updatedAt(OffsetDateTime.now(ZoneId.systemDefault())); + linkService.update(link); + return; + } - if (linkDto.lastUpdated() == null) { - linkDto.lastUpdated(OffsetDateTime.now(ZoneId.systemDefault())); - Link link = linkService - .findById(linkDto.id()) - .orElseThrow(() -> new LinkNotFoundException("Ссылка с ID " + linkDto.id() + " не найдена")); - link.updatedAt(OffsetDateTime.now(ZoneId.systemDefault())); - linkService.update(link); - return; - } + StackOverFlowRequest stackOverFlowRequest = + Utils.parseUrlToStackOverFlowRequest(linkDto.url().toString()); - StackOverFlowRequest stackOverFlowRequest = - Utils.parseUrlToStackOverFlowRequest(linkDto.url().toString()); + Optional questionResponseOptional = + stackOverFlowClient.fetchQuestion(stackOverFlowRequest); + Optional commentResponseOptional = stackOverFlowClient.fetchComment(stackOverFlowRequest); + Optional answersResponseOptional = stackOverFlowClient.fetchAnswer(stackOverFlowRequest); - Optional questionResponseOptional = stackOverFlowClient.fetchQuestion(stackOverFlowRequest); - Optional commentResponseOptional = stackOverFlowClient.fetchComment(stackOverFlowRequest); - Optional answersResponseOptional = stackOverFlowClient.fetchAnswer(stackOverFlowRequest); + StringBuilder answerStringBuilder = new StringBuilder(); + StringBuilder commentStringBuilder = new StringBuilder(); + StringBuilder questionStringBuilder = new StringBuilder(); - StringBuilder answerStringBuilder = new StringBuilder(); - StringBuilder commentStringBuilder = new StringBuilder(); - StringBuilder questionStringBuilder = new StringBuilder(); + if (questionResponseOptional.isPresent()) { + QuestionResponse questionResponseTemp = + questionResponseOptional.orElseThrow(() -> new IllegalStateException("Optional is Empty")); + questionStringBuilder = updateFetchQuestion(linkDto, questionResponseTemp); + } + if (commentResponseOptional.isPresent()) { + CommentResponse commentResponseTemp = + commentResponseOptional.orElseThrow(() -> new IllegalStateException("Optional is Empty")); + commentStringBuilder = updateFetchComment(linkDto, commentResponseTemp); + } + if (answersResponseOptional.isPresent()) { + AnswersResponse answersResponseTemp = + answersResponseOptional.orElseThrow(() -> new IllegalStateException("Optional is Empty")); + answerStringBuilder = updateFetchAnswers(linkDto, answersResponseTemp); + } - if (questionResponseOptional.isPresent()) { - QuestionResponse questionResponseTemp = - questionResponseOptional.orElseThrow(() -> new IllegalStateException("Optional is Empty")); - questionStringBuilder = updateFetchQuestion(linkDto, questionResponseTemp); - } - if (commentResponseOptional.isPresent()) { - CommentResponse commentResponseTemp = - commentResponseOptional.orElseThrow(() -> new IllegalStateException("Optional is Empty")); - commentStringBuilder = updateFetchComment(linkDto, commentResponseTemp); - } - if (answersResponseOptional.isPresent()) { - AnswersResponse answersResponseTemp = - answersResponseOptional.orElseThrow(() -> new IllegalStateException("Optional is Empty")); - answerStringBuilder = updateFetchAnswers(linkDto, answersResponseTemp); - } + if (!answerStringBuilder.isEmpty() || !commentStringBuilder.isEmpty() || !questionStringBuilder.isEmpty()) { + linkDto.lastUpdated(OffsetDateTime.now(ZoneId.systemDefault())); + Link link = linkService + .findById(linkDto.id()) + .orElseThrow(() -> new LinkNotFoundException("Ссылка с ID " + linkDto.id() + " не найдена")); + link.updatedAt(OffsetDateTime.now(ZoneId.systemDefault())); + linkService.update(link); + + StringBuilder temp = new StringBuilder(); + temp.append(CONST_SPACE) + .append(CONST_NEXT_LINE) + .append(CONST_SYMBOL) + .append(CONST_THEME_QUESTION); + questionResponseOptional.ifPresent(questionResponse -> + temp.append(questionResponse.items().get(0).title())); + temp.append(CONST_NEXT_LINE) + .append(answerStringBuilder) + .append(CONST_NEXT_LINE) + .append(commentStringBuilder) + .append(CONST_NEXT_LINE) + .append(questionStringBuilder) + .append(CONST_NEXT_LINE); - if (!answerStringBuilder.isEmpty() || !commentStringBuilder.isEmpty() || !questionStringBuilder.isEmpty()) { - linkDto.lastUpdated(OffsetDateTime.now(ZoneId.systemDefault())); - Link link = linkService - .findById(linkDto.id()) - .orElseThrow(() -> new LinkNotFoundException("Ссылка с ID " + linkDto.id() + " не найдена")); - link.updatedAt(OffsetDateTime.now(ZoneId.systemDefault())); - linkService.update(link); - - StringBuilder temp = new StringBuilder(); - temp.append(CONST_SPACE) - .append(CONST_NEXT_LINE) - .append(CONST_SYMBOL) - .append(CONST_THEME_QUESTION); - questionResponseOptional.ifPresent(questionResponse -> - temp.append(questionResponse.items().get(0).title())); - temp.append(CONST_NEXT_LINE) - .append(answerStringBuilder) - .append(CONST_NEXT_LINE) - .append(commentStringBuilder) - .append(CONST_NEXT_LINE) - .append(questionStringBuilder) - .append(CONST_NEXT_LINE); - - linkDto.descriptionUpdate(temp.toString()); - updatedLinkList.add(linkDto); - } + linkDto.descriptionUpdate(temp.toString()); + updatedLinkList.add(linkDto); + } + }); } public StringBuilder updateFetchQuestion(LinkDto linkDto, QuestionResponse questionResponse) { diff --git a/scrapper/src/main/resources/Metrics.md b/scrapper/src/main/resources/Metrics.md new file mode 100644 index 0000000..5ca2192 --- /dev/null +++ b/scrapper/src/main/resources/Metrics.md @@ -0,0 +1,43 @@ +# RED + +Rate +``` +sum by(uri, method) ( + rate(http_server_requests_seconds_count{job="Scrapper"}[1m]) +) +``` +Errors +``` +sum by(uri, status) ( + rate(http_server_requests_seconds_count{job="Scrapper", status=~"4..|5.."}[1m]) +) +``` +Duration +``` +sum by(uri) ( +rate(http_server_requests_seconds_sum{job="Scrapper"}[1m]) +) +``` + +# Custom + +Active Links +``` +scrapper_links_processed_stackoverflow + +scrapper_links_processed_github +``` + +GitHub Percentile +``` +scrapper_scrape_time_seconds{type="github",quantile="0.5"} +scrapper_scrape_time_seconds{type="github",quantile="0.95"} +scrapper_scrape_time_seconds{type="github",quantile="0.99"} +``` + +StackOverFlow Percentile +``` +scrapper_scrape_time_seconds{type="stackoverflow",quantile="0.5"} +scrapper_scrape_time_seconds{type="stackoverflow",quantile="0.95"} +scrapper_scrape_time_seconds{type="stackoverflow",quantile="0.99"} +``` diff --git a/scrapper/src/main/resources/application.yaml b/scrapper/src/main/resources/application.yaml index 8219e6e..898b98c 100644 --- a/scrapper/src/main/resources/application.yaml +++ b/scrapper/src/main/resources/application.yaml @@ -7,7 +7,7 @@ app: access-token: ${SO_ACCESS_TOKEN:} stack-overflow-url: https://api.stackexchange.com/2.3 link: - telegram-bot-uri: "http://localhost:8080" + telegram-bot-uri: "http://bot:8080" database-access-type: orm message-transport: kafka topic: "updated-topic" @@ -15,7 +15,7 @@ app: scheduler: enable: true - interval: 30000 + interval: 60000 force-check-delay: PT10S batch-size: 250 @@ -86,7 +86,7 @@ spring: datasource: driver-class-name: org.postgresql.Driver - url: jdbc:postgresql://localhost:5433/scrapper_db + url: jdbc:postgresql://postgresql:5432/scrapper_db username: postgres password: postgres @@ -102,7 +102,7 @@ spring: dialect: org.hibernate.dialect.PostgreSQLDialect show_sql: true kafka: - bootstrap-servers: "localhost:29092" + bootstrap-servers: broker:9092 producer: properties: spring.json.add.type.headers: false @@ -115,11 +115,41 @@ springdoc: enabled: true path: /swagger-ui -#logging: -# structured: -# format: -# file: ecs -# console: ecs -# level: -# root: INFO + +management: + server: + port: 8091 # Отдельный порт для метрик + endpoints: + web: + base-path: / + path-mapping: + prometheus: metrics + exposure: + include: "*" + + + distribution: + percentiles-histogram: + http.server.requests: true # Гистограммы для перцентилей + + prometheus: + metrics: + export: + enabled: true # Включить экспорт метрик для Prometheus + web: + server: + auto-time-requests: true # Должен быть здесь! + metrics: + tags: + application: ${spring.application.name} # Добавляет тег с именем приложения + + + +logging: + structured: + format: + file: ecs + console: ecs + level: + root: INFO diff --git a/scrapper/src/test/java/metrics/LinkUpdateProcessorMetricsTest.java b/scrapper/src/test/java/metrics/LinkUpdateProcessorMetricsTest.java new file mode 100644 index 0000000..594a7b1 --- /dev/null +++ b/scrapper/src/test/java/metrics/LinkUpdateProcessorMetricsTest.java @@ -0,0 +1,134 @@ +package metrics; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import backend.academy.scrapper.client.TgBotClient; +import backend.academy.scrapper.repository.TgChatLinkRepository; +import backend.academy.scrapper.service.LinkService; +import backend.academy.scrapper.tracker.client.GitHubClient; +import backend.academy.scrapper.tracker.client.StackOverFlowClient; +import backend.academy.scrapper.tracker.update.LinkUpdateProcessor; +import backend.academy.scrapper.tracker.update.dto.LinkDto; +import io.micrometer.core.instrument.MeterRegistry; +import io.micrometer.core.instrument.Timer; +import io.micrometer.core.instrument.simple.SimpleMeterRegistry; +import java.net.URI; +import java.time.OffsetDateTime; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.mockito.Mock; +import org.mockito.MockitoAnnotations; +import org.springframework.context.annotation.Bean; +import org.springframework.context.annotation.Configuration; + +public class LinkUpdateProcessorMetricsTest { + @Configuration + static class TestMetricsConfig { + @Bean + public MeterRegistry meterRegistry() { + return new SimpleMeterRegistry(); + } + + @Bean + public AtomicInteger githubProcessedLinksCounter() { + return new AtomicInteger(0); + } + + @Bean + public AtomicInteger stackoverflowProcessedLinksCounter() { + return new AtomicInteger(0); + } + + @Bean + public Timer githubScrapeTimer(MeterRegistry registry) { + return Timer.builder("scrapper.scrape.time") + .description("Time taken to scrape GitHub links") + .register(registry); + } + + @Bean + public Timer stackoverflowScrapeTimer(MeterRegistry registry) { + return Timer.builder("scrapper.scrape.time") + .description("Time taken to scrape StackOverflow links") + .register(registry); + } + } + + private MeterRegistry meterRegistry; + private AtomicInteger githubCounter; + private AtomicInteger stackoverflowCounter; + private Timer githubTimer; + private Timer stackoverflowTimer; + private LinkUpdateProcessor linkUpdateProcessor; + + @Mock + private TgBotClient tgBotClient; + + @Mock + private GitHubClient gitHubClient; + + @Mock + private StackOverFlowClient stackOverFlowClient; + + @Mock + private LinkService linkService; + + @Mock + private TgChatLinkRepository tgChatLinkRepository; + + @BeforeEach + void setUp() { + MockitoAnnotations.openMocks(this); + + // Инициализируем тестовые бины + TestMetricsConfig config = new TestMetricsConfig(); + meterRegistry = config.meterRegistry(); + githubCounter = config.githubProcessedLinksCounter(); + stackoverflowCounter = config.stackoverflowProcessedLinksCounter(); + githubTimer = config.githubScrapeTimer(meterRegistry); + stackoverflowTimer = config.stackoverflowScrapeTimer(meterRegistry); + + // Создаем тестируемый процессор с моками и реальными метриками + linkUpdateProcessor = new LinkUpdateProcessor( + tgBotClient, + gitHubClient, + stackOverFlowClient, + linkService, + tgChatLinkRepository, + githubCounter, + stackoverflowCounter, + githubTimer, + stackoverflowTimer); + } + + @Test + void testGithubScrapeTimer() { + // Проверяем начальное состояние таймера + assertEquals(0, githubTimer.count()); + + // Имитируем обработку GitHub ссылки + linkUpdateProcessor.handlerUpdateGitHub( + new LinkDto(1L, URI.create("https://github.com/user/repo"), OffsetDateTime.now(), null)); + + // Проверяем, что таймер зарегистрировал вызов + assertEquals(1, githubTimer.count()); + assertTrue(githubTimer.totalTime(TimeUnit.MILLISECONDS) > 0); + } + + @Test + void testStackoverflowScrapeTimer() { + // Проверяем начальное состояние таймера + assertEquals(0, stackoverflowTimer.count()); + + // Имитируем обработку StackOverflow ссылки + linkUpdateProcessor.handlerUpdateStackOverFlow( + new LinkDto(1L, URI.create("https://stackoverflow.com/questions/123"), OffsetDateTime.now(), null)); + + // Проверяем, что таймер зарегистрировал вызов + assertEquals(1, stackoverflowTimer.count()); + assertTrue(stackoverflowTimer.totalTime(TimeUnit.MILLISECONDS) > 0); + } +}