feat: integration service sync to repositories [CM-865] #3722

mbani01 · 2025-12-30T13:34:57Z

This pull request introduces significant improvements to the repository integration and synchronization logic within the IntegrationService in backend/src/services/integrationService.ts. The main focus is on ensuring that repository mappings between source integrations and the unified public.repositories table are consistent, robust, and transactional across different platforms (GitHub, GitLab, Gerrit, and direct Git). The changes add validation, transactional safety, and better handling of edge cases when mapping, restoring, or soft-deleting repositories.

Key changes include:

Repository Mapping and Synchronization Enhancements

Added a new method mapUnifiedRepositories, which handles inserting, restoring, and soft-deleting repositories in the unified public.repositories table in a transactional manner. This includes robust validation to prevent remapping repositories across different integrations and ensures consistency across platforms.
Introduced helper methods like validateRepoIntegrationMapping and buildRepositoryPayloads to support the main mapping logic, including building payloads with correct associations (segment, integration, insights project, forkedFrom, etc.).

Platform-Aware Repository Synchronization

Updated the gitConnectOrUpdate method and all relevant integration flows (GitHub, GitLab, Gerrit, direct Git) to call mapUnifiedRepositories with the correct platform and integration context, ensuring that repository synchronization is handled appropriately for each platform. [1] [2] [3]
Added a sourcePlatform parameter to gitConnectOrUpdate to control when unified repository mapping should be triggered, allowing for more flexible integration flows.

Integration with Data Access Layer

Expanded imports from the data access layer to support new repository operations, such as fetching, inserting, restoring, and soft-deleting repositories.

Improved Transaction Management

Ensured that all repository mapping operations are performed within a transaction, with proper commit and rollback handling to maintain data integrity in case of errors.

Platform-Specific Handling in Integration Flows

Updated integration flows for GitHub, GitLab, Gerrit, and direct Git to pass the appropriate platform type and ensure that repository synchronization logic is platform-aware. [1] [2] [3] [4] [5] [6]

These changes make the repository integration process more reliable, maintainable, and scalable across different source platforms.

Note

Introduces a unified repository-mapping flow across code platforms (GitHub, GitLab, Gerrit, Git) backed by public.repositories, plus a single API to fetch mappings.

Adds GET /integration/:id/repositories and removes platform-specific repo GETs; frontend now uses IntegrationService.fetchIntegrationRepositories for GitHub, GitLab, and Git
Major IntegrationService refactor: new mapUnifiedRepositories (insert/restore/soft-delete with validations and mirrored-repo handling), getIntegrationRepositories, and enhanced deletion logic to preserve/clean up repos correctly; gitConnectOrUpdate gains sourcePlatform and syncs only owned repos
Data layer: new repositories module (insertRepositories, restoreRepositories, softDeleteRepositories, getRepositoriesBy*, getIntegrationReposMapping), and syncRepositoriesToGitV2 now reuses IDs from github/gitlab/git where available
Frontend UX: Git settings visualize and protect mirrored repos (disabled inputs, tooltips); array-input supports disabled; Gerrit UI defaults enableGit to true and hides the toggle

^{Written by Cursor Bugbot for commit 442550f. This will update automatically on new commits. Configure here.}

github-actions · 2025-12-30T16:51:02Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

github-actions · 2025-12-30T16:51:24Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

github-actions · 2025-12-30T16:51:43Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

github-actions · 2025-12-30T16:54:22Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

github-actions · 2025-12-30T16:54:35Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

cursor · 2025-12-30T17:01:00Z

backend/src/services/integrationService.ts

              }),
            },
            segmentOptions,
+            PlatformType.GITHUB,


Missing repository sync for GitHub and GitLab integrations

The PR adds PlatformType.GITHUB and PlatformType.GITLAB parameters to gitConnectOrUpdate calls, which causes the internal mapUnifiedRepositories call to be skipped (due to the if (!sourcePlatform) check at line 1422). However, unlike githubNangoConnect (which adds an external call at line 961) and gerritConnectOrUpdate (which adds an external call at line 1789), neither mapGithubRepos nor mapGitlabRepos add the required external mapUnifiedRepositories call. This means when these methods are called directly via API endpoints (githubMapRepos.ts, gitlabMapRepos.ts) or from updateGithubIntegrationSettings, repositories won't be synced to public.repositories, breaking the PR's stated goal of consistent repository mapping across all platforms.

Additional Locations (1)

backend/src/services/integrationService.ts#L2865-L2878

cursor · 2025-12-30T17:01:00Z

backend/src/services/integrationService.ts

+
+      const txOptions = { ...this.options, transaction }
+      const txService = new IntegrationService(txOptions)
+      await txService.mapUnifiedRepositories(PlatformType.GERRIT, integration.id, mapping)


Gerrit repository sync fails when Git disabled

The mapUnifiedRepositories call for Gerrit is placed outside the if (integrationData.remote.enableGit) block, meaning it executes unconditionally. However, buildRepositoryPayloads requires a Git integration to exist - it calls IntegrationRepository.findByPlatform(PlatformType.GIT, ...) and throws Error400 if not found. When enableGit is false, no Git integration is created by gitConnectOrUpdate, so the subsequent mapUnifiedRepositories call will fail with "Git integration not found for segment". The mapUnifiedRepositories call should be inside the if (enableGit) block.

github-actions · 2025-12-31T10:40:23Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

backend/src/services/integrationService.ts

github-actions · 2025-12-31T15:04:32Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

cursor · 2025-12-31T15:08:22Z

backend/src/services/integrationService.ts

+    const [gitRepoIdMap, sourceIntegration] = await Promise.all([
+      // TODO: after migration, generate UUIDs instead of fetching from git.repositories
+      getGitRepositoryIdsByUrl(qx, urls),
+      isGitHubPlatform ? IntegrationRepository.findById(sourceIntegrationId, txOptions) : null,


GitLab forkedFrom data not synced to public.repositories

In buildRepositoryPayloads, the sourceIntegration is only fetched for GitHub platforms (line 3118 uses a conditional that returns null for non-GitHub). The forkedFromMap at lines 3157-3166 is populated exclusively from sourceIntegration?.settings?.orgs, which is GitHub-specific. For GitLab repositories, forkedFromMap remains empty, causing all GitLab repos to have forkedFrom: null in public.repositories. However, GitLab's forkedFrom data IS available and passed to gitConnectOrUpdate (lines 2873-2875, 2884-2886), so the information exists but isn't propagated to the unified repositories table.

Additional Locations (1)

backend/src/services/integrationService.ts#L3156-L3166

github-actions · 2025-12-31T15:10:42Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

github-actions · 2025-12-31T16:24:24Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

backend/src/services/integrationService.ts

cursor · 2025-12-31T16:31:48Z

backend/src/services/integrationService.ts

            remotes: repositories.map((url) => ({ url, forkedFrom: null })),
          },
          txOptions,
+          platform,


Missing repository sync in integration update method

The update method now passes platform to gitConnectOrUpdate which skips the internal mapUnifiedRepositories call. However, the update method doesn't add its own mapUnifiedRepositories call afterward. This creates a regression where updating code platform integrations syncs to git.repositories but no longer syncs to public.repositories, breaking the unified repository table consistency.

cursor · 2026-01-05T16:59:15Z

backend/src/services/integrationService.ts

+            const gitIntegration = await IntegrationRepository.findByPlatform(
+              PlatformType.GIT,
+              segmentOptions,
+            )


Integration destroy fails without Git integration

The call to IntegrationRepository.findByPlatform at line 433 lacks error handling. The original code wrapped this in a try-catch with shouldUpdateGit fallback to gracefully handle missing Git integrations. The new code assumes a Git integration always exists for GitHub/GitLab integrations. If a Git integration is missing (due to partial creation failures, manual deletion, or legacy data), findByPlatform throws an error, causing the entire destroy operation to fail and roll back instead of proceeding with cleanup.

cursor · 2026-01-05T16:59:15Z

services/libs/data-access-layer/src/integrations/index.ts

+      SELECT id, url FROM gitlab_repos
+      UNION ALL
+      SELECT id, url FROM git_repos
+    ) combined


DISTINCT ON query without ORDER BY is non-deterministic

The SQL query in syncRepositoriesToGitV2 uses DISTINCT ON (url) without a corresponding ORDER BY clause. When the same URL exists in multiple source tables (github_repos, gitlab_repos, git_repos) with potentially different IDs, PostgreSQL will return an arbitrary row. This could lead to inconsistent repository ID selection across different executions, though in practice this may only manifest if there's existing data inconsistency between the tables.

github-actions · 2026-01-06T10:20:35Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

github-actions · 2026-01-06T10:37:38Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

backend/src/services/integrationService.ts

frontend/src/config/integrations/git/components/git-params.vue

… consistency

mbani01 requested review from themarolt and ulemons December 30, 2025 16:50

mbani01 marked this pull request as ready for review December 30, 2025 16:50

cursor bot reviewed Dec 30, 2025

View reviewed changes

cursor bot reviewed Dec 31, 2025

View reviewed changes

backend/src/services/integrationService.ts Outdated Show resolved Hide resolved

cursor bot reviewed Dec 31, 2025

View reviewed changes

cursor bot reviewed Jan 5, 2026

View reviewed changes

mbani01 force-pushed the feat/integrationService_sync_to_repositories branch from 8f348bc to be61340 Compare January 6, 2026 10:20

mbani01 changed the title ~~feat: integration service sync to repositories~~ feat: integration service sync to repositories [CM-865] Jan 6, 2026

cursor bot reviewed Jan 6, 2026

View reviewed changes

backend/src/services/integrationService.ts Show resolved Hide resolved

frontend/src/config/integrations/git/components/git-params.vue Show resolved Hide resolved

mbani01 added 8 commits January 6, 2026 12:02

feat: implement DAL for new repositories tables

e10ac83

feat: implement repos insertions and enable it for github-nango

f6fc863

feat: enable unified mapping for rest of code platforms

3db29f3

feat: implement restore & delete

deab371

fix: lint

058d85b

fix: formatting

d9fa51b

feat: enable unified mapping for gitlab

15ea1a9

feat: validate repo ownership in deletions

e0c4da1

mbani01 added 9 commits January 6, 2026 12:02

fix: gerrit bug of overriding existing remotes in git integration

f2deee8

fix: always enable git integration for gerrit to ensure code platform…

57f09d3

… consistency

fix: transactions handling & ensure repo id consistency

2f05800

feat: unify get repositories by integration

07080eb

fix: handle edge case & cross integration update/delete

59f1aaa

feat: ui changes to use unified approach

782098c

feat: unify get repositories by integration

91574ed

fix: lint

8563997

chore: remove duplicates

442550f

mbani01 force-pushed the feat/integrationService_sync_to_repositories branch from 644a90c to 442550f Compare January 6, 2026 11:05

mbani01 self-assigned this Jan 6, 2026

mbani01 removed the request for review from themarolt January 6, 2026 11:39

feat: integration service sync to repositories [CM-865] #3722

Are you sure you want to change the base?

feat: integration service sync to repositories [CM-865] #3722

Uh oh!

Conversation

mbani01 commented Dec 30, 2025 • edited by cursor bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Dec 30, 2025

Uh oh!

github-actions bot commented Dec 30, 2025

Uh oh!

github-actions bot commented Dec 30, 2025

Uh oh!

github-actions bot commented Dec 30, 2025

Uh oh!

github-actions bot commented Dec 30, 2025

Uh oh!

cursor bot Dec 30, 2025

Choose a reason for hiding this comment

Missing repository sync for GitHub and GitLab integrations

Uh oh!

cursor bot Dec 30, 2025

Choose a reason for hiding this comment

Gerrit repository sync fails when Git disabled

Uh oh!

github-actions bot commented Dec 31, 2025

Uh oh!

Uh oh!

github-actions bot commented Dec 31, 2025

Uh oh!

cursor bot Dec 31, 2025

Choose a reason for hiding this comment

GitLab forkedFrom data not synced to public.repositories

Uh oh!

github-actions bot commented Dec 31, 2025

Uh oh!

github-actions bot commented Dec 31, 2025

Uh oh!

Uh oh!

cursor bot Dec 31, 2025

Choose a reason for hiding this comment

Missing repository sync in integration update method

Uh oh!

cursor bot Jan 5, 2026

Choose a reason for hiding this comment

Integration destroy fails without Git integration

Uh oh!

cursor bot Jan 5, 2026

Choose a reason for hiding this comment

DISTINCT ON query without ORDER BY is non-deterministic

Uh oh!

github-actions bot commented Jan 6, 2026

Uh oh!

github-actions bot commented Jan 6, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mbani01 commented Dec 30, 2025 •

edited by cursor bot

Loading