70 dockstore refactor into dockstat repository #72
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit introduces a new `@dockstat/repo-cli` package, providing a command-line interface to assist with managing DockStat repositories. Key functionalities include: - `init`: Initialize a new repository configuration file. - `bundle`: Bundle plugins within the repository and update the manifest. - `badges`: Generate SVG badges based on repository content and configuration. Additionally, a new `isRepoType` utility is added to `@dockstat/utils` for validating repository types.
…onfig Deprecate and remove the internal `apps/dockstore` tooling responsible for plugin bundling, README generation, and schema management. This includes `bundler.ts`, `generate-readme.ts`, `.schemas/plugin-meta.schema.json`, and related `package.json` scripts and config files. Replaced the decentralized plugin metadata and manifest generation with a new, centralized `apps/dockstore/repo.json` file. This file now defines the repository's configuration (name, policy, verification_api, content directories) and serves as the single source of truth for repository content metadata (plugins, themes, stacks). The `dockstat-repo-cli`'s `bundle` command has been updated to populate this new `repo.json` file with plugin metadata. Added a new `serve` command to `dockstat-repo-cli`, enabling simple local serving of the repository's static files via `Bun.serve`. This is intended for development, testing, or use behind a reverse proxy. Introduced `packages/dockstat-repo-cli/src/utils/contentType.ts` to support the new `serve` command by providing appropriate `Content-Type` headers.
Add .npmignore to exclude source files and build artifacts from published package. Update package.json with version, description, and 'files' array for explicit publication control. Correct the CLI executable name in the 'bin' field from 'dockstore-repo-cli' to 'dockstat-repo-cli'.
Contributor
Reviewer's GuideIntroduces a new Sequence diagram for bundle command plugin build and repository updatesequenceDiagram
actor User
participant CLI as dockstat_repo_CLI
participant BundleCmd as bundleCommand
participant RepoUtils as loadRepo_saveRepo
participant BunRuntime as Bun
participant Extract as extractMeta
participant AJV as validateMeta
User->>CLI: run dockstat-repo bundle --root repo.json
CLI->>BundleCmd: execute action(options, cmd)
BundleCmd->>RepoUtils: loadRepo(root)
RepoUtils-->>BundleCmd: RepoFile or null
alt repo file missing
BundleCmd-->>User: error and exit(1)
else repo file present
BundleCmd->>BunRuntime: scan plugins (Glob)
BunRuntime-->>BundleCmd: pluginPaths[]
loop for each pluginPath
BundleCmd->>BunRuntime: build({entrypoints,outdir,...})
BunRuntime-->>BundleCmd: BuildResult
BundleCmd->>BunRuntime: import built index.js
BunRuntime-->>BundleCmd: plugin module
BundleCmd->>Extract: extractMeta(plugin or plugin.build())
Extract-->>BundleCmd: PluginMeta
BundleCmd->>AJV: validateMeta(PluginMeta)
alt valid
AJV-->>BundleCmd: true
BundleCmd->>BundleCmd: record success with meta
else invalid
AJV-->>BundleCmd: false
BundleCmd->>BundleCmd: record failure with error
end
end
alt schema option provided
BundleCmd->>BunRuntime: write(schemaPath, PluginMeta schema)
BunRuntime-->>BundleCmd: ok
end
BundleCmd->>RepoUtils: saveRepo(root, updated RepoFile)
RepoUtils-->>BundleCmd: ok
BundleCmd->>BundleCmd: compute success and failure counts
BundleCmd-->>User: summary and exit (nonzero if failures)
end
Sequence diagram for badges command badge generation workflowsequenceDiagram
actor User
participant CLI as dockstat_repo_CLI
participant BadgesCmd as badgesCommand
participant RepoUtils as loadRepo
participant BadgeUtils as createBadge
participant Logger as log
participant BunRuntime as Bun
User->>CLI: run dockstat-repo badges --root repo.json
CLI->>BadgesCmd: execute action(options, cmd)
BadgesCmd->>RepoUtils: loadRepo(root)
RepoUtils-->>BadgesCmd: RepoFile or null
alt repo file missing
BadgesCmd-->>User: error and exit(1)
else repo file present
BadgesCmd->>BadgesCmd: derive style and outputDir
BadgesCmd->>BadgesCmd: initialize badges list
alt plugins option enabled
BadgesCmd->>BadgeUtils: createBadge({label plugins, message count, icon puzzle})
BadgeUtils-->>BadgesCmd: svg
end
alt themes option enabled
BadgesCmd->>BadgeUtils: createBadge({label themes, message count, icon palette})
BadgeUtils-->>BadgesCmd: svg
end
alt stacks option enabled
BadgesCmd->>BadgeUtils: createBadge({label stacks, message count, icon layers})
BadgeUtils-->>BadgesCmd: svg
end
alt type option enabled
BadgesCmd->>BadgeUtils: createBadge({label repo, message config.type, icon from typeConfig})
BadgeUtils-->>BadgesCmd: svg
end
alt status option enabled
BadgesCmd->>BadgeUtils: createBadge({label policy, message config.policy, icon shield or shieldCheck})
BadgeUtils-->>BadgesCmd: svg
end
loop for each badge
BadgesCmd->>BunRuntime: write(outputDir/name.svg, svg)
BunRuntime-->>BadgesCmd: ok
BadgesCmd->>Logger: log("✅", Created badge, path)
end
BadgesCmd->>BadgesCmd: build markdown snippet and README content
BadgesCmd->>BunRuntime: write(outputDir/README.md, content)
BunRuntime-->>BadgesCmd: ok
BadgesCmd-->>User: print completion message
end
Class diagram for new repo-cli core types and utilitiesclassDiagram
direction LR
class Opts {
+string root
+string name
+string policy
+string type
+object themes
+object plugins
+object stacks
}
class ThemesConfig {
+string dir
}
class PluginsConfig {
+string dir
+string bundle
}
class StacksConfig {
+string dir
}
class RepoFile {
+Opts config
+RepoContent content
}
class RepoContent {
+PluginMetaType[] plugins
+unknown[] themes
+unknown[] stacks
}
class BuildResult {
+string name
+boolean success
+PluginMetaType meta
+string error
}
class BadgeOptions {
+string label
+string message
+string color
+string labelColor
+string style
+string icon
}
class BadgeOptionsWithIcon {
+string label
+string message
+string color
+string labelColor
+string style
+IconName icon
}
class IconName {
}
class PluginMetaType {
}
class CreateRepoType {
+string name
+string type
+string policy
}
class RepoType {
+string type
}
class RepoUtils {
+loadRepo(path string) RepoFile
+saveRepo(path string, data RepoFile) void
+extractMeta(plugin Record) PluginMetaType
+isRepoType(str string) boolean
}
class BadgeUtils {
+createBadge(options BadgeOptionsWithIcon) string
}
class LoggerUtils {
+log(icon string, message string, detail string) void
+logError(title string, error unknown) void
+printSummary(succeeded number, failed number) void
}
Opts o-- ThemesConfig : has
Opts o-- PluginsConfig : has
Opts o-- StacksConfig : has
RepoFile o-- RepoContent : has
RepoContent --> PluginMetaType : contains
BadgeOptionsWithIcon --|> BadgeOptions : extends
BadgeOptionsWithIcon --> IconName : uses
RepoUtils ..> RepoFile : reads_writes
RepoUtils ..> PluginMetaType : extracts
RepoUtils ..> RepoType : validates
BadgeUtils ..> BadgeOptionsWithIcon : uses
LoggerUtils ..> BuildResult : reports
Opts ..|> CreateRepoType : based_on
File-Level Changes
Possibly linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Hey - I've found 7 issues, and left some high level feedback:
- The allowed repository types are inconsistent (
isRepoTypeacceptsdefaultbut the CLI help and error message only list github/gitlab/gitea/http/local); consider aligning the enum and user-facing messaging so the valid values are clear and enforced consistently. - In
serveCommand, theportoption is treated as a string and passed directly toBun.serve; parsing it to a number and validating it up front would avoid surprising behavior and give clearer error messages for invalid ports. - The typing for badge
iconis a bit loose (BadgeOptionsusesstringwhilecreateBadgeexpectsIconName); tightening this to consistently useIconNamewould catch invalid icon names at compile time and remove the need for casts.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- The allowed repository types are inconsistent (`isRepoType` accepts `default` but the CLI help and error message only list github/gitlab/gitea/http/local); consider aligning the enum and user-facing messaging so the valid values are clear and enforced consistently.
- In `serveCommand`, the `port` option is treated as a string and passed directly to `Bun.serve`; parsing it to a number and validating it up front would avoid surprising behavior and give clearer error messages for invalid ports.
- The typing for badge `icon` is a bit loose (`BadgeOptions` uses `string` while `createBadge` expects `IconName`); tightening this to consistently use `IconName` would catch invalid icon names at compile time and remove the need for casts.
## Individual Comments
### Comment 1
<location> `packages/repo-cli/src/commands/bundle.ts:56` </location>
<code_context>
- minify: true,
- sourcemap: "external",
- splitting: false,
- env: `${name.toUpperCase()}_*`,
- banner: "/* Bundled by DockStore */",
- target: "bun",
</code_context>
<issue_to_address>
**issue (bug_risk):** The `env` option for `Bun.build` is likely mis-typed here and may not behave as intended.
`env` should be an object map or an array of variable names, not a template string. A value like ``"FOO_*"`` is unlikely to work as a pattern and may be ignored. If you want to forward vars with this prefix, construct an explicit map from `process.env` or verify that Bun supports this syntax; otherwise, adjust or remove this parameter to avoid unexpected build behavior.
</issue_to_address>
### Comment 2
<location> `packages/repo-cli/src/commands/init.ts:15-19` </location>
<code_context>
+ .requiredOption("-n, --name <name>", "The name of the repository")
+ .option("-r, --relaxed", "Use relaxed verification", false)
+ .option("-a, --verification-api <URL>", "Verification API base URL", undefined)
+ .requiredOption("-v, --variant <type>", "Repository type (github, gitlab, gitea, http, local)")
+ .action(async (options, cmd) => {
+ const globalOptions = cmd.optsWithGlobals() as unknown as { root: string }
+
+ if (!repo.isRepoType(options.variant)) {
+ console.error("❌ Invalid variant. Use: github, gitlab, gitea, http, or local")
+ process.exit(1)
</code_context>
<issue_to_address>
**issue:** CLI help and validation for `variant` is slightly inconsistent with the allowed repo types.
`isRepoType` accepts `"default"`, but the CLI help and error message only list `github, gitlab, gitea, http, local`. Please align the accepted values, help text, and error message (either document `default` or stop accepting it) so users have a single, consistent set of allowed types.
</issue_to_address>
### Comment 3
<location> `packages/repo-cli/src/types.ts:28-35` </location>
<code_context>
+}
+
+// cli/types.ts (add or update)
+export interface BadgeOptions {
+ label: string
+ message: string
+ color: string
+ labelColor?: string
+ style?: "flat" | "flat-square"
+ icon?: string
+}
</code_context>
<issue_to_address>
**suggestion:** The `icon` type on `BadgeOptions` conflicts with `BadgeOptionsWithIcon` and `IconName`.
Here `BadgeOptionsWithIcon` narrows `icon` to `IconName`, but `BadgeOptions` uses `icon?: string`, which weakens type safety. Consider either removing `icon` from `BadgeOptions` and only defining it on `BadgeOptionsWithIcon`, or changing `BadgeOptions.icon` to also be `IconName` so the types stay consistent and catch invalid icon names.
```suggestion
export interface BadgeOptions {
label: string
message: string
color: string
labelColor?: string
style?: "flat" | "flat-square"
icon?: IconName
}
```
</issue_to_address>
### Comment 4
<location> `packages/repo-cli/src/utils/contentType.ts:8-11` </location>
<code_context>
+ return "application/yaml"
+ case ".txt":
+ case ".log":
+ case ".md":
+ return "text/plain"
+ default:
+ return "text/plain"
</code_context>
<issue_to_address>
**suggestion (bug_risk):** Missing explicit content type for `.svg` files means badges will be served as plain text.
For SVG badges generated by this CLI, `serve` will currently fall through to the default and send them as `text/plain`. Please add an explicit `.svg` case that returns `image/svg+xml` so that browsers handle rendering and caching correctly.
```suggestion
case ".txt":
case ".log":
case ".md":
return "text/plain"
case ".svg":
return "image/svg+xml"
```
</issue_to_address>
### Comment 5
<location> `packages/repo-cli/src/commands/serve.ts:18-26` </location>
<code_context>
+
+ console.log("Received request:", req.url)
+
+ let filePath = decodeURIComponent(url.pathname)
+
+ if (filePath.charAt(0) === "/") {
+ filePath = filePath.replace("/", "")
+ }
+
+ console.debug("File Path:", filePath)
+
+ const file = Bun.file(filePath)
+ if (!(await file.exists())) {
+ return new Response("Not found", { status: 404 })
</code_context>
<issue_to_address>
**🚨 issue (security):** Serving arbitrary paths from the current working directory without normalization allows path traversal.
The pathname is used directly as a relative filesystem path with only a leading slash removed, so `../` segments can escape the intended directory. Even for a test server, this is a path traversal risk. Please restrict access to a fixed root directory and normalize/validate the resolved path to ensure it cannot point outside that root.
</issue_to_address>
### Comment 6
<location> `packages/repo-cli/README.md:15` </location>
<code_context>
+bun run src/index.ts
+```
+
+This project was created using `bun init` in bun v1.3.1. [Bun](https://bun.com) is a fast all-in-one JavaScript runtime.
</code_context>
<issue_to_address>
**issue (typo):** The Bun website URL appears to be incorrect; consider updating it to the official Bun site.
The official Bun runtime site is https://bun.sh, not https://bun.com. Please update the link so users reach the correct documentation and homepage.
</issue_to_address>
### Comment 7
<location> `packages/repo-cli/src/commands/badges.ts:35` </location>
<code_context>
+
+ const badges: { name: string; svg: string }[] = []
+
+ if (options.plugins) {
+ const count = content.plugins.length
+ badges.push({
</code_context>
<issue_to_address>
**issue (complexity):** Consider refactoring the repeated plugins/themes/stacks badge logic into a single configuration-driven loop to avoid duplication and centralize future changes.
The per‑badge blocks for `plugins`, `themes`, and `stacks` are nearly identical and can be made more declarative without changing behavior.
You can introduce a small config array and a single loop to generate these badges, which will make future changes (e.g. new style options or icon changes) localized:
```ts
// Right after `const badges: { name: string; svg: string }[] = []`
type CountBadgeKey = "plugins" | "themes" | "stacks"
const countBadgeConfigs: {
optionKey: CountBadgeKey
name: string
label: string
color: string
icon: IconName
getCount: () => number
}[] = [
{
optionKey: "plugins",
name: "plugins",
label: "plugins",
color: COLORS.blue,
icon: "puzzle",
getCount: () => content.plugins.length,
},
{
optionKey: "themes",
name: "themes",
label: "themes",
color: COLORS.purple,
icon: "palette",
getCount: () => content.themes.length,
},
{
optionKey: "stacks",
name: "stacks",
label: "stacks",
color: COLORS.teal,
icon: "layers",
getCount: () => content.stacks.length,
},
]
for (const cfg of countBadgeConfigs) {
if (!options[cfg.optionKey]) continue
const count = cfg.getCount()
badges.push({
name: cfg.name,
svg: createBadge({
label: cfg.label,
message: count.toString(),
color: count > 0 ? cfg.color : COLORS.lightgrey,
icon: cfg.icon,
style,
}),
})
}
```
Then you can safely remove the three separate `if (options.plugins) { ... }`, `if (options.themes) { ... }`, and `if (options.stacks) { ... }` blocks.
This keeps all existing behavior (including color choice when count is 0) while reducing duplication and making it easier to add or tweak count‑based badges later.
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
Comment on lines
+28
to
+35
| export interface BadgeOptions { | ||
| label: string | ||
| message: string | ||
| color: string | ||
| labelColor?: string | ||
| style?: "flat" | "flat-square" | ||
| icon?: string | ||
| } |
Contributor
There was a problem hiding this comment.
suggestion: The icon type on BadgeOptions conflicts with BadgeOptionsWithIcon and IconName.
Here BadgeOptionsWithIcon narrows icon to IconName, but BadgeOptions uses icon?: string, which weakens type safety. Consider either removing icon from BadgeOptions and only defining it on BadgeOptionsWithIcon, or changing BadgeOptions.icon to also be IconName so the types stay consistent and catch invalid icon names.
Suggested change
| export interface BadgeOptions { | |
| label: string | |
| message: string | |
| color: string | |
| labelColor?: string | |
| style?: "flat" | "flat-square" | |
| icon?: string | |
| } | |
| export interface BadgeOptions { | |
| label: string | |
| message: string | |
| color: string | |
| labelColor?: string | |
| style?: "flat" | "flat-square" | |
| icon?: IconName | |
| } |
refactor(repo-cli/badges): Consolidate badge creation logic
- Replaced repetitive conditional blocks for plugins, themes, and stacks with a configurable array and a loop.
- Improves code maintainability and extensibility for future badge types.
fix(repo-cli/serve): Prevent directory traversal attacks
- Added `filePath.replaceAll("../", "/")` to sanitize requested file paths.
- Mitigates a potential security vulnerability allowing access to files outside the intended content directory.
refactor(typings, utils): Remove 'default' repo type
- Removed "default" from the `Repo.type` union enum in `db.ts`.
- Removed "default" from the `REPO_TYPES` array in `isRepoType.ts`.
- Aligns type definitions and utility functions with current supported repository types.
This commit significantly overhauls the repository registration process, moving towards a manifest-driven approach.
**Key Changes:**
* **API (`apps/api`)**:
* The `POST /repositories` endpoint now accepts a single `link_to_manifest` (URL) instead of individual fields like name, source, type, and policy.
* The API fetches the manifest from the provided link, utilizes new utility functions to parse its content, and extracts the repository configuration for storage.
* Adds `@dockstat/repo-cli` dependency to facilitate manifest parsing.
* **Core Logic (`packages/utils/repo`)**:
* Introduces `parseRawToDB` to intelligently convert raw manifest URLs (GitHub, GitLab, Gitea, HTTP) into structured database `type` and `source` strings.
* Enhances `parseFromDBToRepoLink` for more robust URL reconstruction based on repository type and source.
* Adds helper functions `parseRepoParts` and `splitDomain` for robust URL decomposition.
* **Database (`packages/db`)**:
* Updates the `repositories_table` schema to include a `paths` column (JSON type) for storing manifest-defined paths for themes, plugins, and stacks.
* Enables automatic database backup functionality with configurable intervals, compression, and maximum backups. Backups are stored in a `.backups` directory.
* **Frontend (`apps/dockstat`)**:
* The "Add Repository" form in the Extensions page is simplified to a single input field for the raw manifest or repository link.
* Removes previously individual fields for name, source, type, policy, and verification API.
* **Typings**:
* `Repo` and `CreateRepo` types updated to include the new `paths` field and remove the "default" option from the `type` enum.
* `RepoFile` interface in `@dockstat/repo-cli` is extended for flexibility.
* **.gitignore**: Updated to exclude the `.backups` directory.
This refactor streamlines the user experience for adding new repositories and prepares the system for richer manifest-based capabilities.
…e publishing - Introduce the new `DockStacks` plugin for managing Docker container stacks. This includes its initial project setup, definition using `@dockstat/plugin-builder`, and configuration types. - Update the API route (`apps/api/src/routes/db.ts`) to store specific `paths` (plugins, stacks, themes) in the repository configuration upon creation. This allows repositories to define structured locations for different asset types. - Standardize the package publishing process across core utility packages (`@dockstat/plugin-builder`, `@dockstat/repo-cli`, `@dockstat/sqlite-wrapper`, `@dockstat/typings`, `@dockstat/utils`) by adding a dedicated `publish` script to each `package.json`. - Add root-level `publish` and `pub` commands to the monorepo `package.json` for simplified execution of package publishing. - Bump versions for `@dockstat/plugin-builder` and `@dockstat/typings`.
Replaces the previous subproject entry with a new implementation of the DockNode application. This commit establishes the foundational structure for DockNode, leveraging Bun runtime and the Elysia web framework. Key components include: - Integration with @dockstat/sqlite-wrapper for local database management. - A StackHandler for dynamic creation, configuration, and orchestration of Docker Compose stacks. - Essential project configuration such as .gitignore, tsconfig, and package.json.
Reordered import statements in `apps/docknode/src/stacks/index.ts`. Standardized comment placement and property delimiters (commas vs. semicolons) in `apps/docknode/tsconfig.json` and `packages/typings/src/v2/extensions/stacks/compose.d.ts`. These changes improve code consistency and readability.
Adds a comprehensive Docker Compose setup for consistent local development, coupled with significant ecosystem and dependency updates.
- **Docker Development Environment:**
- Introduced `docker-compose.dev.yaml` to provide isolated services:
- `socket-proxy`: For secure Docker socket access.
- `sqlite-web`: Two instances for inspecting `dockstat.sqlite` and `verification.db`.
- `prometheus`: For collecting application metrics.
- Added `dockstat-dev.prometheus.yml` to configure Prometheus to scrape metrics from the API.
- Updated `package.json` scripts (`docker-up`, `docker-up:quiet`, `dev:dockstat`) to integrate with the new Docker setup.
- **Bun and Dependency Updates:**
- Upgraded Bun runtime from `1.3.1` to `1.3.6`.
- Migrated core dependencies (`elysia`, `@elysiajs/eden`) to Bun's package catalog for enhanced dependency management and resolution.
- Performed various dependency updates across the monorepo, including `turbo`, `typescript`, `@types/bun`, `@types/react`, `js-yaml`, `@elysiajs/openapi`, `rollup`, `@sinclair/typebox`, `docker-compose`, `framer-motion`, `storybook`, and `@typescript-eslint` packages.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary by Sourcery
Introduce a new DockStat repository CLI package and remove the old dockstore app, centralizing repository management and plugin bundling into a reusable tool.
New Features:
Enhancements:
Documentation: