Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
90509f1
marge master to devel (v1.0.0) (#910)
njzjz Oct 21, 2025
7af8d74
[pre-commit.ci] pre-commit autoupdate (#911)
pre-commit-ci[bot] Oct 28, 2025
e10d0b7
[pre-commit.ci] pre-commit autoupdate (#913)
pre-commit-ci[bot] Nov 5, 2025
47d3d83
[pre-commit.ci] pre-commit autoupdate (#914)
pre-commit-ci[bot] Nov 18, 2025
5e7a335
[pre-commit.ci] pre-commit autoupdate (#915)
pre-commit-ci[bot] Nov 25, 2025
4a84d65
chore(deps): bump actions/checkout from 5 to 6 (#916)
dependabot[bot] Nov 25, 2025
551a1a7
[pre-commit.ci] pre-commit autoupdate (#917)
pre-commit-ci[bot] Dec 2, 2025
a714737
[pre-commit.ci] pre-commit autoupdate (#918)
pre-commit-ci[bot] Dec 10, 2025
6403adc
[pre-commit.ci] pre-commit autoupdate (#919)
pre-commit-ci[bot] Dec 16, 2025
2fa782d
[pre-commit.ci] pre-commit autoupdate (#920)
pre-commit-ci[bot] Dec 23, 2025
6edbd09
[pre-commit.ci] pre-commit autoupdate (#923)
pre-commit-ci[bot] Jan 13, 2026
c396ce7
fix(cp2k): extract scalar from numpy array for math.isclose() compari…
Copilot Jan 14, 2026
61a0cc6
[pre-commit.ci] pre-commit autoupdate (#925)
pre-commit-ci[bot] Jan 20, 2026
9608254
ci: use OIDC for codecov-action (#926)
njzjz-bot Jan 26, 2026
2eadf39
[pre-commit.ci] pre-commit autoupdate (#927)
pre-commit-ci[bot] Jan 28, 2026
8cd814d
implement support for LMDB
Feb 1, 2026
f4c760b
update docstr
Feb 1, 2026
1e347e8
fix example
Feb 1, 2026
eceb452
fix unclosed open
Feb 1, 2026
503a03e
add error handeling
Feb 1, 2026
d3ac7c3
make map_size be argument
Feb 1, 2026
0da5c38
mv implementation to dpdata/lmdb
Feb 1, 2026
b3715c8
add init to module
Feb 1, 2026
a2dcd4a
support non-default data
Feb 2, 2026
56fa9ef
add test case for natom axis
Feb 2, 2026
248ef5a
customize the string format for frame index
Feb 2, 2026
5b39e60
better error handeling
Feb 2, 2026
9ae65de
Merge branch 'devel' into feat-fmt-lmdb
Feb 2, 2026
d2eb6df
support standard dpdata multisystems API
Feb 2, 2026
b3cd20b
feat: index systems in MultiSystems by short names (#929)
njzjz Feb 5, 2026
82ae868
Feat: the LMDB file format (#928)
iProzd Feb 5, 2026
4ae8b2c
[pre-commit.ci] pre-commit autoupdate (#930)
pre-commit-ci[bot] Feb 10, 2026
8cfafc8
feat: support natom padding in deepmd/npy/mixed format (#932)
wanghan-iapcm Feb 14, 2026
58be017
Revert "CI: pin numpy to <2.3 & rdkit to <2025.3.3" (#921)
njzjz Feb 20, 2026
d6a1c81
[pre-commit.ci] pre-commit autoupdate (#935)
pre-commit-ci[bot] Feb 20, 2026
d6b44e7
chore(deps): bump jakebailey/pyright-action from 2 to 3 (#936)
dependabot[bot] Feb 20, 2026
10312fb
feat: add agent skill for dpdata CLI usage (#938)
njzjz-bot Feb 25, 2026
65a97e9
chore: update branch references from devel to master (#937)
njzjz-bot Feb 25, 2026
22bba14
[pre-commit.ci] pre-commit autoupdate (#939)
pre-commit-ci[bot] Feb 25, 2026
ae134fe
chore(pre-commit): use mdformat to format Markdown (#941)
njzjz-bot Feb 26, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .github/dependabot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,3 @@ updates:
directory: "/"
schedule:
interval: "weekly"
target-branch: "devel"
2 changes: 1 addition & 1 deletion .github/workflows/benchmark.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ jobs:
if: ${{ github.repository_owner == 'deepmodeling' }}
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: actions/checkout@v6
- name: Set up Python
uses: actions/setup-python@v6
with:
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/pyright.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ jobs:
with:
python-version: '3.12'
- run: pip install uv
- run: uv pip install --system -e .[amber,ase,pymatgen] 'rdkit<2025.3.3' openbabel-wheel
- uses: jakebailey/pyright-action@v2
- run: uv pip install --system -e .[amber,ase,pymatgen] rdkit openbabel-wheel
- uses: jakebailey/pyright-action@v3
with:
version: 1.1.363
12 changes: 8 additions & 4 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
name: Python package

on:
- push
- pull_request
- push
- pull_request

jobs:
build:
Expand All @@ -12,7 +12,7 @@ jobs:
python-version: ["3.8", "3.12"]

steps:
- uses: actions/checkout@v5
- uses: actions/checkout@v6
# set up conda
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v6
Expand All @@ -26,13 +26,15 @@ jobs:
**/pyproject.toml
cache-suffix: "py${{ matrix.python-version }}"
- name: Install dependencies
run: uv pip install --system .[test,amber,ase,pymatgen] coverage ./tests/plugin 'rdkit<2025.3.3' openbabel-wheel 'numpy<2.3'
run: uv pip install --system .[test,amber,ase,pymatgen] coverage ./tests/plugin rdkit openbabel-wheel
- name: Test
run: cd tests && coverage run --source=../dpdata -m unittest && cd .. && coverage combine tests/.coverage && coverage report
- name: Run codecov
uses: codecov/codecov-action@v5
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
with:
use_oidc: true
pass:
needs: [build]
runs-on: ubuntu-latest
Expand All @@ -42,3 +44,5 @@ jobs:
uses: re-actors/alls-green@release/v1
with:
jobs: ${{ toJSON(needs) }}
permissions:
id-token: write
2 changes: 1 addition & 1 deletion .github/workflows/test_import.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: actions/checkout@v6
- uses: actions/setup-python@v6
with:
python-version: '3.9'
Expand Down
22 changes: 15 additions & 7 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ repos:
# Python
- repo: https://github.com/astral-sh/ruff-pre-commit
# Ruff version.
rev: v0.14.1
rev: v0.15.2
hooks:
- id: ruff
args: ["--fix"]
Expand All @@ -34,10 +34,18 @@ repos:
hooks:
- id: velin
args: ["--write"]
# Python inside docs
- repo: https://github.com/asottile/blacken-docs
rev: 1.20.0
# markdown
- repo: https://github.com/hukkin/mdformat
rev: 1.0.0
hooks:
- id: blacken-docs
ci:
autoupdate_branch: devel
- id: mdformat
exclude: "^tests/.*$"
additional_dependencies:
# - mdformat-myst==0.3.0
# See https://github.com/executablebooks/mdformat-myst/issues/13
- "git+https://github.com/njzjz-bothub/mdformat-myst@d9c414e#egg=mdformat-myst"
- mdformat-ruff==0.1.3
- mdformat-web==0.2.0
- mdformat-config==0.2.1
- mdformat-beautysh==1.0.0
- mdformat-gfm-alerts==2.0.0
21 changes: 19 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,51 +7,60 @@ Always reference these instructions first and fallback to search or bash command
## Working Effectively

- **Bootstrap and install the repository:**

- `cd /home/runner/work/dpdata/dpdata` (or wherever the repo is cloned)
- `uv pip install -e .` -- installs dpdata in development mode with core dependencies (numpy, scipy, h5py, monty, wcmatch)
- Test installation: `dpdata --version` -- should show version like "dpdata v0.1.dev2+..."

- **Run tests:**

- `cd tests && python -m unittest discover` -- runs all 1826 tests in ~10 seconds. NEVER CANCEL.
- `cd tests && python -m unittest test_<module>.py` -- run specific test modules (individual modules take ~0.5 seconds)
- `cd tests && coverage run --source=../dpdata -m unittest discover && coverage report` -- run tests with coverage

- **Linting and formatting:**

- Install ruff: `uv pip install ruff`
- `ruff check dpdata/` -- lint the main package (takes ~1 second)
- `ruff format dpdata/` -- format code according to project style
- `ruff check --fix dpdata/` -- auto-fix linting issues where possible

- **Pre-commit hooks:**

- Install: `uv pip install pre-commit`
- `pre-commit run --all-files` -- run all hooks on all files
- Hooks include: ruff linting/formatting, trailing whitespace, end-of-file-fixer, yaml/json/toml checks

## Validation

- **Always test CLI functionality after making changes:**

- `dpdata --help` -- ensure CLI still works
- `dpdata --version` -- verify version is correct
- Test a basic conversion if sample data is available

- **Always run linting before committing:**

- `ruff check dpdata/` -- ensure no new linting errors
- `ruff format dpdata/` -- ensure code is properly formatted

- **Run relevant tests for your changes:**

- For format-specific changes: `cd tests && python -m unittest test_<format>*.py`
- For core system changes: `cd tests && python -m unittest test_system*.py test_multisystems.py`
- For CLI changes: `cd tests && python -m unittest test_cli.py` (if exists)

## Build and Documentation

- **Documentation:**

- `cd docs && make help` -- see all available build targets
- `cd docs && make html` -- build HTML documentation (requires additional dependencies)
- Documentation source is in `docs/` directory using Sphinx
- **NOTE:** Full docs build requires additional dependencies like `deepmodeling-sphinx` that may not be readily available

- **Package building:**

- Uses setuptools with pyproject.toml configuration
- `uv pip install build && python -m build` -- create source and wheel distributions
- Version is managed by setuptools_scm from git tags
Expand All @@ -61,6 +70,7 @@ Always reference these instructions first and fallback to search or bash command
The following are outputs from frequently run commands. Reference them instead of re-running to save time.

### Repository structure

```
/home/runner/work/dpdata/dpdata/
├── dpdata/ # Main package code
Expand All @@ -82,39 +92,46 @@ The following are outputs from frequently run commands. Reference them instead o
```

### Key dependencies

- Core: numpy>=1.14.3, scipy, h5py, monty, wcmatch
- Optional: ase (ASE integration), parmed (AMBER), pymatgen (Materials Project), rdkit (molecular analysis)
- Testing: unittest (built-in), coverage
- Linting: ruff
- Docs: sphinx with various extensions

### Test timing expectations

- Full test suite: ~10 seconds (1826 tests). NEVER CANCEL.
- Individual test modules: ~0.5 seconds
- Linting with ruff: ~1 second
- Documentation build: ~30 seconds

### Common workflows

1. **Adding a new format:**

- Create module in `dpdata/<format>/`
- Implement format classes inheriting from appropriate base classes
- Add tests in `tests/test_<format>*.py`
- Register format in the plugin system

2. **Fixing bugs:**
1. **Fixing bugs:**

- Write test that reproduces the bug first
- Make minimal fix to pass the test
- Run full test suite to ensure no regressions
- Run linting to ensure code style compliance

3. **CLI changes:**
1. **CLI changes:**

- Modify `dpdata/cli.py`
- Test with `dpdata --help` and specific commands
- Add/update tests if needed

## Troubleshooting

- **Installation timeouts:** Network timeouts during `uv pip install` are common. If this occurs, try:

- Individual package installation: `uv pip install numpy scipy h5py monty wcmatch`
- Use `--timeout` option: `uv pip install --timeout 300 -e .`
- Verify existing installation works: `dpdata --version` should work even if reinstall fails
Expand Down
20 changes: 14 additions & 6 deletions docs/systems/bond_order_system.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@

## BondOrderSystem

A new class {class}`BondOrderSystem <dpdata.BondOrderSystem>` which inherits from class {class}`System <dpdata.System>` is introduced in dpdata. This new class contains information of chemical bonds and formal charges (stored in `BondOrderSystem.data['bonds']`, `BondOrderSystem.data['formal_charges']`). Now BondOrderSystem can only read from .mol/.sdf formats, because of its dependency on rdkit (which means rdkit must be installed if you want to use this function). Other formats, such as pdb, must be converted to .mol/.sdf format (maybe with software like open babel).

```python
import dpdata

Expand All @@ -11,8 +12,10 @@ system_2 = dpdata.BondOrderSystem(
"tests/bond_order/methane.sdf", fmt="sdf"
) # read from .sdf file
```

In sdf file, all molecules must be of the same topology (i.e. conformers of the same molecular configuration).
`BondOrderSystem <dpdata.BondOrderSystem>` also supports initialize from a {class}`rdkit.Chem.rdchem.Mol` object directly.

```python
from rdkit import Chem
from rdkit.Chem import AllChem
Expand All @@ -25,21 +28,26 @@ system = dpdata.BondOrderSystem(rdkit_mol=mol)
```

### Bond Order Assignment

The {class}`BondOrderSystem <dpdata.BondOrderSystem>` implements a more robust sanitize procedure for rdkit Mol, as defined in {class}`dpdata.rdkit.santizie.Sanitizer`. This class defines 3 level of sanitization process by: low, medium and high. (default is medium).
+ low: use `rdkit.Chem.SanitizeMol()` function to sanitize molecule.
+ medium: before using rdkit, the programm will first assign formal charge of each atom to avoid inappropriate valence exceptions. However, this mode requires the rightness of the bond order information in the given molecule.
+ high: the program will try to fix inappropriate bond orders in aromatic hetreocycles, phosphate, sulfate, carboxyl, nitro, nitrine, guanidine groups. If this procedure fails to sanitize the given molecule, the program will then try to call `obabel` to pre-process the mol and repeat the sanitization procedure. **That is to say, if you wan't to use this level of sanitization, please ensure `obabel` is installed in the environment.**
According to our test, our sanitization procedure can successfully read 4852 small molecules in the PDBBind-refined-set. It is necessary to point out that the in the molecule file (mol/sdf), the number of explicit hydrogens has to be correct. Thus, we recommend to use
`obabel xxx -O xxx -h` to pre-process the file. The reason why we do not implement this hydrogen-adding procedure in dpdata is that we can not ensure its correctness.

- low: use `rdkit.Chem.SanitizeMol()` function to sanitize molecule.
- medium: before using rdkit, the programm will first assign formal charge of each atom to avoid inappropriate valence exceptions. However, this mode requires the rightness of the bond order information in the given molecule.
- high: the program will try to fix inappropriate bond orders in aromatic hetreocycles, phosphate, sulfate, carboxyl, nitro, nitrine, guanidine groups. If this procedure fails to sanitize the given molecule, the program will then try to call `obabel` to pre-process the mol and repeat the sanitization procedure. **That is to say, if you wan't to use this level of sanitization, please ensure `obabel` is installed in the environment.**
According to our test, our sanitization procedure can successfully read 4852 small molecules in the PDBBind-refined-set. It is necessary to point out that the in the molecule file (mol/sdf), the number of explicit hydrogens has to be correct. Thus, we recommend to use
`obabel xxx -O xxx -h` to pre-process the file. The reason why we do not implement this hydrogen-adding procedure in dpdata is that we can not ensure its correctness.

```python
import dpdata

for sdf_file in glob.glob("bond_order/refined-set-ligands/obabel/*sdf"):
syst = dpdata.BondOrderSystem(sdf_file, sanitize_level="high", verbose=False)
```

### Formal Charge Assignment

BondOrderSystem implement a method to assign formal charge for each atom based on the 8-electron rule (see below). Note that it only supports common elements in bio-system: B,C,N,O,P,S,As

```python
import dpdata

Expand Down
2 changes: 2 additions & 0 deletions docs/systems/mixed.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,15 @@ This also helps to mixture the type information together for model training with
Here are examples using `deepmd/npy/mixed` format:

- Dump a MultiSystems into a mixed type numpy directory:

```python
import dpdata

dpdata.MultiSystems(*systems).to_deepmd_npy_mixed("mixed_dir")
```

- Load a mixed type data into a MultiSystems:

```python
import dpdata

Expand Down
8 changes: 5 additions & 3 deletions docs/systems/multi.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,16 @@
# `MultiSystems`

The Class {class}`dpdata.MultiSystems` can read data from a dir which may contains many files of different systems, or from single xyz file which contains different systems.
The Class {class}`dpdata.MultiSystems` can read data from a dir which may contains many files of different systems, or from single xyz file which contains different systems.

Use {meth}`dpdata.MultiSystems.from_dir` to read from a directory, {class}`dpdata.MultiSystems` will walk in the directory
Recursively and find all file with specific file_name. Supports all the file formats that {class}`dpdata.LabeledSystem` supports.
Use {meth}`dpdata.MultiSystems.from_dir` to read from a directory, {class}`dpdata.MultiSystems` will walk in the directory
Recursively and find all file with specific file_name. Supports all the file formats that {class}`dpdata.LabeledSystem` supports.

Use {meth}`dpdata.MultiSystems.from_file` to read from single file. Single-file support is available for the `quip/gap/xyz` and `ase/structure` formats.

For example, for `quip/gap xyz` files, single .xyz file may contain many different configurations with different atom numbers and atom type.

The following commands relating to {class}`dpdata.MultiSystems` may be useful.

```python
# load data

Expand Down Expand Up @@ -40,6 +41,7 @@ xyz_multi_systems.to_deepmd_raw("./my_deepmd_data/")
```

You may also use the following code to parse muti-system:

```python
from dpdata import LabeledSystem, MultiSystems
from glob import glob
Expand Down
Loading
Loading