fix(cdk): upgrade unstructured from 0.10.27 to 0.18.18 to fix CVE-2025-64712

[devin-ai-integration]

fix(cdk): upgrade unstructured 0.10.27 → 0.18.18 to fix CVE-2025-64712

Summary

Upgrades the unstructured library to remediate critical path traversal vulnerability GHSA-gm8q-m8mv-jj5m (CVSS 9.8) in partition_msg. Affects file-based connectors: source-gcs, source-azure-blob-storage, source-s3.

The upgrade required adapting unstructured_parser.py to API changes between 0.10.27 and 0.18.18:

• Removed dict lookups → New FileType methods: EXT_TO_FILETYPE[ext] → FileType.from_extension(ext), STR_TO_FILETYPE[mime] → FileType.from_mime_type(mime), FILETYPE_TO_MIMETYPE[ft] → ft.mime_type
• Renamed parameter: detect_filetype(filename=...) → detect_filetype(file_path=...)
• New dependency: Added pi-heif (required at import time by partition_pdf in 0.18.18)
• partition_pdf import made optional: In 0.18.18, partition_pdf requires the heavy unstructured_inference package. The import is now wrapped in try/except; availability is checked only when actually processing PDF files. MD/TXT files no longer fail due to missing PDF dependencies.
• File type detection reordered: Extension-based detection is now checked before content-based sniffing in _get_filetype. Content sniffing in 0.18.18 can return TXT for plain text content on some Python versions, which would mask the actual file type indicated by the extension (e.g., a .csv file with text content). A None guard was also added for FileType.from_extension() which returns None for unknown extensions (rather than raising), ensuring content-based fallback still works for extensionless files.
• Per-filetype availability checks: Removed blanket "all three partitioners must be non-None" guards from _read_file() and _read_file_locally(). Each filetype now checks its own partition function availability, with a clear error message (e.g., PDF parsing requires unstructured_inference).
• Exception propagation: Added except RecordParseError: raise before the generic exception handler in _read_file_locally() to ensure parse errors propagate correctly instead of being re-wrapped.

Test mocks updated to patch module-level globals instead of direct unstructured.partition.* paths, and to mock _import_unstructured to avoid pulling in unstructured_inference (heavy ML dep not needed for unit tests).

Scenario test expectation changes: Because unstructured_inference is not installed in the test environment, PDF files now produce _ab_source_file_parse_error records instead of parsed content. DOCX content rendering also changed from "# Content" to "Content" (the new unstructured version strips markdown heading syntax from .docx output).

Resolves https://github.com/airbytehq/oncall/issues/11267:

• https://github.com/airbytehq/oncall/issues/11267

Review & Testing Checklist for Human

This is a large version jump (0.10.27 → 0.18.18) with significant API and behavioral changes. Unit tests pass locally, but there are meaningful risks that CI and unit tests alone cannot cover:

• Test end-to-end with real files: Run actual file parsing with PDF/DOCX/PPTX/MD files against a real source (e.g., source-s3 or source-gcs) to verify parsing behavior hasn't regressed. Unit tests mock all partition functions, so real parsing with 0.18.18 is completely untested. Critical: DOCX content rendering changed (markdown heading syntax is now stripped), which could affect downstream consumers.
• Verify PDF parsing WITH unstructured_inference installed: In production environments where unstructured_inference is installed, PDFs should parse successfully. Test this end-to-end because the test suite now only validates the "missing unstructured_inference" error path, not actual PDF parsing.
• Verify file type detection priority change: The reordering of extension check before content sniffing could affect edge cases. Test files where extension doesn't match content (e.g., a .txt file containing CSV data) to ensure detection still works as expected.
• Check for other breaking changes: Review unstructured changelog between 0.10.27 and 0.18.18 for any other API or behavioral changes that could affect file parsing (e.g., changes to markdown rendering, element extraction, etc.)
• Verify CI passes on all platforms: Prior attempt to bump to 0.18.15 (airbytehq/airbyte-python-cdk#767) had 12 CI failures. Ensure all CI checks pass, especially on different platforms (pi-heif is a native dependency)

Notes

• Mypy: Added cast(IO[bytes], file) to resolve type incompatibility with detect_filetype. All mypy errors resolved.
• Test coverage gaps:
  • By mocking _import_unstructured, tests no longer verify the lazy import mechanism works correctly with the new version or that partition functions are actually callable.
  • The corrupted_file_scenario no longer tests actual PDF corruption handling - it now tests the "missing unstructured_inference" error path, same as all other PDF tests.
  • PDF parsing is not validated at all in the test suite (only the error path when unstructured_inference is missing).
• New transitive dependencies: poetry.lock shows many new packages (aiofiles, html5lib, nest-asyncio, olefile, unstructured-client, webencodings, eval-type-backport). These increase the dependency surface area.
• Removed dependency: chardet was removed from the lock file (previously a transitive dep).
• CI note: One flaky test failure in test_concurrent_declarative_source.py (unrelated to unstructured changes) - passes locally on both main and PR branch. All required checks pass.

---

Requested by Danylo Jablonski (@DanyloGL) via /ai-fix command
Devin session: https://app.devin.ai/sessions/321f6bedb61f4d809581e073e864a026

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@devin/1771342600-bump-unstructured-0.18.18#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch devin/1771342600-bump-unstructured-0.18.18

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

• /autofix - Fixes most formatting and linting issues
• /poetry-lock - Updates poetry.lock file
• /test - Runs connector tests with the updated CDK
• /prerelease - Triggers a prerelease publish with default arguments
• /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
• /poe <command> - Runs any poe command in the CDK environment

📚 Show Repo Guidance

Helpful Resources

• CDK API Reference

📝 Edit this welcome message.

[github-actions]

PyTest Results (Fast)

590 tests   - 3 279   578 ✅  - 3 279   3m 15s ⏱️ - 3m 23s
  1 suites ±    0    11 💤  -     1 
  1 files   ±    0     1 ❌ +    1 

For more details on these failures, see this check.

Results for commit 76c7faf. ± Comparison against base commit e9144e2.

This pull request removes 3279 tests.

unit_tests.sources.declarative.async_job.test_integration.JobDeclarativeStreamTest ‑ test_when_read_then_call_stream_slices_only_once
unit_tests.sources.declarative.async_job.test_integration.JobDeclarativeStreamTest ‑ test_when_read_then_return_records_from_repository
unit_tests.sources.declarative.async_job.test_job.AsyncJobTest ‑ test_given_status_is_terminal_when_update_status_then_stop_timer
unit_tests.sources.declarative.async_job.test_job.AsyncJobTest ‑ test_given_timer_is_not_out_when_status_then_return_actual_status
unit_tests.sources.declarative.async_job.test_job.AsyncJobTest ‑ test_given_timer_is_out_when_status_then_return_timed_out
unit_tests.sources.declarative.async_job.test_job_orchestrator.AsyncJobOrchestratorTest ‑ test_given_exception_on_single_job_when_create_and_get_completed_partitions_then_return
unit_tests.sources.declarative.async_job.test_job_orchestrator.AsyncJobOrchestratorTest ‑ test_given_exception_to_break_when_start_job_and_raise_this_exception_and_abort_jobs
unit_tests.sources.declarative.async_job.test_job_orchestrator.AsyncJobOrchestratorTest ‑ test_given_exception_when_start_job_and_skip_this_exception
unit_tests.sources.declarative.async_job.test_job_orchestrator.AsyncJobOrchestratorTest ‑ test_given_failure_when_create_and_get_completed_partitions_then_raise_exception
unit_tests.sources.declarative.async_job.test_job_orchestrator.AsyncJobOrchestratorTest ‑ test_given_jobs_failed_more_than_max_attempts_when_create_and_get_completed_partitions_then_free_job_budget
…

♻️ This comment has been updated with latest results.

[github-actions]

PyTest Results (Full)

3 872 tests  ±0   3 860 ✅ ±0   11m 24s ⏱️ +7s
    1 suites ±0      12 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit 76c7faf. ± Comparison against base commit e9144e2.

♻️ This comment has been updated with latest results.