WIP : Experiment [this repo only] : Fast and unsafe development time compilation out of FSharpChecker caches#19267
Open
WIP : Experiment [this repo only] : Fast and unsafe development time compilation out of FSharpChecker caches#19267
Conversation
Add internal CompilationData member on FSharpCheckProjectResults to expose cached typecheck data (TcConfig, TcGlobals, TcImports, CcuThunk, TopAttribs, ILAssemblyRef, CheckedImplFiles). Add public CompileFromCheckedProject method on FSharpChecker that takes check results and an output path, then runs the backend pipeline (sigdata encoding, IL generation, module creation, binary writing) without re-parsing or re-typechecking. Skips optimization and PDB generation for fast dev-loop use.
Save and restore generatedCcu.Contents.Attribs around the compilation to prevent repeated CompileFromCheckedProject calls from appending assembly attributes to the shared cached CCU on each invocation. Includes regression test and surface area baseline update.
Add a 'compile' case to the handleRequest match block in Server.fs that: - Extracts project and output paths from JSON - Resolves project options via projectMgr - Runs ParseAndCheckProject and checks HasCriticalErrors - Calls CompileFromCheckedProject on success - Returns 'OK' on success, 'ERROR: ...' on failure
Use Proto-built FSharp.Compiler.Service.dll (from artifacts/Bootstrap/fsc/) when available, falling back to NuGet PackageReference otherwise. This allows the diagnostics server to access CompileFromCheckedProject API from current source before it ships in a release.
- ProjectManager: Replace single option cache with Dictionary<string, DateTime * FSharpProjectOptions> - ProjectManager: Add path normalization via Path.GetFullPath - ProjectManager: Invalidate accepts optional fsproj path (one or all) - Server: Add resolveProject helper mapping source files to fsproj paths - Server: getOptions now accepts file path parameter - Server: All handlers (parseOnly, check, findRefs, typeHints) pass file to getOptions - Server: checkProject accepts optional 'project' JSON field, defaults to FCS fsproj - Server: Remove hardcoded fsproj variable
- ProjectManager.fs: Dictionary<string, DateTime * FSharpProjectOptions> cache with path normalization and optional Invalidate(?fsprojPath) - ProjectRouting.fs: extracted resolveProject for testability, maps source files to fsproj (ComponentTests or FCS) - Server.fs: uses ProjectRouting.resolveProject, all handlers route via file path - checkProject accepts optional 'project' JSON field, defaults to FCS fsproj - Tests: 10 tests covering resolveProject mapping and ProjectManager.Invalidate
- Add CacheCount, HasCachedProject, InjectTestEntry to ProjectManager for testability - Rewrite ProjectManagerTests with behavioral assertions (cache population, selective/full invalidation, path normalization) - Refactor ResolveProjectTests to use Theory/InlineData, eliminating copy-paste - Shared createManager/dummyOptions helpers eliminate repeated setup - 13 tests: 7 ProjectManager + 6 routing (all pass)
… tests - Mark CacheCount, HasCachedProject, InjectTestEntry as internal - Add InternalsVisibleTo for test project in server fsproj - Add test: Invalidate normalizes path before removal - Add test: InjectTestEntry overwrites existing entry (idempotency) - Add test: resolveProject with out-of-repo file defaults to FCS
- Update guard condition in FastBuildFromCache.targets to also match AssemblyName=='FSharp.Compiler.ComponentTests' - Add Import of FastBuildFromCache.targets to FSharpTests.Directory.Build.targets (mirrors existing import in FSharpBuild.Directory.Build.targets) - Fix XML comment containing '--compile' (invalid in XML comments) - Update header/guard comments to mention ComponentTests
CODE-QUALITY: - Fix unused outPath binding in Server.fs compile handler (let! _ =) - Improve ProjectRouting.resolveProject to use StartsWith prefix check instead of fragile String.Replace, with StringComparison.Ordinal - Add XML doc comment to resolveProject NO-LEFTOVERS: - Remove stale src/FastBuildFromCache.targets (old approach, superseded by eng/targets/) - Remove leftover plan files (FAST_COMPILE_PLAN.md, REUSABLE_COMPILE_PLAN.md) - Revert incorrect Directory.Build.targets import (wrong path) - Add missing FSharpBuild.Directory.Build.targets import for src/ projects - Include uncommitted shell script --compile handler and SKILL.md docs TEST-COVERAGE: - Add DesignTimeBuildTests for config defaults and DtbResult construction - Add ProjectRouting tests: vsintegration path, FSharp.Core path, trailing slash, repoRoot-as-substring edge case - Add ProjectManager tests: 3-project coexistence, selective invalidation TEST-CODE-QUALITY: - Tests use descriptive names and verify specific behaviors - Edge case tests validate the String.Replace fix All 24 tests pass. Server builds with 0 warnings.
…mCheckedProject internal CODE-QUALITY: Extract normalizeAssemblyRefs as TcImports.NormalizeAssemblyRef member to eliminate duplication between service.fs and fsc.fs. Make CompileFromCheckedProject internal since it's a dev-loop-only API. Add InternalsVisibleTo for FSharpDiagServer. NO-LEFTOVERS: Remove redundant comment in ResolveProjectTests.fs. Unstage .ralph/ tracking files. Surface area baseline updated to reflect internal visibility change.
…, TEST-CODE-QUALITY, TEST-COVERAGE verifier feedback - CODE-QUALITY: Break long line in Server.fs, use InvalidOperationException instead of failwith in CompileFromCheckedProject, add project-not-found validation in compile handler, add caching semantics comment - HONEST-ASSESSMENT: Update SKILL.md to reflect ComponentTests support - NO-LEFTOVERS: Remove unused InternalsVisibleTo from FSharpDiagServer.fsproj, update shell script usage/header to include --compile flag - TEST-CODE-QUALITY: Consolidate duplicate Fact tests into Theory InlineData, add meaningful DesignTimeBuildTests for config overrides and edge cases - TEST-COVERAGE: Add ResolveProjectOptions error path test, add FCS fallback and ComponentTests boundary routing tests, add DtbConfig edge cases
PERF: - Replace Dictionary+lock with ConcurrentDictionary in ProjectManager to eliminate lock contention on concurrent cache lookups - Use HashSet instead of ResizeArray for O(1) symbol name lookup in findRefs - Avoid Array.append allocation when one diagnostics array is empty TEST-COVERAGE: - Add concurrent InjectTestEntry+Invalidate thread safety test - Add concurrent InjectTestEntry from multiple threads test - Add Invalidate-during-concurrent-reads test - Add error-does-not-pollute-cache test - Include leftover test consolidation from Fixup #2
Contributor
❗ Release notes required
|
…fsharp into feature/langserver-skill
… filewatcher pre-warming, --times profiling - NormalizeAssemblyRef on TcImports is now member internal (no public API change) - CompileFromCheckedProject uses minimal optimizer with mandatory lowering passes (OptimizeImplFile + LowerLocalMutables + LowerCalls, no detuple/TLR/extra loops) - Added ReportTime instrumentation for --times profiling of emit phases - FastBuildFromCache.targets: added Inputs/Outputs mirroring CoreCompile for proper MSBuild incremental skip, touch all CoreCompile outputs after cache emit - Diagnostics server: filewatcher pre-warming with 5s throttle on src/Compiler/ - Diagnostics server: compile handler with --compile flag, DTB caching, resource embedding
…iateOutputPath field - git rm --cached 99_CI_Fixup_ReleaseNotes.md (NO-LEFTOVERS) - Add IntermediateOutputPath to DtbResult constructions in DesignTimeBuildTests.fs (TEST-CODE-QUALITY)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
⚡ Fast Build from Cache — Prototype
What
Skips
fscentirely for dev builds by emitting DLLs directly fromFSharpChecker's in-memory typecheck cache. Dev-loop only — no optimization, no PDB, not for shipping.How to use
Currently scoped to
FSharp.Compiler.ServiceandFSharp.Compiler.ComponentTestsbyAssemblyNameguard. All other projects build normally. Falls back to normalfscif the cache server is down.Why it's fast
A persistent diagnostics server holds a warm
FSharpCheckerwithuseTransparentCompiler=true. The TransparentCompiler 📚 caches typecheck results per-file, keyed by content hashes. When a file changes:DependencyResolution.mkGraph) determines which files actually depend on the changed one — via a trie of top-level namespaces/modules andopenresolutionAsyncMemoize📚 caches (TcIntermediate,ParseFile,DependencyGraph)FSharpChecker.CompileFromCheckedProjectthen goes straight from 📚 cachedCheckedImplFile[]→ ILX codegen →WriteILBinaryFile→ DLL, skipping parse/check/optimizeEdit file 300 of 420 → files 1–299 stay cached, and peers with no dependency edge to file 300 also stay cached.
Key pieces
eng/targets/FastBuildFromCache.targets—BeforeTargets="CoreCompile", setsSkipCompilerExecution=trueon successFSharpChecker.CompileFromCheckedProjectinservice.fsServer.fs—"compile"commandProjectRouting.fs— maps source files →.fsprojProjectManager.fs—ConcurrentDictionarykeyed by fsproj path + mtime