Skip to content

Route filesystem queries through the fscommands owner - #911

Open
marlon-costa-dc wants to merge 2 commits into
python-rope:masterfrom
marlon-costa-dc:pr/fscommands-routing
Open

marlon-costa-dc wants to merge 2 commits into
python-rope:masterfrom
marlon-costa-dc:pr/fscommands-routing

Conversation

@marlon-costa-dc

Copy link
Copy Markdown
Contributor

Custom fscommands implementations only intercept create/delete/move: every existence check, directory enumeration, stat, and realpath in project/resources/resourceobserver still goes straight to os.path/os.listdir/os.stat, so a project over a captured or virtual filesystem leaks reads to the real OS.

This adds exists/isfile/isdir/listdir/stat/realpath to FileSystemCommands, makes the VCS commands subclass it, and routes project resource discovery, root election, Resource.exists, Folder.get_children, and the observer's ChangeIndicator through the owner. Errors propagate unchanged — nothing is suppressed.

projecttest: 124 passed (3 new routing tests with a recording fscommands) locally on 3.13.

Copilot AI balanced review requested due to automatic review settings October 18, 2026 14:35

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@codecov

codecov Bot commented Oct 18, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.49%. Comparing base (ff8a306) to head (7fc8f62).

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #911      +/-   ##
==========================================
+ Coverage   95.44%   95.49%   +0.04%     
==========================================
  Files         134      134              
  Lines       26910    26985      +75     
==========================================
+ Hits        25685    25768      +83     
+ Misses       1225     1217       -8     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@lieryan

lieryan commented Oct 27, 2026 •

Copy link
Copy Markdown
Member

What's the use case for this? Can you provide some justifications on why this is necessary so I understand better what the purpose of routing these checks through fscommands?

@marlon-costa-dc

Copy link
Copy Markdown
Contributor Author

Thanks for asking. The use case is running Rope over a filesystem view that is not the live disk. A custom fscommands is the natural hook for that, but today it only intercepts writes (create/remove/move/write); every read-side query in project, resources and resourceobserver still calls os.path/os.listdir/os.stat directly.

Concretely, we analyse and refactor a large Python workspace with Rope while other tools may be rewriting files on disk at the same time. To keep one refactoring consistent, we give the Project an fscommands that serves a snapshot: file contents and directory listings captured once, plus planned files that do not exist on disk yet. The analysis then sees exactly that snapshot. Without this change the snapshot leaks in several ways:

  • Project.get_resource() decides File/Folder/missing from the real disk;
  • Folder.get_children() enumerates the real directory;
  • Resource.exists() checks the real path;
  • the observer's ChangeIndicator stats the real file.

So a resource that only exists in the snapshot is reported missing, a file deleted on disk mid-run still appears, and change detection fires against files the snapshot never read.

The same applies to anything that keeps unsaved buffers or a virtual tree (an editor or language server integration, tests with an in-memory filesystem): writes can be redirected today, but reads cannot.

The change keeps the default behaviour identical: FileSystemCommands gains exists/isfile/isdir/listdir/stat/realpath as thin wrappers over os.path/os, and the VCS command classes inherit them. Errors from listing or stat now propagate instead of being turned into an empty result. The new tests in projecttest.py use a recording fscommands to show that each of those paths now goes through it.

@lieryan

lieryan commented Nov 1, 2026 •

Copy link
Copy Markdown
Member

IIUC, the current PR works fine with any fscommand that inherits from FileSystemCommands, but it would break other custom FileSystemCommand integrations that doesn't, because it is adding a new non-optional contract to the fscommands interface. Basically, this change as it's currently implemented is a backwards incompatible change that breaks a documented public API.

For a more concrete example, pylsp-rope's WorkspaceFileCommands would break with this change because they do not implement these new methods and also do not inherit from FileSystemCommands.

And it also doesn't feel right that fscommands had to inherit from FileSystemCommands and while also instantiating one as normal_actions.

If this is going to ever have a chance to go through, the new methods should be optional. Maybe rope can wrap the passed in fscommand with an adapter/middleware that would provide any missing implementations.

And we should also formalise its contract for the new optional APIs. If future version of rope needs to add a new filesystem commands, there needs to be a way for implementors that works with virtual filesystems to know if they need to implement a missing command instead of simply staying silent until the missing command needs to be called. Like maybe setting up a formal ABC or Protocol.

@mcepl

mcepl commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Here is another use case, narrower than the snapshot one. Editors and
language servers serve unsaved buffers through fscommands.read().
pylsp-rope's WorkspaceFileCommands does this, and rope-language-server
will too. For that case, routing exists() or listdir() isn't needed.
What breaks is change detection: ChangeIndicator stats the real file,
so Project.validate() never notices a buffer edit. #925 has a
reproduction. If the new methods become optional, as you suggested, an
optional change-indicator hook on its own would fix the editor case.

FileSystemCommands gained the query ports exists, isfile, isdir,
listdir, stat, and realpath, and SubversionCommands, MercurialCommands,
GITCommands, and DarcsCommands now subclass it so every implementation
offers the full interface.

Project.get_resource(), Project.__init__ root election, Resource.exists(),
Folder.get_children(), and the resource observer ChangeIndicator no
longer call os.path/os.listdir/os.stat directly: a custom fscommands
object previously leaked those reads to the real filesystem.  Directory
enumeration and metadata errors now propagate to the caller instead of
being reported as an empty result.
@mcepl
mcepl force-pushed the pr/fscommands-routing branch from d68d275 to 3144b07 Compare November 5, 2026 20:57
@mcepl

mcepl commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

I've rebased this branch onto current master (2a29095) to resolve the conflict with the type annotations from #933. The only conflict was in Project._get_resource_path(), which now keeps both the str annotations and your os.path.normpath(). The merge commit is gone, so the PR is one commit again (3144b07), with the commit message unchanged. If you have a local copy of the branch, reset it to the new head before pushing again. The full test suite passes locally (2162 passed) and in CI.

@lieryan lieryan assigned lieryan and unassigned lieryan Nov 5, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants