Block pool

README.md

@@ -61,6 +61,57 @@ python3 -m pip install git+https://github.com/zmc/ceph-devstack.git
 
 `ceph-devstack config dump` will output the current configuration.
 
+### Shared block pool (optional)
+
+To back loop devices with slices of a real NVMe device instead of sparse files,
+configure `[block_pool]` in your config. The pool is shared across all
+`testnode_*` containers. Each loop device uses that container's
+`loop_device_size` for its backing region on the pool parent.
+
+```toml
+[block_pool]
+parent = "/dev/nvme0n1p1"   # or a dedicated whole disk
+allow_enroll = true         # first run only; unset afterward
+
+[containers.testnode]
+loop_device_size = "5G"
+```
+
+Requirements:
+
+* Your user must be in the `disk` group and able to read/write the parent
+  device directly (enrollment writes a tail marker to the raw device).
+* On a Fedora CoreOS podman machine (macOS), the `disk` group is defined in
+  `/usr/lib/group`, so `usermod -aG disk core` silently does nothing until the
+  group line is copied into `/etc/group`. `ceph-devstack doctor --fix` does
+  both; by hand, inside `podman machine ssh`:
+
+  ```bash
+  grep -q '^disk:' /etc/group || grep '^disk:' /usr/lib/group | sudo tee -a /etc/group
+  sudo usermod -aG disk core
+  ```
+
+  New `podman machine ssh` sessions then have the group.
+* The parent must be empty, or already enrolled by ceph-devstack (tail marker
+  at the end of the device).
+
+`ceph-devstack block-pool status` shows current allocations.
+
+All pool device access (sysfs, `findmnt`, `blkid`, and the marker and probe
+reads/writes via `dd`) runs on the container host, so a pool inside a podman
+machine (macOS, `RemoteHost`) should work, but that setup is unverified. The
+pool state file and its lock always stay on the local machine.
+
+Each container picks host loop devices at create time: it needs
+`loop_device_count` devices of `loop_device_size`, reuses any existing backing
+files for that container name, then takes the lowest free loop numbers after
+scanning what is already attached or claimed in the `disk_images` directory.
+
+If `block_pool.json` is lost but the on-disk marker remains, start again to
+reclaim the marker; the device is not reformatted. The old allocations are
+forgotten, though, so regions that still hold data are refused as not empty
+until they are wiped.
+
 As an example, the following configuration will use a local image for paddles with the tag `TEST`; it will also create ten testnode containers; and will build its teuthology container from the git repo at `~/src/teuthology`:
 ```
 containers:

ceph_devstack/__init__.py

@@ -15,6 +15,7 @@
 
 PROJECT_ROOT = Path(__file__).parent
 DEFAULT_CONFIG_PATH = Path("~/.config/ceph-devstack/config.toml")
+DEFAULT_DATA_DIR = "~/.local/share/ceph-devstack"
 
 
 def parse_args(args: List[str]) -> argparse.Namespace:
@@ -61,6 +62,11 @@ def parse_args(args: List[str]) -> argparse.Namespace:
         default=False,
         help="Apply suggested fixes for issues found",
     )
+    parser_block_pool = subparsers.add_parser(
+        "block-pool", help="Inspect the shared block device pool"
+    )
+    block_pool_subparsers = parser_block_pool.add_subparsers(dest="block_pool_op")
+    block_pool_subparsers.add_parser("status", help="Show pool allocations and state")
     parser_pull = subparsers.add_parser("pull", help="Pull container images")
     parser_pull.add_argument(
         "image",