From 28343570b3db6a8369e78a436457a5009edf56c7 Mon Sep 17 00:00:00 2001 From: Andrea Date: Fri, 6 Nov 2026 12:08:43 +0100 Subject: [PATCH 1/4] Add installation, usage, and legacy documentation; introduce standalone binary build process - Created `installation.md` to provide detailed installation instructions for various methods (Singularity, pip, Conda, standalone binary). - Added `legacy.md` to document older Bash scripts that are no longer maintained. - Introduced `metrics.md` to describe output files and metrics generated by the collector. - Added `standalone-binary.md` to outline the process for building and using a standalone binary with PyInstaller. - Created `usage.md` to provide quick start guides and detailed usage instructions for metrics collection and analysis. - Added `treecript-builder.def` for building the standalone binary in a Singularity environment. --- README.md | 861 ++------------------------------------ docs/installation.md | 233 +++++++++++ docs/legacy.md | 38 ++ docs/metrics.md | 74 ++++ docs/standalone-binary.md | 299 +++++++++++++ docs/usage.md | 184 ++++++++ treecript-builder.def | 65 +++ 7 files changed, 921 insertions(+), 833 deletions(-) create mode 100644 docs/installation.md create mode 100644 docs/legacy.md create mode 100644 docs/metrics.md create mode 100644 docs/standalone-binary.md create mode 100644 docs/usage.md create mode 100644 treecript-builder.def diff --git a/README.md b/README.md index 19caec0..a75ed61 100644 --- a/README.md +++ b/README.md @@ -1,849 +1,44 @@ # `treecript`: Process Tree Metrics Transcriptor -> Originally named _Execution Process Metrics Collector_ +Originally named *Execution Process Metrics Collector*, treecript monitors Linux +processes and their descendants, collects their metrics, and helps analyze and +visualize the results. It was initially developed for +[ELIXIR STEERS](https://elixir-europe.org/about-us/how-funded/eu-projects/steers). -A set of Python programs to monitor, collect, and digest metrics of a given Linux process or command line, and its descendants. Initially developed for [ELIXIR STEERS](https://elixir-europe.org/about-us/how-funded/eu-projects/steers). +## Documentation ---- +- [Installation](docs/installation.md): Singularity/Apptainer, pip, and Conda. +- [Standalone collector binary](docs/standalone-binary.md): download the existing + Docker-built binary, build with Docker or Apptainer/Singularity, and check + compatibility with HPC systems. +- [Usage and CPU datasets](docs/usage.md): collect, plot, find CPU TDP, and + estimate energy consumption. +- [Metrics and output files](docs/metrics.md): output files and column reference. +- [RO-Crate representation](RO-Crate-REPRESENTATION.md): structure of generated + RO-Crates. +- [Onboarding example](onboarding/README.md) and + [sample workloads](sample-work-to-measure/README.md). +- [Legacy scripts](docs/legacy.md). -## Table of Contents +## Quick start - - - -- [`treecript`: Process Tree Metrics Transcriptor](#treecript-process-tree-metrics-transcriptor) - - [Table of Contents](#table-of-contents) - - [Repository Structure](#repository-structure) - - [Installation](#installation) - - [Prerequisites (singularity)](#prerequisites-singularity) - - [Prerequisites (native)](#prerequisites-native) - - [Not sure which installation method to use?](#not-sure-which-installation-method-to-use) - - [Choosing a constraints file (only needed for option 2, option 3 or option 4)](#choosing-a-constraints-file-only-needed-for-option-2-option-3-or-option-4) - - [Option 1: Singularity](#option-1-singularity) - - [Alternative A: Fetch a pre-built image (only for metrics gathering)](#alternative-a-fetch-a-pre-built-image-only-for-metrics-gathering) - - [Alternative B: Building from source](#alternative-b-building-from-source) - - [Option 2: pip + virtual environment (venv)](#option-2-pip--virtual-environment-venv) - - [Option 3: Conda environment](#option-3-conda-environment) - - [Installing Miniconda (if not already installed)](#installing-miniconda-if-not-already-installed) - - [Creating the treecript conda environment](#creating-the-treecript-conda-environment) - - [Option 4: Standalone binary (PyInstaller)](#option-4-standalone-binary-pyinstaller) - - [Alternative A: Use a pre-built binary](#alternative-a-use-a-pre-built-binary) - - [Alternative B: Building it yourself](#alternative-b-building-it-yourself) - - [Why a container is needed](#why-a-container-is-needed) - - [Building the binary](#building-the-binary) - - [Checking glibc compatibility before deploying](#checking-glibc-compatibility-before-deploying) - - [Troubleshooting](#troubleshooting) - - [Full workflow example: HPC metrics collection + local analysis](#full-workflow-example-hpc-metrics-collection--local-analysis) - - [Verifying the installation](#verifying-the-installation) - - [Quick Start (Singularity/Apptainer)](#quick-start-singularityapptainer) - - [Quick Start (other options)](#quick-start-other-options) - - [Programs Reference](#programs-reference) - - [Collecting metrics](#collecting-metrics) - - [Plotting time series charts](#plotting-time-series-charts) - - [Finding CPU TDP](#finding-cpu-tdp) - - [`tdp-finder` — from a metrics directory](#tdp-finder--from-a-metrics-directory) - - [`cpuinfo-tdp-finder` — from `/proc/cpuinfo`](#cpuinfo-tdp-finder--from-proccpuinfo) - - [`modelname-tdp-finder` — from a processor model string](#modelname-tdp-finder--from-a-processor-model-string) - - [Digesting metrics](#digesting-metrics) - - [CPU Dataset Setup](#cpu-dataset-setup) - - [Output Files Reference](#output-files-reference) - - [Per-process metrics (`metrics-{pid}_{create_time}.csv`)](#per-process-metrics-metrics-pid_create_timecsv) - - [Aggregated metrics (`agg_metrics.tsv`)](#aggregated-metrics-agg_metricstsv) - - [Legacy](#legacy) - - [`execution-metrics-collector.sh`](#execution-metrics-collectorsh) - - [`plotGraph.sh`](#plotgraphsh) - - [`plot-metrics.sh`](#plot-metricssh) - - [License](#license) - - ---- - -## Repository Structure - -``` -treecript/ -├── treecript/ # Core Python package — all program logic lives here -├── installation/ # Constraints and requirements files for reproducible installs -├── legacy/ # Deprecated Bash scripts kept for historical reference -├── sample-series/ # Real metrics from a WfExS workflow execution, used in documentation examples -├── sample-charts/ # Pre-generated charts from the sample series, embedded in this README -├── sample-work-to-measure/ # Example scripts showing how to set up and run a measurement -├── sample_cpuinfo/ # Example /proc/cpuinfo files for testing the TDP finder programs -├── onboarding/ # Full worked example with metrics, charts and a step-by-step walkthrough -├── sample/ # Legacy single-process sample from 2018 (pre-treecript era) -└── tests/ # Unit tests -``` - -| Directory | Description | -| ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | -| `treecript/` | Core Python package — aggregator, collector, parser, plotter, TDP finder | -| `installation/` | Per-version constraints files and requirements for reproducible installs | -| `legacy/` | Deprecated Bash scripts superseded by the Python programs | -| `sample-series/` | Real metrics collected from a WfExS workflow run, used throughout this README as examples | -| `sample-charts/` | Pre-generated chart outputs (SVG/PDF/PNG) from the sample series | -| `sample-work-to-measure/` | Ready-to-use scripts to download and run example workloads to measure | -| `sample_cpuinfo/` | Example `/proc/cpuinfo` files (Intel and AMD) for testing `cpuinfo-tdp-finder` and `modelname-tdp-finder` without needing a real machine | -| `onboarding/` | Self-contained worked example: a full metrics collection, chart generation and aggregation walkthrough for new users | -| `sample/` | Legacy single-process sample from 2018, predating the current process-tree approach | -| `tests/` | Unit tests for the core collector module | - ---- - -## Installation - -### Prerequisites (singularity) - -- Linux OS -- [Singularity](https://sylabs.io/docs/) or [Apptainer](https://apptainer.org/). - -### Prerequisites (native) - -- Linux OS (Ubuntu recommended) -- Python 3.9 or newer -- Git - -### Not sure which installation method to use? - -| I want to... | Use | -| ------------------------------------------------------------------------------ | -------------------------------------------------------------- | -| Keep things simple, already have either Apptainer or Singularity installed and you only want to gather metrics | **Option 1 — Singularity** | -| Keep things simple and already have Python installed | **Option 2 — pip + venv** | -| Already use conda or manage multiple projects/environments | **Option 3 — Conda** | -| Work on an HPC or shared cluster environment (e.g. BSC) | **Option 1 — Singularity** or **Option 3 — Conda** | -| Deploy a single, dependency-free executable to a machine where you can't install Python, pip, or containers directly (e.g. copying onto compute nodes) | **Option 4 — Standalone binary (PyInstaller)** | -| Work on a machine with a corporate or university firewall | Either — both have firewall notes in their respective sections | - ---- - -### Choosing a constraints file (only needed for option 2, option 3 or option 4) - -The repository ships per-version constraints files under the `installation/` directory to ensure a working set of dependencies. Pick the one that matches your setup (including the Python version): - -| Situation | Constraints file to use | -| ----------------------------- | ---------------------------------------------------- | -| Native Linux, Python 3.x | `installation/constraints-3.x.txt` | -| Ubuntu 22.04 on WSL (Windows) | `installation/constraints-3.x_Ubuntu-22.04-wsl.txt` | -| Ubuntu 24.04 on WSL (Windows) | `installation/constraints-3.x_Ubuntu-24.04-wsl.txt` | - -> **WSL** = Windows Subsystem for Linux — Ubuntu running inside Windows rather than directly on hardware. If you are running Ubuntu natively on your machine, use the plain constraints file. To check: -> -> ```bash -> uname -r # if the output contains "microsoft" or "WSL", you are on WSL -> ``` - -To check your Python version: - -```bash -python3 --version -``` - ---- - -### Option 1: Singularity - -#### Alternative A: Fetch a pre-built image (only for metrics gathering) - -Pre-build singularity images are listed at https://github.com/inab/treecript/pkgs/container/treecript . - -They can be fetched using Apptainer/Singularity just using `singularity pull` subcommand: - -```bash -# Replace exec by the tag of the version you want to use -singularity pull oras://ghcr.io/inab/treecript:exec - -# The name of the created file depends on the tag -ls -l treecript-exec.sif -``` - -#### Alternative B: Building from source - -When no file is locally, but you already have either Apptainer or Singularity installed: - -```bash -# TREECRIPT_VER can be either a branch, a tag or a commit hash -TREECRIPT_VER=7c0a20a688518e43952d7bd7080bc34b863e32da - -# If you don't have the recipe, you can fetch it using either curl or wget -mkdir -p treecript_SIF_build/installation -cd treecript_SIF_build/installation -curl -O https://raw.githubusercontent.com/inab/treecript/${TREECRIPT_VER}/installation/Singularity.def -cd .. -``` - -If you have already checked out the code - -```bash -# Alternatively, you can use a checked out copy -cd treecript -TREECRIPT_VER=$(git rev-parse HEAD) -``` - -At last, you can build the SIF image with next command: - -``` -singularity build --build-arg treecript_checkout="${TREECRIPT_VER}" \ - treecript-${TREECRIPT_VER}.sif installation/Singularity.def -``` - ---- - -### Option 2: pip + virtual environment (venv) - -Use this if you already have Python installed on your system and don't use conda. This is the lightest option — it creates an isolated Python environment using only tools that come built into Python, with no additional software required. - -```bash -# 1. Create a virtual environment -python3 -m venv TREECRIPT - -# 2. Activate it -source TREECRIPT/bin/activate - -# 3. Upgrade pip and wheel -pip install --upgrade pip wheel - -# 4. Download the constraints file for your Python version (adjust filename as needed) -wget https://raw.githubusercontent.com/inab/treecript/exec/installation/constraints-3.10.txt - -# 5. Install treecript with constraints -pip install -c constraints-3.10.txt 'treecript [analytics,docker] @ git+https://github.com/inab/treecript.git@exec' -``` - -> **Network issues?** If you are behind a corporate or university firewall (e.g. Fortiguard), add `--no-check-certificate` to the `wget` command. - -To deactivate the environment: - -```bash -deactivate -``` - -To reactivate later: - -```bash -source TREECRIPT/bin/activate -``` - ---- - -### Option 3: Conda environment - -Use this if you already work with Anaconda or Miniconda, or if you prefer conda for managing environments across multiple projects. Conda handles both Python and system-level dependencies, which makes it particularly well suited for HPC or shared computing environments. - -#### Installing Miniconda (if not already installed) - -Miniconda is a minimal conda installer — it gives you the `conda` command and Python without bundling hundreds of extra packages like the full Anaconda distribution does. - -```bash -# Download installer (add --no-check-certificate if behind a firewall) -wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh - -# Run the installer -bash miniconda.sh -# - Accept the license -# - Accept the default install location -# - Answer "yes" when asked to update your shell profile - -# Apply changes to current session -source ~/.bashrc - -# Verify -conda --version -``` - -> **Network issues?** If `repo.anaconda.com` is blocked by your network, use **Miniforge** instead — it is functionally identical to Miniconda but downloads from GitHub and defaults to the `conda-forge` channel, which is actually a better fit for the scientific packages treecript needs: -> -> ```bash -> wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh -O miniconda.sh -> bash miniconda.sh -> ``` - -#### Creating the treecript conda environment - -```bash -# 1. Create a clean environment with Python 3.10 -conda create -n treecript python=3.10 -y - -# 2. Activate it -conda activate treecript - -# 3. Download the constraints file (adjust filename for your Python version / OS) -wget https://raw.githubusercontent.com/inab/treecript/exec/installation/constraints-3.10.txt -# or for WSL Ubuntu 22.04: -# wget https://raw.githubusercontent.com/inab/treecript/exec/installation/constraints-3.10_Ubuntu-22.04-wsl.txt - -# 4. Install treecript and all dependencies in one shot -pip install -c constraints-3.10.txt 'treecript [analytics,docker] @ git+https://github.com/inab/treecript.git@exec' -``` - -To deactivate: - -```bash -conda deactivate -``` - -To remove the environment entirely: - -```bash -conda deactivate -conda remove -n treecript --all -y -``` - ---- - -### Option 4: Standalone binary (PyInstaller) - -Use this if you need to run `execution-metrics-collector` on a machine where you cannot install Python, pip, or a container runtime — for example, a compute node on an HPC cluster that only allows copying pre-built files. [PyInstaller](https://pypi.org/project/pyinstaller/) bundles the Python interpreter, treecript, and all its dependencies into a single, dependency-free executable. - -#### Alternative A: Use a pre-built binary - -A pre-built, ready-to-use binary of `execution-metrics-collector` is published under this repository's [Packages](https://github.com/inab/treecript/pkgs/container/treecript) — the same place the Singularity images live (see [Option 1](#option-1-singularity)). You can find it directly at [`ghcr.io/inab/treecript:exec-binary-linux-x86_64`](https://github.com/inab/treecript/pkgs/container/treecript). - -**What it is:** a monolithic, single-file executable (built with PyInstaller inside a `manylinux_2_28` container, see [Alternative B](#alternative-b-building-it-yourself) below) that bundles the Python interpreter, treecript, and all its dependencies. It requires **glibc >= 2.28** on the target machine and nothing else — no Python, no pip, no virtual environment, no container runtime. - -Binaries are pushed and pulled as [OCI artifacts](https://oras.land/) using [`oras`](https://oras.land/docs/installation), the same underlying tool `singularity pull oras://...` uses for the container images above. - -**Downloading it:** - -```bash -# Install oras once, if you don't have it: https://oras.land/docs/installation -# Replace the tag with the version you want (mirrors the treecript git ref/tag) -oras pull ghcr.io/inab/treecript:exec-binary-linux-x86_64 - -chmod +x execution-metrics-collector -./execution-metrics-collector -``` - -**Using it on an HPC cluster:** since the binary has no dependencies, it can be pulled and run directly on the login node, or inside an interactive Slurm allocation: - -```bash -# Option A: pull and run directly on the login node -oras pull ghcr.io/inab/treecript:exec-binary-linux-x86_64 -chmod +x execution-metrics-collector -./execution-metrics-collector ~/my_metrics my_command --arg1 --arg2 - -# Option B: interactive job via Slurm (salloc) — recommended for anything -# that needs dedicated compute resources rather than sharing the login node -salloc -n 1 -c 4 -t 00:30:00 # adjust cores/time/partition to your cluster -# once the allocation starts and you're placed on a compute node: -./execution-metrics-collector ~/my_metrics my_command --arg1 --arg2 -exit # ends the salloc session when done -``` - -Before relying on this binary on a new target machine, double-check glibc compatibility — see [Checking glibc compatibility before deploying](#checking-glibc-compatibility-before-deploying) below. Since it was built against `manylinux_2_28` (glibc 2.28), it will run on any machine with glibc 2.28 or newer, but **not** on an older one: - -```bash -ldd --version | head -1 # on the target machine; must report glibc >= 2.28 -``` - -If your target machine has an older glibc, or a different CPU architecture, this pre-built binary won't work for you — build your own following [Alternative B](#alternative-b-building-it-yourself) below. - -**Publishing a new version (maintainers):** - -```bash -# 1. Log in to GitHub Container Registry (needs a PAT with write:packages scope) -echo "$GITHUB_TOKEN" | oras login ghcr.io -u your-github-username --password-stdin - -# 2. Build the binary (see "Building the binary" below), then push it as an -# OCI artifact — no Dockerfile/container involved, just the raw file -oras push ghcr.io/inab/treecript:exec-binary-linux-x86_64 \ - dist/execution-metrics-collector:application/octet-stream - -# 3. Confirm it shows up under the repository's Packages tab -``` - -Use a version tag that reflects what the binary actually is (e.g. `exec-binary-linux-x86_64`, or include the treecript ref/commit if you publish multiple versions), so users can tell binaries apart without downloading each one. - -#### Alternative B: Building it yourself - -##### Why a container is needed - -The binary must be built on a machine whose **glibc version is equal to or older than** the target machine's glibc, because glibc is forward-compatible but not backward-compatible: a binary built against a newer glibc will refuse to run on a system with an older one, failing with an error such as: - -``` -/lib64/libc.so.6: version `GLIBC_2.35' not found (required by execution-metrics-collector) -``` - -Development laptops typically run a fairly recent Linux distribution (e.g. Ubuntu 22.04, glibc 2.35), while HPC login/compute nodes often run older, more conservative distributions (e.g. glibc 2.28–2.34). Building directly on the laptop would therefore very likely produce a binary that cannot run on the cluster. - -The safe approach is to build inside a [`manylinux`](https://github.com/pypa/manylinux) container, which ships a deliberately old glibc chosen to be compatible with virtually any modern Linux target: - -```bash -# Check your target machine's glibc version: -ssh your-user@your-hpc-login-node "ldd --version | head -1" - -# Check your build machine's glibc version: -ldd --version | head -1 -``` - -If the target's glibc is **older** than your laptop's, build inside a `manylinux_2_28` (or older, e.g. `manylinux2014`) container instead of building natively. - -> **Note:** the Python interpreters preinstalled inside `manylinux` images (under `/opt/python/cpXXX-cpXXX`) are built **without** a shared `libpython` — they are meant for compiling wheels, not for running applications. PyInstaller requires a shared `libpythonX.Y.so`, so the container build compiles its own Python from source with `--enable-shared`. It also needs `openssl-devel` (and other `-devel` headers) installed **before** compiling Python, otherwise Python's `ssl` module is silently skipped and `pip` cannot reach PyPI. - -##### Building the binary - -Save the following as `Dockerfile` in a directory that also contains your chosen `installation/constraints-3.x.txt` file: - -```dockerfile -FROM quay.io/pypa/manylinux_2_28_x86_64 - -ARG PYVER=3.13.9 -ARG TREECRIPT_GIT_URL=https://github.com/inab/treecript.git -ARG TREECRIPT_REF=exec - -ENV PYSHARED_PREFIX=/opt/python-shared -ENV LD_LIBRARY_PATH=${PYSHARED_PREFIX}/lib - -# Build-time dependencies for compiling Python (openssl-devel is required, -# otherwise Python ends up without the ssl module and pip cannot reach PyPI) -RUN dnf install -y \ - openssl-devel bzip2-devel libffi-devel zlib-devel xz-devel \ - ncurses-devel readline-devel sqlite-devel tk-devel gdbm-devel \ - && dnf clean all - -# Compile Python with --enable-shared, required by PyInstaller -RUN curl -O https://www.python.org/ftp/python/${PYVER}/Python-${PYVER}.tgz \ - && tar xzf Python-${PYVER}.tgz \ - && cd Python-${PYVER} \ - && ./configure --enable-shared --prefix=${PYSHARED_PREFIX} --with-openssl=/usr \ - && make -j"$(nproc)" \ - && make altinstall \ - && cd .. && rm -rf Python-${PYVER} Python-${PYVER}.tgz - -ENV PYBIN=${PYSHARED_PREFIX}/bin/python3.13 -WORKDIR /src - -RUN ${PYBIN} -m venv /opt/venv -ENV PATH=/opt/venv/bin:$PATH - -COPY constraints-3.12.txt /src/constraints-3.12.txt - -RUN /opt/venv/bin/pip install --upgrade pip wheel \ - && /opt/venv/bin/pip install -c /src/constraints-3.12.txt \ - "treecript[analytics,docker] @ git+${TREECRIPT_GIT_URL}@${TREECRIPT_REF}" \ - && /opt/venv/bin/pip install pyinstaller - -CMD ["/opt/venv/bin/pyinstaller", "-F", "-n", "execution-metrics-collector", \ - "/opt/venv/bin/execution-metrics-collector"] -``` - -Build the image once (this recompiles Python from source, so it takes several minutes the first time; subsequent builds reuse the cached layer as long as `PYVER` doesn't change): - -```bash -docker build -t treecript-builder \ - --build-arg TREECRIPT_GIT_URL=https://github.com/inab/treecript.git \ - --build-arg TREECRIPT_REF=exec \ - -f Dockerfile . -``` - -Then generate the binary itself (fast — reuses the already-built image): - -```bash -docker run --rm -v "$PWD/dist:/src/dist" treecript-builder -``` - -The resulting `dist/execution-metrics-collector` is a single, self-contained executable (`--onefile` / `-F` build). - -> **Why only `execution-metrics-collector`?** The whole point of this standalone binary is to run metrics *collection* on a machine with no Python available — typically an HPC compute node. The other treecript programs (`plotGraph`, `tdp-finder`, `metrics-aggregator`) are analysis tools meant to run afterwards on a regular machine (your laptop, a login node with Python, etc.), where a normal [Option 2](#option-2-pip--virtual-environment-venv) or [Option 3](#option-3-conda-environment) install is simpler and works just as well — see the [full workflow example](#full-workflow-example-hpc-metrics-collection--local-analysis) below. It's technically possible to build any of them the same way (just point `pyinstaller` at a different entry point, e.g. `/opt/venv/bin/plotGraph` instead of `/opt/venv/bin/execution-metrics-collector`). - -##### Checking glibc compatibility before deploying - -Before copying the binary anywhere, confirm the minimum glibc version it actually requires: - -```bash -objdump -T dist/execution-metrics-collector | grep GLIBC_ \ - | sed 's/.*GLIBC_\([0-9.]*\).*/\1/' | sort -V | tail -1 -``` - -This should print a version at or below the target machine's `ldd --version`. - -##### Troubleshooting - -| Symptom | Cause | Fix | -| --- | --- | --- | -| `version 'GLIBC_2.XX' not found` when running the binary | Built on a machine/container with newer glibc than the target | Rebuild inside an older `manylinux` base image (e.g. `manylinux2014` instead of `manylinux_2_28`) | -| `Python was built without a shared library` during `pyinstaller` build | Using the manylinux-provided Python instead of a `--enable-shared` build | Compile Python from source with `--enable-shared`, as shown above | -| `SSL module is not available` / pip cannot reach PyPI while building the image | `openssl-devel` (and related `-devel` headers) missing before compiling Python | Install `openssl-devel` (and the other `-devel` packages listed above) before the `./configure && make` step | -| Files generated by `docker run` (`dist/`, `build/`, `*.spec`) cannot be deleted from the host | The container runs as `root`, so generated files are root-owned | `sudo rm -rf build dist *.spec`, or delete them from inside another container run as root, or add `--user "$(id -u):$(id -g)"` to `docker run` to avoid the issue going forward | - -#### Full workflow example: HPC metrics collection + local analysis - -This walks through a complete, real scenario: building a monolithic binary on a laptop, using it to collect metrics on an HPC cluster where no Python environment is available, and then bringing those metrics back to the laptop for plotting, TDP lookup, and energy aggregation — all done locally, since only metrics collection strictly requires running on the HPC node itself. - -```bash -# ── 1. On the laptop: build the binary (see Alternative B above) ────────── -docker build -t treecript-builder -f Dockerfile . -docker run --rm -v "$PWD/dist:/src/dist" treecript-builder - -# Confirm the glibc floor is compatible with the target cluster -objdump -T dist/execution-metrics-collector | grep GLIBC_ \ - | sed 's/.*GLIBC_\([0-9.]*\).*/\1/' | sort -V | tail -1 - -# ── 2. Copy the binary to the HPC login/compute node ─────────────────────── -scp dist/execution-metrics-collector your-user@your-hpc-login-node:~/ - -# ── 3. On the HPC node: collect metrics for the workload you care about ──── -ssh your-user@your-hpc-login-node -chmod +x execution-metrics-collector - -# Option A: direct run on the login node (fine for short/light workloads; -# check your cluster's usage policy before running heavier jobs here) -./execution-metrics-collector ~/my_metrics my_command --arg1 --arg2 - -# Option B: interactive job via Slurm (salloc) — recommended for anything -# that needs dedicated compute resources rather than sharing the login node -salloc -n 1 -c 4 -t 00:30:00 # adjust cores/time/partition to your cluster -# once the allocation starts and you're placed on a compute node: -./execution-metrics-collector ~/my_metrics my_command --arg1 --arg2 -exit # ends the salloc session when done - -# This creates ~/my_metrics/-/ with the raw CSV/TSV metrics -# (from whichever of the two runs above you used) - -# ── 4. Bring the metrics directory back to the laptop ────────────────────── -# (run from the laptop) -scp -r your-user@your-hpc-login-node:~/my_metrics/- ./my_metrics/ - -# ── 5. On the laptop: set up a normal treecript environment for analysis ─── -# (analysis tools — plotGraph, tdp-finder, metrics-aggregator — don't need -# to run on the HPC node itself; a regular venv/conda install is enough) -python3 -m venv TREECRIPT -source TREECRIPT/bin/activate -pip install --upgrade pip wheel -pip install -c constraints-3.12.txt 'treecript[analytics,docker] @ git+https://github.com/inab/treecript.git@exec' - -# ── 6. Get a CPU dataset for TDP lookups (once) ───────────────────────────── -git clone https://github.com/JosuaCarl/cpu-spec-dataset cpu-spec-dataset_Josua - -# ── 7. Plot time series charts ────────────────────────────────────────────── -plotGraph ./my_metrics/-/ ./my_charts/ - -# ── 8. Find the HPC node's CPU TDP ────────────────────────────────────────── -# Use the model name reported by `lscpu` (or /proc/cpuinfo) on the HPC node, -# e.g. "Intel(R) Xeon(R) Platinum 8480+" -tdp-finder ./my_metrics/-/ cpu-spec-dataset_Josua/dataset/*.csv -# Model [Intel(R) Xeon(R) Platinum 8480+] => TDP [MaxTDP] => 350.0 W - -# ── 9. Aggregate and estimate energy consumption ──────────────────────────── -metrics-aggregator ./my_metrics/-/ ./my_agg/ 350.0 -``` - -> **Tip:** if `tdp-finder` prints `Unable to match a valid processor row` for some of the CSV sources you passed, that's normal — different sources use slightly different naming conventions for the same CPU. `tdp-finder` tries each file in order and stops at the first match; passing several sources increases the odds of a hit. The official vendor CSVs (e.g. `intel-cpus.csv`) tend to be the most reliable matches. - ---- - -### Verifying the installation - -> **⚠️ Not applicable to the Option 4 binary itself.** This check requires a Python environment with `treecript` importable, which the standalone binary from [Option 4](#option-4-standalone-binary-pyinstaller) intentionally does **not** provide (that's the whole point of a monolithic executable — see the note right below instead). This check **does** apply to the local analysis environment you create in [Option 4's full workflow example](#full-workflow-example-hpc-metrics-collection--local-analysis) step 5 (the `python3 -m venv TREECRIPT` used to run `plotGraph`, `tdp-finder` and `metrics-aggregator` locally) — it's a completely separate, ordinary Option 2/3-style install, just used alongside the binary rather than instead of it. - -Run this after Option 2 (pip + venv) or Option 3 (Conda) — or after setting up the local analysis environment from Option 4's workflow example — to confirm all dependencies are working correctly: - -```bash -python -c " -import psutil; print('psutil OK:', psutil.__version__) -import docker; print('docker OK:', docker.__version__) -import pandas; print('pandas OK:', pandas.__version__) -import networkx; print('networkx OK:', networkx.__version__) -import matplotlib; print('matplotlib OK:', matplotlib.__version__) -import adjustText; print('adjustText OK:', adjustText.__version__) -import treecript; print('treecript OK') -" -``` - -> For the **Option 4 binary itself**, there is no Python environment to verify this way — simply run the binary, e.g. `./execution-metrics-collector` with no arguments, and confirm it prints its usage message. - ---- - -## Quick Start (Singularity/Apptainer) - -At the moment, it is limited to metrics gathering - -```bash -# 1. Collect metrics for a command in background -my_command --arg1 --arg2 & - -# bash puts the PID of the last background process in $! variable -# This way is needed because the -singularity exec treecript-exec.sif process-metrics-collector $! ~/my_metrics -``` - -## Quick Start (other options) +After a [Python installation](docs/installation.md#option-2-pip--virtual-environment-venv): ```bash -# 1. Collect metrics for a command execution-metrics-collector ~/my_metrics my_command --arg1 --arg2 - -# 2. Plot time series charts -plotGraph ~/my_metrics/2025_01_01-00_00-12345/ ~/my_charts/ - -# 3. Find your CPU's TDP -tdp-finder ~/my_metrics/2025_01_01-00_00-12345/ cpu-spec-dataset_Josua/dataset/*.csv - -# 4. Aggregate and estimate energy consumption -metrics-aggregator ~/my_metrics/2025_01_01-00_00-12345/ ~/my_agg/ 28.0 -``` - ---- - -## Programs Reference - -### Collecting metrics - -`execution-metrics-collector` runs a command and monitors it and all its child processes: - -```bash -execution-metrics-collector {base_metrics_directory} {command} {args...} -``` - -Internally this launches the command, captures its PID, and calls `process-metrics-collector` with a sampling period of 1 second: - -```bash -process-metrics-collector {pid} {base_metrics_directory} [sample_period] -``` - -Example: - -```bash -execution-metrics-collector ~/metrics python myscript.py --input data.txt -``` - ---- - -### Plotting time series charts - -`plotGraph` generates line charts for each monitored process, comparing time series of CPU, memory, I/O and other metrics: - -```bash -plotGraph {metrics_directory} {output_directory} -``` - -Example: - -```bash -plotGraph ~/metrics/2025_01_01-00_00-12345/ ~/charts/ -``` - ---- - -### Finding CPU TDP - -Three programs are available depending on what information you have: - -#### `tdp-finder` — from a metrics directory - -```bash -tdp-finder {metrics_directory} {csv_files...} ``` -Example: +For container-based collection and subsequent analysis, see the +[usage guide](docs/usage.md). The standalone binary only provides the collector; +analysis requires a separate Python installation. -```bash -tdp-finder ~/metrics/2025_01_01-00_00-12345/ cpu-spec-dataset_Josua/dataset/*.csv cpumark_table.csv -``` +## Repository -Use `-q` for quiet mode (outputs only the TDP value, useful for scripting): - -```bash -tdp-finder -q ~/metrics/2025_01_01-00_00-12345/ cpu-spec-dataset_Josua/dataset/*.csv -# Output: 28.0 -``` - -#### `cpuinfo-tdp-finder` — from `/proc/cpuinfo` - -Does not require a metrics directory: - -```bash -cpuinfo-tdp-finder /proc/cpuinfo cpu-spec-dataset_Josua/dataset/*.csv cpumark_table.csv -``` - -Or from a saved copy of `/proc/cpuinfo` — the `sample_cpuinfo/` directory contains example files for Intel and AMD processors you can use for testing: - -```bash -cpuinfo-tdp-finder sample_cpuinfo/cpuinfo-amd.txt cpu-spec-dataset_Josua/dataset/*.csv -``` - -#### `modelname-tdp-finder` — from a processor model string - -```bash -modelname-tdp-finder "11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz" cpu-spec-dataset_Josua/dataset/*.csv -modelname-tdp-finder "AMD EPYC 7742 64-Core Processor" cpu-spec-dataset_Josua/dataset/*.csv cpumark_table.csv -``` - ---- - -### Digesting metrics - -`metrics-aggregator` digests the collected time series and estimates energy consumption per process subtree. It requires the CPU TDP value in Watts. - -```bash -metrics-aggregator {metrics_directory} {output_directory} {TDP_watts} [command_filter] -``` - -The optional `command_filter` argument filters results to show only processes whose command matches the string (e.g. `"docker run"` to focus on Docker steps). - -Example using the included sample series: - -```bash -metrics-aggregator sample-series/Wetlab2Variations_metrics/2025_05_20-02_19-14001/ dest_directory 28.0 "docker run" -``` - -The output directory will contain: - -- A table of energy consumption per task (stdout) -- `graph.pdf` / `graph.svg` — process call graph as a tree -- `spiral-graph.pdf` / `spiral-graph.svg` — process call graph as a spiral -- `consumptions.pdf` / `consumptions.svg` — barplot of task energy and duration -- `timeline.pdf` / `timeline.svg` — lollipop chart of task start, duration, and end - -![Sample process call graph (tree)](sample-charts/graph.svg) -![Sample process call graph (spiral)](sample-charts/spiral-graph.svg) -![Sample task consumptions and duration barplots](sample-charts/consumptions.svg) -![Sample task executions lollipop](sample-charts/timeline.svg) - ---- - -## CPU Dataset Setup - -The TDP programs require one or more CPU specification datasets to look up processor TDP values. Three sources are supported: - -**Recommended — JosuaCarl fork** (better column names): - -```bash -git clone https://github.com/JosuaCarl/cpu-spec-dataset cpu-spec-dataset_Josua -``` - -**Alternative — original felixsteinke repo:** - -```bash -git clone https://github.com/felixsteinke/cpu-spec-dataset -``` - -**CPUBenchmark scrape** (good coverage for AMD server CPUs): - -```bash -python -m treecript.tdp_sources cpumark_table.csv -``` - -You can pass multiple sources to the TDP programs and they will be tried in order: - -```bash -tdp-finder ~/metrics/dir/ cpu-spec-dataset_Josua/dataset/*.csv cpumark_table.csv -``` - -> **Tip:** if `tdp-finder` logs `Unable to match a valid processor row` for one of your CSV sources, that's usually just that particular source's naming convention not matching your `/proc/cpuinfo` model string exactly (e.g. extra vendor symbols, or `[Dual CPU]` prefixes for multi-socket listings). This is expected when passing several sources — `tdp-finder` simply moves on to the next file until it finds a match. The `intel-cpus.csv` / `ampere-cpus.csv` sources (official vendor data) tend to match most reliably for their respective vendors. - ---- - -## Output Files Reference - -Each `execution-metrics-collector` run creates a subdirectory named after the start timestamp and PID. It contains: - -| File | Description | -| ----------------------------------- | -------------------------------------------------------------- | -| `reference_pid.txt` | PID of the root process being monitored | -| `sampling-rate-seconds.txt` | Sampling rate in seconds (usually 1) | -| `pids.txt` | Table of all spawned processes with timestamps and parent PIDs | -| `agg_metrics.tsv` | Time series of aggregated metrics across all processes | -| `metrics-{pid}_{create_time}.csv` | Per-process time series metrics | -| `command-{pid}_{create_time}.txt` | Linearized command line for each process | -| `command-{pid}_{create_time}.json` | JSON representation of the command line | -| `cpu_details.json` | Physical CPU information from `/proc/cpuinfo` | -| `core_affinity.json` | Processor-to-core-to-CPU mapping derived from `/proc/cpuinfo` | - -### Per-process metrics (`metrics-{pid}_{create_time}.csv`) - -| Column | Description | -| ----------------- | ------------------------------------------------------------------------------ | -| `Time` | Sample timestamp | -| `PID` | Process ID | -| `Virt` | Virtual memory size (matches `top` VIRT) | -| `Res` | Resident set size — non-swapped physical memory (matches `top` RES) | -| `CPU` | CPU utilization as a percentage (can exceed 100% for multithreaded processes) | -| `Memory` | RSS memory as a percentage of total physical system memory | -| `TCP connections` | Number of open TCP connections | -| `Thread Count` | Number of threads (non-cumulative) | -| `User` | Time spent in user mode (seconds) | -| `System` | Time spent in kernel mode (seconds) | -| `Children_User` | User time of child processes (always 0 on Windows/macOS) | -| `Children_System` | System time of child processes (always 0 on Windows/macOS) | -| `IO` | Time waiting for blocking I/O (Linux only) | -| `uss` | Unique Set Size — memory freed if this process terminated now | -| `swap` | Memory swapped out to disk | -| `processor_num` | Number of unique CPU processors used | -| `core_num` | Number of unique CPU cores used | -| `cpu_num` | Number of unique physical CPUs used | -| `processor_ids` | IDs of CPU processors used (space-separated) | -| `core_ids` | IDs of CPU cores used (space-separated) | -| `cpu_ids` | IDs of physical CPUs used (space-separated) | -| `process_status` | Process status string (e.g. `sleeping`, `running`) | -| `read_count` | Cumulative number of read syscalls | -| `write_count` | Cumulative number of write syscalls | -| `read_bytes` | Bytes physically read from disk (cumulative) | -| `write_bytes` | Bytes physically written to disk (cumulative) | -| `read_chars` | Bytes passed to read syscalls (cumulative, Linux only) | -| `write_chars` | Bytes passed to write syscalls (cumulative, Linux only) | - -### Aggregated metrics (`agg_metrics.tsv`) - -Each row is a 1-second sample across all monitored processes combined: - -| Column | Description | -| -------------- | ----------------------------------------- | -| Timestamp | Sample time | -| Number of PIDs | Processes monitored at that moment | -| Threads | Total thread count | -| Processors | Number of distinct CPU processors in use | -| Cores | Number of distinct CPU cores in use | -| Physical CPUs | Number of distinct physical CPUs in use | -| CPU IDs | IDs of physical CPUs (space-separated) | -| User memory | Total user memory across all processes | -| Swap memory | Total swap memory across all processes | -| Read ops | Total read operations | -| Write ops | Total write operations | -| Read bytes | Bytes physically read | -| Write bytes | Bytes physically written | -| Read chars | Bytes passed to read syscalls | -| Write chars | Bytes passed to write syscalls | - ---- - -## Legacy - -The `legacy/` directory contains older Bash-based scripts that predate the current Python implementation. They are kept for historical reference but are **no longer maintained or recommended**. - -### `execution-metrics-collector.sh` - -The original Bash wrapper for launching a command and monitoring it. It runs the command in the background, captures the PID, and calls `process-metrics-collector` directly: - -```bash -./legacy/execution-metrics-collector.sh {base_metrics_directory} {command} {args...} -``` - -This has been superseded by `execution-metrics-collector`, which provides the same functionality in a more portable and maintainable way. The sample series included in this repository was originally collected using this script: - -```bash -~/projects/treecript/legacy/execution-metrics-collector.sh \ - ~/projects/treecript/Wetlab2Variations_metrics \ - python WfExS-backend.py -L workflow_examples/local_config.yaml \ - staged-workdir offline-exec 01a1db90-1508-4bad-beb7-7f7989838542 -``` - -### `plotGraph.sh` - -The original gnuplot-based visualization script. It reads the collected CSV files and generates `.pdf` charts using `gnuplot` (requires `apt install gnuplot`). It has been superseded by `plotGraph`, which generates richer charts without requiring gnuplot. - -```bash -./legacy/plotGraph.sh {metrics_csv_files...} -``` - -### `plot-metrics.sh` - -An earlier helper script for plotting individual metric files. Also superseded by `plotGraph`. - -> These scripts are no longer actively maintained. For all new usage, prefer the Python equivalents. - ---- +The Python package is in `treecript/`, installation recipes and constraints are +in `installation/`, and examples are in `sample-series/`, `sample-charts/`, +`sample-work-to-measure/`, and `onboarding/`. Tests are in `tests/`. ## License -Licensed under **GNU GPL v3**. - -This repository is a fork and evolution of [chamilad/process-metrics-collector](https://github.com/chamilad/process-metrics-collector). \ No newline at end of file +Licensed under GNU GPL v3. This repository is a fork and evolution of +[chamilad/process-metrics-collector](https://github.com/chamilad/process-metrics-collector). diff --git a/docs/installation.md b/docs/installation.md new file mode 100644 index 0000000..f65bd39 --- /dev/null +++ b/docs/installation.md @@ -0,0 +1,233 @@ +# Installation + +[Back to the main README](../README.md) + +### Prerequisites (singularity) + +- Linux OS +- [Singularity](https://sylabs.io/docs/) or [Apptainer](https://apptainer.org/). + +### Prerequisites (native) + +- Linux OS (Ubuntu recommended) +- Python 3.9 or newer +- Git + +### Not sure which installation method to use? + +| I want to... | Use | +| ------------------------------------------------------------------------------ | -------------------------------------------------------------- | +| Keep things simple, already have either Apptainer or Singularity installed and you only want to gather metrics | **Option 1 — Singularity** | +| Keep things simple and already have Python installed | **Option 2 — pip + venv** | +| Already use conda or manage multiple projects/environments | **Option 3 — Conda** | +| Work on an HPC or shared cluster environment (e.g. BSC) | **Option 1 — Singularity** or **Option 3 — Conda** | +| Deploy a standalone executable to a machine without Python or a container runtime | [Option 4 — Standalone binary (PyInstaller)](standalone-binary.md) | +| Work on a machine with a corporate or university firewall | Either — both have firewall notes in their respective sections | + +--- + +### Choosing a constraints file (only needed for option 2, option 3 or option 4) + +The repository ships per-version constraints files under the `installation/` directory to ensure a working set of dependencies. Pick the one that matches your setup (including the Python version): + +| Situation | Constraints file to use | +| ----------------------------- | ---------------------------------------------------- | +| Native Linux, Python 3.x | `installation/constraints-3.x.txt` | +| Ubuntu 22.04 on WSL (Windows) | `installation/constraints-3.x_Ubuntu-22.04-wsl.txt` | +| Ubuntu 24.04 on WSL (Windows) | `installation/constraints-3.x_Ubuntu-24.04-wsl.txt` | + +> **WSL** = Windows Subsystem for Linux — Ubuntu running inside Windows rather than directly on hardware. If you are running Ubuntu natively on your machine, use the plain constraints file. To check: +> +> ```bash +> uname -r # if the output contains "microsoft" or "WSL", you are on WSL +> ``` + +To check your Python version: + +```bash +python3 --version +``` + +--- + +### Option 1: Singularity + +#### Alternative A: Fetch a pre-built image (only for metrics gathering) + +Pre-build singularity images are listed at https://github.com/inab/treecript/pkgs/container/treecript . + +They can be fetched using Apptainer/Singularity just using `singularity pull` subcommand: + +```bash +# Replace exec by the tag of the version you want to use +singularity pull oras://ghcr.io/inab/treecript:exec + +# The name of the created file depends on the tag +ls -l treecript-exec.sif +``` + +#### Alternative B: Building from source + +When no file is locally, but you already have either Apptainer or Singularity installed: + +```bash +# TREECRIPT_VER can be either a branch, a tag or a commit hash +TREECRIPT_VER=7c0a20a688518e43952d7bd7080bc34b863e32da + +# If you don't have the recipe, you can fetch it using either curl or wget +mkdir -p treecript_SIF_build/installation +cd treecript_SIF_build/installation +curl -O https://raw.githubusercontent.com/inab/treecript/${TREECRIPT_VER}/installation/Singularity.def +cd .. +``` + +If you have already checked out the code + +```bash +# Alternatively, you can use a checked out copy +cd treecript +TREECRIPT_VER=$(git rev-parse HEAD) +``` + +At last, you can build the SIF image with next command: + +``` +singularity build --build-arg treecript_checkout="${TREECRIPT_VER}" \ + treecript-${TREECRIPT_VER}.sif installation/Singularity.def +``` + +--- + +### Option 2: pip + virtual environment (venv) + +Use this if you already have Python installed on your system and don't use conda. This is the lightest option — it creates an isolated Python environment using only tools that come built into Python, with no additional software required. + +```bash +# 1. Create a virtual environment +python3 -m venv TREECRIPT + +# 2. Activate it +source TREECRIPT/bin/activate + +# 3. Upgrade pip and wheel +pip install --upgrade pip wheel + +# 4. Download the constraints file for your Python version (adjust filename as needed) +wget https://raw.githubusercontent.com/inab/treecript/exec/installation/constraints-3.10.txt + +# 5. Install treecript with constraints +pip install -c constraints-3.10.txt 'treecript [analytics,docker] @ git+https://github.com/inab/treecript.git@exec' +``` + +> **Network issues?** Consult your network administrator about proxy and +> trusted certificate configuration. + +To deactivate the environment: + +```bash +deactivate +``` + +To reactivate later: + +```bash +source TREECRIPT/bin/activate +``` + +--- + +### Option 3: Conda environment + +Use this if you already work with Anaconda or Miniconda, or if you prefer conda for managing environments across multiple projects. Conda handles both Python and system-level dependencies, which makes it particularly well suited for HPC or shared computing environments. + +#### Installing Miniconda (if not already installed) + +Miniconda is a minimal conda installer — it gives you the `conda` command and Python without bundling hundreds of extra packages like the full Anaconda distribution does. + +```bash +# Download installer +wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh + +# Run the installer +bash miniconda.sh +# - Accept the license +# - Accept the default install location +# - Answer "yes" when asked to update your shell profile + +# Apply changes to current session +source ~/.bashrc + +# Verify +conda --version +``` + +> **Network issues?** If `repo.anaconda.com` is blocked by your network, use **Miniforge** instead — it is functionally identical to Miniconda but downloads from GitHub and defaults to the `conda-forge` channel, which is actually a better fit for the scientific packages treecript needs: +> +> ```bash +> wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh -O miniconda.sh +> bash miniconda.sh +> ``` + +#### Creating the treecript conda environment + +```bash +# 1. Create a clean environment with Python 3.10 +conda create -n treecript python=3.10 -y + +# 2. Activate it +conda activate treecript + +# 3. Download the constraints file (adjust filename for your Python version / OS) +wget https://raw.githubusercontent.com/inab/treecript/exec/installation/constraints-3.10.txt +# or for WSL Ubuntu 22.04: +# wget https://raw.githubusercontent.com/inab/treecript/exec/installation/constraints-3.10_Ubuntu-22.04-wsl.txt + +# 4. Install treecript and all dependencies in one shot +pip install -c constraints-3.10.txt 'treecript [analytics,docker] @ git+https://github.com/inab/treecript.git@exec' +``` + +To deactivate: + +```bash +conda deactivate +``` + +To remove the environment entirely: + +```bash +conda deactivate +conda remove -n treecript --all -y +``` + +--- + +### Option 4: Standalone binary + +See the [standalone binary guide](standalone-binary.md) for the existing +Docker-built artifact, Apptainer/Singularity and Docker build instructions, +and system compatibility requirements. + +### Verifying the installation + +This check applies to a Python installation, including a separate environment +used to analyze metrics collected with the [standalone binary](standalone-binary.md). +It does not apply to the binary itself. + +Run this after Option 2 (pip + venv) or Option 3 (Conda) — or after setting up the local analysis environment from Option 4's workflow example — to confirm all dependencies are working correctly: + +```bash +python -c " +import psutil; print('psutil OK:', psutil.__version__) +import docker; print('docker OK:', docker.__version__) +import pandas; print('pandas OK:', pandas.__version__) +import networkx; print('networkx OK:', networkx.__version__) +import matplotlib; print('matplotlib OK:', matplotlib.__version__) +import adjustText; print('adjustText OK:', adjustText.__version__) +import treecript; print('treecript OK') +" +``` + +> To check the binary itself, run it with no arguments and confirm it prints +> its usage message; also test a representative collection on the target node. + +--- diff --git a/docs/legacy.md b/docs/legacy.md new file mode 100644 index 0000000..acf5f8c --- /dev/null +++ b/docs/legacy.md @@ -0,0 +1,38 @@ +# Legacy + +[Back to the main README](../README.md) + +The `legacy/` directory contains older Bash-based scripts that predate the current Python implementation. They are kept for historical reference but are **no longer maintained or recommended**. + +### `execution-metrics-collector.sh` + +The original Bash wrapper for launching a command and monitoring it. It runs the command in the background, captures the PID, and calls `process-metrics-collector` directly: + +```bash +./legacy/execution-metrics-collector.sh {base_metrics_directory} {command} {args...} +``` + +This has been superseded by `execution-metrics-collector`, which provides the same functionality in a more portable and maintainable way. The sample series included in this repository was originally collected using this script: + +```bash +~/projects/treecript/legacy/execution-metrics-collector.sh \ + ~/projects/treecript/Wetlab2Variations_metrics \ + python WfExS-backend.py -L workflow_examples/local_config.yaml \ + staged-workdir offline-exec 01a1db90-1508-4bad-beb7-7f7989838542 +``` + +### `plotGraph.sh` + +The original gnuplot-based visualization script. It reads the collected CSV files and generates `.pdf` charts using `gnuplot` (requires `apt install gnuplot`). It has been superseded by `plotGraph`, which generates richer charts without requiring gnuplot. + +```bash +./legacy/plotGraph.sh {metrics_csv_files...} +``` + +### `plot-metrics.sh` + +An earlier helper script for plotting individual metric files. Also superseded by `plotGraph`. + +> These scripts are no longer actively maintained. For all new usage, prefer the Python equivalents. + +--- diff --git a/docs/metrics.md b/docs/metrics.md new file mode 100644 index 0000000..561fe28 --- /dev/null +++ b/docs/metrics.md @@ -0,0 +1,74 @@ +# Output files and metrics + +[Back to the main README](../README.md) | [Usage](usage.md) + +Each `execution-metrics-collector` run creates a subdirectory named after the start timestamp and PID. It contains: + +| File | Description | +| ----------------------------------- | -------------------------------------------------------------- | +| `reference_pid.txt` | PID of the root process being monitored | +| `sampling-rate-seconds.txt` | Sampling rate in seconds (usually 1) | +| `pids.txt` | Table of all spawned processes with timestamps and parent PIDs | +| `agg_metrics.tsv` | Time series of aggregated metrics across all processes | +| `metrics-{pid}_{create_time}.csv` | Per-process time series metrics | +| `command-{pid}_{create_time}.txt` | Linearized command line for each process | +| `command-{pid}_{create_time}.json` | JSON representation of the command line | +| `cpu_details.json` | Physical CPU information from `/proc/cpuinfo` | +| `core_affinity.json` | Processor-to-core-to-CPU mapping derived from `/proc/cpuinfo` | + +### Per-process metrics (`metrics-{pid}_{create_time}.csv`) + +| Column | Description | +| ----------------- | ------------------------------------------------------------------------------ | +| `Time` | Sample timestamp | +| `PID` | Process ID | +| `Virt` | Virtual memory size (matches `top` VIRT) | +| `Res` | Resident set size — non-swapped physical memory (matches `top` RES) | +| `CPU` | CPU utilization as a percentage (can exceed 100% for multithreaded processes) | +| `Memory` | RSS memory as a percentage of total physical system memory | +| `TCP connections` | Number of open TCP connections | +| `Thread Count` | Number of threads (non-cumulative) | +| `User` | Time spent in user mode (seconds) | +| `System` | Time spent in kernel mode (seconds) | +| `Children_User` | User time of child processes (always 0 on Windows/macOS) | +| `Children_System` | System time of child processes (always 0 on Windows/macOS) | +| `IO` | Time waiting for blocking I/O (Linux only) | +| `uss` | Unique Set Size — memory freed if this process terminated now | +| `swap` | Memory swapped out to disk | +| `processor_num` | Number of unique CPU processors used | +| `core_num` | Number of unique CPU cores used | +| `cpu_num` | Number of unique physical CPUs used | +| `processor_ids` | IDs of CPU processors used (space-separated) | +| `core_ids` | IDs of CPU cores used (space-separated) | +| `cpu_ids` | IDs of physical CPUs used (space-separated) | +| `process_status` | Process status string (e.g. `sleeping`, `running`) | +| `read_count` | Cumulative number of read syscalls | +| `write_count` | Cumulative number of write syscalls | +| `read_bytes` | Bytes physically read from disk (cumulative) | +| `write_bytes` | Bytes physically written to disk (cumulative) | +| `read_chars` | Bytes passed to read syscalls (cumulative, Linux only) | +| `write_chars` | Bytes passed to write syscalls (cumulative, Linux only) | + +### Aggregated metrics (`agg_metrics.tsv`) + +Each row is a 1-second sample across all monitored processes combined: + +| Column | Description | +| -------------- | ----------------------------------------- | +| Timestamp | Sample time | +| Number of PIDs | Processes monitored at that moment | +| Threads | Total thread count | +| Processors | Number of distinct CPU processors in use | +| Cores | Number of distinct CPU cores in use | +| Physical CPUs | Number of distinct physical CPUs in use | +| CPU IDs | IDs of physical CPUs (space-separated) | +| User memory | Total user memory across all processes | +| Swap memory | Total swap memory across all processes | +| Read ops | Total read operations | +| Write ops | Total write operations | +| Read bytes | Bytes physically read | +| Write bytes | Bytes physically written | +| Read chars | Bytes passed to read syscalls | +| Write chars | Bytes passed to write syscalls | + +--- diff --git a/docs/standalone-binary.md b/docs/standalone-binary.md new file mode 100644 index 0000000..ba2ace3 --- /dev/null +++ b/docs/standalone-binary.md @@ -0,0 +1,299 @@ +# Standalone collector binary (PyInstaller) + +[Back to the main README](../README.md) | [Installation](installation.md) | [Usage](usage.md) + +Use this to run `execution-metrics-collector` without installing Python, pip, +or a container runtime on the target machine. [PyInstaller](https://pypi.org/project/pyinstaller/) +bundles the Python interpreter and application dependencies into one executable. +It still needs a compatible Linux system, architecture and glibc; see +[compatibility](#checking-glibc-compatibility-before-deploying). + +#### Alternative A: Use a pre-built binary + +A Docker-built binary of `execution-metrics-collector` is published as the OCI +artifact `ghcr.io/inab/treecript:exec-binary-linux-x86_64` in the repository's +[Packages](https://github.com/inab/treecript/pkgs/container/treecript). It is +distinct from the [Singularity runtime image](installation.md#option-1-singularity). + +**What it is:** a Linux x86-64 single-file executable, built with PyInstaller +inside a `manylinux_2_28` container. Glibc 2.28 is the intended baseline, not a +guarantee that every bundled library works on every system. Verify the actual +artifact and test on the target system. + +Binaries are pushed and pulled as [OCI artifacts](https://oras.land/) using [`oras`](https://oras.land/docs/installation), the same underlying tool `singularity pull oras://...` uses for the container images above. + +**Downloading it:** + +```bash +# Install oras once, if you don't have it: https://oras.land/docs/installation +# Access may require oras login if the package is private. +oras pull ghcr.io/inab/treecript:exec-binary-linux-x86_64 + +chmod +x execution-metrics-collector +./execution-metrics-collector +``` + +**Using it on an HPC cluster:** it can be pulled and run on a compatible login +node, or inside an interactive Slurm allocation: + +```bash +# Option A: pull and run directly on the login node +oras pull ghcr.io/inab/treecript:exec-binary-linux-x86_64 +chmod +x execution-metrics-collector +./execution-metrics-collector ~/my_metrics my_command --arg1 --arg2 + +# Option B: interactive job via Slurm (salloc) — recommended for anything +# that needs dedicated compute resources rather than sharing the login node +salloc -n 1 -c 4 -t 00:30:00 # adjust cores/time/partition to your cluster +# once the allocation starts and you're placed on a compute node: +./execution-metrics-collector ~/my_metrics my_command --arg1 --arg2 +exit # ends the salloc session when done +``` + +Before relying on this binary on a new target machine, check architecture and +glibc on the actual compute node as described [below](#checking-glibc-compatibility-before-deploying): + +```bash +uname -m +ldd --version | head -1 +``` + +An older glibc or a different architecture needs another compatible build; the +recipes below target x86-64 and do not automatically support older systems. + +**Publishing a new version (maintainers):** + +```bash +# 1. Log in to GitHub Container Registry (needs a PAT with write:packages scope) +echo "$GITHUB_TOKEN" | oras login ghcr.io -u your-github-username --password-stdin + +# 2. Build the binary (see "Building the binary" below), then push it as an +# OCI artifact — no Dockerfile/container involved, just the raw file +oras push ghcr.io/inab/treecript:exec-binary-linux-x86_64 \ + dist/execution-metrics-collector:application/octet-stream + +# 3. Confirm it shows up under the repository's Packages tab +``` + +Avoid overwriting an existing tag with an untested build. If publishing a second +binary, use a distinct tag identifying the treecript commit, architecture, +glibc baseline and builder, and provide checksums and test results. A separate +Apptainer-built artifact is not necessary when it offers no tested compatibility +or reproducibility advantage over the existing Docker-built binary. + +#### Alternative B: Building it yourself + +##### Why use a build container + +The binary must be built on a machine whose **glibc version is equal to or older than** the target machine's glibc, because glibc is forward-compatible but not backward-compatible: a binary built against a newer glibc will refuse to run on a system with an older one, failing with an error such as: + +``` +/lib64/libc.so.6: version `GLIBC_2.35' not found (required by execution-metrics-collector) +``` + +Development laptops typically run a fairly recent Linux distribution (e.g. Ubuntu 22.04, glibc 2.35), while HPC login/compute nodes often run older, more conservative distributions (e.g. glibc 2.28–2.34). Building directly on the laptop would therefore very likely produce a binary that cannot run on the cluster. + +One approach is to build inside a [`manylinux`](https://github.com/pypa/manylinux) +container, which provides a defined glibc baseline: + +```bash +# Check your target machine's glibc version: +ssh your-user@your-hpc-login-node "ldd --version | head -1" + +# Check your build machine's glibc version: +ldd --version | head -1 +``` + +If the target's glibc is older than the build host's, choose a compatible build +environment. These recipes use `manylinux_2_28_x86_64`; supporting an older +glibc or another architecture requires adapting and testing the recipe and +its dependencies, not merely changing the image name. + +> **Note:** the Python interpreters preinstalled inside `manylinux` images (under `/opt/python/cpXXX-cpXXX`) are built **without** a shared `libpython` — they are meant for compiling wheels, not for running applications. PyInstaller requires a shared `libpythonX.Y.so`, so the container build compiles its own Python from source with `--enable-shared`. It also needs `openssl-devel` (and other `-devel` headers) installed **before** compiling Python, otherwise Python's `ssl` module is silently skipped and `pip` cannot reach PyPI. + +##### Build with Apptainer or Singularity + +The [builder definition](../treecript-builder.def) builds a **builder image**; +it is not the [runtime image](../installation/Singularity.def). Its `Bootstrap: +docker` line fetches an OCI base image; a Docker daemon is not required. +Run the following from the repository root on Linux x86-64, with Apptainer +installed, network access, and sufficient disk space. Building from a definition +file requires root, a working fakeroot configuration, or another supported +build service; check the cluster's policy. Singularity can be used instead by +replacing `apptainer` with `singularity` if its version supports these options. + +The definition expects `constraints-3.12.txt` in the build context, not in +`installation/`. Stage a copy without changing the source files: + +```bash +mkdir -p build/pyinstaller-context build/pyinstaller-output +cp treecript-builder.def build/pyinstaller-context/ +cp installation/constraints-3.12.txt build/pyinstaller-context/ +( + cd build/pyinstaller-context + apptainer build --fakeroot --build-arg TREECRIPT_REF=exec \ + treecript-builder.sif treecript-builder.def +) + +# The runscript writes build/, dist/ and a .spec file to /work. +apptainer run --bind "$PWD/build/pyinstaller-output:/work" --pwd /work \ + build/pyinstaller-context/treecript-builder.sif + +build/pyinstaller-output/dist/execution-metrics-collector +``` + +The final command should print usage and exit with status 1 because no arguments +were provided; also test a real workload on the target. +The recipe installs the **remote** `TREECRIPT_REF` (default `exec`), not +uncommitted local changes. Override `TREECRIPT_GIT_URL` and `TREECRIPT_REF` +with `--build-arg` to select a repository and preferably an immutable commit. +Record that commit and the image/dependency versions with the artifact. + +The recipe compiles Python 3.13.9 but uses the repository's 3.12 constraints +file. If changing `PYVER`, also update the hard-coded `python3.13` executable +and select appropriate constraints; do not assume arbitrary Python versions +will work. During `%post`, it explicitly exports `LD_LIBRARY_PATH` because +`%environment` applies at runtime, not while building the image. + +##### Build with Docker + +Save the following as `Dockerfile` in a build directory containing a copy of +`installation/constraints-3.12.txt` named `constraints-3.12.txt`. Like the +Apptainer definition, this example uses Python 3.13 with 3.12 constraints and +installs the selected remote ref, not local changes. + +```dockerfile +FROM quay.io/pypa/manylinux_2_28_x86_64 + +ARG PYVER=3.13.9 +ARG TREECRIPT_GIT_URL=https://github.com/inab/treecript.git +ARG TREECRIPT_REF=exec + +ENV PYSHARED_PREFIX=/opt/python-shared +ENV LD_LIBRARY_PATH=${PYSHARED_PREFIX}/lib + +# Build-time dependencies for compiling Python (openssl-devel is required, +# otherwise Python ends up without the ssl module and pip cannot reach PyPI) +RUN dnf install -y \ + openssl-devel bzip2-devel libffi-devel zlib-devel xz-devel \ + ncurses-devel readline-devel sqlite-devel tk-devel gdbm-devel \ + && dnf clean all + +# Compile Python with --enable-shared, required by PyInstaller +RUN curl -O https://www.python.org/ftp/python/${PYVER}/Python-${PYVER}.tgz \ + && tar xzf Python-${PYVER}.tgz \ + && cd Python-${PYVER} \ + && ./configure --enable-shared --prefix=${PYSHARED_PREFIX} --with-openssl=/usr \ + && make -j"$(nproc)" \ + && make altinstall \ + && cd .. && rm -rf Python-${PYVER} Python-${PYVER}.tgz + +ENV PYBIN=${PYSHARED_PREFIX}/bin/python3.13 +WORKDIR /src + +RUN ${PYBIN} -m venv /opt/venv +ENV PATH=/opt/venv/bin:$PATH + +COPY constraints-3.12.txt /src/constraints-3.12.txt + +RUN /opt/venv/bin/pip install --upgrade pip wheel \ + && /opt/venv/bin/pip install -c /src/constraints-3.12.txt \ + "treecript[analytics,docker] @ git+${TREECRIPT_GIT_URL}@${TREECRIPT_REF}" \ + && /opt/venv/bin/pip install pyinstaller + +CMD ["/opt/venv/bin/pyinstaller", "-F", "-n", "execution-metrics-collector", \ + "/opt/venv/bin/execution-metrics-collector"] +``` + +Build the image once (this recompiles Python from source, so it takes several minutes the first time; subsequent builds reuse the cached layer as long as `PYVER` doesn't change): + +```bash +docker build -t treecript-builder \ + --build-arg TREECRIPT_GIT_URL=https://github.com/inab/treecript.git \ + --build-arg TREECRIPT_REF=exec \ + -f Dockerfile . +``` + +Then generate the binary itself (fast — reuses the already-built image): + +```bash +docker run --rm -v "$PWD/dist:/src/dist" treecript-builder +``` + +The resulting `dist/execution-metrics-collector` is a single, self-contained executable (`--onefile` / `-F` build). + +> **Why only `execution-metrics-collector`?** The binary is for collection on +> a node without Python. Run the analysis tools afterwards in a separate +> [Python environment](installation.md), as in the +> [full workflow example](#full-workflow-example-hpc-metrics-collection--local-analysis). + +##### Checking glibc compatibility before deploying + +Check the executable's architecture and the glibc symbols required by its +outer PyInstaller bootloader (adjust the path for the Apptainer output): + +```bash +file dist/execution-metrics-collector +objdump -T dist/execution-metrics-collector | grep GLIBC_ \ + | sed 's/.*GLIBC_\([0-9.]*\).*/\1/' | sort -V | tail -1 +``` + +The reported version must not exceed the target's glibc, but this check alone +is **not sufficient**: one-file bundles also contain native libraries whose +requirements are not checked by `objdump` on the outer executable. Verify +those libraries separately and test a representative collection on the actual +compute nodes. Other system libraries, kernel features, permissions, and CPU +instruction sets can also affect compatibility. PyInstaller extracts a one-file +bundle at startup; its temporary directory must be writable and permit execution. + +##### Troubleshooting + +| Symptom | Cause | Fix | +| --- | --- | --- | +| `version 'GLIBC_2.XX' not found` when running the binary | The bootloader or a bundled library requires newer glibc than the target | Adapt the build and dependencies for an older compatible environment, then test on the target | +| `Python was built without a shared library` during `pyinstaller` build | Using the manylinux-provided Python instead of a `--enable-shared` build | Compile Python from source with `--enable-shared`, as shown above | +| `SSL module is not available` / pip cannot reach PyPI while building the image | `openssl-devel` (and related `-devel` headers) missing before compiling Python | Install `openssl-devel` (and the other `-devel` packages listed above) before the `./configure && make` step | +| Files generated by `docker run` are root-owned | The container runs as `root` | Run with an appropriate user and writable working directory, or ask an administrator to correct ownership | +| Apptainer build fails with a fakeroot error | Fakeroot is unavailable or not configured | Ask the administrator about fakeroot or a supported build service | +| One-file binary fails while extracting or loading libraries | Temporary directory is not writable or is mounted `noexec` | Set `TMPDIR` to a writable executable filesystem allowed by cluster policy | + +#### Full workflow example: HPC metrics collection + local analysis + +Build with either method above or download the published artifact, then check +compatibility before copying it to the cluster. For example, from the repository +root after the Apptainer build: + +```bash +scp build/pyinstaller-output/dist/execution-metrics-collector your-user@your-hpc-login-node:~/ +ssh your-user@your-hpc-login-node +``` + +On the cluster, check the target compute node and follow its job policy: + +```bash +chmod +x execution-metrics-collector +salloc -n 1 -c 4 -t 00:30:00 # adjust to your cluster +uname -m +ldd --version | head -1 +./execution-metrics-collector ~/my_metrics my_command --arg1 --arg2 +exit # end the allocation; then exit the SSH session +``` + +Back on the laptop, replace the example run directory with the one created on +the cluster. Install a separate [Python environment](installation.md) for +analysis, and activate it before running these commands: + +```bash +RUN=2025_01_01-00_00-12345 +mkdir -p my_metrics +scp -r "your-user@your-hpc-login-node:my_metrics/$RUN" my_metrics/ +git clone https://github.com/JosuaCarl/cpu-spec-dataset cpu-spec-dataset_Josua +plotGraph "my_metrics/$RUN" my_charts/ +tdp-finder "my_metrics/$RUN" cpu-spec-dataset_Josua/dataset/*.csv +# Replace 350.0 with the TDP found for your compute node, in watts. +metrics-aggregator "my_metrics/$RUN" my_agg/ 350.0 +``` + +> **Tip:** if `tdp-finder` prints `Unable to match a valid processor row` for some of the CSV sources you passed, that's normal — different sources use slightly different naming conventions for the same CPU. `tdp-finder` tries each file in order and stops at the first match; passing several sources increases the odds of a hit. The official vendor CSVs (e.g. `intel-cpus.csv`) tend to be the most reliable matches. + +--- diff --git a/docs/usage.md b/docs/usage.md new file mode 100644 index 0000000..4f53425 --- /dev/null +++ b/docs/usage.md @@ -0,0 +1,184 @@ +# Usage and CPU datasets + +[Back to the main README](../README.md) | [Metrics reference](metrics.md) + +Run the examples from the repository root. + +## Quick Start (Singularity/Apptainer) + +At the moment, it is limited to metrics gathering + +```bash +# 1. Collect metrics for a command in background +my_command --arg1 --arg2 & + +# bash puts the PID of the last background process in $! +singularity exec treecript-exec.sif process-metrics-collector $! ~/my_metrics +``` + +## Quick Start (other options) + +```bash +# 1. Collect metrics for a command +execution-metrics-collector ~/my_metrics my_command --arg1 --arg2 + +# 2. Plot time series charts +plotGraph ~/my_metrics/2025_01_01-00_00-12345/ ~/my_charts/ + +# 3. Find your CPU's TDP +tdp-finder ~/my_metrics/2025_01_01-00_00-12345/ cpu-spec-dataset_Josua/dataset/*.csv + +# 4. Aggregate and estimate energy consumption +metrics-aggregator ~/my_metrics/2025_01_01-00_00-12345/ ~/my_agg/ 28.0 +``` + +--- + +## Programs Reference + +### Collecting metrics + +`execution-metrics-collector` runs a command and monitors it and all its child processes: + +```bash +execution-metrics-collector {base_metrics_directory} {command} {args...} +``` + +Internally this launches the command, captures its PID, and calls `process-metrics-collector` with a sampling period of 1 second: + +```bash +process-metrics-collector {pid} {base_metrics_directory} [sample_period] +``` + +Example: + +```bash +execution-metrics-collector ~/metrics python myscript.py --input data.txt +``` + +--- + +### Plotting time series charts + +`plotGraph` generates line charts for each monitored process, comparing time series of CPU, memory, I/O and other metrics: + +```bash +plotGraph {metrics_directory} {output_directory} +``` + +Example: + +```bash +plotGraph ~/metrics/2025_01_01-00_00-12345/ ~/charts/ +``` + +--- + +### Finding CPU TDP + +Three programs are available depending on what information you have: + +#### `tdp-finder` — from a metrics directory + +```bash +tdp-finder {metrics_directory} {csv_files...} +``` + +Example: + +```bash +tdp-finder ~/metrics/2025_01_01-00_00-12345/ cpu-spec-dataset_Josua/dataset/*.csv cpumark_table.csv +``` + +Use `-q` for quiet mode (outputs only the TDP value, useful for scripting): + +```bash +tdp-finder -q ~/metrics/2025_01_01-00_00-12345/ cpu-spec-dataset_Josua/dataset/*.csv +# Output: 28.0 +``` + +#### `cpuinfo-tdp-finder` — from `/proc/cpuinfo` + +Does not require a metrics directory: + +```bash +cpuinfo-tdp-finder /proc/cpuinfo cpu-spec-dataset_Josua/dataset/*.csv cpumark_table.csv +``` + +Or from a saved copy of `/proc/cpuinfo` — the `sample_cpuinfo/` directory contains example files for Intel and AMD processors you can use for testing: + +```bash +cpuinfo-tdp-finder sample_cpuinfo/cpuinfo-amd.txt cpu-spec-dataset_Josua/dataset/*.csv +``` + +#### `modelname-tdp-finder` — from a processor model string + +```bash +modelname-tdp-finder "11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz" cpu-spec-dataset_Josua/dataset/*.csv +modelname-tdp-finder "AMD EPYC 7742 64-Core Processor" cpu-spec-dataset_Josua/dataset/*.csv cpumark_table.csv +``` + +--- + +### Digesting metrics + +`metrics-aggregator` digests the collected time series and estimates energy consumption per process subtree. It requires the CPU TDP value in Watts. + +```bash +metrics-aggregator {metrics_directory} {output_directory} {TDP_watts} [command_filter] +``` + +The optional `command_filter` argument filters results to show only processes whose command matches the string (e.g. `"docker run"` to focus on Docker steps). + +Example using the included sample series: + +```bash +metrics-aggregator sample-series/Wetlab2Variations_metrics/2025_05_20-02_19-14001/ dest_directory 28.0 "docker run" +``` + +The output directory will contain: + +- A table of energy consumption per task (stdout) +- `graph.pdf` / `graph.svg` — process call graph as a tree +- `spiral-graph.pdf` / `spiral-graph.svg` — process call graph as a spiral +- `consumptions.pdf` / `consumptions.svg` — barplot of task energy and duration +- `timeline.pdf` / `timeline.svg` — lollipop chart of task start, duration, and end + +![Sample process call graph (tree)](../sample-charts/graph.svg) +![Sample process call graph (spiral)](../sample-charts/spiral-graph.svg) +![Sample task consumptions and duration barplots](../sample-charts/consumptions.svg) +![Sample task executions lollipop](../sample-charts/timeline.svg) + +--- + +## CPU Dataset Setup + +The TDP programs require one or more CPU specification datasets to look up processor TDP values. Three sources are supported: + +**Recommended — JosuaCarl fork** (better column names): + +```bash +git clone https://github.com/JosuaCarl/cpu-spec-dataset cpu-spec-dataset_Josua +``` + +**Alternative — original felixsteinke repo:** + +```bash +git clone https://github.com/felixsteinke/cpu-spec-dataset +``` + +**CPUBenchmark scrape** (good coverage for AMD server CPUs): + +```bash +python -m treecript.tdp_sources cpumark_table.csv +``` + +You can pass multiple sources to the TDP programs and they will be tried in order: + +```bash +tdp-finder ~/metrics/dir/ cpu-spec-dataset_Josua/dataset/*.csv cpumark_table.csv +``` + +> **Tip:** if `tdp-finder` logs `Unable to match a valid processor row` for one of your CSV sources, that's usually just that particular source's naming convention not matching your `/proc/cpuinfo` model string exactly (e.g. extra vendor symbols, or `[Dual CPU]` prefixes for multi-socket listings). This is expected when passing several sources — `tdp-finder` simply moves on to the next file until it finds a match. The `intel-cpus.csv` / `ampere-cpus.csv` sources (official vendor data) tend to match most reliably for their respective vendors. + +--- diff --git a/treecript-builder.def b/treecript-builder.def new file mode 100644 index 0000000..b0478ab --- /dev/null +++ b/treecript-builder.def @@ -0,0 +1,65 @@ +Bootstrap: docker +From: quay.io/pypa/manylinux_2_28_x86_64 + +# Build-time variables, equivalent to the Dockerfile's ARG. Override at +# build time with --build-arg NAME=value (see the build command below). +%arguments + PYVER=3.13.9 + TREECRIPT_GIT_URL=https://github.com/inab/treecript.git + TREECRIPT_REF=exec + +# Copies constraints-3.12.txt from the build context (the directory you +# run `singularity build` from) into the image, same as the Dockerfile's +# COPY instruction. This file must exist alongside this .def file. +%files + constraints-3.12.txt /src/constraints-3.12.txt + +%post + set -e + + # Build-time dependencies for compiling Python (openssl-devel is + # required, otherwise Python ends up without the ssl module and pip + # cannot reach PyPI) + dnf install -y \ + openssl-devel bzip2-devel libffi-devel zlib-devel xz-devel \ + ncurses-devel readline-devel sqlite-devel tk-devel gdbm-devel + dnf clean all + + PYSHARED_PREFIX=/opt/python-shared + mkdir -p /src + + # Compile Python with --enable-shared, required by PyInstaller (the + # manylinux-provided Pythons under /opt/python/cpXXX-cpXXX lack a + # shared libpython) + curl -O https://www.python.org/ftp/python/{{ PYVER }}/Python-{{ PYVER }}.tgz + tar xzf Python-{{ PYVER }}.tgz + cd Python-{{ PYVER }} + ./configure --enable-shared --prefix="${PYSHARED_PREFIX}" --with-openssl=/usr + make -j"$(nproc)" + make altinstall + cd .. + rm -rf Python-{{ PYVER }} Python-{{ PYVER }}.tgz + + # IMPORTANT: unlike Docker's ENV, Apptainer's %environment section only + # applies when the final container is *run* (%runscript, singularity + # run/exec) -- it is NOT in effect during %post (the build itself). So + # the freshly-compiled Python can't find its own shared libpython here + # unless we export this explicitly for the rest of the build. + export LD_LIBRARY_PATH="${PYSHARED_PREFIX}/lib" + + # Virtual environment + treecript + pyinstaller, same as the Dockerfile + "${PYSHARED_PREFIX}/bin/python3.13" -m venv /opt/venv + /opt/venv/bin/pip install --upgrade pip wheel + /opt/venv/bin/pip install -c /src/constraints-3.12.txt \ + "treecript[analytics,docker] @ git+{{ TREECRIPT_GIT_URL }}@{{ TREECRIPT_REF }}" + /opt/venv/bin/pip install pyinstaller + +%environment + export LD_LIBRARY_PATH=/opt/python-shared/lib + export PATH=/opt/venv/bin:$PATH + +# Equivalent to the Dockerfile's CMD: this is what runs by default when +# the resulting .sif is executed with `singularity run`. +%runscript + exec /opt/venv/bin/pyinstaller -F -n execution-metrics-collector \ + /opt/venv/bin/execution-metrics-collector From 35c6e026b50f8a0d90973ef8c40f1274f70ac115 Mon Sep 17 00:00:00 2001 From: Andrea Date: Fri, 6 Nov 2026 12:21:42 +0100 Subject: [PATCH 2/4] Update workflow runners to use ubuntu-24.04 for pip-audit and pre-commit jobs --- .github/workflows/pip-audit.yml | 2 +- .github/workflows/pre-commit.yml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/pip-audit.yml b/.github/workflows/pip-audit.yml index f327782..11ed12d 100644 --- a/.github/workflows/pip-audit.yml +++ b/.github/workflows/pip-audit.yml @@ -7,7 +7,7 @@ on: jobs: pip-audit: - runs-on: ubuntu-latest + runs-on: ubuntu-24.04 strategy: matrix: python-version: [ "3.9", "3.10", "3.11", "3.12", "3.13", "3.14" ] diff --git a/.github/workflows/pre-commit.yml b/.github/workflows/pre-commit.yml index bb6e5b1..5b8b880 100644 --- a/.github/workflows/pre-commit.yml +++ b/.github/workflows/pre-commit.yml @@ -10,7 +10,7 @@ on: jobs: pre-commit: - runs-on: ubuntu-latest + runs-on: ubuntu-24.04 strategy: matrix: python-version: [ "3.9", "3.10", "3.11", "3.12", "3.13", "3.14" ] From 6a520992270dc923bdd55f49df432630bae31f2b Mon Sep 17 00:00:00 2001 From: Andrea Date: Fri, 6 Nov 2026 12:36:58 +0100 Subject: [PATCH 3/4] Add documentation index and enhance README structure --- README.md | 46 +++++++++++++++++++++++++++++++++++++++++++--- docs/README.md | 34 ++++++++++++++++++++++++++++++++++ 2 files changed, 77 insertions(+), 3 deletions(-) create mode 100644 docs/README.md diff --git a/README.md b/README.md index a75ed61..72921d7 100644 --- a/README.md +++ b/README.md @@ -7,6 +7,8 @@ visualize the results. It was initially developed for ## Documentation +The full list is in the [documentation index](docs/README.md). + - [Installation](docs/installation.md): Singularity/Apptainer, pip, and Conda. - [Standalone collector binary](docs/standalone-binary.md): download the existing Docker-built binary, build with Docker or Apptainer/Singularity, and check @@ -34,9 +36,47 @@ analysis requires a separate Python installation. ## Repository -The Python package is in `treecript/`, installation recipes and constraints are -in `installation/`, and examples are in `sample-series/`, `sample-charts/`, -`sample-work-to-measure/`, and `onboarding/`. Tests are in `tests/`. +```text +treecript/ +├── treecript/ Python package (the actual implementation) +│ ├── collector.py Process-tree metrics collection +│ ├── parser.py Reads collected metrics +│ ├── aggregator.py Digests metrics and estimates energy consumption +│ ├── plot_graph.py Time series charts +│ ├── tdp_finder.py CPU TDP lookup (uses tdp_sources.py) +│ ├── rocrate.py RO-Crate generation +│ └── common.py Shared helpers +├── *.py (top level) Thin wrappers of the commands (see table below) +├── tests/ Unit tests +├── docs/ Documentation (start at docs/README.md) +├── installation/ Requirements, constraints and container recipes +├── treecript-builder.def Apptainer/Singularity recipe for the standalone binary +├── setup.py, pyproject.toml Packaging +├── data/ Local CPU datasets (populated as in docs/usage.md) +├── legacy/ Old Bash scripts (unmaintained) +├── onboarding/ First installation and measurement walk-through +├── sample-work-to-measure/ Scripts that run sample workloads +├── sample/, sample-series/, Example metrics, charts and CPU info files +│ sample-charts/, sample_cpuinfo/ +└── .github/workflows/ CI: pre-commit, pip-audit, container builds +``` + +| Command | Module | Purpose | +|---------|--------|---------| +| `execution-metrics-collector` | `treecript.collector` | Run a command and collect its process tree metrics. | +| `process-metrics-collector` | `treecript.collector` | Collect metrics of an already running PID. | +| `plotGraph` | `treecript.plot_graph` | Plot time series charts.* | +| `metrics-aggregator` | `treecript.aggregator` | Aggregate metrics and estimate energy.* | +| `tdp-finder`, `cpuinfo-tdp-finder`, `modelname-tdp-finder` | `treecript.tdp_finder` | Find a CPU TDP from metrics, `/proc/cpuinfo` or a model name.* | + +\* Needs the `analytics` extra (not included in the standalone binary). + +The workflows in `.github/workflows/` are: + +- `pre-commit.yml`, `pre-commit_wsl.yml`: linting, type checks, tests and licence checks. +- `pip-audit.yml`: scheduled security audit that regenerates the constraints files. +- `container.yml`, `container_self.yml`: Singularity/Apptainer image builds + (on `v*` tags or manual dispatch). ## License diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..0643d9f --- /dev/null +++ b/docs/README.md @@ -0,0 +1,34 @@ +# Documentation index + +[Back to the main README](../README.md) + +## Getting started + +| Page | Read it to... | +|------|---------------| +| [Installation](installation.md) | Install `treecript` with Singularity/Apptainer, pip (venv) or Conda. | +| [Standalone collector binary](standalone-binary.md) | Run `execution-metrics-collector` without Python: download, build with Docker or Apptainer/Singularity, check HPC compatibility. | +| [Usage and CPU datasets](usage.md) | Collect metrics, plot charts, find a CPU TDP and estimate energy consumption. | + +## Reference + +| Page | Read it to... | +|------|---------------| +| [Output files and metrics](metrics.md) | Understand the files and columns generated by each collection run. | +| [RO-Crate representation](../RO-Crate-REPRESENTATION.md) | See how results are described in generated RO-Crates. | +| [Legacy scripts](legacy.md) | Look up the old Bash tools (unmaintained). | + +## Examples + +| Location | Content | +|----------|---------| +| [onboarding/](../onboarding/README.md) | Walk-through of a first installation and measurement, with generated charts. | +| [sample-work-to-measure/](../sample-work-to-measure/README.md) | Scripts that run sample workloads to measure. | +| [sample-series/](../sample-series/) | Example metrics of a real workflow execution. | +| [sample-charts/](../sample-charts/) | Example charts produced by `plotGraph`. | + +## Suggested reading order + +1. [Installation](installation.md) (or the [standalone binary](standalone-binary.md) if you only need to collect). +2. [Usage](usage.md) for the typical collect → plot → aggregate workflow. +3. [Metrics](metrics.md) to interpret the results. From e3edd40bab17c84afdf4d8d22c5fa11b4aefd7bc Mon Sep 17 00:00:00 2001 From: Androidbeingx <99746588+Androidbeingx@users.noreply.github.com> Date: Fri, 6 Nov 2026 12:00:47 +0000 Subject: [PATCH 4/4] [create-pull-request] Automatically updated constraints due security reasons --- installation/constraints-3.10.txt | 22 ++++++++++----------- installation/constraints-3.11.txt | 26 ++++++++++++------------- installation/constraints-3.12.txt | 32 +++++++++++++++---------------- installation/constraints-3.13.txt | 32 +++++++++++++++---------------- installation/constraints-3.14.txt | 32 +++++++++++++++---------------- installation/constraints-3.9.txt | 18 ++++++++--------- 6 files changed, 81 insertions(+), 81 deletions(-) diff --git a/installation/constraints-3.10.txt b/installation/constraints-3.10.txt index 704144e..b35a8ae 100644 --- a/installation/constraints-3.10.txt +++ b/installation/constraints-3.10.txt @@ -1,16 +1,16 @@ adjustText==1.4.0 arcp==0.2.1 -certifi==2026.7.22 -charset-normalizer==3.5.1 +certifi==2026.10.23 +charset-normalizer==3.5.2 click==8.5.0 contourpy==1.3.2 cycler==0.12.1 docker==7.2.0 -fonttools==4.64.0 -idna==3.19 +fonttools==4.65.0 +idna==3.20 Jinja2==3.1.6 kiwisolver==1.5.1 -MarkupSafe==3.0.3 +MarkupSafe==3.0.4 matplotlib==3.10.9 networkx==3.4.2 numpy==2.2.6 @@ -18,12 +18,12 @@ packaging==26.3 pandas==2.3.3 pillow==12.3.0 psutil==7.2.2 -pyparsing==3.3.2 -python-dateutil==2.9.0.post0 -pytz==2026.3.post1 +pyparsing==3.3.3 +python-dateutil==3.0.0 +pytz==2026.6 requests==2.34.2 -rocrate==0.15.1 +rocrate==0.16.0 scipy==1.15.3 six==1.17.0 -tzdata==2026.3 -urllib3==2.7.0 +tzdata==2026.6 +urllib3==2.8.0 diff --git a/installation/constraints-3.11.txt b/installation/constraints-3.11.txt index a789787..efe01e9 100644 --- a/installation/constraints-3.11.txt +++ b/installation/constraints-3.11.txt @@ -1,27 +1,27 @@ adjustText==1.4.0 arcp==0.2.1 -certifi==2026.7.22 -charset-normalizer==3.5.1 +certifi==2026.10.23 +charset-normalizer==3.5.2 click==8.5.0 contourpy==1.3.3 cycler==0.12.1 docker==7.2.0 -fonttools==4.64.0 -idna==3.19 +fonttools==4.68.0 +idna==3.20 Jinja2==3.1.6 kiwisolver==1.5.1 -MarkupSafe==3.0.3 -matplotlib==3.11.1 +MarkupSafe==3.0.4 +matplotlib==3.11.2 networkx==3.6.1 numpy==2.4.6 packaging==26.3 -pandas==3.0.5 -pillow==12.3.0 +pandas==3.1.1 +pillow==13.0.0 psutil==7.2.2 -pyparsing==3.3.2 -python-dateutil==2.9.0.post0 +pyparsing==3.3.3 +python-dateutil==4.0.0 requests==2.34.2 -rocrate==0.15.1 +rocrate==0.16.0 scipy==1.17.1 -six==1.17.0 -urllib3==2.7.0 +tzdata==2026.6 +urllib3==2.8.0 diff --git a/installation/constraints-3.12.txt b/installation/constraints-3.12.txt index 54576d5..8cac08f 100644 --- a/installation/constraints-3.12.txt +++ b/installation/constraints-3.12.txt @@ -1,28 +1,28 @@ adjustText==1.4.0 arcp==0.2.1 -certifi==2026.7.22 -charset-normalizer==3.5.1 +certifi==2026.10.23 +charset-normalizer==3.5.2 click==8.5.0 -contourpy==1.3.3 +contourpy==1.4.0 cycler==0.12.1 docker==7.2.0 -fonttools==4.64.0 -idna==3.19 +fonttools==4.68.0 +idna==3.20 Jinja2==3.1.6 kiwisolver==1.5.1 -MarkupSafe==3.0.3 -matplotlib==3.11.1 -networkx==3.6.1 -numpy==2.5.2 +MarkupSafe==3.0.4 +matplotlib==3.11.2 +networkx==3.7 +numpy==2.5.4 packaging==26.3 -pandas==3.0.5 -pillow==12.3.0 +pandas==3.1.1 +pillow==13.0.0 psutil==7.2.2 -pyparsing==3.3.2 -python-dateutil==2.9.0.post0 +pyparsing==3.3.3 +python-dateutil==4.0.0 requests==2.34.2 -rocrate==0.15.1 +rocrate==0.16.0 scipy==1.18.1 -six==1.17.0 -urllib3==2.7.0 +tzdata==2026.6 +urllib3==2.8.0 wheel==0.48.0 diff --git a/installation/constraints-3.13.txt b/installation/constraints-3.13.txt index 54576d5..8cac08f 100644 --- a/installation/constraints-3.13.txt +++ b/installation/constraints-3.13.txt @@ -1,28 +1,28 @@ adjustText==1.4.0 arcp==0.2.1 -certifi==2026.7.22 -charset-normalizer==3.5.1 +certifi==2026.10.23 +charset-normalizer==3.5.2 click==8.5.0 -contourpy==1.3.3 +contourpy==1.4.0 cycler==0.12.1 docker==7.2.0 -fonttools==4.64.0 -idna==3.19 +fonttools==4.68.0 +idna==3.20 Jinja2==3.1.6 kiwisolver==1.5.1 -MarkupSafe==3.0.3 -matplotlib==3.11.1 -networkx==3.6.1 -numpy==2.5.2 +MarkupSafe==3.0.4 +matplotlib==3.11.2 +networkx==3.7 +numpy==2.5.4 packaging==26.3 -pandas==3.0.5 -pillow==12.3.0 +pandas==3.1.1 +pillow==13.0.0 psutil==7.2.2 -pyparsing==3.3.2 -python-dateutil==2.9.0.post0 +pyparsing==3.3.3 +python-dateutil==4.0.0 requests==2.34.2 -rocrate==0.15.1 +rocrate==0.16.0 scipy==1.18.1 -six==1.17.0 -urllib3==2.7.0 +tzdata==2026.6 +urllib3==2.8.0 wheel==0.48.0 diff --git a/installation/constraints-3.14.txt b/installation/constraints-3.14.txt index 54576d5..8cac08f 100644 --- a/installation/constraints-3.14.txt +++ b/installation/constraints-3.14.txt @@ -1,28 +1,28 @@ adjustText==1.4.0 arcp==0.2.1 -certifi==2026.7.22 -charset-normalizer==3.5.1 +certifi==2026.10.23 +charset-normalizer==3.5.2 click==8.5.0 -contourpy==1.3.3 +contourpy==1.4.0 cycler==0.12.1 docker==7.2.0 -fonttools==4.64.0 -idna==3.19 +fonttools==4.68.0 +idna==3.20 Jinja2==3.1.6 kiwisolver==1.5.1 -MarkupSafe==3.0.3 -matplotlib==3.11.1 -networkx==3.6.1 -numpy==2.5.2 +MarkupSafe==3.0.4 +matplotlib==3.11.2 +networkx==3.7 +numpy==2.5.4 packaging==26.3 -pandas==3.0.5 -pillow==12.3.0 +pandas==3.1.1 +pillow==13.0.0 psutil==7.2.2 -pyparsing==3.3.2 -python-dateutil==2.9.0.post0 +pyparsing==3.3.3 +python-dateutil==4.0.0 requests==2.34.2 -rocrate==0.15.1 +rocrate==0.16.0 scipy==1.18.1 -six==1.17.0 -urllib3==2.7.0 +tzdata==2026.6 +urllib3==2.8.0 wheel==0.48.0 diff --git a/installation/constraints-3.9.txt b/installation/constraints-3.9.txt index 92a185c..a85fc69 100644 --- a/installation/constraints-3.9.txt +++ b/installation/constraints-3.9.txt @@ -1,17 +1,17 @@ adjustText==1.4.0 arcp==0.2.1 -certifi==2026.7.22 -charset-normalizer==3.5.1 +certifi==2026.10.23 +charset-normalizer==3.5.2 click==8.1.8 contourpy==1.3.0 cycler==0.12.1 docker==7.2.0 fonttools==4.60.2 -idna==3.19 +idna==3.20 importlib_resources==6.5.2 Jinja2==3.1.6 kiwisolver==1.4.7 -MarkupSafe==3.0.3 +MarkupSafe==3.0.4 matplotlib==3.9.4 networkx==3.2.1 numpy==2.0.2 @@ -19,13 +19,13 @@ packaging==26.3 pandas==2.3.3 pillow==11.3.0 psutil==7.2.2 -pyparsing==3.3.2 -python-dateutil==2.9.0.post0 -pytz==2026.3.post1 +pyparsing==3.3.3 +python-dateutil==3.0.0 +pytz==2026.6 requests==2.32.5 -rocrate==0.15.1 +rocrate==0.16.0 scipy==1.13.1 six==1.17.0 -tzdata==2026.3 +tzdata==2026.6 urllib3==2.6.3 zipp==3.23.1