[dimitri-yatsenko]

Closes #1582.

Why

On PostgreSQL, database.name selects the database, so projects sharing a server keep their schemas apart while pipelines write plain names like dj.Schema("subject"). On MySQL the setting was ignored with a warning, so every pipeline had to build prefixed schema names itself. The deprecated database_prefix stores a value DataJoint never applies, and its message still says it is removed in 2.3.

This PR gives database.name the same role on MySQL: it becomes a prefix that Schema adds to schema names. A pipeline is then written the same way for both backends.

Changes

• Connection.schema_prefix: name + "_" on MySQL when database.name is set; otherwise "", including always on PostgreSQL. On MySQL the name may not contain _, so one namespace's prefix can never begin another's (lab_ begins lab_proj_); such a name raises a DataJointError.
• Connection.qualify_schema_name(name): returns the name on the server. A name that already starts with the prefix is returned unchanged with a UserWarning ("write subject instead"), so code that still prefixes by hand keeps reaching the same schemas.
• Schema.activate: qualifies the name before use, which covers dj.Schema(...), VirtualModule and virtual_schema. schema.database is the name on the server.
• dj.list_schemas(): with a prefix, lists only the namespace's schemas, under their plain names.
• Warnings: the MySQL warning "database.name is set but the MySQL backend does not support database selection" is removed. The database_prefix deprecation message points to database.name and no longer mentions 2.3.
• Settings: the descriptions of database.name and database_prefix are updated.

Compatibility

A MySQL user who sets database.name today gets a warning and no effect. After this change, their schema names gain the prefix. Hence the v2.4 milestone, and a line in the release notes.

PostgreSQL behaviour is unchanged.

Open questions (from #1582)

• the separator _: decided: kept, with _ refused inside a MySQL database.name, so listing a namespace is exact;
• whether to warn or raise on an already-prefixed name: decided: warn and use it as is, so hand-prefixed pipelines keep reaching their schemas. As a consequence, under lab, "lab_notes" and "notes" name the same schema; the docstring says so;
• lineage strings embedding the prefix on MySQL;
• an opt-out for schemas outside the namespace;
• when to remove database_prefix.

The PR implements the proposal's defaults, and each one is easy to change.

Tests

• Unit, tests/unit/test_schema_prefix.py: the prefix rule on both backends, without a server.
• Integration, tests/integration/test_schema_namespace.py: runs on MySQL and PostgreSQL.
  • Two namespaces each hold a subject schema with separate contents.
  • list_schemas returns plain names, while a connection without a namespace sees the prefixed ones on MySQL.
  • VirtualModule resolves inside the namespace.
  • An already-prefixed name is not prefixed twice.
  • Setting database.name on MySQL no longer warns.

Local results: unit 457 passed; integration 694 passed, 16 skipped (testcontainers, both backends).

Docs: datajoint/datajoint-docs#301 (configure-database.md, deploy-production.md, reference/configuration.md), to land with the release.

[lum-agilyti]

A few things I noticed while reading, in case they're useful for the open questions:

1. With database.name = "lab", the prefix is "lab_". A second namespace "lab_proj" has schemas like lab_proj_subject, which also start with "lab_", so from "lab", list_schemas() seems to return "proj_subject" and dj.Schema("proj_subject") would open the other namespace's subject.

2. With database.name = "lab", a pipeline schema that is actually named "lab_notes" looks like it would be used as is (with the warning), so it would be the same schema as dj.Schema("notes").

3. I don't see any CI runs on this branch (no check runs on 688ac4d), so the new tests may not have run in CI yet.

[dimitri-yatsenko]

Thanks, @lum-agilyti. Both cases were real. bd3910e addresses them:

1. One namespace's prefix beginning another's. On MySQL, database.name may no longer contain _. With the prefix name + "_", a name without underscores can never begin another namespace's prefix, so list_schemas() is exact. database.name = "lab_proj" now raises a DataJointError that explains why. PostgreSQL is unchanged, since there the name selects a database.
2. "lab_notes" under lab. This stays as it is, on purpose. A name that already starts with the prefix is used unchanged, with a warning, so pipelines that already add the prefix themselves keep reaching their existing schemas when they set database.name. The cost is the aliasing you describe: "lab_notes" and "notes" name the same schema. The docstring now says so, and the PR description records both decisions.
3. CI. The workflow runs from early to mid October were deleted across the repository. This push runs the checks again.