Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision		 Previous revision
en:entwickler:regression_test_suite [2026/08/24 13:40] – created kainhoferen:entwickler:regression_test_suite [2026/08/24 18:12] (current) – [Continuous integration] kainhofer
Line 1: Line 1:
 ====== Regression test suite ====== ====== Regression test suite ======
  
-The Admidio regression test suite is intended to protect the behavior of Admidio Core against regressions during development.+The Admidio regression test suite protects the behaviour of Admidio Core against regressions during development. It is useful both for developers contributing to Admidio itself and for third-party developers who build modules, plugins or integrations against Admidio.
  
-Once set up, running regression tests after changes to the Admidio codebase is as simple as running ''composer test:all'' in the Admidion installation directory:+The suite does not only test isolated PHP classes. Depending on the layer it exercises the real Admidio database abstraction, Entities, Services, the production installer, the command-line interface, filesystem handling and mail delivery. 
 + 
 +Once the environment is set up, running the tests after a change is a single command ''composer test:all'' in the Admidio installation directory:
  
 <code bash> <code bash>
Line 34: Line 36:
 </code> </code>
  
-It is useful both for developers contributing to Admidio itself and for third-party developers who build modules, plugins, integrations or other extensions against Admidio.+The test count in this example output is only illustrative, it grows with the suite.
  
-The suite does not only test isolated PHP classes. Depending on the test layer it also exercises the real Admidio database abstraction, Entities, Services, the production installer, command-line interface, filesystem handling and mail delivery.+**The suite is intentionally destructive. Never point it at a production database or at a development database whose content you want to keep.** Several guards make this hard to do by accident, see [[#safety_mechanisms|Safety mechanisms]]. Do not disable them.
  
-This page describes how to set up, run and extend the regression test suite.+The suite is not a browser UI test suite. Many tests deliberately stop at the Entity, Service or CLI acceptance boundary. Historic version-to-version database upgrade coverage is a separate lifecycle concern: a green regression run only covers upgrade paths if the current branch actually contains upgrade tests.
  
 For the general setup of an Admidio development installation also see [[en:entwickler:testumgebung_einrichten|Set up a test environment]]. For the general setup of an Admidio development installation also see [[en:entwickler:testumgebung_einrichten|Set up a test environment]].
  
-===== Important safety warning =====+===== Quick start =====
  
-**Never run the database-backed regression tests against a production database or against a development database containing data you want to keep.**+==== First run ====
  
-The regression environment is intentionally destructive.+From the root of your Admidio checkout:
  
-Database-backed test runs recreate the Admidio schema using the production installer. Existing Admidio tables in the configured test database may therefore be removed.+<code bash> 
 +# 1. install the development dependencies (PHPUnit, symfony/process) 
 +composer install
  
-The test harness contains additional safeguards. Among other checks, the configured database name must contain ''test'' as a separate token.+# 2. start the test services (MariaDB, PostgreSQL, Mailpit) 
 +docker compose -f docker-compose.test.yml up -d
  
-Use a dedicated database name such as:+# 3. create the test configuration 
 +cp .env.test.example .env.test
  
-<code> +# 4. prepare and verify the environment 
-admidio_test +php tests/bin/setup-test-env.php
-</code>+
  
-Do not use a shared database and do not try to bypass the safety checks. +# 5. run everything 
- +composer test:all
-Filesystem tests have a similar safeguard. They are only allowed to perform destructive operations below: +
- +
-<code> +
-tests/adm_my_files+
 </code> </code>
  
-and require the regression-test marker file contained in that directory.+Step 2 takes about 30 seconds until the databases accept connections. The setup script in step 4 waits for them, so you do not have to.
  
-===== Test architecture =====+On Windows use ''copy .env.test.example .env.test'' or ''Copy-Item'' instead of ''cp''.
  
-The suite is divided into several layers. A test should be placed in the lowest layer that can reliably test the behavior in question.+The defaults in ''.env.test.example'' match the services in ''docker-compose.test.yml'', so on the recommended setup no value has to be changed.
  
-^ Layer ^ Typical location ^ Purpose ^ +==== Everyday commands ====
-| Unit tests | ''tests/Unit/'' | Test pure production logic without database, filesystem or network access. | +
-| Integration tests | ''tests/Integration/'' | Exercise real Admidio Entities, Services, permissions and database behavior. | +
-| CLI tests | ''tests/Cli/'' | Validate CLI contracts and complete workflows through the real Admidio command-line entry point. | +
-| Filesystem integration tests | ''tests/Integration/Filesystem/'' | Exercise production document, photo, import/export and other filesystem code using the protected test data directory. | +
-| Mail integration tests | ''tests/Integration/Mail/'' | Exercise the real Admidio mail path against a local SMTP sink such as Mailpit. | +
-| Installation tests | ''tests/Cli/'' and support bootstrap | Verify that the current production installer creates a usable Admidio database. |+
  
-The suite should not be interpreted as a browser UI test suite. Many tests deliberately stop at the Entity, Service or CLI acceptance boundary.+^ Command ^ What it runs ^ Database needed ^ 
 +| ''composer test:unit'' | The ''Unit Tests'' suite, ''tests/Unit'' | no | 
 +| ''composer test:integration'' | The ''Integration Tests'' suite, ''tests/Integration'' | yes | 
 +| ''composer test:cli'' | The ''CLI Tests'' suite, ''tests/Cli'' | yes | 
 +| ''composer test'' | Unit and Integration tests | yes | 
 +| ''composer test:all'' | Everything ''phpunit.xml'' defines, all three suites | yes | 
 +| ''composer test:coverage'' | Full run with an HTML report in ''tests/reports/coverage'' | yes | 
 +| ''composer test:setup'' | ''php tests/bin/setup-test-env.php'' | yes |
  
-Historic version-to-version database upgrade coverage is a separate lifecycle concern. A green regression run should only be interpreted as covering upgrade paths when corresponding upgrade tests exist in the current branch.+While developing, run the test you are working on directly:
  
-===== Core testing principle: test Admidio, not the test suite =====+<code bash> 
 +# one file 
 +vendor/bin/phpunit tests/Integration/Inventory/InventoryTest.php
  
-The most important rule when adding regression tests is:+# one method, in the whole suite or in one file 
 +vendor/bin/phpunit --filter testInventoryFieldAndItemLifecycleUsesProductionServices 
 +vendor/bin/phpunit --filter testUploadDownloadAndDelete tests/Integration/Filesystem/DocumentsFilesystemServiceTest.php
  
-**The action being tested must be performed by production Admidio code.**+# one suite with readable descriptions 
 +vendor/bin/phpunit --testsuite="Integration Tests" --testdox 
 +</code>
  
-A test must not reproduce the expected Admidio behavior inside a fixture, helper or mock and then verify the behavior it implemented itself.+''composer test:unit'' is the fast feedback loop: it needs neither database nor Docker, because the Admidio bootstrap is only loaded once a database-backed test asks for it.
  
-For example, a test for a Service operation should normally:+''composer test:coverage'' additionally needs Xdebug or PCOV, otherwise PHPUnit reports that no code coverage driver is available.
  
-  - create only the prerequisites needed by the test; +After a focused test passes, run ''composer test:all'' before you consider the change complete.
-  - call the real Admidio Service; +
-  - let that Service call the normal Admidio Entities and database abstraction; +
-  - verify the resulting state independently, for example through a new Entity instance or a direct prepared database query.+
  
-A test should **not** implement the same database writes itself and then claim that the Service was tested.+==== What a run changes on your machine ====
  
-This distinction is especially important for operations that perform more than one action, for example:+  * **The test database is rebuilt.** The first database-backed test in a PHPUnit process drops every table carrying the Admidio table prefix and reinstalls the schema with the current production installer. Tables without that prefix are left alone, but the database must still be a disposable one. 
 +  * **Files are written below ''tests/adm_my_files''.** For the duration of a run Admidio's ''FOLDER_DATA'' points there, so nothing lands in the ''adm_my_files'' of your checkout. 
 +  * **Mail does not leave your machine.** The mail test talks to the local Mailpit SMTP listener. 
 +  * **Nothing else is touched.** Each database test runs inside a transaction that is rolled back afterwards. The exception are CLI subprocess tests, which commit and therefore clean up after themselves, see [[#cli_tests|CLI tests]].
  
-  - creating reciprocal user relations; +===== Setting up the test environment =====
-  - creating or updating related records; +
-  - enforcing organization boundaries; +
-  - updating changelog information; +
-  - maintaining sequence values; +
-  - applying permissions; +
-  - sending messages or emails; +
-  - creating thumbnails or archive files.+
  
-Fixtures are there to create prerequisites. They are not substitutes for the production workflow being tested.+==== Requirements ====
  
-===== Requirements =====+  * a checkout of the Admidio source tree; 
 +  * PHP as required by ''composer.json'' (currently ''^8.2'') with the extensions ''curl'', ''gd'', ''iconv'', ''json'', ''pdo'', ''simplexml'' and ''zip''; 
 +  * Composer, to install PHPUnit and ''symfony/process''; 
 +  * a dedicated MariaDB, MySQL or PostgreSQL test database and the matching PDO driver (''pdo_mysql'' or ''pdo_pgsql''); 
 +  * Mailpit or another local SMTP sink for the mail test; 
 +  * Docker, if you want the supplied test services instead of your own servers.
  
-Before running the suite, install the normal development dependencies from the Admidio repository.+''ext-gd'' and ''ext-zip'' are not optional for the suite, the photo and import/export regressions use them.
  
-The exact PHP version and required PHP extensions are defined by the current ''composer.json''. +Install the dependencies from the Admidio root directory:
- +
-You need: +
- +
-  - a checkout of the Admidio source tree; +
-  - Composer; +
-  - the PHP extensions required by Admidio; +
-  - a dedicated MariaDB, PostgreSQL or MySQL test database; +
-  - the matching PDO database driver; +
-  - Mailpit when running the mail integration test; +
-  - GD and ZIP support for the photo/filesystem tests. +
- +
-Install the Composer dependencies from the Admidio root directory:+
  
 <code bash> <code bash>
Line 136: Line 130:
 </code> </code>
  
-===== Configure the test environment =====+==== Recommended: the supplied Docker environment ====
  
-Copy the supplied example configuration:+The repository contains ''docker-compose.test.yml'' in the root directory. It provides everything the suite needs: 
 + 
 +^ Service ^ Image ^ Published ports ^ Database ^ User ^ Password ^ 
 +| ''mariadb'' | ''mariadb:10.11'' | 3306 | ''admidio_test'' | ''admidio'' | ''admidio_test'' | 
 +| ''postgres'' | ''postgres:15'' | 5432 | ''admidio_test'' | ''admidio'' | ''admidio_test'' | 
 +| ''mailpit'' | ''axllent/mailpit'' | 1025 (SMTP), 8025 (HTTP API and web UI) | — | — | — |
  
 <code bash> <code bash>
-cp .env.test.example .env.test +# start 
-</code>+docker compose -f docker-compose.test.yml up -d
  
-On Windows, copy the file using Explorer or PowerShell instead.+# status and logs 
 +docker compose -f docker-compose.test.yml ps 
 +docker compose -f docker-compose.test.yml logs -f mariadb
  
-The process environment takes precedence over values stored in ''.env.test'', which is useful in CI environments.+# stop, keeping the data volumes 
 +docker compose -f docker-compose.test.yml down
  
-A typical configuration looks like this:+# stop and throw the databases away 
 +docker compose -f docker-compose.test.yml down -v 
 +</code>
  
-<code ini> +The ''-f docker-compose.test.yml'' is not optional. There is no ''docker-compose.yml'' in the repository root, so a plain ''docker compose up -d'' does not start the test services.
-TEST_DATABASE_ENGINE=mariadb +
-TEST_FILES_PATH=./tests/adm_my_files+
  
-TEST_DB_MARIADB_HOST=127.0.0.1 +The Mailpit web UI is at [[http://localhost:8025|http://localhost:8025]] and is the quickest way to look at what a mail test actually delivered.
-TEST_DB_MARIADB_PORT=3306 +
-TEST_DB_MARIADB_USER=admidio +
-TEST_DB_MARIADB_PASS=admidio_test +
-TEST_DB_MARIADB_NAME=admidio_test+
  
-TEST_DB_POSTGRES_HOST=127.0.0.1 +The Compose file has no MySQL service, because MariaDB already occupies port 3306. MySQL 8 is a scheduled CI target. To test it locally, point the ''TEST_DB_MYSQL_*'' variables at your own instance and make sure it does not collide with the MariaDB container.
-TEST_DB_POSTGRES_PORT=5432 +
-TEST_DB_POSTGRES_USER=admidio +
-TEST_DB_POSTGRES_PASS=admidio_test +
-TEST_DB_POSTGRES_NAME=admidio_test+
  
-TEST_DB_MYSQL_HOST=127.0.0.1 +==== Alternative: your own database server ====
-TEST_DB_MYSQL_PORT=3306 +
-TEST_DB_MYSQL_USER=admidio +
-TEST_DB_MYSQL_PASS=admidio_test +
-TEST_DB_MYSQL_NAME=admidio_test+
  
-TEST_MAIL_HOST=127.0.0.1 +If you already run a database server, you only need a disposable database and a user that may create and drop tables in it.
-TEST_MAIL_PORT=1025+
  
-TEST_MAILPIT_API_HOST=127.0.0.1 +=== MariaDB and MySQL === 
-TEST_MAILPIT_API_PORT=8025+ 
 +<code sql> 
 +CREATE DATABASE admidio_test CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; 
 +CREATE USER 'admidio'@'%' IDENTIFIED BY 'admidio_test'; 
 +GRANT ALL PRIVILEGES ON admidio_test.* TO 'admidio'@'%'; 
 +FLUSH PRIVILEGES;
 </code> </code>
  
-The checked-in ''.env.test.example'' is the authoritative reference for the variables supported by the current branch.+The user needs ''ALL PRIVILEGES'' on that one database because the suite runs the production installer, which creates the whole schema and drops it again at the start of the next run. Grant nothing outside this database.
  
-Only configure credentials for disposable test databases.+=== PostgreSQL ===
  
-===== Using the supplied Docker environment =====+<code sql> 
 +CREATE ROLE admidio LOGIN PASSWORD 'admidio_test'; 
 +CREATE DATABASE admidio_test OWNER admidio ENCODING 'UTF8'; 
 +\c admidio_test 
 +GRANT ALL ON SCHEMA public TO admidio; 
 +</code>
  
-The easiest way to provide the database servers and Mailpit is the Docker Compose test environment supplied with the repository.+The last statement matters from PostgreSQL 15 on, where the ''public'' schema no longer lets everybody create objects. The suite reads and drops the Admidio tables in the ''public'' schema of the configured database.
  
-From the directory containing the Compose configuration, start the test services:+=== A local SMTP sink === 
 + 
 +Without Docker, run Mailpit directly. It is a single binary, or:
  
 <code bash> <code bash>
-docker compose up -d+docker run -d --name admidio-test-mailpit -p 1025:1025 -p 8025:8025 axllent/mailpit
 </code> </code>
  
-The current regression environment provides database services for the normal test matrix and Mailpit for SMTP testing.+The mail test needs both the SMTP port and the HTTP API, because it verifies the delivered message through the API rather than trusting that the connection succeeded.
  
-If you use your own database servers instead, simply adjust ''.env.test'' accordingly.+Whatever you use, adjust ''.env.test'' to match, and keep the database name a throwaway one whose name satisfies the safety check described below.
  
-MySQL can also be tested against an external MySQL instance when one is not part of the local Compose configuration.+==== The test configuration in .env.test ====
  
-==== Mailpit ====+Copy the supplied example and edit it if your services differ from the defaults:
  
-The default regression configuration expects:+<code bash> 
 +cp .env.test.example .env.test 
 +</code>
  
-^ Service ^ Default endpoint ^ +''.env.test'' is not in version control. The checked-in ''.env.test.example'' is the authoritative reference for the variables the current branch supports.
-| SMTP | ''127.0.0.1:1025'' | +
-| HTTP API | ''127.0.0.1:8025'' |+
  
-The Mailpit regression test deliberately does **not** depend on Docker's health status.+^ Variable ^ Meaning ^ Default ^ 
 +| ''TEST_DATABASE_ENGINE'' | Engine the run uses: ''mariadb'', ''mysql'' or ''postgres'' | required | 
 +| ''TEST_FILES_PATH'' | Data directory of the test run, becomes Admidio's ''FOLDER_DATA'' | required, normally ''./tests/adm_my_files'' | 
 +| ''TEST_DB_<ENGINE>_HOST'' | Database host | ''127.0.0.1'' | 
 +| ''TEST_DB_<ENGINE>_PORT'' | Database port | ''3306'', for PostgreSQL ''5432'' | 
 +| ''TEST_DB_<ENGINE>_USER'' | Database user | ''admidio'' | 
 +| ''TEST_DB_<ENGINE>_PASS'' | Password of that user | empty | 
 +| ''TEST_DB_<ENGINE>_NAME'' | Name of the test database | ''admidio_test'' | 
 +| ''TEST_MAIL_HOST'', ''TEST_MAIL_PORT'' | SMTP listener of the mail sink | ''127.0.0.1'', ''1025'' | 
 +| ''TEST_MAILPIT_API_HOST'', ''TEST_MAILPIT_API_PORT'' | HTTP API of Mailpit | ''127.0.0.1'', ''8025'' |
  
-Some Docker environments may show the Mailpit container as ''unhealthy'' even though Mailpit itself is working correctly. The test checks what actually matters:+''<ENGINE>'' is the uppercase engine name, so the MariaDB host is ''TEST_DB_MARIADB_HOST''. One block exists per engine, which is why switching engines is a matter of one variable.
  
-  - whether Admidio can send the message through Mailpit's SMTP listener; +Only ''TEST_DATABASE_ENGINE'' and ''TEST_FILES_PATH'' are strictly required, everything else falls back to the defaults above.
-  - whether the delivered message appears through the Mailpit HTTP API.+
  
-Therefore a Docker ''unhealthy'' label alone is not a reason to disable the mail test.+Two details that regularly cost time:
  
-===== Running the tests =====+  * **Use ''127.0.0.1'', not ''localhost''.** A MySQL client reads ''localhost'' as "connect through the unix socket", which does not reach a database that Docker published on a port. 
 +  * **A variable that is already set in the process environment wins over the file.** That is how CI configures a run without any ''.env.test'', and how you override a single value for one shell:
  
-==== Complete regression suite ====+<code bash> 
 +TEST_DATABASE_ENGINE=postgres composer test:integration 
 +</code>
  
-Run the complete suite from the Admidio root directory:+<code powershell> 
 +$env:TEST_DATABASE_ENGINE = 'postgres'; composer test:integration 
 +</code> 
 + 
 +==== The setup script ====
  
 <code bash> <code bash>
-composer test:all+php tests/bin/setup-test-env.php
 </code> </code>
  
-This is the normal command before submitting a change that can affect several parts of Admidio.+Run it once after creating ''.env.test'', and again whenever you change the engine or recreate the containers. It:
  
-==== Unit tests only ====+  - loads ''.env.test'' and the process environment and fails if the run is not configured; 
 +  - creates the subdirectories of the test data directory (''documents'', ''documents_test'', ''photos'', ''temp'', ''logs'', ''import'', ''export''); 
 +  - connects to the database, retrying for up to a minute, because a container answers on its port before the server accepts connections; 
 +  - reports whether the mail sink is reachable, which is optional and does not stop the setup; 
 +  - writes the ''.test-environment-marker'' file.
  
-For fast feedback while working on pure PHP logic:+The script accepts ''--db=mariadb|postgres|mysql'' to check a different engine than the configured one:
  
 <code bash> <code bash>
-composer test:unit+php tests/bin/setup-test-env.php --db=postgres
 </code> </code>
  
-Unit tests deliberately do not initialize the regression database or filesystem environment.+**This option only affects the script.** The test run itself always reads ''TEST_DATABASE_ENGINE''. The engine is not a command-line option of PHPUnit.
  
-==== Run one test file ====+==== Switching the database engine ====
  
-During development it is often useful to run only the test currently being worked on:+To run the suite against another engine, change the engine in ''.env.test'' (or in the environment), then run the setup script and the tests again:
  
 <code bash> <code bash>
-vendor/bin/phpunit tests/Integration/Inventory/InventoryTest.php+# in .env.test:  TEST_DATABASE_ENGINE=postgres 
 +php tests/bin/setup-test-env.php 
 +composer test:all
 </code> </code>
  
-or, for example:+Each engine has its own connection block, so both configurations can stay in the file side by side. The schema is rebuilt at the start of every run, so no manual cleanup is needed when you switch back and forth.
  
-<code bash> +Before submitting a change that touches SQL, Entities or the installer, run the suite at least against MariaDB and PostgreSQL. Those are the two engines every pull request is tested on.
-vendor/bin/phpunit tests/Integration/Filesystem/DocumentsFilesystemServiceTest.php +
-</code>+
  
-==== Run one test method ====+==== Safety mechanisms ====
  
-PHPUnit filtering can be used for an individual regression:+The suite refuses to run in an environment that does not look like a test environment. The guards are:
  
-<code bash> +^ Guard ^ Enforced in ^ Rule ^ 
-vendor/bin/phpunit --filter testName +| Database name | ''tests/bootstrap.php'' | The name must contain ''test'' as a separate token, delimited by start, end, ''_'' or ''-''. ''admidio_test'', ''test'' and ''test-db'' pass, ''admidiotest'' and ''latest'' do not. | 
-</code>+| Data directory | ''tests/env.php'' | ''TEST_FILES_PATH'' must contain ''test'' and must resolve to a directory //inside// the checkout, because Admidio addresses everything it writes as ''ADMIDIO_PATH . FOLDER_DATA''. | 
 +| Table prefix | ''TestDatabaseInitializer'' | Only tables starting with the configured Admidio table prefix are dropped, so unrelated tables in the same database survive. | 
 +| Filesystem root | ''FilesystemTestCase'' | ''FOLDER_DATA'' has to resolve to ''tests/adm_my_files'' and the committed marker ''tests/adm_my_files/.admidio-regression-test'' has to exist, otherwise destructive operations are refused. Every path that is created or cleaned up is checked against that root again. |
  
-Replace ''testName'' with the actual method name.+There are two marker files in ''tests/adm_my_files'' and they are not interchangeable:
  
-After the focused test passes, run ''composer test:all'' before considering the change complete.+  * ''.admidio-regression-test'' is committed to git and is the guard the filesystem tests check. Do not delete it. 
 +  * ''.test-environment-marker'' is created automatically by the setup script and the bootstrap. It is informational.
  
-===== Database initialization and isolation =====+Never point ''TEST_FILES_PATH'' at the ''adm_my_files'' of a real installation, never use a shared database, and do not work around a guard that fires. It is telling you that the environment is not the one you think it is.
  
-Database-backed PHPUnit runs use the current Admidio production installer to create the schema.+==== Continuous integration ====
  
-This is intentional: an old database dump must not allow an installation or schema regression to remain unnoticed.+''.github/workflows/regression-tests.yml'' runs the suite on every pull request against ''master'', on pushes to that branch and on a schedule. CI configures everything through workflow environment variables and therefore needs no ''.env.test''.
  
-Normal database integration tests run inside a transaction. The base test case rolls the transaction back after the test so that tests remain isolated from one another.+^ Job ^ When ^ What it does ^ 
 +| ''fast-checks'' | always | ''php -l'' over ''src'', ''system'', ''modules'', ''install'', ''tests'', then ''composer validate --strict'' and ''composer test:unit'' | 
 +| MariaDB 10.11 | always, after fast-checks | setup script, ''composer test:integration'', ''composer test:cli'' | 
 +| PostgreSQL 15 | always, after fast-checks | setup script, ''composer test:integration'', ''composer test:cli'' | 
 +| MySQL 8.0 | Mondays 02:00 UTC, or on demand | same as the other database jobs |
  
-Tests should therefore not depend on execution order.+The workflow can also be started manually through //workflow_dispatch// with a choice of engine, which is the quickest way to get a MySQL result for one branch.
  
-Do not assume that data created by another test still exists.+CI runs no mail job. The Mailpit test is a local integration test.
  
-===== A special case: CLI subprocess tests =====+===== How the suite is built =====
  
-CLI subprocess tests are different from ordinary transaction-based integration tests.+==== Test layers ====
  
-They start the real Admidio executable as a separate process. That process has:+The suite is divided into layers. A test belongs in the lowest layer that can reliably test the behaviour in question.
  
-  - its own production bootstrap; +^ Layer ^ Location ^ Purpose ^ 
-  - its own database connection; +| Unit tests | ''tests/Unit/'' | Pure production logic without database, filesystem or network access. | 
-  - its own exit status; +| Integration tests | ''tests/Integration/'' | Real Admidio Entities, Services, permissions and database behaviour. | 
-  - its own standard output and standard error.+| Filesystem integration tests | ''tests/Integration/Filesystem/'' | Production document, photo and import/export code against the protected test data directory. | 
 +| Mail integration tests | ''tests/Integration/Mail/'' | The real Admidio mail path against a local SMTP sink. | 
 +| CLI tests | ''tests/Cli/'' | CLI contracts and complete workflows through the real Admidio command-line entry point. | 
 +| Installation coverage | ''TestDatabaseInitializer'' and ''tests/Cli/'' | The current production installer has to produce a usable Admidio database. |
  
-This makes subprocess tests an important acceptance boundary.+The ''tests/Integration'' directory is organised by subject area — ''Announcements'', ''Cascading'', ''Categories'', ''Changelog'', ''Core'', ''Database'', ''Documents'', ''Events'', ''Exchange'', ''Filesystem'', ''Inventory'', ''Lists'', ''Mail'', ''Menu'', ''Messages'', ''Organizations'', ''Permissions'', ''Photos'', ''ProfileFields'', ''Roles'', ''Security'', ''Services'', ''Users'', ''Weblinks'', ''Workflows''. Add a new directory when a new area gets its first test.
  
-A typical mutating CLI regression should follow this pattern:+==== Directory layout ====
  
-  - process A creates or changes an object; +<code> 
-  - process B reads the object and proves that the change was committed; +tests/ 
-  - another command removes the object again; +├── bootstrap.php               PHPUnit bootstrap: autoloader, session, safety checks 
-  - the test verifies that cleanup succeeded.+├── env.php                     reads .env.test and the process environment 
 +├── bootstrap-admidio.php       full Admidio bootstrap, loaded lazily by DatabaseTestCase 
 +├── bin/ 
 +│   └── setup-test-env.php      one-time preparation and verification of the environment 
 +├── Support/                    base classes, traits and fixtures - contains no tests 
 +│   ├── AdmidioTestCase.php         root base class, generic assertions 
 +│   ├── DatabaseTestCase.php        database connection, schema, transaction isolation 
 +│   ├── AdministratorTestCase.php   runs a test as the real installed administrator 
 +│   ├── FilesystemTestCase.php      guarded filesystem access and cleanup 
 +│   ├── CliSubprocess.php           trait, starts ./admidio as a real process 
 +│   ├── PermissionContext.php       trait, switches organization and current user 
 +│   ├── AdmidioTestFixture.php      creates prerequisite orgs, users, roles, categories 
 +│   └── TestDatabaseInitializer.php runs the production installer against the test database 
 +├── Unit/                       no database, no filesystem, no network 
 +├── Integration/                one directory per subject area 
 +├── Cli/                        CLI contract and workflow tests 
 +└── adm_my_files/               FOLDER_DATA of a test run, git-ignored except the marker 
 +</code>
  
-Do not verify a mutating CLI command only through data held in the test process. A second process or an independent database read should prove persistence.+The three bootstraps are separate on purpose. ''tests/bootstrap.php'' is what ''phpunit.xml'' loads and stays free of Admidio and of the database, so ''composer test:unit'' remains a true unit-test boundary. ''DatabaseTestCase'' pulls in ''tests/bootstrap-admidio.php'' the first time a test really needs Admidio.
  
-Also remember that a CLI subprocess cannot see uncommitted data from the PHPUnit transaction of its parent process.+==== Base classes and helpers ====
  
-Therefore prerequisites for a mutating CLI scenario must either already exist in the committed baseline or be created through subprocess commands as part of the scenario.+The base classes build on each other:
  
-Mutating subprocess tests must clean up after themselves, normally in a ''finally'' block, because their changes are committed and cannot be removed by PHPUnit transaction rollback.+<code> 
 +PHPUnit\Framework\TestCase 
 +    └── AdmidioTestCase          generic assertions, no external state 
 +            └── DatabaseTestCase         database, schema, transaction per test 
 +                    └── AdministratorTestCase    the real administrator as current user 
 +                            └── FilesystemTestCase       guarded test data directory 
 +</code>
  
-The existing ''tests/Support/CliSubprocess.php'' and CLI process tests should be used as the pattern instead of implementing another subprocess launcher.+^ Base class ^ Gives you ^ Choose it when ^ 
 +| ''AdmidioTestCase'' | ''assertValidUuid()'', ''assertValidTimestamp()'', ''assertArrayHasKeys()'' | the code under test needs no database, filesystem or network | 
 +| ''DatabaseTestCase'' | the Admidio bootstrap, an installed schema, ''getDatabase()'', an automatic transaction rollback after each test | the test touches Entities, Services or SQL | 
 +| ''AdministratorTestCase'' | additionally ''$gCurrentUser'' set to the administrator the production installer created, and ''$gValidLogin'' | the production path under test requires administrator rights | 
 +| ''FilesystemTestCase'' | additionally ''getTestDataRoot()'', ''createIsolatedDirectory()'', ''createFixtureFile()'', ''registerCleanupPath()'' and automatic, verified cleanup | production code writes or deletes files |
  
-===== Filesystem tests =====+''AdministratorTestCase'' does not mock anything and injects no rights. It looks up the real administrator membership created by the installer and loads that user through the normal ''User'' Entity, then restores the previous globals in ''tearDown()''.
  
-Tests that exercise documents, photos, imports, exports or other file operations must use the protected filesystem test base class.+Two traits add capabilities to any of those classes:
  
-The regression filesystem root is:+^ Trait ^ Provides ^ 
 +| ''CliSubprocess'' | ''runCli()'' starts ''./admidio'' as a real process against the test database, ''cliJson()'' decodes its JSON output, ''cliConfigurationFile()'' writes the configuration through the production ''Installation'' service | 
 +| ''PermissionContext'' | ''withOrganization()'', ''withCurrentUser()'', ''loadUserInOrganization()'' and ''settingsOf()'' — Admidio resolves rights against globals, so a test has to set and restore them around the code it exercises |
  
-<code> +''AdmidioTestFixture'' is the shared fixture helper. Instantiate it with the test database and use it for prerequisites: 
-tests/adm_my_files+ 
 +<code php> 
 +protected function getFixture(): AdmidioTestFixture 
 +{ 
 +    return new AdmidioTestFixture($this->getDatabase()); 
 +}
 </code> </code>
  
-The ''FilesystemTestCase'' verifies that the configured Admidio data directory resolves to this location and that the regression marker exists.+It offers, among others, ''createAndSaveOrganization()'', ''createAndSaveUser()'', ''createAndSaveRole()'', ''createAndSaveRoleWithRights()'', ''createAndSaveCategory()'', ''assignUserToRole()'', ''assignUserToRolePeriod()'' and ''seedDefaultPreferences()''. An organization created by the fixture starts without preferences, so seed them if the behaviour under test reads a setting.
  
-If either check fails, destructive filesystem operations are refused.+==== Schema creation and test isolation ====
  
-When adding a filesystem test:+Database-backed runs do not load an SQL dump. The first ''DatabaseTestCase'' in a PHPUnit process calls ''TestDatabaseInitializer'', which drops the prefixed tables and then runs the real production installer. An old dump would let an installer or schema regression pass unnoticed; this way a broken ''install/db_scripts'' change fails the run immediately.
  
-  - use ''FilesystemTestCase''; +The installation the tests work on therefore always contains:
-  - create files only below the test data root; +
-  - register created files and directories for cleanup; +
-  - call the actual Admidio Service or Entity that performs the filesystem operation; +
-  - verify both filesystem and database state where applicable; +
-  - verify cleanup explicitly.+
  
-Never point ''TEST_FILES_PATH'' at the ''adm_my_files'' directory of a real Admidio installation.+  * one organization, //Test Organization// with the short name ''TEST''; 
 +  * the default categories, roles and profile fields of a fresh installation; 
 +  * one administrator with the login ''admin''.
  
-===== Mail tests =====+Every individual test then runs inside a transaction that ''DatabaseTestCase'' rolls back in ''tearDown()''. That keeps tests isolated without recreating the schema for each of them, and it means:
  
-Mail regression tests should exercise the real Admidio email stack rather than mocking the mail sender.+  * a test must not depend on data another test created; 
 +  * a test must not depend on execution order — ''phpunit.xml'' uses ''executionOrder="depends,defects"'', so tests do not even run in source order; 
 +  * anything a test writes through the normal database connection disappears afterwards, and nothing has to be deleted by hand.
  
-The existing Mailpit test follows this path:+The two exceptions are CLI subprocesses, which have their own connection and commit, and files, which the filesystem base class removes explicitly.
  
-<code> +==== Core principle: test Admidio, not the test suite ====
-PreferencesService +
-    -> Admidio Email +
-    -> PHPMailer +
-    -> SMTP +
-    -> Mailpit +
-    -> Mailpit HTTP API +
-</code>+
  
-A good mail regression should use a unique recipient or other unique identifier so that it cannot accidentally match a message from an earlier test run.+The most important rule when adding a regression test is:
  
-The test should verify delivery through Mailpit, not merely whether a TCP port is reachable.+**The action being tested must be performed by production Admidio code.**
  
-===== Choosing the correct base test class =====+A test must not reproduce the expected Admidio behaviour inside a fixture, helper or mock and then verify the behaviour it implemented itself. A test for a Service operation should:
  
-^ Test requirement ^ Base class / pattern ^ +  - create only the prerequisites the test needs; 
-| Pure production logic, no external state | ''AdmidioTestCase'' | +  - call the real Admidio Service; 
-| Database, Entities or Services | ''DatabaseTestCase'' | +  - let that Service use the normal Entities and database abstraction; 
-| Managed files below the test data directory | ''FilesystemTestCase'' | +  - verify the resulting state independently, for example through a new Entity instance or a direct prepared query.
-| Real command-line bootstrap | existing ''CliSubprocess'' / CLI process test pattern |+
  
-Do not make a pure Unit test extend ''DatabaseTestCase'' merely because the helper is convenient. Unit tests should remain fast and independent from external infrastructure.+This matters most for operations that do more than one thing: reciprocal user relations, related records, organization boundaries, changelog entries, sequence values, permissions, messages and emails, thumbnails and archives.
  
-===== Adding an Integration or Service regression =====+Fixtures create prerequisites. They are not substitutes for the production workflow under test. If production code is expected to write two reciprocal relationship records, the test must not write those two records itself and then assert that both exist.
  
-When fixing a bug in a Service or adding a new Service feature, the preferred test structure is:+===== Adding a test =====
  
-**Arrange**+==== Where does the test belong? ====
  
-Create the minimum required organizations, users, roles, categories or other prerequisite objects.+^ The behaviour you want to protect ^ Layer ^ Base class ^ 
 +| A pure function or value object, no state | ''tests/Unit/'' | ''AdmidioTestCase'' | 
 +| An Entity, a Service, a query, a cascade | ''tests/Integration/<Area>/'' | ''DatabaseTestCase'' | 
 +| The same, but the production path checks administrator rights | ''tests/Integration/<Area>/'' | ''AdministratorTestCase'' | 
 +| Visibility, role rights, organization isolation | ''tests/Integration/Permissions/'' or the module's area | ''DatabaseTestCase'' plus the ''PermissionContext'' trait | 
 +| Anything that writes, reads or deletes files | ''tests/Integration/Filesystem/'' | ''FilesystemTestCase'' | 
 +| Sending mail | ''tests/Integration/Mail/'' | ''AdministratorTestCase'' | 
 +| A command of the Admidio CLI | ''tests/Cli/'' | ''DatabaseTestCase'' plus the ''CliSubprocess'' trait |
  
-Prefer normal Admidio Entities and Services for fixtures.+When in doubt, pick the lowest layer that can actually fail when the feature breaks.
  
-**Act**+==== Recipe: a regression test for a bug fix ====
  
-Call the real production method whose behavior is being tested.+A regression test for a bug should fail before the fix and pass after it. That is the only way to know it protects anything.
  
-Examples in the current suite include production paths through Services for inventory, profile fields, OIDC, categories, announcements, menu entries, roles, registrations, documents, photos, import/export and messages.+  - **Reproduce the defect** and find the production class that misbehaves: Entity, Service, CLI command or query. 
 +  - **Write the smallest test** that calls that production code and asserts the correct behaviour. Put it in the area directory of the module, name it after the behaviour, not after the ticket. 
 +  - **Run it and watch it fail** — and check //why// it fails. A test that errors out because a fixture is missing has not reproduced the bug. 
 +  - **Implement the production fix.** 
 +  - **Run the focused test again**, it must now pass: ''vendor/bin/phpunit --filter yourTestName''. 
 +  - **Run the neighbouring tests**, for instance the whole area directory or ''composer test:integration''. 
 +  - **Run ''composer test:all''** before you open the pull request.
  
-**Assert independently**+If the fix is in code that only PostgreSQL or only MySQL reaches, also run the suite with that engine, see [[#switching_the_database_engine|Switching the database engine]].
  
-Read the result again independently.+Avoid writing the assertion only after changing the production code, if that makes it impossible to prove that the test actually detects the regression.
  
-Depending on the feature, use:+==== Recipe: a test for a new feature ====
  
-  - a new production Entity object; +Structure the test as arrange, act, assert independently. The skeleton below follows ''tests/Integration/Services/InventoryServicePathTest.php'':
-  - ''$gDb->queryPrepared()''; +
-  - another Service read operation; +
-  - the physical file written by production code; +
-  - a second CLI subprocess; +
-  - the Mailpit HTTP API.+
  
-The assertion should not simply inspect an array or object populated by the test fixture.+<code php> 
 +<?php
  
-===== Fixtures =====+namespace Admidio\Tests\Integration\Inventory;
  
-Reusable fixtures are useful for common prerequisites such as:+use Admidio\Inventory\Service\ItemFieldService; 
 +use Admidio\Tests\Support\AdministratorTestCase;
  
-  - organizations; +class ItemFieldServiceTest extends AdministratorTestCase 
-  - users; +{ 
-  - roles; +    /** 
-  - memberships; +     * @testdox ItemFieldService stores a new inventory field for the current organization 
-  - categories.+     */ 
 +    public function testSaveDataStoresTheFieldForTheCurrentOrganization(): void 
 +    { 
 +        global $gCurrentOrgId;
  
-Whenever possible, fixtures should create these objects through the same Admidio Entities or Services used by production code.+        // Arrange: only the prerequisites, created through production code 
 +        $db = $this->getDatabase(); 
 +        $fieldName = 'Regression asset tag ' . bin2hex(random_bytes(5));
  
-A fixture may prepare state, but must not implement the behavior under test.+        // Act: the real production Service does the work 
 +        $this->assertTrue((new ItemFieldService($db))->saveData(array( 
 +            'inf_name' => $fieldName, 
 +            'inf_type' => 'TEXT', 
 +            'inf_required_input' => 0, 
 +            'inf_disabled' => 0 
 +        )));
  
-For example, if production code is expected to create two reciprocal relationship records, a test fixture must not create those same two records and then assert that both exist.+        // Assert: read the state back independently of the Service 
 +        $row = $db->queryPrepared( 
 +            'SELECT inf_uuid, inf_name_intern 
 +               FROM ' . TBL_INVENTORY_FIELDS . ' 
 +              WHERE inf_org_id = ? 
 +                AND inf_name = ?', 
 +            array($gCurrentOrgId, $fieldName) 
 +        )->fetch();
  
-The production relationship operation must create them.+        $this->assertIsArray($row); 
 +        $this->assertNotSame('', (string)$row['inf_uuid']); 
 +    } 
 +} 
 +</code>
  
-===== Database assertions =====+Points worth copying:
  
-Direct SQL is useful for independent verification of persistence.+  * the namespace mirrors the directory, and the class name ends in ''Test''; 
 +  * a random suffix in names keeps the test independent of leftovers and of parallel data; 
 +  * the Service is called exactly the way a module calls it; 
 +  * the assertion queries the database itself instead of trusting the object the Service returned; 
 +  * ''@testdox'' states the behaviour, so ''--testdox'' output reads like a specification.
  
-Use Admidio's database abstraction and prepared statements:+==== Using fixtures ==== 
 + 
 +For prerequisites that Admidio does not already install — extra organizations, users, roles, memberships, categories — use ''AdmidioTestFixture'' instead of writing rows by hand: 
 + 
 +<code php> 
 +$fixture = $this->getFixture(); 
 + 
 +$org = $fixture->createAndSaveOrganization('Second Organization', 'SECOND'); 
 +$user = $fixture->createAndSaveUser('regression-user', 'user@example.local', $org['org_id']); 
 +$role = $fixture->createAndSaveRoleWithRights('Editors', $org['org_id'], array('rol_edit_user' => 1)); 
 +$fixture->assignUserToRole($user['usr_id'], $role['rol_id']); 
 +$fixture->seedDefaultPreferences($org['org_id']); 
 +</code> 
 + 
 +Each method returns an array with the generated ids and UUIDs. Fixtures create their objects through the same Entities production code uses. 
 + 
 +The short name of an organization is at most ten characters and unique, and ''TEST'' is already taken by the installed organization. 
 + 
 +A fixture may prepare state, but it must never implement the behaviour under test. 
 + 
 +==== Asserting persistence ==== 
 + 
 +Verify the result through a path that is independent of the code you just called: 
 + 
 +^ Verification ^ Use it for ^ 
 +| A new production Entity, read by id or UUID | the normal case for Entity and Service tests | 
 +| ''$this->getDatabase()->queryPrepared(...)'' | proving exactly which columns were written | 
 +| A second Service read operation | read models, lists, exports | 
 +| The file on disk | filesystem tests, next to the database check | 
 +| A second CLI process | proving a CLI command really committed | 
 +| The Mailpit HTTP API | proving a message was really delivered |
  
 <code php> <code php>
Line 422: Line 543:
 </code> </code>
  
-Direct SQL is appropriate for **asserting** what production code wrote.+Direct SQL is the right tool for **asserting** what production code wrote. It is the wrong tool for reproducing the business operation the test claims to exercise.
  
-It should not be used to reproduce the business operation that the test claims to exercise.+Keep the SQL portable across the engines Admidio supports. Use the ''TBL_...'' constants and prepared statements, and avoid engine-specific syntax unless the test exists precisely to verify the database abstraction.
  
-Tests must also remain portable across the database engines supported by Admidio. Avoid database-specific SQL unless the test explicitly verifies database-specific abstraction behavior.+==== Naming and scope ====
  
-===== Testing permissions and organization boundaries =====+  * A test file ends in ''Test.php'', otherwise PHPUnit does not discover it. 
 +  * The namespace mirrors the directory, for example ''Admidio\Tests\Integration\Roles''. 
 +  * Test method names and ''@testdox'' descriptions state the behaviour that is verified. Prefer //PreferencesService sends a real email through Mailpit// over //Email works//. 
 +  * One regression test has one clear reason to fail. It may perform several steps when those steps form one production workflow, such as create, update and delete of the same object.
  
-Permission and organization-isolation tests require special care.+===== Special kinds of tests =====
  
-A weak test can accidentally prove only that the test author knows how to write a secure SQL query.+==== Permissions and organization boundaries ====
  
-For example, manually writing:+Permission tests are the easiest to get wrong. A test that writes
  
 <code sql> <code sql>
Line 440: Line 564:
 </code> </code>
  
-inside the test does not prove that the production Admidio query applies that restriction.+itself proves only that the test author knows how to write a safe query. It says nothing about the production code.
  
-Whenever the regression concerns visibility, permissions or organization isolation, invoke the production Entity, Service, rights object, presenter query or CLI operation that is responsible for enforcing the boundary.+Whenever the regression concerns visibility, role rights or organization isolation, run the production Entity, Service, rights object or query that is responsible for enforcing the boundary, and then verify that inaccessible data really is absent.
  
-Then verify that inaccessible data is really absent.+The ''PermissionContext'' trait exists for this. Admidio resolves rights against globals: ''User::__construct()'' copies ''$gCurrentOrgId'' into the object, and ''Component::isVisible()'' reads ''$gCurrentUser'', ''$gValidLogin'' and ''$gSettingsManager''. The trait sets those globals around a callback and restores them afterwards, so no test leaks its context into the next one:
  
-===== Testing the CLI =====+<code php> 
 +$visible = $this->withCurrentUser($user, $orgId, true, function () { 
 +    // production code that resolves rights for that user 
 +}); 
 +</code>
  
-CLI tests cover two different areas.+Use ''loadUserInOrganization()'' when the rights have to be resolved against a specific organization — the organization is fixed when the ''User'' object is constructed and cannot be changed afterwards.
  
-==== Contract tests ====+==== Filesystem tests ====
  
-Contract tests inspect command registration and validate characteristics such as:+Tests that exercise documents, photos, imports, exports or any other file operation extend ''FilesystemTestCase''. The regression filesystem root is ''tests/adm_my_files''.
  
-  - command name; +The base class verifies that Admidio's data directory really resolves to that path and that the committed marker exists. If either check fails, it refuses to run.
-  - description; +
-  - usage information; +
-  - arguments; +
-  - options; +
-  - callback availability.+
  
-When adding a new command, make sure it satisfies the generic CLI contract tests instead of adding exceptions for incomplete metadata.+When adding a filesystem test:
  
-==== Workflow tests ====+  * create files only below the test data root, using ''createIsolatedDirectory()'' and ''createFixtureFile()''; 
 +  * register everything production code creates with ''registerCleanupPath()''; 
 +  * call the actual Admidio Service or Entity that performs the file operation; 
 +  * verify both the filesystem and the database state; 
 +  * verify the cleanup path of the feature explicitly, where deleting is part of the behaviour.
  
-Workflow tests exercise actual administration operations.+Cleanup runs in ''tearDown()'', also when an assertion failed, and it fails the test if a registered path survives. Every path is re-checked against the test data root before anything is deleted.
  
-For mutating workflows, use the real executable and verify committed state through another process.+==== CLI tests ====
  
-Use machine-readable output such as JSON where the command supports it, rather than parsing human-oriented console formatting.+CLI coverage has two halves.
  
-The test must also verify exit codes and error output where appropriate.+=== Contract tests ===
  
-===== Test naming and scope =====+Contract tests inspect command registration: name, description, usage, arguments, options and callback availability. When you add a command, make it satisfy the generic contract tests rather than adding an exception for incomplete metadata.
  
-Test file names should end in ''Test.php'' so PHPUnit can discover them through the configured test suites.+=== Workflow tests ===
  
-Test names and ''@testdox'' descriptions should state the actual behavior being verified.+Workflow tests start the real Admidio executable through the ''CliSubprocess'' trait. That process has its own bootstrap, its own database connection, its own exit status and its own output — which makes it a genuine acceptance boundary, and which has consequences:
  
-Prefer a description such as:+  * **A subprocess cannot see the PHPUnit transaction.** Prerequisites must either exist in the committed baseline created by the installer, or be created through subprocess commands as part of the scenario. 
 +  * **A subprocess commits.** Its changes survive the rollback, so a mutating test must clean up after itself, normally in a ''finally'' block. 
 +  * **File-writing commands are deliberately not covered here**, because the real CLI bootstrap uses the ''adm_my_files'' of the checkout rather than the test data directory.
  
-<code> +The established pattern is create, verify from a second process, delete, verify the deletion: 
-PreferencesService sends a real email through Mailpit+ 
 +<code php> 
 +$login = 'cli-e2e-' . bin2hex(random_bytes(5)); 
 +$created = false; 
 + 
 +try { 
 +    $create = $this->runCli(array('--as=admin', 'user:add', '--login=' . $login, ...)); 
 +    $this->assertSame(0, $create->getExitCode(), $create->getErrorOutput()); 
 +    $created = true; 
 + 
 +    // A second process has a different connection. Seeing the record here proves 
 +    // the production CLI command committed an actual Admidio database write. 
 +    $show = $this->runCli(array('--as=admin', 'user:show', $login, '--format=json')); 
 +    $this->assertSame(0, $show->getExitCode(), $show->getErrorOutput()); 
 +    $this->assertSame($login, $this->cliJson($show)['login']); 
 +} finally { 
 +    if ($created) { 
 +        $delete = $this->runCli(array('--as=admin', 'user:delete', $login, '--yes')); 
 +        $this->assertSame(0, $delete->getExitCode(), $delete->getErrorOutput()); 
 +    } 
 +}
 </code> </code>
  
-over:+Assert the exit code and the error output, and parse machine-readable output such as ''--format=json'' instead of human-oriented console formatting. Use ''CliSubprocess'', do not write another subprocess launcher. 
 + 
 +==== Mail tests ==== 
 + 
 +Mail regressions exercise the real Admidio email stack instead of mocking the sender. The existing test follows this path:
  
 <code> <code>
-Email works+PreferencesService 
 +    -> Admidio Email 
 +    -> PHPMailer 
 +    -> SMTP 
 +    -> Mailpit 
 +    -> Mailpit HTTP API
 </code> </code>
  
-One regression test should have one clear reason to fail.+Guidelines:
  
-A test may perform several steps when those steps form one production workflow.+  * use a unique recipient address per run, so the assertion cannot match a message from an earlier run; 
 +  * verify the delivered message through the Mailpit API, not merely that a TCP port answered; 
 +  * restore the mail settings and any user data you changed, even though the transaction is rolled back, if the code under test caches them.
  
-===== How to test a bug fix =====+The mail test deliberately ignores Docker's health status. Some environments report the Mailpit container as ''unhealthy'' while Mailpit works perfectly. What matters is whether Admidio can hand the message to the SMTP listener and whether it then appears in the API. A Docker health label alone is not a reason to disable the test.
  
-A regression test for a bug should ideally fail before the production fix and pass after it.+==== Installation coverage ====
  
-A useful workflow is:+Every database-backed run installs Admidio from scratch with the production installer, so the installer is covered implicitly: if ''install/db_scripts'' or the installation service breaks, the whole suite fails during setup rather than in a single test.
  
-  - reproduce the defect; +''tests/Cli'' adds explicit coverage of the installation result and of maintenance mode on top of that.
-  - add the smallest test that demonstrates the incorrect production behavior; +
-  - run the test and confirm that it fails for the expected reason; +
-  - implement the production fix; +
-  - run the focused test again; +
-  - run related Integration or CLI tests; +
-  - finally run ''composer test:all''.+
  
-Avoid writing the assertion only after changing the production code if doing so makes it impossible to prove that the test actually detects the regression.+===== Checklists =====
  
-===== Guidance for third-party developers =====+==== Before you open a pull request ====
  
-Third-party modules and plugins benefit from following the same testing principles even when their tests are maintained outside the Admidio Core repository.+  * Does ''composer test:all'' pass on your primary engine? 
 +  * If the change touches SQL, Entities or the installer: does it also pass on the other engine? 
 +  * Does every new behaviour have a test that would fail without your production change? 
 +  * Do new tests clean up subprocess and filesystem changes? 
 +  * Are new language strings, database changes and update steps covered by the change itself?
  
-Use a checkout of the Admidio version against which the extension is developed and run tests against a dedicated test database.+A green run does not replace code review. Reviewers should still check whether the new tests exercise the correct production path, and whether error, permission and cross-organization cases are covered.
  
-For extension tests:+==== Reviewing a new regression test ====
  
-  - invoke real Admidio APIs rather than duplicating them; +  * Does the test invoke the actual production Entity, Service, CLI command or query it claims to test? 
-  - use Admidio Entities and Services according to the same patterns used by Core; +  * Does it verify the resulting state independently? 
-  - never point tests at a production Admidio database; +  * Does it verify a real database write where persistence is part of the feature? 
-  - keep filesystem fixtures separate from a real ''adm_my_files''; +  * Does it avoid duplicating the production business logic inside the test? 
-  - use Mailpit or another local SMTP sink for mail behavior; +  * Is the fixture limited to prerequisites? 
-  - verify organization and permission boundaries through production code; +  * Is the test isolated from other tests and independent of execution order? 
-  - test against all database engines your extension claims to support.+  * Does cleanup also run when an assertion fails? 
 +  * Is filesystem access restricted to ''tests/adm_my_files''? 
 +  * Does a CLI write become visible to another independent process? 
 +  * Does a mail test verify a message that actually reached Mailpit? 
 +  * Are permissions and organization boundaries tested through production code? 
 +  * Is the SQL portable across the supported engines? 
 +  * **Would the test fail if the production behaviour it protects were removed?**
  
-If a third-party change exposes a regression or missing contract in Admidio Core itself, consider contributing the corresponding regression test to the Core suite.+If the answer to the last question is no, the test is probably testing its own setup rather than Admidio.
  
-===== What not to do =====+==== Anti-patterns ====
  
 Do not add a test that: Do not add a test that:
  
-  - stores expected data only in an in-memory array and reads it from the same array; +  * stores expected data in an array and reads it back from the same array; 
-  - implements a fake Entity or fake Service instead of invoking Admidio; +  * implements a fake Entity or fake Service instead of invoking Admidio; 
-  - manually repeats the database changes the production Service is supposed to make; +  * repeats by hand the database changes the production Service is supposed to make; 
-  - passes when no relevant database record exists; +  * passes when no relevant database record exists; 
-  - catches an unexpected exception and then succeeds unconditionally; +  * catches an unexpected exception and then succeeds unconditionally; 
-  - uses assertions such as ''rowCount() >= 0'' that cannot fail meaningfully; +  * uses assertions such as ''rowCount() >= 0'' that cannot fail meaningfully; 
-  - depends on another test having executed first; +  * depends on another test having run first; 
-  - writes files outside the protected regression directory; +  * writes files outside the protected regression directory; 
-  - points to a non-test database; +  * points at a non-test database; 
-  - assumes a subprocess can see an uncommitted PHPUnit transaction.+  * assumes a subprocess can see an uncommitted PHPUnit transaction.
  
-A regression test that cannot fail when the corresponding production feature is broken provides false confidence and should be corrected.+A test that cannot fail when the corresponding feature is broken provides false confidence and should be corrected.
  
-===== Reviewing a new regression test =====+===== Third-party modules and plugins =====
  
-Before merging a new test, check the following questions:+Third-party modules and plugins benefit from the same principles even when their tests live outside the Admidio Core repository. Use a checkout of the Admidio version you develop against, and a dedicated test database.
  
-  - Does the test invoke the actual production Entity, Service, CLI command or other production path it claims to test? +  * invoke real Admidio APIs rather than duplicating them; 
-  - Does it independently verify the resulting state? +  * use Entities and Services following the same patterns as Core; 
-  - Does it verify a real database write when persistence is part of the feature? +  * never point tests at a production Admidio database; 
-  - Does it avoid duplicating the production business logic in the test? +  * keep filesystem fixtures out of a real ''adm_my_files''; 
-  - Is the fixture limited to prerequisites? +  * use Mailpit or another local SMTP sink for mail behaviour; 
-  - Is the test isolated from other tests? +  * verify organization and permission boundaries through production code; 
-  - Does cleanup also run when an assertion fails? +  * test against every database engine your extension claims to support.
-  - Is filesystem access restricted to ''tests/adm_my_files''? +
-  - Does a CLI write become visible to another independent process? +
-  - Does a mail test verify a message that actually reached Mailpit? +
-  - Are permissions and organization boundaries tested through production code? +
-  - Is the test portable across the relevant supported database engines? +
-  - Would the test fail if the production behavior it protects were removed?+
  
-If the answer to the last question is "no", the test is probably testing its own setup rather than Admidio.+If a third-party change exposes a regression or a missing contract in Admidio Core itself, consider contributing the corresponding regression test to the Core suite.
  
 ===== Troubleshooting ===== ===== Troubleshooting =====
  
-==== Safety check rejects the database ====+==== Safety check failed: database name does not contain "test" ====
  
-Use a dedicated database whose name contains ''test'' as a separate token, for example: +The name must contain ''test'' as a separate token, so ''admidio_test'' works but ''admidiotest'' does not. Check ''TEST_DATABASE_ENGINE'' and the ''TEST_DB_<ENGINE>_NAME'' that belongs to it — with three connection blocks in the file it is easy to edit the wrong one.
- +
-<code> +
-admidio_test +
-</code>+
  
-Check the selected ''TEST_DATABASE_ENGINE'' and the corresponding ''TEST_DB_*'' variables in ''.env.test''.+Note that the setup script checks the name more loosely than the test run does. If ''php tests/bin/setup-test-env.php'' is happy but PHPUnit refuses the database, this is why.
  
 ==== Database connection fails ==== ==== Database connection fails ====
  
-Check that:+Check that the service is running (''docker compose -f docker-compose.test.yml ps''), that host and port are right, that the PDO driver for the engine is installed, that the database exists, and that the user may create and drop tables in it.
  
-  - the selected database service is running; +On MySQL and MariaDB, use ''127.0.0.1'' rather than ''localhost'' so the client does not try the unix socket.
-  - hostname and port are correct; +
-  - the PDO driver is installed; +
-  - the test database exists; +
-  - the configured user has sufficient rights to create and remove the Admidio test tables.+
  
-==== Unit tests work but Integration tests fail immediately ====+==== "Database not initialized" or a connection error at the first integration test ====
  
-Unit tests deliberately do not initialize the Admidio database.+The bootstrap connects lazily, so a wrong configuration only surfaces when the first database-backed test runs. The message repeats the underlying connection error. Run the setup script, it reports the same problem with more context.
  
-Check ''.env.test'' and the database service first.+==== Unit tests pass but integration tests fail immediately ====
  
-==== Filesystem test refuses to run ====+Unit tests deliberately never touch the database. A failure that appears only in the integration suite is almost always the environment: check ''.env.test'' and the database service first.
  
-Verify:+==== The engine did not change ====
  
-<code> +''--db='' is an option of the setup script only. The test run reads ''TEST_DATABASE_ENGINE'' from the environment or ''.env.test''. Also remember that a variable already set in your shell wins over the file. 
-TEST_FILES_PATH=./tests/adm_my_files+ 
 +==== Port 3306 is already in use ==== 
 + 
 +The MariaDB container publishes 3306. A local MySQL or MariaDB service on the same port will conflict. Stop the local service, or change the published port in ''docker-compose.test.yml'' and in ''.env.test''. 
 + 
 +==== PostgreSQL: permission denied for schema public ==== 
 + 
 +From PostgreSQL 15 on, the ''public'' schema does not let every role create objects. Grant it to the test user: 
 + 
 +<code sql> 
 +GRANT ALL ON SCHEMA public TO admidio;
 </code> </code>
  
-and make sure the checked-in file:+==== A filesystem test refuses to run ==== 
 + 
 +Verify that ''.env.test'' contains
  
 <code> <code>
-tests/adm_my_files/.admidio-regression-test+TEST_FILES_PATH=./tests/adm_my_files
 </code> </code>
  
-still exists. +and that the committed marker file ''tests/adm_my_files/.admidio-regression-test'' still exists. Do not build a workaround that disables this protection.
- +
-Do not create a workaround that disables this protection.+
  
 ==== Mailpit is shown as unhealthy ==== ==== Mailpit is shown as unhealthy ====
  
-Ignore the Docker health label initially and check the actual services. +Ignore the Docker health label and check the endpoints the test actually uses:
- +
-The regression test uses:+
  
 <code> <code>
Line 623: Line 781:
 </code> </code>
  
-If those endpoints work, the Mailpit integration test can work even when Docker reports an incorrect health status.+If those work, the mail test can pass even while Docker reports the container as unhealthy.
  
-==== CLI test cannot see a fixture ======+==== A CLI test cannot see its fixture ====
  
-A real CLI subprocess uses another database connection and cannot see uncommitted rows created inside the PHPUnit transaction. +A real CLI subprocess has its own database connection and cannot see uncommitted rows from the PHPUnit transaction. Create the prerequisite through the CLI subprocess itself, or use data from the committed baseline the installer created.
- +
-Create the prerequisite through the CLI subprocess itself or use data that belongs to the committed regression baseline.+
  
 ==== A test leaves data behind ==== ==== A test leaves data behind ====
  
-Normal ''DatabaseTestCase'' changes should disappear through transaction rollback.+Changes made through ''DatabaseTestCase'' disappear with the transaction rollback. Subprocess changes are committed independently and must be removed by the test itself, normally in a ''finally'' block. Filesystem changes must be registered with ''registerCleanupPath()''.
  
-Mutating subprocess changes are committed independently and must therefore be explicitly removed by the test.+==== Code coverage is empty ====
  
-Filesystem changes must be registered for cleanup through ''FilesystemTestCase''.+''composer test:coverage'' needs Xdebug or PCOV. Without a coverage driver PHPUnit runs the tests but writes no report.
  
-===== Before submitting a pull request =====+==== Windows notes ====
  
-Run the focused tests while developing, then run:+''cp'' does not exist in ''cmd.exe'' or PowerShell, use ''copy'' or ''Copy-Item''. Use ''vendor\bin\phpunit'' or ''vendor/bin/phpunit'' depending on the shell. Everything else, including the Docker environment and the setup script, works the same way.
  
-<code bash> +===== See also =====
-composer test:all +
-</code>+
  
-A successful regression run does not replace code review. Reviewers should still check whether the new tests exercise the correct Admidio production path and whether important error, permission and cross-organization cases are covered.+  * [[en:entwickler:testumgebung_einrichten|Set up a test environment]] 
 +  * ''docker-compose.test.yml'', ''.env.test.example'', ''phpunit.xml'' and ''tests/bin/setup-test-env.php'' in the repository 
 +  * ''.github/workflows/regression-tests.yml'' for the exact commands CI runs
  
-The goal of the regression suite is not to maximize the number of tests. The goal is to make real Admidio regressions visible as reliable, understandable test failures.+The goal of the regression suite is not to maximise the number of tests. The goal is to make real Admidio regressions visible as reliable, understandable test failures.
  
  • en/entwickler/regression_test_suite.1787571612.txt.gz
  • Last modified: 2026/08/24 13:40
  • by kainhofer