An enterprise GitLab upgrade changes more than a version number. It may alter policy schemas, evaluation behaviour, API responses, runner interaction and the timing of pipeline creation. Release notes and staging checks are essential, but they do not prove that the controls your organisation depends on still work in your configuration.
Automated integration tests close that gap. They exercise policies through the same project, group, API and pipeline boundaries used in production, then record whether observable behaviour matches the intended control.
The goal is not to reproduce GitLab’s own test suite. It is to verify your organisation’s implementation: the policy project, scopes, exceptions, pipeline components and approval rules that make a generic feature into a working control.
Begin with control intent
Before writing a test, state the behaviour in plain language. For example:
Every in-scope branch pipeline must include the centrally managed secret-detection job, even when the application project does not define that job itself.
That statement identifies the subject, scope and observable outcome. It is much stronger than “test scan execution policy”, which leaves too much open to interpretation.
Build a compact catalogue of intended behaviours. Include positive cases, negative cases and deliberate exceptions. A good catalogue often covers:
- a project that should receive the policy;
- a project outside the policy scope;
- a branch, pipeline source or file condition that changes evaluation;
- an attempted local override;
- the expected failure or approval state;
- the evidence an operator should see.
This catalogue becomes a shared reference for platform, security and assurance stakeholders.
Use disposable test projects
Policy tests change repositories, pipeline definitions and approval state. They are safest when each run creates or resets controlled fixtures rather than borrowing a real application.
A test harness can use the GitLab REST API to create a project from a small template, commit the required files, run a pipeline and wait for a terminal state. GraphQL is useful where the required policy or approval information is easier to query through that schema. The harness should record resource identifiers and clean up what it owns.
Fixtures should be intentionally small. A repository for container scanning needs only enough content to cause the expected job and result. A merge approval fixture needs a predictable finding and a merge request. Smaller fixtures make failures easier to interpret and reduce runtime.
Avoid tests that depend on mutable public images or packages. Pin known artefacts where possible so that a third-party change does not look like a platform regression.
Testing pipeline execution policies
Pipeline execution policies can inject or replace CI configuration at defined stages of pipeline evaluation. Tests should confirm both the presence of the central component and its relationship to project-owned jobs.
Useful assertions include:
- the policy job appears in the resulting pipeline;
- its configuration comes from the expected central source;
- required variables or execution stages are present;
- a conflicting local job cannot silently bypass the control;
- projects outside the scope do not receive the policy;
- invalid policy configuration produces a visible, actionable failure.
Prefer assertions against the resulting pipeline and jobs over assertions that only inspect the YAML policy file. The file can be syntactically correct while scope, permissions or evaluation prevent it from taking effect.
Testing scan execution policies
Security scan execution policies should be tested for scheduling and merge-request contexts that matter to the organisation. Trigger the relevant pipeline source, then query jobs and security results through supported APIs.
An assertion might check that a SAST or secret-detection job ran with the expected status. A deeper test can include a safe, synthetic fixture that the scanner is expected to detect, then verify that the finding is associated with the pipeline or merge request.
Be careful with the distinction between job success and control success. Some scanners complete successfully while reporting vulnerabilities. The test must assert the policy outcome, not assume a green job means no findings.
Testing merge request approval policies
Merge request approval policies combine several moving parts: a qualifying finding, the security report, policy evaluation, approval rules and mergeability. Tests should follow the complete sequence.
Create a branch with a controlled test finding, run the pipeline, open or update the merge request, and wait for policy evaluation. Then assert the approval rule, required approval count, eligible approver group or merge restriction that represents the intended control.
Add a complementary case with no qualifying finding. This prevents a policy that blocks every merge from being mistaken for a successfully enforced risk-based control.
Timing matters. Security report ingestion and policy evaluation may be asynchronous, so the harness needs bounded polling with helpful timeout messages. A fixed sleep is slower and less reliable.
Structure the suite with pytest
Pytest works well for this kind of platform assurance because fixtures can manage API clients and disposable projects, while markers organise slower or more privileged scenarios.
Keep API interaction behind a small client layer. Tests should read as statements of behaviour rather than sequences of HTTP calls. For example:
def test_secret_detection_is_injected(policy_project):
pipeline = policy_project.run_pipeline(ref="main")
jobs = policy_project.wait_for_jobs(pipeline.id)
assert "secret_detection" in jobs.names
assert jobs["secret_detection"].source == "security-policy"The exact API representation will differ, but the intent remains visible. When the platform changes, client parsing can be updated without rewriting every scenario.
Produce evidence people can use
JUnit reports make test results available to CI systems without tying the suite to a particular dashboard. Publish the report and a concise diagnostic log as pipeline artefacts. Include the GitLab version, test environment, policy revision and fixture identifiers so a failure can be reproduced.
Do not expose access tokens, sensitive configuration or full API responses in logs. Redact headers and prefer selected diagnostic fields.
Run a fast critical subset during upgrade rehearsals and the full suite before promotion. The same tests can run on a schedule to detect configuration drift after the upgrade is complete.
Treat tests as part of policy management
Security policy automation is most reliable when policy and tests change together. A merge request that changes central policy should add or update the behavioural scenario that proves the new intent. Reviewers can then assess the control and its evidence in one place.
An automated suite does not replace staged rollout, release-note review or manual exploration of major changes. It provides a repeatable layer of evidence between vendor documentation and production confidence. For centrally managed GitLab controls, that evidence is worth building before the next upgrade window begins.