GitLab Upgrade Cycle on AlmaLinux

This is the reusable upgrade cycle we followed for upgrading GitLab Omnibus safely on AlmaLinux.

1. Check the Current GitLab Version

sudo gitlab-rake gitlab:env:info

Verify the current versions of GitLab, PostgreSQL, Redis, Gitaly, Ruby, and other bundled components.

2. Check GitLab Health Before Upgrading

sudo gitlab-rake gitlab:check SANITIZE=true sudo gitlab-ctl status

Resolve any important errors before starting the next upgrade.

3. Check Background Migrations

sudo gitlab-rake gitlab:background_migrations:status | grep -i -E 'failed|active'

The ideal result before upgrading is no output.

You can also check active and failed migrations separately:

sudo gitlab-rake gitlab:background_migrations:status | grep -ic '^active' sudo gitlab-rake gitlab:background_migrations:status | grep -i failed

Expected result:

Active migrations: 0 Failed migrations: no output

4. Make Sure a Backup Exists

Before important upgrade stops or major-version upgrades, make sure a recent backup exists.

sudo gitlab-backup create

Also preserve the main GitLab configuration and secrets:

/etc/gitlab/gitlab.rb /etc/gitlab/gitlab-secrets.json

5. Determine the Next Required Upgrade Stop

Do not jump directly to the newest GitLab version. GitLab has required upgrade stops between major and minor releases.

The upgrade path followed in this case was:

17.8.7 ↓ 17.11.7 ↓ 18.2.8 ↓ 18.5.7 ↓ 18.8.11 ↓ 18.11.11 ↓ 19.0.8 ↓ 19.2.4

The exact required stops depend on the current and target versions, so the official GitLab upgrade path should always be checked before continuing.

6. Check Available Packages

Example:

sudo dnf --showduplicates list gitlab-ee | grep '19\.2'

Normally use the latest patch available inside the required minor version instead of installing the .0 release.

7. Install the Exact GitLab Version

Example:

sudo dnf install gitlab-ee-19.2.4-ee.0.el8

Allow the complete installation and reconfiguration process to finish. Do not interrupt GitLab while migrations or reconfigure operations are running.

8. Verify the New Version

sudo gitlab-rake gitlab:env:info

Confirm that the expected GitLab version is installed and review any bundled component upgrades such as PostgreSQL or Redis.

9. Check GitLab Services

sudo gitlab-ctl status

Important services normally include:

postgresql redis gitaly puma sidekiq nginx gitlab-workhorse

If the installer explicitly reports that a running bundled service is using an older version than the installed package, restart only that service.

Example:

sudo gitlab-ctl restart redis

10. Run the GitLab Health Check

sudo gitlab-rake gitlab:check SANITIZE=true

The goal is to complete the check without important failures.

11. Wait for Background Migrations

After an upgrade, GitLab may create several new background migrations.

sudo gitlab-rake gitlab:background_migrations:status | grep -i -E 'failed|active'

You can count how many active jobs remain:

sudo gitlab-rake gitlab:background_migrations:status | grep -ic '^active'

It is normal to see the number gradually decrease:

21 13 5 3 0

If migrations do not appear to be progressing, verify Sidekiq:

sudo gitlab-ctl status sidekiq

12. Do Not Start the Next Upgrade Until Migrations Are Clean

Before every next upgrade, run:

sudo gitlab-rake gitlab:background_migrations:status | grep -i -E 'failed|active'

The ideal result is no output.

Also verify:

sudo gitlab-rake gitlab:background_migrations:status | grep -ic '^active' sudo gitlab-rake gitlab:background_migrations:status | grep -i failed

Expected result:

0

and no output for failed migrations.

13. Repeat the Same Upgrade Cycle

CHECK CURRENT VERSION ↓ CHECK GITLAB HEALTH ↓ VERIFY BACKUP ↓ VERIFY ACTIVE MIGRATIONS = 0 ↓ VERIFY FAILED MIGRATIONS = 0 ↓ CHECK NEXT REQUIRED VERSION ↓ CHECK AVAILABLE PACKAGE ↓ INSTALL EXACT VERSION ↓ VERIFY NEW VERSION ↓ CHECK SERVICES ↓ RUN GITLAB CHECK ↓ WAIT FOR BACKGROUND MIGRATIONS ↓ ACTIVE = 0 / FAILED = 0 ↓ CONTINUE TO NEXT REQUIRED VERSION

14. Important Note About Failed Background Migrations

Do not automatically modify the database or manually mark a failed background migration as completed.

First inspect the migration and its logs. Some failures may be known GitLab upgrade issues with a documented upgrade path or workaround.

During this upgrade, the following migration failed temporarily:

BackfillSentNotificationsAfterPartition

The correct action was to follow GitLab's documented handling and continue through the required upgrade path instead of manually changing PostgreSQL partitions or forcing the migration status.

15. Main Rule

Never upgrade to the next GitLab version while important background migrations are still active or failed, unless GitLab documentation explicitly states that a specific known migration is safe to bypass.