Running Leapp against one server is a tutorial. Running it against a large enterprise estate is a campaign, and the failure modes are completely different. These are the traps I hit while automating RHEL 7 to 8 in-place upgrades with Ansible over the last few months, and the playbook patterns that fixed them.

The moving parts

Each migration wave runs through the same pipeline, driven from Ansible Automation Platform:

  1. pre_upgrade.yml - patch to latest RHEL 7, install Leapp, preseed answers, re-register the host against the upgrade content view
  2. rear_backup.yml - install ReaR, prepare local disk space, take a full backup to a remote backup server
  3. leapp upgrade and reboot
  4. post_upgrade.yml - cleanup, re-register against RHEL 8 content, application-specific fixes

Hosts come in as a simple list and get expanded into an in-memory inventory, so a wave is just a vars file:

- name: Build the in-memory inventory for this wave
  hosts: localhost
  gather_facts: false
  tasks:
    - name: Add each host from the wave list
      add_host:
        name: "{{ item.name | lower }}"
        groups: upgrade_wave
      loop: "{{ upgrade_hosts }}"

Pin your migration tooling

The first trap has nothing to do with any single host: it is version skew across the campaign. We started the obvious way:

- name: Install Leapp
  yum:
    name: leapp-upgrade
    state: latest

state: latest means every wave potentially migrates with a different Leapp than the one you validated. Mid-campaign, a new leapp release changed behavior and broke a tested flow, so we pinned the exact NVRs:

 - name: Install Leapp
   yum:
-    name: leapp-upgrade
-    state: latest
+    name:
+      - leapp-<pinned-nvr>.el7_9.noarch
+      - leapp-upgrade-el7toel8-<pinned-nvr>.el7_9.noarch
+      - leapp-upgrade-el7toel8-deps-<pinned-nvr>.el7_9.noarch
+    state: present

Getting there took more iterations than it should have, because the packaging is not obvious: pinning leapp alone still pulls the newest leapp-upgrade-el7toel8, the deps package follows the version stream of leapp-upgrade-el7toel8 rather than leapp itself, and we managed to paste a filename instead of a package name (...noarch.rpm instead of ...noarch) along the way. Pin all three, and treat the pinned set as part of your tested migration artifact.

The same applies to the Leapp data files. We host a known-good snapshot on the Satellite and install exactly that, instead of whatever the metadata channel serves today:

- name: Install the pinned leapp data snapshot
  unarchive:
    src: "{{ satellite_url }}/pub/leapp-data.tar.gz"
    dest: /etc/leapp/files
    remote_src: yes

Preseed the answers Leapp will ask for

leapp preupgrade produces inhibitors that expect a human to confirm choices in the answer file. In a fleet run there is no human, so the playbook answers up front and removes known blockers:

- name: Preseed the PAM PKCS11 answer
  command: leapp answer --section remove_pam_pkcs11_module_check.confirm=True

- name: Unload the pata_acpi kernel module
  command: modprobe -r pata_acpi

- name: Blacklist the pata_acpi kernel module
  copy:
    content: 'blacklist pata_acpi'
    dest: /etc/modprobe.d/pata_acpi-blacklist.conf

pam_tally2 is gone in RHEL 8 and inhibits the upgrade, so it gets stripped from system-auth-ac beforehand. Every estate has its own short list of these; the point is that each one you discover goes into the playbook, not into a wiki page.

One more pre-flight that regularly saved a reboot: RHEL 7 hosts installed years ago still use kernel eth0 style NIC naming, which does not survive the upgrade. Red Hat solution 4067471 describes the fix, and we run it as its own imported playbook before anything else touches the host.

ReaR backups: disk space and timeouts

Rollback for us is a full ReaR backup shipped to a backup server over sshfs, with the ISO staged locally first:

OUTPUT=ISO
BACKUP=NETFS
OUTPUT_URL={{ rear_output_url }}
BACKUP_URL=iso://{{ rear_staging_dir }}
BACKUP_PROG_EXCLUDE={{ rear_excludes | default(rear_default_excludes) }}
export TMPDIR="{{ rear_staging_dir }}"
ISO_DIR="{{ rear_staging_dir }}/output"

Staging locally means you need local disk. On hosts with free space in the volume group we create a dedicated 100G logical volume. On hosts without that luxury (mostly VMs), the playbook finds whatever filesystem holds the backup directory and grows it as far as the VG allows:

- name: Find the device backing the staging directory
  ansible.builtin.shell: "df -h {{ rear_staging_dir }} | awk 'NR==2 {print $1}'"
  register: staging_dev_path
  changed_when: false

- name: Grow the logical volume to the rest of the volume group
  ansible.builtin.shell: "lvextend -l +100%FREE {{ staging_dev_path.stdout }}"

Do the disk math before the campaign, not during. A failed mkbackup at 2 percent free is the single most common way a migration window dies.

The second ReaR lesson: rear mkbackup on a big host runs longer than your SSH connection wants to live. The task would die mid-backup with the backup half-written. The fix is boring and mandatory:

     - name: Run the ReaR backup
       command: rear -d -v mkbackup
+      async: 1800
+      poll: 60

Ansible detaches the job on the host and polls every minute, so a dropped control connection no longer kills the backup.

Per-host variance: migration profiles as custom facts

A hundred hosts means a hundred slightly different hosts. Before the upgrade we scan each machine for the applications that need special handling and write the result as an Ansible custom fact:

[apps]
migrated=True
ha_cluster=False
java_app_installed=False
web_server_installed=True
backup_agent_installed=True
db_installation=Server

Post-migration, the role branches on those facts:

- name: Run the web server post tasks
  include_tasks: post_web_server.yml
  when: ansible_local.upgrade_profile.apps.web_server_installed | bool

This kept one generic playbook for the whole estate while database servers, Java application servers and a handful of in-house agents each got their own small post-upgrade task file.

Post-migration is half the work

The reboot into RHEL 8 is not the finish line. The post playbook asserts it is actually talking to a RHEL 8 host, then:

- name: Make python 3 the default
  command: alternatives --set python /usr/bin/python3

- name: Prune old kernels
  command: "yum -y remove --oldinstallonly --setopt installonly_limit=3 kernel"

- name: Remove the leapp packages
  yum:
    name: 'leapp*'
    state: absent

Plus: clear the sssd cache (stale RHEL 7 entries broke logins), re-register the host in Satellite with the RHEL 8 activation key, flip the inventory variables in the automation controller from RHEL7 to RHEL8, and re-enable the monitoring cron jobs the pre playbook disabled. Every one of these started as a manual “oh, and also” after an early migration and got folded back into the role.

Checklist for your own campaign

  • Pin leapp, leapp-upgrade and the deps packages to exact NVRs; upgrade the pin deliberately, between waves
  • Snapshot the leapp data files and serve them from your own Satellite
  • Preseed every leapp answer your estate needs; treat new inhibitors as code changes
  • Verify and extend backup disk space automatically before running ReaR
  • Run long backups with async/poll
  • Detect per-host application variance up front and store it as custom facts
  • Automate the post-upgrade cleanup with the same rigor as the upgrade

The same structure carries over almost unchanged to RHEL 8 to 9 with leapp-upgrade-el8toel9, which is where the estate is heading next.

A note on support timelines

If you are planning a campaign like this, the clock matters more than the tooling. RHEL 7 reaches the end of its maintenance support phase on 2024-06-30; after that the only supported ways to keep running it are the paid Extended Life Cycle Support (ELS) add-on or, on the major clouds, Extended Update Support style offerings through the marketplace images. RHEL 8 is in full support until mid 2024 and in maintenance support until 2029-05-31, with its own ELS phase after that. In other words, a 7 to 8 migration finished in 2023 buys roughly six years of supported runway, and every month of delay eats into it. Check the Red Hat Enterprise Linux Life Cycle page (linked below) for the current dates before you commit to a target release.

Resources