Friday, 3 August 2018

How to Upgrade a High Availability (HA) Pair

Best Practices for PAN-OS Upgrade

by  on ‎09-06-2016 02:49 PM - edited on ‎06-25-2018 12:35 AM by Community Manager (179,064 Views)
Labels:

Best Practices for PAN-OS Upgrade

The following procedure documents best practices for customers who are new to the PAN-OS upgrade process.  It’s intended as a foundation for customers who want to create their own more-specific upgrade procedures.

About the PAN-OS upgrade and customer responsibilities

  • You cannot skip installation of any feature  release versions in the path to your target PAN-OS
    release. Additionally, it is best practice to always download and install the latest maintenance release
    for each feature release and then reboot before you install the base image for the next feature
    release—this applies to each feature release through which you pass in the upgrade path.
  • In this example, we are upgrading a hypothetical customer (ACME, Inc.) from PAN-OS 7.0.16 to 8.0.6-h3 (with 7.1 as an interim step).
  • ACME firewall is configured in Active/Passive HA cluster managed by Panorama (this is the most common configuration in use today). We are not covering Active/Active, non-HA scenarios or scenarios where there is no Panorama installed.
  • This is a best practice document. It is not meant to be a step-by-step procedure.  Please create your own step-by-step procedure, if necessary.
  • Customer is responsible for verifying all steps before the upgrade.
Terminology
Active firewall
The firewall in an HA cluster that’s passing traffic
Passive firewall
The firewall in an HA cluster that’s not passing traffic
Primary firewall
The firewall in an HA cluster that’s usually the active firewall
Secondary firewall
The firewall in an HA cluster that’s usually the passive firewall
Feature release

Contains new features and bugfixes, typically ends with .0 (i.e. 7.1.0)
Maintenance release
Bug fixes only, typically ends with .number (7.1.2)

Dependencies

  • Before upgrade, make sure  the firewall is running a version of app + threat (content version) that meets the minimum requirement of the new PAN-OS (see release notes: https://www.paloaltonetworks.com/documentation.html).  We recommend always running the latest version of content to ensure the most accurate and effective protections are being applied.
  • Panorama should be running the same or a later version of a feature release than the firewall (up to two feature versions is supported but not recommended as of June 2016).
If the firewall is running
Panorama versions supported are
PAN-OS 7.0.x
7.0.x, 7.1.x, 8.0.x
PAN-OS 7.1.x
7.1.x, 8.0.x
PAN-OS 8.0.x
8.0.x

Table of contents

  1. Determine the need to upgrade
  2. Pre-upgrade checklist
  3. Panorama upgrade procedure
  4. Firewall upgrade procedure (HA)
  5. Post-upgrade checklist
  6. Troubleshooting resources
  7. Downgrade procedure

1. Determine the need to upgrade

  • In most cases, upgrade should be considered for only the following reasons:
  • We highly encourage customers to consult with Palo Alto Networks account team for upgrade decision. Your Palo Alto Networks account team can provide you with a recommended PAN-OS version.
  • For purpose of this document, we will be upgrading from 7.0.16 to 8.0.6-h3 to demonstrate upgrading process across two major releases (7.0 > 7.1 > 8.0).
  • NOTE: 
    For any PAN-OS version prior to PAN-OS 8.0 (so PAN-OS 7.1 and lower) it is recommended to go to the latest maintenance release to prevent running into snags or issues during the upgrade.
  • Note about PAN-OS 8.0 base installation: 
    For PAN-OS 8.0, The additional (Optional when installing from the updates server. When installing from a manual file this step is mandatory) step of installing  and rebooting the base image was added to accomodate the larger size of the base image. This is considered best practice.
  • HA Upgrade NOTE: 
    When upgrading across two major release versions at a time, there will be a time when there will be a network outage. Whereas if the devices are upgraded one major version at a time, HA will remain active, continue to synchronize sessions, and no network outage will be seen.
To maintain HA sync and activity, upgrade the HA pair in tandem one major release at a time. If you upgrade one device by two major upgrades, the newly upgraded device will stay in suspended mode with the error peer OS too old. So when you go to start the first OS upgrade on the second HA device, you will lose network connectivity until the upgrade is completed and the first device is moved out of suspended mode and into passive mode and HA capabilities resume functioning.
  

2. Pre-upgrade checklist

  • Review release notes.
  • Do not schedule Panorama and firewall upgrades at the same time.
  • Upgrade Panorama first, wait at least 24 hours and then upgrade the firewall.
  • A pro-active support case may be desirable for some critical environments, Please review this article: When to request a Support Event
  • Upgrade should be carried out during non-business hours to minimize impact.
  • Allocate sufficient time in the change window for upgrade, troubleshooting and possible downgrade procedures. It may take up to 2-3 hours to upgrade a slower/older system, depending on config. Multiply if upgrading across multiple versions.
  • Identify business contacts who will help verify application and network functionalities after the upgrade.
  • Back up configuration and device state before upgrade.
    2016-09-06_upgrade1.png
    • Device > Setup > Operations > Save Named Configuration Snapshot
    • Device > Setup > Operations > Export Named Configuration Snapshot
    • Device > Setup > Operations > Export Device State
      2016-09-06_ugrade2.png
    • Device > Support > Generate Tech Support File
  • Document any non-standard settings that should be applied post-upgrade
    • Disable tcp state checking
    • Non-syn tcp reject
    • Any debug setting if existing (not recommended)
  • Stage/Download necessary PAN-OS images ahead of time. You will need both the base image and the latest maintenance release. The base image should be installed but not rebooted.  In this case, we need to download the following versions
    • 7.0.18 (it is recommended to bring your current Feature release to the latest recommended maintenance release before preceding)
    • 7.1.0 (base image) (NOTE: If you are on a maintenance release version earlier than 7.0.6, you must install 7.0.6 before 7.1.0 will show up on your software download page)
    • 7.1.14 
    • 8.0.0 for firewalls, 8.0.2 for Panorama (base image) (NOTE: the 8.0 base image and maintenance versions will not become visible in the download section until you have a version of 7.1 installed)
    • 8.0.6-h3
  • Following the PAN-OS upgrade, you may need to upgrade associated software (such as Global Protect agent or User ID agent)
    • See the Associated Software Versions chart in the release notes
  • (Optional but recommended) Arrange for Out-of-Band access (Console access) to the firewall if possible. This is to help recover from any unexpected situations where we lose connectivity to the firewall after upgrade.

3. Panorama upgrade procedure

  • Make sure no policy or configuration changes are being made by acquiring a config lock
  • 2016-09-06_upgrade3.png
  • Click on padlock icon on upper right hand corner of GUI
  • If there are any locks, please remove the locks or talk with the admin who placed the lock in place, and remove or commit..
  • Clear or complete any pending commit job making a commit to panorama before starting the upgrade
  • (Optional but recommended) Post-upgrade failover testing:
  • Suspend Secondary Panorama to fail connection back to Primary Panorama to make sure failover still works after upgrade.
    CLI:
    > request high-availability state suspend
    GUI:
    Device > High Availability > Operations >  click Suspend local device. 
  • Verify suspend status on Secondary Panorama
  • Verify all firewalls are connected to Primary Panorama
  • Re-enable Secondary Panorama and High Availability function 
    CLI:
    > request high-availability state functional
    GUI:
    Device > High Availability > Operations > click Make local device functional. 
  • Primary Panorama Upgrade procedure:
    1. Disable Pre-emption if enabled.
      (Disable preemption on High Availability settings to avoid unexpected failovers. Disabling preempt configuration change must be committed on both peers. Once completed, re-enabling must be committed on both peers).
    2. Go to Device > High Availability > Election Settings and uncheck Preemptive.  Then, commit the change:
      2016-09-06_upgrade4.png
    3. Suspend Primary Panorama to make Secondary Panorama as active
      From Primary Panorama, suspend High Availability function:
      CLI:
      > request high-availability state suspend

      GUI:
      Device > High Availability > Operations > click on Suspend local device. 
      2016-08-31_ha3.png
    4. Verify suspend status on Primary/passive Panorama.
    5. Verify all firewalls are connected to Secondary/active Panorama.
    6. On the Primary Panorama, download, install and reboot 7.0.18
    7. Download and install 7.1.0 (base version).
    8. Download and install 7.1.14, and reboot to complete the upgrade.
    9. Save/export tech support and Device state and save named device config snapshots (this is in case downgrade is needed).
    10. Download 8.0.2 (base version) (Recommended) Install the 8.0 base image and reboot before you install the target maintenance release.
    11. Download and install 8.0.6-h3, and reboot to complete the upgrade.
    12. Save/export tech support and Device state and save named device config snapshots (this is in case downgrade is needed).
    13. Re-enable Pre-emption, if so desired.
    14. This concludes upgrade on Primary Panorama.
  • Secondary Panorama Upgrade procedure
    • Download, install, and reboot 7.0.18
    • Download and install 7.1.0 (base version)
    • Download and install 7.1.14, and reboot to complete the upgrade.
    • Save/export tech support and Device state and save named device config snapshots (this is in case downgrade is needed)
    • Download 8.0.2 (base version) (Recommended) Install the 8.0 base image and reboot before you install the target maintenance release..
    • Download and install 8.0.6-h3, and reboot to complete the upgrade.
    • Save/export tech support and Device state and save named device config snapshots (this is in case downgrade is needed)
    • This concludes upgrade on Secondary Panorama
    • Backup config and device state files just in case:
      https://live.paloaltonetworks.com/t5/Configuration-Articles/Back-Up-Configuration-and-Device-State-f...
(Optional but recommended) Post-upgrade verification
  • Verify connectivity between Panorama and Firewalls. If something is not working, skip to troubleshooting section
    (For example, check if Panorama is receiving logs with correct time stamp from firewalls after upgrade is completed)
  • Test commit-all operations to managed devices, and verify new changes are applied as expected locally on the devices.



4. Firewall upgrade procedure (HA)

It is recommended to upgrade the Primary firewall first and then upgrade the Secondary firewall.  This is done for two reasons:
1.) Ensure that HA failover is functioning properly and 2.) Ensure that the passive firewall is functioning properly and is able to pass traffic without issues.
  • Disable Pre-emption if enabled. Disable preemption on High Availability settings to avoid unexpected failovers. Disabling preempt configuration change must be committed on both peers. Likewise, once completed, re-enabling must be committed on both peers.
    To disable:     Go to Device > High Availability >General > Election Settings <hit edit> and uncheck Preemptive.  Then, perform a commit. 
    2016-09-08_vpc6.png
    NOTE:     This procedure relies on the administrator having foreseen access to their devices at all times, either by being local or having OOB connectivity to the management network which is best practice when upgrading the firewall. In the case where you do not have the option of achieving either, it is a good idea to change the procedure slightly to ensure you dont lose connectivity at the cost of having a less rigid upgrade path.

    Having the preempt enabled will require you to keep this config change in mind during the whole process as it could unexpectedly switch over the active membership while upgrading.
  • Primary firewall Upgrade procedure
    1. On Primary firewall, Suspend Primary firewall to make Secondary firewall active
      CLI
      > request high-availability state suspend
      GUI
      Device > High Availability > Operations > click Suspend local device. 
      NOTE: This will cause an HA failover. We recommend doing this first to verify the HA functionality is working before initiating the upgrade.  Production traffic is now going through the Secondary firewall which is now active.  
    2. Ask your business owners to verify all applications are working on the network.  If there is a problem, skip to troubleshooting section. If there is any problem, fix it before proceeding with upgrade.
    3. Upgrade Primary firewall. You can do this by either directly downloading and installing software onto the firewall itself or via Panorama Device Deployment > Software option.
    4. Download, install and reboot 7.0.18
    5. Download and install 7.1.0 (base version).
    6. Download and install 7.1.14, and reboot to complete the upgrade.
    7. Save/export tech support and Device state and save named device config snapshots (this is in case downgrade is needed).
    8. Download 8.0.0 (base version) (Recommended) Install the 8.0 base image and reboot before you install the target maintenance release..
    9. Download and install 8.0.6-h3, and reboot to complete the upgrade.
    10. On the Primary firewall, verify auto commit completes successfully (FIN OK) by running the command before proceeding to the next step: 
      > show jobs all 
      Run the following command to make Primary firewall functional again: 
      > request high-availability state functional
    11. This concludes upgrade on the Primary firewall.
  • Secondary firewall upgrade procedure:
    1. Suspend Secondary firewall to make Primary firewall active.
      From Secondary firewall, suspend High Availability function
      CLI:
      > request high-availability state suspend

      GUI:
      Device > High Availability > Operations > click Suspend local device. 

      Note: This will cause an HA failover. Production traffic is now going through Primary firewall with new software installed.
    2. Ask your business owners to verify all applications are working on the network. If there is a problem, skip to troubleshooting section. If there is any problem, fix it before proceeding with upgrade.
    3. Upgrade Secondary firewall. You can do this by either directly downloading and installing software onto the firewall itself or via Panorama Device Deployment > Software option
    4. Download, install and reboot 7.0.18
    5. Download and install 7.1.0 (base version)
    6. Download and install 7.1.14. reboot to complete the install
    7. Save/export tech support and Device state and save named device config snapshots (this is in case downgrade is needed)
    8. Download 8.0.0 (base version) (Recommended) Install the 8.0 base image and reboot before you install the target maintenance release.
    9. Download and install 8.0.6-h3. reboot to complete the install
    10. Verify auto commit completes successfully (FIN OK) by running the command before proceeding to the next step: 
      > show jobs all 

      Run the following command to make Secondary firewall functional again: 
      > request high-availability state functional
    11. This concludes upgrade on the Secondary firewall
  • (Optional but recommended) Arrange for Out-of-Band access (Console access) to the firewall if possible. 
    This is again to help recover from any unexpected situation where we are unable to login to the firewall.
  • Backup config and device state files just in case
  • Make sure no policy or configuration changes are being made by acquiring a config lock
    • Click on padlock icon on upper right hand corner of GUI
  • Make sure no pending commit jobs on firewall
  • (Optional but recommended) Post-upgrade verification
    • Now that both Primary and Secondary firewalls are both running the new software, it’s a good idea to test failover functionality again.
    • Run the following comma to suspend Primary firewall to fail traffic to the Secondary firewall
      > request high-availability state suspend
    • Ask your business owners to verify all applications are working on the network through the Secondary firewall.  If there is a problem, skip to troubleshooting section
    • Run the following CLI command to make Primary firewall functional again: 
      > request high-availability state functional
    • Repeat the process to verify traffic works fine through Primary firewall (suspend the Secondary firewall, test functionality on Primary firewall, then re-enable Secondary firewall)
    • This concludes failover test
  • (Optional but recommended) Enable preemption if it was disabled due to upgrade
    • Re-enabling preempt configuration change must be committed on both Likewise, once completed, re-enabling must be committed on both peers.
    • Go to Device > High Availability > Election Settings and check Preemptive.  Then, perform a commit.  
      2016-09-08_vpc6.png
  • This completes upgrade on the HA pair.

5. Post-upgrade checklist

The following Post-Implementation Activities should be performed prior to the change window end time. Performing these Post-Implementation Activities prior to the change window end time allows time to complete any potential corrective action that might be required after performing these activities.
  • Compare Post-Implementation results with Pre-Implementation results
    • Reapply the non-persistence settings from the pre-checklist
    • Check system logs for any unexpected errors
    • Check traffic logs for any traffic that’s unexpectedly allowed or denied
    • If applicable, verify connectivity and network functionality between firewall and panorama, specifically make sure log forwarding from firewall to Panorama is working. Also, validate changes will commit between Panorama and the managed devices.
  • Check system reports and incidents for any relevant outages.
    • Monitor helpdesk tickets post upgrade, this may take 24 to 48 hours
  • Let administrators know about the completion of the change.
    • Contact all stake holders to communicate any change IDs, describe change activity results, and verify that there are no relevant network alarms, incidents, or outages.
    • Monitor the appliance for any anomalies.

6. Troubleshooting resources


7. Downgrade procedure

If the issue cannot be resolved within the allotted change window, you should revert all changes.
    • Verify 7.1.0 (base image version) is still present on the system
    • Verify and install 7.1.14. reboot to complete the install. When prompted, use 7.1.14 snapshot file saved during the upgrade.
    • Verify 7.0.1 (base version) is still on the system
    • Download and install 7.0.18, and reboot to complete the downgrade. When asked, use 7.0.18 snapshot file saved before the upgrade.
      Note: After the Secondary firewall is rebooted, the CLI prompt should show non-functional.
    • On the prrimary firewall, verify auto commit completes successfully (FIN OK) by running the command before proceeding to the next step: 
      > show jobs all 
      Run the following command to make it functional again: 
      > request high-availability state functional
    • This concludes downgrade on the Primary firewall.
  • Downgrade the Primary firewall
    • Suspend the prrimary firewall to fail traffic to Secondary
    • On the Primary device, suspend the unit from the CLI. Run the command:
      > request high-availability state suspend
  • Save the following files for analysis:
    • Save and export Tech support and device state from both active and passive firewalls
    • All core dump files if there is any
    • Packet capture of problem traffic, if observed
  • Assuming the firewall is currently running 8.0.6-h3 already and traffic is currently going through the Primary firewall, follow the same upgrade process but in reverse order.
  • Downgrade the Secondary firewall
    • Verify 7.1.0 (base image version) is still present on the system
    • Verify 7.1.14 is still present on the system and install, reboot to complete the install. When prompted, use 7.1.14 snapshot file saved during the upgrade
    • Verify 7.0.1 (base version) is still on the system
    • Download and install 7.0.18. reboot to complete the install. When prompted, use 7.0.18 snapshot file saved before the upgrade.
      Note: After the Secondary firewall is rebooted, the CLI prompt should show non-functional.
    • On the Secondary firewall, verify auto commit completes successfully (FIN OK) by running the CLI command before proceeding to the next step: 
      > show jobs all 
      Run the following command to make it functional again: 
      > request high-availability state functional
    • This concludes downgrade on the Secondary firewall
  • (Optional but recommended) Enable preemption if it was disabled due to upgrade.
    • Re-enabling preempt configuration change must be committed on both Likewise, once completed, re-enabling must be committed on both peers.
    • Go to Device > High Availability > Election Settings and check Preemptive.  Then, perform a commit.  
      2016-09-08_vpc6.png
  • (Optional but recommended) Ask your business owners to verify all applications are working on the network.  If there is a problem, skip to troubleshooting section.
  • Upload all files to the Palo Alto Networks proactive support case for troubleshooting later.
  • This concludes the downgrade process.
at

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)