The Journey of a Cortex XSOAR Playbook: Building the Engine

The worst upgrade failures I have watched in the field did not happen during the install. They happened at minute thirty of a maintenance window, when something that should have been checked beforehand finally surfaced. A managed firewall pointed at a Panorama on an older release, plus a candidate config nobody committed. The install completed just fine, but the homework was the problem.

That is the difference between a script that automates the steps and an engine your team will trust at 2 AM on a Saturday. A script does what you tell it, and with no exceptions. An engine checks its own preconditions, knows the state of the devices it is about to touch, and refuses to start when something is wrong. This blog is about building that engine.

In Designing Your First Playbook we laid out the blueprint: the manual process mapped to XSOAR constructs, the sub-playbooks we need, and the three-phase flow that becomes the spine of the implementation. Phase 1 initialization, Phase 2 software path calculation, Phase 3 the upgrade loop. Now is the time we build. This post is about the how: how XSOAR's context data, conditional branching, and sub-playbook patterns come together when you are in the trenches.

Phase 1: Initialization and the gates before the engine starts

Phase 1 is where the playbook does its homework. It first establishes context: it associates the device indicator to the incident (so the War Room has full device context), sets the Panorama integration instance the rest of the playbook routes through, and links to a parent incident if the upgrade is part of a batch. None of that is glamorous, but every downstream sub-playbook depends on it. One piece of good news is that Palo Alto Networks has already helped us out with a lot of these processes with the PAN-OS Python SDK, and we're going to be re-using some of their concepts as we can run their SDK as a Docker container in Cortex XSOAR. 

Then the gates fire. Four checks belong here, and each one prevents a specific class of 2 AM failure.

• Panorama version compatibility. Panorama must run a version equal to or greater than the firewalls it manages. Upgrading a managed firewall past Panorama is unsupported and can break pushes, template sync, and log collection, sometimes silently. The playbook queries Panorama's version, compares it to the target, and treats a mismatch as a hard stop. Not a warning. If Panorama is on 11.0 and the operator requests 12.1, the playbook surfaces a clear error and stops. Better to fail at incident creation than at minute thirty of the window. Order matters across the whole management plane, too: Panorama first, then any Dedicated Log Collectors, then the managed firewalls.
• Maintenance window enforcement. The playbook opens with Capture Maintenance Window, a survey that prompts for the planned start and end if an upstream system did not pre-populate them. EnforceMaintenanceWindow then runs as a pre-flight gate with three modes: hard_fail (halt outside the window), warn (log and proceed), and check_only (status only). For ServiceNow or Jira shops, the approved window can come straight from the change ticket.
• Change freeze awareness. Separate from the window, most enterprises freeze changes during peak periods (quarter-end, regulatory cycles, retail holidays). The playbook checks a freeze flag, stored as an XSOAR list entry or queried from the ITSM integration, and halts with a clear reason if a freeze is active. Five minutes to wire in, and it prevents the incident nobody wants to explain to leadership.
• Lab validation gate. CURB (Change and Upgrade Review Board) boards want evidence the target version was validated in a controlled environment first. The playbook checks an XSOAR list for a lab-validated flag against the requested PAN-OS version and halts if it is missing. The lab validation itself can be a lightweight workflow that runs the same upgrade playbook against lab devices and sets the flag on success. This is the gate that turns the CURB review from a gatekeeping exercise into a formality.

These four gates produce no work, but they produce something better: decisions. Pass all four and the playbook moves forward. Fail any one and it halts cleanly, with a structured reason the operator can act on. Three device-level checks run next: CheckLastCommitState flags pending commits on either node, CheckLicenseExpiry flags imminent license expirations against a configurable threshold (default 30 days), and HA sync status surfaces from the Get HA Pair Status output below.

Fail early, fail clearly.

Phase 2: Calculating the upgrade path

PAN-OS upgrades are not always a straight line from A to B. Major jumps may require intermediate versions, and even minor upgrades can have hardware-specific constraints. Two recent features change the math. Simplified Software Upgrade (PAN-OS 10.2) removes the requirement to install the base image of each minor version before the maintenance release. Skip Software Version Upgrade (PAN-OS 11.0) lets you skip an entire feature release. Between them, the number of hops your playbook executes is smaller than it used to be, and the path calculation must understand both to take advantage of them.

The Filter and Select Available Software Images sub-playbook, backed by the FilterAvailableSoftwareImages automation, does the resolution. It queries the device for available images, filters against the current and target versions, and calculates the path. A request for 9.1.4 to 12.1.3-h3 might resolve to [10.1.0, 11.2.0, 12.1.3-h3], three hops the loop executes in sequence, or collapse to [12.1.3-h3] directly on a device that supports Skip Upgrade. If no valid path exists, the playbook stops before any survey or gate. I learned that one the hard way: a target with no path used to sail through the CAB gate and die later at the upgrade call. Now the guard fires about 45 seconds in, names the target and the running version, and points intentional downgrades at the rollback playbook. This is also where you enforce the approved-version matrix: a path with an unapproved intermediate is a failed validation, not a successful calculation.

HA state detection: the sub-playbook everything depends on

Phase 3 is the upgrade loop, and the loop depends on one sub-playbook being right before anything else can work. Get HA Pair Status is that foundation. Before upgrading anything, we need to know which firewall is active, which is passive, and whether the pair is healthy. Every well-designed sub-playbook starts with a clear contract:

Notice the simplicity. The sub-playbook takes a single target and returns the active and passive hostnames. It does not need to know why we need them, whether we are upgrading, running assurance, or just checking health. Separation of concerns in action.

The task flow

The sub-playbook follows a five-step sequence. It calls pan-os-platform-get-ha-state against the target, routed through Panorama via the using parameter, so we need no direct connectivity to each firewall. That single choice has saved deployments where security policy restricts direct firewall management. It then checks whether the PANOS.HAState.peer key is populated: standalone devices exit early; if a peer exists, it runs the same command against the peer. Finally, two tasks run in parallel: one filters PANOS.HAState for the device where active is true and sets ActiveDevice, the other filters for active false and status passive and sets PassiveDevice. The passive task uses SetAndHandleEmpty rather than Set, so a standalone device leaves PassiveDevice empty instead of writing a null.

Those last two run in parallel because both depend on the peer query, not on each other. That is not just an optimization. It is a visual signal that the operations are independent, and that clarity matters when you are troubleshooting at 2 AM.

Context data is the glue. When pan-os-platform-get-ha-state runs, it writes to PANOS.HAState, and later tasks reference that path. The Set Active Peer task reads it declaratively: from the PANOS.HAState array, filtered where active is true, take the hostid. Filter and extract, rather than write procedural logic, is central to XSOAR. It took me a while to stop thinking like a programmer and start thinking like a workflow designer. Once it clicks, most of your playbook logic can be expressed this way.

A script does what you tell it. An engine checks its own preconditions and refuses to start when something is wrong.

Phase 3: The HA-aware upgrade loop

With HA state detection in place, the Device Upgrade Loop orchestrates the actual upgrade. It is the most complex component in the design, but the complexity is intentional and managed through decomposition.

Why a loop? PAN-OS upgrades sometimes require multiple hops. The parent calls the Loop once, and the Loop carries the multi-hop logic internally via a "Have we reached the target version?" condition near the end. If the running version does not match the target, the Loop re-enters from the top with fresh HA state and processes the next hop. For a path like [11.0.0, 11.1.0, 11.1.4], it self-iterates three times, so every hop gets the same passive-first treatment with fresh state and the parent never has to count hops.

Each iteration follows an HA-aware sequence:

• HA discovery. Call Get HA Pair Status again, even if a previous iteration determined it, because that iteration may have triggered a failover.
• Pre-flight snapshot. Pre-Upgrade Snapshots captures both members' operational state via pan-os-platform-get-system-info. It also stamps each member's pre-change running version on the incident, which matters more than it looks: that is the exact value the rollback reads if the upgrade has to be backed out.
• Passive-first upgrade. The Loop hands the passive device to one sub-playbook, Single Device Upgrade, which runs the whole sequence for one member (save state, resolve the path including any base image, download, install, reboot, wait for the device to return, verify the version) and wraps the Download Software and Install Software mechanics underneath. That one wrapper is why the forward upgrade and the backout share the same machinery.
• Failover and second upgrade. Once the passive is healthy, a controlled failover via pan-os-platform-update-ha-state shifts traffic to the freshly upgraded node, and the original active gets the same treatment.
• Post-flight and topology restore. Post-Upgrade Snapshots captures both members again, then the Loop optionally restores the original active/passive arrangement, controlled by the reset_ha_topology input.

Why passive first? If the upgrade fails, only the passive node is affected. The active keeps forwarding traffic. That is the kind of HA awareness that separates automation you run in a maintenance window from automation you trust during business hours.

Failback, and how it differs from rollback

Failback is one of the most underestimated parts of HA upgrades and the root cause of more than a few escalations. The active/passive arrangement is often deliberate, driven by routing policy, asymmetric flows, ECMP hashing, or regulatory requirements. As the pack stands, failback runs inline in the Loop via reset_ha_topology (suspend the active, unsuspend the passive to flip back).

Be precise about the vocabulary, because failback and rollback get conflated constantly. Failback restores which member is active. Rollback backs out the software change entirely. Failback lives inline in the loop; backing out a failed upgrade is a separate, dedicated playbook that our blog in Part 5 walks through.

A gotcha that has burned more than one window: HA preemption. If it is enabled, the higher-priority device reclaims the active role the moment it comes back, which can flip the pair mid-upgrade and fight the controlled sequence you just ran. Check the preempt setting before the loop touches anything, and either disable it for the window or validate around the automatic failback.

Error handling and human checkpoints

Production-grade automation does not pretend failures will not happen. A download failure stops the run rather than installing an image that is not present. An install timeout is flagged if the device does not come back. After each reboot, the Loop verifies the pair re-synchronizes, and a device that reboots but fails to rejoin is a critical failure. Named confirmation gates show up in the War Room at the moments that matter. Mature environments remove them; teams building confidence keep them.

Those gates are no longer confined to the War Room. The pack ships a notification layer that fires on the moments that matter (upgrade start, each member reaching target, the failover, a step failure, the readiness abort), so the operator does not have to sit and watch the incident. It stays dormant until you configure the settings list, and Part 6 of our series will cover how those same events feed the change record. Alongside the snapshots, Pre-Upgrade and Post-Upgrade Reachability Checks confirm traffic is actually flowing, not just that the firewall came back up.

One more design point: every PAN-OS command routes through Panorama. XSOAR needs API credentials only for the Panorama instance, not for every firewall, so the per-firewall key sprawl operators dread is not part of the design, and it works with the management-network security model rather than against it.

Context flow: shared versus separate

How context flows between sub-playbooks has two modes. Separate context gives a sub-playbook its own namespace: nothing leaks in or out except through declared inputs and outputs, so it tests cleanly in isolation. Shared context is the opposite: the sub-playbook reads and writes the parent's context directly.

The rule I follow is shared context for sub-playbooks bound to one workflow, separate context for anything meant for reuse. The HA detection and Loop pieces run shared, because they write keys the parent reads back (OriginalActive, OriginalPassive, ActiveDevice, PassiveDevice), and forcing separate context would add an output bridge for every key without changing behavior. Take Snapshot is a candidate for separate context the next time I extract it.

Why this design works

The architecture is not decoration. Every choice is there because someone lived through what happens when it is not. Modularity lets a team build in parallel and onboard a new engineer without three weeks of ramp. Passive-first sequencing turns a maintenance-window change into one the business barely notices. And the patterns travel: the HA detection logic works any time you need the state of an HA pair, and the context-flow patterns apply to any multi-step automation. You are not just learning how to upgrade firewalls. You are learning how to think about automation design.

There is a strategic version of that worth stating on purpose. This is a SecOps automation platform doing NetOps work, and that is the wedge, not an accident. A network team that adopts XSOAR to take the pain out of firewall upgrades has, as a side effect, stood up the same platform the SOC runs on: the same incident model, War Room, integration framework, RBAC, and audit trail. I have watched that on-ramp turn a single NetOps use case into an enterprise SecOps footprint.

What's next

We have built the two core engines, HA state detection and the HA-aware upgrade loop, and seen how context flows and how passive-first keeps the network alive. What they are missing is assurance.

In Part 4 we bring it together with the upgrade assurance workflow: the pre-flight/post-flight snapshot comparison, the layered verdict, and what happens when that verdict comes back as something other than PASS and the automated rollback takes over. 

Part 5 covers the roadmap and advanced patterns, including the Automated Rollback that now ships and reuses the Single Device Upgrade machinery. 

And Part 6 closes with governance, turning everything the playbook records into a change-evidence package a CAB or CURB board signs off on without an argument.