As AI accelerates vulnerability discovery, Cisco IT has cut fleet operating system (OS) upgrade times from months to days, shrinking the window attackers have to exploit a known flaw without sacrificing uptime.
By integrating upgrade decisions into our network inventory source of truth and enabling our Catalyst Center, Nexus Dashboard, and Meraki dashboard implementations to carry out these decisions, we are now able to apply hardened releases to over 95% of our Cisco IT-managed devices within our service-level objective of 7 days from availability.
Our challenge: advisories outpaced remediation #
Frontier AI models are discovering vulnerabilities faster than the traditional disclosure-and-patch model can absorb, and the gap between disclosure and exploitation is shrinking from weeks to minutes. The Cisco Product Security Incident Response Team (PSIRT) responded with security-hardening releases on a regular cadence each month, with seven days of advance notice for these planned releases. Russ Smoak’s (VP, Information Security) “Strengthening the Foundation” blog covers the details. Building those fixes still takes skilled human judgment, as Jonathan Felten (Vice President, Engineering) describes in his “Cryptography Rarely Fails Because of the Math” blog. Here we cover the final stretch: deploying them.
Cisco IT runs the same hardware and software as our customers and our old workflow treated each advisory as an isolated investigation. We spent valuable time parsing inventory sources of truth, backup configs, and live device states just to answer one question: Are we vulnerable?
At our scale, that approach broke down:
- Volume and diversity: With more than 60,000 devices including IOS-XE, IOS-XR, NX-OS, Application Centric Infrastructure (ACI), and others, each OS type needed its own assessment approach.
- Version sprawl: Advisories arrived faster than we could remediate within our service-level objectives (SLOs), which can be as short as three to seven days for a critical issue. Each OS type fragmented into many versions.
- Manual execution: Device-by-device scripts and hand-authored change requests tied remediation speed to engineer availability, not patch availability.
How AI-discovered vulnerabilities change the landscape #
Hardened releases include many fixes and do not detail each bug, but must still be deployed. The question is no longer “Are we vulnerable?” but rather, “How fast can we upgrade safely?” Leaders weighed the tradeoff: a brief, planned disruption to upgrade against an unplanned security event that could cost far more. Security and network leadership agreed early that the second risk is more impactful, and they committed to upgrading by default. The challenge then became delivery.
Our approach: optimizing OS image management #
Our approach spans four layers: the source of truth is leveraged for decisions, Cisco controllers execute and verify, logs of the process are recorded in Splunk Cloud Platform, and AI agents cover the gaps.
After a human sets the golden image, automation performs the upgrade:
- Maintenance and eligibility checks
- Change record creation, conflict resolution, and approvals
- Image distribution
- Pre-checks
- Upgrade device
If any of these steps fail, the device upgrade will not proceed. Human judgment stays where it matters most: overseeing the process and resolving issues as they arise. Fewer engineers overseeing each of our components can now effectively scale to oversee thousands of changes occurring over a few days or even hours.
The key requirements that allowed us to implement this strategy:
- Accurate source of truth: We must operate from a reliable, known state to make proper decisions. In Cisco IT’s case, we leveraged our configuration management database (CMDB) because it contains our inventory with additional device attribution, and can be used to decide what gets upgraded, when it will be upgraded, and in what order.
- Change management: Upgrades continue to follow our established change management standards, including tracking every change and identifying dependencies and conflicts. Automation does not replace approvals. It helps upgrades move through our existing change process more consistently. Changes that meet our documented criteria for a standard change (low risk, minimal impact and repeatable) follow a pre-approved path defined in our change management policy, so they don’t need case-by-case review.
- Assurance capabilities: Since automation drives this process, we must have confidence in reporting the state of the network and the progress of upgrades. Automation can make decisions based on topology (as defined in the single source of truth) to schedule upgrades to avoid outages. Upgrade progress is logged toSplunk Cloud Platform , where we use this data to create dashboards that identify compliance issues blocking device upgrades. This enables our team to quickly resolve blockers and allow upgrades to proceed.
- Simplify success criteria: We had to evolve our mindset about how we measure success of upgrades. Success is more heavily weighted to whether the device upgraded successfully vs operational issues that may arise. If we encounter an operational issue, it is addressed through separate incidents and escalated, if necessary, instead of being treated as an upgrade failure.
Once these decisions have been made, execution becomes a product capability, not a custom project. Software Image Management (SWIM), available to customers in Catalyst Center, distributes and activates images across thousands of devices with built-in pre-checks, post-checks, and rollback. Nexus Dashboard and Meraki dashboard have similar capabilities in their domains.
Where a device is not yet managed by a controller, automation and agentic operations fill the gap. For example, the Cisco Network IT team has leveraged an AI operations agent to upgrade 200 non-Catalyst Center managed devices within three days of the upgrade bundle becoming available. This AI operations agent also accelerated deployment of upgrades to Identity Services Engine (ISE) and assists with day-to-day network operations.
Matching speed to blast radius #
The approach is the same everywhere – how fast we move depends on known impacts.
At the edge, we go fast. Taking a single switch, access point, or even half a building offline is acceptable, especially outside business hours. Topology rules keep the other half online. Where business-critical teams operate around the clock, we coordinate change windows with them directly. Everywhere else, upgrades run autonomously. Using this strategy has enabled us to upgrade more than 30,000 devices within their service-level objectives.
In the data center, we move quickly, but don’t rush. Each Application Centric Infrastructure (ACI) fabric change affects far more workloads, so we take a more cautious yet still aggressive approach. In the past, upgrading 1200 ACI fabric switches took six months. It now takes five weeks, a fivefold reduction. Three changes made the difference:
- Better code quality: High-quality releases from Cisco engineering let us upgrade confidently during business hours.
- Faster approvals: Change requests better account for non-impacting changes to network configuration items, removing most approval delays.
- Automation:Application Policy Infrastructure Controller (APIC) upgrades switches in staggered groups so redundant pairs never go offline In-house validation replaces manual checks, comparing endpoints, protocol neighbors, and interfaces against a pre-upgrade baseline before the next group starts. Batches now reach 34 switches at a time.
Outcomes and what we learned #
The measure that matters most is the last mile: how long a device stays exposed after a fix is available. Our mean time to remediate (MTTR) has dropped from months to days, and hardened release with the exception of NX-OS has been applied within our service-level objectives. Being customer zero is ingrained in our culture at Cisco – we deploy and operationalize Cisco products at a global scale. The success of this customer zero deployment means upgrades aren’t just tested in a lab, they are deployed to over 60,000 devices including:
- IOS-XE devices, including cloud-managed switches
- IOS-XR devices
- NX-OS devices (includes ACI fabrics and standalone)
- Access points: Catalyst & Meraki cloud-managed
- Meraki Hybrid Worker Bundles
- Adaptive Security Appliance (ASA) and Firepower Threat Defense (FTD) Firewalls
- Multiple global Identity Services Engine (ISE) clusters
- Nexus Dashboard controllers
- SD-WAN control components
- Webex devices
This was accomplished with minimal operational impacts caused by hardened release upgrades.
Most of the gain came from streamlining change management processes and reducing wait times. Additionally, Cisco controllers scale the implementation by deploying upgrades in parallel instead of device by device. Engineers are now able to focus more of their time on exceptions. Every change is traceable in Splunk Cloud Platform, and fewer one-off OS variants mean the next release lands even faster.
If you are facing the same pressure, here is what worked for us:
- Agree on priority first: Treat hardened releases as must-deploy work.
- Get the CMDB right: Producing an accurate inventory is a prerequisite, then layer OS standards, change rules, and topology rules on top of it.
- Use the product capability you already have: Start with SWIM in Catalyst Center before writing custom automation.
- Pre-approve what you repeat: Move well-understood, low-risk upgrades to standard changes to allow for automated approvals. Utilize AI operations agents with human-approval guardrails to handle processes not yet automated.
- Match speed to blast radius: Go fast where failure is contained. Where it is not, move quickly, but don’t rush.
- Use the seven-day notice: Pre-stage upgrade bundles on controllers and devices, perform lab validation, and automate approvals when PSIRT publishes the technology list.
What’s next #
Cisco PSIRT “Hardened Release” advisories are released on a schedule, but exploits are not. Cisco IT and security teams are now working together on the harder case: a known, severe vulnerability that cannot wait weeks for an upgrade. Our goal is to mitigate and remediate within hours to days.
- Find and contain: Our Attack Surface Management program and platform will surface critical exposures quickly. Until a patch lands, we will contain each exposure with the most precise control for that asset, such asCisco Live Protect on network devices, endpoint detection and response (EDR) on hosts, or targeted network policy, rather than utilizing blunt instruments such network isolation.
- Make room to patch: We are removing hurdles to change windows and establishing standing windows during which teams can upgrade or patch. Routine upgrades move toward an ‘inform’ model, where infrastructure and application teams get visibility into what is changing instead of approving each change when no impact is expected.
- Design for patching: Every application has different resilience needs. We are defining Cisco IT-validated reference designs matched to each level of criticality. Teams that implement these designs know their applications will withstand routine patching.
AI has collapsed the time between discovery and exploitation. Cisco IT’s evolving practices are designed to keep pace by minimizing change hurdles, automating upgrades through business processes on Cisco controllers, and integrating human governed AI agents into our engineering workflows. That is digital resilience in practice – patching near the speed of discovery while the business keeps running with minimal disruption.
Moving this fast will surface new friction before it removes the old. We plan to share what we learn as we go.