{"description":"Standing quarterly operator workflow that verifies alternate storage, alternate processing, and diverse telecommunications capability (UC-BCDR-04) and validates the safe-mode, alternate-communications, and alternate-security-mechanism design configured on critical systems (UC-BCDR-11). Each quarterly instance runs against the EXISTING UC-BCDR-04 alternate-capability Control item in the Control library (frequency quarterly), with the UC-BCDR-11 degraded-mode Control item linked as the second in-scope control — it enriches the evidence trail on those existing controls, never creating a duplicate control. Self-originating: it consumes no upstream workflow handoff, reading the two governing Control items (control_id UC-BCDR-04 and UC-BCDR-11), the prior quarter's archived readiness instance, and the open corrective-action Issues carried forward on those controls; all operational reference data (backup schedule, alternate-site media/replication log, site records, telecom inventory, system design specs and configs) enters as PBC uploads on the verifying step, since none of it is item-typed. In scope: confirming that this alternate capability and degraded-mode design remain current, hazard-separated, secured to primary-site parity, and ready to assume operations within RTO. Out of scope: the live failover exercise that actually cuts over to the alternate site, run separately by the assure-line DR test workflow that consumes the readiness state this workflow maintains. Named deliverables: six verification memos (storage-site, processing-capability, telecom, safe-mode, alternate-communications, alternate-security), a readiness dashboard and a signed readiness summary, and a corrective-action register — every gap converts into an owned corrective-action Issue (issue_type: deficiency, source: self_assessment) linked to the impaired Control, with any recovery-time-impacting gap escalated immediately rather than held for end-of-cycle closure. Downstream handoff: the archived readiness summary and dashboard serve as the readiness-state package the assure-line DR test workflow reads (no terminal handoff node exists yet — see the closure step).","edges":[{"id":"e-verify-alternate-processing-capability-classify-readiness-disposition","label":"Confirmed ready","source":"verify-alternate-processing-capability","target":"classify-readiness-disposition","whenValue":"confirmed_ready"},{"id":"e-verify-alternate-processing-capability-escalate-processing-capability-gap","label":"At risk","source":"verify-alternate-processing-capability","target":"escalate-processing-capability-gap","whenValue":"activation_at_risk"},{"id":"e-escalate-processing-capability-gap-classify-readiness-disposition","source":"escalate-processing-capability-gap","target":"classify-readiness-disposition"},{"id":"e-classify-readiness-disposition-log-corrective-actions","label":"Gaps","source":"classify-readiness-disposition","target":"log-corrective-actions","whenValue":"gaps_identified"},{"id":"e-classify-readiness-disposition-close-and-archive","label":"Healthy","source":"classify-readiness-disposition","target":"close-and-archive","whenValue":"healthy"},{"id":"e-log-corrective-actions-close-and-archive","source":"log-corrective-actions","target":"close-and-archive"}],"isPublic":true,"metadata":{"capabilities":[],"controlVerbs":{},"controls":["UC-BCDR-04","UC-BCDR-11"],"department":"it","domains":["controls"],"library":{"aliases":[],"canonicalUrl":"https://workflow-library.com/all/?w=controls-resilience-failover-readiness-verification","contentDigest":"sha256:f38e4776db3dc7198958fcc6b8b422244265219259df3fd62ccf20545bd3db9b","prerequisites":{"status":"undeclared"},"provenance":[],"releaseId":"sha256:f38e4776db3dc7198958fcc6b8b422244265219259df3fd62ccf20545bd3db9b","schemaVersion":1,"sourceTemplateId":"workflow-library:controls-resilience-failover-readiness-verification"},"lineOfDefense":"operate","mappingStatus":"mapped","risks":[],"slug":"controls-resilience-failover-readiness-verification","source":"coworkcanvas-gallery","standards":["nist-800-53","nist-csf-2","iso-27001"],"teams":["it"]},"name":"Resilience & Failover Readiness Verification","nodes":[{"data":{"decisionField":"processing_capability_status","description":"Agent verifies the alternate processing site's separation from primary-site hazards, activation readiness within RTO, and facility security parity; human decides whether activation is confirmed ready or at risk","formData":{"fields":[{"key":"processing_capability_status","label":"Processing Capability Status","options":[{"label":"Confirmed ready","value":"confirmed_ready"},{"label":"Activation at risk","value":"activation_at_risk"}],"required":true,"type":"select"}],"resultType":"form","submittedAt":null,"values":{}},"instructions":"**Objective** — Decide whether the alternate processing capability is confirmed ready or activation-at-risk, resolving whether it is genuinely separated from primary-site hazards, can assume operations within the recovery time objective (RTO), and is secured to primary-site parity (UC-BCDR-04). Owned by the resilience lead.\n\n**Inputs**\n- The alternate processing site record — a PBC upload at this step (site record + runbook, PDF/DOCX): location, infrastructure dependencies (power grid, network path, flood or seismic zone), and shared-service footprint. There is no native Facility/Site item type, so the site enters as an upload.\n- The site's capacity figures, standby staffing plan, and activation runbook, with the runbook's last-reviewed date and the site's last capacity or activation test result (same upload).\n- The primary site's security baseline for a parity comparison — a PBC upload (baseline document); the individual safeguards it summarizes exist as Control items (control_category: physical | technical | administrative) and are queryable via query-data for the parity check.\n- The RTO for the systems that would fail over to this site — a PBC upload (BIA / recovery-objectives extract); no System/Asset item type carries an rto field.\n\n**Decision criteria**\n- Select **confirmed_ready** when ALL three hold: (a) genuine separation — the alternate site shares no single point of failure (power grid, network path, hazard zone, or shared service) with the primary; (b) activation within RTO — current capacity and standby staffing suffice and the activation runbook is current (reviewed within cadence) with a last-test result that met the RTO; (c) security parity — physical, environmental, and logical safeguards match the primary baseline.\n- Select **activation_at_risk** when ANY one dimension is deficient — for example shared-hazard exposure, insufficient capacity or staffing, a stale activation runbook, or a security-control gap. A single deficient dimension is enough; do not average across dimensions.\n\n**Record in AssureSwarm**\n- Submit the `processing_capability_status` SELECT (confirmed_ready | activation_at_risk).\n- In the step result, cite the separation, activation-within-RTO, and security-parity evidence behind the pick, naming the deficient dimension when at-risk. Attach the processing-capability assessment (document upload). Set the step's approver record to the decision owner or approver.\n\n**Exit criteria** — SELECT submitted with an evidence-referenced rationale; the processing-capability assessment attached; the unused branch is prunable so only the chosen path proceeds.","kind":"decision","label":"Verify alternate processing capability","performedBy":{"primitives":["coach-query-data","coach-document-upload"]}},"id":"verify-alternate-processing-capability"},{"data":{"description":"Agent packages the deficiency and an interim mitigation for immediate escalation; human confirms leadership is notified and the interim mitigation is in force","instructions":"**Objective** — Escalate an at-risk alternate processing capability immediately, with an interim mitigation in force, rather than holding it for end-of-cycle closure, since this capability underpins recovery for every dependent system.\n\n**Inputs**\n- The processing-capability assessment and the step result naming the deficient dimension: shared-hazard exposure, insufficient capacity or staffing, a stale runbook, or a security-control gap.\n- The recovery time objective and the list of systems that fail over to this site (the impact scope).\n- Resilience leadership and the accountable system owners (escalation recipients).\n\n**Procedure**\n1. Isolate the specific deficiency from the assessment and quantify its RTO impact — which systems could not be recovered within objective if the alternate site were needed now.\n2. Choose an interim mitigation proportional to the deficiency: temporary or expanded standby staffing, a compensating security control, an expedited runbook refresh, or a temporary secondary-site arrangement.\n3. Create an urgent corrective action as an Issue item capturing the deficiency, its RTO impact, the interim mitigation, an accountable owner, and a due date ahead of the next quarterly interval; link it to the UC-BCDR-04 Control item and the workflow anchor.\n4. Draft an escalation notice to resilience leadership and the affected system owners stating the deficiency, the interim mitigation now in force, and the residual risk until remediation completes.\n\n**Record in AssureSwarm**\n- Create the urgent corrective action as an Issue item (item create — issue_type: deficiency, severity: high | critical scaled to RTO impact, source: self_assessment, root_cause = the named deficient dimension, remediation_plan = the interim mitigation, issue_owner, identified_date, target_remediation_date before the next quarterly interval).\n- Link the Issue to the UC-BCDR-04 Control item it impairs and to the workflow anchor (items link); the processing-capability assessment is named in the Issue description, since step documents are not linkable targets.\n- Attach the escalation notice as a step document (DOCX/PDF) and flag the Issue for priority follow-up ahead of the standard cadence.\n\n**Exit criteria** — Corrective-action item created with a named owner and due date; the interim mitigation confirmed actually in force by leadership; escalation notice attached; resilience leadership confirms the escalation was received before the cycle's readiness disposition is judged.","label":"Escalate processing capability gap","performedBy":{"primitives":["coach-item-create","coach-items-link","coach-document-upload"]}},"id":"escalate-processing-capability-gap"},{"data":{"decisionField":"readiness_disposition","description":"Judge storage retrievability, telecom diversity, alternate safeguards, safe-mode and fallback tests in one complete readiness disposition.","formData":{"fields":[{"key":"readiness_disposition","label":"Readiness Disposition","options":[{"label":"Healthy, within tolerance","value":"healthy"},{"label":"Gaps identified","value":"gaps_identified"}],"required":true,"type":"select"}],"resultType":"form","submittedAt":null,"values":{}},"instructions":"**Objective** — Judge storage retrievability, telecom diversity, alternate safeguards, safe-mode and fallback tests in one complete readiness disposition.\n\n**Inputs**\n- The workflow anchor: the existing UC-BCDR-04 alternate-capability Control item (with UC-BCDR-11 linked), queryable in the Control library, plus the open corrective-action Issues carried forward on it from prior cycles (issue_type: deficiency, source: self_assessment).\n- The backup schedule — a PBC upload at this step (backup-tool schedule extract, CSV/XLSX): every scheduled backup set, its source system, cadence, and recovery point objective (RPO). There is no native BackupSet item type, so it enters as an upload, not a queryable item.\n- The alternate storage site's media and replication log — a PBC upload at this step (site media/replication log extract): what media and replicated volumes the site currently holds, by system and by date, plus rotation and retention windows. No native type; enters as an upload.\n- The RTO/RPO per in-scope critical system — a PBC upload at this step (BIA / recovery-objectives extract); no System/Asset item type carries these fields.\n- The prior-cycle storage verification memo (a step document on the prior quarter's archived instance) and any open storage corrective-action Issues carried forward.\n- The cycle context: routine quarterly cadence, a triggering change (new system added, site relocated, carrier changed), or a follow-up on a prior gap. This workflow is self-originating; it has no upstream workflow input.\n- The telecom circuit inventory — a PBC upload at this step (carrier circuit inventory, CSV/XLSX): every primary and alternate or diverse circuit, its carrier, physical path, conduit and last-mile facility, and last path-diversity verification date. There is no native Circuit/Asset item type, so it enters as an upload.\n- Priority-of-service records — a PBC upload (TSP provisioning records): carrier restoration priority (for example Telecommunications Service Priority), priority calling or dialing provisioning, and renewal dates, per in-scope circuit and critical communications line.\n- The prior-cycle telecom memo (a step document on the prior quarter's archived instance) and any lapsed-circuit finding Issues carried forward.\n- Each critical system's primary security safeguards (authentication, encryption, monitoring and logging) and their documented alternate mechanism — a PBC upload at this step (safeguard design spec); the safeguards themselves may also exist as Control items (control_category: technical | administrative) queryable via query-data. There is no native System/Asset item type, so per-system design enters as an upload.\n- The deployment and enablement status and last validation or test date for each alternate mechanism — a PBC upload (config export).\n- The protection level each primary safeguard provides, for an equivalence assessment.\n- The critical-systems register — a PBC upload at this step (which systems carry a safe-mode and alternate-communications design); there is no native System/Asset item type, so the register enters as an upload.\n- Each critical system's safe-mode design specification — a PBC upload (design spec): the trigger conditions and the restricted-function set safe mode drops to.\n- Each critical system's communications design — a PBC upload (design spec): the primary protocol and its documented alternate protocol or channel (secondary API endpoint, out-of-band messaging path, satellite or cellular fallback).\n- The live configuration for each critical system — a PBC upload (config export, for drift comparison); recent connectivity or heartbeat evidence on each alternate path — a PBC upload; and any non-production or staging instance available for a non-disruptive trigger or fallback test.\n- The other verification memos from this cycle: storage-site, processing-capability, telecom, and alternate-security-mechanism (each an evidence document in this checkpoint or the processing-capability decision).\n- The escalated processing-capability corrective action and its interim mitigation, if the processing-capability decision went at-risk.\n- The prior-cycle disposition and trend data, plus the open corrective-action Issues on the two Controls (query-data).\n\n**Procedure**\n_This checkpoint absorbs “Verify alternate storage site and media”, “Verify alternate telecom and priority of service”, “Verify alternate security mechanisms”. The agent runs the preparation, evidence assembly and record updates below; the named owners retain the substantive decisions and approvals stated in the procedure._\n1. Verify alternate storage site and media: Confirm the alternate storage site holds current, complete, retrievable backup media so recovery never depends on data that was never actually shipped or replicated off the primary site (UC-BCDR-04).\n2. Reconcile every scheduled backup set against what the alternate site actually holds, system by system and date by date. Read the uploaded backup schedule and alternate-site media/replication log and build a one-row-per-backup-set reconciliation.\n3. Flag three gap classes: (a) sets scheduled but absent at the alternate site, (b) media past its rotation or retention window, (c) replication lag exceeding the RPO for that system.\n4. Sample a subset of alternate-site media or replicated volumes and confirm retrievability — readable, uncorrupted, catalogued — without a full restore (full restore is reserved for the separate DR test exercise). Size the sample to cover the highest-criticality systems first; at minimum one volume per critical system.\n5. For each gap, record the affected system, its recovery point objective, and whether the gap breaches RPO.\n6. Draft the storage-site verification memo listing every reconciled set, each gap, and each retrievability result.\n7. Verify alternate telecom and priority of service: Confirm diverse or alternate telecommunications services with priority-of-service provisions are actually in force — not merely contracted — so connectivity survives a primary-path outage (UC-BCDR-04).\n8. List every primary and alternate circuit pair from the inventory with carrier, physical path, and last diversity-verification date.\n9. Confirm true diversity: each alternate circuit shares no carrier point of failure, conduit, or last-mile facility with its primary. Flag any pair that cannot be positively confirmed diverse — a different carrier alone is insufficient if the circuits share last-mile fiber.\n10. Confirm priority-of-service provisions are currently active and renewed (not lapsed) for every in-scope circuit and critical communications line; check each renewal date against today.\n11. Draft the telecom verification memo listing each circuit, its diversity status, and its priority-of-service status.\n12. Verify alternate security mechanisms: Confirm alternative security mechanisms are ready to substitute for a critical system's primary security safeguard when it is unavailable, so degraded mode never becomes unprotected mode (UC-BCDR-11).\n13. For each critical system, list each primary safeguard and the alternate mechanism that substitutes when the primary is unavailable.\n14. Confirm every alternate is presently deployed and enabled — not dormant or unconfigured — and record its last validation or test date.\n15. Assess equivalence: does the alternate provide materially equivalent protection, or would it leave a meaningful security gap? Flag any alternate that materially weakens protection.\n16. Draft the alternate-security-mechanism verification memo listing each system, its primary and alternate safeguard, and any readiness or equivalence gap.\n17. Classify readiness disposition: Verify the degraded-mode configuration on every critical system (safe mode and alternate communications), then judge whether the alternate capability and degraded-mode design verified this cycle are within tolerance or carry gaps needing tracked corrective action, so only the relevant closure path continues. Owned by the resilience lead.\n18. For each critical system, list the documented safe-mode trigger conditions and the restricted-function set from its design spec.\n19. Pull the live configuration and compare it against the spec; flag drift in the trigger condition, the restricted-function set, or the activation logic.\n20. Where a non-production or staging instance exists, confirm the trigger fires correctly and the restricted-function set matches design, without disrupting production.\n21. Draft the safe-mode verification memo listing each system, its configured trigger and restricted-function set, and any drift or untested finding (UC-BCDR-11).\n22. For each critical system, identify the primary communications protocol and its documented alternate channel.\n23. Confirm the alternate is currently configured and reachable — not merely documented — using recent connectivity or heartbeat evidence on the alternate path.\n24. Where a non-disruptive check is available, trigger a fallback on a non-production instance and confirm the system maintains continuity on the alternate protocol.\n25. Draft the alternate-communications verification memo listing each system, its alternate protocol, and its configured-and-reachable status; any system without a working fallback is a finding.\n26. Consolidate the six verification memos — storage-site, processing-capability, telecom, and alternate-security from their own steps, plus the safe-mode and alternate-communications memos produced above — together with any open corrective-action Issues on the two Controls (query-data), and build the readiness dashboard (dashboard create) showing each UC-BCDR-04 and UC-BCDR-11 element against its result and the trend against prior cycles.\n27. Judge the disposition against the criteria below and submit the branch with an evidence-referenced rationale.\n\n**Decision criteria**\n- Select **healthy** when every UC-BCDR-04 and UC-BCDR-11 element verified within tolerance this cycle — no open storage, processing, telecom, safe-mode, alternate-communications, or alternate-security finding remains, and any escalated processing gap has a confirmed interim mitigation and remediation plan.\n- Select **gaps_identified** when any element carries an open finding — for example missing or lapsed backup media or replication beyond RPO, an at-risk processing capability, a non-diverse or lapsed circuit, a drifted or untested safe-mode configuration, a missing alternate communications fallback, or a non-equivalent alternate security mechanism.\n\n**Record in AssureSwarm**\n- Attach the storage-site verification memo (reconciliation + retrievability sample + gap list) as a step document (DOCX/PDF, reconciliation tab as XLSX).\n- The backup schedule and alternate-site media/replication log enter this step as PBC uploads (they are not native items); query-data reads the anchor UC-BCDR-04/UC-BCDR-11 Control items and the open corrective-action Issues carried forward.\n- Attach the telecom verification memo (per-circuit diversity + priority-of-service status) as a step document (DOCX/PDF, circuit table as XLSX); the circuit inventory and priority-of-service records enter this step as PBC uploads, not native items.\n- Attach the alternate-security-mechanism verification memo (deployment, currency, equivalence gaps) as a step document (DOCX/PDF); per-system safeguard configs enter this step as PBC uploads, while query-data reads any safeguard Control items used for the equivalence check.\n- Attach the safe-mode verification memo (per-system config vs design, drift/untested findings) and the alternate-communications verification memo (per-system fallback configured-and-reachable status) as step documents (DOCX/PDF); the design specs, live configurations, and connectivity evidence enter this step as PBC uploads, not native items.\n- Build the readiness dashboard (dashboard create) across all six elements and attach the readiness summary with the signed disposition as a step document (DOCX/PDF).\n- Submit the `readiness_disposition` SELECT (healthy | gaps_identified); in the step result cite the consolidated findings list and dashboard; set the step's approver record.\n\n**Exit criteria**\n- Every scheduled backup set reconciled against alternate-site holdings; the retrievability sample documented; each gap tagged with its system and RPO-breach status; memo attached and the resilience lead confirms the media are current, complete, and retrievable and lists any gap that must carry into the readiness disposition.\n- Every in-scope circuit pair assessed for genuine path diversity; priority-of-service status confirmed active or flagged lapsed; memo attached and the resilience lead confirms diversity and provisioning and lists each lapsed or unconfirmed circuit as a finding.\n- Every critical system's alternate security mechanism confirmed deployed, current, and materially equivalent, or flagged; memo attached and the resilience lead confirms coverage leaves no gap and records each deficiency as a finding.\n- Every critical system's safe-mode configuration compared to its documented design and its alternate protocol confirmed configured and reachable or flagged, with both memos attached; all six verification results consolidated on the dashboard; SELECT submitted with an evidence-referenced rationale; any escalated processing gap confirmed still tracked; the unused branch is prunable.","kind":"decision","label":"Classify readiness disposition","performedBy":{"primitives":["coach-query-data","coach-document-upload","coach-dashboard-create"]}},"id":"classify-readiness-disposition"},{"data":{"description":"Agent converts every open finding into an owned corrective action; human confirms every gap is tracked and escalated where required","instructions":"**Objective** — Convert every gap identified during verification into an owned, tracked corrective action so no deficiency in the alternate capability or degraded-mode design goes unaddressed.\n\n**Inputs**\n- The readiness summary and consolidated findings list from the disposition decision, each finding tagged with its root cause and the verification memo that surfaced it.\n- The escalated processing-capability corrective action, if already open, to roll forward rather than duplicate.\n- The owner and recovery time objective per affected system.\n\n**Procedure**\n1. List each open finding with its root cause and the verification memo that surfaced it.\n2. Create an Issue item per finding capturing root cause, owner, due date, and interim mitigation where applicable; link each to the UC-BCDR-04 or UC-BCDR-11 Control item it impairs, naming the driving verification memo in the Issue description.\n3. Roll the already-escalated processing-capability gap forward as the same tracked Issue — do not create a duplicate.\n4. Escalate any finding with recovery-time-objective impact to resilience leadership if it is not already escalated.\n\n**Record in AssureSwarm**\n- Create one Issue item per finding (item create — issue_type: deficiency | observation, source: self_assessment, severity scaled to RTO impact, root_cause, issue_owner, target_remediation_date).\n- Link each Issue to the UC-BCDR-04 or UC-BCDR-11 Control item it impairs (items link); name the source verification memo in the Issue description, since step documents are not linkable targets.\n- Attach the corrective-action register as a step document (XLSX).\n\n**Exit criteria** — Every open finding has a named owner and due date; recovery-time-objective-impacting gaps escalated; the escalated processing gap rolled forward without duplication; nothing left untracked before closure; the resilience lead confirms.","label":"Log corrective actions","performedBy":{"primitives":["coach-item-create","coach-items-link","coach-document-upload"]}},"id":"log-corrective-actions"},{"data":{"description":"Automatically archive the authorized cycle record and carry open actions into the next cycle.","instructions":"**Objective** — Automatically preserve the authorized cycle record and its carry-forward actions after the preceding decision.\n\n**Inputs**\n- The full verification record: all verification memos, the readiness dashboard and summary, the disposition decision, and the corrective-action register (present only when the cycle routed through the gaps-identified branch).\n- Open corrective actions and any deferred verification, such as a retrievability sample that must expand next cycle.\n- The evidence repository location and retention policy, and the quarterly cadence schedule.\n\n**Procedure**\n1. Export the full verification record (workflow export) and archive it in the designated evidence repository under retention controls; record the archive location and reference.\n2. Leave open corrective-action Issues OPEN and create carry-forward Issues for deferred verification (for example an expanded retrievability sample), linking each to the UC-BCDR-04/UC-BCDR-11 Control items so the next instance reads them as existing open Issues.\n3. Update the control execution log with the cycle result and key metrics (elements verified, findings raised, escalations). The Control items carry no execution-log field, so the archived workflow instance IS the execution-log entry; confirm the next quarterly cadence review is scheduled.\n4. Hand off the readiness state to the assure-line DR test workflow: the readiness summary and dashboard serve as the de facto readiness-state package that workflow's first step reads (there is no terminal handoff node yet).\n5. Attach the closure record.\n\n**Record in AssureSwarm**\n- Export the workflow record (workflow export) and archive the run itself on the anchor Control as the durable audit trail (workflow instance).\n- Create/keep-open the carry-forward Issue items (item create) and link them to the UC-BCDR-04/UC-BCDR-11 Control items (items link); note deferred-verification scope in the closure record.\n- Attach the closure record as a step document (DOCX/PDF).\n\n**Exit criteria** — The archived record is immutable and retrievable; carry-forward items created and linked; the next review scheduled; nothing remains open without a tracked owner; the authorized cycle record is complete.","label":"Close and archive","performedBy":{"primitives":["coach-workflow-export","coach-item-create","coach-items-link","coach-document-upload"]},"requiredApprovals":0},"id":"close-and-archive"}],"sourceTemplateId":"workflow-library:controls-resilience-failover-readiness-verification"}