High Availability (HA) and Disaster Recovery (DR)

High Availability (HA) and Disaster Recovery (DR) in SAP BASIS

I. Understanding High Availability (HA) vs. Disaster Recovery (DR)

HA and DR are often used interchangeably, but they serve distinct purposes:

High Availability (HA):
- Goal: To minimize downtime and ensure continuous operation in the event of localized failures (e.g., server crash, power outage in a rack, network card failure, software bug).
- Scope: Typically within the same data center, often geographically close.
- Recovery Point Objective (RPO): Near-zero or zero data loss.
- Recovery Time Objective (RTO): Minutes to a few hours.
- Key Concept: Redundancy and automatic failover.
Disaster Recovery (DR):
- Goal: To recover from catastrophic failures affecting an entire site or region (e.g., natural disaster, widespread power outage, major fire, cyber-attack).
- Scope: Involves a geographically separate data center (DR site).
- RPO: Can tolerate some data loss (minutes to hours), depending on the DR strategy.
- RTO: Hours to days, depending on the DR strategy.
- Key Concept: Replication of data and infrastructure to a remote site for recovery.

II. Components Covered by HA/DR in SAP Systems

Both HA and DR strategies focus on protecting these critical SAP components:

Database Layer: The most critical component as it holds all business data.
- Examples: SAP HANA, Oracle, SQL Server, IBM DB2, Sybase ASE.
Central Services (ASCS/ERS): The Single Point of Failure (SPOF) for the SAP system.
- ASCS (ABAP SAP Central Services): Contains the Enqueue server (lock management) and Message Server (communication manager between application servers).
- ERS (Enqueue Replication Server): Provides HA for the Enqueue server by replicating the lock table to a secondary server. In case of ASCS failure, ERS takes over, and the original Enqueue server state is restored.
Application Servers (Dialog Instances): Provide the processing power for user requests and background jobs.
- These are generally easy to scale out (add more instances) and are inherently redundant if multiple are running. HA for these involves load balancing and failover mechanisms.
Shared Storage: Essential for SAP system files (executables, profiles, data files, log files).
- Typically a Network File System (NFS), Storage Area Network (SAN), or highly available clustered file system.

III. High Availability (HA) Strategies

HA primarily relies on redundancy and clustering within a single data center.

Clustering (Software Cluster Solutions):
- Principle: Two or more physical servers are configured to work together as a single logical unit. If one server fails, the other takes over its services.
- Common Cluster Software:
  - Linux: Pacemaker, Corosync, SLES HAE (High Availability Extension), RHEL HA Add-on.
  - Windows: Windows Server Failover Clustering (WSFC).
- Components Clustered: Primarily ASCS/ERS and the Database. Application servers usually run as individual instances.
- Active/Passive (Failover Clustering):
  - One node is active, running the service. The other is passive, waiting.
  - In case of active node failure, the passive node takes over the virtual IP and shared storage, starting the services.
  - Pros: Simpler to configure, widely used.
  - Cons: Resources on the passive node are idle; recovery involves a short downtime for failover.
- Active/Active (Workload Distribution/Clustering):
  - Both nodes are active, potentially running different services or sharing the same workload.
  - Often used for database mirroring/replication where both databases are active.
  - Pros: Better resource utilization.
  - Cons: More complex to configure and manage; failure of one node might reduce performance of the other.
SAP-Specific HA (ASCS/ERS):
- ASCS Instance: Contains Message Server (MS) and Enqueue Server (ES).
- Enqueue Replication Server (ERS): A separate instance that runs on a different host from the ASCS.
  - It receives a copy of the Enqueue lock table from the ASCS.
  - If the ASCS host fails, the cluster software brings the ERS instance online first, which recovers the last state of the Enqueue table. Then, the ASCS instance is brought online on a surviving cluster node.
  - Ensures: No loss of lock entries, preventing data inconsistency.
- Message Server HA: The Message Server state is not critical like the Enqueue server. Application servers simply re-register with the Message Server after a failover.
Load Balancing:
- Purpose: Distributes user requests across multiple application servers.
- SAP Web Dispatcher: Acts as a software load balancer for HTTP/HTTPS requests (web GUI, Fiori). Provides HA for web-based access.
- Message Server: For traditional SAP GUI connections, the Message Server handles load balancing based on logon groups (SMLG).
- HA for Web Dispatcher: Typically deployed in a redundant setup (e.g., two Web Dispatchers with a hardware load balancer in front of them, or a virtual IP managed by a cluster).
Redundant Infrastructure:
- Network: Multiple Network Interface Cards (NICs) with bonding/teaming, redundant switches, redundant routers.
- Storage: SAN or NAS with redundant controllers, power supplies, disks (RAID), and multiple paths to storage.
- Power: Dual power supplies, Uninterruptible Power Supplies (UPS), generators.

Important HA Configuration Considerations:

Virtual Hostnames and IPs: All SAP instances and database services should use virtual hostnames and IPs. This allows the underlying physical server to change during failover without requiring clients to reconfigure.
Shared Storage: Essential for ASCS and database (unless database replication is used). The shared storage must be highly available itself and accessible from all potential cluster nodes.
SAP Profile Parameters:
- enq/serverhost, enq/serverinst: Point to the virtual hostname of the ASCS instance.
- rdisp/mshost, ms/http_port: Point to the Message Server's virtual hostname/port.
- login/group_selection_enabled = 1: For Message Server-based load balancing.
- rdisp/auto_logout: To disconnect idle users.
Cluster Software Configuration: Correct fencing/STONITH (Shoot The Other Node In The Head) mechanisms to prevent split-brain scenarios.
DNS Configuration: DNS entries for virtual hostnames must point to the virtual IP.
Network Firewall Rules: Ensure all necessary ports are open between cluster nodes, SAP instances, and clients.
Application Server Redundancy: Deploy multiple dialog instances across different physical hosts.

IV. Disaster Recovery (DR) Strategies

DR aims to recover the SAP system at a remote site after a catastrophic event. The choice of strategy depends heavily on the acceptable RPO (data loss) and RTO (downtime).

Backup & Restore (Lowest Cost, Highest RPO/RTO):
- Principle: Regular backups are taken at the primary site and transferred to the DR site.
- Recovery: In a disaster, the last valid backup is restored at the DR site.
- RPO: Up to the age of the last full backup (e.g., 24 hours), plus any transaction logs since then.
- RTO: Can be very long (hours to days) due to restoration time.
- Considerations: Network bandwidth for transferring backups, storage at DR site.
Log Shipping / Database Replication (Moderate RPO/RTO):
- Principle: Transaction logs (database changes) are continuously shipped from the primary database to the standby database at the DR site and applied.
- Recovery: In a disaster, the standby database is activated.
- RPO: Minutes to a few hours (depending on log shipping frequency).
- RTO: Hours to a few hours (less than backup/restore).
- Database-Specific Technologies:
  - Oracle Data Guard: Physical Standby (block-level replication) or Logical Standby.
  - SQL Server AlwaysOn Availability Groups (Basic): Limited to one database, typically asynchronous.
  - HANA System Replication (HSR) - Asynchronous: Log blocks are replicated to the secondary system's disk.
  - DB2 HADR (High Availability Disaster Recovery):
- Considerations: Network bandwidth, latency, and the need to keep the DR database in a "recovery" or "standby" state.
Storage Replication (Low RPO/RTO):
- Principle: Entire storage volumes (SAN-to-SAN replication) are replicated from the primary data center to the DR site.
- Types:
  - Synchronous Replication: Changes are committed on both primary and secondary storage before acknowledgment to the host.
    - Pros: Zero data loss (RPO = 0).
    - Cons: High latency sensitive (limits distance), high bandwidth requirement. Best for short distances.
  - Asynchronous Replication: Changes are committed locally first, then asynchronously replicated.
    - Pros: Less latency sensitive (can be used over longer distances).
    - Cons: Small RPO (potential for a few seconds/minutes of data loss if primary fails before replication completes).
- Recovery: DR site storage is brought online, and servers at DR site connect to it.
- RPO: Near-zero (synchronous) or very low (asynchronous).
- RTO: Minutes to a few hours (depending on server startup time).
- Considerations: Cost of storage, network bandwidth, and the need for identical storage arrays.
Database AlwaysOn / HANA System Replication (HSR) - Synchronous (Very Low RPO/RTO):
- Principle: Real-time, continuous replication of data at the database layer, often with automatic failover capabilities within a region, and manual failover for cross-site DR.
- HANA System Replication (HSR) - Synchronous: Log blocks are replicated to the secondary system's memory.
  - Pros: Very low RPO (often zero in the same data center), fast failover.
  - Cons: High network bandwidth, sensitive to latency.
- SQL Server AlwaysOn Availability Groups (Advanced): Can span multiple data centers for DR.
- Oracle Data Guard - Far Sync: For zero data loss over long distances.
- Recovery: Rapid activation of the secondary database.
- RPO: Near-zero.
- RTO: Minutes to a few hours.

Important DR Configuration Considerations:

DR Site Readiness: The DR site must have identical or compatible hardware, network infrastructure, and sufficient resources to run the SAP system.
Network Configuration:
- Dedicated high-bandwidth, low-latency network links between primary and DR sites.
- DNS management to switch SAP virtual IPs/hostnames to the DR site.
Application Server Replication:
- Often, application servers are not replicated but are built clean at the DR site and then connected to the recovered database. Their configuration (profiles, kernel) can be copied.
- Alternatively, server virtualization allows VM replication.
SAP Profiles: Have up-to-date SAP profiles available at the DR site. These may need slight adjustments (e.g., target server names if not using virtual hostnames globally).
Test Drills: Regular and thorough DR testing is absolutely crucial. Without testing, you cannot be sure your DR strategy will work. This includes:
- Regularly restoring backups.
- Performing full DR failover drills.
- Verifying data consistency after failover.
Documentation: Comprehensive documentation of all HA/DR procedures, configurations, and contacts.
Monitoring: Implement robust monitoring for all replication processes and DR site health.

30 Interview Questions and Answers (One-Liner) for High Availability and Disaster Recovery in SAP BASIS

Q: What is the primary goal of High Availability (HA)?
- A: To minimize downtime from localized failures.
Q: What is the primary goal of Disaster Recovery (DR)?
- A: To recover from catastrophic site-wide failures.
Q: Which SAP component is considered a Single Point of Failure (SPOF) for HA?
- A: ASCS (ABAP SAP Central Services).
Q: What does ERS stand for in SAP HA?
- A: Enqueue Replication Server.
Q: What is the main purpose of ERS?
- A: To prevent loss of Enqueue lock entries during ASCS failover.
Q: What is a typical RPO for synchronous storage replication?
- A: Near-zero or zero data loss.
Q: What does RPO stand for?
- A: Recovery Point Objective.
Q: What does RTO stand for?
- A: Recovery Time Objective.
Q: Name two common Linux clustering solutions for SAP HA.
- A: Pacemaker and Corosync (or SLES HAE, RHEL HA Add-on).
Q: What is WSFC?
- A: Windows Server Failover Clustering.
Q: How does SAP Web Dispatcher contribute to HA?
- A: It acts as a software load balancer for HTTP/HTTPS requests, distributing workload.
Q: What is the purpose of using virtual hostnames/IPs in HA?
- A: To allow services to failover to another physical host without reconfiguring clients.
Q: What SAP profile parameter points to the Message Server host?
- A: rdisp/mshost.
Q: What database technology provides physical standby for Oracle DR?
- A: Oracle Data Guard.
Q: What is HSR in the context of SAP HANA DR?
- A: HANA System Replication.
Q: Which type of storage replication is sensitive to network latency over long distances?
- A: Synchronous replication.
Q: What is the main disadvantage of a Backup & Restore DR strategy?
- A: High RPO (potential data loss) and high RTO (long recovery time).
Q: Are SAP application servers typically replicated in a DR setup?
- A: No, they are often built clean at the DR site and configured to connect to the recovered database.
Q: What is the purpose of STONITH in a cluster setup?
- A: To prevent split-brain scenarios by ensuring a failed node is truly shut down.
Q: How do you perform load balancing for SAP GUI users?
- A: Via Logon Groups in SMLG and the Message Server.
Q: Which file system is commonly used for SAP shared directories in a clustered environment?
- A: NFS (Network File System) or a clustered file system.
Q: Why are regular DR test drills crucial?
- A: To validate the DR strategy and ensure it works as expected.
Q: What is the difference between active/passive and active/active clustering?
- A: Active/passive has one active node and one standby; active/active has both nodes actively participating in workload/services.
Q: What is the SAP ASCS instance responsible for?
- A: Enqueue Server (locks) and Message Server (communication, load balancing).
Q: What network consideration is vital for synchronous replication?
- A: Low latency and high bandwidth.
Q: What does "split-brain" refer to in clustering?
- A: When both cluster nodes believe they are the active node, leading to data corruption.
Q: Can one SAP system serve as both primary and DR for different SAP systems?
- A: Yes, a DR site can be a primary site for other systems (mutual DR).
Q: How does the Message Server help HA for application servers?
- A: It redirects users to active instances and balances the load across them.
Q: What is the role of DNS in an HA/DR setup?
- A: To map virtual hostnames to the correct active virtual IP (primary or DR site).
Q: What are the two main types of HSR (HANA System Replication)?
- A: Synchronous and Asynchronous.

5 Scenario-Based Hard Questions and Answers for High Availability and Disaster Recovery in SAP BASIS

Scenario: Your company wants to implement HA for their critical SAP S/4HANA system running on Red Hat Enterprise Linux (RHEL) with HANA DB. They have two physical servers in the primary data center. The business demands near-zero RPO for both the HANA database and the ASCS instance.
- Q: Design the HA solution for this setup, outlining the specific technologies, their configuration requirements, and how near-zero RPO is achieved for both HANA and ASCS.
- A:
  - HA Design:
    - OS: Red Hat Enterprise Linux (RHEL) with RHEL High Availability Add-On (based on Pacemaker/Corosync).
    - Database HA (HANA): HANA System Replication (HSR) in Synchronous Mode (Mode=sync, Operation Mode=logreplay).
      - Configuration:
        
        Primary HANA instance on Server A, Secondary HANA instance on Server B.
        
        HSR configured to replicate log buffers from primary to secondary in memory. A commit on primary only occurs after secondary acknowledges receipt of the log buffer.
        
        HANA HA/DR provider hooks (e.g., saptune for HANA, systemd integration with Pacemaker) for automatic failover.
        
        Virtual IP for HANA DB managed by Pacemaker, that floats between Server A and Server B.
      - RPO: Near-zero. Since logreplay mode is synchronous, no committed transaction is lost if the primary fails.
    - ASCS/ERS HA: Linux Cluster (Pacemaker/Corosync) managing ASCS and ERS resources.
      - Configuration:
        
        ASCS resource group (virtual IP, ASCS instance) configured to run on Server A initially.
        
        ERS instance running on Server B (the alternate node). ERS constantly replicates the Enqueue lock table from ASCS.
        
        Pacemaker manages the ASCS and ERS resources, ensuring they run on separate nodes for redundancy.
        
        Shared storage (e.g., highly available NFS or clustered file system like GFS2) mounted to both servers for SAP global host directories (/sapmnt/<SID>).
        
        Virtual IP for ASCS managed by Pacemaker.
      - RPO: Near-zero. In an ASCS failover:
        
        Pacemaker detects ASCS failure on Server A.
        
        ERS on Server B is activated as a standalone Enqueue server, restoring the last replicated lock table.
        
        ASCS is brought up on Server B (now taking over the ASCS role), connecting to the restored Enqueue Server.
        
        Application servers reconnect to the ASCS Message Server on Server B.
    - Application Servers (Dialog Instances):
      - Deploy at least one dialog instance on Server A and one on Server B.
      - Utilize SAP Message Server load balancing (SMLG logon groups) for user distribution.
      - These instances do not typically require clustering; their HA is inherent in their redundancy. If one fails, users are routed to others.
    - Shared Storage: Essential for sapmnt global host files, profiles, kernel, etc., accessible to all cluster nodes.
  - How Near-Zero RPO is Achieved:
    - HANA: Synchronous HSR ensures that transaction logs are replicated to the secondary's memory before the transaction is committed on the primary.
    - ASCS: ERS continuously replicates the Enqueue lock table. In a failover, ERS ensures the lock table is re-instantiated without loss, hence near-zero RPO for lock entries.
Scenario: Your production SAP ECC system is running on Windows Server with SQL Server database. You have an HA cluster for ASCS/ERS and a separate SQL Server AlwaysOn Availability Group (AG) for the database, both in your primary data center. Now, your business wants to establish a DR strategy to a remote data center (500 km away) with an RPO of less than 30 minutes and an RTO of less than 4 hours for a full SAP system recovery.
- Q: Propose a suitable DR strategy for this setup, detailing the specific technologies for SQL Server, ASCS/ERS, and application servers, and explaining how the RPO/RTO targets are met.
- A:
  - Proposed DR Strategy:
    - Database DR (SQL Server): SQL Server AlwaysOn Availability Groups (AG) with Asynchronous Replication across Data Centers.
      - Primary Site (DC1): AG with a synchronous replica for HA (e.g., Node 1, Node 2).
      - DR Site (DC2): Add a third replica (Node 3) to the same AG, configured for Asynchronous Replication.
      - Configuration:
        
        Databases in the AG on DC1 are in synchronous commit mode.
        
        The replica on DC2 is in asynchronous commit mode, allowing for some transaction log latency.
        
        A separate listener IP for the AG will exist at the DR site, or DNS needs to be updated during failover.
      - RPO: Achieved (less than 30 minutes). Asynchronous replication means some minimal data loss might occur (seconds to a few minutes) if DC1 fails before logs are hardened at DC2, but it's well within the 30-minute RPO.
      - RTO: This allows for a quick manual failover of the database to the DR site (minutes).
    - ASCS/ERS DR: No direct real-time replication. Instead, a "Warm Standby" approach with cluster software (WSFC) at the DR site.
      - Configuration:
        
        At the DR site, set up two servers (e.g., Node 4, Node 5) with WSFC configured, ready to host the ASCS/ERS roles.
        
        The SAP global host files (/sapmnt/<SID>) would be replicated to the DR site. This can be done via:
        
        Storage Replication (Asynchronous): If shared storage is used for sapmnt at DC1, replicate the entire volume to DC2.
        
        File-Level Replication: Tools like DFSR (Distributed File System Replication for Windows) or rsync for Linux to replicate the sapmnt directory.
        
        In a disaster: After database recovery, the ASCS/ERS cluster at DC2 is manually started (or a script triggers it), connecting to the replicated sapmnt and the recovered database.
      - RPO: Dependent on sapmnt replication frequency, but typically very low for critical files. The main RPO concern is the database.
      - RTO: Hours (setting up ASCS/ERS on DR cluster, connecting to DB).
    - Application Servers (Dialog Instances) DR: "Cold Standby" / Build-on-Demand.
      - Configuration:
        
        No running application servers at the DR site.
        
        Server images/templates or pre-built VMs for SAP application servers are available at the DR site.
        
        During a DR event, new application servers are rapidly deployed/booted, configured to connect to the recovered database and ASCS instance at the DR site.
        
        SAP profiles and kernel executables should be available from the replicated sapmnt.
      - RPO: Not applicable to application servers directly.
      - RTO: Hours (deployment and startup time).
    - Network & DNS:
      - Dedicated WAN link between DC1 and DC2 for AG replication.
      - Manual or automated DNS update to redirect SAP system's virtual IP/hostname to the DR site during failover.
  - Meeting RPO/RTO Targets:
    - RPO (< 30 minutes): Achieved by SQL Server AlwaysOn AG with asynchronous replication, which limits data loss to minutes.
    - RTO (< 4 hours):
      - Database failover is quick (minutes).
      - ASCS/ERS startup at DR site is manual but relatively fast (e.g., 30-60 minutes).
      - Bringing up application servers: By having pre-built images/VMs and automated scripts for configuration, multiple application servers can be started within 1-2 hours.
      - Total time adds up to within the 4-hour RTO.
Scenario: Your company performs monthly full DR drills for its SAP ECC system (Oracle DB on Linux). During the last drill, the sapinst tool, used to install additional application servers at the DR site, failed repeatedly with "Disk full" errors, significantly extending the RTO. The storage team confirms the logical volumes for /sapmnt and /usr/sap are sized correctly per SAP recommendations at the DR site.
- Q: What is the most likely underlying cause of the "Disk full" error during sapinst execution in a DR drill, despite correct logical volume sizing, and what specific Basis-level checks and remedies would you implement?
- A:
  - Most Likely Underlying Cause: The "Disk full" error during sapinst installation, despite correctly sized logical volumes for /sapmnt and /usr/sap, strongly indicates that the /tmp (temporary directory) or the /var (variable data, including logs and temporary files for installation) file system on the Linux DR servers is insufficient or not correctly sized/mounted. sapinst extensively uses /tmp for temporary files, logs, and unpacking during the installation process. Even if the target directories are large, a small /tmp can halt the installation.
  - Specific Basis-Level Checks and Remedies:
    1. Check /tmp and /var Size and Usage:
      - Action: Before running sapinst again (or during analysis of the failed attempt), log onto the DR server and execute df -h to check the free space on all mounted file systems, specifically /tmp, /var, /usr/sap, and /sapmnt.
      - Action: Also, check /var/tmp and /var/log for large files.
      - Rationale: Confirm the actual free space on the temporary file systems.
    2. sapinst Log Analysis:
      - Action: Review the sapinst_dev.log and sapinst.log files from the failed installation attempt (located in /tmp/sapinst_instdir or a specific directory defined by SAPINST_INSTDIR environment variable).
      - Look for: Specific messages indicating which file or directory couldn't be written to due to lack of space, often explicitly mentioning /tmp or /var/tmp.
      - Rationale: Pinpoints the exact location of the disk full error.
    3. Verify SAP Pre-Requisites and OS Notes:
      - Action: Consult the SAP installation guide for the specific S/4HANA version and OS. Pay close attention to OS pre-requisites for /tmp and /var sizing.
      - Action: Check relevant SAP Notes for sapinst issues or specific OS requirements (e.g., large /tmp for HANA installations).
      - Rationale: Ensure all basic requirements are met.
    4. Remedies for Insufficient /tmp / /var:
      - Increase File System Size:
        
        Action: Coordinate with the OS/storage team to extend the /tmp and/or /var file systems' logical volumes/partitions. This is the most robust solution.
        
        Rationale: Provides permanent space.
      - Clean Up Existing /tmp:
        
        Action: If /tmp has old, accumulated temporary files from previous attempts or other processes, clean them up.
        
        Rationale: Frees up space immediately.
      - Redirect sapinst Temporary Directory:
        
        Action: Before starting sapinst, set the environment variable SAPINST_TEMPDIR to a path on a larger file system (e.g., /usr/sap/tmp if /usr/sap has free space).
        
        Command: export SAPINST_TEMPDIR=/usr/sap/tmp then run sapinst.
        
        Rationale: A quick workaround to bypass a small /tmp partition without resizing.
      - Clean up /var logs: If /var/log or /var/tmp are full, clean up old logs.
    5. DR Drill Process Enhancement:
      - Action: Update the DR drill runbook to include pre-checks for /tmp and /var free space on DR servers before starting sapinst.
      - Rationale: Prevent this issue in future drills.
Scenario: Your company recently upgraded its SAP ECC system to S/4HANA 2023. Post-upgrade, you are conducting the annual DR drill. The recovery of the ASCS/ERS cluster on Windows Server Failover Clustering (WSFC) at the DR site is failing repeatedly. The ASCS virtual IP comes online, but the Enqueue Server resource fails to start, and the cluster logs show "Resource 'SAP <SID> <Instance Number> ENQ' failed to start." There's no specific error in the SAP ASCS dev_ms or dev_enq logs at the OS level during startup attempts.
- Q: What is the most probable cause for an Enqueue Server resource failing to start within a WSFC after an S/4HANA upgrade, especially if SAP logs are uninformative, and what specific areas would you investigate and rectify?
- A:
  - Most Probable Cause: The highly probable cause for this specific Enqueue Server resource failure after an S/4HANA upgrade, particularly with uninformative SAP logs, is an issue with the shared storage access or permissions from the DR cluster nodes to the globally mounted SAP file systems (e.g., sapmnt share).
    - Upgrade Impact: S/4HANA upgrades often involve kernel updates, changes to service users (e.g., SAPService<SID>), or stricter security requirements. These changes might implicitly affect how the cluster service or the SAP instance service account accesses the shared sapmnt drive or network shares.
    - Lack of SAP Logs: If dev_enq and dev_ms logs are clean, it implies the SAP process itself isn't even getting far enough to generate its own specific error messages, suggesting a fundamental external dependency (like shared storage access or critical binaries) is failing first.
  - Specific Areas to Investigate and Rectify:
    1. Shared Storage Accessibility (Crucial):
      - Action: On both DR cluster nodes, manually try to access the sapmnt share (e.g., dir \\<fileshare>\sapmnt\<SID>) using the SAPService<SID> user context (or the user running the cluster service).
      - Check: Verify the share permissions on the file server providing sapmnt (usually a separate NAS/SAN). Ensure SAPService<SID> or the cluster service account has full control/modify permissions.
      - Action: If sapmnt is a local mount (e.g., GFS2), verify mount points and permissions on the underlying LUN.
      - Rationale: The Enqueue Server needs to access its executable, profiles, and potentially the replication server files from sapmnt. If it cannot, it cannot start.
    2. SAPService<SID> User Permissions:
      - Action: In Active Directory (if used), verify that the SAPService<SID> user (used for SAP services) has not been locked, expired, or had its permissions revoked.
      - Action: Ensure the SAPService<SID> user has "Log on as a service" right on both DR cluster nodes.
      - Action: Check the local security policy on both DR cluster nodes (secpol.msc) to ensure SAPService<SID> is not explicitly denied any rights.
      - Rationale: This user runs the SAP instance services, including the Enqueue Server. Upgrade processes can sometimes affect service user configurations.
    3. ASCS/ERS Resource Dependencies in WSFC:
      - Action: In Failover Cluster Manager, inspect the properties and dependencies of the "SAP <SID> <Instance Number> ENQ" resource.
      - Check: Does it correctly depend on the "SAP <SID> <Instance Number> FS" (File Share) resource or the virtual IP? Ensure no new incorrect dependencies were introduced during the upgrade.
      - Rationale: Misconfigured dependencies can prevent resources from starting.
    4. Network Firewall between Cluster Nodes and File Server:
      - Action: Ensure no new firewall rules (local or network-based) are blocking access from the DR cluster nodes to the file server hosting sapmnt after the upgrade.
      - Rationale: Critical for network share access.
    5. SAP Profile Parameter Review:
      - Action: Check the DEFAULT.PFL and ASCS instance profile at the DR site. Ensure enq/serverhost and enq/serverinst point to the correct virtual hostname and instance number of the ASCS instance.
      - Rationale: Although usually stable, a profile inconsistency can cause startup issues.
    6. Kernel Patch Level (Minor possibility):
      - Action: Ensure the kernel on the DR site servers (specifically the sapstartsrv and other executables) is compatible with the upgraded S/4HANA version.
      - Rationale: While less likely to cause a "resource failed to start" without specific SAP errors, kernel mismatches can lead to unpredictable behavior.
    7. WSFC Event Logs:
      - Action: Check the Windows Event Viewer on both DR cluster nodes under "Failover Clustering" for more detailed error messages related to the resource startup failure.
      - Rationale: Provides system-level context.
Scenario: Your company's SAP system (AnyDB on Linux) utilizes a multi-tier DR strategy: synchronous HANA System Replication (HSR) within the primary data center (for HA), and asynchronous HSR to a DR site located 1000 km away. During a recent network outage affecting the inter-DC link for 15 minutes, you observed a significant performance degradation in the production SAP system at the primary site. The network link has since been restored, and HSR replication is catching up, but the business wants to understand why production performance was impacted by a DR link issue for asynchronous HSR.
- Q: Explain the technical reason why an asynchronous HSR link outage could impact primary SAP system performance, and what configuration adjustments or architectural considerations could mitigate this without switching to a different DR strategy.
- A:
  - Technical Reason for Performance Impact:
    - Even though HSR to the DR site is asynchronous, the primary HANA system still has to manage the asynchronous replication queue and acknowledge that the log segments are written to disk on the secondary system.
    - When the inter-DC network link fails, the asynchronous replication queue on the primary HANA system starts to build up. Log segments accumulate on the primary's disk, waiting to be sent to the secondary.
    - This backlog of unsent log segments can lead to:
      1. Increased Disk I/O on Primary: The primary system needs to write the log segments to its own disk and keep them there until they are sent to the secondary. This can increase disk I/O, especially if the volume of data changes is high.
      2. Resource Contention: The HANA logwriter process or other replication-related processes on the primary system might consume more CPU or memory as they try to manage the growing queue and retry sending.
      3. Log Volume Full (Extreme Case): In a prolonged outage or very high transaction volume, the primary's log volume could potentially become full, which would halt all transactions on the primary.
      4. Backpressure: Although asynchronous, if the queue on the primary becomes too large, it might eventually exert a form of backpressure on transactions, causing them to wait for log segments to be offloaded, even if not directly waiting for remote acknowledgment.
  - Configuration Adjustments / Architectural Considerations to Mitigate:
    1. Dedicated Network for HSR:
      - Adjustment: Ensure HSR traffic between primary and DR sites uses a dedicated, high-bandwidth, and low-latency network link, separate from other business traffic.
      - Rationale: Isolates HSR performance from general network congestion and reduces the chance of replication-related issues impacting other business operations.
    2. HANA Log Volume Sizing:
      - Adjustment: Ensure the primary HANA system's log volume is generously sized (e.g., at least 2-3 times the size of the logical memory volume, plus additional buffer for replication backlog) to accommodate potential log backlogs during network outages.
      - Rationale: Provides more buffer space for unsent log segments, delaying the "log volume full" scenario.
    3. HANA global.ini Parameters (HSR Specific):
      - Adjustment: Review and fine-tune HSR-related parameters in global.ini (especially [replication], [system_replication]).
      - Example parameters (check SAP documentation for specific versions):
        
        system_replication_max_delay_size: Limits the maximum size of the log shipping backlog. If exceeded, the primary might temporarily pause commits or go into "catch-up" mode to reduce the backlog, impacting performance. Adjusting this might allow a larger backlog before impact.
        
        system_replication_queue_mode: For asynchronous, typically "ASYNC_FULL".
      - Rationale: These parameters control how HANA behaves under replication pressure. Careful tuning can allow more tolerance for network issues.
    4. Network Quality of Service (QoS):
      - Adjustment: Implement QoS policies on network devices to prioritize HSR traffic over the inter-DC link.
      - Rationale: Ensures that even if the link is congested, HSR traffic gets preferential treatment, helping to clear the backlog faster once connectivity is restored.
    5. Monitoring and Alerting for HSR Backlog:
      - Adjustment: Implement robust monitoring (e.g., SAP Solution Manager, custom scripts) to alert Basis/DBA teams when the HSR replication backlog (log buffer or log segment queue) exceeds predefined thresholds.
      - Rationale: Allows proactive intervention (e.g., temporarily pausing non-critical operations, or initiating a manual "log replay" if the secondary database starts falling too far behind) before performance significantly degrades.
    6. "Performance Optimized" HSR Asynchronous Mode (if available for version):
      - Adjustment: Some HANA versions offer a "Performance Optimized" asynchronous mode which might have different behavior regarding primary system impact during replication outages. Investigate if this applies and is suitable.
      - Rationale: Offers alternative asynchronous behavior.
    7. Temporary Deactivation/Resynchronization (Manual Intervention):
      - Architectural Consideration: For very prolonged outages, consider a procedure to temporarily deactivate asynchronous replication to prevent primary system performance degradation or log volume issues. Once the network is stable, replication can be re-initialized (which might involve a full data copy if the gap is too large).
      - Rationale: A last resort to protect primary system performance.

This deep dive into HA and DR concepts, strategies, and configurations provides a strong foundation for managing SAP system reliability and resilience.

Rakshit Ranjan Singh

Search This Blog