Skip to content

[patch] Fix DB2 HADR Setup - #559

Open
tomklapiscak wants to merge 8 commits into
mainfrom
fixhadrs3
Open

[patch] Fix DB2 HADR Setup#559
tomklapiscak wants to merge 8 commits into
mainfrom
fixhadrs3

Conversation

@tomklapiscak

@tomklapiscak tomklapiscak commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Description

This PR resolves three separate problems relating to the use of S3 as part of the HADR setup process. These issues were discovered while testing HADR in GovCloud. The first issue is specific to GovCloud, the other two are bugs that I happened to hit, and may be hit in Commercial too.

GovCloud Compatibility

S3 URL is different in GovCloud (e.g. https://s3-fips.us-gov-east-1.amazonaws.com vs https://s3.us-east-1.amazonaws.com), which was causing downloads from S3 to fail. This PR fixes this by passing in the value of the SERVER variable in .PROPS to the --endpoint-url argument of the aws s3 cp command.


Issues with large backup files

A subsequent issue was discovered where the db 2RemStgManager command was segfaulting when attempting to upload a large backup file.

I had 2 backup parts: 001 is 25GB, and 002 is 97MB.

The db2RemStgManager command code used to upload backup files to S3 is failing for the 25GB 001 part with:

Memory fault(coredump)

but works fine for the 97MB 002 part.

Not sure why it's failing for the large file only:

  • There is sufficient storage in the DB2_OBJECT_STORAGE_LOCAL_STAGING_PATH (>150GB)
  • The db2u pod has 32GB memory limit
  • No ulimits set on db2inst1 user
  • DB2_ENABLE_COS_SDK isn't set (so defaults to `ON), so no file limit of 5GB and multipart upload should be supporteds
  • multipartsizemb is set to the default of 100MB

This PR fixes this by replacing the db2RemStgManager command with a simple call to aws s3 cp (which is already used on the standby to download the backup files).


HADR job fails when a backup file is not yet present in COS

If a backup file does not exist in COS at the time the postsync-hadr job runs then $timestamp will be empty and the following condition will (always) fail:

if [[ $(( `date +%Y%m%d` - ${timestamp:0:8} )) -lt 2 || -z ${timestamp} ]]; then 

with error:

/bin/sh: line 229: 20260814 -  : syntax error: operand expected (error token is "-  ")

The condition is incorrect, the -z ${timestamp} gets evaluated AFTER the first condition, so if timestamp is empty, it will always error.

Besides, the logic itself is not correct - the code branch it covers should only be run if there is a backup (and hence a timestamp) and that timestamp is less that 2 days old.

This PR changes the condition to:

if [[ -n ${timestamp} && $(( `date +%Y%m%d` - ${timestamp:0:8} )) -lt 2 ]]; then

with that change in place, the postsync-hadr job that was previously failing now works as expected.

Testing

Verified that HADR setup now works in GovCloud when applied to existing and fresh Manage and Facilities DB2 instances in GovCloud.

@tomklapiscak
tomklapiscak marked this pull request as draft August 13, 2026 13:19
db2RemStgManager was segfaulting on 25GB backup files
…is getting forced to "manage""

This reverts commit 626fe54.
@tomklapiscak tomklapiscak changed the title [patch] Fix HADR in GovCloud [patch] Fix DB2 HADR Setup Aug 14, 2026
@tomklapiscak
tomklapiscak marked this pull request as ready for review August 14, 2026 15:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants