Monat: Februar 2023

How-To: Move and delete old Partitions made easy

Hello fellow DBAs,

it’s time to share another tidbit from my daily work.

Given, that we have a table in our schema, that has a range partiton by month, our application people wanted to have a procedure to move all partitions older than a month to an archive tablespace and all older than 90 Days to be deleted, I came up with this small bit of code to make their lives easier.

First, let’s take a look at how the table is created.

CREATE TABLE DB_USER.PART_TABLE_1 
   (	
   ID01 NUMBER(19,0) NOT NULL ENABLE, 
   DATA1 varchar2(4000), 
   DATA2 varchar2(4000), 
   CREATE_TIMESTAMP TIMESTAMP (6) NOT NULL ENABLE 
   ) TABLESPACE DATA 
   PARTITION BY RANGE (CREATE_TIMESTAMP) INTERVAL (NUMTOYMINTERVAL(1, 'MONTH')) 
   (
     PARTITION OLD_DATA VALUES LESS THAN (TIMESTAMP' 2021-03-01 00:00:00') SEGMENT CREATION IMMEDIATE TABLESPACE DATA
   ); 
ALTER TABLE DB_USER.PART_TABLE_1 ADD UNIQUE (ID01)
  TABLESPACE DATA  ENABLE;

As you can see, I partitioned, I created a unique Index which will give me an Index on the table.
Then we fill and fill and fill the Table with lots and lots of data, partitions will be created.

And now to the interesting part, the procedures.

First, let’s move all partitons that are older than one month to the Archive Tablespace.

create or replace procedure DB_USER.MOVE_PART_TABLE_1_PARTITIONS AS
begin
for i in
(
select table_name, partition_name,high_value_in_date_format from (
   select 
          table_name,
          partition_name,
          to_date (
          trim (
             '''' from regexp_substr (
                        extractvalue (
                          dbms_xmlgen.getxmltype (
                          'select high_value from user_tab_partitions where table_name='''
                                   || table_name
                                   || ''' and partition_name = '''
                                   || partition_name
                                   || ''''),
                                '//text()'),
                             '''.*?''')),
             'syyyy-mm-dd hh24:mi:ss')
             high_value_in_date_format
     FROM user_tab_partitions
    WHERE (table_name = 'PART_TABLE_1') AND TABLESPACE_NAME = 'DATA'
 )
WHERE high_value_in_date_format < trunc(sysdate, 'MONTH')
)
LOOP
    execute immediate 'alter table '||i.table_name||' move partition '||i.partition_name||' tablespace ARCHIVE online update indexes';
end loop;

end;
/

The ONLINE Keyword in the move command works only on Enterprise Editions, keep that in mind.
After we moved everything, we want to get rid of old data regularly, so let’s delete old partitions.

create or replace procedure DB_USER.DELETE_PART_TABLE_1_PARTITIONS AS
begin
for i in
(
select table_name, partition_name,high_value_in_date_format from (
   select 
          table_name,
          partition_name,
          to_date (
          trim (
             '''' from regexp_substr (
                        extractvalue (
                          dbms_xmlgen.getxmltype (
                          'select high_value from user_tab_partitions where table_name='''
                                   || table_name
                                   || ''' and partition_name = '''
                                   || partition_name
                                   || ''''),
                                '//text()'),
                             '''.*?''')),
             'syyyy-mm-dd hh24:mi:ss')
             high_value_in_date_format
     FROM user_tab_partitions
    WHERE (table_name = 'PART_TABLE_1')  AND TABLESPACE_NAME = 'ARCHIVE'
 )
WHERE high_value_in_date_format < trunc(sysdate-90, 'MONTH')
)
LOOP
    execute immediate 'alter table '||i.table_name||' drop partition '||i.partition_name||' online update indexes';
end loop;

end;
/

As with the former procedure, the online keywork only works in enterprise editions. And I have a little Failsave, that it only takes partitions in the ARCHIVE Tablespace into account.

By adjusting the Where clauses you can easily transform this to work with multiple tables, different high values, etc.

Now that you have these procedures, call them in a scheduler job, cronjob or whatever you desire.

I welcome your comments.

TDE Wallets – Best Practices and How-Tos

Hello fellow Oracle DBAs. In this Article I will give you a few established „Best Practices“ that have emerged in my work when using TDE (Transparent Data Encryption) Wallets. Also, I will give you a few commands for handling Wallets and Keys and a Guide for a little bit of Troubleshooting.

CAVE: On Premises TDE is a feature that needs to be licensed. You need to have "Advanced Security" licensed in order to use TDE. The license is included in Autonmous Databases and Database Cloud Services within the OCI (Oracle Cloud Infrastructure)

Versions that are covered by this Article: 18c, 19c

Best Practices

Naturally this list is not complete and probably never will be and some of these practices may not be for you. Nonetheless, let me know, if you have something that could be missing.

Use TDE Wallets only for TDE Masterkeys. No Credentials and no Certificates

Storing Credentials or Certificates in an TDE Wallet can lead to the database not being able to open the Wallet at startup and thus not being able to access TDE encrypted Data in the Database.

This is one possible reason for the error: „ORA-28365 Wallet not open“

Manage TDE Wallets only with SQL*Plus

When managing TDE Wallets and Keys use only the SQL*Plus „ADMINISTER KEY MANAGEMENT“ commands.

These command require a backup parameter whenever there is a change done to the wallet which then backs up the file.

Store Wallet in Filesystem

Store the Wallet in a Filesystem that is part of the Filesystembackup, thus we regularly have a backup. On RAC Systems this should be a shared ACFS Filesystem.

Don’t use ASM to store your TDE Wallets, or have a really good concept for backups whenver you make a change to the Wallet.

Store the Wallet Password securely in a Password Manager

Don’t lose the Password. Losing Access to the Wallet and Keys within can lead to not being able to access the Data

Use „wallet_root“ Parameter in Database

Don’t use the old sqlnet.ora Parameters to define the path to your Oracle TDE Wallet this is deprecated in future Database Versions.

Use the Database Parameter „wallet_root“, which overrides any old sqlnet.ora parameters.

How-Tos

Step by Step Instructions to Configure a Wallet in a Non-CDB or CDB Database

Create Wallet Directory in Non-Rac Environment and secure directory

[oracle@dbhost01:ORCL]/home/oracle: 
[oracle@dbhost01:ORCL]/home/oracle: mkdir /u01/tde_wallet
[oracle@dbhost01:ORCL]/home/oracle: cd /u01/tde_wallet
[oracle@dbhost01:ORCL]/u01/tde_wallet: ls -la .
total 0
drwxrwxr-x  2 oracle dba   6 Jan 11 14:14 .
drwxrwxr-x 13 oracle dba 140 Jan 11 14:14 ..
[oracle@dbhost01:ORCL]/u01/tde_wallet: chmod 750 .
[oracle@dbhost01:ORCL]/u01/tde_wallet: ls -la .
total 0
drwxr-x---  2 oracle dba   6 Jan 11 14:14 .
drwxrwxr-x 13 oracle dba 140 Jan 11 14:14 ..
[oracle@dbhost01:ORCL]/u01/tde_wallet:

Create shared ACFS Wallet Directory in RAC Environment and secure directory

Create ACFS Volume
[oracle@dbhost01:+ASM1]/u01/tde_wallet: asmcmd
ASMCMD> volcreate -G DATA -s 10G vol_wallets
ASMCMD> volinfo -G DATA vol_wallets
Diskgroup Name: DATA
 
         Volume Name: VOL_WALLETS
         Volume Device: /dev/asm/vol_wallets-377
         State: ENABLED
         Size (MB): 10240
         Resize Unit (MB): 64
         Redundancy: HIGH
         Stripe Columns: 8
         Stripe Width (K): 1024
         Usage:
         Mountpath:
ASMCMD> exit
[oracle@dbhost01:+ASM1]/u01/tde_wallet:
Create Filesystem on ACFS Volume
[oracle@dbhost01:+ASM1]/u01/tde_wallet: /sbin/mkfs -t acfs /dev/asm/vol_wallets-377
mkfs.acfs: version                   = 19.0.0.0.0
mkfs.acfs: on-disk version           = 46.0
mkfs.acfs: volume                    = /dev/asm/vol_wallets-377
mkfs.acfs: volume size               = 10737418240  (  10.00 GB )
mkfs.acfs: Format complete.
Register Filesystem and wait until it is mounted on both nodes
[root@dbhost01:+ASM1]/root:
[root@dbhost01:+ASM1]/root: /sbin/acfsutil registry -a /dev/asm/vol_wallets-377 /u01/tde_wallets
acfsutil registry: mount point /u01/tde_wallets successfully added to Oracle Registry
[root@dbhost01:+ASM1]/root: df -h /u01/tde_wallets
Filesystem                Size  Used Avail Use% Mounted on
/dev/asm/vol_wallets-377   10G  570M  9.5G   6% /u01/tde_wallets
[root@dbhost01:+ASM1]/root: chown oracle:dba /u01/tde_wallets
[root@dbhost01:+ASM1]/root: ls -la /u01/tde_wallets
total 100
drwxr-xr-x  4 oracle dba  32768 Jan 11 14:35 .
drwxr-xr-x 11 root   dba   4096 Jan 11 14:35 ..
drwx------  2 root   root 65536 Jan 11 14:35 lost+found
[root@dbhost01:+ASM1]/root:

Set Database Parameters for Wallet and restart Database

Set wallet_root Parameter
[oracle@dbhost01:ORCL]/u01/tde_wallet: sqlplus / as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Wed Jan 11 15:04:43 2023
Version 19.16.0.0.0
 
Copyright (c) 1982, 2022, Oracle.  All rights reserved.
 
 
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.16.0.0.0
 
SQL> alter system set wallet_root='/u01/tde_wallets/ORCL' scope=spfile;
 
System altered.
 
SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.16.0.0.0
[oracle@dbhost01:ORCL]/u01/tde_wallet:
Restart Database and create Wallet directory
[oracle@dbhost01:ORCL]/u01/tde_wallet: srvctl stop database -db ORCL
[oracle@dbhost01:ORCL]/u01/tde_wallet: mkdir -p /u01/tde_wallets/ORCL/tde
[oracle@dbhost01:ORCL]/u01/tde_wallet: srvctl start database -db ORCL
Set tde_configuration parameter
[oracle@dbhost01:ORCL]/u01/tde_wallet: sqlplus / as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Wed Jan 11 15:22:46 2023
Version 19.16.0.0.0
 
Copyright (c) 1982, 2022, Oracle.  All rights reserved.
 
 
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.16.0.0.0
 
SQL> alter system set tde_configuration='KEYSTORE_CONFIGURATION=FILE' scope=spfile;
 
System altered.
 
SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.16.0.0.0
Create Wallet and TDE Masterkey
  • Create Wallet
  • Open Wallet
  • Create and Set Masterkey
  • Create Autologin
  • Restart Database to test Autologin Wallet
[oracle@dbhost01:ORCL]/u01/tde_wallets/ORCL/tde: sqlplus / as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Thu Jan 12 08:24:53 2023
Version 19.17.0.0.0
 
Copyright (c) 1982, 2022, Oracle.  All rights reserved.
 
 
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.17.0.0.0
 
SQL> ADMINISTER KEY MANAGEMENT CREATE KEYSTORE '/u01/tde_wallets/ORCL/tde' IDENTIFIED BY "SuperSecretPassword";
 
keystore altered.
 
SQL> ADMINISTER KEY MANAGEMENT SET KEYSTORE OPEN IDENTIFIED BY "SuperSecretPassword";
 
keystore altered.
 
SQL> ADMINISTER KEY MANAGEMENT SET KEY IDENTIFIED BY "SuperSecretPassword" with backup;
 
keystore altered.
 
SQL> ADMINISTER KEY MANAGEMENT CREATE AUTO_LOGIN KEYSTORE FROM KEYSTORE '/u01/tde_wallets/ORCL/tde' IDENTIFIED BY "SuperSecretPassword";
 
keystore altered.
 
SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.17.0.0.0
[oracle@dbhost01:ORCL]/u01/tde_wallets/ORCL/tde: srvctl stop database -db ORCL
[oracle@dbhost01:ORCL]/u01/tde_wallets/ORCL/tde: srvctl start database -db ORCL
[oracle@dbhost01:ORCL]/u01/tde_wallets/ORCL/tde: sqlplus / as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Thu Jan 12 08:28:03 2023
Version 19.17.0.0.0
 
Copyright (c) 1982, 2022, Oracle.  All rights reserved.
 
 
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.17.0.0.0
 
SQL> select * from gv$encryption_wallet;
 
   INST_ID WRL_TYPE
---------- --------------------
WRL_PARAMETER
--------------------------------------------------------------------------------
STATUS                         WALLET_TYPE          WALLET_OR KEYSTORE FULLY_BAC
------------------------------ -------------------- --------- -------- ---------
    CON_ID
----------
         2 FILE
/u01/tde_wallets/ORCL/tde/
OPEN                           AUTOLOGIN            SINGLE    NONE     NO
         1
 
 
         1 FILE
/u01/tde_wallets/ORCL/tde/
OPEN                           AUTOLOGIN            SINGLE    NONE     NO
         1
 
 
2 rows selected.
 
SQL> select inst_id, key_id, creation_time, CREATOR_DBNAME from gv$encryption_keys;
 
   INST_ID
----------
KEY_ID
------------------------------------------------------------------------------
CREATION_TIME
---------------------------------------------------------------------------
CREATOR_DBNAME
--------------------------------------------------------------------------------
         1
AUC/+0hT/E9pv4p+f5I7KQcAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
12-JAN-23 09.25.39.988752 AM +02:00
ORCL
 
         2
AUC/+0hT/E9pv4p+f5I7KQcAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
12-JAN-23 09.25.39.988752 AM +02:00
ORCL
 
 
SQL>
Open Wallet in Pluggable Database and Create/Set Masterkey

For this to work, you have to have the CDB Steps completed. Thus, a functioning Wallet already has to exist
Change into Pluggable Database and show Wallet Status

SQL> show pdbs
 
    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 ORCL_P01                 READ WRITE NO
SQL> alter session set container=ORCL_P01;
 
Session altered.
 
SQL> SELECT wrl_parameter, status, wallet_type FROM v$encryption_wallet;
 
WRL_PARAMETER
--------------------------------------------------------------------------------
STATUS                         WALLET_TYPE
------------------------------ --------------------
 
OPEN_NO_MASTER_KEY             AUTOLOGIN

If Wallet is Closed, open in CDB.

Create Masterkey in Pluggable Database and Check Key
[oracle@dbhost01:ORCL]/DBA/users/oracle: sqlplus / as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Wed Jan 18 14:24:44 2023
Version 19.17.0.0.0
 
Copyright (c) 1982, 2022, Oracle.  All rights reserved.
 
 
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.17.0.0.0
 
SQL> alter session set container=ORCL_P01;
 
Session altered.
 
SQL> ADMINISTER KEY MANAGEMENT SET KEY USING TAG 'ORCL_P01' FORCE KEYSTORE IDENTIFIED BY "SuperSecretPassword" WITH BACKUP;
 
keystore altered.
 
SQL> SELECT wrl_parameter, status, wallet_type FROM v$encryption_wallet;
 
WRL_PARAMETER
--------------------------------------------------------------------------------
STATUS                         WALLET_TYPE
------------------------------ --------------------
 
OPEN                           AUTOLOGIN
 
 
SQL> set line window
SQL> set pagesize 50
SQL> select key_id, con_id, CREATOR_PDBNAME from gv$encryption_keys order by con_id,key_id;
 
KEY_ID                                                                             CON_ID
------------------------------------------------------------------------------ ----------
CREATOR_PDBNAME
--------------------------------------------------------------------------------------------------------------------------------
AefCYCk3qk+Hv+MPk9F/dJgAAAAAAAAAAAAAAAAAAAAAAAAAAAAA                                    3
ORCL_P01
 
AefCYCk3qk+Hv+MPk9F/dJgAAAAAAAAAAAAAAAAAAAAAAAAAAAAA                                    3
ORCL_P01
 
SQL>

Various Commands for Wallet Handling and Troubleshooting

Possible Troubleshooting for „Wallet not open“ Error

Various Steps can now be taken to check for the most common reasons that lead this error. I give you here a small checklist and the necessary commands to check your wallet.

  • Check Wallet Parameters and Keys
  • Check if Wallet and Autologin Wallet are existing and have correct Permissions
  • Try to show contents of Wallet as OS User oracle

Check Wallet Parameters and Keys

Here’s an example output of a correct open autologin wallet with various keys

[oracle@dbhost01:ORCL]/DBA/users/oracle: sqlplus / as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Wed Jan 18 16:16:22 2023
Version 19.17.0.0.0
 
Copyright (c) 1982, 2022, Oracle.  All rights reserved.
 
 
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.17.0.0.0
 
SQL> set line 400
SQL> set pagesize 50
SQL> col STATUS for a20
SQL> col WRL_PARAMETER for a100
SQL> col STATUS for a20
SQL> col WALLET_TYPE for a10
SQL> select INST_ID,WRL_PARAMETER,STATUS, WALLET_TYPE, con_id from gv$encryption_wallet;
 
   INST_ID WRL_PARAMETER                                                                                        STATUS               WALLET_TYP     CON_ID
---------- ---------------------------------------------------------------------------------------------------- -------------------- ---------- ----------
         1 /u01/tde_wallets/ORCL/tde/                                                                     OPEN                 AUTOLOGIN           1
         1                                                                                                      OPEN                 AUTOLOGIN           2
         1                                                                                                      OPEN                 AUTOLOGIN           3
         2 /u01/tde_wallets/ORCL/tde/                                                                     OPEN                 AUTOLOGIN           1
         2                                                                                                      OPEN                 AUTOLOGIN           2
         2                                                                                                      OPEN                 AUTOLOGIN           3
 
6 rows selected.
 
SQL> select key_id, con_id, CREATOR_PDBNAME from gv$encryption_keys order by con_id,key_id;
 
KEY_ID                                                                             CON_ID CREATOR_PDBNAME
------------------------------------------------------------------------------ ---------- --------------------------------------------------------------------------------------------------------------------------------
AUC/+0hT/E9pv4p+f5I7KQcAAAAAAAAAAAAAAAAAAAAAAAAAAAAA                                    1 CDB$ROOT
AUC/+0hT/E9pv4p+f5I7KQcAAAAAAAAAAAAAAAAAAAAAAAAAAAAA                                    1 CDB$ROOT
AefCYCk3qk+Hv+MPk9F/dJgAAAAAAAAAAAAAAAAAAAAAAAAAAAAA                                    3 ORCL_P01
AefCYCk3qk+Hv+MPk9F/dJgAAAAAAAAAAAAAAAAAAAAAAAAAAAAA                                    3 ORCL_P01
 
SQL>

Check if Wallet and Autologin Wallet are existing and have correct Permissions

Files ewallet.p12 for Wallet and also cwallet.sso for autologin Wallet should be present in the Wallet directory. Owner oracle, permissions rw for Oracle and rwx on the directory

[oracle@dbhost01:ORCL]/DBA/users/oracle: ls -la /u01/tde_wallets/ORCL/tde/
total 136
drwxrwxr-x 2 oracle dba 20480 Jan 18 14:25 .
drwxrwxr-x 3 oracle dba 20480 Jan 11 15:20 ..
-rw------- 1 oracle dba  5704 Jan 18 14:25 cwallet.sso
-rw------- 1 oracle dba  2555 Jan 12 08:25 ewallet_2023011207253993.p12
-rw------- 1 oracle dba  3995 Jan 18 14:25 ewallet_2023011813250035.p12
-rw------- 1 oracle dba  5659 Jan 18 14:25 ewallet.p12

The files ewallet_BunchOfNumbers.p12 are Backups created by „ADMINISTER KEYSTORE“ Commands. Better keep them also and don’t delete them.

Try to show contents of Wallet as OS User oracle

You can show the Wallet contents with the following command as OS User „oracle“.

If a valid Autologin Wallet is missing, you will be prompted for the Wallet password.

[oracle@dbhost01:ORCL]/DBA/users/oracle: cd /u01/tde_wallets/ORCL/tde/
[oracle@dbhost01:ORCL]/u01/tde_wallets/ORCL/tde: orapki wallet display -wallet .
Oracle PKI Tool Release 19.0.0.0.0 - Production
Version 19.4.0.0.0
Copyright (c) 2004, 2022, Oracle and/or its affiliates. All rights reserved.
 
Requested Certificates:
Subject:        CN=oracle
User Certificates:
Oracle Secret Store entries:
ORACLE.SECURITY.DB.ENCRYPTION.AefCYCk3qk+Hv+MPk9F/dJgAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
ORACLE.SECURITY.DB.ENCRYPTION.AUC/+0hT/E9pv4p+f5I7KQcAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
ORACLE.SECURITY.DB.ENCRYPTION.MASTERKEY
ORACLE.SECURITY.DB.ENCRYPTION.MASTERKEY.E04A6D293EC456C4E0539618A10A9767
ORACLE.SECURITY.ID.ENCRYPTION.
ORACLE.SECURITY.KB.ENCRYPTION.
ORACLE.SECURITY.KM.ENCRYPTION.AefCYCk3qk+Hv+MPk9F/dJgAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
ORACLE.SECURITY.KM.ENCRYPTION.AUC/+0hT/E9pv4p+f5I7KQcAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
ORACLE.SECURITY.KT.ENCRYPTION.AefCYCk3qk+Hv+MPk9F/dJgAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
Trusted Certificates:
[oracle@dbhost01:ORCL]/u01/tde_wallets/ORCL/tde:

Be aware: The Wallet must only contain TDE Keys in the „Oracle Secret Store entries“.

If there are other types of contents in the Wallet, this could also lead to the Error „Wallet not open“, because then the Database will not be able to open the TDE Wallet.

Things that happen on christmas morning

Hello friends of the Oracle Database Administration and Troubleshooting,

here’s another story for you. Once upon a time on christmas day (that would be December the 25th) I got a call from work because I was the Oracle DBA On-Call that day. Yay.

Usually On-Call is pretty calm with no calls on most days, but this christmas we had an unfortunate coincidence, that the Unix Patch Days in our company fell right upon Christmas, so I got a nice little Christmas Present which was a call, because on one database server the Oracle Grid/ASM Software didn’t start correctly.

So, my Christmas Present was to investigate as to why this happened. And boy did I have fun, as this was a mixture of something quite unusual and later on a small human error which lead to a second error that came up after I found and corrected the first one.

So after checking that some processes like the HAS Daemon where running, I took a look into the Alert Log of the ASM Instance and I found some quite interesting Error Messages in between some status messages.

2022-12-25T06:38:21.239192+01:00
Errors in file /u01/oracle/diag/asm/+asm/+ASM/trace/+ASM_rbal_11371.trc:
ORA-15183: ASMLIB initialization error [driver/agent not installed]
2022-12-25T06:38:21.239280+01:00
WARNING: FAILED to load library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so


SQL> ALTER DISKGROUP ALL MOUNT /* asm agent call crs // {0:0:208} */
2022-12-25T06:38:21.326369+01:00
NOTE: Diskgroups listed in ASM_DISKGROUP are
ASMFR1


ASMDG1


2022-12-25T06:38:21.327622+01:00
NOTE: ASM instance +ASM is discoverable by local clients on node dbhost01
2022-12-25T06:38:21.336612+01:00
NOTE: cache registered group ASMDG1 1/0x41816392
NOTE: cache began mount (first) of group ASMDG1 1/0x41816392
NOTE: cache registered group ASMFR1 2/0x41816393
NOTE: cache began mount (first) of group ASMFR1 2/0x41816393
2022-12-25T06:38:21.371829+01:00
ERROR: no read quorum in group: required 2, found 0 disks
2022-12-25T06:38:21.372163+01:00
NOTE: cache dismounting (clean) group 1/0x41816392 (ASMDG1)
NOTE: messaging CKPT to quiesce pins Unix process pid: 11383, image: oracle@dbhost01 (TNS V1-V3)
NOTE: dbwr not being msg'd to dismount
NOTE: LGWR not being messaged to dismount
NOTE: cache dismounted group 1/0x41816392 (ASMDG1)
NOTE: cache ending mount (fail) of group ASMDG1 number=1 incarn=0x41816392
NOTE: cache deleting context for group ASMDG1 1/0x41816392
2022-12-25T06:38:21.408229+01:00
GMON dismounting group 1 at 2 for pid 24, osid 11383
2022-12-25T06:38:21.409859+01:00
ERROR: diskgroup ASMDG1 was not mounted

So, okay. Failed to load library. Let’s check if the library exists and how the permissions are set:

[oracle@dbhost01:+ASM]/home/oracle: ls -la /opt/oracle/extapi/64/asm/orcl/1/libasm.so
-rwxr-xr-x 1 root root 32992 Feb 25 2020 /opt/oracle/extapi/64/asm/orcl/1/libasm.so
[oracle@dbhost01:+ASM]/home/oracle: ldd /opt/oracle/extapi/64/asm/orcl/1/libasm.so
linux-vdso.so.1 (0x00007fff144c5000)
libc.so.6 => /lib64/libc.so.6 (0x00007f3a0c3fe000)
/lib64/ld-linux-x86-64.so.2 (0x00007f3a0c9cc000)
[oracle@dbhost01:+ASM]/home/oracle:

Well, this looks quite good. So, what else could it be?
I had no idea, so I asked the well known Senior DBA (okay…I typed the error into a well known search engine).
The results haven’t been good. Well, yes. I obviously checked file permissions already, but then there was a hint towards the oracleasm tool to show the actual configuration. So I did:

[oracle@dbhost01:+ASM]/home/oracle: /usr/sbin/oracleasm configure
ORACLEASM_ENABLED=true
ORACLEASM_UID=
ORACLEASM_GID=
ORACLEASM_SCANBOOT=true
ORACLEASM_SCANORDER="dm"
ORACLEASM_SCANEXCLUDE="sd"
ORACLEASM_SCAN_DIRECTORIES=""
ORACLEASM_USE_LOGICAL_BLOCK_SIZE="false"
[oracle@dbhost01:+ASM]/home/oracle:

So, let’s compare this with another ASM installation.

[oracle@dbhost02:+ASM]/home/oracle: /usr/sbin/oracleasm configure
ORACLEASM_ENABLED=true
ORACLEASM_UID=5000
ORACLEASM_GID=300
ORACLEASM_SCANBOOT=true
ORACLEASM_SCANORDER="dm"
ORACLEASM_SCANEXCLUDE="sd"
ORACLEASM_SCAN_DIRECTORIES=""
ORACLEASM_USE_LOGICAL_BLOCK_SIZE="false"

Well, there is a slight difference between these two. On the second server UID and GID have been set to the appropriate values for the oracle user and dba group.
So, by calling the command „/usr/sbin/oracleasm configure -i“ I was able to set the needed values.
ASM was now able to start and find all necessary devices. A quick reboot of the server to check wether this is reboot-proof showed the ASM looked good.

But now we had another problem, because the Database did not want to start. So, let’s take a look at what’s going on there.

The Alert Log showed the following errors:

Sys-V shared memory will be used for creating SGA




2022-12-25T06:57:35.084474+01:00
Error: ENOSPC while creating shared memory segment of size 23555211264 bytes
Check the system shared memory parameters
2022-12-25T06:57:35.084533+01:00
Error: ENOSPC while creating shared memory segment of size 23555211264 bytes
Check the system shared memory parameters
.
.
.
2022-12-25T06:57:35.084803+01:00
PAGESIZE AVAILABLE_PAGES EXPECTED_PAGES ALLOCATED_PAGES ERROR(s)
2022-12-25T06:57:35.084830+01:00
4K Configured 14 0 ORA-27125
2022-12-25T06:57:35.084874+01:00
2048K 11265 11265 5 ORA-27125
2022-12-25T06:57:35.084916+01:00


2022-12-25T06:57:35.084958+01:00
SGA: Realm creation failed

Okay. What is going on here? Aren’t these enough hugepages?

[oracle@dbhost01:ORCL]/u01/oracle/ORCL/logs: cat /proc/meminfo | grep Huge
AnonHugePages: 784384 kB
ShmemHugePages: 0 kB
FileHugePages: 0 kB
HugePages_Total: 11275
HugePages_Free: 31
HugePages_Rsvd: 21
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 23091200 kB

Well…something is using the databases hugepages.
Let’s check the parameters.
The Databases Alert Log says, it want’s about 20G.

Starting ORACLE instance (normal) (OS id: 17486)
2022-12-25T07:09:20.609947+01:00




Instance SGA_TARGET = 22528 MB and SGA_MAX_SIZE = 22528 MB

That’s fine. One Hugepage is 2M. So we should have enough if we look at the values above.
There is no other database running on the machine, so where have they gone?

Well…not database exactly, but we have an ASM Instance.
Let’s check the SGA Parameter there:

SQL> show parameter sga


NAME TYPE VALUE




sga_max_size big integer 22G
sga_target big integer 22G

Wow…22G for an ASM Instance, that’s a lot. Let’s scale that down to 2G and restart the ASM.

alter system set sga_max_size=2G scope=spfile;
alter system set sga_target=2G scope=spfile;
shutdown immediate;
startup;

After succesful restart of the ASM instance we have now been able to restart the Database.

But how did this happen?
Well, as often is the case, human error. The longer you are working in IT the more often things like these will happen.
It was planned to change the parameter in the Database, but the DBA changed into the ASM environment by accident without really registering what they did and changed the parameters there. After restarting the database and seeing that seemingly nothing changed they changed the parameters again, this time being in the correct environment and thus having succes after a database restart.

So, this was fine until the Server rebooted and thus the ASM Instance restarted with a bigger SGA.

I hope you like these kind of stories as they might give you hints how to investigate similar problems.