ORA-27125 When attempting to start 12c Database On Solaris 11

Issue: When attempting to start the 12c Database on Solaris 11, we hit the below error.

ORA-27125: unable to create shared memory segment
Invalid Argument.

Cause: The issue was due to an undersized project.max-shm-memory setting, where project.max-shm-memory was set to 8 GB while the RAM was 200 GB.

Solution:

Increase project.shm-max-memory to a value greater than 1/4 of physical memory.

The project.max-shm-memory resource control limits the total amount of shared memory of one project.
The PROJECT.MAX-SHM-MEMORY should be set larger than the sum of all segments used by the project.

In case, we don't set project.shm-max-memory but shmmax is defined, system assumes "shm-max-memory" as shmmax*shmmni. If shmmax is undefined as well,
then default value is 1/4 of physical memory:

 http://docs.oracle.com/cd/E19082-01/819-2724/6n50b0793/index.html


For example, to change the setting for project.max-shm-memory to 10 GB for the project default you can use either the 'prctl' command or the 'projmod' command. Changes made with the 'prctl' command are good only until system reboot. The 'projmod' command should be used to make changes that persist across reboot.

The following command requires reboot:

projmod -sK "project.max-shm-memory=(privileged,10G,deny)" user.dba

OR

To modify a parameter until next system reboot:

prctl -n project.max-shm-memory -v 6gb -r -i project user.dba

What is Cluster Time Synchronization Service CTSS ?

The Cluster Time Synchronization Service (CTSS) is one of the newprocesses in the Oracle Clustware 11gR2 layer , the CTSS is installed duringthe GRID infrastructure installation.

The Daemon is responsible for the time management within the cluster. Makingsure each node in the cluster is using the same time. But before doing this theCTSS will first determine if a Network Time Protocol (NTP) daemon is running inthe system. If this is not the case CTSS will run in active mode and performthe same task as the NTP daemon. If NTP daemon is running CTSS will be started,but in observer mode. The reason Oracle has implemented CTSS is to make suretime management within the cluster is performed. CTSS is Oracles implementationof the NTP daemons normally setup by the OS administrators

The first node in the cluster where CTSS is started will become the mastertime manager. Other CTSS daemons will communicate with the master CTSS andvalidate the time. If a time difference between hosts in the cluster isdetected it will adjust the time, similar to the NTP daemon. CTSS will never goback into time. If time differences are taking place, it will be reported inthe alert.log. If the time difference between hosts during startup is too large( 1000 msec) Oracle Clusterware will not startup on the newly joined nodes. Analert will be written to the alert.log of the Oracle Clusterware home e.g. (/u01/app/11.2.0/grid/log//alert.log).You manually need to modify the time and start Oracle Clusterware in thatcase.

Solution

How does Oracle Clusterware decide to start CTSS in observe or active mode?

CTSS is a process which runs as root on each node. As soon as OracleClusterware is started the CTSS daemon validates if the /etc/ntp.conf fileexists, if this file exists CTSS will run in observer mode. To determine if NTPdaemon is active cluster verification utility to get that part of heinformation. 

root 3582 1 0 11:24 ? 00:00:12 /u01/app/11.2.0/grid/bin/octssd.binreboot

How to validate CTSS is running in observer mode or active mode?

Of course there is the alert.log on cluster level which will report the statusas well as the trace file. But the two easiest way is to use crsctl or cluvfufor this purpose. Crsctl will tell if CTSS is running and when the role isActive report the Offset in msec. Cluvfy will report much more information.Below are examples:

$ crsctl check ctss
$ cluvfy comp clocksync -n all
$ cluvfy comp clocksync -verbose

This will list the status of CTSS, if it is running and after that it willreport the current mode, active or observer. If the mode is active it will alsoreport if a time synchronization issue exists. 

Sample output using CRSCTL:

[oracle@server1 log]$  /u01/app/11.2.0/grid/bin/crsctlcheck ctss
CRS-4701: The Cluster Time Synchronization Service is in Active mode.
CRS-4702: Offset (in msec): 0

Sample output CTSS in Observer mode but NTP not active:

[oracle@server1 server1]$ cluvfy comp clocksync
Verifying Clock Synchronization across the cluster nodes 

Checking if Clusterware is installed on all nodes...
Check of Clusterware install passed
Checking if CTSS Resource is running on all nodes...

CTSS resource check passed

Querying CTSS for time offset on all nodes...
Query of CTSS for time offset passed
Check CTSS state started...

CTSS is in Observer state. Switchingover to clock synchronization checks using NTP

Starting Clock synchronization checks using NetworkTime Protocol(NTP)...
NTP Configuration file check started...
NTP Configuration file check passed 

Checking daemon liveness...
Liveness check failed for "ntpd"
Check failed on nodes: server1

PRVF-5415: Check to see if NTP daemon is running failed
Clock synchronization check using Network Time Protocol(NTP) failed
PRVF-9652: Cluster Time Synchronization Services checkfailed

Verification of Clock Synchronization across thecluster nodes was unsuccessful on all the specified nodes. 

Here we see the error messagesPRVF-5415followed byPRVF-9652,indicating there is an issue with NTP. This is correct as it wasnotconfigured.

Sample output CTSS in Observer mode but NTP not active in verbose mode:
[oracle@server1]$ cluvfy comp clocksync -verbose
Verifying Clock Synchronization across the cluster nodes
Checking if Clusterware is installed on all nodes...
Check of Clusterware install passed
Checking if CTSS Resource is running on all nodes...
Check: CTSS Resource running on all nodes

Node Name Status

------------------------------------------------------------
server1 passed
Result: CTSS resource check passed

Querying CTSS for time offset on all nodes...
Result: Query of CTSS for time offset passed
Check CTSS state started...
Check: CTSS state
Node Name State

------------------------------------------------------------

server1 Observer

CTSS is in Observer state. Switching over to clocksynchronization checks using NTP 
Starting Clock synchronization checks using NetworkTime Protocol(NTP)...

NTP Configuration file check started...
The NTP configuration file "/etc/ntp.conf" is available on all nodes
NTP Configuration file check passed

Checking daemon liveness...
Check: Liveness for "ntpd"
Node Name Running?

------------------------------------------------------------

server1 no
Result: Liveness check failed for "ntpd"
PRVF-5415 : Check to see if NTP daemon is running failed
Result: Clock synchronization check using Network Time Protocol(NTP) failed
PRVF-9652 : Cluster Time Synchronization Services check failed

Verification of Clock Synchronization across thecluster nodes was unsuccessful on all the specified nodes. 

Same result as above, although easier to see the State and that NTP is notactive.

Sample output when CTSS is in active mode using cluvfy:

[oracle@server1~]$ cluvfy comp clocksync -verbose

Verifying Clock Synchronization across the clusternodes 
Checking if Clusterware is installed on all nodes...
Check of Clusterware install passed

Checking if CTSS Resource is running on all nodes...
Check: CTSS Resource running on all nodes
NodeName                            Status
------------------------------------  ------------------------
 server1                              passed
Result: CTSS resource check passed

Querying CTSS for time offset on all nodes...
Result: Query of CTSS for time offset passed
Check CTSS state started...
Check: CTSS state
  NodeName                            State
  ------------------------------------  ------------------------
  server1                              Active
CTSS is in Active state. Proceeding with check of clock time offsets on allnodes...
Reference Time Offset Limit: 1000.0 msecs
Check: Reference Time Offset
  Node Name     TimeOffset              Status
  ------------  ------------------------ ------------------------
  server1      0.0                      passed

Time offset is within the specified limits on thefollowing set of nodes:
"[server1]"
Result: Check of clock time offsets passed

Oracle Cluster Time Synchronization Services checkpassed

Verification of Clock Synchronization across the cluster nodes was successful.
[oracle@server1 ~]$ 

Sample output when time offset is violated using cluvfy:

[oracle@server1 ~]$ cluvfy comp clocksync -n all
Verifying Clock Synchronization across the cluster nodes
Checking if Clusterware is installed on all nodes...
Check of Clusterware install passed
Checking if CTSS Resource is running on all nodes...

CTSS resource check passed
Querying CTSS for time offset on all nodes...
Query of CTSS for time offset passed

Check CTSS state started...
CTSS is in Active state. Proceeding with check of clock time offsets on allnodes...
PRVF-9661 : Time offset is NOT within the specified limits on the followingnodes:"[server2]"

PRVF-9652 : Cluster Time Synchronization Services check failed

Verification of Clock Synchronization across the cluster nodes wasunsuccessful.
Checks did not pass for the following node(s): server2

 [oracle@server2 ~]$ crsctl check ctss
CRS-4701: The Cluster Time Synchronization Service is in Active mode.
CRS-4702: Offset (in msec): 13700 

How to switch between observer mode to active mode in either direction?

To execute this task it is very simple. Make sure the /etc/ntp.conf fileis not available. Based on the existence of ntp.conf file CTSS will be inactive or observe mode. So remove/rename the file. 

Every 30 second CTSS will check if the current state is still correct.When CTSS discovers this state is incorrect it will automatically switch fromobserver to active mode when the file is removed. If you don’t want to use theCTSS for the time management create the ntp.conf file again, and on the fly thestate will change.

Trace output explaining the above:

2009-09-28 16:27:13.768: [CTSS][3010210704]sclsctss_gvss1: NTP default config file found
2009-09-28 16:27:13.768: [CTSS][3010210704]sclsctss_gvss8: Return [0] and NTP status [2].
2009-09-28 16:27:13.768: [ CTSS][3010210704]ctss_check_vendor_sw: Vendor timesync software is detected. status [2].
2009-09-28 16:27:15.375: [ CTSS][3020700560]ctsscomm_prh: Handler called
[ CTSS][3020700560]ctss_process_request_handler: Master: Received sync messageevent
2009-09-28 16:27:15.375: [ CTSS][3020700560]ctsscomm_pi: Received sync msg
2009-09-28 16:27:15.375: [ CTSS][3020700560]ctsscomm_pi: Received from slave (mode [0x46] nodenum [2] hostname [server2] )
2009-09-28 16:27:23.378: [ CTSS][3020700560]ctsscomm_prh: Handler called
[ CTSS][3020700560]ctss_process_request_handler: Master: Received sync messageevent
2009-09-28 16:27:23.378: [ CTSS][3020700560]ctsscomm_pi: Received sync msg
2009-09-28 16:27:23.378: [ CTSS][3020700560]ctsscomm_pi: Received from slave (mode [0x46] nodenum [2] hostname [server2] )
2009-09-28 16:27:31.389: [ CTSS][3020700560]ctsscomm_prh: Handler called
[ CTSS][3020700560]ctss_process_request_handler: Master: Received sync messageevent
2009-09-28 16:27:31.389: [ CTSS][3020700560]ctsscomm_pi: Received sync msg
2009-09-28 16:27:31.389: [ CTSS][3020700560]ctsscomm_pi: Received from slave (mode [0x46] nodenum [2] hostname [server2] )
2009-09-28 16:27:39.389: [ CTSS][3020700560]ctsscomm_prh: Handler called
[ CTSS][3020700560]ctss_process_request_handler: Master: Received sync messageevent
2009-09-28 16:27:39.389: [ CTSS][3020700560]ctsscomm_pi: Received sync msg
2009-09-28 16:27:39.389: [ CTSS][3020700560]ctsscomm_pi: Received from slave (mode [0x46] nodenum [2] hostname [server2] )
2009-09-28 16:27:43.383: [ CTSS][2978741136]ctss_checkcb: clsdm requested checkalive. Returns [6e]
2009-09-28 16:27:43.769: [ CTSS][3010210704]sclsctss_gvss2: NTP default pid file not found<==== here /etc/ntp.conf is renamed.
2009-09-28 16:27:43.770: [CTSS][3010210704]sclsctss_gvss8: Return [0] and NTP status [1].
2009-09-28 16:27:43.770: [ CTSS][3010210704]ctss_check_vendor_sw: Vendor timesync software is not detected. status [1].
2009-09-28 16:27:43.786: [ CTSS][3010210704]ctsselect_determine_role: node [1]with mode [0x4e] in the modes table
2009-09-28 16:27:43.799: [ CTSS][3010210704]ctsselect_determine_role: node [2]with mode [0x46] in the modes table
2009-09-28 16:27:43.799: [ CTSS][3010210704]ctsselect_determine_role: Vendor time synchronizationsoftware is not detected on any node in the cluster. Switched to active role.

Output from the alert.log when there is en Time synchronizationissue.[ctssd(3416)]CRS-2408:The clock on host server2 has been updated by theCluster Time Synchronization Service to be synchronous with the mean clustertime.

2009-10-01 13:50:51.727[ctssd(3416)]CRS-2411:The Cluster TimeSynchronization Service will take a long time to perform time synchronizationas local time is significantly different from mean cluster time.Details in/u01/app/11.2.0/grid/log/server2/ctssd/octssd.log.

You can find similar output in the operation system logfile. 

Remark: 
- CTSS will run in observe mode as soon as an NTPconfiguration is found. This doesn’t tell if the NTP daemon is really workingproperly. Be aware of this! Default Linux installation will have the ntp.conffile in /etc/
- Use either NTP configuration or CTSS for time management. Don’t “play’ withCTSS on production environments. So discuss what you require. 

Additional trace information:

When you look into the details when using cluvfy you will find out thefollowing checks are performed.

1. Validate     if this is a cluster environment, does ocr.loc exists?
2. Check     if CTSS is running using: /u01/app/11.2.0/grid/bin/crsctl check ctss 
3. Check     if ntp configuration file exists (when found mark as exists):/tmp/CVU_11.2.0.1.0_oracle/exectask.sh     -chkfile /etc/ntp.conf
4. Validate     if NTP daemon is really active using:/tmp/CVU_11.2.0.1.0_oracle/exectask.sh     -chkalive ntpd

Alert.log will show when inobserver mode: 

[ctssd(3582)]CRS-2403:The Cluster TimeSynchronization Service on host server1 is in observer mode.
2009-09-27 21:24:46.766[ctssd(3582)]CRS-2407:The new Cluster TimeSynchronization Service reference node is host server2.
2009-09-27 21:24:46.938[ctssd(3582)]CRS-2412:The Cluster Time SynchronizationService detects that the local time is significantly different from the meancluster time. Details in /u01/app/11.2.0/grid/log/server1/ctssd/octssd.log.
2009-09-27 21:24:46.986[ctssd(3582)]CRS-2409:The clock on host server1 is notsynchronous with the mean cluster time. No action has been taken as the ClusterTime Synchronization Service is running in observer mode.
2009-09-27 21:24:47.277[ctssd(3582)]CRS-2401:The Cluster Time SynchronizationService started on host server1.
2009-09-27 21:26:38.926.....
[ctssd(3582)]CRS-2409:The clock on host server1 is not synchronous with themean cluster time. No action has been taken as the Cluster Time SynchronizationService is running in observer mode.
2009-09-27 22:00:32.725[ctssd(3582)]CRS-2409:The clock on host server1 is notsynchronous with the mean cluster time. No action has been taken as the ClusterTime Synchronization Service is running in observer mode.

Here you can read the state of CTSS as well, but also see there is asynchronization issue No action is taken to fix this issue as CTSS is inobserver mode. 

Alert.log will show when in active mode:

[ctssd(3578)]CRS-2401:TheCluster Time Synchronization Service started on host server1 is in active mode. 

Advise: As time management is extremely important in clusterenvironments you must make sure NTP is running correctly. In general NTP willbe configured as standard on each system. But as soon as you are not able toconfirm this I would advise that the file /etc/ntp.conf is removed so CTSS willtake over this responsibility and become active. Make sure this is done on eachnode in the cluster.

ASM DISKGROUP SHOWS USABLE_FILE_MB VALUE IN NEGATIVE

A negative value for USABLE_FILE_MB means that you do not have sufficient free space to tolerate a disk failure. If a disk were to fail, the subsequent rebalance would run out of space before full redundancy could be restored to all files.

The box i was working on was exadata server eighth rack, so it had 3 storage server. Each storage server on an exadata server has 12 cell disk. Grid disk are created within Cell Disks.In a simple configuration, One Grid Disk can be created per Cell Disk and Grid disks are what the storage cell presents to db servers. So basically

GRID DISK = ASM DISK.

When creating disk groups, ASM automatically puts all grid disks from the same storage cell into the same failgroup. The failgroup is then named after the storage cell.

[oracle@test~]$ asmcmd lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  NORMAL  N         512   4096  4194304  40697856  8464936         13565952        -2550508              0             N  DATA1/
MOUNTED  NORMAL  N         512   4096  4194304    415296   367220           138432          114394              0             Y  DBFS_DG/
MOUNTED  NORMAL  N         512   4096  4194304  10176480  9018276          3392160         2813058              0             N  RECO1/

compute sum Label total_FG of total_mb on FAILGROUP
compute sum Label total of total_mb on report
col diskgroup for a20
col failgroup for a30
col name for a30
select g.name diskgroup, d.failgroup,  d.name, d.total_mb from v$asm_disk d, v$asm_diskgroup g where g.name = 'DATA1' and d.GROUP_NUMBER = g.GROUP_NUMBER order by g.name, d.failgroup;

DISKGROUP            FAILGROUP                      NAME                                   TOTAL_MB
-------------------- ------------------------------ ------------------------------ ----------------
DATA1               CELL01                         DATA1_CD_00_CELL01             2260992
DATA1                                              DATA1_CD_05_CELL01             2260992
DATA1                                              DATA1_CD_03_CELL01             2260992
DATA1                                              DATA1_CD_04_CELL01             2260992
DATA1                                              DATA1_CD_01_CELL01             2260992
DATA1                                              DATA1_CD_02_CELL01             2260992
                     ******************************                                ----------------
                     total_FG                                                              13565952
DATA1               CELL02                         DATA1_CD_01_CELL02             2260992
DATA1                                              DATA1_CD_05_CELL02             2260992
DATA1                                              DATA1_CD_02_CELL02             2260992
DATA1                                              DATA1_CD_03_CELL02             2260992
DATA1                                              DATA1_CD_00_CELL02             2260992
DATA1                                              DATA1_CD_04_CELL02             2260992
                     ******************************                                ----------------
                     total_FG                                                              13565952
DATA1               CELL03                         DATA1_CD_02_CELL03             2260992
DATA1                                              DATA1_CD_05_CELL03             2260992
DATA1                                              DATA1_CD_01_CELL03             2260992
DATA1                                              DATA1_CD_04_CELL03             2260992
DATA1                                              DATA1_CD_03_CELL03             2260992
DATA1                                              DATA1_CD_00_CELL03             2260992
                     ******************************                                ----------------
                     total_FG                                                              13565952
                                                                                   ----------------
total                                                                                      40697856

For DATA1 diskgroup the USABLE_FILE_MB shows value in Negative (-2550508 MB).

SQL> select name, state, type, total_mb, free_mb, required_mirror_free_mb req_free,  usable_file_mb use_mb from v$asm_diskgroup where name = 'DATA1';

NAME                      STATE       TYPE     TOTAL_MB    FREE_MB   REQ_FREE     USE_MB
------------------------- ----------- ------ ---------- ---------- ---------- ----------
DATA1                      MOUNTED     NORMAL   40697856    8464936   13565952   -2550508
                                                                                                              ----------
total                                                                                                           40697856

TOTAL_MB:- Refers to total capacity of the diskgroup
FREE_MB :- Refers to raw free space available in diskgroup in MB.

FREE_MB = (TOTAL_MB – (HOT_USED_MB + COLD_USED_MB))

REQUIRED_MIRROR_FREE_MB :- Indicates how much free space is required in an ASM disk group to restore redundancy after the failure of an ASM disk or ASM failure group.In exadata it is the disk capacity of one failure group.

USABLE_FILE_MB :- Indicates how much space is available in an ASM disk group considering the redundancy level of the disk group.

Its calculated as :-

USABLE_FILE_MB=(FREE_MB – REQUIRED_MIRROR_FREE_MB ) / 2 –> For Normal Redundancy
USABLE_FILE_MB=(FREE_MB – REQUIRED_MIRROR_FREE_MB ) / 3 –> For High Redundancy

Also to note here is ASM diskgroup do not set aside the space based on reuqired_mirror_free_mb. Its merely calculated and used to derive usable_file_mb.

While reading Mos Doc Id 1551288.1 i came across some interesting terms and script which i wanted to share to everyone (atleast some of you who might not have been familiar)

Failure coverage refers to the amount of space in a disk group that will be used to re-mirror data in the event of some storage failure.

1. Disk Failure Coverage :- Refers to having enough free space to allow data to be re-mirrored (rebalanced) after a single disk failure in Normal redundancy.

2. Cell Failure Coverage :- Refers to having enough free space to allow data to be re-mirrored after loss of One entire Cell Disk.

Reserving space in the disk group means that you monitor the disk group to ensure that FREE_MB never goes below minimum amount needed for disk or cell failure coverage.

I ran the script provided in Mos Docid 1551288 and below was the output :-

Description of Derived Values:
One Cell Required Mirror Free MB : Required Mirror Free MB to permit successful rebalance after losing largest CELL regardless of redundancy type
Disk Required Mirror Free MB     : Space needed to rebalance after loss of single or double disk failure (for normal or high redundancy)
Disk Usable File MB              : Usable space available after reserving space for disk failure and accounting for mirroring
Cell Usable File MB              : Usable space available after reserving space for SINGLE cell failure and accounting for mirroring
.  .  .
ASM Version: 11.2.0.4
.  .  .
----------------------------------------------------------------------------------------------------------------------------------------------------
|          |         |     |          |            |            |            |Cell Req'd  |Disk Req'd  |            |            |    |    |       |
|          |DG       |Num  |Disk Size |DG Total    |DG Used     |DG Free     |Mirror Free |Mirror Free |Disk Usable |Cell Usable |    |    |PCT    |
|DG Name   |Type     |Disks|MB        |MB          |MB          |MB          |MB          |MB          |File MB     |File MB     |DFC |CFC |Util   |
----------------------------------------------------------------------------------------------------------------------------------------------------
|DATA1    |NORMAL   |   18| 2,260,992|  40,697,856|  32,233,944|   8,463,912|  14,922,547|   2,761,008|   2,851,452|  -3,229,318|PASS|FAIL|  79.2%|
|DBFS_DG  |NORMAL   |   12|    34,608|     415,296|      48,076|     367,220|     152,275|      59,425|     153,898|     107,472|PASS|PASS|  11.6%|
|RECO1    |NORMAL   |   18|   565,360|  10,176,480|   1,171,220|   9,005,260|   3,731,376|     703,460|   4,150,900|   2,636,942|PASS|PASS|  11.5%|
----------------------------------------------------------------------------------------------------------------------------------------------------
Cell Failure Coverage Freespace Failures Detected. Warning Message Follows.
Enough Free Space to Rebalance after loss of ONE cell: WARNING (However, cell failure is very rare)
.  .  .
Script completed.

https://aprakash.wordpress.com/2014/09/17/asm-diskgroup-shows-usable_file_mb-value-in-negative/