Loading

×Sorry to interrupt
Uncaught ReferenceError: addCTracking is not defined
throws at https://kb.vmware.com/resource/KM_TealiumTrackingJS?v=0.5:100:1

Refresh

Cookie Preference Center

General Information on cookies

When you visit our website, we use cookies to ensure that we give you the best experience. This information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies by clicking on the different category headings to find out more and change your settings. However, blocking some types of cookies may impact your experience on the site and the services we are able to offer. Further information can be found in our

Cookie Policy.

Strictly Necessary

Strictly Necessary

Always Active

Strictly Necessary

Strictly necessary cookies are always enabled since they are essential for our website to function. They enable core functionality such as security, network management, and website accessibility. You can set your browser to block or alert you about these cookies, but this may affect how the website functions. For more information please visit www.aboutcookies.org or www.allaboutcookies.org.

Cookie Details‎

Performance

Performance

Performance

Performance cookies are used to analyze the user experience to improve our website by collecting and reporting information on how you use it. They allow us to know which pages are the most and least popular, see how visitors move around the site, optimize our website and make it easier to navigate.

Cookie Details‎

Please confirm your settings by refreshing the page.

Functional

Functional

Functional

Functional cookies help us keep track of your past browsing choices so we can improve usability and customize your experience. These cookies enable the website to remember your preferred settings, language preferences, location and other customizable elements such as font or text size. If you do not allow these cookies, then some or all of these services may not function properly.

Cookie Details‎

Advertising

Advertising

Advertising

Advertising cookies are used to send you relevant advertising and promotional information. They may be set through our site by third parties to build a profile of your interests and show you relevant advertisements on other sites. These cookies do not directly store personal information, but their function is based on uniquely identifying your browser and internet device.

Cookie Details‎

Advertising Cookies

Consent Leg.Interest

Select All Vendors

Select All Vendors

Select All Hosts

Select All

Clear Filters

Information storage and access

Apply

Confirm My Choices Reject All Allow All

My Subscriptions

Loading

Knowledge Base

"503 Service Unavailable (Failed to connect to endpoint: _serverNamespace = /websso", vCenter Server stops responding due to Secure Token Service (vmware-stsd) OutOfMemoryError (78966)

Last Updated: 8/4/2020Categories: TroubleshootingTotal Views: 18689Language:
English

3 like this article

Subscribe to get an email
newsletter with new updates

3 people found this helpful Please provide article feedback:

Symptoms

You are running with vCenter Server 6.7 and intermittently UI client (vsphere-ui) becomes unresponsive with below error message

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http16LocalServiceSpecE:0x00007f36c81e2f50] _serverNamespace = /websso action = Allow _port = 7080)

You will notice multiple Java heap dumps under /var/log/vmware/sso

Example:
[ /var/log/vmware/sso ]# ls *.hprof
java_pid1774.hprof  java_pid1798.hprof  java_pid1803.hprof

You are using Smart Cards to login vCenter Server and CRL are being checked due to smart card login

Log file /var/log/vmware/sso/utils/vmware-stsd.err shows below entries

INFO: Server startup in 20715 ms

java.lang.OutOfMemoryError: GC overhead limit exceeded

at java.util.Arrays.copyOf(Arrays.java:3236)

at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)

at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)

at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)

at java.io.OutputStream.write(OutputStream.java:75)

at sun.security.util.DerValue.encode(DerValue.java:434)

at sun.security.util.DerValue.toByteArray(DerValue.java:867)

at sun.security.x509.X509CRLEntryImpl.parse(X509CRLEntryImpl.java:456)

at sun.security.x509.X509CRLEntryImpl.<init>(X509CRLEntryImpl.java:133)

at sun.security.x509.X509CRLImpl.parse(X509CRLImpl.java:1160)

at sun.security.x509.X509CRLImpl.<init>(X509CRLImpl.java:146)

at sun.security.provider.X509Factory.engineGenerateCRL(X509Factory.java:390)

at java.security.cert.CertificateFactory.generateCRL(CertificateFactory.java:497)

at com.vmware.identity.idm.server.clientcert.IdmCrlCache.downloadCrl(IdmCrlCache.java:168)

at com.vmware.identity.idm.server.clientcert.IdmCrlCache.refresh(IdmCrlCache.java:101)

at com.vmware.identity.idm.server.clientcert.TenantCrlCache.refreshCrl(TenantCrlCache.java:50)

at com.vmware.identity.idm.server.IdentityManager.refreshTenantCrlCache(IdentityManager.java:7322)

at com.vmware.identity.idm.server.IdentityManager.access$400(IdentityManager.java:217)

at com.vmware.identity.idm.server.IdentityManager$IdmCrlCachePeriodicChecker.run(IdentityManager.java:282)

java.lang.OutOfMemoryError: Java heap space

Log file /var/log/vmware/sso/vmware-identity-sts-default.log shows below entries

[2020-03-15T11:33:00.848+03:00 Thread-9 IdmCrlCachePeriodicChecker ERROR com.vmware.identity.idm.server.IdentityManager] IdmCrlCachePeriodicChecker refreshTenantCrl failed : GC overhead limit exceeded

java.lang.OutOfMemoryError: GC overhead limit exceeded

at sun.security.util.DerInputBuffer.dup(DerInputBuffer.java:66) ~[?:1.8.0_221]

at sun.security.util.DerValue.<init>(DerValue.java:284) ~[?:1.8.0_221]

at sun.security.util.DerInputStream.getDerValue(DerInputStream.java:451) ~[?:1.8.0_221]

at sun.security.x509.X509CRLEntryImpl.parse(X509CRLEntryImpl.java:459) ~[?:1.8.0_221]

at sun.security.x509.X509CRLEntryImpl.<init>(X509CRLEntryImpl.java:133) ~[?:1.8.0_221]

at sun.security.x509.X509CRLImpl.parse(X509CRLImpl.java:1160) ~[?:1.8.0_221]

Cause

This issue occurs when vCenter is given an extremely large CRL to process.

Resolution

This is a known issue affecting VMware vCenter Server 6.7 and 7.0.

Workaround

To workaround this issue, increase the heap size of vmware-stsd or change the certificate revocation checking method from CRL to OCSP.

Heap Size
Increase the heap size of vmware-stsd service by following below steps. These steps are applicable only from vCenter Server 6.7 Update 3 and above builds. Update the vCenter Server to 6.7 Update 3 or above builds before proceeding with below steps :

Connect to VCSA using SSH

Change the shell to Bash

Connected to service

* List APIs: "help api list"
* List Plugins: "help pi list"
* Launch BASH: "shell"

Command> shell
Shell access is granted to root
root@vcsa [ ~ ]#

Increase the Heap Size to 1024MB. Please note, you might have to increase the Memory (RAM) of vCenter Server before increasing the heap size. Refer to Related Information of this KB for more details.

cloudvm-ram-size -C 1024 vmware-stsd 

Restart the vmware-stsd Service

service-control --stop vmware-stsd && service-control --start vmware-stsd

Certification Revocation Checking
Updating the Certificate revocation settings to OCSP can also be used as a workaround. Your PKI infrastructure must support his method of certificate revocation to work properly. You can update the certificate revocation settings in the vSphere Client.

From the Home menu, select Administration.

Under Single Sign On, click Configuration.

Click Smart Card Authentication.

Under Smart card authentication settings, click Certificate revocation and click Edit.

Change the Revocation check to Use OCSP only and provide the location.

For more information on configuring certificate revocation, reference Set Revocation Policies for Smart Card Authentication in the VMware Documentation

Related Information

The cloudvm-ram-size utility is an utility of VC/PSC that given the total available memory divides and distributes said memory between individual processes that make up VC/PSC. Therefore, if cloudvm-ram-size is used to increase a memory size of one process, it will automatically decrease the size of all other processes because the total memory available is static.

Therefore, it is important to increase the total memory (RAM) of VC/PCS. By increasing total memory of VM that is hosting VC/PCS, and then increasing a memory footprint of a single process, results in not starving other processes of their memory, and does not introduce more instability into the system due other processes starving.

Follow below best practices for increasing the heap size of the process:

SSH into the VC/PSC as a root user

Execute the command: "cloudvm-ram-size -l". This will list the current memory distribute and division of memory between each service. Take note of these allocations

Increase the heap size by executing the command: "cloudvm-ram-size -C 1024 vmware-stsd"

Execute "cloudvm-ram-size -l" again and compare the result to make sure no other service has been adjusted down.

If other services were adjusted down too much (more than 5%), then increase the total available memory (RAM) of vCenter Server

Restart the VM and execute "cloudvm-ram-size -C 1024 vmware-stsd" again so that utility will recalculate memory allocations based on the new total memory

Check "cloudvm-ram-size -l" allocation again, and compare the distributions to the originals.

The goal is to make sure that cloudvm-ram-size has enough total memory to work with and that it adjusts vmware-stsd up without compromising other services and adjusting anything else down.

Detectable by VMware SkylineTM

Actions

Copy

link to clipboard
copied!

Print

Language:
English

Attachments

Additional Resources

KB
•
End Of General Availability of the Free vSphere Hypervisor (ESXi 7.x and 8.x) (2107518)

KB
•
将 Cloud Director 9.5 迁移到 10.0 的过程中，出现“无法读取或写入配置文件: 密钥库已被篡改，或密码不正确”错误 (78924)

KB
•
VMware Update Manager の診断情報の収集 (1003693)

KB
•
VMware Intelligent Hub 20.04 for iOS リリースのお知らせ (78889)

KB
•
从 6.0 迁移到 6.7 失败，并显示错误“无法加载 sqljdbc_auth.dll 原因: no sqljdbc_auth in java.library.path” (76091)

Results 1-5 of 5

Ask The Community

Get answers quickly from VMware experts in the community

Post Subject

CONTINUE IN COMMUNITIES

Related Products:

VMware vCenter Server

Related Versions:

VMware vCenter Server 7.0.x

VMware vCenter Server 6.7.x

Take Our Survey

Support Assistant