Getting continuous "MBean attribute access denied" in SOA logs due to OEM Agent
Recently, we started getting continuous errors every few seconds in our Oracle SOA Suite logs (specifically in the SOA managed server standard out log).
The entire exception is shown here:
<Sep 11, 2019 9:27:41 PM EDT> <Error> <oracle.as.jmx.framework.generic.spi.security.AbstractMBeanSecurityInterceptor> <J2EE JMX-46335> <MBean attribute access denied.
MBean: oracle.soa.config:name="PayPal",j2eeType=SOAFolder,Application=soa-infra
Getter for attribute State
Detail: access denied ("oracle.fabric.permission.CompositePermission" "PayPal" "read")
java.security.AccessControlException: access denied ("oracle.fabric.permission.CompositePermission" "PayPal" "read")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
at java.security.AccessController.checkPermission(AccessController.java:884)
at oracle.security.jps.util.JpsAuth$AuthorizationMechanism$3.checkPermission(JpsAuth.java:527)
at oracle.security.jps.util.JpsAuth.checkPermission(JpsAuth.java:587)
at oracle.security.jps.util.JpsAuth.checkPermission(JpsAuth.java:623)
at oracle.fabric.permission.internal.InternalSOAPermissionCheckHelper$2.run(InternalSOAPermissionCheckHelper.java:204)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at oracle.fabric.permission.internal.InternalSOAPermissionCheckHelper.internalCheckSOAPermission(InternalSOAPermissionCheckHelper.java:202)
at oracle.fabric.permission.internal.InternalSOAPermissionCheckHelper.checkSOAPermission(InternalSOAPermissionCheckHelper.java:173)
at oracle.fabric.permission.management.SOAPermissionCheckPluginFactory$SOAMBeanCustomSecurityPlugin.checkGetAttribute(SOAPermissionCheckPluginFactory.java:60)
at oracle.as.jmx.framework.MBeanCustomSecurityHelper.checkGetAttribute(MBeanCustomSecurityHelper.java:193)
at oracle.as.jmx.framework.generic.spi.security.AbstractMBeanSecurityInterceptor.checkAttributeAccess(AbstractMBeanSecurityInterceptor.java:269)
at oracle.as.jmx.framework.generic.spi.security.AbstractMBeanSecurityInterceptor.internalGetAttribute(AbstractMBeanSecurityInterceptor.java:128)
at oracle.as.jmx.framework.generic.spi.interceptors.AbstractMBeanInterceptor.doGetAttribute(AbstractMBeanInterceptor.java:86)
at oracle.security.jps.ee.jmx.JpsJmxInterceptor$GetAttributeDelegator.delegate(JpsJmxInterceptor.java:634)
at oracle.security.jps.ee.jmx.JpsJmxInterceptor$3.run(JpsJmxInterceptor.java:540)
at java.security.AccessController.doPrivileged(Native Method)
at oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:315)
at oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:649)
at oracle.security.jps.ee.jmx.JpsJmxInterceptor.jpsInternalInvoke(JpsJmxInterceptor.java:558)
at oracle.security.jps.ee.jmx.JpsJmxInterceptor.internalGetAttribute(JpsJmxInterceptor.java:265)
at oracle.as.jmx.framework.generic.spi.interceptors.AbstractMBeanInterceptor.doGetAttribute(AbstractMBeanInterceptor.java:86)
at oracle.as.jmx.framework.generic.spi.interceptors.ContextClassLoaderMBeanInterceptor.internalGetAttribute(ContextClassLoaderMBeanInterceptor.java:63)
at oracle.as.jmx.framework.generic.spi.interceptors.AbstractMBeanInterceptor.doGetAttribute(AbstractMBeanInterceptor.java:86)
at oracle.as.jmx.framework.generic.spi.interceptors.MBeanRestartInterceptor.internalGetAttribute(MBeanRestartInterceptor.java:67)
at oracle.as.jmx.framework.generic.spi.interceptors.AbstractMBeanInterceptor.doGetAttribute(AbstractMBeanInterceptor.java:86)
at oracle.as.jmx.framework.standardmbeans.spi.OracleStandardEmitterMBean.getAttribute(OracleStandardEmitterMBean.java:631)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)
at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$17.run(WLSMBeanServerInterceptorBase.java:466)
at java.security.AccessController.doPrivileged(Native Method)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.getAttribute(WLSMBeanServerInterceptorBase.java:464)
at weblogic.management.mbeanservers.internal.JMXContextInterceptor.getAttribute(JMXContextInterceptor.java:165)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$17.run(WLSMBeanServerInterceptorBase.java:466)
at java.security.AccessController.doPrivileged(Native Method)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.getAttribute(WLSMBeanServerInterceptorBase.java:464)
at weblogic.management.mbeanservers.internal.SecurityInterceptor.getAttribute(SecurityInterceptor.java:294)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$17.run(WLSMBeanServerInterceptorBase.java:466)
at java.security.AccessController.doPrivileged(Native Method)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.getAttribute(WLSMBeanServerInterceptorBase.java:464)
at weblogic.management.mbeanservers.internal.MBeanCICInterceptor.access$101(MBeanCICInterceptor.java:38)
at weblogic.management.mbeanservers.internal.MBeanCICInterceptor$1.call(MBeanCICInterceptor.java:134)
at weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:284)
at weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:269)
at weblogic.management.mbeanservers.internal.MBeanCICInterceptor.getAttribute(MBeanCICInterceptor.java:130)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$17.run(WLSMBeanServerInterceptorBase.java:466)
at java.security.AccessController.doPrivileged(Native Method)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.getAttribute(WLSMBeanServerInterceptorBase.java:464)
at weblogic.management.mbeanservers.internal.PartitionJMXInterceptor.getAttribute(PartitionJMXInterceptor.java:303)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$17.run(WLSMBeanServerInterceptorBase.java:466)
at java.security.AccessController.doPrivileged(Native Method)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.getAttribute(WLSMBeanServerInterceptorBase.java:464)
at weblogic.management.mbeanservers.internal.CallerPartitionContextInterceptor.getAttribute(CallerPartitionContextInterceptor.java:177)
at weblogic.management.jmx.mbeanserver.WLSMBeanServer.getAttribute(WLSMBeanServer.java:283)
at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder$5$1.run(JMXConnectorSubjectForwarder.java:308)
at java.security.AccessController.doPrivileged(Native Method)
at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder$5.run(JMXConnectorSubjectForwarder.java:306)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:368)
at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder.getAttribute(JMXConnectorSubjectForwarder.java:301)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1445)
at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)
at java.security.AccessController.doPrivileged(Native Method)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1408)
at javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:639)
at javax.management.remote.rmi.RMIConnectionImpl_WLSkel.invoke(Unknown Source)
at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:645)
at weblogic.rmi.internal.BasicServerRef$2.run(BasicServerRef.java:534)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:368)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:163)
at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:531)
at weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:137)
at weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:348)
at weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:333)
at weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:54)
at weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41)
at weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:617)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:397)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:346)
>
<Sep 11, 2019 9:27:41 PM EDT> <Warning> <RMI> <BEA-080003> <A RuntimeException was generated by the RMI server: javax.management.remote.rmi.RMIConnectionImpl.getAttribute(Ljavax.management.ObjectName;Ljava.lang.String;Ljavax.security.auth.Subject;)
javax.management.RuntimeMBeanException: java.lang.SecurityException: MBean attribute access denied.
MBean: oracle.soa.config:name="PayPal",j2eeType=SOAFolder,Application=soa-infra
Getter for attribute State
Detail: access denied ("oracle.fabric.permission.CompositePermission" "PayPal" "read").
javax.management.RuntimeMBeanException: java.lang.SecurityException: MBean attribute access denied.
MBean: oracle.soa.config:name="PayPal",j2eeType=SOAFolder,Application=soa-infra
Getter for attribute State
Detail: access denied ("oracle.fabric.permission.CompositePermission" "PayPal" "read")
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:839)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:852)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:651)
at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$17.run(WLSMBeanServerInterceptorBase.java:466)
Truncated. see log file for complete stacktrace
Caused By: java.lang.SecurityException: MBean attribute access denied.
MBean: oracle.soa.config:name="PayPal",j2eeType=SOAFolder,Application=soa-infra
Getter for attribute State
Detail: access denied ("oracle.fabric.permission.CompositePermission" "PayPal" "read")
at oracle.as.jmx.framework.generic.spi.security.AbstractMBeanSecurityInterceptor.checkAttributeAccess(AbstractMBeanSecurityInterceptor.java:312)
at oracle.as.jmx.framework.generic.spi.security.AbstractMBeanSecurityInterceptor.internalGetAttribute(AbstractMBeanSecurityInterceptor.java:128)
at oracle.as.jmx.framework.generic.spi.interceptors.AbstractMBeanInterceptor.doGetAttribute(AbstractMBeanInterceptor.java:86)
at oracle.security.jps.ee.jmx.JpsJmxInterceptor$GetAttributeDelegator.delegate(JpsJmxInterceptor.java:634)
at oracle.security.jps.ee.jmx.JpsJmxInterceptor$3.run(JpsJmxInterceptor.java:540)
Truncated. see log file for complete stacktrace
Caused By: java.security.AccessControlException: access denied ("oracle.fabric.permission.CompositePermission" "PayPal" "read")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
at java.security.AccessController.checkPermission(AccessController.java:884)
at oracle.security.jps.util.JpsAuth$AuthorizationMechanism$3.checkPermission(JpsAuth.java:527)
at oracle.security.jps.util.JpsAuth.checkPermission(JpsAuth.java:587)
at oracle.security.jps.util.JpsAuth.checkPermission(JpsAuth.java:623)
Truncated. see log file for complete stacktrace
>By observing the log, you will notice the following error in specific:
access denied ("oracle.fabric.permission.CompositePermission" "PayPal" "read")Here, "PayPal" is the name of our SOA partitions. The error repeats repeatedly for all partitions in our domain. So some process is trying to read some composite information from all our partitions.
Analysis
The culprit here turned out to be the Oracle Enterprise Manager (OEM) Cloud Control 13c. Agent.
The OEM Agent is configured against the weblogic domain not using the weblogic account, but a custom created account called oemagent. This account was giving the Operators group. It appears that the OEM Agent would probably need more privileges.
Resolution
- Shut down and clear the state of the agent.
emctl stop agent
emctl clearstate agent2. Assign the Administrators group to the oemagent account (note that oemagent is a weblogic account, and this account is used by the agent to log in to the WebLogic target).
3. Restart the agent.