Group memberships updates reflected in delay

In a multiple Content Servers Documentum environment you can encounter the following issue: after you change a group memberships (add/delete users and groups) using one CS, the changes are not reflected when you connect to the repository with other Content Servers, for a period of time (up to several hours). So the Content Server that performs the memberships change reflects the changes immediately, while the others will have a delay, usually a big one.
Why is this happening?
That's because the caching mechanism the Content Servers are using. The group memberships are cached from the DB into Content Server memory, so the CS does not query the DB every time it needs this information. When you are connected with a CS and a group is updated, this CS takes care to update its cache accordingly for that group. But what if you have more CS for the same repository? They will not update their cache, because they don't know there was a change. That's why on the other CS the group change will not be reflected until they don't refresh their cache.
How to solve that?
There is a configuration in server.ini which the CS Installation Guide does not pay attention to: upd_last_chg_time_from_db . The CS Administration Guide provides the following description for the key:
Optional key. Specifies that all Content Servers in a clustered environment have timely access to all changes in group membership. Valid values are: * TRUE: Should only be used for environments with multiple Content Servers. The value should be set to TRUE for all running Content Servers. * FALSE: The default value.
So the solution is to modify the server.ini files on all Content Severs setting this key value to TRUE:
# A boolean flag specifying whether user entries and groups list are synchronized from db.
upd_last_chg_time_from_db=T

D2 Installation Troubleshooting

During D2 installation there are some steps you should pay attention at, while others are not described in the installation guide.
Below you will find some notes and issues faced during installation of D2 product.
I Back-end installation on the Content Server
1. Stop all Documentum services before the instalations.
2. If you're upgrading D2, delete the preferences objects (DQL: delete d2c_preferences objects) and remove the poi*.jar files from ServerApps.ear lib folder.
3. While installing D2 libraries for the Content Server and JMS, enter a correct path to the ServerApps.ear lib folder on JMS (for Dctm 6.7 SP1 it does not contain APP-INF), as well as the path to the DFS SDK of same version as your CS.
4. Remove from the ServerApps.ear lib folder the following jars: itext.2.0.2.jar, xercesImpl-2.7.1.jar
(optional: if a recent patch has been installed for the CS, replace the dfc.jar from ServerApps.ear lib folder with the newer one)
5. Check and add (if not done by installer) the D2.jar file location to the CLASSPATH environment variable of the server hosting the CS.
6. Configure logback.xml file (which contains logging settings) from ServerApps.ear/[APP-INF/]classes folder. Note: In Document 6.7 SP1 the classes folder is created by the installer directly in ServerApps.ear folder, though it is normaly present in ServerApps.ear/APP-INF folder. Most probably it's a bug, so I suggest to copy the newly created xml files (after log configuration) from ServerApps.ear/classes to the ServerApps.ear/APP-INF/classes folder.
7. For DARs installation: if using Dctm 6.6 and later in the installation wizzard select 'Do not install DocApp/DAR', just select a path for extraction and consequently install the dars (D2.dar, D2Widget-DAR.dar) manually with DarDeployer. For version 4.1 install also CollaborationServices.dar (also on global repository if it's not the same one).
If your installation owner is not 'dmadmin', create an installparam file with following content:
<?xml version="1.0" encoding="UTF-8"?>
<installparam:InputFile xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:installparam="installparam">
    <parameter key="dmadmin" value="my_install_owner"/>
</installparam:InputFile>
Use this installparam file during DARs installation.

8. Create registered table for auditing:
register table dm_audittrail_s (event_name string(64), user_name string(32), time_stamp time, object_name string(255), string_1 string(200), string_2 string(200), string_3 string(200), string_4 string(200), string_5 string(200))
update dm_registered object set object_name = 'D2 Audits', set owner_table_permit = 1, set group_table_permit = 1, set world_table_permit = 1 where object_name = 'dm_audittrail_s';

II Front-end installation on Application Server
1. After extracting the webapps with the installer, configure the following files per application:
 a) D2-Config:
    - Path: D2-Config.war\WEB-INF\classes
    - Files: D2-Config.properties, dfc.properties, log4j.properties (add if missing), logback.xml
 b) D2-Client:
    - Path: D2-Client.war\WEB-INF\classes
    - Files: D2-Client.properties, dfc.properties, logback.xml
 c) D2FS (not present in 4.1):
    - Path: D2FS.war\WEB-INF\classes
    - Files: dfc.properties, logback.xml
 d) D2:
    - Path: D2.war\WEB-INF\classes
    - Files: applicationContext.xml, logback.xml (+ 4.1: logback-base.xml, D2FS.properties, dfc.properties)
 * Note: URL of D2FS in applicationContext.xml (not required for 4.1) must contain the host name, not localhost (because this URL also serves as download URL for files)
2. Widget View plugin configuration:
a) If you don't find Widget plugin jars in the package, launch the D2—Widget-Install.jar installer to deploy the jars to some location
b) Copy in \D2-Config\WEB-INF\lib the libraries: D2-Constants.jar, D2-Widget-API.jar (in addition D2-Widget.jar in case of patch 02)
c) For versions previous to 4.1: Copy D2-Widget-Plugin.jar to D2-Config\WEB-INF\classes\plugins
(Note: Weblogic does not locate correctly the plugin jar when relative path is specified, so you have to deploy the jar to external location and specify full path in D2-Config.properties, for example: D:\EMC\D2\plugins\D2-Widget-Plugin.jar)
d) For versions previous to 4.1: Add the path to the plugin in \D2-Config\WEB-INF\classes\D2-Config.properties: plugin_1=plugins/D2-Widget-Plugin.jar (or full path to jar location for Weblogic)
Note: In version 4.1 the plugins are included and configured by default in D2-Config (however if Weblogic does not find them, use an external path as shown above).

III Common issues

Issue 1 - Error while installing the DAR file: Owner object (user OR group) 'dmadmin' not found in the target repository
Solution:
As described above, create a file nodmadmin.installparam with the following contents:
<?xml version="1.0" encoding="UTF-8"?>
<installparam:InputFile xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:installparam="installparam">
    <parameter key="dmadmin" value="my_install_owner"/>
</installparam:InputFile>
On the DarDeployer interface, under DAR Details, click Browse next to Input File, and select the nodmadmin.installparam you've created.

Issue 2 - File transfer fails in D2
Solution: Check the D2FS URL in the \D2\WEB-INF\classes\applicationContext.xml - it should use the server name, not 'localhost'.

Issue 3 - Tomcat PermGen Space error
Solution: Add/modify the following Java option in your Tomcat environment: -XX:PermSize=256m -XX:MaxPermSize=512m

Issue 4 - D2-Config menus do not show all items (not sized correctly)
Solution: Configure Internet Explorer as follows:
  - Allow popup windows
  - Allow windows to resize by script without size or position constraints
  - Allow the browser to use tabbed browser settings when encountering a popup window

Documentum 6.7 CTS troubleshooting: after installation

If you've came here I guess you're in same trouble I was while installing and configuring Documentm CTS in 6.7.x versions.
If you've read the first part of my CTS Troubleshooting Guide, but didn't find the solution, try your luck by reading this article too.
1. Server returns HTTP response code 404
If you find in the logs the following error:
com.documentum.cts.plugin.advancedpdf.AdvancedPDFProcessor - Exception within  processResponse() method: Server returned HTTP response code: 404 for URL: http://localhost/exponentwsa/exponentwsa.asmx/DeleteJob

Open in the browser the following URL: http://localhost:80/exponentwsa/exponentwsa.asmx/DeleteJob
The page shows "HTTP 404.2 - Not Found" error.
Solution: To fix this, open IIS Manager (from Server Manager) then find and open 'ISAPI and CGI Restriction'. The entries ASP.NET vXXX must have Restriction value 'Allowed'.

2. Server returns HTTP response code 500
This error occurs when you find in the logs something like:
com.documentum.cts.plugin.advancedpdf.AdvancedPDFProcessor - Exception within  processResponse() method: Server returned HTTP response code: 500 for URL: http://localhost/exponentwsa/exponentwsa.asmx/AddJob

Open the URL in the browser to be check you really have 500 code response  Then open IIS Manager and go to Application Pools. Check all the pool ExponentWSA is started. Open the pool settings and check that .NET Framework v2.0.XXs is selected. If you changed the value, restart the server.

3. CTS does not create renditions
One possible cause of CTS malfunction is the user set to run the CTS & Adlib services. There are 7 services, from which 3 of them (Adlib FMR, Adlib Process Manager and Documentum CTS Admin. Agent) must run as 'Local System', while the remaining 4 (Adlib Exponent Connector, Adlib Exponent Manager, Documentum Content Transformation Services, Documentum Content Transformation Monitor Services) with [DOMAIN\]SUPERUSER (SUPERUSER - normaly the installation owner).
Restart the CTS services if you've changed any settings.

4. CTS can perform a transformation only when the server is remotely connected (Windows 2008 R2 x64)
There's an known issues for Windows 2008 R2 x64: renditions work only when the server is remotely connected. The process adexps.exe is closed when the install owner logs off.
To fix this, you have to edit the file: ..\Program Files (x86)\Adlib\Process Manager\ProcessManagerInitSettings.xml (default path):
Change lines (all found):
<ProcessLaunchType>LaunchAndWatch</ProcessLaunchType>
<ProcessSessionType>UserSession</ProcessSessionType>
to
<ProcessLaunchType>LaunchAndWatchSession</ProcessLaunchType>
<ProcessSessionType>SystemSession</ProcessSessionType>

Save the file and restart the CTS services.

These are the main issues encountered with CTS. If you face other issues, feel free to post them into comments, I could give some ideas.

Documentum 6.7 CTS troubleshooting: before installing

Installing Documentum CTS (including DTS/ADTS/MTS/etc.) became a bit tricky in versions 6.7.x since it requires more pre-requisite components to be installed:
1. IIS Server
2. ASP.NET
3. Message Queuing
4. .NET Framework

Moreover you have to install a certain set of subfeatures. These services/features can be installed in Windows using Server Manager. Start by adding the role Web Server (IIS), then check the following features/services to be installed for it:

- Under Application Development:
* ASP.NET
* .NET Extensibility
* ISAPI Extensions
* ISAPI Filters

- Under Management Tools:
* IIS Management Console
* IIS Management Compatibility + all subfeatures

Then go to Add features section and add the feature Message Queuing Services with the following subfeatures:

* Message Queuing Server

* Directory Service Integration

* HTTP Support

Well this could be enough to go on with the CTS products installation, however your system still might miss some configurations that will prevent your CTS from working properly.

First, go to IIS Manager, open Sites and check you have a site called: Default Web Site

Open the browser (for ex. IE) and open the following address: http://localhost 

If default IIS web site is working you will see IIS welcome page (Welcome in different languages).

Ensure you have Microsoft Office installed (Word, Excel, PowerPoint). 

If you've done all the steps above, you can go on with installation of CTS.

During the installation you might encounter the following error: The installation cannot continue until the following conditions are met: Microsoft Office must be installed.

Well, even you've installed Microsoft Office, the CTS installer might not 'see' it, so you have to manually intervene in Windows Registry. Check if you have the following keys in the Registry (if you don't - create them):

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall\office_outlook

Name: DisplayName

Type: String

Value: Microsoft Office Outlook

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall\office_word

Name: DisplayName

Type: String

Value: Microsoft Office Word

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall\office_excel

Name: DisplayName

Type: String

Value: Microsoft Office Excel

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall\office_powerpoint

Name: DisplayName

Type: String

Value: Microsoft Office PowerPoint

Now the installation should complete successfully. You can start the CTS services. All looks fine, but... no, some errors, again!

If you have a 64-bit OS you might find in the CTS log the following kind of errors:

Unable to instantiate the following MP: com.documentum.cts.plugin.advancedpdf.AdvancedPDFPlugin

java.lang.UnsatisfiedLinkError: D:\Documentum\CTS\lib\JNI_WindowsService.dll: Can't load AMD 64-bit .dll on a IA 32-bit platform

The cause is the CTS installer which, unlike the CS installer, includes just a 32-bit JDK, while for your OS you need the 64-bit one. So, preferably before configuring a CTS instance for a repository, you have to perform the following workaround, (described also in the CTS installation guide):

1. Install a 64-bit 1.6.x JVM in a folder, then rename the Documentum's 32-bit java folder (for ex: %Documentum%\java\1.6.0_17) by adding _32 (%Documentum%\java\1.6.0_17_32), then create in the same path a new folder with the original java name (%Documentum%\java\1.6.0_17). Then copy all the contents from the 64-bit Java to the newly created folder (%Documentum%\java\1.6.0_17)

2. Set environment variables (JAVA_HOME, PATH) to point to this Java version installation.

3. Once the Java has been updated, update the Windows registry value for CTS Admin Agent to use the older 32-bit Java:

Key:

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Appache Software

Foundation\Procrun 2.0\CtsAdminAgent\Parameters\Java

Property Name: JVM

Property Value: C:\PROGRA~1\DOCUME~1\java\1.6.0_17_32\jre\bin\server\jvm.dll

Now you can configure CTS instances for your repository(ies).

If you encounter further issues with CTS performance, see the second part of this CTS Troubleshooting Guide.

JMS 6.7 SP1 error connecting to inexisting docbase

Recently I've installed Content Server 6.7 SP1 and the patch 08. When I start the Java Method Service, I find the following error in the logs:
INFO [STDOUT] (main) 10:02:37,932 ERROR [main] com.documentum.mthdservlet.MethodConfig - DfNoServersException:: THREAD: main; MSG: [DM_DOCBROKER_E_NO_SERVERS_FOR_DOCBASE]error: "The DocBroker running on host ([HOST]:1489) does not know of a server for the specified docbase ([INSTALLATION_OWNER])"; ERRORCODE: 100; NEXT: null

So it seems JMS client tries to connect to a docbase which has a name equal to the installation owner of the current docbase. Error in JMS configuration? Nope - checked all configurations, everything's correct. So what's the problem?
Ok, I checked again the stack trace and saw the problem is coming from method populateDocbaseNames of MethodConfig class, which is in mthdservlet.jar library.
Decompiled the jar, opened the method and here's the big surprise from EMC developers:
...
  String str1 = (String)localEnumeration.nextElement();
  if ((!Utils.isNull(str1)) && (str1.toLowerCase().startsWith("docbase")))
...

these lines read all the docbase names from web.xml, found in ...\ServerApps.ear\DmMethods.war\WEB-INF\
Opening the file we see the tags:
    <init-param>
      <param-name>docbase-my_docbase</param-name>
      <param-value>my_docbase</param-value>
    </init-param>

So it should read this docbase and all other available & configured repositories, the tag <param-name> having values of format 'docbase-[DOCBASE_NAME]'.
Ok, but I have only 1 repository configured. Scrolling a bit, I find another tag:
    <init-param>
      <param-name>docbase_install_owner_name</param-name>
      <param-value>dmadmin</param-value>
    </init-param>

Having the code above - startsWith("docbase") it will read also this tag and interpret it as a docbase name. Ok, then I decompiled an older version of mthdservlet.jar and found a bit different code:
  if ((!Utils.isNull(str1)) && (str1.toLowerCase().startsWith("docbase-")))

Here it is! A genious EMC developer removed that dash after docbase: startsWith("docbase-") Well, sh*t happens, even to geniuses.

So while we wait for a patch for this patch :) we can use the old version of mdthdservlet.jar or just ignore this error, as it has no impact on JMS work.

How to unlock a job in running state

There are situations when a running job fails and remains locked in running state. It usually happens when the JMS is stopped or other fatal errors occur.
To unlock such a job, the following steps should be performed (in this order until it's unlocked).

1) Check if the job is locked, using the following DQL:
select r_lock_date from dm_job where job_name='[JOB_NAME]'

if the returned value is not nulldate, the job is locked. Unlock it by executing the API command:
unlock,c,[r_object_id]

2) If the job is still in running state, perhaps the job's process is still active (hung) and must be killed.
a. Get the process ID, using the DQL:
select a_last_process_id from dm_job where object_name='[JOB_NAME]'
b. Using Process Explorer (or similar) locate the process by id and kill it
c. Check the job state (in DA)
3) Checked the job and it's still running? Don't give up, we're almost done!
Check the a_current_status attribute value (DQL: select a_current_status from dm_job where object_name='[JOB_NAME]').
If it's 'STARTED', we should change it to 'FAILED'. Use the following DQL:
update dm_job object set a_current_status='FAILED' where object_name='[JOB_NAME]'

4) If your job still didn't get rid of the running status, perhaps it has the reference to the application that locked it.
Execute the DQL query:
select r_object_id, a_last_invocation, a_last_completion, a_special_app from dm_job where (((a_last_invocation IS NOT NULLDATE)
 and (a_last_completion IS NULLDATE)) or (a_special_app = 'agentexec')) and (i_is_reference = 0 OR i_is_reference is NULL)
 and (i_is_replica = 0 OR i_is_replica is NULL)

If the query returns your job, execute the following DQL:
update dm_job objects set a_special_app='' where object_name='[JOB_NAME]'

That's it, your job is unlocked now and ready to be run again.

LDAP Sync - Solve user name conflicts

When an AD (Active Directory) user object name is changed (userPrincipalName,sAMAccountName, or other attributes) and later another AD object is created with the original name of the first user object, a conflict in the LDAPSync'ed repository occurs.
This is caused by the fact that user_name attribute of the dm_user objects is not changed during LDAP Synchronization (by default user rename option is disabled in LDAP config object) - thus another object with same value can't be created (as this field is unique).
Another conflict situation could be when 1 repository is sync'ed against several AD domains and some user name values might be equal across domains.

The steps to solve this conflict are:
1. Rename the old user object in the repository
(user_login_name has changed while user_name still holds the old value).
For this: open DA, Users section, search for the user, right-click on it and choose Reassign User option.
In the Reassign field enter the new user name (should be current user_login_name value).
Select the option 'Run the Reassign job Now'.
If the user has locked objects, perhaps it's better to select Checked Out Objects Leave all locked (though if the user is inactive, perhaps it's better to unlock).
Press Ok to perform the changes, which will start the dm_UserRename job shortly.
2. Check the dm_UserRename job report
Search for the dm_UserRename job and check if it is running or already completed (after your change at step 1), or if it didn’t start yet (CS can be loaded with other jobs) wait until it runs. Notice that usually the job is in inactive state, but this is normal.
After the run has completed (see Last Run column value), view the job report (from context menu) and check that the user has been renamed, if there were some issues or if some repository objects still have references to the old value.

3. Run immediately the dm_LDAPSynchronization job (optional)
Running this job manually is not mandatory as it usually runs by schedule. However, if the user updates are urgent, the job can be run manually, but an additional trick must be performed in order to not disturb the time intervals of the synchronizations. Perform the following steps:

a) Retrieve the a_last_run and a_last_no from the ldap config object and store them:
DQL> select a_last_run,a_last_no from dm_ldap_config
a_last_run                            a_last_no        
================ ============
20091105210022.0Z         195290415        

b) Store the current value of dm_ldap_config.per_search_filter, then change it (to update only the 1 required user) to:
DQL> update dm_ldap_config object set per_search_filter='(&(userPrincipalName=MY_USER*)(mail=*))'
(assuming 'MY_USER' is the new user to be created, which failed due to the conflict. You can leave the old filter and just add an additional contidion: (userPrincipalName=MY_USER*) )

c) Run the dm_LDAPSynchronization job. Check the report and see if the user has been created properly. Also check the user is indeed created (using DA/samson/other client)

d) Restore previous ldap config object settings:
update dm_ldap_config object set a_last_run='20091105210022.0Z',set a_last_no='195290415', set per_search_filter='[ORIGINAL_VALUE]'

Error while creating a copy of ACL template in DFC

If you try to create a copy of an ACL template (which has alias references) using the method saveAsNew(), you will receive an error like this:

2011-12-02 11:53:01,138 ERROR [main] com.mycompany.documentum.job.method - [DM_ACL_E_USER_NOT_EXIST]error: "The owner_name or accessor_name '%alias' given in the ACL 'myACL' does not exist."

DfException:: THREAD: main; MSG: [DM_ACL_E_USER_NOT_EXIST]error: "The owner_name or accessor_name '%alias' given in the ACL 'myACL' does not exist."; ERRORCODE: 100; NEXT: null

This is caused by a bug in DFC method saveAsNew() of DfACL - it does not copy the value of attribute acl_class, which defines the ACL type (template, regular, etc.)

So you have to make a clone with custom code, something like this:

public IDfACL cloneACL(IDfACL acl, String name) throws DfException {

IDfACL copy = (IDfACL) acl.getSession().newObject("dm_acl");

copy.setObjectName(acl.getObjectName());

copy.setDescription(acl.getDescription());

copy.setString("owner_name", acl.getString("owner_name"));

copy.setACLClass(acl.getACLClass());

for (int i = 0; i < acl.getAttrCount(); i++) {

if (acl.getAttr(i).isRepeating()) {

String attrName = acl.getAttr(i).getName();

for (int j = 0; j < acl.getValueCount(attrName); j++) {

copy.setRepeatingValue(attrName, j, acl.getRepeatingValue(attrName, j));

}

}

}

copy.save();

return copy;

}

xPlore Index Agent login failure

If you are using Documentum xPlore for fulltext search, you might face the following issue: after starting the Index Agent process, you have to start the Index Agent from the Administration page (http://URL:port/IndexAgent/login_dss.jsp), but the login fails with the message:
User is not an Admin User!
even if your user is a superuser

The cause is a bug in the login_dss.jsp code:
IDfUser userObject = doc_session.getUserByOSName(uname, domain_name);
…
if (userObject != null)
isSysAdmin = userObject.isSysAdmin();

The bug here is that if the userObject is null (was not found), the flag isSysAdmin remains false, so the error message doesn’t say that the user was not found, instead you have the message above telling it’s not an Admin user.

Now why the userObject can happen to be null? (Note: at this point of code the user is already logged in against the repository). It’s because the user is searched using getUserByOSName(uname, domain_name) method. So if the user_os_domain is not set in the dm_user object and the Documentum repository runs in domain required mode (dm_docbase_config.auth_protocol='domain_required') : if you don’t specify the domain on login page – login doesn’t succeed, if you specify it – the userObject is not found – thus null.

To solve this issue it's enough to modify the login_dss.jsp page substituting the getUserByOSName(uname, domain_name) method with getUserByLoginName(uname, domain_name) or getUser(uname), or you could just set the user_os_domain in the dm_user object (however first option is preferred).