Introduction

This is a follow-up to a blog post I did in March highlighting the need for TWS administrators to prepare for the expiration of the default SSL certificates contained within key TWS components. Since then, IBM have conducted several training sessions to help users understand the impact of the issue and the steps they must take to ensure they are not affected when the default certificates expire in Feb 2014 or they don’t have additional tasks to perform before Dec 10, 2013. For details of these training sessions and IBM-supplied documentation, follow this link http://www-01.ibm.com/support/docview.wss?uid=swg21628636

A recap of the main points:

  • The default SSL certificates within several TWS components are due to expire on Feb 10, 2014. If your environment uses SSL to communicate between components after this date, communication between components will  fail.
  • You must evaluate your TWS networks and understand to what degree this issue will affect you. Perform an inventory of your estate and the configuration options you have employed. e.g. Do you use SSL comms between Master Domain Manager (MDM)/Backup Master Domain Manager (BKM) and FTAs? Have you implemented dynamic scheduling in your networks with jobs being launched from the Tivoli Dynamic Workload Broker (TDWB)? Are you using Job Scheduling Console or Tivoli Dynamic Workload Console?
  • Depending on how your TWS networks are configured and the options you use to schedule your batch dictates the amount of actions you will need to perform. All TWS administrators will need to perform at least some actions.
  • Applying the latest fixpack to servers or agents is only half of the process. You must manually perform a final step on each affected component (referred to as Step 2) in addition to applying the latest fixpack.
  • Running Step 1 or Step 2 multiple times on a component will not cause an issue. The scripts recognise if the default certificates have been replaced and will skip the ones that have already been updated.
  • The action that must remain until last is running the “Step 2” procedure on the Master Domain Manager
  • All actions must be completed by Dec 10, 2013. The default certificates expire on Feb 10, 2014 but the Websphere server will begin to produce it’s own set of  certificates from Dec 10, 2013 so you are advised to perform all tasks before this date, or disable automatic generation of certificates within Websphere.

 

Overview

If you are running a basic TWSd network where you have a MDM and BKM communicating with FTAs and you haven’t implemented SSL communication between them, you will have a small number of tasks to perform.  These relate to updating certificates between MDM/BKM and TDWC server or JSC clients, depending on the versions you use. You will not have to go and apply fixpacks and run Step 2 on your agents before Feb 10, 2014 as those agent components don’t use any of the SSL-based communications that require their keystores to be updated with certificates that are due to expire shortly.  Working in this simple MDM-FTA configuration represents a large number of TWS users in my experience.

At the opposite end of the scale, if you have enabled SSL communications between MDM/BKM and FTA, you must replace the default certificates on your Fault Tolerant Agents either by installing the latest fixpack and running Step 2, or running Step1 and Step2 from the standalone certificate bundle. Similarly, if you have enabled Dynamic Scheduling in your estate, you must replace the default certificates your dynamic agents use on whichever servers they have been installed on. If you use either or both of these features, you will have significantly more actions than a “vanilla” TWS network as you must  physically patch your agents, and run “Step 2” manually, rather than just updating the management and UI components in your estate.

The instructions below show you how to update the default certificates which must be done before Feb 10, 2014. As highlighted in the main points above, it is recommended you have all steps completed before Dec 10, 2013 to avoid hitting the issue with the Websphere servers creating a new set of certificates that cannot be replaced using IBM’s supplied methods and will require further manual intervention.

In this example, I am using TWS 8.6 GA on a Red Hat Enterprise Linux 5.8 server installed into /opt/ibm/TWA.  The instructions below show how to view expiration dates of all of the certificates at each stage so you can confirm the “before and after” status of running Step 1,  Step 2, or both.  Some of the utilities used are either built into the operating system or supplied with TWS,  but the keytool utility is also available within Java Runtime Environment (JRE) deployments which most servers and PCs will already have installed. You can use this utility to confirm the expiration dates of certificates for JSC users or servers where only the Dynamic Agent is installed.

 

Step-by-Step Guide to Replacing the Default Certificates

TWS 8.6 Master Domain Manager GA  (Red Hat Linux 5)

The steps below show how to replace the expiring default TWS certificates in a TWS 8.6 Master Domain Manager on Linux and confirming they have been updated. All certificates should be replaced by December 10th, 2013.
 
Logon to the MDM as the TWS engine account and source the TWA environment:

 

[twsadmin@MYMDM ~]$ . /opt/ibm/TWA/twa_env.sh
Setting CLI environment variables….
Tivoli Workload Automation Environment Successfully Set !!!

 

The commands below show the default certificate status for a TWS 8.6 GA MDM
 
PEM certificate ( OpenSSL format ):
openssl x509 -in /opt/ibm/TWA/TWS/ssl/OpenSSL/TWSClient.cer -noout -enddate
notAfter=Feb 10 13:43:12 2014 GMT

 

kdb certificate ( GSKit certificate )
/usr/Tivoli/TWS/GSKit64/8/bin/gsk8capicmd_64 -keydb -expiry -db /opt/ibm/TWA/TWS/ssl/GSKit/TWSClientKeyStore.kdb -pw default
Password Expiry Time : 0     (Does not expire)

 

jks certificate ( Java Key Store)

/opt/ibm/TWA/eWAS/java/bin/keytool -list -v -keystore /opt/ibm/TWA/eWAS/profiles/TIPProfile/etc/TWSServerKeyFile.jks -storepass default | grep Valid
Valid from: 11/24/05 1:43 PM until: 2/10/14 1:43 PM

/opt/ibm/TWA/eWAS/java/bin/keytool -list -v -keystore /opt/ibm/TWA/eWAS/profiles/TIPProfile/etc/TWSServerTrustFile.jks -storepass default | grep Valid
Valid from: 11/24/05 1:43 PM until: 2/10/14 1:43 PM
Valid from: 11/24/05 1:43 PM until: 2/10/14 1:43 PM

/opt/ibm/TWA/eWAS/java/bin/keytool -list -v -keystore /opt/ibm/TWA/eWAS/profiles/TIPProfile/etc/TWSClientKeyFile.jks -storepass default | grep Valid
Valid from: 11/24/05 1:43 PM until: 2/10/14 1:43 PM

/opt/ibm/TWA/eWAS/java/bin/keytool -list -v -keystore /opt/ibm/TWA/eWAS/profiles/TIPProfile/etc/TWSClientTrustFile.jks -storepass default | grep Valid
Valid from: 11/24/05 1:43 PM until: 2/10/14 1:43 PM

TWS 8.6 MDM after TWS 8.6 FP2 Installed

Installing the fixpack updates the Trust Store – referred to as Step 1

 
Logon to the MDM as the TWS engine account and source the TWA environment:

 

[twsadmin@MYMDM ~]$ . /opt/ibm/TWA/twa_env.sh
Setting CLI environment variables….
Tivoli Workload Scheduler Environment Successfully Set !!!
Tivoli Workload Automation Environment Successfully Set !!!

 

PEM certificate ( OpenSSL format ):
openssl x509 -in /opt/ibm/TWA/TWS/ssl/OpenSSL/TWSClient.cer -noout -enddate
notAfter=Feb 10 13:43:12 2014 GMT

 

kdb certificate ( GSKit certificate )
/usr/Tivoli/TWS/GSKit64/8/bin/gsk8capicmd_64 -keydb -expiry -db /opt/ibm/TWA/TWS/ssl/GSKit/TWSClientKeyStore.kdb -pw default
Password Expiry Time : 0     (Does not expire)
/usr/Tivoli/TWS/GSKit64/8/bin/gsk8capicmd_64 -keydb -expiry -db /opt/ibm/TWA/TWS/ITA/cpa/ita/cert/TWSClientKeyStore.kdb -pw default
Password Expiry Time : 0     (Does not expire)

 

jks certificate ( Java Key Store)

/opt/ibm/TWA/eWAS/java/bin/keytool -list -v -keystore /opt/ibm/TWA/eWAS/profiles/TIPProfile/etc/TWSServerKeyFile.jks -storepass default | grep Valid
Valid from: 11/24/05 1:43 PM until: 2/10/14 1:43 PM

/opt/ibm/TWA/eWAS/java/bin/keytool -list -v -keystore /opt/ibm/TWA/eWAS/profiles/TIPProfile/etc/TWSServerTrustFile.jks -storepass default | grep Valid
Valid from: 11/14/12 8:51 AM until: 11/9/32 8:51 AM
Valid from: 11/14/12 8:48 AM until: 11/9/32 8:48 AM

/opt/ibm/TWA/eWAS/java/bin/keytool -list -v -keystore /opt/ibm/TWA/eWAS/profiles/TIPProfile/etc/TWSClientKeyFile.jks -storepass default | grep Valid
Valid from: 11/24/05 1:43 PM until: 2/10/14 1:43 PM

/opt/ibm/TWA/eWAS/java/bin/keytool -list -v -keystore /opt/ibm/TWA/eWAS/profiles/TIPProfile/etc/TWSClientTrustFile.jks -storepass default | grep Valid
Valid from: 11/24/05 1:43 PM until: 2/10/14 1:43 PM
Valid from: 11/14/12 8:48 AM until: 11/9/32 8:48 AM

Note: The old certificates are left in the directory and renamed as TWSServerTrustFile.jks.bck and TWSClientTrustFile.jks.bck. In the event a backout is required, these files can be reinstated.

Result: Applying the latest fixpack has added certificates to the TrustStore with expiration dates of 2032.

TWS 8.6 MDM – post TWS 8.6 FP2 Applied (Linux)

Update the KeyStore – referred to as Step 2

 
Important Note: Performing Step 2 on the Master Domain Manager is the last step to be taken across your whole estate.

 

You should already have run Step 1 on your agents, BKM, and TDWC servers by either installing the latest fixpack for your agent’s release level and then running the Step 2 scripts manually,or, if you cannot install the latest TWS fixpack on your components, run Step 1 and Step 2 from the certificate packages from IBM which can be downloaded from Fix Central.
 
Logon to the TWS MDM as root. You cannot use the TWS administrator account is it will not have the permissions to rename some files.

 

Stop all Websphere processes before commencing.

 

[twsadmin@MYMDM ~]$ . /opt/ibm/TWA/twa_env.sh
Setting CLI environment variables….
Tivoli Workload Scheduler Environment Successfully Set !!!
Tivoli Workload Automation Environment Successfully Set !!

 

Go to the updCertsScripts directory under TWSHOME:
cd /opt/ibm/TWA/TWS/updCertsScripts
 
Set the environment variables:

 

. /opt/ibm/TWA/TWS/updCertsScripts/env.sh /opt/ibm/TWA/TWS/updCertsScripts/updTrustKeyStoresCerts.sh /opt/ibm/TWA

 

 

[root@MYMDM updCertsScripts]# /opt/ibm/TWA/TWS/updCertsScripts/updTrustKeyStoresCerts.sh /opt/ibm/TWA

Skipping /opt/ibm/TWA/TWS/../eWAS/profiles/TIPProfile/etc/TWSServerTrustFile.jks because it is customized or already updated
Skipping /opt/ibm/TWA/TWS/../eWAS/profiles/TIPProfile/etc/TWSClientTrustFile.jks because it is customized or already updated
Updating BrokerWorkstation.properties
Skipping /opt/ibm/TWA/TWS/TDWB_CLI/certs/TWSClientTrustFile.jks because it does not exist
Skipping /opt/ibm/TWA/TWS/ITA/cpa/ita/cert/TWSClientKeyStore.kdb because it is customized or already updated
Skipping /opt/ibm/TWA/TWS/ssl/OpenSSL/TWSTrustCertificates.cer because it is customized or already updated
Skipping /opt/ibm/TWA/TWS/ssl/sslDefault/TWSCertificateChainFile.pem because it does not exist
SUCCESS
Updating /opt/ibm/TWA/TWS/../eWAS/profiles/TIPProfile/etc/TWSServerKeyFile.jks
Updating /opt/ibm/TWA/TWS/../eWAS/profiles/TIPProfile/etc/TWSClientKeyFile.jks
Skipping /opt/ibm/TWA/TWS/TDWB_CLI/certs/TWSClientKeyFile.jks because it does not exist
Updating /opt/ibm/TWA/TWS/ITA/cpa/ita/cert/TWSClientKeyStore.kdb
Updating /opt/ibm/TWA/TWS/ssl/OpenSSL/TWSClient.cer
Updating /opt/ibm/TWA/TWS/ssl/OpenSSL/TWSClient.key
Skipping /opt/ibm/TWA/TWS/ssl/sslDefault/TWSPublicKeyFile.pem because it does not exist
Skipping /opt/ibm/TWA/TWS/ssl/sslDefault/TWSPrivateKeyFile.pem because it does not exist
SUCCESS

 

PEM certificate ( OpenSSL format ) 
openssl x509 -in /opt/ibm/TWA/TWS/ssl/OpenSSL/TWSClient.cer -noout -enddate
notAfter=Nov  9 08:51:44 2032 GMT
 
kdb certificate ( GSKit certificate ) 
/usr/Tivoli/TWS/GSKit64/8/bin/gsk8capicmd_64 -keydb -expiry -db /opt/ibm/TWA/TWS/ssl/GSKit/TWSClientKeyStore.kdb -pw default
Password Expiry Time : 0 
(Does not expire)

 

/usr/Tivoli/TWS/GSKit64/8/bin/gsk8capicmd_64 -keydb -expiry -db /opt/ibm/TWA/TWS/ITA/cpa/ita/cert/TWSClientKeyStore.kdb -pw default
Password Expiry Time : 0  
(Does not expire)

 

jks certificate ( Java Key Store)

/opt/ibm/TWA/eWAS/java/bin/keytool -list -v -keystore /opt/ibm/TWA/eWAS/profiles/TIPProfile/etc/TWSServerKeyFile.jks -storepass default | grep Valid
Valid from: 11/14/12 8:48 AM until: 11/9/32 8:48 AM

Note: The old certs is renamed TWSServerKeyFile.jks.bck

 

/opt/ibm/TWA/eWAS/java/bin/keytool -list -v -keystore /opt/ibm/TWA/eWAS/profiles/TIPProfile/etc/TWSServerTrustFile.jks -storepass default | grep Valid
Valid from: 11/14/12 8:51 AM until: 11/9/32 8:51 AM

Valid from: 11/14/12 8:48 AM until: 11/9/32 8:48 AM
Valid from: 11/24/05 1:43 PM until: 2/10/14 1:43 PM
Valid from: 11/24/05 1:43 PM until: 2/10/14 1:43 PM


/opt/ibm/TWA/eWAS/java/bin/keytool -list -v -keystore /opt/ibm/TWA/eWAS/profiles/TIPProfile/etc/TWSClientKeyFile.jks -storepass default | grep Valid
Valid from: 11/14/12 8:51 AM until: 11/9/32 8:51 AM

 

/opt/ibm/TWA/eWAS/java/bin/keytool -list -v -keystore /opt/ibm/TWA/eWAS/profiles/TIPProfile/etc/TWSClientTrustFile.jks -storepass default | grep Valid
Valid from: 11/24/05 1:43 PM until: 2/10/14 1:43 PM
Valid from: 11/14/12 8:48 AM until: 11/9/32 8:48 AM

 

Note: The old cert is renamed TWSClientKeyFile.jks.bck

End Result: The OpenSSL PEM certificate is updated to expire in 2032.

The TrustStore and KeyStore certificates have been updated to not expire until 2032
The GSKit certificates are not affected before or after running these processes and do not expire but are included for example purposes.

Mark Delaney
SystemsManaged Ltd