Mar 092015
 

A New Dawn

Starting in January 2015, Avaya has changed it’s official policy with regards to Microsoft Hotfix updates to AACC servers. Prior to this policy update, all Microsoft Hotfixes were approved for installation only when tested and approved specifically by Avaya. There were numerous Hotfixes that were not approved and if those Hotfixes were installed, Avaya could (and sometimes did) decline to support the customer site. As of the January 2015 policy update, only those Hotfixes specifically listed by Avaya as not compatible are restricted from installation.

What this means for the traditional customer is that the standard IT Security policy of installing the latest Microsoft Hotfixes to ensure OS security is now part of the approved processes for Avaya Aura Contact Center Servers. As long as the Hotfix was released prior to the last published date of the bulletin, and as long as Avaya has not discovered a specific fault, the Hotfix is supported for installation on AACC systems.

As of this blog post, all Microsoft Hotfixes released by Microsoft on or before 10 Feb 2015 are approved for installation on Avaya Aura Contact Center, if the AACC is Release 6.4 SP14. Service Pack 14 was released mid-December 2014. For older systems (AACC SP13 or earlier, or any NES CC or Symposium systems), the older policy remains in force. Only those specifically tested and approved by Avaya are allowed to be installed, and for extremely old systems (NES CC or Symposium) installed on Windows 2003 Server or earlier operating systems, the Microsoft end of life is relevant.

Avaya Aura Contact Center runs on Windows 2008 Server R2 with specific server hardware engineering requirements. [Avaya credentials required] For more information about server specifications, please refer to the linked documentation or contact your support partner for assistance in ensuring hardware compliance.

Take Away

From a partner support perspective, this makes checking compatibility a much simpler endeavor– as long as the system is on SP14 or later, if the Hotfix isn’t listed then it’s OK to install. So the business partner need only look to see if any patches were installed after the “released before” date on the bulletin and only check those (or look for a limited number of specifically restricted hotfixes.)

From a customer support perspective, this ensures that AACC server OS security is capable of being much more current than it ever has been before in the history of the AACC product line.

This is great news for all concerned!

Recommendations

First, consult your support partner. Take their direction over anything you read on the internet. Installation of Service Packs for AACC is (these days) virtually a full dot release upgrade instead of the simple patch window we used to have with early AACC Service Packs or NES CC Service Updates. My experience is that instead of having a 2-5 hour window, windows are now consistantly 4-7 hours, and potentially much longer if the system is Highly Available. And that doesn’t even take into account the pre-upgrade engineering that is necessary to ensure you don’t upgrade and then find yourself exceeding the hardware requirements on the AACC’s Windows 2008 Server hardware.

Second, if you are on anything prior to AACC 6.4 Service Pack 14, you should update to SP14 ASAP. This addresses many of the most common and well known issues on the AACC. Similarly, if you are on anything prior to AACC 6.x you should upgrade now. Windows 2003 Server will soon reach end of life. This will obsolete NES CC6 and NES CC7 even more so than it is obsolete now (since those systems are “functionally stable” and there are no “corrective content” plans for this manufacture discontinued product version.) There are many reasons why you should upgrade, but to keep this focused on OS Security and Microsoft Hotfix compatibility, Windows 2008 Server will continue to receive additional Hotfix content. Windows 2003 Server, and earlier, will not.

Third, in the process of upgrading to SP14, you or your support partner should carefully review the readme to determine all of the known issues and known fixes for associated systems. There are engineering considerations on the PBX, PBX patches, Callpilot versioning (if you have ACCESS ports) and other considerations that should be taken into account. Some considerations aren’t part of the standard PBX DEPLIST, and by updating the DEPLIST the PBX patch required by the AACC Readme gets removed, resulting in recurring maintenance issues.

Nov 032013
 

Recently worked on an AACC (Avaya Aura Contact Center) where the partitioning of the server was determined to be the cause of the problem. While Disk Management (diskmgmt.msc) is easily accessible from START>>RUN, a screenshot is not quite as portable as text. To that end (and as a recommendation for addition to the Nortel Enterprise Audit Tool, or NEAT, used to survey Contact Center servers for Avaya engineering), I put together a script to query WMI (Windows Management Instrumentation) for the necessary information.

WMI Objects:

  • Win32_DiskDrive
  • Win32_DiskDriveToDiskPartition
  • Win32_DiskPartition
  • Win32_LogicalDiskToPartition

Using WMI queries against these objects you can derive:

  • Win32_DiskDrive => Physical Device ID (.\\.\PHYSICALDRIVE0\)
  • Win32_DiskPartition => Partition Device ID (Disk #0, Partition #1) and a derived type (e.g., Simple Volume? Primary Partition? Extended Partition/Logical Drive?)
  • Win32_LogicalDiskToPartition => Logical Drive Device ID (D:)

For quick “automated” checks of a system to verify compliance with engineering guidelines, this is a must.

Sample output:

\\.\PHYSICALDRIVE0,Disk #0, Partition #2,Basic,True,C:,Primary Partition
\\.\PHYSICALDRIVE0,Disk #0, Partition #3,Basic,False,D:,Extended Partition/Logical Drives
\\.\PHYSICALDRIVE0,Disk #0, Partition #3,Basic,False,F:,Extended Partition/Logical Drives
\\.\PHYSICALDRIVE0,Disk #0, Partition #3,Basic,False,G:,Extended Partition/Logical Drives
\\.\PHYSICALDRIVE0,Disk #0, Partition #3,Basic,False,T:,Extended Partition/Logical Drives

and

\\.\PHYSICALDRIVE0,Disk #0, Partition #2,Dynamic,True,C:,Simple Volume?
 \\.\PHYSICALDRIVE0,Disk #0, Partition #3,Dynamic,True,D:,Simple Volume?
 \\.\PHYSICALDRIVE0,Disk #0, Partition #3,Dynamic,True,F:,Simple Volume?
 \\.\PHYSICALDRIVE0,Disk #0, Partition #3,Dynamic,True,G:,Simple Volume?

The cool thing is that the script is applicable for all systems going back to Windows 2000 (Symposium 4 if I recall correctly) when the WMI query objects were instantiated in the OS by Microsoft.

Apr 032013
 

  • LD 48
  • Enabling Application Module Link (AML) traces
    • enxp <msgi/msgo> <aml_id> <exclude_priority> … <exclude_priority>
      Example:
      enxp msgo 16 1
      enxp msgi 16 1
    • enl <msgi/msgo> <aml_id>
      Example:
      enl msgi 16
      enl msgo 16
  • Disabling
    • dis <msgi/msgo> <aml_id>
    • dsxp <msgi/msgo> <aml_id>

 

Mar 092013
 

Here are all of the trace messages I’ve been able to identify.

 

Acquire

Acquire_Resp

Answer

AnswerIndication

Answer_Resp

Application_Polling

CallAbandonedQueue

CallOffered

CallRouted

Release

ReleaseAcquire

ReleaseAcquire_Resp

ReleaseIndication

Release_Resp

RouteCall

RouteCall_Resp

RouteRequest

SF_StartRecord

SF_StartRecord_Resp

SF_StopRecord

SF_StopRecord_Resp

SFN_ActivityCode

SFN_Login

SFN_Logout

SFN_MakeSetBusy

SFN_MakeSetInService

SFN_NotReady

SFN_Ready

SFN_Return

SFN_StartRecord

SFN_StopRecord

SFN_WalkAway

StatusChange

 

  • An MLSM Trace message always begins:
    ff 0a 00
  • The fourth word is always the word-length of the message (including the three word header)
  • The fifth word is always the Associated ID
  • From there it gets a lot more complicated– I wish there was some documentation I could lay hands on.
    • 96 06 + 4 words is the Call ID which matches AML trace or Call By Call stats
    • 0e 00 02 + 10 words is the Calling Line ID associated with a call. Each digit in a CLID is broken out into a separate word ranging from 30 for 0 to 39 for 9. As an example, the number 877-555-1212 would be represented by 38 37 37 35 35 35 31 32 31 32.

I do have a breakdown of several CallRouted, RouteCall, RouteCall_Resp and RouteRequest messages, but unfortunately the pattern is consistant and I know for certain there are other call types (sources, numbering plans, etc.) that could be represented in the traces… I just don’t have examples of them.

It was kind of fun to write a parse utility to grab all of the clear-text headers and then write another procedure that would use those headers to collate all of the responses so that I could look for patterns. It’s similar to DCH traces. Given enough time I’m sure I can find out a number of obvious patterns within the hex codes… (like 96 06 and 0e 00 02… I’ve already noticed that oe 00 02 represents only one type of calling line ID. There are other hex patterns that might also include CLID information. Origination Type, Origination Address, Destination Type, Destination Address… there might be a few more.

Mar 072013
 

I’ve been dealing with a number of complicated cases recently for work requiring that I review mass quantity of multi-meg trace files (text files outputting raw log information). The reason for this is because the customer that I’m working with is using a CTI application (Chrysalis Cloudburst) to control call routing and perform screen-pop functionality at the agent’s desktop when the call arrives. As part of that process, Cloudburst accepts the call from the IVR application and then performs a CTI route via MLSM (Meridian Link Services Manager).

--------    Output from mlsm.exe        Tue Feb 19 09:13:43.781 2013   (13:43.781)
ITR Route
                  03 1f 00 00 00 00 1e 34 a5 8b 00 00    header (12 bytes)       
                                             95 01 05    Subtype (Route)      
                                    96 04 00 00 2f c1    Call Id       
                                          4b 02 66 26    CDN (6626)      
                                          31 02 66 a3    Terminating DN (6603)      
                                             67 01 00    Conditional (unconditional)      
=======================================================================================
--------    Input                       Tue Feb 19 09:13:43.781 2013   (13:43.796)
ITS Route
                  03 22 00 00 00 00 1e 35 a5 8b 00 00    header (12 bytes)       
                                             95 01 05    Subtype (Route)      
                                    96 04 00 00 2f c1    Call Id       
                                          4b 02 66 26    CDN (6626)      
                                          31 02 66 a3    Terminating DN (6603)      
                                             67 01 00    Conditional (unconditional)      
                                             aa 01 00    Status (Successful request)      
=======================================================================================

When you’re looking for twenty or thirty trace entries across three to twelve trace files, representing hundreds (or potentially thousands) of trace entries… it gets to be both time consuming and tedious. I decided to code a parse utility to make my life easier. Part that process is converting the decimal call ID (in the example above, Decimal 12225) to a hex call ID and properly formatting that hex value (hex 2fc1) so that it can be searched for in the trace files.

While I have no intention of releasing the full parse utility to the public (if you really want a copy, contact me via LinkedIn, or comment on this blog and we’ll discuss it), I thought it might be helpful for those “do it yourselfers” who might want a hint at how to overcome one of the challenges I had to overcome to get this utility functional.

I’m using VBScript running as a cscript:

'**
'* function convertCIDtoCallID integerCallID
'* returns stringFormattedHexCallID
'*
'* Converts a decimal call ID to a hex string call ID with spaces between each byte (for use in searching AML logs)
'*

function convertCIDtoCallID( iCID )
 ' convert decimal Call ID to hex, then convert to lowercase
 hCID = lcase(hex(iCID))
 sRetVal = ""

 ' if the number of characters in the hex Call ID is odd, then we need a leading 0 inserted
 if (len(hCID)%2)=1 then
  hCID = "0" & hCID
 end if

 ' for each character from 1 to the length of the hCID string
 for i=1 to len(hCID)
  ' group hex values in groups of two characters
  if ((i mod 2) = 1) and (i>2) then
   ' insert a space between hex value groups
   sRetVal = sRetVal & " "
  end if
  ' append the current character to the return value
  sRetVal = sRetVal & mid(hCID,i,1)
 next

 ' return the formatted hex call ID string
 convertCIDtoCallID = sRetVal

end function

What this does is convert a decimal value to hex (12225 to 2fc1) and then formats it the way it might appear in an AML trace (2f c1). It also automatically adjusts for scenarios where the hex value is 3 hex digits (hex fc1 = dec 4033) to something that can be searched for (0f c1) instead of mishandling (fc 1, which would not be found in the traces.)

This should save me hours in reviewing log files.