Apr 282013

While working for a Nortel distributor a few years ago, after answering repeat questions (sometimes the same question 4+ times from the same individual) I put together an internal WIKI and documented things for all employees. This information ranged from corporate policy, forms, etc. and went even so far as to document technical procedures. It was a one-stop-shop for information– any time anyone asked me a question, I put the answer on the Wiki and then linked them to the wiki. I called it, appropriately, RTFM. I could have used Sharepoint, but we were operating on a budget. So I commandered a directory on an IIS server, installed mySQL and mediawiki. I’m sure there was more to it than that, but the point of this article isn’t to talk about my mediawiki installation experience.


One of the features introduced with the Nortel CP-PM (Call Processor, Pentium Mobile) and CS1000E chassis (I forget the exact chassis shown in the pictures below) was the redundant Network Interfaces for the Call Server in the CS slot.

The Call Server (CS card in the middle of the picture above) supports a HSP (High Speed Pipe), TLAN (Telephony LAN) and ELAN (Equipment LAN) network interface, but the CS only uses the ELAN (and maybe the HSP if you were in a redundant CPU configuration). The CS does not use or need the TLAN, since it did not process any VoIP traffic.

The MGC (Media Gateway Controller– the card on the bottom of the picture above) supports both the TLAN and ELAN interface, as well as pass-through for both TLAN and ELAN.

The back of the chassis has a redundant TLAN and ELAN interface for the MGC card.


The reason that this particular wiki page came about is because I was sent to three or four sites over a month period to do clean up on sites that had been installed by our “lead installer.” I say cleanup because in each installation, the problem was system instability due to customer maintenance of their data network.

You see, if you connect both the primary and redundant ELAN connections on the MGC, you can take either data switch down and the system will keep on chugging without impact… and, by plugging the CS into the ELAN-passthrough port, the CS could simulate an increase in network redundancy (i.e., as long as the MGC remained up, the CS would retain communication with all other components over the ELAN through whichever ELAN port was still live and working on the MGC– of course, if the MGC rebooted or if both NICs went down, became unplugged, the data switches were rebooted, whatever– then the whole house of cards came down, same as if you didn’t have redundancy.)

So after going out to each of these and being asked to install the redundancy, I went back to our “lead installer” and asked him to make it part of his installs in the future. His response: It doesn’t work that way.

So I went to a co-worker, verified my understanding of how things worked, and then asked my co-worker to approach him and try again. The lead installer’s response: Yea, the senior remote engineer said the same thing to me, but it doesn’t work that way.

When asked how he thought it worked, he said that the redundant ports were not redundant and that the passthrough port didn’t work.

So I proceeded to document, with photos (the ones you see above), how it worked and put together a wiki page.


Then I involved management.


For a small distributor to have this kind of inefficiencies (the lead installer does a bad job, then someone else has to go out and clean up the work), it was not only killing budgets on projects and eating “warranty” hours after the customer went live… it was also putting me on-site, which meant less time to focus on my primary responsibilities.


I took our lab system and set it up the proper way, then I invited management to a demonstration where I demonstrated by unplugging and plugging in data cables how stability/redundancy worked. Then I showed them what happened when certain data cables were left unplugged or unused (the way the lead installer was doing it).


When management addressed the issue with the lead installer, he quit rather than amend his ways. Needless to say, I was shocked by this response.

Apr 272013

For all I know, this system is still there. Certainly the company that ended up buying the Firstworld call center still exists and still operates facilities at this location. (Shaw & Minnewawa, Clovis, California).

My introduction to Nortel systems was kind of round-about. I was working for a company in Fresno that was on a Lucent Definity G3S (?) and things were not looking very good, so I started shopping my options and got a job that was actually a step up in both pay and stability (Firstworld) as a report writer. I crash-coursed on SQL (I was actually MSCE certified on SQL but had very limited need to use the knowledge on a day-to-day business before this) and when I got the job I did nothing but review the design of the call center & write reports… for about 90 days.

Then, the guy who got me hired took a job with Quantum (which closed down that facility within 6 months of hiring him on) and left me the job of running the technical part of the Call Center. I was given an incredible opportunity and I leveraged it into the career I have today.

From this job I went to Shared Technologies Fairchild (an Intermedia company) as a field engineer.

Apr 272013

Doing some housecleaning on my harddrive and as I was going through files that I’ve never accessed nor updated, I came across this ancient visio diagram. I received the Visio stencils from an awesome sales person at Shared Technologies Fairchild (back during the Intermedia days) and used them regularly to produce high-quality documentation for big customers.

I don’t mind posting this because I happen to know that the customer for whom this was produced has forklifted their Nortel system in favor of another phone system manufacturer. They were a customer of my current employer back when I started 5 years ago, and they were in the process of migrating away from Nortel at that time.

I can’t say that I’m surprised, looking back. When I was working with that customer 12 years ago they canned their telecom manager who knew something about the Nortel systems (for reasons I never fully understood) and hired this guy who knew absolutely nothing about Nortel systems. He was aggressively ignorant and self absorbed. After more than a month listening to him complain endlessly about how he thought the Nortel system was a piece of garbage, I took him out to lunch and asked him a simple question. After which, he never complained to me again… but I had no doubt that he agitated to remove the system from his start date until he left. (He is no longer there.)

I think it was the following year we upgraded this system from a 61C to an 81C, pulled out the Meridian Mail and put in a Callpilot (as Callpilot-Symposium integration got better), added a Telstrat IDVR. I was pretty proud of all the technology that was there and how much I learned in the short amount of time I worked with this customer.

Apr 032013

  • LD 48
  • Enabling Application Module Link (AML) traces
    • enxp <msgi/msgo> <aml_id> <exclude_priority> … <exclude_priority>
      enxp msgo 16 1
      enxp msgi 16 1
    • enl <msgi/msgo> <aml_id>
      enl msgi 16
      enl msgo 16
  • Disabling
    • dis <msgi/msgo> <aml_id>
    • dsxp <msgi/msgo> <aml_id>


Mar 092013

Here are all of the trace messages I’ve been able to identify.




































  • An MLSM Trace message always begins:
    ff 0a 00
  • The fourth word is always the word-length of the message (including the three word header)
  • The fifth word is always the Associated ID
  • From there it gets a lot more complicated– I wish there was some documentation I could lay hands on.
    • 96 06 + 4 words is the Call ID which matches AML trace or Call By Call stats
    • 0e 00 02 + 10 words is the Calling Line ID associated with a call. Each digit in a CLID is broken out into a separate word ranging from 30 for 0 to 39 for 9. As an example, the number 877-555-1212 would be represented by 38 37 37 35 35 35 31 32 31 32.

I do have a breakdown of several CallRouted, RouteCall, RouteCall_Resp and RouteRequest messages, but unfortunately the pattern is consistant and I know for certain there are other call types (sources, numbering plans, etc.) that could be represented in the traces… I just don’t have examples of them.

It was kind of fun to write a parse utility to grab all of the clear-text headers and then write another procedure that would use those headers to collate all of the responses so that I could look for patterns. It’s similar to DCH traces. Given enough time I’m sure I can find out a number of obvious patterns within the hex codes… (like 96 06 and 0e 00 02… I’ve already noticed that oe 00 02 represents only one type of calling line ID. There are other hex patterns that might also include CLID information. Origination Type, Origination Address, Destination Type, Destination Address… there might be a few more.

Mar 072013

I’ve been dealing with a number of complicated cases recently for work requiring that I review mass quantity of multi-meg trace files (text files outputting raw log information). The reason for this is because the customer that I’m working with is using a CTI application (Chrysalis Cloudburst) to control call routing and perform screen-pop functionality at the agent’s desktop when the call arrives. As part of that process, Cloudburst accepts the call from the IVR application and then performs a CTI route via MLSM (Meridian Link Services Manager).

--------    Output from mlsm.exe        Tue Feb 19 09:13:43.781 2013   (13:43.781)
ITR Route
                  03 1f 00 00 00 00 1e 34 a5 8b 00 00    header (12 bytes)       
                                             95 01 05    Subtype (Route)      
                                    96 04 00 00 2f c1    Call Id       
                                          4b 02 66 26    CDN (6626)      
                                          31 02 66 a3    Terminating DN (6603)      
                                             67 01 00    Conditional (unconditional)      
--------    Input                       Tue Feb 19 09:13:43.781 2013   (13:43.796)
ITS Route
                  03 22 00 00 00 00 1e 35 a5 8b 00 00    header (12 bytes)       
                                             95 01 05    Subtype (Route)      
                                    96 04 00 00 2f c1    Call Id       
                                          4b 02 66 26    CDN (6626)      
                                          31 02 66 a3    Terminating DN (6603)      
                                             67 01 00    Conditional (unconditional)      
                                             aa 01 00    Status (Successful request)      

When you’re looking for twenty or thirty trace entries across three to twelve trace files, representing hundreds (or potentially thousands) of trace entries… it gets to be both time consuming and tedious. I decided to code a parse utility to make my life easier. Part that process is converting the decimal call ID (in the example above, Decimal 12225) to a hex call ID and properly formatting that hex value (hex 2fc1) so that it can be searched for in the trace files.

While I have no intention of releasing the full parse utility to the public (if you really want a copy, contact me via LinkedIn, or comment on this blog and we’ll discuss it), I thought it might be helpful for those “do it yourselfers” who might want a hint at how to overcome one of the challenges I had to overcome to get this utility functional.

I’m using VBScript running as a cscript:

'* function convertCIDtoCallID integerCallID
'* returns stringFormattedHexCallID
'* Converts a decimal call ID to a hex string call ID with spaces between each byte (for use in searching AML logs)

function convertCIDtoCallID( iCID )
 ' convert decimal Call ID to hex, then convert to lowercase
 hCID = lcase(hex(iCID))
 sRetVal = ""

 ' if the number of characters in the hex Call ID is odd, then we need a leading 0 inserted
 if (len(hCID)%2)=1 then
  hCID = "0" & hCID
 end if

 ' for each character from 1 to the length of the hCID string
 for i=1 to len(hCID)
  ' group hex values in groups of two characters
  if ((i mod 2) = 1) and (i>2) then
   ' insert a space between hex value groups
   sRetVal = sRetVal & " "
  end if
  ' append the current character to the return value
  sRetVal = sRetVal & mid(hCID,i,1)

 ' return the formatted hex call ID string
 convertCIDtoCallID = sRetVal

end function

What this does is convert a decimal value to hex (12225 to 2fc1) and then formats it the way it might appear in an AML trace (2f c1). It also automatically adjusts for scenarios where the hex value is 3 hex digits (hex fc1 = dec 4033) to something that can be searched for (0f c1) instead of mishandling (fc 1, which would not be found in the traces.)

This should save me hours in reviewing log files.

Aug 092012

Zone Bandwidth Strategy

One of the things I run into regularly with deployments is an issue where the installer improperly configures the Zone Bandwidth Strategy. This information can be checked by printing the interzone and intrazone configurations in LD 117.

=> prt zone

=> prt interzone
 |Near end  |Far end   |State| Type  |Stra|MO/ |QoS| Bandwidth | Sliding |  Usage  |  Quota  |Peak|  Calls  |  Alarm  |
 |          |          |     |       |tegy|BMG/|Fac|Configured |   max   |         |         |    |         |         |
 |          |          |     |       |    |VTRK|tor|           |         |         |         |    |         |         |
 |Zone|VPNI |Zone|VPNI |     |       |    |    | % |    kbps   |   kbps  |   kbps  |   kbps  | %  |   Cph   |   Aph   |
 |   1|     |    |     | ENL |SHARED |  BB|  MO|   |    1000000|         |      380|        0|   0|         |         |
 |   5|     |    |     | ENL |SHARED |  BB|VTRK|   |    1000000|         |      380|        0|   0|         |         |
 |  42|     |    |     | ENL |SHARED |  BB|  MO|   |    1000000|         |        0|        0|   0|         |         |

=> prt intrazone
 |Zone|State| Type  |Strategy|MO/ | Bandwidth |  Usage  |  Quota  | Peak |
 |    |     |       |        |BMG/|    kbps   |   kbps  |   kbps  |   %  |
 |    |     |       |        |VTRK|           |         |         |      |
 |   1| ENL |SHARED |   BQ   |  MO|    1000000|      380|        0|    0 |
 |   5| ENL |SHARED |   BQ   |VTRK|    1000000|      380|        0|    0 |
 |  42| ENL |SHARED |   BQ   |  MO|    1000000|        0|        0|    0 |
  Number of Zones configured = 3


The Strategy column for intrazone and interzone can be configured as either:

  • BQ aka Best Quality aka G.711 codec selection
  • BB aka Best Bandwidth aka G.729 codec selection

BB should be used when a WAN interface is handling VoIP with the following limitations/restrictions:

  • Bandwidth shaping or rate limiting restrictions
  • Bandwidth availability limitations (either the pipe is too small, or other usage + VoIP traffic exceeds available bandwidth)
  • No QoS policies configured

Even if you believe your WAN can support the bandwidth demands of VoIP + other usage, if you are experiencing any quality issues with VoIP, it’s worth changing your bandwidth strategy to BB (i.e., Best Bandwidth, G.729) as part of your troubleshooting efforts. I also recommend that you configure the available bandwidth to match the actual available bandwidth.


As the word root suggests, the Interzone Strategy is used for all VoIP calls that leave the zone wherein the caller is configured. This strategy is also used for calls that are inter-site.


As the word root suggests, the Intrazone Strategy is used for all VoIP calls within the zone.

Purpose of Zones

Zones provide a way of controlling the behavior of VoIP and QoS within a site (and between sites). Zones work best when used to represent physical locations.


Zone 1 is set up for all phones within a site (on the LAN). BQ strategy is used for intrazone and BB is used for interzone.

Zone 2 is set up for all phones at a branch office. BQ strategy is used for intrazone and BB is used for interzone.

Zone 3 is set up for all softphones connected via VPN (remote employees). BB strategy is used for both intrazone and interzone.

If a caller in Zone 1 calls anyone in Zone 1, they use G.711 (BQ). If they call anyone outside of Zone 1 (a VPN user softphone, or the branch office) they use the G.729 (BB strategy) codec.

If a caller in Zone 2 calls anyone in Zone 2, they use G.711 (BQ), but calls out of Zone 2 use G.729 (BB). Just like Zone 1.

If a caller in Zone 3 calls anyone they use G.729 (BB).

Mixed VoIP/TDM environments

TDM resources are sometimes configured to be in a separate zone by installers. I’ve never understood this, because following the logic above, if all the TDM equipment (Media Gateways, MGCs, MC32s, IP Trunks, etc.) are put in Zone 4 and everyone uses BB for interzone, then all calls between Zone 1 and TDM resources in Zone 4 will use the G.729 codec (BB) (even if those Zone 4 resources are physically located in the same site as Zone 1).

From my way of thinking, all physical TDM equipment should be configured in the zone appropriate to its physical location in the network. Zone 1 VoIP users should share the zone with Zone 1 TDM equipment/users. But, if there is TDM resources at the branch office, they would go into Zone 2 (using the above example).

Virtual Trunking would be a special case, in that Virtual Trunks are not used by IP Phones, so putting them in a separate zone has no negative impact (and the fact that they show up as a zone type in the Intrazone/Interzone print out tells us it was Nortel’s Design Intent that they be put in their own zone.)

Jul 282012

IP Phones LLDP/ADAC Boot Procedure

All Avaya IP Phones, in factory default mode, are configured with ADAC/LLDP enabled. LLDP, or Link Layer Discovery Protocol (802.1ab), and ADAC, or Auto-Detection Auto-Configuration, can operate independently. ADAC provides automatic configuration of the Data Switch port, while LLDP provides a method for the device to communicate it’s nature to the switch it connects to. ADAC can also utilize LLDP to identify the connected device type and configure the Data Switch port. Additionally, LLDP supports LLDP-MED, Link Layer Discovery Protocol Media Endpoint Discovery which permits configuration information to be provided to the Media Endpoint (i.e., IP Phone) automatically from the data switch without the need to communicate with additional components on the network.

LLDP can delay the IP Phone boot process if ADAC/LLDP is not enabled on the network. LLDP is only recommended for those environments where LLDP or ADAC is enabled on the data network.


See the documentation for precedence of configuration methods. Manual configuration always takes precedence over automatic configuration values. If any unexpected behavior is experienced, consider factory defaulting the IP Phone as part of the troubleshooting process.


This LLDP Process diagram does not include LLDP-MED behavior.

Configuration Methods

  • By default, LLDP is enabled on the IP Phone. If LLDP is enabled on the Phone, ADAC can use LLDP to identify the connected device for ADAC configuration.
    If LLDP is disabled on the IP Phone, ADAC can use the MAC Address of the connected device for ADAC configuration.
  • After DHCP Provisioning or “Manual Provisioning” via TFTP, LLDP can be disabled. However, if LLDP is manually disabled on the IP Phone, it cannot be enabled via DHCP or “Manual Provisioning”.
  • LLDP-MED can configure the following settings on the IP Phone:
    • Voice VLAN ID
    • Control pBits (priority bits, Layer 2)
    • Media pBits (priority bits, Layer 2)
    • Data pBits (priority bits, Layer 2)
    • DSCP Override of Voice and Media DSCP (DiffServ Code Point, Layer 3)

Recommended Scenarios

  • LLDP on the IP Phone is recommended where the data network is using LLDP or ADAC. If LLDP or ADAC are not enabled on the data network, then LLDP should be disabled on the phone to speed the IP Phone boot process.
  • ADAC is recommended on data networks where security is critical, or where the size of the voice network creates a requirement to distribute the boot-provisioning-registration load.
  • LLDP-MED on the data network is recommended to speed the boot procedure where Voice VLAN, pBits and DSCP values are currently configured via DHCP or TFTP (i.e., “Manual Provisioning”) and/or where security is critical.

For environments where Auto-VLAN ID is enabled, the IP Phone must request DHCP (acquiring an IP address and the Auto-VLAN ID from the DHCP server), then release the DHCP IP address and perform DHCPDISCOVERy a second time to obtain the DHCP Provisioning information. By configuring the Voice VLAN ID via LLDP/ADAC or LLDP-MED, you can eliminate the second DHCP request (reducing load on the DHCP server.)

Additionally, ADAC/LLDP and LLDP-MED increase network security by configuring the network port to the Voice VLAN instead of trusting the 801.1q VLAN ID supplied by the connected device. (A standard converged scenario is to configure the data port to trust the VLAN ID tag provided by the attached device, but this does introduce the potential for security issues and is not recommended for a secure environment.)


Auto-Detection, Auto-Configuration (ADAC) provides an automatic configuration method for Avaya (formerly-Nortel) IP Phones.

ADAC Modes configured on the Ethernet Routing Switch (ERS):

  • Untagged-Frames-Basic – Used when the IP phones are sending untagged traffic.
  • Untagged-Frames-Advanced – Used when the IP Phones are sending untagged traffic.
  • Tagged Frames – Used when the IP Phones are sending tagged traffic.

Detection Modes configured on each port (one or both must be assigned to each port to enable ADAC for that port):

  • MAC address
  • Link Layer Discovery Protocol, or LLDP (IEEE 802.1ab)

MAC address detection supports up to 32 devices per port. LLDP detection supports up to 16 devices per port. A standard scenario is one IP Phone and one PC on each port.

Ports are polled every two (2) seconds for their auto-configuration state. ADAC will be applied against a port when one of the following two conditions are true:

  • op-mode untagged-frames-basic or op-mode untagged-frames-advanced, at least one IP phone is detected on the port and no non-IP phones are detected on the port.
  • op-mode tagged frames, and at least one IP phone is detected on the port.

ADAC is removed if any of these conditions become true:

  • Auto-detect becomes disabled on the port.
  • The ports operational state becomes disabled.
  • op-mode untagged-frames-basic or op-mode untagged-frames-advanced, and at least one non-IP phone device is detected on the port.
  • There are no IP Phones detected on the port and the link is down.
  • If the link is still up but there are no IP phones on the port, auto-configuration is disabled after an aging period of about 90 seconds.
  • If all MAC addresses belonging to Nortel IP Phones on a port age out, the auto-configuration settings are removed from the port.
ADAC will automatically update VLAN ID and PVID settings under the above specified conditions. This will cause manual settings to be over-written and is design intent. Michael McNamaranoted in 2008 that an earlier release of the ERS software will maintain the VLAN ID and PVID as the port is configured when ADAC is enabled. For example, if you configure the default VLAN ID and PVID as 10 and the ADAC Voice VLAN and PVID as 20, then change the VLAN ID and PVID to 30, when ADAC resets the port configuration (either enabling ADAC or disabling ADAC), the configuration will go back to what it was configured at the time ADAC was enabled. (VLAN/PVID 10 if ADAC is disabling, VLAN/PVID 20 if ADAC is enabling.)I do not have a lab ERS to play with this configuration at this time, so I’ll just make a note that it’s something to watch for.

Feb 132012

So by it’s definition an access port can only belong to one VLAN while a trunk port can belong to multiple VLANs. It’s important to distinguish that we’re talking about single ports. A trunk group or trunk port group is made up of multiple ports which are combined into a single virtual port. Protocols such as MultiLink Trunking (Avaya), EtherChannel (Cisco) and LACP provide the ability to combine multiple trunk ports into a single virtual interface providing redundancy and additional bandwidth.

via untagAll vs tagAll on Avaya Ethernet Routing Switches | Michael McNamara.

The above is an oversimplification, but I thought Mr. McNamara did a reasonable job of explaining the four 802.1q tagging methods (untagAll, untagPvidOnly, tagPvidOnly, tagAll).

My attribution of oversimplification comes when you combine the above claim that access ports can only belong to one VLAN, with the definition of untagPvidOnly then combine it with the instruction to use untagPvidOnly with converged VoIP/Desktop solutions (plug the PC in to the phone, which plugs in to the LAN). Since converged VoIP solutions that place the VoIP and desktop traffic in different VLANs recommend untagPvidOnly and because doing so causes the port to belong to more than one VLAN which creates a conflict with his summry of access vs trunk.

If you skip over this one little self contradiction, the rest of the information is quite good. Once I get through my CS1000 IP Phone registration process, I might do a series on the Ethernet Routing Switches.

Jan 232012

This article provides an overview of the Avaya IP Phone registration procedure (for UNIStim IP Phones)

When the phone is powered up, the following happens:

  1. NVRAM (non-volatile RA) is loaded, including the local configuration information. Any configuration options set to manual on the phone will overwrite automatic configuration information received.
    If you experience any problems with any part of the process, use the IP Phone Factory Default reset procedure to clear all local configuration settings.


  2. Phone then boots and determines if data switch provides LLDP or ADAC. This setting can be disabled manually, via DHCP or via manual provisioning. Unless this is disabled manually, the phone will always check LLDP/ADAC when it first boots.
    Leaving LLDP/ADAC enabled when it is not supported by the Layer 2 switching equipment installed at the site can extend boot times for IP Phone devices. While LLDP/ADAC is enabled in a factory default configuration, it is recommended that this be disabled unless it is specifically supported by the networking environment.


  3. The phone then requests DHCP. If DHCP is available it processes the DHCP information.
  4. If a provisioning server is provided via DHCP Option 66, DHCP or manually configured on the IP Phone, then the the IP Phone requests the system.prv and <TYPE>.cfg from the HTTP or TFTP servers. While there is a lot more available under manual provisioning than just firmware upgrades (and while I will be writing an article to cover those topics later), I have only written the manual firmware upgrade article.
  5. Then the phone attempts to contact the S1 and S2 (primary signaling server and failover signaling server). If the phone cannot make a connection to the signaling server (or that information isn’t provided via any of the configuration methods available: manual, DHCP or provisioning server) then the Phone reboots and tries again.
  6. If a connection is made to either the primary or failover signaling server, then the phone will register and proceed with attempting to connect to External Application Servers (XAS) such as the Application Server 1000. A lot of the functionality that was originally relegated to an External Server (screen savers, backgrounds, some directory functions) have been incorporated in to the base firmware/functionality of the IP Phones. Others still require an XAS. For more information on this, contact an authorized Avaya distributor.

The only information that is critical to an IP phone for the boot process is:

  1. Set IP address, subnet mask and gateway
  2. Primary signaling server (S1) IP address, Port, Action and Retry values
  3. Node and TN

When troubleshooting, eliminate variables by resetting the unit back to factory default and then configure only the minimum number of settings needed to establish connectivity (start with manually configuring the phone, then migrate components of the configuration back to auto to determine where the process fails.)