Nov 122013
 

Issue:

  • Ports, Cards or entire Shelves disable during midnight routine.
  • NWS messages generate during midnight routine
% NWS301 8 0 : -1 -2 -3 -4 -5 -6
%
% NWS101 1 : 24
%
% NWS211 24 : 0 1

Environment:

  • Avaya CS1000, all releases
  • Nortel Meridian-1, all releases
  • Digital phones only

Cause:

  • Cabling issues and/or unplugged phones cause “continuity test” failures during midnight routines.
  • After sufficient number of port-based continuity tests fail a card reports a failure
  • After sufficient number of card-based continuity tests report failures the shelf reports a failure

Solution:

  • If a phone is removed from the jack, restore or de-program
  • If a phone cabling issue exists, fix

Comments:

  • I saw this for the first time when I was working for HellerEhrman. The site’s telecom tech would deploy phones where needed, moving phones from existing workspaces to new workspaces and document in a personal document all unused terminal numbers (TNs) for later re-use but did not de-program them. This was “speedier” for them than removing & reprogramming TNs. Doing this allowed them to save the time of programming the entire TN, they just plugged a new phone in, re-enabled the port, changed the DN and they were good.
  • However, users began reporting phones were disabling during midnight routine and had to be manually re-enabled next business morning.
  • Issue escalated to me (Firm-wide Telecom team).
  • I’d never seen this particular issue before and did not know root cause.
  • I performed routine troubleshooting and recommended several corrective actions, including routine maintenance (cleaning up TNs, etc.) but having no authority over the site tech (not being able to force them to do the recommended work and I did not know that the absence of routine maintenance was the proximate cause) I was told to escalate to Nortel (via our Service Provider).
  • Service provider had not seen it before and escalated to Nortel
  • Nortel indicated performance of routine maintenance. i.e., clean up all programmed TNs that were not going to be put back into service or reconnect a phone to any TN that needed to remain.
  • Issue resolved.

I’ve seen a couple of these tickets recently at my place of employment. So far each one appears to be the same cause/solution. I’ll post a comment later if I learn anything new.

Nov 072013
 

Issue

When IP Phones enter a reboot loop, attempt to “upgrade”, fail, then reboot again, or
When IP Phones enter a reboot loop, attempt to “upgrade”, fail with “FW authentication failure”, then reboot again

Environment

Avaya CS1000

UNIStim 5.0 or earlier

Avaya IP Phone 1100, Avaya IP Phone 1200

Cause

UNIStim firmware is digitally signed.

Signature has an expiration date.

UNIStim versions prior to 5.0 had shorter expiration dates.

New IP Phone hardware will not load firmware with expired signatures.

Source: http://downloads.avaya.com/css/P8/documents/100152833

Solution

Use UNIStim 5.1 or later firmware.

Avaya has applied a digital signature with a 10 year expiration date to UNIStim 5.1 and later.

UNIStim 5.5.1 (C8T) released in Aug 2013.

I updated my Google drive table of UNIStim firmware releases.

 

Nov 032013
 

Recently worked on an AACC (Avaya Aura Contact Center) where the partitioning of the server was determined to be the cause of the problem. While Disk Management (diskmgmt.msc) is easily accessible from START>>RUN, a screenshot is not quite as portable as text. To that end (and as a recommendation for addition to the Nortel Enterprise Audit Tool, or NEAT, used to survey Contact Center servers for Avaya engineering), I put together a script to query WMI (Windows Management Instrumentation) for the necessary information.

WMI Objects:

  • Win32_DiskDrive
  • Win32_DiskDriveToDiskPartition
  • Win32_DiskPartition
  • Win32_LogicalDiskToPartition

Using WMI queries against these objects you can derive:

  • Win32_DiskDrive => Physical Device ID (.\\.\PHYSICALDRIVE0\)
  • Win32_DiskPartition => Partition Device ID (Disk #0, Partition #1) and a derived type (e.g., Simple Volume? Primary Partition? Extended Partition/Logical Drive?)
  • Win32_LogicalDiskToPartition => Logical Drive Device ID (D:)

For quick “automated” checks of a system to verify compliance with engineering guidelines, this is a must.

Sample output:

\\.\PHYSICALDRIVE0,Disk #0, Partition #2,Basic,True,C:,Primary Partition
\\.\PHYSICALDRIVE0,Disk #0, Partition #3,Basic,False,D:,Extended Partition/Logical Drives
\\.\PHYSICALDRIVE0,Disk #0, Partition #3,Basic,False,F:,Extended Partition/Logical Drives
\\.\PHYSICALDRIVE0,Disk #0, Partition #3,Basic,False,G:,Extended Partition/Logical Drives
\\.\PHYSICALDRIVE0,Disk #0, Partition #3,Basic,False,T:,Extended Partition/Logical Drives

and

\\.\PHYSICALDRIVE0,Disk #0, Partition #2,Dynamic,True,C:,Simple Volume?
 \\.\PHYSICALDRIVE0,Disk #0, Partition #3,Dynamic,True,D:,Simple Volume?
 \\.\PHYSICALDRIVE0,Disk #0, Partition #3,Dynamic,True,F:,Simple Volume?
 \\.\PHYSICALDRIVE0,Disk #0, Partition #3,Dynamic,True,G:,Simple Volume?

The cool thing is that the script is applicable for all systems going back to Windows 2000 (Symposium 4 if I recall correctly) when the WMI query objects were instantiated in the OS by Microsoft.

May 062013
 

Ever have a massive block of D-Channel (DCH) traces that you need to parse into something meaningful? I do. In fact, for a couple of customers it’s become a regular occurrence the last few months. So, in an effort to make life easier for myself I wrote a nifty little parse utility then converted it to a PHP/HTML page.

My web hosting company has some hard limits– like you can’t use more than 30 seconds of CPU time without a script being killed… so super-long parse logs is out of the question. (They’ll simply get killed by the web host.)

Also, to ensure that it doesn’t get hammered, I wrote into the code a lock so that only one-person can access it at the same time. There isn’t yet an ability to IP Ban, but that will get added– so that if anyone hammers this I can block them from being able to use it.

So, a few etiquette requests:

  • be gentle– keep log sizes under 10k lines and don’t hammer it over and over.
  • If you see a warning that someone else is using it, come back in 30-seconds to 5-minutes.
  • If you’d like something like this, but don’t want to be limited by the freeware constraints, contact me and we can negotiate something.

Apr 272013
 

For all I know, this system is still there. Certainly the company that ended up buying the Firstworld call center still exists and still operates facilities at this location. (Shaw & Minnewawa, Clovis, California).

My introduction to Nortel systems was kind of round-about. I was working for a company in Fresno that was on a Lucent Definity G3S (?) and things were not looking very good, so I started shopping my options and got a job that was actually a step up in both pay and stability (Firstworld) as a report writer. I crash-coursed on SQL (I was actually MSCE certified on SQL but had very limited need to use the knowledge on a day-to-day business before this) and when I got the job I did nothing but review the design of the call center & write reports… for about 90 days.

Then, the guy who got me hired took a job with Quantum (which closed down that facility within 6 months of hiring him on) and left me the job of running the technical part of the Call Center. I was given an incredible opportunity and I leveraged it into the career I have today.

From this job I went to Shared Technologies Fairchild (an Intermedia company) as a field engineer.

Apr 272013
 

Doing some housecleaning on my harddrive and as I was going through files that I’ve never accessed nor updated, I came across this ancient visio diagram. I received the Visio stencils from an awesome sales person at Shared Technologies Fairchild (back during the Intermedia days) and used them regularly to produce high-quality documentation for big customers.

I don’t mind posting this because I happen to know that the customer for whom this was produced has forklifted their Nortel system in favor of another phone system manufacturer. They were a customer of my current employer back when I started 5 years ago, and they were in the process of migrating away from Nortel at that time.

I can’t say that I’m surprised, looking back. When I was working with that customer 12 years ago they canned their telecom manager who knew something about the Nortel systems (for reasons I never fully understood) and hired this guy who knew absolutely nothing about Nortel systems. He was aggressively ignorant and self absorbed. After more than a month listening to him complain endlessly about how he thought the Nortel system was a piece of garbage, I took him out to lunch and asked him a simple question. After which, he never complained to me again… but I had no doubt that he agitated to remove the system from his start date until he left. (He is no longer there.)

I think it was the following year we upgraded this system from a 61C to an 81C, pulled out the Meridian Mail and put in a Callpilot (as Callpilot-Symposium integration got better), added a Telstrat IDVR. I was pretty proud of all the technology that was there and how much I learned in the short amount of time I worked with this customer.

Aug 092012
 

Zone Bandwidth Strategy

One of the things I run into regularly with deployments is an issue where the installer improperly configures the Zone Bandwidth Strategy. This information can be checked by printing the interzone and intrazone configurations in LD 117.

=> prt zone
Use the PRT INTERZONE or PRT INTRAZONE

=> prt interzone
 ______________________________________________________________________________________________________________________
 |Near end  |Far end   |State| Type  |Stra|MO/ |QoS| Bandwidth | Sliding |  Usage  |  Quota  |Peak|  Calls  |  Alarm  |
 |          |          |     |       |tegy|BMG/|Fac|Configured |   max   |         |         |    |         |         |
 |          |          |     |       |    |VTRK|tor|           |         |         |         |    |         |         |
 |----------|----------|-----|-------|----|----|---|-----------|---------|---------|---------|----|---------|---------|
 |Zone|VPNI |Zone|VPNI |     |       |    |    | % |    kbps   |   kbps  |   kbps  |   kbps  | %  |   Cph   |   Aph   |
 |----|-----|----|-----|-----|-------|----|----|---|-----------|---------|---------|---------|----|---------|---------|
 |   1|     |    |     | ENL |SHARED |  BB|  MO|   |    1000000|         |      380|        0|   0|         |         |
 |--------------------------------------------------------------------------------------------------------------------|
 |   5|     |    |     | ENL |SHARED |  BB|VTRK|   |    1000000|         |      380|        0|   0|         |         |
 |--------------------------------------------------------------------------------------------------------------------|
 |  42|     |    |     | ENL |SHARED |  BB|  MO|   |    1000000|         |        0|        0|   0|         |         |
 |--------------------------------------------------------------------------------------------------------------------|

=> prt intrazone
 _________________________________________________________________________
 |Zone|State| Type  |Strategy|MO/ | Bandwidth |  Usage  |  Quota  | Peak |
 |    |     |       |        |BMG/|    kbps   |   kbps  |   kbps  |   %  |
 |    |     |       |        |VTRK|           |         |         |      |
 |----|-----|-------|--------|----|-----------|---------|---------|------|
 |   1| ENL |SHARED |   BQ   |  MO|    1000000|      380|        0|    0 |
 |-----------------------------------------------------------------------|
 |   5| ENL |SHARED |   BQ   |VTRK|    1000000|      380|        0|    0 |
 |-----------------------------------------------------------------------|
 |  42| ENL |SHARED |   BQ   |  MO|    1000000|        0|        0|    0 |
 |-----------------------------------------------------------------------|
  Number of Zones configured = 3

=>

The Strategy column for intrazone and interzone can be configured as either:

  • BQ aka Best Quality aka G.711 codec selection
  • BB aka Best Bandwidth aka G.729 codec selection

BB should be used when a WAN interface is handling VoIP with the following limitations/restrictions:

  • Bandwidth shaping or rate limiting restrictions
  • Bandwidth availability limitations (either the pipe is too small, or other usage + VoIP traffic exceeds available bandwidth)
  • No QoS policies configured

Even if you believe your WAN can support the bandwidth demands of VoIP + other usage, if you are experiencing any quality issues with VoIP, it’s worth changing your bandwidth strategy to BB (i.e., Best Bandwidth, G.729) as part of your troubleshooting efforts. I also recommend that you configure the available bandwidth to match the actual available bandwidth.

Interzone

As the word root suggests, the Interzone Strategy is used for all VoIP calls that leave the zone wherein the caller is configured. This strategy is also used for calls that are inter-site.

Intrazone

As the word root suggests, the Intrazone Strategy is used for all VoIP calls within the zone.

Purpose of Zones

Zones provide a way of controlling the behavior of VoIP and QoS within a site (and between sites). Zones work best when used to represent physical locations.

Example

Zone 1 is set up for all phones within a site (on the LAN). BQ strategy is used for intrazone and BB is used for interzone.

Zone 2 is set up for all phones at a branch office. BQ strategy is used for intrazone and BB is used for interzone.

Zone 3 is set up for all softphones connected via VPN (remote employees). BB strategy is used for both intrazone and interzone.

If a caller in Zone 1 calls anyone in Zone 1, they use G.711 (BQ). If they call anyone outside of Zone 1 (a VPN user softphone, or the branch office) they use the G.729 (BB strategy) codec.

If a caller in Zone 2 calls anyone in Zone 2, they use G.711 (BQ), but calls out of Zone 2 use G.729 (BB). Just like Zone 1.

If a caller in Zone 3 calls anyone they use G.729 (BB).

Mixed VoIP/TDM environments

TDM resources are sometimes configured to be in a separate zone by installers. I’ve never understood this, because following the logic above, if all the TDM equipment (Media Gateways, MGCs, MC32s, IP Trunks, etc.) are put in Zone 4 and everyone uses BB for interzone, then all calls between Zone 1 and TDM resources in Zone 4 will use the G.729 codec (BB) (even if those Zone 4 resources are physically located in the same site as Zone 1).

From my way of thinking, all physical TDM equipment should be configured in the zone appropriate to its physical location in the network. Zone 1 VoIP users should share the zone with Zone 1 TDM equipment/users. But, if there is TDM resources at the branch office, they would go into Zone 2 (using the above example).

Virtual Trunking would be a special case, in that Virtual Trunks are not used by IP Phones, so putting them in a separate zone has no negative impact (and the fact that they show up as a zone type in the Intrazone/Interzone print out tells us it was Nortel’s Design Intent that they be put in their own zone.)

Jan 252012
 

Pairing the Jawbone Prime or Jawbone 2 Bluetooth Headset to Your Cell Phone

Put your phone into pairing mode

This can usually be accomplished by going under settings in your menu and selecting Bluetooth. Follow the prompts to “find a new device.” If you are having problems, refer to your phone’s user guide.

Put your Jawbone 2 Bluetooth headset into pairing mode

The first time you turn your Jawbone on it will immediately go into pairing mode. If you need to manually put the Jawbone into pairing mode, start with headset off. Hold down the NoiseAssasin button and power on. Continue to hold down button for 2 seconds. Headset will flash red and white when it is in paring mode.

If you are having trouble with this step, try pushing in the NoiseAssassin button just a half a second before you push in the talk button.

Select device and enter in universal keyYour phone may or may not ask you to enter in a key code. If it does, the code is always four zeros: 0000

You are paired up!

via Jawbone Bluetooth Headset Pairing Guide | Headsets.com – America’s Headset Specialists.

Jan 232012
 

This article provides an overview of the Avaya IP Phone registration procedure (for UNIStim IP Phones)

When the phone is powered up, the following happens:

  1. NVRAM (non-volatile RA) is loaded, including the local configuration information. Any configuration options set to manual on the phone will overwrite automatic configuration information received.
    NOTE
    If you experience any problems with any part of the process, use the IP Phone Factory Default reset procedure to clear all local configuration settings.

     

  2. Phone then boots and determines if data switch provides LLDP or ADAC. This setting can be disabled manually, via DHCP or via manual provisioning. Unless this is disabled manually, the phone will always check LLDP/ADAC when it first boots.
    NOTE
    Leaving LLDP/ADAC enabled when it is not supported by the Layer 2 switching equipment installed at the site can extend boot times for IP Phone devices. While LLDP/ADAC is enabled in a factory default configuration, it is recommended that this be disabled unless it is specifically supported by the networking environment.

     

  3. The phone then requests DHCP. If DHCP is available it processes the DHCP information.
  4. If a provisioning server is provided via DHCP Option 66, DHCP or manually configured on the IP Phone, then the the IP Phone requests the system.prv and <TYPE>.cfg from the HTTP or TFTP servers. While there is a lot more available under manual provisioning than just firmware upgrades (and while I will be writing an article to cover those topics later), I have only written the manual firmware upgrade article.
  5. Then the phone attempts to contact the S1 and S2 (primary signaling server and failover signaling server). If the phone cannot make a connection to the signaling server (or that information isn’t provided via any of the configuration methods available: manual, DHCP or provisioning server) then the Phone reboots and tries again.
  6. If a connection is made to either the primary or failover signaling server, then the phone will register and proceed with attempting to connect to External Application Servers (XAS) such as the Application Server 1000. A lot of the functionality that was originally relegated to an External Server (screen savers, backgrounds, some directory functions) have been incorporated in to the base firmware/functionality of the IP Phones. Others still require an XAS. For more information on this, contact an authorized Avaya distributor.

The only information that is critical to an IP phone for the boot process is:

  1. Set IP address, subnet mask and gateway
  2. Primary signaling server (S1) IP address, Port, Action and Retry values
  3. Node and TN

When troubleshooting, eliminate variables by resetting the unit back to factory default and then configure only the minimum number of settings needed to establish connectivity (start with manually configuring the phone, then migrate components of the configuration back to auto to determine where the process fails.)

Jan 212012
 

The 2033 Conference phone firmware versions are approximations. I’m recreating this table from documentation, since Avaya doesn’t seem to care to keep such a thing current. I do have some old UNIStim 3.0 documentation. I’m debating whether or not to include the now-manufacture-discontinued IP phones (i.e., i2001, i2002, i2004). I’m very tempted to do so, since I still see a lot of 4.5 and earlier systems (which still support the older IP Phones.)