Jun 112014
 

https://support.avaya.com/downloads/download-details.action?contentId=C20145311538319080_3&productId=P0599&releaseId=UNIStim%205.x

Updated my Google doc table of IP Phone firmware:

Nov 072013
 

Issue

When IP Phones enter a reboot loop, attempt to “upgrade”, fail, then reboot again, or
When IP Phones enter a reboot loop, attempt to “upgrade”, fail with “FW authentication failure”, then reboot again

Environment

Avaya CS1000

UNIStim 5.0 or earlier

Avaya IP Phone 1100, Avaya IP Phone 1200

Cause

UNIStim firmware is digitally signed.

Signature has an expiration date.

UNIStim versions prior to 5.0 had shorter expiration dates.

New IP Phone hardware will not load firmware with expired signatures.

Source: http://downloads.avaya.com/css/P8/documents/100152833

Solution

Use UNIStim 5.1 or later firmware.

Avaya has applied a digital signature with a 10 year expiration date to UNIStim 5.1 and later.

UNIStim 5.5.1 (C8T) released in Aug 2013.

I updated my Google drive table of UNIStim firmware releases.

 

Sep 192013
 

LLDP-MED for Avaya CS1000 IP PhonesIncreasing boot efficiency is one of those things I’m working on. My personal or work PC, my IP Phone, systems I manage. The less time I have to spend sitting around waiting for something to boot up is more time doing something productive. On the PC, that involves looking at your startup folder, your registry run folders and removing any unnecessary services from automatic startup.

For Avaya CS1000 IP Phones, that involves looking at the config and determining which features can be added or removed to achieve an optimal boot up sequence.

Although my 4st post is not live yet (when it is, it will be here), in it I cover Link Layer Discovery Protocol (LLDP) and how it applies to Avaya CS1000 IP Phone deployment. On of the biggest inefficiencies I’ve found in CS1000 IP Phone deployments is where customers leave LLDP enabled but don’t use it.

ZzzWaiting for LLDP-MED (Link Layer Discovery Protocol, Media Endpoint Discovery) can add as much as 30 seconds delay to the boot process… So disable it if you’re not using it!

With stickiness, you can configure the Phone to not use LLDP on bootup, or you can disable it manually at each phone by turning it off.

On the other hand, if use LLDP you might increase boot efficiency by distributing the configuration of the IP Phones and reducing dependency upon DHCP. If you want to configure the Voice VLAN but don’t use LLDP, your options are to manually configure each IP Phone or use the VLAN-A option to assign a Voice VLAN ID.

Avaya CS1000 IP Phone, DHCP provisioning behaviorIf you use DHCP though, you’re going to be querying the DHCP server (or multiple servers) multiple times.

It’s certainly faster than waiting for LLDP-MED to time out, but using LLDP-MED is faster than multiple DHCP queries (Although talking a fraction of the delay caused by LLDP-MED being enabled and unused.)

It’s also a good idea to reduce the number of retries to allow the IP Phone to failover to an alternate signaling server (i.e. Connect Server) more quickly.

Take away:

  • If you’re not using a feature, disable it. Your phones will boot faster and you’ll recover more quickly from maintenance windows or disaster.
  • Nortel-i2004-B,s1ip=10.10.10.10;p1=4100;a1=1;r1=3;s2ip=10.10.10.20;p2=4100;a2=1;r2=3;vq=y;st=y;lldp=n;vvsource=a;

Sep 172013
 

Avaya CS1000 IP Phone registration process, DHCP (c) 2013 John Williams, DATARAVEIn my 3rd company blog article, I talked about DHCP and auto configuration via DHCP. There are fifteen different feature groups (IP Deskphones Fundamentals, NN43001-368, Appendix B: Provisioning the IP Phones) and 100+ different settings configurable via DHCP (I didn’t count them, I’m estimating). Some of the gotchas I’ve learned over the years are as follows:

  • As noted in the 3rd blog post, Stickiness can cause deployment issues for refurb phones– Always add a factory reset to your troubleshooting efforts (**RENEW[MAC]##), which was introduced in UNIStim 3.0, otherwise your refurb phone may not operate as expected when first deployed.
  • When you have multiple Signaling Servers– Always make sure that both Sig Servers are using the same IP Phone firmware, otherwise your phones may “upgrade” to an earlier firmware release when they failover, resulting on extended and unexpected downtime.
  • Know your current UNIStim firmware. There have been issues in the past where newer IP Phone hardware requires a minimum firmware level, which means that all IP Phones in the environment must be upgraded in order to deploy new phones.
  • Learn how to use PRT TNB commands in LD 20 and STAT IP commands in LD 117 (I’m actually planning on doing a future blog post on this topic, since the LD 117 commands can be used to also inventory your IP phones)–
    • Make sure that the TNB is programmed, otherwise you’ll get SRPT062/SRPT0062 – Request to register TN rejected. Reason = UNEQUIPPED.
    • Make sure the set isn’t deployed elsewhere, otherwise you’ll get SRPT062/SRPT0062 – Request to register TN rejected. Reason = DUPLICATE.
    • Make sure the set is the right TYPE, otherwise you’ll get SRPT062/SRPT0062 – Request to register TN rejected. Reason = WRONGTYPE.
    • Make sure the set is of the right TN_TYPE (covered in part 2 of the company series), otherwise you’ll get SRPT062/SRPT0062 – Request to register TN rejected. Reason = CANNOTCONVERT
      • 2004P1, 2004P2, 2210, 2211, 2007, 1140, 2050PC, 2050MC, 6140
      • 2002P1, 2002P2, 1120, 1220, 6120
      • 2001P2, 2033, 1110, 1210
      • 1230
      • 1150
  • Know how to troubleshoot your data network– Always make sure that your provisioning method is supported in the subnet where the IP phone is being deployed and know how to check the data network configuration, otherwise your phone might get stuck at “Starting DHCP…” or “Connecting to S1…” (or “Connecting to S2…”)
    • I can’t tell you how many times I’ve seen a telecom tech install an IP Phone into a brand new subnet which has no DHCP configuration options, or a new data switch without correct VLAN settings, and then request help– only to be landed with a bill for not setting up the new subnet correctly.

Another thing that I find extremely helpful is learning the provisioning cycle and the status messages which display on the IP Phone during boot up:

Provisioning Step Display message
LLDP Waiting for Cfg Data…
DHCP Starting DHCP…
Manual Provisioning Prov. 192.168.0.254 (or whatever IP you have)
system.prv reading…
(system.prv failed may display)Attempting TFTP…
Registration (pre-connect) Connecting to S1
Connecting to S2
Registration (post-connect) Connect Svc
Node & TN
Connect Svc
Forwarding…

This way, if you get stuck at a particular phase, you know where and can use that to determine your next troubleshooting step. For example;

  • If you get stuck at “Starting DHCP…” then you should check (a) is your DHCP server still running, (b) are you in a subnet that provides the DHCP options you need, (c) can your IP phone reach the DHCP server, (d) does the IP Phone firmware support the features you’re trying to push to it?
  • If you get stuck at “Attempting TFTP…”, (a) is your TFTP server still running, (b) can you IP Phone reach your TFTP server, (c) are your PRV files correctly configured?
  • If you get stuck at “Connecting to S1” (or S2), (a) is your IP Phone in the right VLAN, (b) Can the IP Phone reach the signaling server, (c) is the signaling server still running?
  • If you get stuck at “Connect Svc Forwarding…” then there is something wrong with your signaling server– it’s responded to the initial connect request, and it’s forwarded the request to the signaling server but the signaling server isn’t responding correctly to complete the registration process.

Aug 272013
 

Last week I talked about an overview of  the registration process and the different phases for the Avaya IP Phone registration process. This week my contribution to the company blog covers the essential information needed to deploy any IP phone. Understanding IP Phone deployment, essential information needed for all deployment scenarios. Part two of a six part series.

Getting started

I start with the assumption that you already know a little bit about the CS1000 architecture and programming– I tried to keep this article to 600-800 words, and even discounting some of the captures I put in the article it still came out to almost double the size. To do a really in depth review of this topic you need a lot more than just a six part series. But, I have plenty of other topics that need attention and while IP Phone deployment is an interesting topic (to me, because of the number of features available compared to those that are implemented in most sites) as a topic, it’s just one of many. Writing white papers is best left to people who have that as a job– amiright?

Unlike completely creative writing, the technical writing comes very easily. The problem with creative writing is always figuring out the answers that you don’t even really understand the questions to– if X happens, how does Y feel about it, how do they react, etc. When a writer gets stuck, often the problem is that either they’re not able to answer the question, or they may not even realize what question they need to answer. But, with technical writing, all you have to be is knowledgeable about your topic and have time to flip through the documentation.

Trimming the fat

But, I removed a lot of in depth detail that I started to slide into the original blog article and instead referenced the documentation. I figure I trimmed at least another 800 words from the article that gave a lot more detail on the use of the Nortel-i2004-A string, as well as a few other B-string mnemonic that I really like when I’m in charge of IP Phone deployment (or tasked with consulting with a customer.)

Trimming takes at least two passes.

Fact checking

Speaking of documentation, it was a challenge getting some of this history straight. I was around during the UNIStim 1.x days (we’re at 5.4 now), but the documentation doesn’t really talk about when features were implemented, changed or removed.

For instance, it wasn’t until 2.2-2.3 that the new Nortel-i2004-B string was introduced, but the latest UNIStim 5.4 documentation doesn’t say when it was introduced. This is important because if you’re on firmware prior to 2.3, you might not be able to support the B string format (I know this is unlikely– but I know of customers that are still on Meridian-1 software that was released before I got into Telecom in 1999.)

In addition to filling in the facts that I could validate through documentation, I also had to make sure that I hadn’t introduced any errors through typos or omissions. Being thorough takes a couple of passes.

Formatting

Selecting images to fill out a blog is tough for me– I struggle with the artistic creativity required for selecting images that work with the blog post. And, when I cannot find what I’m looking for (and when a google search fails me), I whip something up in Photoshop. You might think that’s creative, but it doesn’t feel that way.

Then, it’s on to getting the text formatting right. Another couple of passes there. Headers are properly coded with the correct H1,H2,H3 tag, words and phrases get the correct bold/underline/italics emphasis. Colored text, when used, can convey information subtly. (e.g., On my company blog articles, I use a dark green bolded text format to represent terms which are universal and not specific to the subject of my blog.)

Approval

Then I submit the blog post to the company approval process and if it gets approved, we schedule it for posting.

 

And thus, Part 2, IP Phone deployment, essential information goes live.

 

The full series will be:

  • Part 1, IP Phone registration process overviewA review of the basic registration process, the phases, a couple of tips and some hints of topics to come.
  • Part 2, Manual IP deployment, Partial and Full DCHP deployment – the essential information needed for all phone deployment scenarios.
  • Part 3, Full DHCP in more detail – the power of DHCP and auto configuration.
  • Part 4, integrating switch level registration – creating efficiencies in phone registration using LLDP/ADAC.
  • Part 5, Using TFTP provisioning for security and redundancy – advanced IP phone deployment techniques using TFTP provisioning.
  • Part 6, remote worker scenarios, NAT, VPN, Firewalls – advanced networking scenarios.

 

Aug 232013
 

The IP Phone Registration process articles are some of the most visited articles on my personal blog. So when my employer began a new initiative asking employees to contribute content to the business blog, I proposed and was greenlit to rewrite and publish my IP Phone Registration overview as a six part blog series.

There will be some rehash of information previously posted here, but there will also be parts that haven’t yet been covered.

The full series will be:

  • Part 1, IP Phone registration process overviewA review of the basic registration process, the phases, a couple of tips and some hints of topics to come.
  • Part 2, Manual IP deployment, Partial and Full DCHP deployment – the essential information needed for all phone deployment scenarios.
  • Part 3, Full DHCP in more detail – the power of DHCP and auto configuration.
  • Part 4, integrating switch level registration – creating efficiencies in phone registration using LLDP/ADAC.
  • Part 5, Using TFTP provisioning for security and redundancy – advanced IP phone deployment techniques using TFTP provisioning.
  • Part 6, remote worker scenarios, NAT, VPN, Firewalls – advanced networking scenarios.

Next week, I anticipate getting part 2 approved and published.

Now, for the “Call to Action”– I get special recognition at my job if my articles happen to be the most viewed articles over the next 6 months (compared to any of my co-worker’s articles.) So, I know this topic is popular and I’m hoping you’ll be tempted by more information on this topic.

Go ahead– do us both a favor and click through to the first article Understanding IP Phone deployment, registration process overview.

If you’d like to influence future articles, register to stay informed and ask your questions either via the registration process or the comments section via the Comments section.

Apr 282013
 

While working for a Nortel distributor a few years ago, after answering repeat questions (sometimes the same question 4+ times from the same individual) I put together an internal WIKI and documented things for all employees. This information ranged from corporate policy, forms, etc. and went even so far as to document technical procedures. It was a one-stop-shop for information– any time anyone asked me a question, I put the answer on the Wiki and then linked them to the wiki. I called it, appropriately, RTFM. I could have used Sharepoint, but we were operating on a budget. So I commandered a directory on an IIS server, installed mySQL and mediawiki. I’m sure there was more to it than that, but the point of this article isn’t to talk about my mediawiki installation experience.

Anyway…

One of the features introduced with the Nortel CP-PM (Call Processor, Pentium Mobile) and CS1000E chassis (I forget the exact chassis shown in the pictures below) was the redundant Network Interfaces for the Call Server in the CS slot.

The Call Server (CS card in the middle of the picture above) supports a HSP (High Speed Pipe), TLAN (Telephony LAN) and ELAN (Equipment LAN) network interface, but the CS only uses the ELAN (and maybe the HSP if you were in a redundant CPU configuration). The CS does not use or need the TLAN, since it did not process any VoIP traffic.

The MGC (Media Gateway Controller– the card on the bottom of the picture above) supports both the TLAN and ELAN interface, as well as pass-through for both TLAN and ELAN.

The back of the chassis has a redundant TLAN and ELAN interface for the MGC card.

 

The reason that this particular wiki page came about is because I was sent to three or four sites over a month period to do clean up on sites that had been installed by our “lead installer.” I say cleanup because in each installation, the problem was system instability due to customer maintenance of their data network.

You see, if you connect both the primary and redundant ELAN connections on the MGC, you can take either data switch down and the system will keep on chugging without impact… and, by plugging the CS into the ELAN-passthrough port, the CS could simulate an increase in network redundancy (i.e., as long as the MGC remained up, the CS would retain communication with all other components over the ELAN through whichever ELAN port was still live and working on the MGC– of course, if the MGC rebooted or if both NICs went down, became unplugged, the data switches were rebooted, whatever– then the whole house of cards came down, same as if you didn’t have redundancy.)

So after going out to each of these and being asked to install the redundancy, I went back to our “lead installer” and asked him to make it part of his installs in the future. His response: It doesn’t work that way.

So I went to a co-worker, verified my understanding of how things worked, and then asked my co-worker to approach him and try again. The lead installer’s response: Yea, the senior remote engineer said the same thing to me, but it doesn’t work that way.

When asked how he thought it worked, he said that the redundant ports were not redundant and that the passthrough port didn’t work.

So I proceeded to document, with photos (the ones you see above), how it worked and put together a wiki page.

 

Then I involved management.

 

For a small distributor to have this kind of inefficiencies (the lead installer does a bad job, then someone else has to go out and clean up the work), it was not only killing budgets on projects and eating “warranty” hours after the customer went live… it was also putting me on-site, which meant less time to focus on my primary responsibilities.

 

I took our lab system and set it up the proper way, then I invited management to a demonstration where I demonstrated by unplugging and plugging in data cables how stability/redundancy worked. Then I showed them what happened when certain data cables were left unplugged or unused (the way the lead installer was doing it).

 

When management addressed the issue with the lead installer, he quit rather than amend his ways. Needless to say, I was shocked by this response.

Jan 142013
 

The networking giant warned of the vulnerability on Wednesday, almost two weeks after a security expert demonstrated how people with physical access to the phones could cause them to execute malicious code. Cisco plans to release a stop-gap software patch later this month for the weakness, which affects several models in the Cisco Unified IP Phone 7900 series. The vulnerability can also be exploited remotely over corporate networks, although Cisco has issued workarounds to make those hacks more difficult.

via Hack turns the Cisco phone on your desk into a remote bugging device | Ars Technica.

Aug 092012
 

Zone Bandwidth Strategy

One of the things I run into regularly with deployments is an issue where the installer improperly configures the Zone Bandwidth Strategy. This information can be checked by printing the interzone and intrazone configurations in LD 117.

=> prt zone
Use the PRT INTERZONE or PRT INTRAZONE

=> prt interzone
 ______________________________________________________________________________________________________________________
 |Near end  |Far end   |State| Type  |Stra|MO/ |QoS| Bandwidth | Sliding |  Usage  |  Quota  |Peak|  Calls  |  Alarm  |
 |          |          |     |       |tegy|BMG/|Fac|Configured |   max   |         |         |    |         |         |
 |          |          |     |       |    |VTRK|tor|           |         |         |         |    |         |         |
 |----------|----------|-----|-------|----|----|---|-----------|---------|---------|---------|----|---------|---------|
 |Zone|VPNI |Zone|VPNI |     |       |    |    | % |    kbps   |   kbps  |   kbps  |   kbps  | %  |   Cph   |   Aph   |
 |----|-----|----|-----|-----|-------|----|----|---|-----------|---------|---------|---------|----|---------|---------|
 |   1|     |    |     | ENL |SHARED |  BB|  MO|   |    1000000|         |      380|        0|   0|         |         |
 |--------------------------------------------------------------------------------------------------------------------|
 |   5|     |    |     | ENL |SHARED |  BB|VTRK|   |    1000000|         |      380|        0|   0|         |         |
 |--------------------------------------------------------------------------------------------------------------------|
 |  42|     |    |     | ENL |SHARED |  BB|  MO|   |    1000000|         |        0|        0|   0|         |         |
 |--------------------------------------------------------------------------------------------------------------------|

=> prt intrazone
 _________________________________________________________________________
 |Zone|State| Type  |Strategy|MO/ | Bandwidth |  Usage  |  Quota  | Peak |
 |    |     |       |        |BMG/|    kbps   |   kbps  |   kbps  |   %  |
 |    |     |       |        |VTRK|           |         |         |      |
 |----|-----|-------|--------|----|-----------|---------|---------|------|
 |   1| ENL |SHARED |   BQ   |  MO|    1000000|      380|        0|    0 |
 |-----------------------------------------------------------------------|
 |   5| ENL |SHARED |   BQ   |VTRK|    1000000|      380|        0|    0 |
 |-----------------------------------------------------------------------|
 |  42| ENL |SHARED |   BQ   |  MO|    1000000|        0|        0|    0 |
 |-----------------------------------------------------------------------|
  Number of Zones configured = 3

=>

The Strategy column for intrazone and interzone can be configured as either:

  • BQ aka Best Quality aka G.711 codec selection
  • BB aka Best Bandwidth aka G.729 codec selection

BB should be used when a WAN interface is handling VoIP with the following limitations/restrictions:

  • Bandwidth shaping or rate limiting restrictions
  • Bandwidth availability limitations (either the pipe is too small, or other usage + VoIP traffic exceeds available bandwidth)
  • No QoS policies configured

Even if you believe your WAN can support the bandwidth demands of VoIP + other usage, if you are experiencing any quality issues with VoIP, it’s worth changing your bandwidth strategy to BB (i.e., Best Bandwidth, G.729) as part of your troubleshooting efforts. I also recommend that you configure the available bandwidth to match the actual available bandwidth.

Interzone

As the word root suggests, the Interzone Strategy is used for all VoIP calls that leave the zone wherein the caller is configured. This strategy is also used for calls that are inter-site.

Intrazone

As the word root suggests, the Intrazone Strategy is used for all VoIP calls within the zone.

Purpose of Zones

Zones provide a way of controlling the behavior of VoIP and QoS within a site (and between sites). Zones work best when used to represent physical locations.

Example

Zone 1 is set up for all phones within a site (on the LAN). BQ strategy is used for intrazone and BB is used for interzone.

Zone 2 is set up for all phones at a branch office. BQ strategy is used for intrazone and BB is used for interzone.

Zone 3 is set up for all softphones connected via VPN (remote employees). BB strategy is used for both intrazone and interzone.

If a caller in Zone 1 calls anyone in Zone 1, they use G.711 (BQ). If they call anyone outside of Zone 1 (a VPN user softphone, or the branch office) they use the G.729 (BB strategy) codec.

If a caller in Zone 2 calls anyone in Zone 2, they use G.711 (BQ), but calls out of Zone 2 use G.729 (BB). Just like Zone 1.

If a caller in Zone 3 calls anyone they use G.729 (BB).

Mixed VoIP/TDM environments

TDM resources are sometimes configured to be in a separate zone by installers. I’ve never understood this, because following the logic above, if all the TDM equipment (Media Gateways, MGCs, MC32s, IP Trunks, etc.) are put in Zone 4 and everyone uses BB for interzone, then all calls between Zone 1 and TDM resources in Zone 4 will use the G.729 codec (BB) (even if those Zone 4 resources are physically located in the same site as Zone 1).

From my way of thinking, all physical TDM equipment should be configured in the zone appropriate to its physical location in the network. Zone 1 VoIP users should share the zone with Zone 1 TDM equipment/users. But, if there is TDM resources at the branch office, they would go into Zone 2 (using the above example).

Virtual Trunking would be a special case, in that Virtual Trunks are not used by IP Phones, so putting them in a separate zone has no negative impact (and the fact that they show up as a zone type in the Intrazone/Interzone print out tells us it was Nortel’s Design Intent that they be put in their own zone.)