Ian Dudley

Hi all, I'm new to the forum having just got an IP Office for my company a week ago. Unfortunately we've been having a lot of very serious problem, to the point where our call volume has dropped from a thousand calls a day to around 300. Avaya seem stumped and say none of these problems have ever happened before. I was wondering if anyone here recognised any of the problems or had any ideas about what may be the cause.

First up our setup.

IP Office 412 with 2x 20 channel VCM (see bellow) and 8 port Analogue module.
Initialy V2.1.15, then .24 currently on .27
Phone Manager iPro V2.1.8 x 40 users in agent mode with queuing.
2 x Hardphone in meeting rooms (hardly used).
XP SP1 with Plantronics DSP-100 headsets

Note we are only using Phone Manager for the users, no hardphones at all except for general meeting rooms where there are no PC's.

The worst problem was calls being unexpectedly dropped, or cross connected to another user. e.g. User 1 is talking to a client. User 2 dials a number (can be internal, external or even dialing voicemail) User 1 suddenly finds themselves talking to the destination of user 2s call, often whlie it is stil ringing. User 2 is cut off and so is the client user 1 was originaly talking to. The call dropping typicaly happens after 8 seconds of the call, the cross connection could happen at any time.

After a process of elimination, including 4 different VCM modules and 2 different 412s, we worked out that it happens when there is a VCM fitted in a particular slot on the 412. So if you have one card in that slot, or a card in both slots, the problems happen. If you have one card in the other slot it's ok. Not a hardware fault, or if it is it's affected two seperate brand new 412s. We are curently running with just a single VCM module fitted.

The other problem is with Phone Manager. User is on an outbound call, again it can be to any destination internal, external, voicemail, whatever, and suddenly the ringing wav is played in their ear. It is loud enough that it drowns out the call. The person on the other end of the call hears it as well. The ringing will continue until the end of the call, there is no way to stop it. When the call is hung up, a missed call is registered from the number the user was talking to. There is nothing registered on the switch. The only workaround we've found for this is to untick 'Play Sounds', but of course that disables genuine rings as well, so far from ideal. Unticking play sounds doesnt work when already in a call, so once the ringing has started you're stuck with it. This problem happens no matter what VCMs are fitted where.

In an attempt to fix these we've completely wiped and re-typed the system, upgraded the version to first .24, then the just released .27, and all the hardware (bar the analogue module) has been replaced at least once.

The combination of the above has made the system nearly unusuable, and while we have workarounds in place now which minimise the impact we're a long way from a properly working system. The bit that mystefies me is how such serious issues got out to us as end users without being spotted during testing. As I said at the top, Avaya say we are unique in having these problems, but I can't see how as our setup isn't particularly odd and the hardware has been replaced to (hopefully) eliminate manufacturing faults.

40 users, decent network with QoS switches, 30 channel ISDN to the teleco. It's an ordinary SME configuration, the only unusual bit is we are using Phone Manager as our primary phone system, rather than hardphones, but I can't see how that would cause switching issues on the IPO.

Any comments, thoughts, musing or advice welcome!

Cephas

Wow. Quite the buffet table of challenges you've got there!

Personally I've never run into anything remotely like the problems you are having. Then again, I would never attempt to deploy a system with this configuration. Why did you decide to go all softphones for inhouse agents, instead of 4406D digital sets enhanced with PhoneManager pro? You would have avoided all these issues for a very similiar price. Personally I'm of the mindset that voip phones are for remote or homeworkers, and digital phones are for inhouse.

When you say a fairly decent network - the more specific you can be, the more some of our resident networking gurus here on the forum can offer their input, which I suspect we will need for this one!

Which VCM slot in your 412 was the problem one?

Peter

PBXtech.info

 

Ian Dudley

Originaly we planned to roll out softphones with wireless networking, so as to avoid structured cabling altogether in the office. i.e each desk would only require a power point and all comms to the desk would go over the wireless LAN. This fell through as Avaya won't warranty QoS over wireless yet (well they will but only with one AP for every three users) , but we stuck with the softphone as it was cheaper and we were looking for close integration with our other IT systems. When asked dirertly and specificaly by myself, Avaya said the softphone was a full commercial grade alternative to a traditional hardphone. In rtropspect it seems this may have been somewhat of an exageration.

The network is two brand new 24 port HP 2626 procurve switches, bought to spec to go with the system. We have enabled QoS on the XP machines. the two are linked with a gigbit connection and all voicetraffic is contained within those two switches. There is a third switch, which is an old D-Link unit and not Layer 3 compliant, but I only use that for network devices like print servers, which don't require any voice traffic. Everything runs at 100Mb full duplex, except for a couple of the servers (fileshare and database) which are on gigabit.

With just 35-40 odd PC's on the LAN, spread over the two switches, the traffic is quiet and there is plenty of headroom. Leaving the port monitor running on the switches for a day or two, the utilization on the port the 412 is connected to has never risen above 1 percent, and none of the client PC's go above 35 percent, average is too low to register. I'm fairly confident the network is up to the job, appart from anything else there is no way I can imagine an IP layer issue could cause calls to be cross connected to another user, as that is only relevent at the application layer.

Not sure which slot is which, but the one we are using now (the good one) is hard up against the front panel and to the right, so it's the other one that's dodgy. Note that the exact same problem was found on a replacement 412, suggesting an inherant firmware or software issue rather than a fault. Obvisouly there are lots of 412's out there with two VCM modules though, so possibly something to do with using softphones. It's all rather baffling really.

big70

I don,t think someone can help you, because anyone who placed many IPO,s whil not try this kind of configuration.
Is there anybody out there who hase a configuration more than 40 IP-Stations and have no problems.
I only use them for home-workers and some remote locations but in smale ammount.
And really not for agents in a call-center.

Greet Peter

Ian Dudley

I'm getting increasingly concerned at the general opinion here that IP telephony is not a business grade alternative to traditional structured cabling and DT phones.

We went to Avaya to put in a greenfield site purely on the basis of IP telephony, and as such only put in enough structured cabling for datacomms to each desk. i.e. one Cat-5e per desk rather than the traditional two. This saved us a lot of money in structured cabling, on top of which we saved money on not having to buy hardware phones.

This was obviously a fairly radical departure in terms of technology from the way things used to done. As background we used to use an old Index system with DT3 handsets, the IPOffice was installed when we moved to a new office that had no in place infrastructure. Essentialy the IPOffice was paid for by the savings we made on the infrastructure, we probably wouldn't have bought a new phone system otherwise.

At every stage from beginging to end I made both Avaya and the reseller aware of what we did, what we wanted to do and how we wanted to do it. At every stage they were absolutely firm that the IP Office and Phone Manager combo was a fully featured and resiliant alternative to traditional digital telephony. If it isn't then fair enough, a product can only do what it can do. But if it isnt and Avaya sold it to me saying it was then we'll be after a full refund and compensation for selling us a system which is not fit for the purpose it was sold for.

cg111

I would agree with Cephas by not going to a total VoIP solution. I have seen many vendors try this with out success. The only solution that I have seen this type of design work properly is the Avaya S8xxx platform. The S8300 would work great fro 40 IP stations. Your PCs, regardless of whoever's softphone, need to have plenty of memory and ponies for softphones to work well. Also, digital stations should be placed is key locations in the event of a network failure (i.e. receptionists, CEOs, etc.)

Before doing anything, you may want to have your network sniffed to ensure that QOS is indeed functioning properly. I have seen instances, especially with Dell PCs, that QOS is enabled but does NOT work. We were shocked to find this was occuringin a very high profile installation.

__________________
ACE.....

PBXtech.info

 

Ian Dudley

If all goes well we'll have Avaya tier 2 engineers in this afternoon to take a look themselves. It seems they put up an identical setup to ours in their lab and got weird problems. Unfortunately they apparently have different weird problems to us.

WRT to the Pc's, we have a range from a few old P3 800's up to brand new P4 2.8Ghz. All have plenty of memory (min 256Mb), including my own dev PC which has 1Gb RAM. The problems have been just as prevalent on my PC as on the oldest slowest machines, there is no correlation at all between PC spec and the issues. They have also happened out of hours when my PC is the only one on the network. I'm keeping an eye on the QoS stuff, but I really don't think it relevent to what we're seeing. The issue isn't the quality of the call, which is fine, but rather the call being misrouted or unexpected things happening like the ringing over a call. All speak to me of an application issue, i.e. bugs, rather than network traffic. My background is in coding and IT systrems (I'm an IT Manager, not a telecomms expert) but I do know how networks operate and I find it hard to beleive QoS could cause this. It would be like llimited network bandwidth causing someone else to get my e-mail, it simply doesn't work that way.

Again, a serious conversation is due with Avaya. They sold me a purely IP based system on the basis they had plenty of existing installations like this and it was tried and tested technology. This was clearly them being economical with the truth. That is completely unacceptable for something as crucial as a companies phone system.

cg111

Definately give Avaya a chance to figure out what's causing the problem. Just make sure they use their escalation procedures. It's not uncommon for them to go as high aas tier 4 to get a fix. Avaya has a good reputation and if something's broke they'll work hard to make it right.

For a quick fix, I would ask them what version of code they are running at these "working locations" and install it on your 412 so that you can run normally for the time being. Then you can have piece of mind and Avaya can have time to find the bug(s).

__________________
ACE.....

brainey

I am a Business Partner and I have experienced the same problem. I had a customer on a Magix who wants to go with the same configuration. I let him know I was against this type of setup but I bought a USB headset to test with. I am the only user , at my office, using Phone Manager IP Pro and have the same problems. I opened a ticket with Avaya and it went up to tier 4. I done everything they asked me to do and I still have the same problem. During a call, I will get a ringing and the only way to get rid of it is to HANG UP! Avaya closed the ticket and I am still putting up with it. I agree with the person above, go with digital sets in the same building and maybe have remote users using voice over IP.

__________________
brainey
www.primelineusa.com

Ian Dudley

Would you be willing to provide me your ticket reference so I can pass it on to Avaya? They are insisting that we are the only customer to have had this problem and it's something unique to our setup. If I can show that it is happening on other places it will hopefully indiciate that it is a fault or bug with the system, and may also provide them with a broad enough spread of example data to work out what's causing it.

Sadly it's too late for us to go to digital telephony. I'd need to run a new set of structured cabling through the office, which would be extremely expensive now the fit out is complete and the users in place. The original cabling was put in while the office was still empty, with no walls or furniture.

brainey

I do not have the ticket number. It has been several months ago since this took place and I can't remember the tech's name either. Sorry!

__________________
brainey
www.primelineusa.com

big70

We have many side,s with S8300 and S8700 and a lot of IP-Phone,s and Softphone,s.
It works witout big problem,s.
You can use your same IP-Phones.
Try to change the platform and give the ipo 412 back.
It will cost more but a lot of more pleasure.
The IP-Office is really great if you stay on the right track.

corhavelaar

I have seen quite some sites only with IP hardphones and IPO412 without problems, so it is quite possible to use it as sold to you. I expect it has to do with some configuration problems and i believe it is not a network problem.
I'm quite confident Avaya will find a solution for you, they are really helpfull and experienced.

cg111

I concur with Big70. If Avaya don't find the problem ask your rep. how much would it cost to take you to an S8300 with IP Softphones. The cost shouldn't be too great. The S8xxx platform works perfectly.

__________________
ACE.....

sjered

Just wanted to jump briefly into the discussion:

I have a customer that is running a pure IP configuration on an IP Office 412 unit with about 50 hardphones. So far, everything is running great. They've had a few issues with some echo coming over the phones, but with a few tweaks and adjustments, we've been able to fix this. They are currently running QoS on a new Cisco 6400 series switch. They are running some instances of IPhoneManger Pro (softphone) and it is working fine for them.

Nothing more to add to the discussion as far as possible solution. Just wanted to alert you to the fact that it is possible to run a pure IP solution on IP412 and make it work. Just a thought: maybe you coudl try using IP hardphones (the 4602's are very affordable) and see how it works?