From: Church, Chuck (cchurch@wamnetgov.com)
Date: Tue Apr 20 2004 - 09:22:18 GMT-3
So to sum it up (correct me if I'm wrong):
Both hosts on 5500 - no problem
Both hosts on 3550 - no problem
FTP from 5500 host to 3550 host - SLOW
FTP from 3550 host to 5500 host - no problem
No difference trying ISL, dot1q, or no trunk at all between the two
switches
UDP transfer didn't help, actually it was slow in both directions
Swapping GBICs, cables didn't help
I assume the CAM/Mac address tables show the entries correctly - verify
The 5500s could be oversubscribed, but you can prioritize certain ports
with 'set port level'. It's a longshot, but if you do a 'sh system' and
see the peak near %100, it might help if the peak is recent.
On the Unix side, take a look at the protocol statistics. I assume it's
a netstat -'something'. Look for retransmits, out of orders, etc.
Any bugs on the 5500 code that you're running in Bug Navigator? Set the
test variable to 'complete' and reset it. Maybe 'sh test' after it
reloads will show something funky.
Perhaps it's a problem with a gigabit controller on either of the
switches. Try a copper 100 mb link between the switches, if possible.
You mentioned full speed as being 10mbps. Are these 10mb NICs?
Nothing showing up in the switch logs?
I'm kind of running out of ideas....
Chuck Church
Lead Design Engineer
CCIE #8776, MCNE, MCSE
Wam!Net Government Services - Design & Implementation Team
13665 Dulles Technology Dr. Ste 250
Herndon, VA 20171
Office: 703-480-2569
Cell: 703-819-3495
cchurch@wamnetgov.com
PGP key:
http://pgp.mit.edu:11371/pks/lookup?op=index&search=cchurch%40wamnetgov.
com
-----Original Message-----
From: Minihane Kevin-kminih01 [mailto:Kevin.Minihane@motorola.com]
Sent: Monday, April 19, 2004 10:35 PM
To: 'Brian McGahan'; Minihane Kevin-kminih01; Kenneth Wygand; Church,
Chuck; ccielab@groupstudy.com
Subject: RE: Strange trunking issue
Hi Brian
I've actually used a number of different hosts to try and resolve this.
Initially, I thought it was an issue with HostB. However, I've
connected 3 different hosts to various ports on the 3550, and 3 other
hosts to the 5500. The same result every time.
Anything being "pushed" from the 5500 side is slow.
I've tried Chuck's suggestion of using TFTP to put a file across the
network. The speed is about the same in both directions! (Not very
fast!!! about 500k per sec)
I'm going to look at the flow control on both sides of the trunk, as
suggested earlier by Rob Fischer. I've been playing around with the
flow control on the 5500 side, but it hasnt made much of a difference.
-----Original Message-----
From: Brian McGahan [mailto:bmcgahan@internetworkexpert.com]
Sent: 20 April 2004 02:22
To: Minihane Kevin-kminih01; Kenneth Wygand; Church, Chuck;
ccielab@groupstudy.com
Subject: RE: Strange trunking issue
Kevin,
Try changing the MAC addresses of the hosts. I've seen some odd
bugs where certain MAC specific addresses can cause problems depending
on traffic load, so it's not unfathomable that the direction of flow
could cause it as well.
HTH,
Brian McGahan, CCIE #8593
bmcgahan@internetworkexpert.com
Internetwork Expert, Inc.
http://www.InternetworkExpert.com
Toll Free: 877-224-8987 x 705
Outside US: 775-826-4344 x 705
> -----Original Message-----
> From: Minihane Kevin-kminih01 [mailto:Kevin.Minihane@motorola.com]
> Sent: Monday, April 19, 2004 7:44 PM
> To: 'Kenneth Wygand'; Brian McGahan; Church, Chuck
> Subject: RE: Strange trunking issue
>
> Hi Kenneth
>
> Thanks for all the help from everyone.
> When I said I set up a packet sniffer, I SPANned the affected port to
> another port on the 3550, and used a packet analyser to look at the
> traffic passing between the hosts. I saw nothing untoward at all.
I've
> also cleared the counters on the interfaces concerned, and I'm not
seeing
> any CRC errors, dropped packets etc
>
> I'm not performing any QoS at all! I can open a TAC case, which I was
> going to do in the morning, as this is starting to depress me!
>
> It must be nearly quitting time for all you guys over there in the
> US....it's coming up to 1:45 am here.
>
> Going to stick with this another little while, and see if I cant crack
it.
> If not, will have to open a case with TAC
>
> Thanks
>
> Kevin
>
> -----Original Message-----
> From: Kenneth Wygand [mailto:KWygand@customonline.com]
> Sent: 20 April 2004 01:33
> To: Minihane Kevin-kminih01; Brian McGahan; Church, Chuck
> Cc: ccielab@groupstudy.com
> Subject: RE: Strange trunking issue
>
>
> Kevin,
>
> A few days ago you said:
>
> <snip>
> I've used a packet sniffer to check for windowing (or rather,
> > for continuous changes in the windowing) and everything appears
> normal.
> > I'm not seeing excessive traffic, or changes in the window sizes.
> </snip>
>
> Did you SPAN a port to a packet sniffer on another workstation or just
> load a packet sniffer on one of the devices? Try SPANning the
affected
> port (monitor command) on the 3550 to another port on the 3550 and
sniff
> that. Do you see any other types of traffic other than TCP traffic?
Do
> you see CRC errors or retransmissions? How about the interface
> statistics on the affected port of the 3550?
>
> I'm also assuming you aren't doing ANY type of QoS on the equipment...
> do you have any type of WRED configuration that could've gone nuts?
Any
> type of QoS configuration at all that could be bug infested?
>
> Last but not least, do you have a SmartNet contract? Can you open a
TAC
> case? Is it feasible to post your configurations here after masking
> your sensitive IP's, descriptions, hostnames, etc.?
>
> Kenneth E. Wygand
> Systems Engineer, Project Services
> CISSP #37102, CCNP, CCDP, ACSP, Cisco IPT Design Specialist, MCP, CNA,
> Network+, A+
> Custom Computer Specialists, Inc.
> "The only unattainable goal is the one not attempted."
> -Anonymous
>
> -----Original Message-----
> From: nobody@groupstudy.com [mailto:nobody@groupstudy.com] On Behalf
Of
> Minihane Kevin-kminih01
> Sent: Monday, April 19, 2004 8:14 PM
> To: 'Brian McGahan'; MinihaneKevin-kminih01; Church, Chuck
> Cc: ccielab@groupstudy.com
> Subject: RE: Strange trunking issue
>
> Hi Brian
>
> I've put a host onto the 5500, in the same VLAN as hostB. I've also
> placed all the GBIC's into the same access VLAN. The slowness is
still
> there.
>
> I've placed another host onto the 3550, in the same VLAN as hostB, and
> the speed is what you would expect (10Mb per sec).
>
> At times, the speed between the 5500 and the 3550 is down to 300k per
> second. At it's worst, I've seen speeds of 24k per sec! And again,
this
> is only in one direction! It happends when I try to "push" something
> from hostA to hostB. If I try to push from hostB to hostA, then
> everything is ok!!!!!!
>
>
>
> -----Original Message-----
> From: Brian McGahan [mailto:bmcgahan@internetworkexpert.com]
> Sent: 20 April 2004 00:21
> To: Minihane Kevin-kminih01; Church, Chuck
> Cc: ccielab@groupstudy.com
> Subject: RE: Strange trunking issue
>
>
> Kevin,
>
> Try setting both hosts and the GBICs between the switches in the
same
> access VLAN. This will let you know whether or not it is a trunking
> related issue.
>
> HTH,
>
> Brian McGahan, CCIE #8593
> bmcgahan@internetworkexpert.com
>
> Internetwork Expert, Inc.
> http://www.InternetworkExpert.com
> Toll Free: 877-224-8987 x 705
> Outside US: 775-826-4344 x 705
>
>
> > -----Original Message-----
> > From: nobody@groupstudy.com [mailto:nobody@groupstudy.com] On Behalf
> Of
> > Minihane Kevin-kminih01
> > Sent: Monday, April 19, 2004 6:01 PM
> > To: 'Church, Chuck'
> > Cc: 'ccielab@groupstudy.com'
> > Subject: RE: Strange trunking issue
> >
> > I've even tried a different 3550 !....At the moment, I'm running
> version
> > 12.1(19) on both my 3550's. I've tried ftp'ing between two hosts on
> the
> > 3550, and it works perfectly.
> >
> > -----Original Message-----
> > From: Church, Chuck [mailto:cchurch@wamnetgov.com]
> > Sent: 19 April 2004 23:56
> > To: Minihane Kevin-kminih01; Kenneth Wygand; Robert Fischer; MADMAN
> > Cc: ccielab@groupstudy.com
> > Subject: RE: Strange trunking issue
> >
> >
> > Maybe try a different GBIC port and/or copper port on the 3550?
Newer
> > code for the 3550?
> >
> >
> > Chuck Church
> > Lead Design Engineer
> > CCIE #8776, MCNE, MCSE
> > Wam!Net Government Services - Design & Implementation Team
> > 13665 Dulles Technology Dr. Ste 250
> > Herndon, VA 20171
> > Office: 703-480-2569
> > Cell: 703-819-3495
> > cchurch@wamnetgov.com
> > PGP key:
> >
>
http://pgp.mit.edu:11371/pks/lookup?op=index&search=cchurch%40wamnetgov.
> > com
> > -----Original Message-----
> > From: Minihane Kevin-kminih01 [mailto:Kevin.Minihane@motorola.com]
> > Sent: Monday, April 19, 2004 6:38 PM
> > To: 'Kenneth Wygand'; Robert Fischer; Church, Chuck; MADMAN
> > Cc: ccielab@groupstudy.com
> > Subject: RE: Strange trunking issue
> >
> > Well, this is really doing my head in now! I've replaced the fibre,
> > GBIC's on the 3550. Have changed the encapsulation to ISL (from
> dot1q).
> > have reversed the fibres, etc. I've just placed hostB onto the 5500
> > again, and the ftp takes off like a rocket. I'm beginning to
suspect
> > the 3550> Has anyone ever seen "slowness" on a device connected to
a
> > 3550? I'd have thought it would have been ok....
> >
> >
> >
> > -----Original Message-----
> > From: nobody@groupstudy.com [mailto:nobody@groupstudy.com]On Behalf
Of
> > Minihane Kevin-kminih01
> > Sent: 19 April 2004 21:15
> > To: 'Kenneth Wygand'; Robert Fischer; Church, Chuck; MADMAN
> > Cc: ccielab@groupstudy.com
> > Subject: RE: Strange trunking issue
> >
> >
> > Hi
> > Sorry for the delay in this. Was sent away with work. I've tried
> > swapping the strands around, still no joy. I've used a different
> cable,
> > and different GBIC's as well in the 3550. I'm going to take Chuck's
> > advice, and try changing the encapsulation to ISL, just in case
there
> is
> > some sort of bug causing me to go grey prematurely. I'll let you
know
> > how that works out. It's 9 pm here in ireland, so I'll try this
later
> > on tonight and let u all know the result
> >
> > Thanks for all the help so far!
> >
> > Kevin
> >
> > -----Original Message-----
> > From: Kenneth Wygand [mailto:KWygand@customonline.com]
> > Sent: 16 April 2004 12:46
> > To: Minihane Kevin-kminih01; Robert Fischer; Church, Chuck; Minihane
> > Kevin-kminih01; MADMAN
> > Cc: ccielab@groupstudy.com
> > Subject: RE: Strange trunking issue
> >
> >
> > Try swapping the fiber strands on both sides (take the receive port
on
> > the 5500 and swap it with the transmit port on the 5500, then do the
> > same thing on the 3550 side). Then try your transmissions again and
> let
> > us know.
> >
> > Good luck,
> > Ken
> >
> > -----Original Message-----
> > From: nobody@groupstudy.com on behalf of Minihane Kevin-kminih01
> >
> > Sent: Fri 4/16/2004 5:13 AM
> > To: 'Robert Fischer'; 'Church, Chuck'; Minihane Kevin-kminih01;
> > 'MADMAN'
> > Cc: ccielab@groupstudy.com
> > Subject: RE: Strange trunking issue
> >
> >
> >
> > One of the hosts is running Solaris, (hostB), while HostA is a
> > Linux (ReHat) machine. I've also put another Solaris host onto the
> > 5500, with the same effect.
> >
> > I've used a packet sniffer to check for windowing (or rather,
for
> > continuous changes in the windowing) and everything appears
> normal.
> > I'm not seeing excessive traffic, or changes in the window sizes.
> >
> > All hosts are running at 100 Mbs Full duplex. I've played
around
> > with auto settings on each switch, and still have the same result.
> >
> > The mtu on both is 1500, and (using a packet sniffer) I've
checked
> > the packet size of the FTP data. This is 1518, which is correct?
> > (to include the headers etc?)
> >
> > Both hosts are in the same VLAN.
> >
> > This is doing my head in!
> >
> >
> > -----Original Message-----
> > From: Robert Fischer [mailto:rfischer@broadcom.com]
> > Sent: 16 April 2004 01:09
> > To: 'Church, Chuck'; 'Minihane Kevin-kminih01'; 'MADMAN'
> > Cc: ccielab@groupstudy.com
> > Subject: RE: Strange trunking issue
> >
> >
> > What are the hosts running, Windows/ linux? What are the host
> > uplinks,
> > 10/100? The host connected to the 5500 may be overrunning the
> > 5500's port
> > buffer. Capture some packets on the 5500 size and watch for the
> > windowing
> > size to continuously change and the host to retransmit. Newer
> > versions of
> > Windows like XP are defaulting to 64K window sizes and causing
port
> > buffer
> > issues with older gear. Some versions of Red Hat have difficulty
> > negotiating
> > this too.
> >
> > -Robb
> >
> > -----Original Message-----
> > From: nobody@groupstudy.com [mailto:nobody@groupstudy.com] On
> > Behalf Of
> > Church, Chuck
> > Sent: Thursday, April 15, 2004 10:35 AM
> > To: Minihane Kevin-kminih01; MADMAN
> > Cc: ccielab@groupstudy.com
> > Subject: RE: Strange trunking issue
> >
> >
> > So now put the Host B on the 3550, and see if the problem
exists.
> > Any
> > chance it's an MTU issue? Are A and B in the same VLAN? If
not,
> > who's
> > routing, the 3550?
> >
> >
> > Chuck Church
> > Lead Design Engineer
> > CCIE #8776, MCNE, MCSE
> > Wam!Net Government Services - Design & Implementation Team 13665
> > Dulles
> > Technology Dr. Ste 250 Herndon, VA 20171
> > Office: 703-480-2569
> > Cell: 703-819-3495
> > cchurch@wamnetgov.com
> > PGP key:
> >
> >
>
http://pgp.mit.edu:11371/pks/lookup?op=index&search=cchurch%40wamnetgov.
> > com
> > -----Original Message-----
> > From: Minihane Kevin-kminih01
> > [mailto:Kevin.Minihane@motorola.com]
> > Sent: Thursday, April 15, 2004 12:23 PM
> > To: 'MADMAN'; Minihane Kevin-kminih01
> > Cc: ccielab@groupstudy.com
> > Subject: RE: Strange trunking issue
> >
> > Hiya
> >
> > Thanks for the input. I moved hostB onto the 5500, and the
problem
> > disapeared. I put it back onto the 3550, and the same problem
> > exists
> >
> >
> >
> > -----Original Message-----
> > From: MADMAN [mailto:dmadlan@qwest.com]
> > Sent: 15 April 2004 17:04
> > To: Minihane Kevin-kminih01
> > Cc: ccielab@groupstudy.com
> > Subject: Re: Strange trunking issue
> >
> >
> >
> > Your assuming you have a switching problem accross the trunk.
> > Plug host
> > B into the 5500 and see if you still have the problem.
> >
> > Dave
> >
> > Minihane Kevin-kminih01 wrote:
> >
> > > Hi
> > >
> > > Apologies in advance if this seems long winded, but I wanted
to
> > attach
> > the configurations, and output of various commands.
> > >
> > > I'm experiencing a very strange trunking issue, and was
wondering
> > if
> > anyone could perhaps have some input which might help?
> > >
> > >
> > > The following is my layout.
> > >
> > >
> > > Host A ------- Cat5500 ------------------------------ Cat3550
> > > ------------ Host B
> > >
> > > Each host is connected to it's switch using a 100Mb full
duplex
> > > connection
> > >
> > > I have two switches (cat 5500 running CatOS, and a 3550). I'm
> > trunking between both using dot1q encapsulation. Vtp
information
> > is flowing
> > freely between them, and I can configure vlans on both,
propagate
> > the vlans
> > between each switch, etc. The trunk between the switches is a
> > gigabit fibre
> > connection.
> > >
> > > The problem I am having is when I try to ftp from host A to
host
> > B,
> > and vice versa.
> > >
> > > If I ftp from Host A to Host B, I experience very slow speeds
> > > (sometimes as low as 24k per second). If I come from Host B
to
> > host
> > > A, the speed increases to 8Mbs per second! I've checked the
> > usual
> > > (speed, duplex, check for CRC, dropped packets) and I get
nothing
> > >
> > > The following is the config of the interface g0/1 on the 3550
> > >
> > > Lab_3550_1#show run int g0/1
> > > Building configuration...
> > >
> > > Current configuration : 169 bytes
> > > !
> > > interface GigabitEthernet0/1
> > > description Trunk to main lab 5500 port 1/1 switchport trunk
> > > encapsulation dot1q switchport mode dynamic desirable no ip
> > address
> > > end
> > >
> > > This is the output when I type the "show int g0/1" command on
the
> > 3550
> > >
> > > Lab_3550_1#show int g0/1
> > > GigabitEthernet0/1 is up, line protocol is up (connected)
> > > Hardware is Gigabit Ethernet, address is 000d.bc49.44b1 (bia
> > 000d.bc49.44b1)
> > > Description: Trunk to main lab 5500 port 1/1
> > > MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
> > > reliability 255/255, txload 1/255, rxload 1/255
> > > Encapsulation ARPA, loopback not set
> > > Keepalive set (10 sec)
> > > Full-duplex, 1000Mb/s, link type is auto, media type is SX
> > > output flow-control is off, input flow-control is off
> > > ARP type: ARPA, ARP Timeout 04:00:00
> > > Last input 00:00:00, output 00:00:00, output hang never
> > > Last clearing of "show interface" counters 01:52:54
> > > Input queue: 0/75/0/0 (size/max/drops/flushes); Total output
> > drops:
> > 0
> > > Queueing strategy: fifo
> > > Output queue: 0/40 (size/max)
> > > 5 minute input rate 1962000 bits/sec, 471 packets/sec
> > > 5 minute output rate 1707000 bits/sec, 428 packets/sec
> > > 2818750 packets input, 1023156797 bytes, 0 no buffer
> > > Received 293096 broadcasts (0 multicast)
> > > 0 runts, 0 giants, 0 throttles
> > > 1 input errors, 1 CRC, 0 frame, 0 overrun, 0 ignored
> > > 0 watchdog, 224737 multicast, 0 pause input
> > > 0 input packets with dribble condition detected
> > > 2635991 packets output, 1397912368 bytes, 0 underruns
> > > 0 output errors, 0 collisions, 0 interface resets
> > > 0 babbles, 0 late collision, 0 deferred
> > > 0 lost carrier, 0 no carrier, 0 PAUSE output
> > > 0 output buffer failures, 0 output buffers swapped out
> > >
> > > And finally, on the 3550, I "show int trunk"
> > >
> > > Lab_3550_1#show int trunk
> > >
> > > Port Mode Encapsulation Status Native
> > vlan
> > > Fa0/45 desirable isl trunking 44
> > > Fa0/46 desirable isl trunking 44
> > > Fa0/47 on isl trunking 41
> > > Gi0/1 desirable 802.1q trunking 1
> > > Gi0/2 desirable 802.1q trunking 41
> > >
> > > Port Vlans allowed on trunk
> > > Fa0/45 1-4094
> > > Fa0/46 1-4094
> > > Fa0/47 1-4094
> > > Gi0/1 1-4094
> > > Gi0/2 1-4094
> > >
> > > Port Vlans allowed and active in management domain
> > > Fa0/45
> >
> >
>
1,10,12,20,25,28,30-33,35-41,44-45,48,52,54-55,60,130,341,351-352,361,36
> > 9,400-406,419,995
> > > Fa0/46
> >
> >
>
1,10,12,20,25,28,30-33,35-41,44-45,48,52,54-55,60,130,341,351-352,361,36
> > 9,400-406,419,995
> > > Fa0/47
> >
> >
>
1,10,12,20,25,28,30-33,35-41,44-45,48,52,54-55,60,130,341,351-352,361,36
> > 9,400-406,419,995
> > > Gi0/1
> >
> >
>
1,10,12,20,25,28,30-33,35-41,44-45,48,52,54-55,60,130,341,351-352,361,36
> > 9,400-406,419,995
> > > Gi0/2
> >
> >
>
1,10,12,20,25,28,30-33,35-41,44-45,48,52,54-55,60,130,341,351-352,361,36
> > 9,400-406,419,995
> > >
> > > Port Vlans in spanning tree forwarding state and not
> > pruned
> > > Fa0/45 1,41,44,351,361,400
> > > Fa0/46 1
> > > Fa0/47
> >
> >
>
1,10,12,20,25,28,30-33,35-41,44-45,48,52,54-55,60,130,341,351-352,361,36
> > 9,400-406,419,995
> > > Gi0/1
> >
> >
>
1,10,12,20,25,28,30-33,35-41,44-45,48,52,54-55,60,130,341,351-352,361,36
> > 9,400-406,419,995
> > > Gi0/2 1,41,403-406
> > >
> > >
> > >
> > > On my 5500 I do the following:
> > >
> > > 5500-OMCLAB (enable) show port capabilities 1/1
> > > Model WS-X5530
> > > Port 1/1
> > > Type 1000BaseSX
> > > Speed 1000
> > > Duplex full
> > > Trunk encap type 802.1Q,ISL
> > > Trunk mode on,off,desirable,auto,nonegotiate
> > > Channel no
> > > Broadcast suppression percentage(0-100)
> > > Flow control
> > receive-(off,on,desired),send-(off,on,desired)
> > > Security no
> > > Membership static
> > > Fast start yes
> > > QOS n/a
> > > Rewrite no
> > > UDLD Capable
> > >
> > >
> > >
> > >
> > > 5500-OMCLAB (enable) show port 1/1
> > > Port Name Status Vlan Level Duplex
> > Speed
> > Type
> > > ----- ------------------ ---------- ---------- ------ ------
> > -----
> > ------------
> > > 1/1 Trunk to lab_3550_ connected trunk normal full
> > 1000
> > 1000BaseSX
> > >
> > > Port Trap IfIndex
> > > ----- -------- -------
> > > 1/1 disabled 224
> > >
> > > Port Broadcast-Limit Broadcast-Drop
> > > -------- --------------- --------------
> > > 1/1 - 0
> > >
> > > Port Send FlowControl Receive FlowControl RxPause
> > TxPause
> > Unsupported
> > > admin oper admin oper
> > opcodes
> > > ----- -------- -------- -------- -------- -------
> > -------
> > -----------
> > > 1/1 desired off off off 0 0
> > 0
> > >
> > >
> > > Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize
> > > ----- ---------- ---------- ---------- ---------- ---------
> > > 1/1 0 0 0 0 0
> > >
> > > Port Single-Col Multi-Coll Late-Coll Excess-Col Carri-Sen
Runts
> > Giants
> > > ----- ---------- ---------- ---------- ---------- ---------
> > ---------
> > ---------
> > > 1/1 0 0 0 0 0
> > 0
> > -
> > >
> > > Last-Time-Cleared
> > > --------------------------
> > > Thu Apr 15 2004, 10:04:41
> > >
> > >
> > > 5500-OMCLAB (enable) show trunk
> > > Port Mode Encapsulation Status Native
> > vlan
> > > -------- ----------- ------------- ------------
> > -----------
> > > 1/1 on dot1q trunking 1
> > > 1/2 on dot1q trunking 1
> > >
> > >
> >
This archive was generated by hypermail 2.1.4 : Mon May 03 2004 - 19:48:50 GMT-3