Re: OT: Nexus deployment with 802.3ad LACP team

Hello all,

I wanted to give an update on this and put the rumors to rest. We got
in touch with TAC and apparently the latest version of the IOS
(4.1(3)N2(1a)) has a bug in the LACP timeout negotiation. The bug ID
and symptoms are as follows:

 CSCta60232 Bug Details

Add configurable short timeout to Nexus 5000

Symptom: The Nexus 5000 does not currently have support to request
short timeout.

NOTE: The Nexus 5000 does move to a short timeout when the partner
timeout is short. This is because the partner dictates the timeout for
the actor. The N5k cannot get the partner to reciprocate due to it not
being able to be configured for short timeout.

Workaround: The channel should still form, but each side is using
different timeouts. Whichever timeout is lost first will trigger the
channel to re-initialization.
It is recommended to use long-timers on both sides until 4.2(1)N1(1)

Further Problem Description:

To identify the partner timeout:

show lacp interface eth X/Y
Neighbor: 0/0
MAC Address= 0-0-0-0-0-0
System Identifier=0x0,0-0-0-0-0-0
Port Identifier=0x0,0x0
Operational key=0
LACP_Activity=unknown
LACP_Timeout=short Timeout (1s)<<<
Synchronization=NOT_IN_SYNC
Collecting=false
Distributing=false

Status Fixed (Resolved)

Severity 3 - moderate

Last Modified In Last month

Product Cisco Nexus 5000 Series Switches

Technology

1st Found-In 4.1(3)N1(0.164)

Fixed-In Release-Pending

Component(s) lacp

Hope this helps anyone else that is having the same problem.

Usama.

On Mon, Mar 1, 2010 at 9:28 PM, <istong_at_stong.org> wrote:
> That's an understatement but it's worth a little risk to
> enjoy the cutting edge aspects :)
> Also recently found out the 5K's don't support VTP and the
> 7k's only support VTP transparent as of 4.2.3
>
>
> Ian Stong
> www.CCIE4u.com
> Rack Rentals and discounted Lab Scenarios
>
>
>> I tell you, this whole Nexus / UCS line is so unbaked and
>> there are very many questions more than answers. I
>> understand this is new technology - but I've seen more
>> issues and marketing vs actual engineering supported
>> scenarios and numbers than I would like to see for this.
>> Lots of GREAT things are happening within DC from Cisco
>> but there is sooo much SMOKE - more than I've seen with
>> any other product line thus far. And I love Cisco's
>> products.
>>
>>
>>
>> > From: istong_at_stong.org
>> > To: adrianlazar_at_gmail.com; rmur_at_ipexpert.com
>> > CC: pbhatkoti_at_gmail.com; chaudri_at_gmail.com;
>> > ccielab_at_groupstudy.com Subject: RE: OT: Nexus deployment
>> > with 802.3ad LACP team Date: Sun, 28 Feb 2010 20:55:13
>> -0500 >
>> > For what it's worth the 2K's are fabric extenders and
>> > not full blown switches. You can't connect a switch to
>> > them and they don't do vPC. If you want redundancy then
>> > one option is to dual connect your server to two
>> different 2K's (with each 2K going to a different 5K) and
>> then configure the
>> > server for NIC teaming.
>> >
>> > Alternatively you can single home your server to one 2K
>> that connects to two
>> > different 5K's . Then configure the 5K's with a vPC.
>> Note: The 5K currently
>> > has a 12 hardware etherchannel limit (but much higher
>> > software etherchannel limit).
>> >
>> > Yet another option is to connect your servers to two
>> > different 5K's (versus 2K's) and configure the 5K with a
>> > vPC so it looks like one switch to the downstream
>> server. Switch or blade chassis switch. I like the 3120X
>> > 10Gig switches for uplinks to the 5K for performance as
>> well as having the ability
>> > to stack two 3120X switches together as one.
>> >
>> > There are a few other options as well but the above
>> should give you an idea.
>> >
>> >
>> > Thanks,
>> >
>> > Ian Stong
>> > www.CCIE4u.com
>> > Rack Rentals and discounted Lab Scenarios
>> >
>> >
>> >
>> >
>> > -----Original Message-----
>> > From: nobody_at_groupstudy.com
>> > [mailto:nobody_at_groupstudy.com] On Behalf Of Lazar Adrian
>> > Sent: Sunday, February 28, 2010 10:56 AM
>> > To: Rick Mur
>> > Cc: Radioactive Frog; Usama Pervaiz; Cisco certification
>> > Subject: Re: OT: Nexus deployment with 802.3ad LACP team
>> >
>> > Hi,
>> >
>> > The N2K is like a linecard for the Nexus 5k so it should
>> > support anything that the N5K supports (or most of it).
>> > vPC is a Nexus 5K feature and I am pretty sure that can
>> be used through the 2K (when the server is connected to
>> > the 2K via LACP port-channel).
>> > In most of the materials from Cisco regarding vPC is
>> > described the exact topology that Usama wants to use so
>> if it's not working this is really a big
>> > problem for Cisco :). One of the materials is here:
>> >
>>
> http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/configurat
>> > ion_guide_c07-543563.html
>> > Coincidentally, I am working on a similar setup using
>> > N5Ks and N2ks so I will check if this is actually
>> working or not. >
>> > Regards,
>> >
>> > Adrian
>> >
>> > On Sun, Feb 28, 2010 at 4:30 PM, Rick Mur
>> <rmur_at_ipexpert.com> wrote: >
>> > > I indeed can't imagine that the N2k support the vPC
>> > feature yet. You can do
>> > > teaming on the server though. Teaming on servers is
>> > > usually just using 1 interface and the second as a
>> > > backup. The server really has to support balancing,
>> > like ESX (not by default as well). >
>> > > Then again. AFAIK it's not possible to configure a vPC
>> > on 2 N2k's. >
>> > >
>> > > --
>> > > Regards,
>> > >
>> > > Rick Mur
>> > > CCIE2 #21946 (R&S / Service Provider)
>> > > Sr. Support Engineer IPexpert, Inc.
>> > > URL: http://www.IPexpert.com
>> > >
>> > > On 28 feb 2010, at 13:11, Radioactive Frog wrote:
>> > >
>> > > > thanks, let us know how did you go.
>> > > >
>> > > > On Sun, Feb 28, 2010 at 5:47 AM, Usama Pervaiz
>> > > <chaudri_at_gmail.com> wrote:
>> > > >
>> > > >> We are using the management vrf for the keepalives.
>> > > otherwise its the >> default vrf. We didnt want to get
>> > > too fancy with the config as we dont >> need it.
>> > > Another member on the list replied and said that the
>> 2K's >> currently do not support this yet. I will have to
>> > > call Cisco and get >> more info.
>> > > >>
>> > > >> Thank you all for your help!!
>> > > >>
>> > > >> Usama
>> > > >>
>> > > >> On Sat, Feb 27, 2010 at 7:33 AM, Radioactive Frog
>> <pbhatkoti_at_gmail.com>
>> > > >> wrote:
>> > > >>> Hi,
>> > > >>>
>> > > >>> The config looks okay. I'd add this, just to make
>> > > sure the right vrf (if
>> > > >>> you're using any).
>> > > >>>
>> > > >>> vpc domain 1
>> > > >>> peer-keepalive destination 10.0.0.1 source
>> > 10.10.0.2 vrf vpc-keepalive
>> > > >>>
>> > > >>> Are you issueing show vpc status command from base
>> > system i.e. mgmt vdc
>> > > >> or
>> > > >>> one of the vdc which has peering with another one
>> > > on a different box? >>>
>> > > >>>
>> > > >>> use switchto vdc <vdcName> command and then check
>> > > the vpc status. >>>
>> > > >>>
>> > > >>>
>> > > >>>
>> > > >>> On Sat, Feb 27, 2010 at 4:27 PM, Usama Pervaiz
>> > > <chaudri_at_gmail.com> >> wrote:
>> > > >>>>
>> > > >>>> Following is how i configured the vPC:
>> > > >>>>
>> > > >>>> 5K-A
>> > > >>>>
>> > > >>>> vpc domain 1
>> > > >>>> peer-keepalive destination 10.0.0.1
>> > > >>>>
>> > > >>>> interface port-channel1
>> > > >>>> switchport mode trunk
>> > > >>>> vpc peer-link
>> > > >>>> spanning-tree port type network
>> > > >>>> speed 10000
>> > > >>>>
>> > > >>>> interface port-channel10
>> > > >>>> switchport access vlan 100
>> > > >>>> vpc 10
>> > > >>>> speed 1000
>> > > >>>>
>> > > >>>> interface Ethernet1/17
>> > > >>>> switchport mode trunk
>> > > >>>> channel-group 1 mode active
>> > > >>>>
>> > > >>>> interface Ethernet1/18
>> > > >>>> switchport mode trunk
>> > > >>>> channel-group 1 mode active
>> > > >>>>
>> > > >>>> int e100/1/10
>> > > >>>> switchport access vlan 100
>> > > >>>> spanning-tree port type edge
>> > > >>>> channel-group 10 mode active
>> > > >>>>
>> > > >>>>
>> > > >>>>
>> > > >>>>
>> > > >>>> This is the config on 5K-B:
>> > > >>>>
>> > > >>>> vpc domain 1
>> > > >>>> peer-keepalive destination 10.0.0.2
>> > > >>>>
>> > > >>>> interface port-channel1
>> > > >>>> switchport mode trunk
>> > > >>>> vpc peer-link
>> > > >>>> spanning-tree port type network
>> > > >>>>
>> > > >>>> interface port-channel10
>> > > >>>> switchport access vlan 100
>> > > >>>> vpc 10
>> > > >>>> speed 1000
>> > > >>>>
>> > > >>>> interface Ethernet1/17
>> > > >>>> switchport mode trunk
>> > > >>>> channel-group 1 mode active
>> > > >>>>
>> > > >>>>
>> > > >>>> interface Ethernet1/18
>> > > >>>> switchport mode trunk
>> > > >>>> channel-group 1 mode active
>> > > >>>>
>> > > >>>> interface Ethernet100/1/10
>> > > >>>> switchport access vlan 100
>> > > >>>> spanning-tree port type edge
>> > > >>>> channel-group 10 mode active
>> > > >>>>
>> > > >>>>
>> > > >>>>
>> > > >>>>
>> > > >>>> sh vpc brief
>> > > >>>> Legend:
>> > > >>>> (*) - local vPC is down, forwarding
>> > > via vPC peer-link >>>>
>> > > >>>> vPC domain id : 1
>> > > >>>> Peer status : peer adjacency
>> > > formed ok >>>> vPC keep-alive status : peer
>> > > is alive >>>> Configuration consistency status:
>> > > success >>>> vPC role :
>> > > secondary >>>>
>> > > >>>> vPC Peer-link status
>> > > >>>>
>> ----------------------------------------------------------
>> > > ----------- >>>> id Port Status Active vlans
>> > > >>>> -- ---- ------
>> --------------------------------------------------
>> > > >>>> 1 Po1 up 1,14-15,100
>> > > >>>>
>> > > >>>> vPC status
>> > > >>>>
>> > > >>>>
>> > > >>
>> > >
>> >
>> ----------------------------------------------------------
>> > > ------------------ >>>> id Port Status
>> > > Consistency Reason Active
>> > > >>>> vlans
>> > > >>>> ------ ----------- ------ -----------
>> > > -------------------------- >>>> -----------
>> > > >>>> 100 Po100 up success success
>> > > 100 >>>> 114 Po114 down*
>> > > failed Consistency Check Not - >>>>
>> > > Performed >>>>
>> > > >>>>
>> > > >>>> I dont understand why the status says down and
>> > > the consistency says >>>> failed. Please let me know
>> > > if i configured something wrong or if i >>>> need
>> > > addition steps. >>>>
>> > > >>>> Thanks,
>> > > >>>> Usama
>> > > >>>>
>> > > >>>>
>> > > >>>> On Fri, Feb 26, 2010 at 10:47 PM, Radioactive
>> > > Frog < pbhatkoti_at_gmail.com
>> > > >>>
>> > > >>>> wrote:
>> > > >>>>> My question is do i even need to configure
>> > > anything if the servers are
>> > > >>>>> doing NIC teaming?
>> > > >>>>>
>> ----------------------------------------------------------
>> > > ------ >>>>> ah...
>> > > >>>>> the answer is simply "yes", you need to
>> > > configure 802.13ad on your >>>>> switch as
>> > > >>>>> well where those 2 teamed NIC's are connected.
>> > > If this is not done >> then
>> > > >>>>> the
>> > > >>>>> incoming traffic to server will not load
>> > > balanced. >>>>>
>> > > >>>>> Server to Switch traffic flow:
>> > > >>>>> --------------------------------------
>> > > >>>>> Outgoing from server = Teaming servers ports, if
>> > servers are Linux it
>> > > >>>>> creates a "BOND0" interface which is logically
>> > > interface created by >> the
>> > > >>>>> OS
>> > > >>>>> when teaming is enabled. The OS sends any
>> > traffic originated from (in
>> > > >>>>> virtulization environment VM's) OS through the
>> > BOND0 interface out to
>> > > >> a
>> > > >>>>> switch.
>> > > >>>>>
>> > > >>>>> same concept applies fomr incoming traffic.
>> > > >>>>>
>> > > >>>>> HTH
>> > > >>>>>
>> > > >>>>>
>> > > >>>>>
>> > > >>>>>
>> > > >>>>> On Sat, Feb 27, 2010 at 2:24 PM, Usama Pervaiz
>> > > <chaudri_at_gmail.com> >>>>> wrote:
>> > > >>>>>>
>> > > >>>>>> The uplink vPC to the 6509's is working. The
>> > > VSS consists of 2 6509's
>> > > >>>>>> which are multi-chassis portchannel to the 2
>> > > 5K's (which are using >>>>>> vPC) My concern is at the
>> > > host level. If i configure the port-channel
>> > > >>>>>> to the servers using LACP the port channel says
>> > > its down and the port
>> > > >>>>>> itself is in Individual mode (I). If i
>> > > configure it with just this >>>>>> command (as stated
>> > > in the quick start guide for vPC) >>>>>>
>> > > >>>>>> channel-group 10
>> > > >>>>>>
>> > > >>>>>> The port-channel shows up but I do not see any
>> > > increase in throughput
>> > > >>>>>> from the server.
>> > > >>>>>>
>> > > >>>>>> My question is do i even need to configure
>> > > anything if the servers >> are
>> > > >>>>>> doing NIC teaming?
>> > > >>>>>>
>> > > >>>>>> Thanks for your response.
>> > > >>>>>>
>> > > >>>>>> Usama
>> > > >>>>>>
>> > > >>>>>> On Fri, Feb 26, 2010 at 10:11 PM, Radioactive
>> > > Frog >>>>>> <pbhatkoti_at_gmail.com>
>> > > >>>>>> wrote:
>> > > >>>>>>> vPC is not supported within the BOX (even it
>> > > has 2 vdc/vss). You >> must
>> > > >>>>>>> have 2
>> > > >>>>>>> physical boxes i.e. 2x6509.
>> > > >>>>>>>
>> > > >>>>>>>
>> > > >>>>>>> On Sat, Feb 27, 2010 at 11:08 AM, Usama
>> Pervaiz <chaudri_at_gmail.com
>> > > >>>
>> > > >>>>>>> wrote:
>> > > >>>>>>>>
>> > > >>>>>>>> Hello all,
>> > > >>>>>>>>
>> > > >>>>>>>> We are rolling out a deployment of Nexus 5k's
>> > > and 2k's with VSS. >>>>>>>> following is the design
>> > > that we have chosen. The >>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>>>
>> > > ___________ >>>>>>>>
>> > > N2K1------>N5K1 --------------- | 6509 |
>> >>>>>>>> Server< || -----vPC---- |
>> > > VSS | >>>>>>>> LACP N2K2------>N5K2
>> > > -------------- |___________| >>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>>> The N5K1 and N5K2 are using vPC to connect to
>> > > the 6509's. This >> link
>> > > >>>>>>>> is
>> > > >>>>>>>> up and functioning properly.
>> > > >>>>>>>> My question is if we are using active/active
>> LACP from the server
>> > > >>>>>>>> side
>> > > >>>>>>>> NIC's do we have to configure port-channel on
>> > the N2K1 and N2K2? I
>> > > >>>>>>>> have tried it with and without and in both
>> > > cases my maximum >>>>>>>> throughput
>> > > >>>>>>>> is just below 1Gbps. I would have though that
>> > teaming up the NIC's
>> > > >>>>>>>> on
>> > > >>>>>>>> the server would give me at least something
>> > > over a gig of >> throughput
>> > > >>>>>>>> or am I wrong in my assumption.
>> > > >>>>>>>>
>> > > >>>>>>>> Also when I specify the mode for the
>> > > channel-group as active >>>>>>>> (channel-group 10
>> > > mode active) my port-channel status shows as >> down
>> > > >>>>>>>> and my port status is (I). Is LACP not
>> > supported on host ports for
>> > > >>>>>>>> the
>> > > >>>>>>>> N2K's?
>> > > >>>>>>>>
>> > > >>>>>>>> If I have confused anyone then I apologize!!
>> > > >>>>>>>>
>> > > >>>>>>>> Thanks in advance!
>> > > >>>>>>>> Usama
>> > > >>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>>> Blogs and organic groups at
>> > > http://www.ccie.net >>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>>>
>> > > >>
>> __________________________________________________________
>> > > _____________ >>>>>>>> Subscription information may be
>> > > found at: >>>>>>>>
>> > > http://www.groupstudy.com/list/CCIELab.html >
>> > > >
>> > > > Blogs and organic groups at http://www.ccie.net
>> > > >
>> > > >
>> __________________________________________________________
>> > > > _____________ Subscription information may be found
>> > > > at: http://www.groupstudy.com/list/CCIELab.html
>> > >
>> > >
>> > > Blogs and organic groups at http://www.ccie.net
>> > >
>> > >
>> __________________________________________________________
>> > > _____________ Subscription information may be found
>> > > at: http://www.groupstudy.com/list/CCIELab.html
>> >
>> >
>> > Blogs and organic groups at http://www.ccie.net
>> >
>> >
>> __________________________________________________________
>> > _____________ Subscription information may be found at:
>> > http://www.groupstudy.com/list/CCIELab.html
>> >
>> >
>> > Blogs and organic groups at http://www.ccie.net
>> >
>> >
>> __________________________________________________________
>> > _____________ Subscription information may be found at:
>> > http://www.groupstudy.com/list/CCIELab.html
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>>
>> __________________________________________________________
>> _______ Your E-mail and More On-the-Go. Get Windows Live
>> Hotmail Free.
>> http://clk.atdmt.com/GBL/go/201469229/direct/01/
>>
>>
>> Blogs and organic groups at http://www.ccie.net
>>
>> __________________________________________________________
>> _____________ Subscription information may be found at:
>> http://www.groupstudy.com/list/CCIELab.html
>>
>>
>>
>>
>>
>>
>>
>
>
>
>
> Bigger. Stronger. Faster.
> Sign up for a New Free Email Account at http://MyEmail.com
>
>
> Blogs and organic groups at http://www.ccie.net
>
> _______________________________________________________________________
> Subscription information may be found at:
> http://www.groupstudy.com/list/CCIELab.html

Blogs and organic groups at http://www.ccie.net
Received on Tue Mar 02 2010 - 19:02:25 ART