From: Jian Gu (guxiaojian@gmail.com)
Date: Tue Feb 12 2008 - 20:43:36 ARST
Hi, all,
Thanks everybody for your help, the problem is still not resolved yet after
trying those suggestions. I did a TCP dump on the server in siteA when
transferring a small file and large file to remote back up server, looks
like when transferring a larger file (64K bytes only), the TCP connection
did not get established, traffic from remote back up server was dropped
somehow.
Jian
### successful transfer of a 18 bytes file from 10.138.45.10 to 10.138.1.41#####
4:04:37.069224 IP 10.138.45.10.58301 > 10.138.1.41.ftp: P 12:20(8) ack 113
win 1460 <nop,nop,timestamp 578782183 3195080185>
14:04:37.129638 IP 10.138.1.41.ftp > 10.138.45.10.58301: P 113:144(31) ack
20 win 5792 <nop,nop,timestamp 3195091695 578782183>
14:04:37.129658 IP 10.138.45.10.58301 > 10.138.1.41.ftp: . ack 144 win 1460
<nop,nop,timestamp 578782244 3195091695>
14:04:37.129792 IP 10.138.45.10.58301 > 10.138.1.41.ftp: P 20:26(6) ack 144
win 1460 <nop,nop,timestamp 578782244 3195091695>
14:04:37.189145 IP 10.138.1.41.ftp > 10.138.45.10.58301: P 144:193(49) ack
26 win 5792 <nop,nop,timestamp 3195091756 578782244>
14:04:37.189269 IP 10.138.45.10.58307 > 10.138.1.41.50102: S
3372932458:3372932458(0) win 5840 <mss 1460,sackOK,timestamp 578782303
0,nop,wscale 2>
14:04:37.229124 IP 10.138.45.10.58301 > 10.138.1.41.ftp: . ack 193 win 1460
<nop,nop,timestamp 578782343 3195091756>
14:04:37.248309 IP 10.138.1.41.50102 > 10.138.45.10.58307: S
1090359299:1090359299(0) ack 3372932459 win 5792 <mss 1380,sackOK,timestamp
3195091817 578782303,nop,wscale 0>
14:04:37.248322 IP 10.138.45.10.58307 > 10.138.1.41.50102: . ack 1 win 1460
<nop,nop,timestamp 578782362 3195091817>
14:04:37.248346 IP 10.138.45.10.58301 > 10.138.1.41.ftp: P 26:37(11) ack 193
win 1460 <nop,nop,timestamp 578782362 3195091756>
14:04:37.307221 IP 10.138.1.41.ftp > 10.138.45.10.58301: P 193:215(22) ack
37 win 5792 <nop,nop,timestamp 3195091877 578782362>
14:04:37.307238 IP 10.138.45.10.58301 > 10.138.1.41.ftp: . ack 215 win 1460
<nop,nop,timestamp 578782421 3195091877>
14:04:37.307380 IP 10.138.45.10.58307 > 10.138.1.41.50102: P 1:16(15) ack 1
win 1460 <nop,nop,timestamp 578782421 3195091817>
14:04:37.307400 IP 10.138.45.10.58307 > 10.138.1.41.50102: F 16:16(0) ack 1
win 1460 <nop,nop,timestamp 578782421 3195091817>
14:04:37.366592 IP 10.138.1.41.50102 > 10.138.45.10.58307: . ack 16 win 5792
<nop,nop,timestamp 3195091939 578782421>
14:04:37.366704 IP 10.138.1.41.ftp > 10.138.45.10.58301: P 215:237(22) ack
37 win 5792 <nop,nop,timestamp 3195091939 578782421>
14:04:37.366717 IP 10.138.45.10.58301 > 10.138.1.41.ftp: . ack 237 win 1460
<nop,nop,timestamp 578782481 3195091939>
14:04:37.366738 IP 10.138.1.41.50102 > 10.138.45.10.58307: F 1:1(0) ack 17
win 5792 <nop,nop,timestamp 3195091939 578782421>
14:04:37.366746 IP 10.138.45.10.58307 > 10.138.1.41.50102: . ack 2 win 1460
<nop,nop,timestamp 578782481 3195091939>
### unsucceful ftp transfer a 64K bytes file from 10.138.45.10 to
10.138.1.41 #####
14:04:43.869301 IP 10.138.45.10.58301 > 10.138.1.41.ftp: P 37:43(6) ack 237
win 1460 <nop,nop,timestamp 578788984 3195091939>
14:04:43.928340 IP 10.138.1.41.ftp > 10.138.45.10.58301: P 237:284(47) ack
43 win 5792 <nop,nop,timestamp 3195098658 578788984>
14:04:43.928363 IP 10.138.45.10.58301 > 10.138.1.41.ftp: . ack 284 win 1460
<nop,nop,timestamp 578789044 3195098658>
14:04:43.928495 IP 10.138.45.10.58308 > 10.138.1.41.7723: S
3370651285:3370651285(0) win 5840 <mss 1460,sackOK,timestamp 578789044
0,nop,wscale 2>
14:04:43.988117 IP 10.138.1.41.7723 > 10.138.45.10.58308: S
1677726584:1677726584(0) ack 3370651286 win 5792 <mss 1380,sackOK,timestamp
3195098718 578789044,nop,wscale 0>
14:04:43.988135 IP 10.138.45.10.58308 > 10.138.1.41.7723: . ack 1 win 1460
<nop,nop,timestamp 578789103 3195098718>
14:04:43.988167 IP 10.138.45.10.58301 > 10.138.1.41.ftp: P 43:54(11) ack 284
win 1460 <nop,nop,timestamp 578789103 3195098658>
14:04:44.047471 IP 10.138.1.41.ftp > 10.138.45.10.58301: P 284:306(22) ack
54 win 5792 <nop,nop,timestamp 3195098780 578789103>
14:04:44.047520 IP 10.138.45.10.58301 > 10.138.1.41.ftp: . ack 306 win 1460
<nop,nop,timestamp 578789163 3195098780>
14:04:44.047605 IP 10.138.45.10.58308 > 10.138.1.41.7723: . 1:1369(1368) ack
1 win 1460 <nop,nop,timestamp 578789163 3195098718>
14:04:44.047610 IP 10.138.45.10.58308 > 10.138.1.41.7723: . 1369:2737(1368)
ack 1 win 1460 <nop,nop,timestamp 578789163 3195098718>
14:04:44.306746 IP 10.138.45.10.58308 > 10.138.1.41.7723: . 1:1369(1368) ack
1 win 1460 <nop,nop,timestamp 578789422 3195098718>
14:04:44.824719 IP 10.138.45.10.58308 > 10.138.1.41.7723: . 1:1369(1368) ack
1 win 1460 <nop,nop,timestamp 578789940 3195098718>
14:04:45.860671 IP 10.138.45.10.58308 > 10.138.1.41.7723: . 1:1369(1368) ack
1 win 1460 <nop,nop,timestamp 578790976 3195098718>
14:04:47.932591 IP 10.138.45.10.58308 > 10.138.1.41.7723: . 1:1369(1368) ack
1 win 1460 <nop,nop,timestamp 578793048 3195098718>
14:04:52.075413 IP 10.138.45.10.58308 > 10.138.1.41.7723: . 1:1369(1368) ack
1 win 1460 <nop,nop,timestamp 578797192 3195098718>
~
On Feb 12, 2008 1:20 PM, Rik Guyler <rik@guyler.net> wrote:
> All too true Steve, especially working in an Enterprise shop where you're
> always understaffed and overworked. I adopted a policy of adjusting MSS
> to
> 1400 and clearing the DF-bit some time ago when PMTUD became totally
> worthless to me for my 50+ VPN remotes. After that, performance may not
> have been optimal but my phone was much less active. ;-)
>
> Rik
>
> -----Original Message-----
> From: nobody@groupstudy.com [mailto:nobody@groupstudy.com] On Behalf Of
> Steve
> Sent: Tuesday, February 12, 2008 3:15 AM
> To: Joseph Brunner
> Cc: Luan Nguyen; Paul Cosgrove; Jian Gu; ccie forum
> Subject: Re: Traffic extremely slow in one direction
>
> The theory is good but more and more I am finding security products are
> blocking ICMP wholesale.
> I have lost count on the number of times I have had to clear DF bit simply
> because end to end hosts have various AV/FW products installed that don't
> allow for path discovery.
> Many system admins have been taught from birth "ICMP is evil......block it
> at all costs..." so even if you identify the issue, you often have a hard
> time convincing them to adjust security policies to allow the ICMP
> through.
>
> Yes - this does cause fragments....is this the right way to run good
> networks - no.......by clearing DF bit, does the throughput increase so
> users are happy? - yes.
>
> Sigh...such is life.
>
>
> Steve
>
>
>
> On 12 Feb 2008, at 05:35, Joseph Brunner wrote:
>
> > Why would you clear the df bit?
> >
> > You DON'T WANT FRAGMENTS!!!! perhaps you should put the ip mtu/tcp mss
> > commands on the LAN FACING interface...
> >
> > IMHO the 2801 is the WORST of the bunch... I tried using it as a large
> > scale dmvpn hub... LOL the 2811 actually held up quite nicely, all the
> > way to
> > 20Mbps+ what killed it was the sheer number of dmvpn tunnels...
> >
> > -Joe
> >
> > -----Original Message-----
> > From: nobody@groupstudy.com [mailto:nobody@groupstudy.com] On Behalf
> > Of Luan Nguyen
> > Sent: Monday, February 11, 2008 9:20 PM
> > To: Paul Cosgrove
> > Cc: Jian Gu; ccie forum
> > Subject: Re: Traffic extremely slow in one direction
> >
> > For me, setting ip mtu, and ip tcp adjust-mss are never enough. I
> > have to clear the df bit as well.
> > access-list 111 permit tcp any any
> > route-map clear-df-bit permit 10
> > match ip address 111
> > set ip df 0
> > int WAN
> > ip policy route-map clear-df-bit
> >
> > On a side note, the 2811 is the worst performer of the 2800 series
> > family of ISR regarding GRE/IPSEC. We had to drop it even with the
> > AIM-VPN/
> > SSL-2
> > card. And don't take performance number for its face value...it's
> > unidirectional and best case scenario. You should take that number
> > divide by 4 to get real world value :)
> >
> > -lmn
> >
> > On Feb 11, 2008 3:30 PM, Paul Cosgrove <paul.cosgrove@heanet.ie>
> > wrote:
> >
> >> Jian Gu wrote:
> >>> hi, all,
> >>>
> >>> I have a real world problem that I am scratching my head to find the
> >> root
> >>> cause. We have a local data center at site A and a remote backup
> >>> server
> >> at
> >>> site B, site A and site B has its own service provider, the
> >>> connection bandwidth is well above 50M in either site, traffic will
> >>> traverse a GRE/IPsec tunnel which is configured between two 2800
> >>> routers, both
> >> routers
> >>> have hardware accelerated IPSec encryption/decrytion module
> >>> installed.
> >>>
> >>> It is interesting that traffic from site A to site B is extremely
> >>> slow, performance is only around 10s packets/second, but FTP
> >>> transferring data from site B to site A is reasonably fast, which is
> >>> around 8Mb/sec.
> >>> I am
> >>> pretty sure service provider of site A is not rate limiting incoming
> >> traffic
> >>> (even it does, noway they will rate limit to 10s packets/sec). What
> >> could be
> >>> the root cause of such poor performance, any server experts here can
> >> give me
> >>> a clue?
> >>>
> >>> Thanks,
> >>> Jian
> >>>
> >>> ____________________________________________________________________
> >>> ___ Subscription information may be found at:
> >>> http://www.groupstudy.com/list/CCIELab.html
> >>>
> >>
> >> Hi Jian Gu,
> >>
> >> Perhaps you do not notice delay or drops with FTP transfer because
> >> you do not see the individual packets being dropped/retransmitted?
> >> With pings this will be more readily apparent. How are you testing
> >> the link?
> >> Which direction does most of the traffic flow in?
> >>
> >> The number of packets per second is very important for encryption,
> >> more so than the throughput in terms of Kbps. Sending lots of small
> >> packets will normally have more of an impact than sending a few large
> >> ones (such as FTP).
> >>
> >> Other than pps, also watch for fragmentation. Check what GRE IP MTU
> >> you are using, as unless your GRE IP MTU allows for the encryption
> >> header being added later your GRE packets will be fragmented.
> >>
> >> GRE adds 24 bytes, IPSEC is variable because of the padding but (I
> >> think) is about 80 bytes.
> >>
> >> Also keep in mind that the router will also have to perform
> >> fragmentation if the incoming IP packet is larger than the GRE IP
> >> MTU, and setting the TCP MSS lower using "ip tcp adjust-mss" can help
> >> with that (but only for TCP). Path MTU discovery on end
> >> hosts/servers is also a good way to reduce fragmentation, and thereby
> >> reduce load on the device.
> >>
> >> According to the folowing link the 2811 is capable of 55 Mbps with
> >> AES or 3DES, but you often have to quite cautious when shown
> >> encryption stats without pps or packet size.
> >>
> >>
> >>
> >
>
> http://www.cisco.com/web/partners/downloads/765/tools/quickreference/vpn_per
> > formance_eng.pdf
> >>
> >>
> >> Paul.
> >> --
> >> Paul Cosgrove
> >> HEAnet Limited, Ireland's Education and Research Network
> >> 1st Floor, 5 George's Dock, IFSC, Dublin 1
> >> Registered in Ireland, no 275301
> >> tel: +353-1-660 9040 fax: +353-1-660 3666
> >> web: http://www.heanet.ie/
> >>
> >> _______________________________________________________________________
> >> Subscription information may be found at:
> >> http://www.groupstudy.com/list/CCIELab.html
> >
> > _______________________________________________________________________
> > Subscription information may be found at:
> > http://www.groupstudy.com/list/CCIELab.html
> >
> > _______________________________________________________________________
> > Subscription information may be found at:
> > http://www.groupstudy.com/list/CCIELab.html
>
> _______________________________________________________________________
> Subscription information may be found at:
> http://www.groupstudy.com/list/CCIELab.html
This archive was generated by hypermail 2.1.4 : Sat Mar 01 2008 - 16:54:48 ARST