Any PC storage gurus?

[Diluted]

I currently run 2 servers, 1 ESXi box and the other running Ubuntu Server acting as a NAS. I'm using the space on the NAS to host my Virtual Machines/Storage. Currently the NAS is RAID-5 using 4 640GB SATA drives running at 1.5GB/s. EDIT: The space from the NAS is served up via NFS, not iSCSI

The ESXi and NAS boxes are connected @ 1Gbps through a Cisco 3550 switch.

I was wondering if breaking the array and making it a RAID-10 would yield any performance benefits? It takes forever to copy files and create new VM's currently. I know that RAID-5 generally sucks with small writes.

The ESXi box is currently running 4 VM's and the specs of that box are...
Intel Core2Duo E6600
8GB DDR2-800 RAM
74GB WD Raptor to host the ESXi OS

Unfortunately with the RAID card isn't the greatest so I can't run it in the ESXi box with all the storage, otherwise the whole box will freeze. I believe its because it doesn't have enough memory (64MB) to support everything...using it in the NAS provides no issues which is strange. EDIT: Do you think putting the storage back in the ESXi box with RAID-10 would make a difference to having it RAID-5?

Any assistance with improving the performance would be greatly appreciated.

 

[bmw675]

:stupidpc:

D, all of the above???

 

[Diluted]

:stupidpc:

D, all of the above???

lol :iconbeer:

 

[dart1963]

is it mission critical enough that you can't just stripe with some decent backups? Even stripe it locally with some synch sw between it and a nas (which may slow it down just as much as a raid 5)? just tossing out a few ideas:iconbeer:

 

[Diluted]

is it mission critical enough that you can't just stripe with some decent backups? Even stripe it locally with some synch sw between it and a nas (which may slow it down just as much as a raid 5)? just tossing out a few ideas:iconbeer:

It's really nothing critical...it's just my home setup. I run an AD, DHCP and File Server. The problem with local storage is that ESXi apparently doesn't like my RAID Card (WTB Dell PERC5i lol), or it didn't when I had it in there as RAID-5. It would freeze the whole box. ESXi also only recognizes 2 of my 4 SATA ports on the motherboard. I don't really need all the space so RAID-10 wouldn't kill me. I only have about 100GB worth of stuff, that'd still leave me 900GB for VM's and other stuff.

 

[Diluted]

Just had a thought. What about setting the drives up in a RAID-10, then finding a way to connect that box via eSATA to the ESXi host box? Would that box need an OS? I don't know that it's possible, but I'd imagine it's been done before.

 

[dart1963]

Just had a thought. What about setting the drives up in a RAID-10, then finding a way to connect that box via eSATA to the ESXi host box? Would that box need an OS? I don't know that it's possible, but I'd imagine it's been done before.

See, you're already talking above me... I had to google ESXi to have somewhat an idea WTF that was....

I've shy'd away from VM setups... IT is my part time for work... and I really don't have time to figure it out (it sounds like the way to go, I just really don't have time to learn it), I thought about playing with it at home, but I have my priorities... kids/wife first, bike/biker friends/biker forums second/anything&everything else third/ work, at the very last, least, minuscule thought of my off time!

edit: oh, and I threw my home pc together... I could go into details... but just a quick, core i7 930, Asus P6ST deluxe MB, and everything else fairly weak/generic, 3G corsair mem, 1G 9500GT vid, 2 640 WD cav black HD... and everything else.

 

[Detrich]

when u say slow, what type of files & size/ throughput figures are u getting? and,
what RPM are your SATA drives? if they are fast drives & on a robust NAS, then im not sure that changing from a RAID5 to a RAID10 will yield earth-shattering gains. have you looked at the resource utilization on your ESXi host box and optimized that? (swap files, # of hosts etc.) and, is your host adequately sized?

we use a SAN (via iSCSI) to store our ESX VMS, and generally don't have any issue. but, we have 5 hosts running on really high-end dell servers server up 50-60 hosts of all flavors (linux, windows etc.) and don't seem to have any problems.

 

[Diluted]

when u say slow, what type of files & size/ throughput figures are u getting? and,
what RPM are your SATA drives? if they are fast drives & on a robust NAS, then im not sure that changing from a RAID5 to a RAID10 will yield earth-shattering gains. have you looked at the resource utilization on your ESXi host box and optimized that? (swap files, # of hosts etc.) and, is your host adequately sized?

we use a SAN (via iSCSI) to store our ESX VMS, and generally don't have any issue. but, we have 5 hosts running on really high-end dell servers server up 50-60 hosts of all flavors (linux, windows etc.) and don't seem to have any problems.

Throughput isn't great...I figure I copied about 100gb in more than 5 hours...They are 7200RPM and 2 320GB Platters per disk. We use FC storage at work with ESX VM's on higher end dells as well and don't see any issues.

Host is only a Dual Core Proc, with 8GB of RAM. You figure the Domain Controller and DHCP server just sit there, only the file server would be getting accessed regularly. I tried to install Windows Home Server to test something for my mom and it took well over and hour and a half just to format the space...

 

[Diluted]

Here's an HDPARM from the NAS Device on the RAID-5 Array...
/dev/sdb:
Timing buffered disk reads: 68 MB in 3.03 seconds = 22.45 MB/sec

Here's the HDPARM from the Flash Memory that runs the OS...
/dev/sda:
Timing buffered disk reads: 158 MB in 3.01 seconds = 52.50 MB/sec

 

[Detrich]

my coworker recently copied a disk image on our network from a nas to the san, and 136gb took about 4 hours... surprisingly. so, maybe your home speeds are not quite as bad as previously thought?

hmm... just as a test, did you try copying files from a workstation or notebook over the network directly to the nas? that might shed some light where the bottleneck is- since it removes the esx host out of the picture. a 100mb file should copy in about/ under 10 seconds. then, try larger files, like maybe a 2 or 3gb disk image, and that should only copy in a matter of minutes. if these copies go fast, then maybe it's the esx host slowing things down? but, if they also copy slow, then it could be either the network or the nas...

i have a cheap, consumer-grade nas device at home running on a 100baset link speeds. and, it copies kinda slow too. 100gb would for sure take a few hours. but, i'm also using smb protocol which isn't the fastest. but, copy speeds also depend on the number of files and the size of files. if it's a gazllion small files, then it always copies significantly slower, whereas if it's a smaller number of large files it copies noticeably faster.

 

[Diluted]

Here's the result of a 167MB ISO file transfered over SFTP to the NAS

Status: File transfer successful, transferred 167,276,544 bytes in 103 seconds

So almost 2 minutes, this is over 100BaseT to a 1Gbps connection.

 

[toddjcruz]

Don't bother with changing the array, that won't be your problem. You major limitation here is your network.

As you are running TCP/IP (easy guess) you are only going to get around 25-35% utilization out of your network. Granted, RAID10 would improve your disk performance and if your RAID card does not support enhanced RAID5 opertations (battery backed write cache usually required for this), the RAID10 will solve some of that, but not the overall performance issue.

1st thing I would do is enable jumbo frames on your switch and both boxes (ESXi and your NAS). This will allow you to improve your network performance considerably. I would also switch to iSCSI rather than NFS if you can, as block level commands will yield better performance than NFS will. Although I don't think ESXi support iScsi, that may be a licensed feature.

Regardless, you can verify this by monitoring your switch performance while you move data around. A read from the NAS is your best way to test performance, as RAID5 reads will be fast and you take the write overhead of RAID5 out of the picture.

 

[Detrich]

hmm, maybe i'm remembering wrong then. i just copied a 120mb file in 27 sec from a 100baset to 1000baset via windows explorer at work. for some reason i thought it it should be faster than that- ie 1/2 the time it actually took...

but, i think when u have mixed duplex settings, it always negotiates at the lower one. so, 100 to 1000 inevitably transfers at 100... maybe everything will go faster if u peg the duplex settings to all match at whatever max u can get? ie 100full to 100full? and, see if that helps? my stuff at home always runs slower than work tho, for obvious reasons.

 

[99vengeur]

Nothing useful to add here....

It's all gibberish to me!

 

[Diluted]

Don't bother with changing the array, that won't be your problem. You major limitation here is your network.

As you are running TCP/IP (easy guess) you are only going to get around 25-35% utilization out of your network. Granted, RAID10 would improve your disk performance and if your RAID card does not support enhanced RAID5 opertations (battery backed write cache usually required for this), the RAID10 will solve some of that, but not the overall performance issue.

1st thing I would do is enable jumbo frames on your switch and both boxes (ESXi and your NAS). This will allow you to improve your network performance considerably. I would also switch to iSCSI rather than NFS if you can, as block level commands will yield better performance than NFS will. Although I don't think ESXi support iScsi, that may be a licensed feature.

Regardless, you can verify this by monitoring your switch performance while you move data around. A read from the NAS is your best way to test performance, as RAID5 reads will be fast and you take the write overhead of RAID5 out of the picture.

Bingo on the TCP/IP Network...

My Cisco 3550 switch doesn't support jumbo frames...I don't think it was meant to be used like I am, which is having two servers plugged into the two fiber ports. Those are generally left to tie back into a core...anyway...iSCSI is supported by ESXi, I've tried it and the performance is pretty much the exact same. I think the RAID controller may be the bottle neck. I have a pretty high end network setup here, here's the components...

Cable Modem->Netgear FVS338 Router->Cisco 3550 Switch->Cisco 2620 Router (for Router on a Stick)

This allows me to run different VLAN's so I can have a separate network for public devices, my servers, my other devices and I am using ACL's to keep the public stuff from touching everything else. All that said, I'll go hook my laptop up to the 3550 and send a file to my desktop downstairs and report back with the speed.

EDIT: Transfered the same file as above, 167MB from my desktop to my laptop in ~15 seconds. Gonna go with the RAID controller, I may swap it over to 10 and see what happens. May give iSCSI another go.

EDIT 2: Here is an updated HDPARM on the Array with -t and -T...
chris@nas:~$ sudo hdparm -t /dev/sdb

/dev/sdb:
Timing buffered disk reads: 68 MB in 3.03 seconds = 22.44 MB/sec
chris@nas:~$ sudo hdparm -T /dev/sdb

/dev/sdb:
Timing cached reads: 920 MB in 2.00 seconds = 459.41 MB/sec

 

[Detrich]

cool. yah i think comparing copy speeds helps. 167mb in 15 seconds is within the range of what you'd expect to see on a fast 100baset network i think...

 

[Diluted]

cool. yah i think comparing copy speeds helps. 167mb in 15 seconds is within the range of what you'd expect to see on a fast 100baset network i think...

I agree. I'm going to set the port for the ESXi host to 100 Full and transfer that file to the local storage on that box and see how that goes.

EDIT: Well can't set the speed/duplex on a trunk port. And I can only set the NIC as 1000/Full in ESXi. That same 167MB file took around 2 minutes to transfer from my desktop to the ESXi local storage.

 

[Diluted]

I'll be seeing if I can snag a copper NIC from work that will work with ESXi so I can test and see if its my Fiber NIC's causing problems.

 

[toddjcruz]

If your NAS and ESXi are connected to fiber ports (1000) on the 3550, you can't set duplex, it's always full on a fiber port.

3550, yeah, I see 2000 is the largest you can set, that won't buy you much in TCP overhead.

without that, you just won't get significant network throughput on your network.

I always go direct attached or SAN for best performance on my disk sub system. Worst case you dig up an old PCI SCSI RAID card and grab an external enclosure to add to your server.

Or, find a 3750 or some other GIG switch that will support jumbo frames so that you can increase your network throughput.