Bonjour,
 
J'ai une VM linux et j'ai effectué deux tests et je ne comprends pas les résultats. J'utilise le logiciel iozone pour les tests.
J'ai deux datastores sur mon ESXi : un sur un agrégat FC et l'autre sur un agrégat SATA
 
aggrFC => raid-dp, 13 disk 15K 600G
 
aggrSATA => raid-dp, 13 disks 7200k 500G
 
Sur aggrFC :
Le premier test est cette commande. Il y a 10 tests pour pouvoir avoir une moyenne.
 
for i in 1 2 3 4 5 6 7 8 9 10; do h=`date +%H%M`; echo "##### $h: $i #####"; /root/iozone/iozone -I DIRECT I/O -R -l 1 -u 1 -r 4k -s 8m -O -i0 -i1 -i2 -i8 -+p50 -F /production/test1 > ${h}-r4k_s8m_worker1_directio_p50 2>&1; sleep 60; done
 
Une fois terminé, j'ai migré la VM de datastore via vmotion dans l'agrégat aggrSATA
 
J'ai lancé la même commande (donc environ 20 min plus tard).
 
Une fois terminé, voici les résultats :
 
# grep "Initial write" aggrFC/*
 
aggrFC/1605-r4k_s8m_worker1_directio_p50:"  Initial write "    2841.82  
 
aggrFC/1606-r4k_s8m_worker1_directio_p50:"  Initial write "    2269.83  
 
aggrFC/1608-r4k_s8m_worker1_directio_p50:"  Initial write "    1795.11  
 
aggrFC/1609-r4k_s8m_worker1_directio_p50:"  Initial write "    3023.86  
 
aggrFC/1611-r4k_s8m_worker1_directio_p50:"  Initial write "    2688.28  
 
aggrFC/1612-r4k_s8m_worker1_directio_p50:"  Initial write "    3032.19  
 
aggrFC/1614-r4k_s8m_worker1_directio_p50:"  Initial write "    2530.15  
 
aggrFC/1615-r4k_s8m_worker1_directio_p50:"  Initial write "    2917.37  
 
aggrFC/1617-r4k_s8m_worker1_directio_p50:"  Initial write "    3505.62  
 
aggrFC/1618-r4k_s8m_worker1_directio_p50:"  Initial write "    1506.58  
 
 
[root@F6M001T01ASUproduction]
 
# grep "Initial write" aggrSATA/*
 
aggrSATA/1750-r4k_s8m_worker1_directio_p50:"  Initial write "    2623.89  
 
aggrSATA/1751-r4k_s8m_worker1_directio_p50:"  Initial write "    2908.46  
 
aggrSATA/1753-r4k_s8m_worker1_directio_p50:"  Initial write "    2897.90  
 
aggrSATA/1754-r4k_s8m_worker1_directio_p50:"  Initial write "    3335.04  
 
aggrSATA/1756-r4k_s8m_worker1_directio_p50:"  Initial write "    3028.79  
 
aggrSATA/1758-r4k_s8m_worker1_directio_p50:"  Initial write "    2607.58  
 
aggrSATA/1759-r4k_s8m_worker1_directio_p50:"  Initial write "    3386.53  
 
aggrSATA/1801-r4k_s8m_worker1_directio_p50:"  Initial write "    3356.94  
 
aggrSATA/1802-r4k_s8m_worker1_directio_p50:"  Initial write "    2592.27  
 
aggrSATA/1803-r4k_s8m_worker1_directio_p50:"  Initial write "    3151.19
 
 
 
 
# grep "Read" aggrFC/*
 
aggrFC/1605-r4k_s8m_worker1_directio_p50:"           Read "    4133.05  
 
aggrFC/1606-r4k_s8m_worker1_directio_p50:"           Read "    4083.46  
 
aggrFC/1608-r4k_s8m_worker1_directio_p50:"           Read "    4054.55  
 
aggrFC/1609-r4k_s8m_worker1_directio_p50:"           Read "    4479.86  
 
aggrFC/1611-r4k_s8m_worker1_directio_p50:"           Read "    3723.38  
 
aggrFC/1612-r4k_s8m_worker1_directio_p50:"           Read "    4026.06  
 
aggrFC/1614-r4k_s8m_worker1_directio_p50:"           Read "    4301.60  
 
aggrFC/1615-r4k_s8m_worker1_directio_p50:"           Read "    4109.03  
 
aggrFC/1617-r4k_s8m_worker1_directio_p50:"           Read "    2690.19  
 
aggrFC/1618-r4k_s8m_worker1_directio_p50:"           Read "    2378.36  
 
 
[root@F6M001T01ASUproduction]
 
# grep "Read" aggrSATA/*
 
aggrSATA/1750-r4k_s8m_worker1_directio_p50:"           Read "    4799.14  
 
aggrSATA/1751-r4k_s8m_worker1_directio_p50:"           Read "    1690.01  
 
aggrSATA/1753-r4k_s8m_worker1_directio_p50:"           Read "    2063.44  
 
aggrSATA/1754-r4k_s8m_worker1_directio_p50:"           Read "    1066.23  
 
aggrSATA/1756-r4k_s8m_worker1_directio_p50:"           Read "    4807.54  
 
aggrSATA/1758-r4k_s8m_worker1_directio_p50:"           Read "    4471.69  
 
aggrSATA/1759-r4k_s8m_worker1_directio_p50:"           Read "    4286.77  
 
aggrSATA/1801-r4k_s8m_worker1_directio_p50:"           Read "    4510.91  
 
aggrSATA/1802-r4k_s8m_worker1_directio_p50:"           Read "    4253.37  
 
aggrSATA/1803-r4k_s8m_worker1_directio_p50:"           Read "    4481.57  
 
 
Mon agrégat SATA obtient de meilleur résultat que mon agrégat FC.
 
Avez-vous des idées de comment cela est possible ?
Si il vous faut plus de précision sur la configuration faite le moi savoir car là je comprends vraiment pas.
 
Merci.
Cordialement,

Quel est le mode RAID de chaque stockage ? 10? 5? 1? 0?

Ah oui pardon.
 
Le modèle de baie est un VFiler 3240 Netapp.
Donc les agrégats sont en raid-dp.

y a de l'activité sur les agregats pendant tes tests ?  
lance un sysstat -x  1 pour voir ou ca peut pecher ...

Merci pour ton retour.
Il y a 6 vm sur l'agrégat FC et 13 sur l'agrégat SATA. Les datastores sont montées en NFS et il n'y a pas d'autres utilisation à part ça.

Je me demandais si il y n'y avait pas une limitation possible de configurée (je sais pas quel paramètre par contre) par exemple au niveau nfs/réseau/cache/etc...
Est-il possible que l'agrégat SATA soit privilégié au niveau ESX ou bien qu'il est du cache que pour lui ? ou autre chose ?

En fait les perf de l'agrégat FC ne sont pas mauvaise en soient mais c'est surtout celle du SATA qui sont trop hautes je pense pour moi.

Si je fais un calcul basique même si il n'est pas forcément bon mais au moins ça donne une idée (peut-être que je me trompe dans ce cas merci de me corriger) :
disk FC = 180iops => 13 disks = 2340iops
disk SATA = 75iops =>  13 disks = 975iops
(il y a aussi la notion de raid que ne prend pas en compte mais bon c'est pas grave)

J'ai tout de même essayé une analyse sur un point. J'ai fais une moyenne sur 11 secondes de sysstat soit le temps du test iozone.
- le nombre d'opérations NFS est plus important sur le SATA que sur le FC (on voit par contre une différence quand le test n'est pas lancé (à froid))
- beaucoup plus de trafic réseau sur le SATA que sur le FC. En sortie (out) on est à 10 fois plus
- il y a plus d'écriture lors du test SATA que sur le FC

Je vais pas afficher la totalité du sysstat mais voici les principales colonnes :
A froid :
NFS => 329
Net in kB/s => 3895
Net out kB/s => 295
Disk read kB/s => 2724
Disk write kB/s => 6707

Agrégat FC :
NFS => 790
Net in kB/s => 4423
Net out kB/s => 1418
Disk read kB/s => 5031
Disk write kB/s => 7583

Agrégat SATA :
NFS => 1069
Net in kB/s => 5449
Net out kB/s => 15426
Disk read kB/s => 5338
Disk write kB/s => 8762

Voici le résultat de iozone # /root/iozone/iozone -I DIRECT I/O -R -l 1 -u 1 -r 4k -s 8m -O -i0 -i1 -i2 -i8 -+p50 -F /production/test1 :
Pour FC :
"  Initial write "    3173.77

"        Rewrite "    2664.25

"           Read "    4421.38

"        Re-read "    4258.16

"    Random read "    4118.86

" Mixed workload "    3109.89

"   Random write "    2699.23

Pour SATA :
"  Initial write "    2580.83

"        Rewrite "    3216.17

"           Read "    4174.34

"        Re-read "    4404.61

"    Random read "    3556.04

" Mixed workload "    3344.87

"   Random write "    2948.47

Avez-vous des idées ?

Voici le détail de la commande sysstat -x 1 (j'ai enlevé les colonnes inutiles) :
"a froid","CPU","NFS","Total","Net","kB/s","Disk","kB/s","Cache","CP","Disk","FCP","FCP","kB/s"
,,,,"in","out","read","write","hit","time","util",,"in","out"
,"11%",766,1053,20573,701,3032,41352,"99%","100%","15%",55,721,1233
,"8%",274,309,2346,188,692,5232,"100%","26%","17%",27,137,707
,"1%",133,166,828,104,380,0,"99%","0%","5%",33,276,838
,"2%",191,235,2074,244,460,24,"100%","0%","5%",44,282,1119
,"10%",192,209,694,82,244,0,"99%","0%","7%",17,140,421
,"3%",350,408,1875,685,628,0,"99%","0%","9%",50,414,1249
,"2%",202,202,1858,212,268,32,"99%","0%","2%",0,0,0
,"8%",463,530,3013,261,1684,0,"99%","0%","8%",67,548,1672
,"2%",221,283,1891,134,528,0,"99%","0%","6%",62,417,1601
,"3%",249,265,2614,288,840,24,"99%","0%","9%",16,138,419
,"13%",434,456,6362,406,800,0,"100%","0%","7%",17,134,415
,"17%",297,298,2120,206,25380,51420,"100%","72%","43%",0,0,0
,"3%",277,335,2598,361,4720,20472,"98%","55%","20%",58,542,1383
,"12%",309,378,1977,358,1044,0,"100%","0%","6%",69,567,1733
,"5%",554,604,7158,461,940,0,"99%","0%","12%",50,421,1248
,"6%",666,705,9240,489,1376,24,"99%","0%","21%",34,277,831
,"5%",348,371,2352,174,1464,8,"99%","0%","6%",23,267,549
,"2%",223,257,849,182,424,0,"99%","0%","7%",34,272,835
,"4%",329,370,2934,216,980,24,"100%","0%","5%",41,407,967
,"6%",454,"506",5870,327,492,0,"96%","0%","6%",52,424,1254
,"4%",343,407,7520,290,944,0,"100%","0%","7%",57,547,1382
,"4%",431,441,8948,501,748,32,"100%","0%","17%",10,0,286
,"20%",269,296,5020,276,24088,42340,"100%","42%","30%",27,137,705
,"6%",261,291,1223,361,3804,33464,"100%","88%","35%",30,215,707
,"3%",458,475,4091,406,1620,24,"99%","0%","10%",17,141,414
,"4%",181,220,636,122,156,0,"99%","0%","6%",34,282,836
,"2%",156,189,1159,157,280,8,"99%","0%","7%",33,272,834
,"2%",244,278,1851,213,352,24,"99%","0%","4%",34,280,834
,"5%",293,327,3308,172,628,0,"99%","0%","5%",34,275,837
,,"329,9310344828","374,6206896552","3895,9310344828","295,7586206897",2724,"6707,0344827586","#DIV/0!","#DIV/0!","#DIV/0!","35,3448275862","294,2413793103","872,724137931"
,,,,,,,,,,,,,
,,,,,,,,,,,,,
"FC","CPU","NFS","Total","Net","kB/s","Disk","kB/s","Cache","CP","Disk","FCP","FCP","kB/s"
,,,,"in","out","read","write","hit","time","util",,"in","out"
,"13%",1429,1429,7294,521,5124,24,"99%","0%","7%",0,0,0
,"14%",1007,1054,6105,553,3664,0,"99%","0%","6%",47,133,1270
,"3%","356",408,2434,298,912,0,"100%","0%","5%",52,408,1317
,"34%",2198,2239,10208,740,26012,33912,"100%","52%","32%",34,274,837
,"10%",468,494,4404,1268,3892,23480,"99%","75%","22%",24,139,420
,"5%",651,685,4415,471,3540,0,"99%","0%","20%",32,343,550
,"9%",2300,2347,2421,8983,3824,24,"100%","0%","6%",47,346,1118
,"5%",252,293,1092,192,424,0,"99%","0%","9%",41,418,974
,"10%",2264,2329,2633,10185,1672,0,"99%","0%","3%",57,581,1390
,"3%",493,546,2528,584,1220,32,"100%","0%","5%",53,274,1407
,"6%",269,336,4326,279,700,0,"99%","0%","5%",67,553,1661
,"2%",283,317,1681,1462,1284,0,"98%","0%","7%",34,272,835
,"2%",249,381,1817,219,344,24,"100%","0%","4%",54,494,1250
,"11%",2246,2257,2999,8959,304,0,"100%","0%","4%",0,0,0
,"16%",213,230,5168,228,19495,11224,"100%","23%","32%",17,133,413
,"18%",334,358,1596,324,12635,42595,"100%","100%","40%",24,271,547
,"10%",236,293,1826,240,1004,13608,"100%","38%","17%",57,543,1379
,"13%",1632,1830,7281,560,256,0,"100%","0%","4%",37,136,1055
,"9%",951,1046,9520,773,1800,24,"99%","0%","16%",86,825,1941
,"10%",178,188,1336,108,460,8,"98%","0%","10%",7,143,132
,"4%",402,402,6932,347,2448,520,"99%","0%","10%",0,0,0
,"3%",344,398,4478,436,1784,24,"100%","0%","7%",54,279,1405
,"18%",2595,2645,15847,949,2024,0,"100%","0%","11%",50,411,1254
,"2%",214,275,1752,112,280,0,"100%","0%","4%",51,415,1251
,"4%",463,463,10102,448,1744,32,"98%","0%","20%",0,0,0
,"7%",215,249,1811,225,448,0,"99%","0%","6%",34,283,838
,"23%",282,345,3048,928,30860,31484,"100%","62%","35%",63,410,1537
,"5%",220,237,1311,581,14224,38276,"93%","100%","24%",17,139,420
,"10%",188,193,1902,156,3524,24628,"100%","78%","16%",0,0,0
,,"790,7586206897","836,7931034483",4423,"1418,2413793104","5031,1034482759","7583,4137931035","#DIV/0!","#DIV/0!","#DIV/0!","35,8275862069","283,5517241379",869
,,,,,,,,,,,,,
,,,,,,,,,,,,,
"SATA","CPU","NFS","Total","Net","kB/s","Disk","kB/s","Cache","CP","Disk","FCP","FCP","kB/s"
,,,,"in","out","read","write","hit","time","util",,"in","out"
,"4%",360,404,1146,15955,1248,0,"96%","0%","3%",44,277,1128
,"21%",2341,2424,11825,17108,8684,24,"98%","0%","10%",83,1206,1759
,"14%",834,861,2902,18667,9231,3245,"98%","12%","14%",27,139,691
,"10%",432,449,2376,7966,11668,48368,"99%","100%","49%",17,136,417
,"20%",2518,2556,11217,12410,3272,18808,"99%","47%","24%",33,277,831
,"9%",1053,1113,5645,25985,3356,0,"99%","0%","18%",60,414,1543
,"7%",995,1028,6272,8843,1163,0,"98%","0%","16%",33,277,830
,"10%",1965,1999,2061,17939,3277,24,"99%","0%","6%",34,276,838
,"6%",667,684,1895,23717,748,8,"98%","0%","8%",17,155,419
,"11%",2603,2659,4172,19332,776,0,"99%","0%","6%",51,412,1256
,"13%",449,480,929,21477,1215,22,"97%","0%","4%",31,255,768
,"4%",539,582,5282,5300,1328,0,"98%","0%","8%",37,356,832
,"4%",461,501,4704,8315,992,0,"98%","0%","9%",34,269,832
,"16%",1291,1307,21430,18926,3152,32,"96%","0%","15%",16,137,417
,"26%",1906,1926,2166,18482,23332,41416,"99%","60%","56%",14,280,264
,"7%",617,661,1625,20909,10480,40628,"99%","100%","39%",44,268,1118
,"13%",773,806,6917,25404,13328,19876,"97%","65%","25%",33,277,836
,"9%",562,620,5202,10606,2000,0,"98%","0%","6%",58,755,1226
,"19%",2563,2590,12121,21700,2964,8,"99%","0%","5%",27,135,703
,"10%",501,528,2174,15558,2700,24,"99%","0%","5%",20,0,567
,"2%",260,309,1551,326,276,0,"99%","0%","4%",49,406,1248
,"4%",576,603,3583,15713,2812,0,"99%","0%","5%",27,136,704
,"8%",375,413,1823,10459,1348,24,"99%","0%","12%",38,277,971
,"18%",2477,2506,11192,16228,1984,8,"99%","0%","4%",29,277,699
,"4%",538,594,3217,15691,2008,0,"99%","0%","5%",51,420,1247
,"10%",752,819,3407,20756,2972,24,"99%","0%","6%",67,538,1662
,"21%",1220,1269,7289,8769,19904,18648,"99%","29%","27%",49,417,1244
,"13%",1103,1145,11835,17833,15724,62900,"98%","77%","42%",42,416,1035
,"8%",290,323,2070,6987,2872,32,"98%","0%","5%",33,275,837
,,"1069,6896551724","1108,9310344828","5449,2413793104","15426,2413793103","5338,4137931035","8762,724137931","#DIV/0!","#DIV/0!","#DIV/0!","37,8620689655","326,3103448276","928,3448275862"

Merci pour votre aide.
Cordialement,

Bonjour,
 
Sauf erreur de ma part, mais il me semble qu'il y a une confusion entre la bande passante et les iops.
 
les disques fc et sata ont les memes caracteristique en bande passantes mais ce qui les différencie ce sont les IO par seconde.
 
effectivement :
disk FC = 180iops => 13 disks = 2340iops
disk SATA = 75iops =>  13 disks = 975iops
 
ce nombre d'io par seconde est en lien direct avec la taille du segment ecrit (4k je crois sur ta commande).
 
ce qui serait interressant c'est d'avoir ce nombre d'io/ seconde sur tes aggregats.
 

Bonjour,
 
Merci pour ton retour.
Effectivement les IO différencies les deux types de disques cependant plus on va générer d'IO et plus on aura de bande passante (mais c'est vrai que pour un même nombre d'IO généré on aura la même bande passante).
 
Effectivement la taille des blocks est de 4k.
 
Le nombre d'IO par agrégat n'est-il pas ce que j'avais déjà mis ? :
 
Pour FC :
"  Initial write "    3173.77
"        Rewrite "    2664.25
"           Read "    4421.38
"        Re-read "    4258.16
"    Random read "    4118.86
" Mixed workload "    3109.89
"   Random write "    2699.23
 
Pour SATA :
"  Initial write "    2580.83
"        Rewrite "    3216.17
"           Read "    4174.34
"        Re-read "    4404.61
"    Random read "    3556.04
" Mixed workload "    3344.87
"   Random write "    2948.47
 
Merci.
Cordialement,

quel est ta charge processeur ?
tes agregats sont -ils sur le meme controleur ?
 
Tu peux utiliser des outils netapp d'analyse :
http://support.netapp.com/NOW/download/tools/perfstat/
et
http://support.netapp.com/NOW/download/tools/sio_ntap/
 
cela permettra d'affiner tes résultats.
tes resultats peuvent etre faussé par la charge déjà existante. la charge de tes serveurs physiques.

Merci pour ton retour.
 
Je me suis arrangé pour lancé les tests à des moments où la charge du contrôleur était comprise entre 5% et 15%.
Il n'y pas de serveur physique uniqement des ESX qui contiennent des VM (je veux dire qu'il n'y a de lun pour partage nfs monté sur un serveur final).
 
Oui les agrégats sont sur le même controleur.
 
Perfstats je connais mais je n'arrive pas à l'analyser parce que c'est illisible.
SIO_ntap connais pas, je vais chercher.
 
J'ai fais des tests avec iometer et j'ai eu grosso modo les mêmes résultats (ce qui exclu un problème d'outil).
 
Cordialement,

Si tu as les resultats de perfstat, envoies ces resultats a Nettap ou a ton prestataire pour t'aider a analyser les resultats.  
tu as peut etre un probleme de conf.
http://support.netapp.com/NOW/download/tools/sio_ntap/  cet outil permet de simuler des IO. puis d'avoir des stats basic. io/s Kb /s transferé .
 
si tu as déjà ces infos, ce n'est peut etre pas utile de recommencer.

Oui c'est ce que j'ai fais ce matin j'ai ouvert un ticket auprès du support netapp et je leur ai envoyé un perfstat
 
Effectivement si sio_ntap est pareil que iometer ou iozone ça ne sert à rien de refaire des tests.

je serais curieux d'avoir le resultat de leur analyse.

Oui tkt pas  
je l'ai ouvert ce matin donc je pense que d'ici la fin de la semaine j'aurai une réponse et je la posterai
par contre j'espère qu'ils trouveront

J'ai eu un retour du support Netapp et il semblerait que iometer ne soit pas optimal pour faire des tests de performance sur des VM avec une baie Netapp. Voici la raison :
 
-----------
Symptoms
 
During VMware performance measuring, results from benchmarking tools may be inaccurate
 
Cause
 
Performance measurement with tools like IOmeter and ntap_sio within virtual machines send back incorrect values because there is virtualization of the CPU cycles.
 
Solution
 
Since benchmarking tools may not be accurate it is highly recommended to refer to VMware documentation. This is a common issue Mware has described here:
 
Performance Tuning Best Practice
http://www.vmware.com/pdf/vi_performance_tuning.pdf
 
Timing numbers reported within the virtual machine can be inaccurate, especially when the processor is over committed.
 
The problem increases when there are more parallel virtual systems on one physical hardware environment. The technical details of the issue is described in the following VMware Technical Report:
 
Timekeeping in VMware Virtual Machines
http://www.vmware.com/pdf/vmware_timekeeping.pdf
 
There is no advantage in retrieving meaningful values with tools like IOmeter or ntap_sio.
 
To get accurate, actual performance, look at the storage controller.
 
Note:
VMware recommends the use of the tool, vmark.
 
------
 
Voilà je fais faire d'autre tests avec cet outil.
 
Merci encore.
Cordialement,