jueves, 11 de octubre de 2012

Análisis de rendimiento de servidores GNU/Linux ( y parte II)

De la serie: Manos a la obra


Del anterior articulo de la serie Rendimiento de servidores GNU/Linux (Parte I)


Repasemos las tareas planificadas con su estado de ejecución:
  1. (Realizada) Selección del hardware para la comparación.
  2. (Realizada) Instalación y configuración del sistema operativo en los servidores, evidentemente GNU/Linux.
  3. (Realizada) Instalación y configuración del software de estudio de rendimiento y revisión de configuración.
  4. (Realizada) Definición de los juegos de pruebas a realizar.
  5. (No realizada) Ejecución de la prueba.
  6. (No realizada) Análisis de resultados y conclusiones.

En esta 2a parte (y final) se documentarán las tareas 5 y 6.


Ejecución de la prueba


Utilizando los valores de ejecución de la entrada anterior se generan los siguientes resultados para cada una de las pruebas ejecutadas:

CPU
Para el hardware HP8200:


$ sysbench --test=cpu --cpu-max-prime=20000 run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1

Doing CPU performance benchmark

Threads started!
Done.

Maximum prime number checked in CPU test: 20000


Test execution summary:
    total time:                          23.5936s
    total number of events:              10000
    total time taken by event execution: 23.5928
    per-request statistics:
         min:                                  2.28ms
         avg:                                  2.36ms
         max:                                  5.08ms
         approx.  95 percentile:               2.46ms

Threads fairness:
    events (avg/stddev):           10000.0000/0.00
    execution time (avg/stddev):   23.5928/0.00

Para el hardware virtual:

$ sysbench --test=cpu --cpu-max-prime=20000 run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1

Doing CPU performance benchmark

Threads started!
Done.

Maximum prime number checked in CPU test: 20000


Test execution summary:
    total time:                          24.9474s
    total number of events:              10000
    total time taken by event execution: 24.9460
    per-request statistics:
         min:                                  2.48ms
         avg:                                  2.49ms
         max:                                  3.23ms
         approx.  95 percentile:               2.55ms

Threads fairness:
    events (avg/stddev):           10000.0000/0.00
    execution time (avg/stddev):   24.9460/0.00



Se ha remarcado el dato más interesante en fondo azul.



Threads
Para el hardware HP8200:


$ sysbench --num-threads=128 --test=threads --thread-yields=200 --thread-locks=4 run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 128

Doing thread subsystem performance test
Thread yields per test: 200 Locks used: 4
Threads started!
Done.


Test execution summary:
    total time:                          0.5518s
    total number of events:              10000
    total time taken by event execution: 70.0153
    per-request statistics:
         min:                                  0.05ms
         avg:                                  7.00ms
         max:                                 61.09ms
         approx.  95 percentile:              23.76ms

Threads fairness:
    events (avg/stddev):           78.1250/7.49
    execution time (avg/stddev):   0.5470/0.00


Para el hardware virtual:

$ sysbench --num-threads=128 --test=threads --thread-yields=200 --thread-locks=4 run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 128

Doing thread subsystem performance test
Thread yields per test: 200 Locks used: 4
Threads started!
Done.


Test execution summary:
    total time:                          0.6431s
    total number of events:              10000
    total time taken by event execution: 80.9499
    per-request statistics:
         min:                                  0.05ms
         avg:                                  8.09ms
         max:                                317.48ms
         approx.  95 percentile:              29.18ms

Threads fairness:
    events (avg/stddev):           78.1250/9.55
    execution time (avg/stddev):   0.6324/0.00



Se ha remarcado el dato más interesante en fondo azul.


Entrada/Salida de ficheros

Esta prueba requiere de un poco de preparación:
  1. Se verifica que tengamos disponible la cantidad de espacio con el que se quiere realizar la prueba. Yo la realizaré con un fichero de 5GB, por ello verifico que en el filesystem en el que me encuentre disponga de este espacio.
  2. Se crean el/los ficheros para realizar la prueba mediante en comando:
    sysbench --test=fileio --file-total-size=5G prepare
    Una vez ejecutado han aparecido (en mi caso) 128 ficheros 'test_file.NN'. Se pueden observar las opciones de ejecución para este test con el comando:
    sysbench --test=fileio help   
    Se debe tener en cuenta que es muy importante que la medida del fichero debe ser mayor que la memoria RAM disponible para que este no quepa en la memoria RAM e invalide las pruebas.
  3. Ahora ya se puede ejecutar el comando de la prueba:
    sysbench --test=fileio --file-total-size=5G --file-test-mode=rndrw --init-rng=on --max-time=300 --max-requests=0 run      
  4. ¿Que os parece que nos falta?. Nos hemos dejado los ficheros de prueba, nos estaran ocupando (en este caso) 5GB del disco. Ejecutamos:
    rm test_file.*



Para el hardware HP8200:

 sysbench --test=fileio --file-total-size=5G --file-test-mode=rndrw --init-rng=on --max-time=300 --max-requests=0 run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1
Initializing random number generator from timer.


Extra file open flags: 0
128 files, 40Mb each
5Gb total file size
Block size 16Kb
Number of random requests for random IO: 0
Read/Write ratio for combined random IO test: 1.50
Periodic FSYNC enabled, calling fsync() each 100 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random r/w test
Threads started!
Time limit exceeded, exiting...
Done.

Operations performed:  13736 Read, 9157 Write, 29184 Other = 52077 Total
Read 214.62Mb  Written 143.08Mb  Total transferred 357.7Mb  (1.1923Mb/sec)
   76.31 Requests/sec executed

Test execution summary:
    total time:                          300.0076s
    total number of events:              22893
    total time taken by event execution: 191.3679
    per-request statistics:
         min:                                  0.00ms
         avg:                                  8.36ms
         max:                                226.02ms
         approx.  95 percentile:              26.28ms

Threads fairness:
    events (avg/stddev):           22893.0000/0.00
    execution time (avg/stddev):   191.3679/0.00

Para el hardware virtual:

 $ sysbench --test=fileio --file-total-size=5G --file-test-mode=rndrw --init-rng=on --max-time=300 --max-requests=0 run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1
Initializing random number generator from timer.


Extra file open flags: 0
128 files, 40Mb each
5Gb total file size
Block size 16Kb
Number of random requests for random IO: 0
Read/Write ratio for combined random IO test: 1.50
Periodic FSYNC enabled, calling fsync() each 100 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random r/w test
Threads started!
Time limit exceeded, exiting...
Done.

Operations performed:  87491 Read, 58327 Write, 186624 Other = 332442 Total
Read 1.335Gb  Written 911.36Mb  Total transferred 2.225Gb  (7.5945Mb/sec)
  486.05 Requests/sec executed

Test execution summary:
    total time:                          300.0063s
    total number of events:              145818
    total time taken by event execution: 262.3669
    per-request statistics:
         min:                                  0.00ms
         avg:                                  1.80ms
         max:                                415.03ms
         approx.  95 percentile:               8.66ms

Threads fairness:
    events (avg/stddev):           145818.0000/0.00
    execution time (avg/stddev):   262.3669/0.00


Se ha remarcado el dato más interesante en fondo azul.

Análisis de resultados y conclusiones

De forma general, se puede se han visto alguna de las características de la utilidad sysbench. Esta herramienta se creó por parte de MySQL AB para el análisis de rendimiento de su base de datos con licencia GPLv2 (cabe recordar que esta empresa sueca fue comprada por Sun Microsystems y esta en el 2009 fue comprada por Oracle Corporation). Existen 2 opciones mas de análisis que no han sido exploradas (una de ellas si comentada), con las tres ejecutadas he tenido suficiente para verificar el (bastante) buen comportamiento del servidor virtual respecto al físico.

CPU

 Los resultados son muy parecidos,incluso en el percentíl del 95% el servidor virtual se comporta bastante bien. En este punto hubiera esperado más diferencia.


Threads
 En este aspecto, así como en el de la CPU, las diferencias de resultados son bastante contenidas. Cabe destacar el elevado tiempo de respuesta máximo para alguna de las operaciones, probablemente muy pocas ya que el percentíl sigue siendo bastante parejo al de la máquina física.


 Entrada/Salida de ficheros
 Este es el indicador sorpresa, a pesar del elevadísimo tiempo de respuesta máximo para alguna de las operacionesel percentíl 95% ya es mejor, pero el resultado de la tasa de transferencia es del orden de 7 veces más entre la máquina física y la virtual. En este indicador la diferencia entre el disco ata del PC físico y los super discos del armario de almacenamiento de la cabina de máquinas virtuales juega un papel muy importante.


En definitiva, se puede afirmar que, aunque exista una pequeña perdida de rendimiento unitario en el sistema huésped, los beneficios de la administración de sistemas de forma global supera con creces esa pequeña perdida de rendimiento. En el aspecto económico de optimización de sistemas también ya que permite un mejor aprovechamiento de los recursos. Sin comentarios en cuanto a la alta disponibilidad de los sistemas, tiempos de recuperación, de backups, restauraciones y un largo etcétera.

Con esto se finalizan las tareas planificadas:
  1. (Realizada) Selección del hardware para la comparación.
  2. (Realizada) Instalación y configuración del sistema operativo en los servidores, evidentemente GNU/Linux.
  3. (Realizada) Instalación y configuración del software de estudio de rendimiento y revisión de configuración.
  4. (Realizada) Definición de los juegos de pruebas a realizar.
  5. (Realizada) Ejecución de la prueba.
  6. (Realizada) Análisis de resultados y conclusiones.


Un saludo a mis 2 o 3 lectores!



domingo, 7 de octubre de 2012

Internet access through a Micro$ft proxy from a GNU/Linux client


From the serie: Get down to the job

Maybe it never happened to you, but there are some working scenarios where we are behind a Micro$otf Proxy and we still want to keep using out GNU/Linux client box in a normal way. For the unpatient: It's possible but you have to follow some actions and be carefully.

Before we start, I don't want to waste your time, if the one and only thing you want to get is an Internet access with a browser, you don't need to continue reading, all of the browsers in the net as the possibility of connecting to a web proxy. All you need is: Window$ Domain user credentials and the information where proxy is.

In this article we are going to explain how to configure a GNU/Linux box behind a MS-Proxy and be able to do some of this tasks (i.e.):
  • Update system
  • Install distribution packages this system tools
  • Instant Messaging (IM) with a specific client (not browser)

 Requirements

 All the information we have to gather to do a right configuration for the access through the MS-Proxy is:
  • Web proxy for downloading distribution packages for the system.
  • DHCP client in our GNU/Linux box
  • Network information of the proxy server (IP adress, port number)
  • Credentials in Window$ Domain to go through the proxy

Planning

 As always, we should plan the task for the execution:
  1. Gather necessary information from the net
  2. Config webproxy in browser for downloading GNU/Linux Debian packages.
  3. Download, installation of the cntlm package  and all its dependences (if are needed)
  4. GNU/Linux box cntlm config to authenticate and access through the proxy
  5. Test deployment


Gather necessary information from the net

What do we need to begin our project?
  • Output proxy information (IP address and port). Note: M$-proxy protocol is not necessary, it's going to be find out by the cntlm.
  • Do we have a windows domain user with go through the proxy grant?


Config webproxy in browser

Second important thing to be care of, are we able to download Debian packages via web?. It's very important because the cntlm and all its dependencies are needed. Let's see how to do that ...

Normally, M$-proxy is open as a webproxy service, that means it's possible to config a browser to use it to go to the Internet. You just have to go to (for Firefox): Menu edit -> Preferences -> Advanced icon -> Network tab -> Settings button -> Manual proxy configuration checkbox

[IMG]

(for the good observers :-)  yes the capture was taken with an Ubuntu box, it's the machine I had on my hands in that moment)



Download, install of the cntlm package

Put you browser to Debian GNU/Linux repository at: http://packages.debian.org/wheezy/cntlm, this is the homepage for the package. Be sure to select you architecture and click on it to download.



In my case it's http://packages.debian.org/wheezy/amd64/cntlm/download because, you know, I'm using wheezy version with amb64. Be sure to choose the right one.

After downloading, and before install, we have to be sure all cntlm packages dependences are fulfilled. Know do we know these?, Here comes aptitude to save us:

$aptitude show cntlm 

Package: cntlm                           
State: installed
Automatically installed: no
Version: 0.92.3-1
Priority: optional
Section: net
Maintainer: David Watson 
Architecture: amd64
Uncompressed Size: 180 k
Depends: adduser, libc6 (>= 2.10)
Replaces: ntlmaps
Description: Fast NTLM authentication proxy with tunneling
 Cntlm is a fast and efficient NTLM proxy, with support for TCP/IP tunneling, authenticated connection caching, ACLs, proper
 daemon logging and behaviour and much more. It has up to ten times faster responses than similar NTLM proxies, while using
 by orders or magnitude less RAM and CPU. Manual page contains detailed information.
Homepage: http://cntlm.sourceforge.net/


This command shows information about the package been asked for and, of courses its dependencies. Line 'Depends' shows it has two, adduser and libc6. Know you have to confirm these packages are installed:


$dpkg -l 'adduser'
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                      Version           Architecture      Description
+++-=========================-=================-=================-=======================================================
ii  adduser                   3.113+nmu3        all               add and remove users and groups 
$ dpkg -l 'libc6'
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                      Version           Architecture      Description
+++-=========================-=================-=================-=======================================================
ii  libc6:amd64               2.13-35           amd64             Embedded GNU C Library: Shared libraries

Please, look in man about dpkg command to learn more uses for it. And just a note, all these commands has been executed with a non privileged user (not root).



GNU/Linux box cntlm config to authenticate and access through the proxy

Once you have installed cntlm package is time to config authentication against M$-Proxy. First of any other action: Make a backup of the conf file your are going to modify (this is something not necessary to remember every time but sometimes you, me and everybody forgets). This is what I do:

$ cp /etc/cntlm.conf /etc/cntlm.conf.orig

Now edit the file we just backed up, modify these entries:
  • Username
  • Domain
  • (Comment line) Password
  • proxy
  • NoProxy
If it's possible you should use the highest authentication protocol your proxy serves. How can you know it? good question, read man page ...

... now that you have already read man page for cntlm you should know there is a parameter to query that to MS-Proxy via cntlm executable:
cntlm -I -M http://www.google.com

These parameters give us the best (or the highest) security protocol to communicate against the proxy server. Use the hash string that comment give us to authenticate at proxy server. See that example (captured from http://cntlm.sourceforge.net/):



$ cntlm -I -M http://test.com
Config profile  1/11... OK (HTTP code: 200)
Config profile  2/11... OK (HTTP code: 200)
Config profile  3/11... OK (HTTP code: 200)
Config profile  4/11... OK (HTTP code: 200)
Config profile  5/11... OK (HTTP code: 200)
Config profile  6/11... Credentials rejected
Config profile  7/11... Credentials rejected
Config profile  8/11... OK (HTTP code: 200)
Config profile  9/11... OK (HTTP code: 200)
Config profile 10/11... OK (HTTP code: 200)
Config profile 11/11... OK (HTTP code: 200)
----------------------------[ Profile  0 ]------
Auth            NTLMv2
PassNTLMv2      4AC6525378DF8C69CF6B6234532943AC
------------------------------------------------


And modify that parameter in cntlm.conf (see string mark in yelow) in this case my proxy is able to authenticate in NTLMv2.



#
# Cntlm Authentication Proxy Configuration
#
# NOTE: all values are parsed literally, do NOT escape spaces,
# do not quote. Use 0600 perms if you use plaintext password.
#

Username    adrian
Domain        MYDOMAIN
#Password    ******
# NOTE: Use plaintext password only at your own risk
# Use hashes instead. You can use a "cntlm -M" and "cntlm -H"
# command sequence to get the right config for your environment.
# See cntlm man page
# Example secure config shown below.
# PassLM          1AD35398BE6565DDB5C4EF70C0593492
# PassNT          77B9081511704EE852F94227CF48A793
### Only for user 'testuser', domain 'corp-uk'
PassNTLMv2      6E2567JFCGHIIIKLL23445CC4ED58D24

# Specify the netbios hostname cntlm will send to the parent
# proxies. Normally the value is auto-guessed.
#
# Workstation    netbios_hostname

# List of parent proxies to use. More proxies can be defined
# one per line in format <proxy_ip>:<proxy_port>
#
Proxy        proxyxoc3.micro$oft.com:80

# List addresses you do not want to pass to parent proxies
# * and ? wildcards can be used
#
NoProxy        localhost, 127.0.0.*, 10.*, 192.168.*, *.microSoft.com
# Specify the port cntlm will listen on
# You can bind cntlm to specific interface by specifying
# the appropriate IP address also in format <local_ip>:<local_port>
# Cntlm listens on 127.0.0.1:3128 by default
#
Listen        3128

# If you wish to use the SOCKS5 proxy feature as well, uncomment
# the following option. It can be used several times
# to have SOCKS5 on more than one port or on different network
# interfaces (specify explicit source address for that).
#
# WARNING: The service accepts all requests, unless you use
# SOCKS5User and make authentication mandatory. SOCKS5User
# can be used repeatedly for a whole bunch of individual accounts.
#
#SOCKS5Proxy    8010
#SOCKS5User    dave:password

# Use -M first to detect the best NTLM settings for your proxy.
# Default is to use the only secure hash, NTLMv2, but it is not
# as available as the older stuff.
#
# This example is the most universal setup known to man, but it
# uses the weakest hash ever. I won't have it's usage on my
# conscience. :) Really, try -M first.
#
#Auth        LM
#Flags        0x06820000

# Enable to allow access from other computers
#
#Gateway    yes

# Useful in Gateway mode to allow/restrict certain IPs
# Specifiy individual IPs or subnets one rule per line.
#
#Allow        127.0.0.1
#Deny        0/0

# GFI WebMonitor-handling plugin parameters, disabled by default
#
#ISAScannerSize     1024
#ISAScannerAgent    Wget/
#ISAScannerAgent    APT-HTTP/
#ISAScannerAgent    Yum/

# Headers which should be replaced if present in the request
#
#Header        User-Agent: Mozilla/4.0 (compatible; MSIE 5.5; Windows 98)

# Tunnels mapping local port to a machine behind the proxy.
# The format is <local_port>:<remote_host>:<remote_port>
# 
#Tunnel        11443:remote.com:443



Test deployment 

In fact we have test access through proxy yet with command line when we was trying to configure protocol and passwd.

I usually use local proxy to update and install system packages for linux clients. For access to Internet I have more that enough this the webproxy service. If this is your case you can use system packages update to test your environment. This is how I do it:
  1. Configure apt to access through the proxy. Modify (or create if it doesn't exist) this file: /etc/apt/apt.conf.d/80apt-proxy and add:   Acquire::http::Proxy "http://localhost:3128";   
  2. Now you can test proxy updating the packages list, execute: aptitude update.  If every thing is OK, you should have updated you package list with no error in screen.

References (thanks a lot to every one)





What's next

  •  I will continue working and writing for my 3 or 4 readers to configure a desktop environment using GNU/Linux



Goodbye my friends ...