Thursday, January 24, 2008

Benchmarking CouchDB

After installation of CouchDB, I was wondering about it's performance, so I made some benchmarking, and it looks pretty good to me - 2,000 concurrent requests.

I only had to raise number of open files per process by using

echo "204854" > /proc/sys/fs/file-max

ulimit -n 200000

to get rid of this message
OS error code 24: Too many open files

I'll publish results as Google docs, so they are posted to this blog automatically and I'll provide just a link to results.

This are read tests :

http://ihatecubicle.blogspot.com/2008/02/benchmarking-couchdb-read.html

http://ihatecubicle.blogspot.com/2008/02/benchmarking-couchdb-read-2000.html

http://ihatecubicle.blogspot.com/2008/02/benchmarking-couchdb-read-6000.html

I'm working on creation of files with sample data to emulate document for insertion, so we can test write efficiency.

Wednesday, January 23, 2008

Amazon EC2 - server response 400 (Bad Request)

etch:/mnt# ec2-upload-bundle -b khaz_debian_elasticdrive041_02 -m /mnt/image.manifest.xml -a key -s secretkey

Setting bucket ACL to allow EC2 read access ...
Error: could not create or access bucket khaz_debian_elasticdrive041_02: server response 400 (Bad Request)
ec2-upload-bundle failed

trying --debug option showed, that create bucket operation fails.

Searching Amazon EC2 forums didn't give anything valuable, so I'd tried to create bucket manually and voila - it spits error that sound like "You've tried to create more buckets than allowed". So deleting some garbage buckets helped to get rid of this hard to trace error (at least for me).

Setup Elastic Drive in Jeos/VMWare

Sometimes if you try to run Elastic Drive

root@khazJeosED:/opt/e# sudo elasticdrive /etc/elasticdrive.ini -d

and get following error:

Traceback (most recent call last): File "/usr/bin/elasticdrive", line 5, in pkg_resources.run_script('ElasticDrive==0.4.1-FREE-5-r54', 'elasticdrive') File "build/bdist.linux-i686/egg/pkg_resources.py", line 448, in run_script File "build/bdist.linux-i686/egg/pkg_resources.py", line 1173, in run_script File "/usr/bin/elasticdrive", line 175, in
File "/usr/bin/elasticdrive", line 156, in main
File "build/bdist.linux-i686/egg/elasticdrive/app_server.py", line 99, in start_drives File "build/bdist.linux-i686/egg/elasticdrive/s3_simple.py", line 57, in __init__ File "/usr/lib/python2.5/site-packages/boto-0.9d-py2.5.egg/boto/s3/connection.py", line 103, in create_bucket raise S3ResponseError(response.status, response.reason, body) boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden RequestTimeTooSkewedThe difference between the request time and the current time is too large.2008-01-23T18:47:06Z439AD07AEA9A14C6900000a7RNB5bY7n1muIce2Ht40zcMUj4qkwJV6vtHdgWFzA6ezn3aov171Ov1UrHXWweZWed, 23 Jan 2008 01:31:49 GMT Exception exceptions.AttributeError: "s3_simple_engine instance has no attribute 'locker'" in > ignored

it means that local time should be adjusted to comply Amazon S3 requirement (max difference might be 10 minutes).

Extract time part from response 2008-01-23T18:47:06Z minus 5 hours, and use it to set time in sync. Run following command:

date --set="2008-01-23 13:47:43"

run again

root@khazJeosED:/opt/e# sudo elasticdrive /etc/elasticdrive.ini -d

if no errors, lets check it all goes well

#ps aux
...
root 5480 0.2 1.0 18176 5284 ? Ssl 13:59 0:00 /usr/bin/python /usr/bin/elasticdrive /etc/elasticdrive.ini -d
root 5482 0.2 1.0 18128 5348 ? Ssl 13:59 0:00 /usr/bin/python /usr/bin/elasticdrive /etc/elasticdrive.ini -d
root 5485 0.0 0.9 9660 4836 ? S 13:59 0:00 /usr/bin/python /usr/bin/elasticdrive /etc/elasticdrive.ini -d
root 5488 0.0 0.1 2620 1000 pts/0 R+ 14:00 0:00 ps aux

let's mount filesystem

root@khazJeosED:/opt/e# sudo mount -o loop /fuse2/ed0 /s3
mount: you must specify the filesystem type

this might mean that you need to run

#mke2fs -b 4096 /fuse2/ed0

mke2fs 1.40.2 (12-Jul-2007)
/fuse2/ed0 is not a block special device.Proceed anyway? (y,n) y
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
65536 inodes, 65536 blocks
3276 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=67108864
2 block groups
32768 blocks per group, 32768 fragments per group
32768 inodes per group
Superblock backups stored on blocks:
32768
Writing inode tables: done

Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 33 mounts or180 days, whichever comes first. Use tune2fs -c or -i to override.
root@khazJeosED:/opt/e#


now we can run the previous command:

root@khazJeosED:/opt/e# sudo mount -o loop /fuse2/ed0 /s3
root@khazJeosED:/opt/e#

Let check it went fine:

root@khazJeosED:/opt/e# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 2.8G 474M 2.2G 18% /
varrun 252M 44K 252M 1% /var/run
varlock 252M 0 252M 0% /var/lock
udev 252M 44K 252M 1% /dev
devshm 252M 0 252M 0% /dev/shm
/fuse2/ed0 248M 144K 236M 1% /s3

Monday, January 21, 2008

Running Selenium Grid on Amazon EC2

Synopsis:
This article describes a process of setting up and running Selenium Grid web testing tool in distributed fashion.

Pre-requisistes:
You should have an Amazon Elastic Compute Cloud account
It is implied, that you have basic Unix/Linux administration skills and some experience with Amazon EC2 tools
Some knowledge of Selenium testing ( read more at http://www.openqa.org/selenium-grid/ )


How to start rolling:
1. Run instance of AMI ami-fd37d294 and get Public DNS of the instance.

2. Login into ssh terminal, using Public DNS.

3. Run following commands:

#cd /usr/share/selenium-grid-0.9.3
#ant sanity-check


Instances of this AMI can be run as "hub" or "remote control" depending on your layout. If this is going to be a remote control:

#ant -Dport= -Dhost= -DhubURL= launch-remote-control
ant -Dport=5555 -Dhost=testhost -DhubURL=http://your-hub-url:4444 -Denvironment="Firefox on Linux" launch-remote-control


Future work:
- pass parameters before launch, so it runs in hub or rm mode, all neat and automated
- add Firefox on Windows and Safari on OS X profiles

Sunday, January 20, 2008

Geotourism in Kazakhstan



What is geotourism ?
National Geographic defines geotourism as tourism that sustains or enhances the geographical character of a place -its environment, culture, aesthetics, heritage, and the well-being of its residents. Geotourism incorporates the concept of sustainable tourism -that destinations should remain unspoiled for future generations- while allowing for enhancement that protects the character of the locale. Geotourism also adopts a principle from its cousin, ecotourism -that tourism revenue can promote conservation- and extends that principle beyond nature travel to encompass culture and history as well: all distinctive assets of a place.

Challenge
Nominate your favorite example of geotourism -- defined by National Geographic as tourism that sustains or enhances the geographical character of a place: its environment, heritage, culture, aesthetics, and the well-being of its residents.
All nominees will receive a personal email inviting them to enter the National Geographic-Ashoka's-Changemakers Geotourism Challenge launching on jan 30, 2008. All nominators who tell their first-person story become eligible to win one of two whl.travel prizes.
http://www.changemakers.net/en-us/competition/geotourism

My nomination - Falcon hunt in Central Kazakhstan

Hunting with golden eagles in Kazakhstan, is ancient and deeply esteemed tradition, and hunters-berkutchi are one of the most esteemed persons in a society.
The first world falconry festival took place on July14-15, 2007 in Reading, Berkshire, United Kingdom http://www.falconryfestival.com/
Festival gathered teams from 33 countries, including USA, France, Germany, Belgium, Japan, Brazil, China, South Korea, Turkmenistan, England and others. Kazakhstan team was recognized the best one.

If you are new to falconry, Alan Gates has good article about hunting with falcon at http://www.avmv20.dsl.pipex.com/Articles/The%20Hunt.htm

I would like to present my friend's falconry estate at Central Kazakhstan, that offers falcons, hawks and golden eagles to hunt wolf, fox, hare, black-cocks, ducks, partridges and pigeons. Since 1995 they hold international competitions “Salburn” - hunting with birds and Kazakh “tazy”- wolfhound, while riding horses.

For more details visit their website at http://www.pmicro.kz/~falcon/falcon_e/
or just give them a call:
Tileukabyl Esembekuly: 7 7212 45-41-02 , 7 300 244 65 69, tleu.e@rambler.ru
Karlygash Makataeva: 7 7272 71-26-17, 7 300 755 20 86, kmakataeva@yahoo.com
[UPDATE]

Thursday, January 10, 2008

Benchmarking Windows 2003 Server in Amazon EC2

Abstract: Some notes on trying to run Windows 2003 Server EE on Amazon EC2 extralarge type of instances (4 way).

SMP
I tried to use Qemu's -smp n feature on Amazon Elastic Compute Cloud, but it still uses only one processor. It is possible to set affinity and run each Qemu instance on individual processor, but this is very nontypical scenario. Also this article explains Qemu's "rdtsc" usage on SMP hosts http://lists.gnu.org/archive/html/qemu-devel/2007-03/msg00652.html

ramfs
Moving system disk to RAM decreased disk-related IO by 10-15%, but CPU-bound bottleneck still slowed system down.

ACPI SMP
Windows Device Manager showed that our installation uses MP kernel according to "ACPI Multiprocessor PC", ACPI APIC MP HAL (Halmacpi.dll).

I'll keep you informed on my other findings.

Monday, January 7, 2008

How to setup LDAP authentication in Alfresco

Synopsis: This articles describes a process of setting up an LDAP authentication in Afresco content management system.


OpenLDAP
Setting it up is pretty trivial, I used yum. It is important to add initial entries into fresh installation, since it comes totally empty and spits errors. This is how I did it:
#nano khaz.ldif
dn: dc=khaz-domain,dc=com
objectClass: domain
dc: khaz-domain

#slapadd -l khaz.ldif
#chown ldap:ldap /var/lib/ldap/objectClass.bdb
#/etc/rc.d/init.d/ldap restart

Atlassian Crowd
Crowd is a web-based single sign-on (SSO) tool that simplifies application provisioning and identity management. I used it as front-end tool for OpenLDAP to manage users.
Install the software and login into administration panel at something like http://yoursite.com:8095/crowd/console
choosing Directories tab and click on add directory. Choose "Crowd supports several connectors such as Active Directory, Sun ONE and Open Directory. " Connector button and fill in details (of OpenLDAP installation).

Alfresco
I used bundled version (tomcat + alfresco) with HSQL database, which might be switched to another one like MySQL.
Firstly, I tried it on my desktop in VMware server and then on Amazon EC2 instance, running under Fedora Core 6.
During initial stage I turned on debug mode to see exactly what was going on, and it really helped me to trace LDAP communication messages between my OpenLDAP server and Alfresco.
Use this settings as a guidance:
/opt/alfresco/tomcat/shared/classes/alfresco/extension/chaining-authentication-context.xml
/opt/alfresco/tomcat/shared/classes/alfresco/extension/ldap-authentication-context.xml

http://s3.amazonaws.com/khaz_download/chaining-authentication-context.xml

http://s3.amazonaws.com/khaz_download/ldap-authentication-context.xml

Adding users to Alfresco
Log in to Crowd panel.
Choose Principals tab > then OpenLDAPForAlfresco (this is how I named it, yours might have different name) in Directory dropdown and hit Search button
This should bring a list of users in directory. To add new user, locate Add Principal in Principal Browser tab and click on it. This will change to form, where you fill in user details and select proper directory for user to belong to.
Upon successful creation of user account, you can test it in Alfresco at http://youralfrescoinstallation.com:8080/alfresco At this point all users are managed outside of alfresco and might be easily attached to other services like single sign-on and OpenID.

Cloud Computing Google Group