Key-value-stores and other non-relational databases are a hot topic right now, as it increasingly turns out that the traditional (= relational) approach is difficult to scale horizontally. In other words, it may be time — as Bob Ippolito put it — to “Drop ACID and think about data”.

If you couldn’t make it to the first “NoSQL Meetup Berlin” which recently took place at newthinking store — all the talks have been recorded and are on vimeo now:

• Consistency in Distributed Key/Value Stores, Monika Moser
• Redis — Fast and Furious, Mathias Meyer
• Peer to Peer Applications with CouchDB, Jan Lehnardt
• Riak, Martin Scholl
• MongoDB, Mathias Stearn
• 4th Generation Object Databases, Prof. Stefan Edlich

Heise has a good summary (in german).

Here at StudiVZ we are following this topic very closely and are actively investigating some of the alternatives. At the moment, Cassandra looks very promising to us.

Some more links:

• Quick Reference to Alternative data storages
• Comparing Key/Value Stores, Round 2
• NoSQL East 2009 – Summary of Day 1, Day 2

Yesterday we had our 2nd GeekNight and it was at least as successfull as our first event of that manner. It was a premiere in two ways:

• We broadcasted the sessions live through uStream
• Our partner Wooga showed BrainBuddies, the first OpenSocial Gadget running on our platforms. It will become available when we launch OpenSocial

We had about 70 visitors at the Volksbar and over 30 people watched the live stream.
As requested you can download our presentations on the vCard and on BrainBuddies in German.

Watch out for upcoming dates so you can come and join us next time!

Today you have the last chance to get your hands dirty! Our friends at Box119 open their doors for DevHouseBerlin.

DevHouseBerlin is a fun-packed weekend of hacking and sharing knowledge. The Box119 office is open for a weekend and invites hackers of all sorts to join us working on projects, sharing code and ideas and just hanging out among fellow geeks.

DevHouseBerlin is highly influenced by the SuperHappyDevHouse:

SuperHappyDevHouse is a non-exclusive event intended for creative and curious people interested in technology. We’re about knowledge sharing, technology exploration, and ad-hoc collaboration. Come to have fun, build things, learn things, and meet new people. It’s called hacker culture, and we’re here to encourage it.

VZ is happy to provide you with first class Spreeschnittchen Catering. Enjoy!

We are proud to announce that our Gadget Sandbox with OpenSocial 0.8.1 integration is up and running. Developers can upload and test their gadgets against our platforms and request the approval to make them available to our users.
All interested developers are invited to join our OpenSocial support group on meinVZ and studiVZ where more information can be found about how to become a VZ OpenSocial Gadget developer.

We strongly believe that OpenSocial should also been taken literally so we started a monthly GeekNight where all interested developers are informed about our current projects and the release status of our OpenSocial implementation.

OpenSocial will become fully available by the end of the year. This leaves enough room to be out with stable and tested gadgets for the container’s OpenSocial launch.
However we already use GoogleGadgets to provide interactive ads and games on our platforms.

Endlich ist es soweit, die mobile Variante der VZNetzwerke – studiVZ, meinVZ und schülerVZ – ist seit gestern im Android Market kostenlos verfügbar.

Über den mobilen Browser gelangt man direkt über folgende Links zur entsprechenden Anwendung im Market:

market://search?q=pname:net.studivz.android.studivz
market://search?q=pname:net.studivz.android.meinvz
market://search?q=pname:net.studivz.android.schuelervz

Die derzeitige Version 1.0b verfügt unter Anderen über folgende Funktionen:

• Buschfunk
• Lesen und schreiben von Nachrichten
• Anzeigen der Freundesliste
• Anzeige der letzten Besucher
• Gruscheln

Die erste Version stellt somit die wesentlichen Funktionen bereit und wird nun zügig mit weiteren Features, wie sie bereits von der normalen Web-Oberfläche bekannt sind, erweitert.

Bis dahin freuen wir uns natürlich über jegliches Feedback zu den Anwendungen.

On April 26th 2007 Steve Souders wrote:

The user’s proximity to your web server has an impact on response times. Deploying your content across multiple, geographically dispersed servers will make your pages load faster from the user’s perspective. [...] Remember that 80-90% of the end-user response time is spent downloading all the components in the page: images, stylesheets, scripts, Flash, etc. [...] A content delivery network (CDN) is a collection of web servers distributed across multiple locations to deliver content more efficiently to users.

Steve posted this approx. e^3.25809654 days after(!) we started to use a CDN for our web sites. Just some days later we noticed the desired effect. Our users started to make more and more traffic. The activity grew. Of course a CDN is some kind of luxury but it’s worth to invest into such a service at a special time. And from our point of view we thought it was time to. We were right.

Actually round about 286.356,421^2 objects will be requested per month by our users. More than the half of that (5,4E10 objects) are photos. Small, medium and big sized ones. So each of all photo files we store will be loaded round(pow(2,4.91)) times in a month. That makes a monthly traffic volume of more ore less 265.334.489.612.288 bytes only for these kind of objects. The total traffic of all delivered objects per month is something about 1,402939962446178 times higher.

At high traffic times there are over 11000011010100000₂ requests per second hitting our CDN and we are happy that our origin servers only get the (5^5)th part of it.

As a side effect we can learn something about the behaviour of our users because the performance graphs can show us for example what they do in the evening. Maybe the Schimanski serials on 26th of July was one of the reasons for the spikes after 8 pm (see graph above) which are nothing else than commercial breaks. Have a break, have a visit at studiVZ.

Heute, am 20.08.2009, veranstalten wir eine GeekNight, um euch über den aktuellen Entwicklungsstand in Hinsicht auf die Einführung von OpenSocial auf unseren Plattformen zu informieren. Die Veranstaltung ist offen für alle Entwickler, Agenturen und technisch Interessierte.

Nährere Infos sind auf dem entsprechenden Edelprofil bei studiVZ und meinVZ zu finden.

Wir freuen uns auf viele Interessante Gespräche in gepflegter Atmosphäre!

Update
Wir werden Ende September eine weitere GeekNight in Berlin veranstalten, um euch über den aktuellen Stand zu informieren. Der Termin wird rechtzeitig angekündigt, versprochen ;).

Originally we built our automated acceptance tests within our agile development process on a continuous integration server using Selenium with only one Selenium Remote Control. The tests were executed on a fixed browser under a specific operating system. With the growth of the test cases in the test suite the execution time of the builds extended rapidly, so the tests could not directly identify defects and thus a part of their function was lost.

The common strategy to solve this problem is to install Selenium Grid on the machine that formerly hosted the one Selenium RC to parallelize the execution of the tests. By connecting only 4 Selenium RC’s to this Grid Hub the execution time of these tests is reduced by a factor of 4 without any additional hardware and without rewriting the tests. The only prerequisite for this is that the used testing framework supports a parallelized test execution, i.e. it must be able to start more than one test of a test suite simultaneously and assign the answers supplied by Selenium Grid to the right test again.

Although our used testing framework PHPUnit does not provide a parallel execution of tests yet we found a way to use the benefits of Selenium Grid in our testing environment. Our continuous integration server provides the possibility to set up more than one build agent to run the Selenium driven acceptance tests. If we would do this with one single Selenium RC these agents would stress this RC rapidly because there is no possibility to check its state. The agents would start new tests no matter how many tests are already running at this RC.

So we installed Selenium Grid with 4 connected Remote Controls as described above. We can now control the number of simultaneously running tests, because Selenium Grid starts only so many test suites as RC’s are connected. Other incoming requests are queued in the Selenium Grid Hub until one of the connected RC’s has finished its test suite. Unlike the common usage of Selenium Grid we have not yet a real parallelization with this solution, since the test suites from the build agents of the continuous integration server run simultaneously, but each is still to be processed sequentially. But we have always the option of switching to a real parallelization when our testing framework supports it.

We wanted to speed up our web-applications by alleviating our database-loads. So we decided to use the distributed memory object caching system, memcached. Due to the many requests of our memcached-systems (about 1.5 million requests per second), we built a tool (called mckoy), which is capable to perform statistics and debugging information about all memcache-requests in our network.

mckoy is a memcache protocol sniffer (based on pcap library) and statistics builder. It automatically detects and parses each key (and its value) and memcache-api methods.  At  the  end of the sniffing session, the results are used to build the statisticis. mckoy was written to analyse our web application and its usage of  memcache-api in PHP. For example: We wanted to know how many set() and get() methods were invoked in a given time. Based on these results,  we had to make changes to improve the usage of memcache-api for PHP. You can run mckoy on any UNIX based systems. It was tested on many *BSD and Linux systems. mckoy is licensed under GPLv3 and completely published as opensource project!

You can run mckoy in various modes (see manpage!). For example, if you want to sniff pattern “foobar” for all memcache-api methods and with live capturing, use:

mckoy -i <interface> -e “port 11211″ -m 5 -k foobar -v

And this is, how it looks like:

Unfortunately, there are some known bugs. :) For example: An SIGSEGV will encounter when ^C is sent from user. Also, we noticed that mckoy isn’t able to handle memcached-1.2.8 <= 1.4.* correctly. These bugs will be fixed in the next version as soon as possible! For the next version I also planned to build in udp and binary support.

You can offcially download mckoy from:
http://www.lamergarten.de/releases.html
or
http://sourceforge.net/projects/mckoy/

cheers.

I attended an awesome talk by Kenneth Lundin about Erlang/OTP at the Erlang Factory in London. The main topic was SMP and it’s improvements it in the latest release(s). That’s exactly one of the main reasons for Erlang, parallelize computations on many cores, without worrying about locks in shared memory.

Some of the issues they’ve been working on:

1. Erlang now detects CPU Topology automatically at startup.
2. Multiple run-queues
3. You can lock schedulers to logical CPU’S
4. Improved message passing – reduced lock time

They improved more things of course but considering SMP these are the most important ones.

1. Erlang now detects the CPU topology of your system automatically at startup. You may still override this automatic setup using:
   erl +sct L0-3c0-3
erlang:system_flag(cpu_topology,CpuTopology).
2. Multiple run queues … what does that mean? We should first take a look at how Erlang does SMP:
   • Erlang without SMP:
     Without SMP support the Erlang VM had one Scheduler for one runqueue. So all the jobs were pushed on one queue and fetched by one scheduler.
   • Erlang SMP / before R13
     They started more schedulers that were pulling jobs from one queue. Sounds more parallel but still not performing as good as desired on many cores.
   • Erlang SMP R13
     Several schedulers like in the former solution but each of them has it’s own runqueue. The problem with this approach is that it can of course happen that you end up with some empty and some full queues because of the different runtime of the processes. So they build something called migration logic that is controlling and balancing the different runqueues.

   They migration logic does:

   • collect statistics about the maxlength of all scheduler’s runqueues
   • setup migration paths
   • Take away jobs from full-load schedulers and pushing jobs on low load scheduler queues

   Running on full load or not! If all schedulers are not fully loaded, jobs will be migrated to schedulers with lower id’s and thus making some schedulers inactive.

   This makes perfectly sense because the more schedulers and runqueues you need the more migrating has to be done. Using SMP support with many schedulers makes only sense if you’re really optimizing for many cores and you will have decreased performance on systems with few cores.

3. Binding schedulers to CPU’s is really worth looking at it. The more cores your CPU has the more important it’ll be and the more performance improvement you’ll gain. You can force the erlang VM to do scheduler binding by:
   erl +sbt db
erlang:system_flag(scheduler_bind_type,default_bind).1>erlang:system_info(cpu_topology).
[{processor,[{core,{logical,0}},
{core,{logical,3}},
{core,{logical,1}},
{core,{logical,2}}]}]
2> erlang:system_info(scheduler_bindings).
{unbound,unbound,unbound,unbound}
fabrizio@machine:~$ erl +sbt db
1> erlang:system_info(scheduler_bindings).
{0,1,3,2}

Source: presentation Kenneth Lundin – Erlang-Factory

You can test and benchmark SMP using following flags:
fabrizio@machine:~$ erl -smp disable       //default is auto
fabrizio@machine:~$ erl +S 2:4               //Number of Schedulers : Schedulers online

With erlang:system_info/1 you can use the following atoms

# cpu_topology
# multi_scheduling
# scheduler_bind_type
scheduler_bindings
logical_processors
multi_scheduling_blockers
scheduler_id
schedulers
# schedulers_online
smp_support

The ones marked with # can be set using system_flag/2