My Journey with HHVM-Pt.1

Facebook released HHVM, a new PHP interpreter that runs as a virtual machine. It is suppose to out perform PHP5-FPM. Currently one of our projects is running PHP5-FPM. In this series of posts I will outline our thought process in evaluating HHVM as a viable solution for us.

The first step before switching over to HHVM is to see if we get any performance benefits from doing so. In this article I am going to compare the performance of PHP5-FPM against HHVM. My experience with HHVM is limited so I appreciate your feedback!

Summary

  • HHVM out of the box build (master: 5b00837d9b2ecc28e6225336296be88449637b2e) over a TCP socket takes 2.83 times the amount of time it takes PHP5-FPM over a TCP socket to serve the same number of request.
  • PHP5-FPM over a TCP socket takes 1.08 times the amount of time it takes PHP5-FPM over a unix socket to serve the same number of requests.
  • HHVM over a TCP socket takes 1.26 times the amount of time it takes HHVM over a unix socket to serve the same requests.
  • Increasing the CoreFileSize for HHVM to 100Mb does not improve it’s performance.

Please read post #2 to see why the conclusions drawn in this article are incorrect.

My setup

Before we jump into the thick of it, here is my setup:

  • AWS EC2 c3.large instance
  • Ubuntu 13.10
  • Built HHVM from source commit# 5b00837d9b2ecc28e6225336296be88449637b2e
  • PHP5-FPM: PHP 5.5.3
  • nginx version: nginx/1.4.1 (Ubuntu)

I will not be going over how to compile HHVM code. You can find the instructions here.

My process

I run this test in each scenario with apache bench:


ab -c 100 -n 10000 http://buyable.me/product/view/1561

Running PHP5-FPM over a Unix socket

The project does not get much load so it is not load balanced. PHP5-FPM is running locally as is nginx, which is connecting to PHP5-FPM via a Unix socket.

I run the test. Here are the results:


Document Path:          /product/view/1561
Document Length:        27716 bytes

Concurrency Level:      100
Time taken for tests:   40.926 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      282506490 bytes
HTML transferred:       277160000 bytes
Requests per second:    244.34 [#/sec] (mean)
Time per request:       409.264 [ms] (mean)
Time per request:       4.093 [ms] (mean, across all concurrent requests)
Transfer rate:          6741.01 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        1    1  10.0      1    1001
Processing:    24  406  71.8    389     930
Waiting:       20  404  71.8    388     929
Total:         28  407  72.4    390    1385

Mean time per request 409ms and total time for the test 40.926 seconds.

Here is a bit of the nginx config:


location ~ .php$ {
        fastcgi_split_path_info ^(.+\.php)(.*)$;
        fastcgi_pass unix:/var/run/php5-fpm.sock;
        fastcgi_index  index.php;
        include /etc/nginx/fastcgi_params;
        fastcgi_param  SCRIPT_FILENAME /var/www/application/www$fastcgi_script_name;
        fastcgi_param  QUERY_STRING     $query_string;
        fastcgi_param  REQUEST_METHOD   $request_method;
        fastcgi_param  CONTENT_TYPE     $content_type;

PHP5-FPM over a TCP socket

For applications that experience a lot of load, PHP5-FPM can be run behind a load balancer. Unix sockets work only when connecting to a service locally. Typically the load balancer would be remote(not on the same machine as the web server). To connect to it, one would need a TCP socket. To simulate this setup I run PHP5-FPM on 127.0.0.1:9000.

I rerun the test. Here are the results


Document Path:          /product/view/1561
Document Length:        27716 bytes

Concurrency Level:      100
Time taken for tests:   44.333 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      282506340 bytes
HTML transferred:       277160000 bytes
Requests per second:    225.56 [#/sec] (mean)
Time per request:       443.334 [ms] (mean)
Time per request:       4.433 [ms] (mean, across all concurrent requests)
Transfer rate:          6222.96 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        1    1  10.0      1     998
Processing:    22  440 126.1    418    1634
Waiting:       21  439 126.1    417    1632
Total:         27  441 126.5    419    1635

As you can see the performance is a bit slower (8.31%) 443ms with a total test running time of 44 seconds. But the result is still comparable.

Here is a bit of the nginx config:


location ~ .php$ {
        fastcgi_split_path_info ^(.+\.php)(.*)$;
        fastcgi_pass   127.0.0.1:9000;
        fastcgi_index  index.php;
        include /etc/nginx/fastcgi_params;
        fastcgi_param  SCRIPT_FILENAME /var/www/application/www$fastcgi_script_name;
        fastcgi_param  QUERY_STRING     $query_string;
        fastcgi_param  REQUEST_METHOD   $request_method;
        fastcgi_param  CONTENT_TYPE     $content_type;

HHVM over a Unix socket

Now I need to rerun the above two tests with HHVM. I shut down PHP5-FPM and update my nginx config to connect to HHVM that is running on a Unix socket

Here is the command to run HHVM:


sudo hhvm --mode server -vServer.Type=fastcgi -vServer.FileSocket=/var/run/hhvm/hhvm.sock

Here are the results:


Document Path:          /product/view/1561
Document Length:        27716 bytes

Concurrency Level:      100
Time taken for tests:   100.118 seconds
Complete requests:      10000
Failed requests:        6465
   (Connect: 0, Receive: 0, Length: 6465, Exceptions: 0)
Write errors:           0
Non-2xx responses:      6465
Total transferred:      111660110 bytes
HTML transferred:       105559505 bytes
Requests per second:    99.88 [#/sec] (mean)
Time per request:       1001.184 [ms] (mean)
Time per request:       10.012 [ms] (mean, across all concurrent requests)
Transfer rate:          1089.14 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        1    1   0.5      1       7
Processing:   109  998 260.2    900    4391
Waiting:      108  998 259.8    900    4391
Total:        115  999 260.3    901    4395

This mean request time is about twice as much as PHP5-FPM over TCP. The total test time is 100 seconds.

Here is a bit of the nginx config:


location ~ .php$ {
        fastcgi_split_path_info ^(.+\.php)(.*)$;
        fastcgi_pass unix:/var/run/hhvm/hhvm.sock;
        fastcgi_index  index.php;
        include /etc/nginx/fastcgi_params;
        fastcgi_param  SCRIPT_FILENAME /var/www/application/www$fastcgi_script_name;
        fastcgi_param  QUERY_STRING     $query_string;
        fastcgi_param  REQUEST_METHOD   $request_method;
        fastcgi_param  CONTENT_TYPE     $content_type;

HHVM over a TCP socket

The next test is to run HHVM over a TCP socket and see if it produces different results.


sudo hhvm --mode server -vServer.Type=fastcgi -vServer.Port=9000 

I rerun the test. Here are the results:


Document Path:          /product/view/1561
Document Length:        27716 bytes

Concurrency Level:      100
Time taken for tests:   125.841 seconds
Complete requests:      10000
Failed requests:        176
   (Connect: 0, Receive: 0, Length: 176, Exceptions: 0)
Write errors:           0
Non-2xx responses:      176
Total transferred:      278607904 bytes
HTML transferred:       272488432 bytes
Requests per second:    79.47 [#/sec] (mean)
Time per request:       1258.414 [ms] (mean)
Time per request:       12.584 [ms] (mean, across all concurrent requests)
Transfer rate:          2162.07 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        1    1   0.4      1      11
Processing:   242 1254 199.0   1260    3007
Waiting:      240 1252 199.0   1259    3007
Total:        247 1255 199.0   1261    3008

As expected the performance is slower but by larger amount than PHP5-FPM over a unix socket vs PHP5-FPM over a TCP socket. HHVM over a TCP socket is about 25.7% slower than HHVM over a unix socket.

Here is a bit of the nginx config:


location ~ .php$ {
        fastcgi_split_path_info ^(.+\.php)(.*)$;
        fastcgi_pass   127.0.0.1:9000;
        fastcgi_index  index.php;
        include /etc/nginx/fastcgi_params;
        fastcgi_param  SCRIPT_FILENAME /var/www/application/www$fastcgi_script_name;
        fastcgi_param  QUERY_STRING     $query_string;
        fastcgi_param  REQUEST_METHOD   $request_method;
        fastcgi_param  CONTENT_TYPE     $content_type;

HHVM with a larger CoreFileSize

At this point I am want to see if adjusting some of the HHVM settings will produce a different result. HHVM offers many runtime options. I choose to tweak a ResourceLimit called CoreFileSize by setting it to ~100mb.


sudo hhvm --mode server -vServer.Type=fastcgi -vServer.Port=9000 -vResourceLimiCoreFileSize=100000000

I rerun the test. Here are the results:


Document Path:          /product/view/1561
Document Length:        27716 bytes

Concurrency Level:      100
Time taken for tests:   128.016 seconds
Complete requests:      10000
Failed requests:        29
   (Connect: 0, Receive: 0, Length: 29, Exceptions: 0)
Write errors:           0
Non-2xx responses:      29
Total transferred:      282510166 bytes
HTML transferred:       276390253 bytes
Requests per second:    78.11 [#/sec] (mean)
Time per request:       1280.165 [ms] (mean)
Time per request:       12.802 [ms] (mean, across all concurrent requests)
Transfer rate:          2155.10 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        1    1   0.4      1       6
Processing:   191 1275 198.6   1282    2099
Waiting:      189 1274 198.6   1280    2098
Total:        195 1276 198.6   1283    2100

As you can see it does not process more requests. It actually does worse. I try other values (1000,10000) for the CoreFileSize and the results do not get any better. Before I go down the rabbit hole of settings-tweaking I decide to stop and take a step back. May be I am missing something.

Summary

Based on the above tests HHVM does not seem to out perform PHP5-FPM. As of now, here are the results.

  • HHVM out of the box build (master: 5b00837d9b2ecc28e6225336296be88449637b2e) over a TCP socket takes 2.83 times the amount of time it takes PHP5-FPM over a TCP socket to serve the same number of request.
  • PHP5-FPM over a TCP socket takes 1.08 times the amount of time it takes PHP5-FPM over a unix socket to serve the same number of requests.
  • HHVM over a TCP socket takes 1.26 times the amount of time it takes HHVM over a unix socket to serve the same requests.
  • Increasing the CoreFileSize for HHVM to 100Mb does not improve it’s performance.

Please read post #2 to see why the conclusions drawn in this article are incorrect.

Next Steps

The next steps are to benchmark the code and see exactly where HHVM is slower than PHP5-FPM. Once again your feedback is always welcome.

Appendix

Here is the code that is run when that url is visited

public function view_get()
 	{
 		$this->load->library("form_validation");
 		$this->form_validation->set_rules('product_id', 'product_id', 'required|integer|xss_clean');
 		if ($this->form_validation->run() == FALSE)
		{
			show_error(_("Invalid input provided."), 400);
		}

 		$fields = $this->form_validation->get_fields();

        
        $product_is_watched_by_user = $this->user_product_model->is_product_watched_by_user($this->sess->id, $product_id);

        //get detailed product info
		if($product_is_watched_by_user) {
			$product_details = $this->user_product_model->get_product_for_user($this->sess->id, $product_id);
		}
		else
		{
 			$product_details = $this->product_model->load($fields['product_id'], "*"); 			
		}

		//calculate price delta and appropriate color
		$product_details->price_delta = round((($product_details->new_price - $product_details->original_price) / $product_details->original_price) * 100, 2);  
		$product_details->price_color = $product_details->price_delta >= 0 ? 'redFont' : 'greenFont';
		
		$product_details->url_domain = parse_url($product_details->url, PHP_URL_HOST);

		$product_created_date = new DateTime($product_details->created_on);
		$product_details->created_on = $product_created_date->format('M j, Y');

 		$first_item = array('previous_price' => $product_details->original_price, 'created_on' => $product_details->created_on);
 		
		$product_history_array = array();
 		$product_history_array[] = $this->get_display_value_for_product_history($first_item);
 		$product_history = $this->model->get_price_change_history_for_product($fields['product_id']);
 		foreach($product_history as $history_item)
 		{
 			$product_history_array[] = $this->get_display_value_for_product_history($history_item);
 		}

 		$latest_item = array('previous_price' => $product_details->new_price, 'created_on' => $product_details->updated_on);
 		$product_history_array[] = $this->get_display_value_for_product_history($latest_item);
 		$product_history_array = $this->data_interpolate($product_history_array);

 		$result = array("product_history" => $product_history_array, "product" => $product_details, "product_is_watched_by_user" => $product_is_watched_by_user);
 		$result = $this->escape_values_for_ui($result);

 		
		$this->load->view('chart_viewer', $result);
		
 	}	
  • Sebastien Barre

    Very interesting post, thanks for sharing all your setup and configuration data!

    • Prashant

      I have revised my results in the next blog post, give it a read and let me know what you think.

  • Matthew Glinski

    EC2 Micro Instances are not suitable for performance bench-marking or comparisons like this. They are subject to heavy preformance fluctuations. Try running these tests again on a c3.large instance, where CPU preformance is much more stable and a core feature of the instance your testing on.

    • Prashant

      @matthewglinski:disqus I will make sure to do that and post my results.

      • http://www.stephen3.com/ Stephen Smith

        Results? This is the first time I have seen HHVM not performing better than PHP, and it would indicate a huge waste of time on Facebook’s part. I have personally seen very consistent, low response times as a result of HHVM, and I will probably be migrating most of my stuff to a “LAMH” stack now that mysqli is supported.

        • Prashant

          Hi I have revised my results in my next blog post. Give it a read and let me know what you think