Archive for the ‘php’ Category

之前使用的加速器是eAccelerator,新公司使用的是APC,转载文章一篇。

zz from:http://devzone.zend.com/1812/using-apc-with-php/

Using APC with PHP

Cache Cow

If you’ve been around PHP for a while, you’ve probably heard about APC, the Alternative PHP Cache. APC is an opcode cache that can significantly speed up your PHP applications, by caching both PHP code and user variables. Adding APC to an application usually results in improved application response times, reduced server load and happier users.

In this article, I’ll introduce you to APC, guiding you through the process of installing and configuring it and showing you a few examples of how it works. I’ll also walk you through the APC administrator interface, which lets you view APC performance in real time, and show you how you can use it with the Zend Framework. So come on in, and let’s get started!

Getting Started

First up, a quick description of what APC is and how it works.

As you probably already know, PHP is an interpreted language (unlike, say, Java or C++). Whenever a client requests a PHP page, the server will read in the source code of the page, compile it into bytecode and then execute it. In the normal scenario, this process is performed on every request…although PHP is so speedy that you probably won’t even notice it!

If you’re running an application or Web site that has many hundreds or thousands of requests coming in per minute, though, you’re going to want to speed things up as much as possible. And that’s where APC comes in. In the words of its Web site, APC is “a free, open, and robust framework for caching and optimizing PHP intermediate code.” Very simply, APC caches the compiled output of each PHP script run and reuses it for subsequent requests. This reduces the time and processing cycles needed to fully satisfy each request, leading to better performance and lower response times.

Does it work? You bet (there are some benchmarks at the end of the article). And it’s easy to set up as well. To install it, use the pecl command, as shown below:

shell> pecl install apc-3.1.4

The PECL installer will now download the source code, compile it and install it to the appropriate location on your system.

Alternatively, manually download the source code archive (v3.1.4 at this time) and compile it into a loadable PHP module with phpize:

shell> cd apc-3.1.4
shell# phpize
shell# ./configure
shell# make
shell# make install

This procedure should create a loadable PHP module named apc.so in your PHP extension directory. You should now enable the extension in the php.ini configuration file, restart your Web server, and check that the extension is enabled with a quick call to phpinfo():

Windows users have a much easier time of it; pre-compiled Windows versions of php_apc.dll can be downloaded from here. Once you’ve got the file, place it in your PHP extension directory, activate it via your php.ini configuration file, and restart your Web server. You should now be able to see the extension active with phpinfo(), as described above.

Digging Deeper

The APC source code archive includes a script named apc.php. This script serves as the administrator interface for the cache, allowing you to look inside the cache at any time to view usage or inspect cached variables. It’s a good idea to get familiar with how this works before starting to write any code.

To begin, extract the script from the source code archive and copy it to your Web server document root. Then, open it in a text editor and set the administrator password (you’ll find it near the top of the script). Once you’ve got that done, try accessing the script through your Web browser. You should see something like this:

As you can see, the script provides a birds-eye view of the current cache status, displaying general cache information, usage and statistics on hits and misses. The information is presented for both the system cache (which handles opcodes) and the user cache (which handles user variables). You’ll also see some interesting data derived from these statistics, such as the number of cache requests per second, the hit rate and miss rate.

This information is useful to understand how well your cache is working, and identify areas that are under-optimized. In particular, note the cache full count value, which indicates how often the cache has filled up; if this is a high number, it’s an indication of high cache churn and suggests that you should perhaps assign more memory to the cache.

The “System Cache Entries” tab lists the PHP scripts that are currently being cached, together with their filename, size and number of hits. Note that APC will automatically cache script opcode.

The “User Cache Entries” tab lists user variables that have been stored in the cache, together with their identifier, size, and creation and modification times. You can select any of these entries to look inside the cache entry and see what it contains. Note that user cache entries must be manually created by a PHP script – you’ll see how on the next page.

Remember that you can clear the opcode cache or the user cache at any time with the “Clear cache” buttons at the top right corner of the page.

A Galaxy Far, Far Away

Now that you have a better idea of how APC works, let’s write some code. Caching user variables in APC is mainly done through the apc_add() and apc_fetch() methods, which allow variable to added to, and retrieved from, the cache respectively. Here’s a simple example that illustrates:

<?php
if ($quote = apc_fetch('starwars')) {
  echo $quote;
  echo " [cached]";
} else {
  $quote = "Do, or do not. There is no try. -- Yoda, Star Wars";
  echo $quote;
  apc_add('starwars', $quote, 120);
}
?>

Now, the first time, you run this script, the page will be generated from scratch and you’ll see something like this:

Next, try refreshing the page. This time, the output will be retrieved from the cache:

The business logic to use a cache is fairly simple. The first step is to check if the required data already exists in the cache. If it doesn’t, it should be generated from the original data source, and a copy saved to the cache for future use. If it does, you can use it right away – write it to a file, pipe it to an external program or output it to the client.

In the previous example, checking whether the data already exists in the cache is accomplished with the apc_fetch() method, while writing a fresh snapshot to the cache is done with the apc_add() method. Notice that both methods require an ID; this ID uniquely identifies the object in the cache and serves as a key for saving and retrieving cache data. The apc_add()method additionally lets you specify a duration (in seconds) for which the cache is valid.

Take a look in the administrator interface, and you should see your cached data, together with statistics on cache hits and misses:

A-rray of Sunshine

In addition to caching strings, APC also allows you to cache arrays, objects, functions and references. Consider the following example, which caches an array of values:

<?php
// if available, use cached array
// if not, create and cache array
if ($data = apc_fetch('heroes')) {
  echo 'Cached data: ';
} else {
  $data = array('Spiderman', 'Superman', 'Green Lantern', 'Human Torch');
  apc_add('heroes', $data, 120);
}
echo $data[1];  // Superman
?>

You can also cache nested or multi-dimensional arrays, as shown below:
<?php
// if available, use cached data
// if not, create and cache nested array
if ($data = apc_fetch('config')) {
  echo 'Cached data: ';
} else {
  $data = array(
    'site1' => array(
      'smtp' => array(
        'host' => '192.168.0.1',
        'user' => 'user1',
        'pass' => 'guessme'
      )
    ),
    'site2' => array(
      'smtp' => array(
        'host' => '192.168.10.10',
        'user' => 'user2',
        'pass' => 's3cr3t'
      )
    ),
  );
  apc_add('config', $data, 120);
}

// display data
echo $data['site2']['smtp']['pass']; // s3cr3t
?>

An Object Lesson

In addition to caching arrays, APC also allows you to store objects in the cache. To illustrate, consider the next example, which initializes a simple User object, stores it in the cache, and then retrieves it back from the cache:

<?php
// define class
class User {

  private $name;
  private $location;

  function setName($value) {
    $this->name = $value;
  }

  function setLocation($value) {
    $this->location = $value;
  }

  function getName() {
    return $this->name;
  }

  function getLocation() {
    return $this->location;
  }
}

// if cached object available, use cached object
// if not, create new object instance and cache it
if (apc_exists('user')) {
  $obj = apc_fetch('user');
  echo "Cached data: ";
} else {
  $obj = new User;
  $obj->setName('John');
  $obj->setLocation('London');
  apc_add('user', $obj, 120);
}

// print object properties
echo 'My name is ' . $obj->getName() . ' and I live in ' . $obj->getLocation();
?>

Here’s what the output looks like:

Another approach to arrive at the same result is to serialize the object into a string, and then store the string in the cache instead of the object. Here’s what that would look like:

<?php
// define class
class User {

  private $name;
  private $location;

  function setName($value) {
    $this->name = $value;
  }

  function setLocation($value) {
    $this->location = $value;
  }

  function getName() {
    return $this->name;
  }

  function getLocation() {
    return $this->location;
  }
}

// serialize object and cache
// retrieve from cache as needed and deserialize
if ($str = apc_fetch('user')) {
  $obj = unserialize($str);
  echo "Cached data: ";
} else {
  $obj = new User;
  $obj->setName('John');
  $obj->setLocation('London');
  $str = serialize($obj);
  apc_add('user', $str, 120);
}
echo 'My name is ' . $obj->getName() . ' and I live in ' . $obj->getLocation();
?>

Getting Closure

You can also use APC to cache references and (with a little tweaking) anonymous functions. Let’s take a look:

<?php
class Calendar {
  public $year = 2001;
  public function &getYear() {
    return $this->year;
  }
}

$obj = new Calendar;
$a = &$obj->getYear();  // 2001
$obj->year = 2010;      // 2010
apc_add('ref', $a, 60);
$ref = apc_fetch('ref');
$ref++;
echo $ref;              // 2011
?>

Anonymous functions or closures, new in PHP 5.3, offer an easy way to define functions “on the fly”. By default, however, closures cannot be cached with APC, as the Closure class does not implement serialization. Here’s a simple example that illustrates the problem:
<?php
// check if closure exists in cache
// if yes, retrieve and use
// if not, define and add to cache
if (!apc_exists('area')) {
  // simple closure
  // calculates area from length and width
  $area = function($length, $width) {
      return $length * $width;
  };
  apc_store('area', $area);
  echo 'Added closure to cache.';
} else {
  $func = apc_fetch('area');
  echo 'Retrieved closure from cache. ';
  echo 'The area of a 6x5 polygon is: ' . $func(6,5);
}
?>

When you try accessing this script, you’ll see an error about serialization, as shown below:

What’s the solution? Well, Jeremy Lindblom has extended the Closure class and created a custom SuperClosure class that supports both serialization and reflection. If you implement your closure using this class, you will be able to cache it. Here’s a revision of the previous script that demonstrates:

<?php
// include SuperClosure class
include 'SuperClosure.class.php';

// check if closure exists in cache
// if yes, retrieve and use
// if not, define and add to cache
if (!apc_exists('area')) {
  // simple closure
  // calculates area given length and width
  $area = new SuperClosure(
    function($length, $width) {
      return $length * $width;
    }
  );
  apc_store('area', $area);
  echo 'Added closure to cache.';
} else {
  $func = apc_fetch('area');
  echo 'Retrieved closure from cache. ';
  echo 'The area of a 6x5 polygon is: ' . $func(6,5);
}
?>

Here’s what the output looks like:

Note that these examples use apc_store() instead of apc_add(). In most cases, you can use these two functions interchangeably. The primary difference lies in how they behave when you attempt to store a value using an identifier that already exists in the cache: apc_add() will return false, while apc_store() will overwrite the previous value with the new value.

You can download the SuperClosure class definition from Jeremy Lindblom’s Github account.

Utility Belt

The APC extension also comes with a few other methods of note. For example, there’s theapc_inc() and apc_dec() methods, which can be used to increment or decrement cached values respectively:

<?php
// store a value
apc_store('a', 20);

// increment and decrement stored value
apc_inc('a');         // 21
apc_inc('a');         // 22
apc_inc('a');         // 23
apc_dec('a');         // 22

// retrieve final value
echo apc_fetch('a');  // 22
?>

The apc_bin_dump() method dumps the current contents of the cache in binary form, while theapc_bin_load() method loads a binary dump into the cache. Consider the following example, which illustrates:
<?php
// clear cache
apc_clear_cache();
apc_clear_cache('user');

// store some values
apc_store('a', 20001);
apc_store('b', 7494);

// dump cache in binary form
$dump = apc_bin_dump();

// clear cache
apc_clear_cache();
apc_clear_cache('user');

// try accessing a stored value
if (apc_fetch('a')) {
  echo apc_fetch('a');
} else {
  echo 'Nothing in cache';
}
// reload cache from binary dump
apc_bin_load($dump);

// try accessing a stored value
if (apc_fetch('a')) {
  echo apc_fetch('a');     // 20001
} else {
  echo 'Nothing in cache';
}
?>

The apc_clear_cache() method can be used to clear the opcode cache or the user cache:
<?php
// clear opcode cache
apc_clear_cache();

// clear user cache
apc_clear_cache('user');
?>

The apc_cache_info() method presents information on current cache status and memory allocation:
<?php
print_r(apc_cache_info());
?>

Here’s what the output looks like:

Tweet Tweet

With all this background information at hand, let’s try APC with a real-world example. This next script uses APC to cache the result of a Twitter search:

<html>
  <head>
    <style type="text/css">
      div.outer {
      	border-bottom: dashed orange 1px;
      	padding: 4px;
      	clear: both;
      	height: 50px;
      }
      div.img {
        float:left;
        padding-right: 2px;
      }
      span.attrib {
        font-style: italic;
      }
    </style>
  </head>
  <body>
    <h2>Twitter Search</h2>
    <form action="<?php echo htmlentities($_SERVER['PHP_SELF']); ?>" method="post">
    Search term: <input type="text" name="q" />
    <input type="submit" name="submit" />
    </form>
<?php
    // if form submitted
    if (isset($_POST['submit'])):
      // sanitize query terms
      $q = strip_tags($_POST['q']);

      // generate cache id from query term
      $id = md5($q);

      // check if this search already exists in cache
      // use if yes, generate fresh results and add to cache if no
      if (apc_exists($id)) {
        $records = apc_fetch($id);
      } else {
        // search Twitter for query term
        $result = simplexml_load_file("http://search.twitter.com/search.atom?q=$q&lang=en");

        // process Atom feed of search results
        $records = array();
        foreach ($result->entry as $entry) {
          $item['image'] = (string)$entry->link[1]['href'];
          $item['owner'] = (string)$entry->author->name;
          $item['uri'] = (string)$entry->author->uri;
          $item['tweet'] = (string)$entry->content;
          $item['time'] = date('d M Y, h:i', strtotime($entry->published));
          $records[] = $item;
        }

        // cache for 5 minutes
        apc_store($id, $records, 300);
      }    

      // display search results
?>
    <h2>Twitter Search Results for '<?php echo $q; ?>'</h2>
      <?php foreach ($records as $r): ?>
      <div>
        <div><img width=48" height="48" src="<?php echo $r['image']; ?>" /></div>
        <div><?php echo $r['tweet']; ?><br/>
        <span>By <a href="<?php echo $r['uri']; ?>"><?php echo $r['owner']; ?></a>
        on <?php echo $r['time']; ?></span></div>
      </div>
      <?php endforeach; ?>
    <?php endif; ?>
  </body>
</html>

Despite its length, this is actually a very simple script. It begins by creating a search form for the user to enter search terms into. Once this form is submitted, it connects to the Twitter Search API, retrieves an Atom-formatted list of search results matching the search term, process the Atom feed and render the final output as an HTML table. The results of the search are cached for five minutes, so that they can be used for subsequent searches containing the same search terms. Notice that each search query is assigned a unique identifier in the APC cache, by using its MD5 signature as key.

You will realize that there are two levels of caching in this script. First, APC’s opcode cache is automatically caching the compiled bytecode of the script, and using this cached bytecode for subsequent requests instead of recompiling it anew. Second, APC’s user cache is caching the results of each Twitter search, and reusing these results (instead of connecting to Twitter afresh) for subsequent searches containing the same query terms. As a result, subsequent searches for the same term will be served from the cache, leading to a noticeable reduction in load time (try it for yourself and see).

Here’s an example of what the output looks like:

In The Frame

If you’re a fan of the Zend Framework, you’ll be happy to hear that Zend_Cache comes with built-in support for APC, allowing you to begin using it out of the box. To illustrate, consider the following Zend Framework controller, which revises the previous example into a Zend Framework controller:

<?php
class IndexController extends Zend_Controller_Action
{
    public function indexAction()
    {
        // action body
    }

    public function searchAction()
    {
      // initialize cache
      $cache = Zend_Cache::factory( 'Core',
                                    'APC',
                                    array('lifeTime' => 300, 'automatic_serialization' => true));

      // create form and attach to view
      $form = new SearchForm();
      $this->view->form = $form;       

      // validate input
      if ($this->getRequest()->isPost()) {
        if ($form->isValid($this->getRequest()->getPost())) {
          // get sanitized input
          $values = $form->getValues();        

          // calculate MD5 hash
          $id = md5($values['q']);

          // look for records in cache
          if (!$records = $cache->load($id) ){
            // if not present in cache
            // search Twitter for query term
            $result = simplexml_load_file("http://search.twitter.com/search.atom?q=" . $values['q'] . "&lang=en");

            // process Atom feed of search results
            $records = array();
            foreach ($result->entry as $entry) {
              $item['image'] = (string)$entry->link[1]['href'];
              $item['owner'] = (string)$entry->author->name;
              $item['uri'] = (string)$entry->author->uri;
              $item['tweet'] = (string)$entry->content;
              $item['time'] = date('d M Y, h:i', strtotime($entry->published));
              $records[] = $item;
            }

            // save to cache
            $cache->save($records, $id);
          }

          // assign results to view
          $this->view->records = $records;
          $this->view->q = $values['q'];
        }
      }
    }
}

// search form
class SearchForm extends Zend_Form
{
  public function init()
  {
    // initialize form
    $this->setMethod('post');

    // create text input for search term
    $q = new Zend_Form_Element_Text('q');
    $q->setLabel('Search term:')
         ->setOptions(array('size' => '35'))
         ->setRequired(true)
         ->addValidator('NotEmpty', true)
         ->addFilter('HTMLEntities')
         ->addFilter('StringTrim');            

    // create submit button
    $submit = new Zend_Form_Element_Submit('submit');
    $submit->setLabel('Search');

    // attach elements to form
    $this->addElement($q)
         ->addElement($submit);
  }
}

Here, the searchAction() method first sets up the Zend_Cache instance, with the Core frontend and the APC backend. The form object, which extends Zend_Form, is then added to the view, together with all necessary validators and filters, and the view is rendered.

When the user submits the form, control transfers back to the action controller, which checks the input and retrieves the filtered values. It then checks the cache to see if a search result already exists for this search term, and uses it if available; if not, it connects to the Twitter Search API, retrieves a result set, and saves it to the cache for future use. The results are then rendered through the view script. On subsequent searches for the same term, the cached result set will be used, producing a much faster response.

Here’s the code for the view script:

<style type="text/css">
  div.outer {
  	border-bottom: dashed orange 1px;
  	padding: 4px;
  	clear: both;
  	height: 50px;
  }
  div.img {
    float:left;
    padding-right: 2px;
  }
  span.attrib {
    font-style: italic;
  }
</style>
<h2>Twitter Search</h2
<?php echo $this->form; ?>

<?php if ($this->records): ?>
  <h2>Twitter Search Results for '<?php echo $this->q; ?>'</h2>
  <?php foreach ($this->records as $r): ?>
  <div>
    <div><img width=48" height="48" src="<?php echo $r['image']; ?>" /></div>
    <div><?php echo $r['tweet']; ?><br/>
    <span>By <a href="<?php echo $r['uri']; ?>"><?php echo $r['owner']; ?></a>
    on <?php echo $r['time']; ?></span></div>
  </div>
  <?php endforeach; ?>
<?php endif; ?>

And here’s a sample of the output:

The Need For Speed

At this point, there’s only one question left to answer: does APC’s opcode caching really deliver the goods and produce a verifiable increase in performance?

A good way to test this is by benchmarking a PHP script with and without APC, and evaluating the performance differential if any. ApacheBench (ab) is my tool of choice for this test, and my testbed will be the default welcome page of a new Zend Framework project. You can create this by installing the Zend Framework and then using the zf command-line tool to initialize a new, empty project, like this:

shell> zf create project example

Now, turn off APC, by disabling the extension in your php.ini configuration file and restarting the Web server. Then, use ab to benchmark the application welcome page by sending it 1000 requests with a concurrency level of 5, as follows:
shell> ab -n 1000 -c 5 http://example.localhost/default/index/index

On my development system, this produces output like the following:

The main numbers to look at here are the requests per second and the average time per request. The lower the average time per request, the better the performance. Similarly, the greater the number of requests served, the better the performance.

Next, re-enable APC, restart the Web server and try the test again. On my development system, this produces output like the following:

As you can see, enabling APC has resulted in an almost 185% increase in performance, with the server now being able to handle 71 requests per second (up from 25 earlier) and the average time per request coming down to 69 ms (from 194 ms earlier).

The above test was run with APC’s default settings. However, APC comes with a number of configuration settings that you can tweak further to squeeze even better performance from it. Here are some of the important ones:

  • ‘apc.shm_size’ controls the size of the APC memory cache;
  • ‘apc.stat’ controls whether APC checks each script to see if it has been modified and needs to be recompiled and recached;
  • ‘apc.optimization’ determines the degree of optimization to apply;
  • ‘apc.filters’ specifies which files should be cached;
  • ‘apc.write_lock’ places an exclusive lock for caching compiled script bytecode;
  • ‘apc.lazy_functions’ and ‘apc.lazy_classes’ enables lazy loading for functions and classes.

You can read more about these and other configuration directives here.

That’s about all I have for the moment. I hope this tutorial has given you some insight into how APC works, and how you can use it to improve the performance of your PHP applications. Try it out the next time you have a performance optimization problem, and see what you think!

有同事分享了array_merge与+的区别,以前也遇到过,备忘如下:array_merge和+,遇到相同key的elem时,array_merge是覆盖,而+是抛弃后面的值。

<?php

$arr1 = array(‘a’ => 1);
$arr2 = array(‘a’ => 2);

var_dump(array_merge($arr1, $arr2));
var_dump($arr1 + $arr2);

[chengy@jx-uc-rd00.jx.baidu.com test]$ ./test.php
array(1) {
[“a”]=>
int(2)
}
array(1) {
[“a”]=>
int(1)
}

nginx能够通过upstream的方式,把请求分配到不同的phpcgi服务上。

#phpcgi/etc/php-fpm.conf
listen = 10.241.133.144:9000   # 本地ip,非127.0.0.1
listen.allowed_clients = 10.241.133.137 # nginx server ip,非127.0.0.1

#nginx/conf/nginx.conf
upstream multiphp{
  server 10.241.133.144:9000 weight=1;
  server 127.0.0.1:9000 weight=1;
}
server{
  ……
  location ~ .*\.php{
    fastcgi_pass  multiphp;
    #fastcgi_pass  127.0.0.1:9000;
    fastcgi_index index.php;
    fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;
    include fastcgi.conf;
  }
}

taint是大牛风雪之隅写的一个php扩展,用来检测GET、POST、COOKIE等中可能有的漏洞。其中用了很多php扩展开发必须的知识,简单记录代码逻辑如下。

首先,看到代码最末端的PHP_MINIT_FUNCTION,该方法在每个php进程初始化时执行,如果需要的话,则加载taint扩展模块,包括注册新函数,以及override旧函数,这里主要是一些与字符串有关的函数。前者使用zend_set_user_opcode_handler方法,设置echo、include等的钩子函数。

同样需要注意的是PHP_RINIT_FUNCTION,该方法在每个web请求开始时执行。所以,如果没有sapi支持的话,就不必要检查了。对于POST、GET、COOKIE数组中的每个值,标记它为possible tainted。这样,如果后续有使用到这些值,即op的两端有这些值的话,result也就是possible tainted了。

RINIT里使用php_taint_mark_strings方法标识tainted字符串,在其末尾追加一个unsigned字符0x6A8FCE84作为标识符。在该函数开始,会先检查是否有嵌套:

|     if (++ht->nApplyCount > 1) {
|         ht->nApplyCount–;
|         return;
|     }

不过我不是很明白GET、POST、COOKIE在什么情况下会有嵌套存在。

打标记,由于这里仅改变了Z_STRVAL_PP(ppzval)的内存大小和内容,但是并没有改变Z_STRLEN_PP(ppzval)。所以之后常规方法是用ppzval的时候,不会读取到打上的标记值:

| Z_STRVAL_PP(ppzval) = erealloc(Z_STRVAL_PP(ppzval), Z_STRLEN_PP(ppzval) + 1 + PHP_TAINT_MAGIC_LENGTH);
| PHP_TAINT_MARK(*ppzval, PHP_TAINT_MAGIC_POSSIBLE);

以echo的钩子函数为例,说明单操作符。
static int php_taint_echo_handler(ZEND_OPCODE_HANDLER_ARGS) /* {{{ */ {
    zend_op *opline = execute_data->opline;
        zval *op1 = NULL;
        zend_free_op free_op1;

        switch(TAINT_OP1_TYPE(opline)) {
                case IS_TMP_VAR:
                        op1 = php_taint_get_zval_ptr_tmp(TAINT_OP1_NODE_PTR(opline), execute_data->Ts, &free_op1 TSRMLS_CC);
                        break;
                case IS_VAR:
                        op1 = TAINT_T(TAINT_OP1_VAR(opline)).var.ptr;
                        break;
                case IS_CV: {
                                zval **t = TAINT_CV_OF(TAINT_OP1_VAR(opline));
                                if (t && *t) {
                                        op1 = *t;
                                } else if (EG(active_symbol_table)) {
                                        zend_compiled_variable *cv = &TAINT_CV_DEF_OF(TAINT_OP1_VAR(opline));
                                        if (zend_hash_quick_find(EG(active_symbol_table), cv->name, cv->name_len + 1, cv->hash_value, (void **)&t) == SUCCESS) {
                                                op1 = *t;
                                        }
                                }
                    }
                        break;
        }

        if (op1 && IS_STRING == Z_TYPE_P(op1) && PHP_TAINT_POSSIBLE(op1)) {
                if (ZEND_ECHO == opline->opcode) {
                        php_taint_error("function.echo" TSRMLS_CC, "Attempt to echo a string that might be tainted");
                } else {
                        php_taint_error("function.echo" TSRMLS_CC, "Attempt to print a string that might be tainted");
                }
        }

        return ZEND_USER_OPCODE_DISPATCH;
} /* }}} */

这里就是检查echo的内容,是否在之前被打伤tainted标记。这里区分不同类型的变量,获取其值。摘录网上的说明如下:

IS_CV:这种类型的操作数比较重要,此类型是在PHP后来的版本中(大概5.1)中才出现,CV的意思是compiled variable,即编译后的变量,变量都是保存在一个符号表中,这个符号表是一个哈希表,试想如果每次读写变量的时候都需要到哈希表中去检索,势必会对效率有一定的影响,因此在执行上下文环境中,会将一些编译期间生成的变量缓存起来,此过程以后再详细介绍。此类型操作数一般以!开头表示,比如变量$a=123;$b=”hello”这段代码,$a和$b对应的操作数可能就是!0和!1, 0和1相当于一个索引号,通过索引号从缓存中取得相应的值。

IS_TMP_VAR:表示临时变量,临时变量一般在前面加~来表示,这是一些OP执行过程中需要用到的中间变量,例如初始化一个数组的时候,就需要一个临时变量来暂时存储数组zval,然后将数组赋值给变量。

IS_VAR: 一般意义上的变量,以$开发表示。

IS_TMP_VAR, 顾名思义,这个是一个临时变量,保存一些op_array的结果,以便接下来的op_array使用,这种的操作数的u保存着一个指向变量表的一个句柄(整数),这种操作数一般用~开头,比如~0,表示变量表的0号未知的临时变量

IS_VAR 这种就是我们一般意义上的变量了,他们以$开头表示

IS_CV 表示ZE2.1/PHP5.1以后的编译器使用的一种cache机制,这种变量保存着被它引用的变量的地址,当一个变量第一次被引用的时候,就会被CV起来,以后对这个变量的引用就不需要再次去查找active符号表了,CV变量以!开头表示。

这样,逻辑就很清晰了!最后,还有一个小疑问,mark taint时分配的那些unsigned字节,会在什么时候释放呢?由于这些空间是用erealloc分配的,初步猜测是在这个php进程结束时被释放。

本文参考了:http://www.grepmymind.com/2008/01/11/memcached-php-semaphore-cache-expiration-handling/

问题描述

memcached作为互联网经常使用的缓存工具,大大减少了数据库的压力。但是当并发度较高时,可能会有多个请求都检测到expire,从而发起数据库请求,于是会对数据库造成冲击。

问题重现

当采用传统的设置expire time时,使用httpload工具并发100,在数据库请求函数里sleep 1秒钟。memcache封装里打印是从mc读取还是db读取。expire time设置为2秒。可以看到有重复的数据库请求:

[2012-02-21 10:17:13] NOTICE [1]: mc(127.0.0.1,/test.php)
[2012-02-21 10:17:14] NOTICE [1]: mc(127.0.0.1,/test.php)
[2012-02-21 10:17:14] NOTICE [1]: db(127.0.0.1,/test.php)
[2012-02-21 10:17:14] NOTICE [1]: db(127.0.0.1,/test.php)
[2012-02-21 10:17:15] NOTICE [1]: mc(127.0.0.1,/test.php)
[2012-02-21 10:17:15] NOTICE [1]: mc(127.0.0.1,/test.php)

解决方法

                function call_high($func, $args, $expire=60){/*{{{*/
                        // Calulate key
                        $key = md5(json_encode($func).json_encode($args));

                        // Get from memcached
                        return $this->get_high($key, $func, $args, $expire);

                }/*}}}*/

                // lock_expire should longer than db_func running time
                function get_high($key/* string or array*/, $db_func/*mixed*/, $db_args/*mixed*/, $expire, $lock_expire=10){/*{{{*/
                        if (!$this->mem){
                                return false;
                        }

                        $val = $this->get($key);

                        // if(!$val){ // Maybe empty when FIRST get, otherwise, must something wrong, like the lock&work thread haven't refresh item before expire time. So we use a short lock expire time to allow more thread trying to refresh. }   

                        if ( $val && (!isset($val['t']) || !isset($val['i'])) ){ // don't have expire index? must be wrong. For logic layer to process.
                                #Core_Log::get_instance()->notice("mc, no expire index");
                                return $val;
                        }

                        if ($val && ($val['t'] - $this->get_high_cal_time($expire) > time())){ // not expired
                                #Core_Log::get_instance()->notice("mc");
                                return $val['i'];
                        }

                        // expired, allowed one process to refresh it
                        if (!($this->get_high_lock($key, $lock_expire))){ // other process has refreshing it, so we quit with old data(or may be empty)
                                #Core_Log::get_instance()->notice("other lock, we quit");
                                return $val?$val['i']:false;
                        }

                        #Core_Log::get_instance()->notice("db");
                        // get from user func, and set into memcache
                        $db_info = call_user_func_array($db_func, $db_args); // may be empty, we also set into mc
                        $this->set_high($key, $db_info, MEMCACHE_COMPRESSED, $expire, $this->get_more_expire_time($expire));

                        $this->free_high_lock($key);
                        #Core_Log::get_instance()->notice("save and free lock");

                        return $db_info;
                }

                // These two functions should guarantee the refresh thread can do all work before realy expired.
                // So the work_time < cal_time + more_time                 
                protected function get_high_cal_time($expire){                         
                        return intval(min(10, max(0, $expire/5))); // [0, 10]                 
                }                 
                protected function get_more_expire_time($expire){                          
                        return intval(max(100, $expire*0.1)); // [100, ...)                 
                }                 

                function get_high_lock($key, $expire=100){                         
                        return $this->mem->add($key.self::HIGH_MC_LOCK_TAIL, 1, 0, $expire); // return true when not exists already, otherwise return false, use as mutex
                        //return $this->mem->increment($key.self::HIGH_MC_LOCK_TAIL, 1); //can't, because increment will return false, if elem not exists already
                }

                function free_high_lock($key){
                        return $this->del($key.self::HIGH_MC_LOCK_TAIL);
                        //return $this->mem->decrement ($key.self::HIGH_MC_LOCK_TAIL, 1);
                }

                function set_high($key, $val, $flag, $expire/*second*/, $more_expire=100){
                        if (!$this->mem){
                                return false;
                        }

                        // save expire time
                        $expire_time = time()+$expire;
                        $val = array('t'=>$expire_time, 'i'=>$val);

                        $expire += $more_expire; // need more expire time, because we don't want return null when get_high
                        return $this->set($key, $val, $flag, $expire);
                }/*}}}*/

基本思想就是利用mutex锁,保证仅有一个进程更新该item。

有几个点说明一下:

  • 首次请求时,非refresh线程会返回为空,在业务逻辑上是否可接受?解决方法可以是等待一定时间后再次获取
  • 如果refresh线程工作太久,导致memcached中相应item已经expire了,那么其他线程get from mc即为空,则也会尝试获取lock后从db读取。这里通过设置lock time、cal time和more expire time已解决,允许其他线程也请求db,这种情况一般是数据库阻塞导致的,允许重复读取可能会加大数据库压力,所以存疑。
  • 如果memcached内存不足,导致item expire,则正常情况下也只有一个refresh线程从db读取
  • 如果memcached内存不足,导致lock expire,则会有多个refresh线程从db读取(一般不会设置这么小的memcached吧?)

php的foreach与引用(&)一起,如果使用不当,会引起相当隐蔽的bug。

  $arr = array(1,2,3);
  foreach($arr as &$a){
      echo $a."\t";
  }
  echo "\n";

  foreach($arr as $a){
      echo $a."\t";
  }

输出为:

:!./test.php
1       2       3
1       2       2
而期望输出为:
:!./test.php
1       2       3
1       2       3

可以看到,数组的最后一个元素被修改了!

实际上,是数组的最后一个元素,被N次改写了。具体过程为:

  1. 第一次foreach结束之后,$a与$arr[last]指向同一块地址
  2. 第二次foreach里,每次,$a的内容都被改写!
    1. $arr[2] = $a = $arr[0] = 1
    2. $arr[2] =  $a = $arr[1] = 2
    3. $arr[2] =  $a = $arr[2] = 2

这样就很明显了!通过以下代码可以看出具体过程:

 $arr = array(1,2,3);
  foreach($arr as &$a){
      echo $a."\t";
  }
  echo "\n";

  foreach($arr as $a){
      echo "$a( $arr[2] )\t";
  }

输出为:

:!./test.php
1       2       3
1( 1 )  2( 2 )  2( 2 )

使用php发送邮件很简单,实际利用的是linux自带的smtp服务。这里不加赘述。

邮件服务,我的理解是数据包与包的交互。发送邮件时,smtp服务接收client程序的邮件标题、正文、from、to、优先级等信息,拼装为smtp数据包,网络发送给to地址所在的邮件服务器。接收邮件时,smtp服务从网络上的邮件服务器接收数据包,解析出to的email地址,利用推或拉的方式传递给client。

php中的mail函数,作为一种client形式,在其help里写的很清楚:

reject note 返回值

Returns TRUE if the mail was successfully accepted for delivery, FALSE otherwise.

It is important to note that just because the mail was accepted for delivery, it does NOT mean the mail will actually reach the intended destination.

即mail的返回值仅代表php.ini里配置的smtp server是否接收发送请求成功,而并不代表最终的to email接收成功。

进而推测,smtp对mail请求应当是异步的,那么也许smtp的log中会有邮件接收的真正状态。进入centos系统的/var/log,果然有maillog!

Feb  7 09:21:45 v080106 sendmail[32612]: q171LiwL032612: to=abc@sohu.com, ctladdr=admin (500/500), delay=00:00:01, xdelay=00:00:01, mailer=relay, pri=32760, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (Ok: queued as AC58F1060001)

这是一条发送成功的邮件记录。各字段意义为:

Feb  7 09:21:45 发送时间
v080106 发送者机器名称
sendmail[32612]: 进程号
q171LiwL032612: 唯一Message-ID,可以根据该字段,找到log里与本次发送相关的所有记录
to=abc@sohu.com, 接收者邮箱
ctladdr=admin (500/500)
delay=00:00:01, xdelay=00:00:01, 延时
mailer=relay, 邮件服务器,是relay,则参考下面的relay字段可知具体信息
pri=32760, 优先级
relay=[127.0.0.1] [127.0.0.1], 中继
dsn=2.0.0, smtp code (200表示成功)
stat=Sent (Ok: queued as AC58F1060001) 状态说明,Sent代表发送成功

Feb  8 01:38:48 v080106 postfix/smtpd[11264]: F1E521060007: client=localhost.localdomain[127.0.0.1]

Feb  8 01:38:49 v080106 postfix/cleanup[11267]: F1E521060007: message-id=<201202071738.q17HcmEB011273@v080106.isv>

Feb  8 01:38:49 v080106 postfix/qmgr[18249]: F1E521060007: from=<admin@v080106.isv>, size=3258, nrcpt=1 (queue active)

Feb  8 01:38:49 v080106 sendmail[11273]: q17HcmEB011273: to=abc@vip.sohu.com, ctladdr=admin (500/500), delay=00:00:01, xdelay=00:00:01, mailer=relay, pri=32807, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (Ok: queued as F1E521060007)

Feb  8 01:38:49 v080106 postfix/smtp[11270]: F1E521060007: to=<abc@vip.sohu.com>, relay=mx.vip.sohu.com[61.135.132.86]:25, delay=0.25, delays=0.05/0/0.15/0.05, dsn=4.1.8, status=deferred (host mx.vip.sohu.com[61.135.132.86] said: 450 4.1.8 <admin@v080106.isv>: Sender address rejected: Domain not found (in reply to RCPT TO command))

Feb  8 02:04:59 v080106 postfix/qmgr[18249]: F1E521060007: from=<admin@v080106.isv>, size=3258, nrcpt=1 (queue active)

Feb  8 02:05:00 v080106 postfix/smtp[22089]: F1E521060007: to=<abc@vip.sohu.com>, relay=mx.vip.sohu.com[61.135.132.86]:25, delay=1572, delays=1571/0.85/0.13/0.04, dsn=4.1.8, status=deferred (host mx.vip.sohu.com[61.135.132.86] said: 450 4.1.8 <admin@v080106.isv>: Sender address rejected: Domain not found (in reply to RCPT TO command))

Feb  8 02:38:26 v080106 postfix/qmgr[18249]: F1E521060007: from=<admin@v080106.isv>, size=3258, nrcpt=1 (queue active)

Feb  8 02:38:26 v080106 postfix/smtp[28707]: F1E521060007: to=<abc@vip.sohu.com>, relay=mx.vip.sohu.com[61.135.132.86]:25, delay=3577, delays=3577/0.01/0.14/0.05, dsn=4.1.8, status=deferred (host mx.vip.sohu.com[61.135.132.86] said: 450 4.1.8 <admin@v080106.isv>: Sender address rejected: Domain not found (in reply to RCPT TO command))

Feb  8 03:45:54 v080106 postfix/qmgr[18249]: F1E521060007: from=<admin@v080106.isv>, size=3258, nrcpt=1 (queue active)

Feb  8 03:45:54 v080106 postfix/smtp[10772]: F1E521060007: to=<abc@vip.sohu.com>, relay=mx.vip.sohu.com[61.135.132.86]:25, delay=7625, delays=7625/0.03/0.08/0.05, dsn=4.1.8, status=deferred (host mx.vip.sohu.com[61.135.132.86] said: 450 4.1.8 <admin@v080106.isv>: Sender address rejected: Domain not found (in reply to RCPT TO command))

Feb  8 05:08:20 v080106 postfix/qmgr[18249]: F1E521060007: from=<admin@v080106.isv>, size=3258, nrcpt=1 (queue active)

Feb  8 05:08:30 v080106 postfix/smtp[19312]: F1E521060007: to=<abc@vip.sohu.com>, relay=mx.vip.sohu.com[61.135.132.86]:25, delay=12581, delays=12572/9.5/0.11/0.04, dsn=4.1.8, status=deferred (host mx.vip.sohu.com[61.135.132.86] said: 450 4.1.8 <admin@v080106.isv>: Sender address rejected: Domain not found (in reply to RCPT TO command))

Feb  8 06:31:47 v080106 postfix/qmgr[18249]: F1E521060007: from=<admin@v080106.isv>, size=3258, nrcpt=1 (queue active)

Feb  8 06:31:47 v080106 postfix/smtp[21697]: F1E521060007: to=<abc@vip.sohu.com>, relay=mx.vip.sohu.com[61.135.132.86]:25, delay=17578, delays=17578/0/0.08/0.05, dsn=4.1.8, status=deferred (host mx.vip.sohu.com[61.135.132.86] said: 450 4.1.8 <admin@v080106.isv>: Sender address rejected: Domain not found (in reply to RCPT TO command))

Feb  8 07:55:00 v080106 postfix/qmgr[18249]: F1E521060007: from=<admin@v080106.isv>, size=3258, nrcpt=1 (queue active)

Feb  8 07:55:00 v080106 postfix/smtp[22783]: F1E521060007: to=<abc@vip.sohu.com>, relay=mx.vip.sohu.com[61.135.132.86]:25, delay=22572, delays=22571/0.02/0.11/0.37, dsn=4.1.8, status=deferred (host mx.vip.sohu.com[61.135.132.86] said: 450 4.1.8 <admin@v080106.isv>: Sender address rejected: Domain not found (in reply to RCPT TO command))

Feb  8 09:01:41 v080106 postfix/qmgr[18249]: F1E521060007: from=<admin@v080106.isv>, size=3258, nrcpt=1 (queue active)

Feb  8 09:01:41 v080106 postfix/smtp[23869]: F1E521060007: to=<abc@vip.sohu.com>, relay=mx.vip.sohu.com[61.135.132.86]:25, delay=26572, delays=26572/0/0/0, dsn=4.1.8, status=deferred (host mx.vip.sohu.com[61.135.132.86] said: 450 4.1.8 <admin@v080106.isv>: Sender address rejected: Domain not found (in reply to RCPT TO command))

Feb  8 10:08:21 v080106 postfix/qmgr[18249]: F1E521060007: from=<admin@v080106.isv>, size=3258, nrcpt=1 (queue active)

Feb  8 10:08:21 v080106 postfix/smtp[24922]: F1E521060007: to=<abc@vip.sohu.com>, relay=mx.vip.sohu.com[61.135.132.86]:25, delay=30572, delays=30572/0/0/0, dsn=4.1.8, status=deferred (host mx.vip.sohu.com[61.135.132.86] said: 450 4.1.8 <admin@v080106.isv>: Sender address rejected: Domain not found (in reply to RCPT TO command))

以上是一组失败的记录,唯一号是F1E521060007。可以看到,当接收到失败之后,sendmail进行了多次重试。里面还有一些疑问,之后再解决,现在主要关心错误信息:

relay=mx.vip.sohu.com[61.135.132.86]:25, 中继
dsn=4.1.8, # 错误码
status=deferred (host mx.vip.sohu.com[61.135.132.86] said: 450 4.1.8 <admin@v080106.isv>: Sender address rejected: Domain not found (in reply to RCPT TO command))  # 具体描述,延迟发送,mx.vip.sohu.com认为无法找到发送者的domain

还有一类统计数据:

Feb  8 10:50:21 v080106 postfix/scache[25440]: statistics: start interval Feb  8 10:41:42

Feb  8 10:50:21 v080106 postfix/scache[25440]: statistics: domain lookup hits=1692 miss=209 success=89%

Feb  8 10:50:21 v080106 postfix/scache[25440]: statistics: address lookup hits=947 miss=9428 success=9%

Feb  8 10:50:21 v080106 postfix/scache[25440]: statistics: max simultaneous domains=4 addresses=9 connection=30

有了以上这些信息,就可以准备的跟踪真实的邮件到达率了。当然,是否被认为是垃圾邮件之类,是无法从这里看出来的。

———————————————————

以gmail为例,可以点击邮件右上方的more按钮->show original查看到邮件头信息:

Delivered-To: abc@gmail.com
Received: by 10.182.176.71 with SMTP id cg7cs142014obc;
        Tue, 7 Feb 2012 15:44:47 -0800 (PST)
Received: by 10.68.218.167 with SMTP id ph7mr39734334pbc.110.1328658286305;
        Tue, 07 Feb 2012 15:44:46 -0800 (PST)
Return-Path: <201202072344452bac1460e73a42f2b86762ab94c6272d-C29QW58DB99VUH@bounces.amazon.com>
Received: from smtp-out-184-237.amazon.com (smtp-out-184-237.amazon.com. [207.171.184.237])
        by mx.google.com with ESMTP id e6si262857pbj.138.2012.02.07.15.44.45;
        Tue, 07 Feb 2012 15:44:46 -0800 (PST)
Received-SPF: pass (google.com: domain of 201202072344452bac1460e73a42f2b86762ab94c6272d-C29QW58DB99VUH@bounces.amazon.com designates 207.171.184.237 as permitted sender) client-ip=207.171.184.237;
Authentication-Results: mx.google.com; spf=pass (google.com: domain of 201202072344452bac1460e73a42f2b86762ab94c6272d-C29QW58DB99VUH@bounces.amazon.com designates 207.171.184.237 as permitted sender) smtp.mail=201202072344452bac1460e73a42f2b86762ab94c6272d-C29QW58DB99VUH@bounces.amazon.com
Date: Tue, 7 Feb 2012 23:44:45 +0000 (UTC)
From: "Amazon.cn" <store-news@amazon.cn>
To: "abc@gmail.com" <abc@gmail.com>
Message-ID: <677952103.17204211328658285803.JavaMail.em-build@massmail-sender-fe-9004.sea19.amazon.com>

SPF的状态我看到的有pass(可信)、neutral(中性),有些垃圾邮件的状态也是pass的。
大批量发送广告邮件时,目前收集到的需要注意:

  1. 控制发送频率,10个以上ip并行发送,一天控制在几十万的数量
  2. 更真实的header信息,通过配置发件域名等信息
  3. 如果可能,同邮件服务商沟通,争取进入白名单

邮件域名配置:

  1. 主机A记录abc.com
  2. 主机A记录mail.abc.com
  3. 主机A记录www.abc.com
  4. 主机A记录ftp.abc.com
  5. abc.com 到 mail.abc.com的MX记录

另外还有反向解析相关的配置。比如我们的发件地址是news@mail.flykobe.com,则需要在邮件服务器里将HELO地址配成mail.flykobe.com,并且按照上面的方法配置好各种相关域名到固定ip。同时,还需要将ip到mail.flykobe.com做一个映射(这个需要联系DNS提供商)。我们既无固定ip,就无法实现这个了!可以参考:http://hi.baidu.com/yangming/blog/item/323d034ff87a103fafc3ab8a.html

————————————————————-

记录一下利用postfix,即开源的邮件传输代理,搭建邮件服务器的过程。

网上资料可见,需要以下几个套件:

  • SMTP 認證套件: cyrus-sasl ( cyrus-sasl-1.5.27-5mdk 以及其他認證機制函式庫 )
  • Postfix 郵件伺服器: postfix ( postfix-1.1.11-4mdk )
  • POP3 伺服器: imap ( imap-2001a-9mdk, imap-devel-2001a-9mdk 兩個 ) # 最后没有用这个
  • 郵件分析軟體:procmail ( procmail-3.22-3mdk )

我们的支付宝悬赏项目今天上线了,整理一下开发过程中遇到的问题,以及相应的解决方法。我们线上项目地址:http://yingyong.taobao.com/show.htm?app_id=155001

测试环境的ip限制

支付宝项目大多涉及到money,所以对安全性要求较高。支付宝一般要求第三方提供固定ip的白名单,仅允许白名单ip访问其系统接口(注:测试开发阶段,访问页面接口,也需通过固定ip)。

测试开发阶段,也同样需要固定ip的服务器,并在这些服务器上配置支付宝提供的hosts。我们的测试、开发服务器出口ip不固定,但是可以通过ssh的端口转发、本地代理等方式解决。以下,将称呼该固定ip服务器为代理服务器。

为chrome安装proxy switch,并用putty建立一个到代理服务器的ssh代理通道,指定给proxy switch。在本地开发机上,将hosts配置为淘宝所需。这样就可以访问其页面接口了。下面通过linux curl函数代理解决固定ip访问系统接口的问题。

Linux curl函数代理

绑定帐号之类的页面接口访问,可以通过浏览器代理实现。但是之后的创建悬赏任务等,都是系统接口,在server端执行的,需要设置linux下curl函数的sock代理。原理一样,也是通过ssh代理实现:ssh -TnNv -D local-port -p ssh-proxy-port  user@ssh-proxy-ip # 如果需要后台执行,则ssh -qTfnN 。

之后,在代码里设置代理信息:

curl_setopt($process, CURLOPT_PROXY, ‘127.0.0.1:7070′);
curl_setopt($process, CURLOPT_PROXYTYPE, CURLPROXY_SOCKS5);

ssh代理建立完成后,可以写一些小测试代码,检查代理是否正确。这时,有些情况会没有响应,这时可以换一个目标url试试。比如我之前用baidu测试,就没有响应,但是退出ssh代理时可以看到:

debug1: channel 2: free: direct-tcpip: listening port 7070 for 119.75.218.77 port 80, connect from 127.0.0.1 port 21856, nchannels 3
debug1: channel 3: free: direct-tcpip: listening port 7070 for 119.75.217.56 port 80, connect from 127.0.0.1 port 21857, nchannels 2
debug1: channel 4: free: direct-tcpip: listening port 7070 for 119.75.217.56 port 80, connect from 127.0.0.1 port 21864, nchannels 1

说明还是连接上了,但是baidu没给影响而已。

这样就彻底解决了固定IP的问题,以下正式开发。

绑定帐号

我们的功能需求是,定期给我们的用户打款,人和金额都不定。

由于支付宝接口中,用户都是用支付宝唯一用户号(2088开头,16位纯数字)来标识的,可以通过用户绑定的方式获取其支付宝id号(也有其他途径,比如taobao.user.get)。针对悬赏系列接口而言,仅创建任务者需要通过该接口绑定。其他如收款者,只需要获取到支付宝唯一号即可,不需要强制绑定。

例如我们的用户flykobe登录之后,希望绑定支付宝帐号,则页面会由站内跳转到alipay,flykobe在那里输入支付宝的帐号、密码手工授权,授权成功后,会跳转到return_url指定的我们的站内页面来,提示授权成功,并且该return_url会将绑定的结果(包括支付宝唯一用户号、部分屏蔽的支付宝帐号、绑定key)记录在我们的本地存储里,留待后用。

sns.account.bind参数和返回值解释:

Args:sns_user_id – 第三方站内用户id;sns_user_name – 第三方站内用户名; type – 我使用的是common,因为不清楚buyer、seller、common之后的区别。注意:之后悬赏接口的outer_account_name和outer_account_id就对应这里的sns_user_name和sns_user_id。

Returns:alipay_login_id – 支付宝帐号,可能是邮箱、手机号等;key – 在解除绑定时需要,之后的悬赏接口未见使用;alipay_card_no – 支付宝用户号,16位纯数字,目前以2088开头。

注意:在测试阶段,使用的支付宝帐号是测试帐号,需要请支付宝帮忙生成,否则就会一直报“请求参数不正确。”

注意2:对支付宝返回值进行sign verify的时候,仅需要将支付宝提供的参数作为verify的对象,而return_url等中自带的参数不需要!否则就会verify失败!

创建任务

可以在https://shang.alipay.com/qiugou/index.htm看看支付宝的悬赏任务。我的理解是,由一些商家或者组织,发布任务提供赏金(即冻结这部分钱),一旦用户完成了这些任务,并由商家人工审核通过后,就可以标记用户,并自动完成支付赏金的过程。创建任务接口,顾名思义,就是创建一个新的悬赏任务。

在支付宝提供的悬赏接口中分为页面接口和系统接口。前者实际上是带着第三方提供的url参数跳转到支付宝里,需要人工输入一些密码、验证码等信息来完成的;后者则是可以完全代码后台运行,不需要人工干预的。

创建任务是一个页面接口,解释一下参数和返回值。

Args:outer_task_id – 第三方的任务id,由第三方保证其唯一性;outer_task_freeze_no – 第三方流水号,也由第三方保证唯一性;task_amount – 赏金金额(总额);task_type – 目前测试阶段,由支付宝提供了一个字符串作为我们的任务类型,不清楚之后应该传入什么值;task_title – 标题;task_expired_time – 截止时间;outer_account_name – 绑定的第三方用户名;outer_account_id – 绑定的第三方用户id。未使用增值服务,所以增值字段都置为空了。

Returns:仅多返回了paid_time – 支付时间。其他还返回了outer_task_id、outer_task_freeze_no、task_amount、additional_profit_amount、additional_profit_transfer_no。所以,具体支付的细节,需要在调用该create接口前,第三方自己记录下来。

注意:在测试环境下,验证码都是“8888”!

注意2:线上环境里,任务的创建者必须是与支付宝签约的帐号,否则在进行后续的pay动作时,会报错:PLATFORM_AUTHORITY_ILLEGAL

注意3:线上环境,刚上线时,创建的任务截止时间不要太长、金额不要太大。如果用户有未结束的创建任务,则无法解除绑定。非签约用户可创建任务,但该任务之后无法支付赏金,也无法结束。

打款

alipay.witkey.task.pay.by.platform接口是真正付款的接口,是一个系统接口。可以针对不同的用户,支付不同的金额。需要创建者的sns_user_id或者支付宝id,以及收款用户的支付宝id和sns_user_name,收款用户可以不绑定帐号。支付宝id可以通过taobao.user.get获取(app授权后可用)。

在测试时,开始少传递了return_url和notify_url,则返回ILLEGAL_ARGUMENT错误。

该接口调用时,有以下几点需要注意的:

  1. 请求参数必须使用POST方式传递
  2. 返回分为同步和异步两种:
    1. 同步返回:仅表示支付宝是否成功受理该请求,若成功,第三方可以修改支付为“处理中”的状态;若失败,则代表参数有问题,或者金额不足等,需要自行处理
    2. 异步返回:至notify_url。表示赏金是否支付成功,第三方可修改支付为“已支付”状态。并且,第三方必须打印“success”作为响应,否则支付宝会多次调用notify_url,直至超时。

解释下参数:

Args:outer_task_id – 对应create里的outer_task_id;outer_account_id – 创建者的第三方用户id;alipay_user_id – 创建者的支付宝id;transfer_detail – 字符串,放款明细。记录数不超过1000。 多条记录之间用(*@|$)分割, 记录内部属性之间用(~*@^)分割。 具体内容: 赏金分配流水号~*@^打款金额 ~*@^支付宝用户号~*@^合作网站用户号。其中,赏金分配流水号由第三方自行指定;打款金额以元为单位,精确到分。支付宝用户号是收款人的支付宝id;合作网站用户号是收款人的第三方用户号,如果绑定的话,需要跟绑定时传入的sns_user_id一致。

异步返回给notify_url的数据是分条的。及transfer_detail中若含有N条数据,则会异步返回N次。异步返回成功后,可以在创建者绑定的支付宝帐号的账单明细中看到支付记录。

注意:失败时返回的错误提示码并不是很友好,比如,当传入的收款人不存在时,返回的是ILLEGAL_SIGN。并且如果传入的第三方用户号与支付宝用户号不一致的话,也报ILLEGAL_SIGN。

注意2:传入的notify_url中,不可以有GET参数!比如http://abc.com/ri.php?action=notify,否则支付宝会解析为:“http://abc.com/ri.php?action=notify,charset=utf-8,params={name=sign,”

最后,感谢支付宝合作方的各种帮助,我在开发过程中,提了N多的傻问题,不好意思了 :(

在php代码里写sql句子,我喜欢用sprintf,这样看起来比较清晰,但是今天就遇到了一个隐藏很深的bug。

有这样一段代码:

        public function update_catogery_ad($id, array $info){/*{{{*/
                if (!$id || !$info){
                        return false;
                }

                $sql = null;
                foreach($info as $a){
                        list($type, $key, $val) = $a;
                        switch($type){
                                case 'int':
                                        $val = intval($val);
                                        break;
                                case 'string':
                                        $val = "'"._es($val)."'";
                                        break;
                        }
                        $sql .= sprintf(" `%s` = $val,", _es($key)); #1
                }
                $sql = substr($sql, 0, -1);

                $sql = sprintf("update %s set $sql where id = %d", $this->_get_catogery_ad(), intval($id)); #2
                return $this->db->update($sql);
        }/*}}}*/

它执行的是,根据传入的info数组中的字段,更新某表。在测试环境,运行的很好,但是上线之后,有同事就反映无法更新数据,也不报错。

出问题的info数组是:

$info = array(

‘a’ => ‘bbbb’,

‘c’ => ‘<area target=”_blank” alt=”” href=”http%3A%2F%2Fitry.try8.info%2Ftaobao%2Ftry%2F%3Fcatogery%3D%E4%BC%98%E8%B4%A8%E8%89%AF%E5%93%81″ coords=”0,100,180,170″ shape=”rect”>’,

);

这里字段a可以被正常更新,但是字段c就维持原样。

仔细观察这段数据,可以看到字段c的value中含有%!在sprintf里,它会被当作特殊字符,于是第一条sql句子会报warning:Too few arguments,由于页面上display_errors关闭了,所以没有看到任何提示,这段代码会默默的执行下去,完成整个功能,看起来就好像c字段被忽略掉一样。

于是动手修改第一条sql句子为:$sql .= sprintf(” `%s` = %s,”, _es($key), $val);

再次运行,直接报sql句子为空!这时发现第二条sql句子也有同样的问题。于是将第二条句子修改为:$sql = sprintf(“update %s set %s where id = %d”, $this->_get_catogery_ad(), $sql, intval($id));

这些人为的bug会隐藏的很深!得多加小心,所以,编程习惯很重要!在使用sprintf的时候,如果变量中可能带有特殊字符,则一定要放在arg里,不能直接写入format里。

tokyo cabinet与tyrant一起,组成了一个可用的NoSQL数据库,cabinet是lib,tyrant是server。其性能和应用性毋庸置疑,希望可以学习,同时也对其稳定性持怀疑态度,希望可以看看是我用的不对,还是其源码有可改进的地方。

从tyrant/ttserver.c的main函数入手:

main(){
    ……
    g_serv = ttservnew();
    int rv = proc(dbname, host, port, thnum, tout, dmn, pidpath, kl, logpath,
                  ulogpath, ulim, uas, sid, mhost, mport, rtspath, ropts,
                  skelpath, mulnum, extpath, extpcs, mask);
    ttservdel(g_serv);
    if(extpcs) tclistdel(extpcs);
    return rv;
  }
}

省略的部分是命令行参数处理,ttservnew函数对TTSERV进行初始化,其中初始化了4个锁:
ttservnew(){
    ……
    if(pthread_mutex_init(&serv->qmtx, NULL) != 0) tcmyfatal("pthread_mutex_init failed");
    if(pthread_cond_init(&serv->qcnd, NULL) != 0) tcmyfatal("pthread_cond_init failed");
    if(pthread_mutex_init(&serv->tmtx, NULL) != 0) tcmyfatal("pthread_mutex_init failed");
    if(pthread_cond_init(&serv->tcnd, NULL) != 0) tcmyfatal("pthread_cond_init failed");
    ……
}

typedef struct _TTSERV {
    ……
    pthread_mutex_t qmtx;                  /* mutex for the queue */
    pthread_cond_t qcnd;                   /* condition variable for the queue */
    pthread_mutex_t tmtx;                  /* mutex for the timer */
    pthread_cond_t tcnd;                   /* condition variable for the timer */
    ……
}

这里就牵涉到两个重要的概念,timer线程和worker线程,稍候会介绍。qmtx和qcnd主要是供主线程和worker线程使用,tmtx和tcnd主要供主线程和timer线程使用。

可以看出,真正的处理逻辑是在proc函数里。这个函数里进行了大量的稳定、log、容错等处理,先都忽略(但是这些才保证了一份好代码的存在!)。

proc函数中主要进行了以下工作:

  1. 如果以daemon模式运行,则调用ttdaemonize函数,建立可运行的子子进程,父进程和子进程都退出
  2. tcadbopen函数,根据数据库文件,为TCADB对象赋值,对于hash类型的db而言,会将数据库文件mmap到共享内存中
  3. 调用ttservaddtimedhandler函数,建立一个tserv的timer对象,其handler是do_slave
  4. 如果有外部脚本相关的,则再调用ttservaddtimedhandler函数,建立N个timer对象,其handler是do_extpc
  5. 调用ttservsettaskhandler函数,将tserv的do_task赋值为do_task函数
  6. 调用ttservsettermhandler函数,将tserv的do_term赋值为do_term函数,处理退出
  7. do{ ttservstart(); }while(g_restart); 之类的g_restart值可以有signal改变,从而使ttserver可以再次执行ttservstart,以达到重启、reload的效果
  8. 在ttservstart函数中,是真正的业务逻辑

以下简单看看ttservstart函数:

  1. 根据host和port,建立socket,可以是unix原始套接口,也可以是TCP套接口
  2. 新建timer个数个线程,等待执行timer->do_timer函数
  3. 新建thnum个worker线程,以空的reqs初始化,等待执行timer->do_task函数
  4. epoll_wait:
    1. 如果是请求建立连接,则accept连接,并将accept的socket端口加入到epoll的监听队列里
    2. 如果是其他请求,则请求qmtx锁,将该端口push到serv->queue里,释放qmtx锁,并发送qcnd信号,触发worker线程

worker线程的主体就是do_task函数,其读取socket中的数据,如果以magic word打头,则是二进制格式的命令;否则就是memcached兼容的命令格式。根据请求内容,再分别调用如do_put、do_get等函数进行处理。

以do_get函数为例,主要是调用tcadbget函数,读取到内容,再通过socket返回给客户端。这里的tcadbget函数,会根据数据库的类型,调用其get函数进行读取,比如hash database,就使用tchdbget方法。tchdbget中,首先请求HDBLOCKMETHOD 的read lock(允许其他读,但是不允许写),对整个db加锁,然后调用tchdbbidx函数计算出key对应在hash数组中的位置,和二次hash的key(为了解决hash碰撞,对于第一次hash key相同的元素,使用tree再进行存储)。调用tchdbgetimpl函数,进行真正的获取数据。

tchdbgetimpl函数中:由于hash database使用了on-memory hash database作为缓存,所以,如果cache不为空,则先会执行tcmdbget在cache中查找,如果找到则返回,否则继续在真正的hash database里查找。首先找到bucket的偏移位置,对于hash database来说,一部分数据会被map到共享内存,如果bucket属于这部分数据,则从map中读取;否则从数据文件读取(可能存在bucket的一部分数据在map,另一部分在数据文件的情况)。

bucket的header数据结构为:

magic word | hash | left child off | right child off | padding size | size of key | size of value |

buckey里的数据是以树存储的,用上面计算出的二次hash的key,与bucket key比较,若二次key较小,则向左子树找;若较大则向右子树找;否则相等时,比较关键字,若相等,则找到结果,返回;如果虽然二次hash的值相等,但是比较关键字不等,则需要解决二次 hash的碰撞。

二次hash碰撞的解决方法,是沿着当前节点的左子树或者右子树下行直至找到二次hash值等、关键字也等的节点,或者找不到匹配的节点,则返回失败。

———————————- 小分割 ——————————————

2012-1-19

今天想解决3个问题:

  1. 复制数据库文件,与tcrmgr copy的区别;后者对锁的使用情况
  2. B+tree数据文件格式,为什么异常退出会导致数据丢失?
  3. 尝试用c代码读取ulog和数据文件的内容
tcrmgr copy的奥秘

tcrmgr copy的响应函数是proccopy,首先tcrdbnew数据库,并init了一个mutex锁,继而调用tcrdbcopy函数,先请求这个mutex锁,再进行具体的copy操作。

这里请求mutex锁的tcrdblockmethod函数,在多处被调用,比如get、put等等。这里是简单的CLI模式,每次数据库都是new的,mutex锁也是新建的,但是可以猜想,在多线程模式下,线程间的读写操作都是互斥的。(这里存疑,为什么用互斥锁,而非条件变量呢?)

tcrdbcopyimpl组装socket的copy包,通过tcrdbsend发送给ttserver。并通过ttsockgetc获取返回值。

可以看出,copy的奥秘不在tcrmgr这里,而是在ttserver怎样处理copy请求里。

ttserver.c的do_copy函数具体处理copy请求,针对B+tree,其实际处理函数为tcbdbcopy。

首先对数据库加写锁,排除一切的读写操作,从cache里读取叶节点和非叶节点,分别存储到lids和nids里。解锁。由于cache里的数据还未写入到数据文件里,所以针对每个节点,调用对应的save函数,写入到数据文件中(写入每个节点的前后加写锁、解锁)。

调用tcbdbtranbegin启动事务,加读锁,调用tchdbcopy函数(据说B+tree的底层存储其实是hash格式)将数据写到目的文件中,解锁。

实际执行复制操作的是tchdbcopyimpl,这里封装了两种操作,一种是直接调用tccopyfile进行复制,另一种是当目的文件以@开头,则执行命令。仅关注前者,很简单,就是将数据库文件中的内容读取出来,写入到目标文件。

那么tcrmgr copy和直接copy有哪些区别呢?

  • tcrmgr copy会先将cache中的数据写回到文件中;而直接copy不会,所以cache中的数据就丢失了
  • tcrmgr copy写备份文件的时候,会加读锁,排斥写,保证备份文件的数据格式是正确的;而直接copy时没有锁,如果正好遇到cache中的数据同步到数据库文件,则备份文件的格式可能是损坏的
B+tree数据库异常情况分析

从上面的copy过程,猜想,异常退出导致文件格式损坏的原因可能有:

  • 数据写在cache里,没有存入文件,cache里的新数据丢失
  • 存入文件过程中,异常退出,导致文件格式损坏,可能很多数据甚至整个文件都会丢失

具体需要看tcbdbputimpl函数。在该函数中,首先从history里查找key对应的叶节点的pid,如果找不到则从根节点开始二分查找,直至找到或者新建了一个leaf节点的pid(细节没看)。然后将新内容作为rec写入pid对应的leaf位置。之后如果需要,会对整个树进行调整

如果该节点所在的recs数目超过了启动ttserver时的预设值lmemb,或者该节点的的size超过了lsmax,则调整。调整的具体过程忽略,可以参考算法中的B+ tree部分。

否则,如果该节点的recs数目<1,则说明put失败,需要调用tcbdbleafkill函数切断该leaf节点与树的联系(疑问,这个leaf里应该还有其他旧数据,岂不是也丢失了?)。

如果不在事务里,则将数据写入cache。如果页节点的cache数目大于设定值lcnum,则需要将叶节点的cache内容写入到数据文件(又调用了hash的put函数,不清楚那边是否也有cache缓存),并清除叶节点cache的内容。同样处理非叶节点。那么,如果非正常退出,cache中的数据就有可能会丢失。

在将数据写入数据文件的过程中,注意到其rbuf的大小是BDBPAGEBUFSIZ,即32768,这就意味着,一个leaf的最大字节数是32768(数据部分应该是比这个要小)。

留个问题,cache里的数据难道不存在于数据文件中吗?