Archive for 六月, 2012

http://blog.csdn.net/MONKEY_D_MENG/article/details/6647488

树形结构的数据库表Schema设计

 

程序设计过程中,我们常常用树形结构来表征某些数据的关联关系,如企业上下级部门、栏目结构、商品分类等等,通常而言,这些树状结构需要借助于数据库完成持久化。然而目前的各种基于关系的数据库,都是以二维表的形式记录存储数据信息,因此是不能直接将Tree存入DBMS,设计合适的Schema及其对应的CRUD算法是实现关系型数据库中存储树形结构的关键。

理想中树形结构应该具备如下特征:数据存储冗余度小、直观性强;检索遍历过程简单高效;节点增删改查CRUD操作高效。无意中在网上搜索到一种很巧妙的设计,原文是英文,看过后感觉有点意思,于是便整理了一下。本文将介绍两种树形结构的Schema设计方案:一种是直观而简单的设计思路,另一种是基于左右值编码的改进方案。

一、基本数据

本文列举了一个食品族谱的例子进行讲解,通过类别、颜色和品种组织食品,树形结构图如下:

 

二、继承关系驱动的Schema设计

对树形结构最直观的分析莫过于节点之间的继承关系上,通过显示地描述某一节点的父节点,从而能够建立二维的关系表,则这种方案的Tree表结构通常设计为:{Node_id,Parent_id},上述数据可以描述为如下图所示:

这种方案的优点很明显:设计和实现自然而然,非常直观和方便。缺点当然也是非常的突出:由于直接地记录了节点之间的继承关系,因此对Tree的任何CRUD操作都将是低效的,这主要归根于频繁的“递归”操作,递归过程不断地访问数据库,每次数据库IO都会有时间开销。当然,这种方案并非没有用武之地,在Tree规模相对较小的情况下,我们可以借助于缓存机制来做优化,将Tree的信息载入内存进行处理,避免直接对数据库IO操作的性能开销。

三、基于左右值编码的Schema设计

在基于数据库的一般应用中,查询的需求总要大于删除和修改。为了避免对于树形结构查询时的“递归”过程,基于Tree的前序遍历设计一种全新的无递归查询、无限分组的左右值编码方案,来保存该树的数据。

第一次看见这种表结构,相信大部分人都不清楚左值(Lft)和右值(Rgt)是如何计算出来的,而且这种表设计似乎并没有保存父子节点的继承关系。但当你用手指指着表中的数字从1数到18,你应该会发现点什么吧。对,你手指移动的顺序就是对这棵树进行前序遍历的顺序,如下图所示。当我们从根节点Food左侧开始,标记为1,并沿前序遍历的方向,依次在遍历的路径上标注数字,最后我们回到了根节点Food,并在右边写上了18。

第一次看见这种表结构,相信大部分人都不清楚左值(Lft)和右值(Rgt)是如何计算出来的,而且这种表设计似乎并没有保存父子节点的继承关系。但当你用手指指着表中的数字从1数到18,你应该会发现点什么吧。对,你手指移动的顺序就是对这棵树进行前序遍历的顺序,如下图所示。当我们从根节点Food左侧开始,标记为1,并沿前序遍历的方向,依次在遍历的路径上标注数字,最后我们回到了根节点Food,并在右边写上了18。

    依据此设计,我们可以推断出所有左值大于2,并且右值小于11的节点都是Fruit的后续节点,整棵树的结构通过左值和右值存储了下来。然而,这还不够,我们的目的是能够对树进行CRUD操作,即需要构造出与之配套的相关算法。

四、树形结构CRUD算法

(1)获取某节点的子孙节点

只需要一条SQL语句,即可返回该节点子孙节点的前序遍历列表,以Fruit为例:SELECT* FROM Tree WHERE Lft BETWEEN 2 AND 11 ORDER BY Lft ASC。查询结果如下所示:

    那么某个节点到底有多少的子孙节点呢?通过该节点的左、右值我们可以将其子孙节点圈进来,则子孙总数 = (右值 – 左值– 1) / 2,以Fruit为例,其子孙总数为:(11 –2 – 1) / 2 = 4。同时,为了更为直观地展现树形结构,我们需要知道节点在树中所处的层次,通过左、右值的SQL查询即可实现,以Fruit为例:SELECTCOUNT(*) FROM Tree WHERE Lft <= 2 AND Rgt >=11。为了方便描述,我们可以为Tree建立一个视图,添加一个层次数列,该列数值可以写一个自定义函数来计算,函数定义如下:

  1. CREATE FUNCTION dbo.CountLayer
  2. (
  3.     @node_id int
  4. )
  5. RETURNS int
  6. AS
  7. begin
  8.     declare @result int
  9.     set @result = 0
  10.     declare @lft int
  11.     declare @rgt int
  12.     if exists(select Node_id from Tree where Node_id = @node_id)
  13.     begin
  14.         select @lft = Lft, @rgt = Rgt from Tree where node_id = @node_id
  15.         select @result = count(*) from Tree where Lft <= @lft and Rgt >= @rgt
  16.     end
  17.     return @result
  18. end
  19. GO

基于层次计算函数,我们创建一个视图,添加了新的记录节点层次的数列:

  1. CREATE VIEW dbo.TreeView
  2. AS
  3. SELECT Node_id, Name, Lft, Rgt, dbo.CountLayer(Node_id) AS Layer FROM dbo.Tree ORDER BY Lft
  4. GO

创建存储过程,用于计算给定节点的所有子孙节点及相应的层次:

  1. CREATE PROCEDURE [dbo].[GetChildrenNodeList]
  2. (
  3.     @node_id int
  4. )
  5. AS
  6. declare @lft int
  7. declare @rgt int
  8. if exists(select Node_id from Tree where node_id = @node_id)
  9.     begin
  10.         select @lft = Lft, @rgt = Rgt from Tree where Node_id = @node_id
  11.         select * from TreeView where Lft between @lft and @rgt order by Lft ASC
  12.     end
  13. GO

现在,我们使用上面的存储过程来计算节点Fruit所有子孙节点及对应层次,查询结果如下:

 

 

从上面的实现中,我们可以看出采用左右值编码的设计方案,在进行树的查询遍历时,只需要进行2次数据库查询,消除了递归,再加上查询条件都是数字的比较,查询的效率是极高的,随着树规模的不断扩大,基于左右值编码的设计方案将比传统的递归方案查询效率提高更多。当然,前面我们只给出了一个简单的获取节点子孙的算法,真正地使用这棵树我们需要实现插入、删除同层平移节点等功能。

(2)获取某节点的族谱路径

假定我们要获得某节点的族谱路径,则根据左、右值分析只需要一条SQL语句即可完成,以Fruit为例:SELECT* FROM Tree WHERE Lft < 2 AND Rgt > 11 ORDER BY Lft ASC ,相对完整的存储过程:

  1. CREATE PROCEDURE [dbo].[GetParentNodePath]
  2. (
  3.     @node_id int
  4. )
  5. AS
  6. declare @lft int
  7. declare @rgt int
  8. if exists(select Node_id from Tree where Node_id = @node_id)
  9.     begin
  10.         select @lft = Lft, @rgt = Rgt from Tree where Node_id = @node_id
  11.         select * from TreeView where Lft < @lft and Rgt > @rgt order by Lft ASC
  12.     end
  13. GO

(3)为某节点添加子孙节点

    假定我们要在节点“Red”下添加一个新的子节点“Apple”,该树将变成如下图所示,其中红色节点为新增节点。

 

仔细观察图中节点左右值变化,相信大家都应该能够推断出如何写SQL脚本了吧。我们可以给出相对完整的插入子节点的存储过程:

  1. CREATE PROCEDURE [dbo].[AddSubNode]
  2. (
  3.     @node_id int,
  4.     @node_name varchar(50)
  5. )
  6. AS
  7. declare @rgt int
  8. if exists(select Node_id from Tree where Node_id = @node_id)
  9.     begin
  10.         SET XACT_ABORT ON
  11.         BEGIN TRANSCTION
  12.         select @rgt = Rgt from Tree where Node_id = @node_id
  13.         update Tree set Rgt = Rgt + 2 where Rgt >= @rgt
  14.         update Tree set Lft = Lft + 2 where Lft >= @rgt
  15.         insert into Tree(Name, Lft, Rgt) values(@node_name, @rgt, @rgt + 1)
  16.         COMMIT TRANSACTION
  17.         SET XACT_ABORT OFF
  18.     end
  19. GO

(4)删除某节点

如果我们想要删除某个节点,会同时删除该节点的所有子孙节点,而这些被删除的节点的个数为:(被删除节点的右值 – 被删除节点的左值+ 1) / 2,而剩下的节点左、右值在大于被删除节点左、右值的情况下会进行调整。来看看树会发生什么变化,以Beef为例,删除效果如下图所示。

则我们可以构造出相应的存储过程:

  1. CREATE PROCEDURE [dbo].[DelNode]
  2. (
  3.     @node_id int
  4. )
  5. AS
  6. declare @lft int
  7. declare @rgt int
  8. if exists(select Node_id from Tree where Node_id = @node_id)
  9.     begin
  10.         SET XACT_ABORT ON
  11.         BEGIN TRANSCTION
  12.             select @lft = Lft, @rgt = Rgt from Tree where Node_id = @node_id
  13.             delete from Tree where Lft >= @lft and Rgt <= @rgt
  14.             update Tree set Lft = Lft – (@rgt – @lft + 1) where Lft > @lft
  15.             update Tree set Rgt = Rgt – (@rgt – @lft + 1) where Rgt > @rgt
  16.             COMMIT TRANSACTION
  17.         SET XACT_ABORT OFF
  18.     end
  19. GO

五、总结

我们可以对这种通过左右值编码实现无限分组的树形结构Schema设计方案做一个总结:

(1)优点:在消除了递归操作的前提下实现了无限分组,而且查询条件是基于整形数字的比较,效率很高。

(2)缺点:节点的添加、删除及修改代价较大,将会涉及到表中多方面数据的改动。

当然,本文只给出了几种比较常见的CRUD算法的实现,我们同样可以自己添加诸如同层节点平移、节点下移、节点上移等操作。有兴趣的朋友可以自己动手编码实现一下,这里不在列举了。值得注意的是,实现这些算法可能会比较麻烦,会涉及到很多条update语句的顺序执行,如果顺序调度考虑不周详,出现Bug的话将会对整个树形结构表产生惊人的破坏。因此,在对树形结构进行大规模修改的时候,可以采用临时表做中介,以降低代码的复杂度,同时,强烈推荐在做修改之前对表进行完整备份,以备不时之需。在以查询为主的绝大多数基于数据库的应用系统中,该方案相比传统的由父子继承关系构建的数据库Schema更为适用。

 

 

https://communities.bmc.com/communities/docs/DOC-9902

Trees in SQL: Nested Sets and Materialized Path

by Vadim Tropashko

 

Relational databases are universally conceived of as an advance over their  predecessors network and hierarchicalmodels.  Superior in every querying respect, they turned out to be surprisingly  incomplete when modeling transitive dependencies. Almost every couple of months  a question about how to model a tree in the database pops up at the  comp.database.theory newsgroup. In this article I’ll investigate two out of four  well known approaches to accomplishing this and show a connection between them.  We’ll discover a new method that could be considered as a “mix-in” between  materialized path and nested sets.

Adjacency List

Tree structure is a special case of Directed Acyclic Graph (DAG). One way to  represent DAG structure is:

 

create table emp (
ename   varchar2(100),
mgrname varchar2(100)
);

 

Each record of the emp table identified by ename is referring to its parent  mgrname. For example, if JONES reports to KING, then the emp table contains  <ename=’JONES’, mgrname=’KING’> record. Suppose, the emp table also  includes <ename=’SCOTT’, mgrname=’JONES’>. Then, if the emp table doesn’t  contain the <ename=’SCOTT’, mgrname=’KING’> record, and the same is true  for every pair of adjoined records, then it is called adjacency  list. If the opposite is true, then the emp table is a transitively closed relation.

A typical hierarchical query would ask if SCOTT indirectly reports to KING.  Since we don’t know the number of levels between the two, we can’t tell how many  times to selfjoin emp, so that the task can’t be solved in traditional SQL. If  transitive closure tcemp of the emp table is known, then the query is  trivial:

 

select ‘TRUE’ from tcemp
where ename = ‘SCOTT’ and mgrname = ‘KING’

 

The ease of querying comes at the expense of transitive closure  maintenance.

Alternatively, hierarchical queries can be answered with SQL extensions:  either SQL3/DB2 recursive query

 

with tcemp as (
select ename,mgrname from tcemp
union
select tcemp.ename,emp.mgrname from tcemp,emp
where tcemp.mgrname = emp.ename
) select ‘TRUE’ from tcemp
where ename = ‘SCOTT’ and mgrname = ‘KING';

 

that calculates tcemp as an intermediate relation, or Oracle proprietary  connect-by syntax

 

select ‘TRUE’ from (
select ename from emp
connect by prior mgrname = ename
start with ename = ‘SCOTT’
) where ename = ‘KING';

 

in which the inner query “chases the pointers” from the SCOTT node to the  root of the tree, and then the outer query checks whether the KING node is on  the path.

Adjacency list is arguably the most intuitive tree model. Our main focus,  however, would be the following two methods.

Materialized Path

In this approach each record stores the whole path to the root. In our  previous example, lets assume that KING is a root node. Then, the record with  ename = ‘SCOTT’ is connected to the root via the path SCOTT->JONES->KING.  Modern databases allow representing a list of nodes as a single value, but since  materialized path has been invented long before then, the convention stuck to  plain character string of nodes concatenated with some separator; most often ‘.’  or ‘/’. In the latter case, an analogy to pathnames in UNIX file system is  especially pronounced.

In more compact variation of the method, we use sibling numerators instead of  node’s primary keys within the path string. Extending our example:

 

 

ENAME PATH
KING 1
JONES 1.1
SCOTT 1.1.1
ADAMS 1.1.1.1
FORD 1.1.2
SMITH 1.1.2.1
BLAKE 1.2
ALLEN 1.2.1
WARD 1.2.2
CLARK 1.3
MILLER 1.3.1

 

Path 1.1.2 indicates that FORD is the second child of  the parent JONES.

Let’s write some queries.

1. An employee FORD and chain of his supervisors:

 

select e1.ename from emp e1, emp e2
where e2.path like e1.path || ‘%’
and e2.name = ‘FORD’

 

2. An employee JONES and all his (indirect) subordinates:

 

select e1.ename from emp e1, emp e2
where e1.path like e2.path || ‘%’
and e2.name = ‘JONES’

 

Although both queries look symmetrical, there is a fundamental difference in  their respective performances. If a subtree of subordinates is small compared to  the size of the whole hierarchy, then the execution where database fetches e2  record by the name primary key, and then performs a range scan of  e1.path,  which is guaranteed to be quick.

On the other hand, the “supervisors” query is roughly equivalent to

 

select e1.ename from emp e1, emp e2
where e2.path > e1.path and e2.path < e1.path || ‘Z’
and e2.name = ‘FORD’

 

Or, noticing that we essentially know e2.path, it can further be reduced  to

 

select e1.ename from emp e1
where e2path > e1.path and e2path < e1.path || ‘Z’

 

Here, it is clear that indexing on path doesn’t work (except for “accidental”  cases in which e2path happens to be near the domain boundary, so that  predicate e2path > e1.path is selective).

The obvious solution is that we don’t have to refer to the database to figure  out all the supervisor paths! For example, supervisors of 1.1.2 are 1.1 and 1. A  simple recursive string parsing function can extract those paths, and then the  supervisor names can be answered by

 

select e1.ename from emp where e1.path in (‘1.1′,’1′)

 

which should be executed as a fast concatenated plan.

Nested Sets

Both the materialized path and Joe  Celko’s nested sets provide the capability to answer hierarchical  queries with standard SQL syntax. In both models, the global  position of the node in the hierarchy is “encoded” as opposed to an adjacency  list of which each link is a local connection between immediate  neighbors only. Similar to materialized path, the nested sets model suffers from  supervisors query performance problem:

 

select p2.emp from Personnel p1, Personnel p2
where p1.lft between p2.lft and p2.rgt
and p1.emp = ‘Chuck’

 

(Note: This query is borrowed from the previously  cited Celko article). Here, the problem is even more explicit than in  the case of a materialized path: we need to find all the intervals that cover a  given point. This problem is known to be difficult. Although there are  specialized indexing schemes like R-Tree, none of them is as universally  accepted as B-Tree. For example, if the supervisor’s path contains just 10 nodes  and the size of the whole tree is 1000000, none of indexing techniques could  provide 1000000/10=100000 times performance increase. (Such a performance  improvement factor is typically associated with index range scan in a similar,  very selective, data volume condition.)

 

Unlike a materialized path, the trick by which we computed all the nodes  without querying the database doesn’t work for nested sets.

 

Another — more fundamental — disadvantage of nested sets is that nested sets  coding is volatile. If we insert a node into the middle of the  hierarchy, all the intervals with the boundaries above the insertion point have  to be recomputed. In other words, when we insert a record into the database,  roughly half of the other records need to be updated. This is why the nested  sets model received only limited acceptance for static hierarchies.

 

Nested sets are intervals of integers. In an attempt to make the nested sets  model more tolerant to insertions, Celko suggested we give up the property that  each node always has (rgt-lft+1)/2 children. In my opinion, this is a half-step  towards a solution: any gap in a nested  set model with large gaps and spreads in the numbering still could be  covered with intervals leaving no space for adding more children, if those  intervals are allowed to have boundaries at discrete points  (i.e., integers) only. One needs to use a dense domain like rational, or real  numbers instead.

Nested Intervals

Nested intervals generalize nested sets. A node [clft, crgt] is an (indirect)  descendant of [plft, prgt] if:

 

plft <= clft and crgt >= prgt

 

The domain for interval boundaries is not limited by integers anymore: we  admit rational or even real numbers, if necessary. Now, with a reasonable  policy, adding a child node is never a problem. One example of such a policy  would be finding an unoccupied segment [lft1, rgt1] within a parent interval  [plft, prgt] and inserting a child node [(2*lft1+rgt1)/3,  (rgt1+2*lft)/3]:

trees-in-sql-fig1.bmp

After insertion, we still have two more unoccupied segments  [lft1,(2*lft1+rgt1)/3] and [(rgt1+2*lft)/3,rgt1] to add more children to the  parent node.

We are going to amend this naive policy in the following sections.

Partial Order

Let’s look at two-dimensional picture of nested intervals. Let’s assume that  rgt is a horizontal axis x, and lft is a vertical one – y. Then, the nested  intervals tree looks like this:

trees-in-sql-fig2.bmp

Each node [lft, rgt] has its descendants bounded within the two-dimensional  cone y >= lft & x <= rgt. Since the right interval boundary  is always less than the left one, none of the nodes are allowed above the  diagonal y = x.

The other way to look at this picture is to notice that a child node is a  descendant of the parent node whenever a set of all points defined by the child  cone y >= clft & x <= crgt is a subset of the parent cone y >= plft  & x <= prgt. A subset relationship between the cones on the plane is a  partial order.

Now that we know the two constraints to which tree nodes conform, I’ll  describe exactly how to place them at the xy plane.

The Mapping

Tree root choice is completely arbitrary: we’ll assume the interval [0,1] to  be the root node. In our geometrical interpretation, all the tree nodes belong  to the lower triangle of the unit square at the xy plane.

 

We’ll describe further details of the mapping by induction. For each node of  the tree, let’s first define two important points at the xy plane. The  depth-first convergence point is an intersection between the diagonal  and the vertical line through the node. For example, the depth-first convergence  point for <x=1,y=1/2> is  <x=1,y=1>. The breadth-first convergence point  is an intersection between the diagonal and the horizontal line through the  point. For example, the breadth-first convergence point for  <x=1,y=1/2> is  <x=1/2,y=1/2>.

 

Now, for each parent node, we define the position of the first child as a  midpoint halfway between the parent point and depth-first convergence point.  Then, each sibling is defined as a midpoint halfway between the previous sibling  point and breadth-first convergence point:

trees-in-sql-fig3.bmp

For example, node 2.1 is positioned at x=1/2, y=3/8.

 

Now that the mapping is defined, it is clear which dense domain we are using:  it’s not rationals, and not reals either, but binary fractions (although, the  former two would suffice, of course).

 

Interestingly, the descendant subtree for the parent node “1.2” is a scaled  down replica of the subtree at node “1.1.” Similarly, a subtree at node 1.1 is a  scaled down replica of the tree at node “1.” A structure with self-similarities  is called a fractal.

Normalization

Next, we notice that x and y are not completely independent. We can tell what  are both x and y if we know their sum. Given the  numerator and denominator of the rational number representing the sum of the  node coordinates, we can calculate x and y coordinates back as:

 

function x_numer( numer integer, denom integer )
RETURN integer IS
ret_num integer;
ret_den integer;
BEGIN
ret_num := numer+1;
ret_den := denom*2;
while floor(ret_num/2b = ret_num/2 loop
ret_num := ret_num/2;
ret_den := ret_den/2;
end loop;
RETURN ret_num;
END;

function x_denom( numer integer, denom integer )

RETURN ret_den;
END;

 

 

in which function x_denom body differs from x_numer in the return variable  only. Informally, numer+1 increment would move the ret_num/ret_den point  vertically up to the diagonal, and then x coordinate is half of the value, so we  just multiplied the denominator by two. Next, we reduce both numerator and  denominator by the common power of two.

Naturally, y coordinate is defined as a complement to the sum:

 

function y_numer( numer integer, denom integer )
RETURN integer IS
num integer;
den integer;
BEGIN
num := x_numer(numer, denom);
den := x_denom(numer, denom);
while den < denom loop
num := num*2;
den := den*2;
end loop;
num := numer – num;
while floor(num/2) = num/2 loop
num := num/2;
den := den/2;
end loop;
RETURN num;
END;

function y_denom( numer integer, denom integer )

RETURN den;
END;

 

 

Now, the test (where 39/32 is the node 1.3.1):

 

select x_numer(39,32)||’/’||x_denom(39,32),
y_numer(39,32)||’/’||y_denom(39,32) from dual

 

5/8     19/32

 

select 5/8+19/32, 39/32 from dual

 

1.21875 1.21875

 

I don’t use a floating point to represent rational numbers, and wrote all the  functions with integer arithmetics instead. To put it bluntly, the floating  point number concept in general, and the IEEE standard in particular, is useful  for rendering 3D-game graphics only. In the last test, however, we used a  floating point just to verify that 5/8 and 19/32, returned by the previous  query, do indeed add to 39/32.

 

We’ll store two integer numbers — numerator and denominator  of the sum of the coordinates x and y — as an encoded  node path. Incidentally, Celko’s nested sets use two integers as well.  Unlike nested sets, our mapping is stable: each node has a  predefined placement at the xy plane, so that the queries involving  node position in the hierarchy could be answered without reference to the  database. In this respect, our hierarchy model is essentially a materialized  path encoded as a rational number.

Finding Parent Encoding and Sibling Number

Given a child node with numer/denom encoding, we find the node’s parent like  this:

 

function parent_numer( numer integer, denom integer )
RETURN integer IS
ret_num integer;
ret_den integer;
BEGIN
if numer=3 then
return NULL;
end if;
ret_num := (numer-1)/2;
ret_den := denom/2;
while floor((ret_num-1)/4) = (ret_num-1)/4 loop
ret_num := (ret_num+1)/2;
ret_den := ret_den/2;
end loop;
RETURN ret_num;
END;

function parent_denom( numer integer, denom integer )

RETURN ret_den;
END;

 

 

The idea behind the algorithm is the following: If the node is on the very  top level — and all these nodes have a numerator equal to 3 — then the node has  no parent. Otherwise, we must move vertically down the xy plane at a  distance equal to the distance from the depth-first convergence point. If the  node happens to be the first child, then that is the answer. Otherwise, we must  move horizontally at a distance equal to the distance from the breadth-first  convergence point until we meet the parent node.

Here is the test of the method (in which 27/32 is the node 2.1.2, while 7/8  is 2.1):

 

select parent_numer(27,32)||’/’||parent_denom(27,32) from dual

7/8

 

 

In the previous method, counting the steps when navigating horizontally would  give the sibling number:

 

function sibling_number( numer integer, denom integer )

RETURN integer IS
ret_num integer;
ret_den integer;
ret integer;
BEGIN
if numer=3 then
return NULL;
end if;
ret_num := (numer-1)/2;
ret_den := denom/2;
ret     := 1;
while floor((ret_num-1)/4) = (ret_num-1)/4 loop
if ret_num=1 and ret_den=1 then
return ret;
end if;
ret_num := (ret_num+1)/2;
ret_den := ret_den/2;
ret     := ret+1;
end loop;
RETURN ret;
END;

 

For a node at the very first level a special stop condition, ret_num=1  and ret_den=1 is needed.

The test:

 

select sibling_number(7,8) from dual

1

 

Calculating Materialized Path and Distance between nodes

Strictly speaking, we don’t have to use a materialized path, since our  encoding is an alternative. On the other hand, a materialized path provides a  much more intuitive visualization of the node position in the hierarchy, so that  we can use the materialized path for input and output of the data if we provide  the mapping to our model.

Implementation is a simple application of the methods from the previous  section. We print the sibling number, jump to the parent, then repeat the above  two steps until we reach the root:

 

function path( numer integer, denom integer )
RETURN varchar2 IS
BEGIN
if numer is NULL then
return ”;
end if;
RETURN path(parent_numer(numer, denom),
parent_denom(numer, denom))
|| ‘.’ || sibling_number(numer, denom);
END;

 

select path(15,16) from dual

.2.1.1

 

Now we are ready to write the main query: given the 2 nodes, P and C,  when P is the parent of C? A more general query would return the number of  levels between P and C if C is reachable from P, and some exception indicator;  otherwise:

 

function distance( num1 integer, den1 integer,
num2 integer, den2 integer )
RETURN integer IS
BEGIN
if num1 is NULL then
return -999999;
end if;
if num1=num2 and den1=den2 then
return 0;
end if;
RETURN 1+distance(parent_numer(num1, den1),
parent_denom(num1, den1),
num2,den2);
END;

 

select distance(27,32,3,4) from dual

2

 

Negative numbers are interpreted as exceptions. If the num1/den1 node is not  reachable from num2/den2, then the navigation converges to the root, and  level(num1/den1)-999999 would be returned (readers are advised to find a less  clumsy solution).

 

The alternative way to answer whether two nodes are connected is by simply  calculating the x and y coordinates, and checking if the parent interval  encloses the child. Although none of the methods refer to disk, checking whether  the partial order exists between the points seems much less expensive! On the  other hand, it is just a computer architecture artifact that comparing two  integers is an atomic operation. More thorough implementation of the method  would involve a domain of integers with a unlimited range (those kinds of  numbers are supported by computer algebra systems), so that a comparison  operation would be iterative as well.

 

Our system wouldn’t be complete without a function inverse to the path, which  returns a node’s numer/denom value once the path is provided. Let’s introduce  two auxiliary functions, first:

 

function child_numer
( num integer, den integer, child integer )
RETURN integer IS
BEGIN
RETURN num*power(2, child)+3-power(2, child);
END;

 

function child_denom
( num integer, den integer, child integer )
RETURN integer IS
BEGIN
RETURN den*power(2, child);
END;

 

select child_numer(3,2,3) || ‘/’ ||
child_denom(3,2,3) from dual

19/16

 

For example, the third child of the node 1 (encoded as 3/2) is the node 1.3  (encoded as 19/16).

The path encoding function is:

function path_numer( path varchar2 )
RETURN integer IS
num integer;
den integer;
postfix varchar2(1000);
sibling varchar2(100);
BEGIN
num := 1;
den := 1;
postfix := ‘.’ || path || ‘.';

 

while length(postfix) > 1 loop
sibling := substr(postfix, 2,
instr(postfix,’.’,2)-2);
postfix := substr(postfix,
instr(postfix,’.’,2),
length(postfix)
-instr(postfix,’.’,2)+1);
num := child_numer(num,den,to_number(sibling));
den := child_denom(num,den,to_number(sibling));
end loop;

 

RETURN num;
END;

 

function path_denom( path varchar2 )

RETURN den;
END;

 

select path_numer(‘2.1.3′) || ‘/’ ||
path_denom(‘2.1.3′) from dual
51/64

The Final Test

Now that the infrastructure is completed, we can test it. Let’s create the  hierarchy

create table emps (
name varchar2(30),
numer integer,
denom integer
)

alter table emps
ADD CONSTRAINT uk_name UNIQUE (name) USING INDEX
(CREATE UNIQUE INDEX name_idx on emps(name))
ADD CONSTRAINT UK_node
UNIQUE (numer, denom) USING INDEX
(CREATE UNIQUE INDEX node_idx on emps(numer, denom))

 

and fill it with some data:

 

insert into emps values (‘KING’,
path_numer(‘1′),path_denom(‘1′));
insert into emps values (‘JONES’,
path_numer(‘1.1′),path_denom(‘1.1′));
insert into emps values (‘SCOTT’,
path_numer(‘1.1.1′),path_denom(‘1.1.1′));
insert into emps values (‘ADAMS’,
path_numer(‘1.1.1.1′),path_denom(‘1.1.1.1′));
insert into emps values (‘FORD’,
path_numer(‘1.1.2′),path_denom(‘1.1.2′));
insert into emps values (‘SMITH’,
path_numer(‘1.1.2.1′),path_denom(‘1.1.2.1′));
insert into emps values (‘BLAKE’,
path_numer(‘1.2′),path_denom(‘1.2′));
insert into emps values (‘ALLEN’,
path_numer(‘1.2.1′),path_denom(‘1.2.1′));
insert into emps values (‘WARD’,
path_numer(‘1.2.2′),path_denom(‘1.2.2′));
insert into emps values (‘MARTIN’,
path_numer(‘1.2.3′),path_denom(‘1.2.3′));
insert into emps values (‘TURNER’,
path_numer(‘1.2.4′),path_denom(‘1.2.4′));
insert into emps values (‘CLARK’,
path_numer(‘1.3′),path_denom(‘1.3′));
insert into emps values (‘MILLER’,
path_numer(‘1.3.1′),path_denom(‘1.3.1′));
commit;

 

All the functions written in the previous sections are conveniently combined  in a single view:

 

create or replace
view hierarchy as
select name, numer, denom,
y_numer(numer,denom) numer_left,
y_denom(numer,denom) denom_left,
x_numer(numer,denom) numer_right,
x_denom(numer,denom) denom_right,
path (numer,denom) path,
distance(numer,denom,3,2) depth
from emps

 

And, finally, we can create the hierarchical reports.

    • Depth-first enumeration, ordering by left interval  boundary

 

select lpad(‘ ‘,3*depth)||name
from hierarchy order by numer_left/denom_left

 

LPAD(”,3*DEPTH)||NAME
———————————————–
KING
CLARK
MILLER
BLAKE
TURNER
MARTIN
WARD
ALLEN
JONES
FORD
SMITH
SCOTT
ADAMS

 

    • Depth-first enumeration, ordering by right interval  boundary

 

select lpad(‘ ‘,3*depth)||name
from hierarchy order by numer_right/denom_right desc

 

LPAD(”,3*DEPTH)||NAME
—————————————————–
KING
JONES
SCOTT
ADAMS
FORD
SMITH
BLAKE
ALLEN
WARD
MARTIN
TURNER
CLARK
MILLER

 

    • Depth-first enumeration, ordering by path (output identical to  #2)

 

select lpad(‘ ‘,3*depth)||name
from hierarchy order by path

 

LPAD(”,3*DEPTH)||NAME
—————————————————–
KING
JONES
SCOTT
ADAMS
FORD
SMITH
BLAKE
ALLEN
WARD
MARTIN
TURNER
CLARK
MILLER

    • All the descendants of JONES, excluding himself:

 

select h1.name from hierarchy h1, hierarchy h2
where h2.name = ‘JONES’
and distance(h1.numer, h1.denom,
h2.numer, h2.denom)>0;

NAME
——————————
SCOTT
ADAMS
FORD
SMITH

    • All the ancestors of FORD, excluding himself:

 

select h2.name from hierarchy h1, hierarchy h2
where h1.name = ‘FORD’
and distance(h1.numer, h1.denom,
h2.numer, h2.denom)>0;

 

NAME
——————————
KING
JONES

Vadim  Tropashko works for Real World Performance group at Oracle  Corp. In prior life he was application programmer and translated “The C++  Programming Language” by B.Stroustrup, 2nd edition into Russian. His current  interests include SQL Optimization, Constraint Databases, and Computer Algebra  Systems.

记录log,对于很多人而言是很简单或者低级的事情。但是,随着项目经验的增长,遇到生产环境中bug数的增多,至少对于我来说,日志的重要性日益增加。

这次,需要对项目中log类进行重构,主要希望实现4个目的:

  1. 建立日志监控机制,第一时间发现问题
  2. 协助定位问题,帮助快速解决问题
  3. 记录用户行为,协助解答客户疑问
  4. 记录用户行为,协助制定安全与个性化等策略

除了这些功能性的目的,由于log类在一次请求中的调用频率相对较高,且与基本业务无关,如果性能方面有问题的话,就太本末颠倒了,所以先从性能说起。

log一般记录在文件里,所以其本质上是写文件,使用php作为开发语言的话,主要考虑了3个方面:

  1. 选择fwrite还是file_put_contents?
  2. 是否使用debug_backtrace函数获取文件名和行号?
  3. 怎样保证并发写的原子性?

选择fwrite还是file_put_contents?

php有多种写文件的函数,fwrite需要先fopen获取到文件句柄,而file_put_contents直接使用文件名即可,且传入的data参数可以是字符串,也可以是数组,甚至stream,使用较简单。

zend框架的Zend_Log_Writer_Stream类,使用的是fwrite函数;而公司内部多个team的log封装都使用了file_put_contents函数。首先考虑,我们的log类,给谁用?内部使用,暂时没考虑开源。传入的参数,是否需要支持string,array or stream?记录log而已,支持string即可,而且log的基本样式是每次记录一行。所以,比较两者在写入多行数据之间的性能区别即可:

$str = "abc\n";
$n = 10000;
$filename = 'test_write.txt';

$start_time = mytime();
write_with_open($filename, $str, $n);
$used_time = mytime() - $start_time;
printf("%20s%s\n","fwrite","Time used: $used_time ");

$start_time = mytime();
write_without_open($filename, $str, $n);
$used_time = mytime() - $start_time;
printf("%20s%s\n","file_put_contents","Time used: $used_time ");

$start_time = mytime();
write_with_errorlog($filename, $str, $n);
$used_time = mytime() - $start_time;
printf("%20s%s\n","error_log","Time used: $used_time ");

function write_with_open($filename, $str, $n){
        $fp = fopen($filename, 'a') or die("can't open $filename for write");

        while($n--){
                fwrite($fp, $str);
        }

        fclose($fp);
}

function write_without_open($filename, $str, $n){
        while($n--){
                file_put_contents($filename, $str, FILE_APPEND);
        }
}

function write_with_errorlog($filename, $str, $n){
        while($n--){
                error_log($str, 3, $filename);
        }
}

执行该测试脚本的结果是:

fwriteTime used: 0.018175840377808
file_put_contentsTime used: 0.22816514968872
error_logTime used: 0.2338011264801

可见fwrite的性能要远远大于另外两者,直观上看,fwrite仅关注于写,而文件句柄的获取仅由fopen做一次,关闭操作也尽有一次。如果修改write_with_open函数,把fopen和fclose函数放置到while循环里,则3者的性能基本持平。

以上结论,也可以通过查看PHP源代码得知,fwrite和file_put_contents的源码都在php-5.3.10/ext/standard/file.c里,file_put_contents不但逻辑较为负载,还牵涉到open/锁/close操作,对于只做一次写操作的请求来说,file_put_contents可能较适合,因为其减少了函数调用,使用起来较为方便。而对于log操作来说,fwrite从性能角度来说,较为适合。

2012-06-22补充:

以上只是单纯从“速度”角度考虑,但是在web的生产环境里,如果并发数很高,导致系统的open files数目成为瓶颈的话,情况就不同了!fwrite胜在多次写操作只用打开一次文件,持有file handler的时间较长;而file_put_contents每次都在需要的时候打开文件,写完就立即关闭,持有file handler的时间较短。如果open files的值已无法调高,那么使用file_put_contents在这种情况下,就会是另外一种选择了。

是否使用debug_backtrace函数获取文件名和行号?

在gcc,标准php出错信息里,都会给出出错的文件名和行号,我们也希望在log里加上这两个信息,那么是否使用debug_backstrace函数呢?

class A1{
        public static $_use_debug=true;
        function run(){
                # without debug_backtrace
                if (!self::$_use_debug){
                        #echo "Quit\n";
                        return;
                }

                # with debug_backtrace
                $trace = debug_backtrace();
                $depth = count($trace) - 1;
                if ($depth > 1)
                        $depth = 1;
                $_file = $trace[$depth]['file'];
                $_line = $trace[$depth]['line'];

                #echo "file: $_file, line: $_line\n";
        }
}

class A2{
        function run(){
                $obj = new A1();
                $obj->run();
        }
}

class A3{
        function run(){
                $obj = new A2();
                $obj->run();
        }
}
class A4{
        function run(){
                $obj = new A3();
                $obj->run();
        }
}
class A5{
        function run(){
                $obj = new A4();
                $obj->run();
        }
}
class A6{
        function run(){
                $obj = new A5();
                $obj->run();
        }
}

$n = 1000;
$start_time = mytime();
A1::$_use_debug = true;
$obj = new A6();
while($n--){
        $obj->run();
}
$used_time = mytime() - $start_time;
printf("%30s:%s\n", "With Debug Time used", $used_time);

$n = 1000;
$start_time = mytime();
A1::$_use_debug = false;
$obj = new A6();
while($n--){
        $obj->run();
}
$used_time = mytime() - $start_time;
printf("%30s:%s\n", "Without Debug Time used", $used_time);

function mytime(){
        list($utime, $time) = explode(' ', microtime());
        return $time+$utime;
}

运行的结果是:

flykobe@desktop test $ php test_debug_trace.php
With Debug Time used:0.005681037902832
Without Debug Time used:0.0021991729736328

但是若多次运行,少数情况下,with debug版本甚至与快于without版本。综合考虑,还是决定不用debug_backtrace,而由调用者传入__FILE__和__LINE__值。

怎样保证并发写的原子性?

写文件时,大致有这3种情况:

  1. 一次写一行,中间被插入其他内容
  2. 一次写多行,行与行之间被插入其他内容
  3. 多次写多行,行与行之间被插入其他内容

对于web访问这种高并发的写日志而言,一条日志一般就是一行,中间绝不允许被截断,覆盖或插入其他数据。但行与行之间,是否被插入其他内容,并不care。既然之前是决定采用fwrite,php手册上说的很清楚:

If handle was fopen()ed in append mode, fwrite()s are atomic (unless the size of string exceeds the filesystem’s block size, on some platforms, and as long as the file is on a local filesystem). That is, there is no need to flock() a resource before callingfwrite(); all of the data will be written without interruption.

即当fwrite使用的handler是由fopen在append模式下打开时,其写操作是原子性的。不过也有例外,如果一次写操作的长度超过了文件系统的块大小,或者使用到NFS之类的非local存储时,则可能出问题。其中文件系统的块大小(在我的centos5虚拟机上是4096 bytes)可以通过以下命令查看:

sudo /sbin/tune2fs -l /dev/mapper/VolGroup00-LogVol00 | grep -i block

这同样可以通过模拟多种不同情况的fwrite操作来验证,由于比较简单不再赘述代码。

————————-

2012-07-03添加

《unix环境高级编程》里说:Unix系统提供了一种方法使这种操作成为原子操作,即打开文件时,设置O_APPEND操作,就使内核每次对这种文件进行读写之前,都将进程的当前偏移量设置到该文件的尾端处。

而php手册中,指的某些platform应该不包含*nix实现。故可认为,在我们的环境下,php的fwrite函数是原子性的。

在以上几种比较的基础上,初步完成了我们的Log类封装,进行了千行和万行级别的log写入测试,性能提高了3倍,但应该仍然有优化余地。

国内开放平台泛滥,大多使用的是OAuth或其变形,但对于openID则了解较少。

当前有一个需求,核心其实是用户的身份验证,从目前看来,并没有后续资源访问了,使用oauth感觉小题大做。而如果自己定义协议,又增加了用户的学习成本,且有安全性等疏漏的隐患。

OpenID和OAuth的区别及第三方登录的安全隐患分析一文中,作者分析了openID与OAuth的区别:

OpenID和OAuth完全是为了两种不同的需求而生

OpenID的目标是为了帮助网站确认一个用户的身份
OAuth的目标是为了授权第三方在可控范围下访问用户资源

同时,http://www.slideshare.net/gongjt/oauthopenid 里也比对比了两者,并给出国内目前可用的openID案例:比如http://www.lepu.com/login使用了google的openID.

—————————

综合以上的分析,对于需要开放资源的平台而言,oauth更合适。而仅提供用户身份验证的平台,openID就足够用了。考虑到我们的需求,以下详细研究OAuth2.0。

具体的计划是:

  1. 泛读OAuth2.0 draft协议
  2. 泛读百度内部文档中,关于OAuth的部分,关注有哪些部门选用了OAuth,是哪个版本或者变种,实现机制是什么?
  3. 结合步骤2,以及网络上关于OAuth实现版本的分析,快速浏览两到三种OAuth2.0 server端的php开源代码,并进行搭建,分析其性能/安全性/可扩展性/协议实现的完整程度
  4. 选用一种实现代码,进行必要的修改
  5. 选择对应的OAuth client SDK,完成授权流程
  6. 结合业务需求,开发open api,完成一期api开发流程

出于时间的考虑,目前仅focus在web server scenarios上,不考虑js/mobile等场景。

大多数OAuth2.0实现的授权过程如下:

1. app方,根据appkey/return_url(授权成功后返回的url)等,拼成授权登录的url,header user 到这个url(该url部署在授权服务器上)

2.user在这个url页面里,登录,并同意授权给该app

3.授权服务器验证用户登录和授权成功,生成request_token/request_token_secret,拼在return_url后面,header user到该return_url(加密和签名后的)

4.app方,根据该request_token/appkey,生成获取access_token的api url(该url也部署在授权服务器上),调用该api;授权服务器检查requset_token和appkey是否匹配,生成access_token/access_token_secret,作为返回值;app方,获取返回值中的access_token/access_token_secret,并保存。

5.app方,利用access_token,调用资源服务器提供的api

资料:

1. oauth2-11 版本中文译文 http://wenku.baidu.com/view/b37ed7260722192e4536f66e.html

2.oauth2-26 英文版 http://tools.ietf.org/html/draft-ietf-oauth-v2-26

3.Using OAuth 2.0 to Access Google APIs https://developers.google.com/accounts/docs/OAuth2?hl=zh-CN

4.OAuth官网 http://oauth.net/

之前使用的加速器是eAccelerator,新公司使用的是APC,转载文章一篇。

zz from:http://devzone.zend.com/1812/using-apc-with-php/

Using APC with PHP

Cache Cow

If you’ve been around PHP for a while, you’ve probably heard about APC, the Alternative PHP Cache. APC is an opcode cache that can significantly speed up your PHP applications, by caching both PHP code and user variables. Adding APC to an application usually results in improved application response times, reduced server load and happier users.

In this article, I’ll introduce you to APC, guiding you through the process of installing and configuring it and showing you a few examples of how it works. I’ll also walk you through the APC administrator interface, which lets you view APC performance in real time, and show you how you can use it with the Zend Framework. So come on in, and let’s get started!

Getting Started

First up, a quick description of what APC is and how it works.

As you probably already know, PHP is an interpreted language (unlike, say, Java or C++). Whenever a client requests a PHP page, the server will read in the source code of the page, compile it into bytecode and then execute it. In the normal scenario, this process is performed on every request…although PHP is so speedy that you probably won’t even notice it!

If you’re running an application or Web site that has many hundreds or thousands of requests coming in per minute, though, you’re going to want to speed things up as much as possible. And that’s where APC comes in. In the words of its Web site, APC is “a free, open, and robust framework for caching and optimizing PHP intermediate code.” Very simply, APC caches the compiled output of each PHP script run and reuses it for subsequent requests. This reduces the time and processing cycles needed to fully satisfy each request, leading to better performance and lower response times.

Does it work? You bet (there are some benchmarks at the end of the article). And it’s easy to set up as well. To install it, use the pecl command, as shown below:

shell> pecl install apc-3.1.4

The PECL installer will now download the source code, compile it and install it to the appropriate location on your system.

Alternatively, manually download the source code archive (v3.1.4 at this time) and compile it into a loadable PHP module with phpize:

shell> cd apc-3.1.4
shell# phpize
shell# ./configure
shell# make
shell# make install

This procedure should create a loadable PHP module named apc.so in your PHP extension directory. You should now enable the extension in the php.ini configuration file, restart your Web server, and check that the extension is enabled with a quick call to phpinfo():

Windows users have a much easier time of it; pre-compiled Windows versions of php_apc.dll can be downloaded from here. Once you’ve got the file, place it in your PHP extension directory, activate it via your php.ini configuration file, and restart your Web server. You should now be able to see the extension active with phpinfo(), as described above.

Digging Deeper

The APC source code archive includes a script named apc.php. This script serves as the administrator interface for the cache, allowing you to look inside the cache at any time to view usage or inspect cached variables. It’s a good idea to get familiar with how this works before starting to write any code.

To begin, extract the script from the source code archive and copy it to your Web server document root. Then, open it in a text editor and set the administrator password (you’ll find it near the top of the script). Once you’ve got that done, try accessing the script through your Web browser. You should see something like this:

As you can see, the script provides a birds-eye view of the current cache status, displaying general cache information, usage and statistics on hits and misses. The information is presented for both the system cache (which handles opcodes) and the user cache (which handles user variables). You’ll also see some interesting data derived from these statistics, such as the number of cache requests per second, the hit rate and miss rate.

This information is useful to understand how well your cache is working, and identify areas that are under-optimized. In particular, note the cache full count value, which indicates how often the cache has filled up; if this is a high number, it’s an indication of high cache churn and suggests that you should perhaps assign more memory to the cache.

The “System Cache Entries” tab lists the PHP scripts that are currently being cached, together with their filename, size and number of hits. Note that APC will automatically cache script opcode.

The “User Cache Entries” tab lists user variables that have been stored in the cache, together with their identifier, size, and creation and modification times. You can select any of these entries to look inside the cache entry and see what it contains. Note that user cache entries must be manually created by a PHP script – you’ll see how on the next page.

Remember that you can clear the opcode cache or the user cache at any time with the “Clear cache” buttons at the top right corner of the page.

A Galaxy Far, Far Away

Now that you have a better idea of how APC works, let’s write some code. Caching user variables in APC is mainly done through the apc_add() and apc_fetch() methods, which allow variable to added to, and retrieved from, the cache respectively. Here’s a simple example that illustrates:

<?php
if ($quote = apc_fetch('starwars')) {
  echo $quote;
  echo " [cached]";
} else {
  $quote = "Do, or do not. There is no try. -- Yoda, Star Wars";
  echo $quote;
  apc_add('starwars', $quote, 120);
}
?>

Now, the first time, you run this script, the page will be generated from scratch and you’ll see something like this:

Next, try refreshing the page. This time, the output will be retrieved from the cache:

The business logic to use a cache is fairly simple. The first step is to check if the required data already exists in the cache. If it doesn’t, it should be generated from the original data source, and a copy saved to the cache for future use. If it does, you can use it right away – write it to a file, pipe it to an external program or output it to the client.

In the previous example, checking whether the data already exists in the cache is accomplished with the apc_fetch() method, while writing a fresh snapshot to the cache is done with the apc_add() method. Notice that both methods require an ID; this ID uniquely identifies the object in the cache and serves as a key for saving and retrieving cache data. The apc_add()method additionally lets you specify a duration (in seconds) for which the cache is valid.

Take a look in the administrator interface, and you should see your cached data, together with statistics on cache hits and misses:

A-rray of Sunshine

In addition to caching strings, APC also allows you to cache arrays, objects, functions and references. Consider the following example, which caches an array of values:

<?php
// if available, use cached array
// if not, create and cache array
if ($data = apc_fetch('heroes')) {
  echo 'Cached data: ';
} else {
  $data = array('Spiderman', 'Superman', 'Green Lantern', 'Human Torch');
  apc_add('heroes', $data, 120);
}
echo $data[1];  // Superman
?>

You can also cache nested or multi-dimensional arrays, as shown below:
<?php
// if available, use cached data
// if not, create and cache nested array
if ($data = apc_fetch('config')) {
  echo 'Cached data: ';
} else {
  $data = array(
    'site1' => array(
      'smtp' => array(
        'host' => '192.168.0.1',
        'user' => 'user1',
        'pass' => 'guessme'
      )
    ),
    'site2' => array(
      'smtp' => array(
        'host' => '192.168.10.10',
        'user' => 'user2',
        'pass' => 's3cr3t'
      )
    ),
  );
  apc_add('config', $data, 120);
}

// display data
echo $data['site2']['smtp']['pass']; // s3cr3t
?>

An Object Lesson

In addition to caching arrays, APC also allows you to store objects in the cache. To illustrate, consider the next example, which initializes a simple User object, stores it in the cache, and then retrieves it back from the cache:

<?php
// define class
class User {

  private $name;
  private $location;

  function setName($value) {
    $this->name = $value;
  }

  function setLocation($value) {
    $this->location = $value;
  }

  function getName() {
    return $this->name;
  }

  function getLocation() {
    return $this->location;
  }
}

// if cached object available, use cached object
// if not, create new object instance and cache it
if (apc_exists('user')) {
  $obj = apc_fetch('user');
  echo "Cached data: ";
} else {
  $obj = new User;
  $obj->setName('John');
  $obj->setLocation('London');
  apc_add('user', $obj, 120);
}

// print object properties
echo 'My name is ' . $obj->getName() . ' and I live in ' . $obj->getLocation();
?>

Here’s what the output looks like:

Another approach to arrive at the same result is to serialize the object into a string, and then store the string in the cache instead of the object. Here’s what that would look like:

<?php
// define class
class User {

  private $name;
  private $location;

  function setName($value) {
    $this->name = $value;
  }

  function setLocation($value) {
    $this->location = $value;
  }

  function getName() {
    return $this->name;
  }

  function getLocation() {
    return $this->location;
  }
}

// serialize object and cache
// retrieve from cache as needed and deserialize
if ($str = apc_fetch('user')) {
  $obj = unserialize($str);
  echo "Cached data: ";
} else {
  $obj = new User;
  $obj->setName('John');
  $obj->setLocation('London');
  $str = serialize($obj);
  apc_add('user', $str, 120);
}
echo 'My name is ' . $obj->getName() . ' and I live in ' . $obj->getLocation();
?>

Getting Closure

You can also use APC to cache references and (with a little tweaking) anonymous functions. Let’s take a look:

<?php
class Calendar {
  public $year = 2001;
  public function &getYear() {
    return $this->year;
  }
}

$obj = new Calendar;
$a = &$obj->getYear();  // 2001
$obj->year = 2010;      // 2010
apc_add('ref', $a, 60);
$ref = apc_fetch('ref');
$ref++;
echo $ref;              // 2011
?>

Anonymous functions or closures, new in PHP 5.3, offer an easy way to define functions “on the fly”. By default, however, closures cannot be cached with APC, as the Closure class does not implement serialization. Here’s a simple example that illustrates the problem:
<?php
// check if closure exists in cache
// if yes, retrieve and use
// if not, define and add to cache
if (!apc_exists('area')) {
  // simple closure
  // calculates area from length and width
  $area = function($length, $width) {
      return $length * $width;
  };
  apc_store('area', $area);
  echo 'Added closure to cache.';
} else {
  $func = apc_fetch('area');
  echo 'Retrieved closure from cache. ';
  echo 'The area of a 6x5 polygon is: ' . $func(6,5);
}
?>

When you try accessing this script, you’ll see an error about serialization, as shown below:

What’s the solution? Well, Jeremy Lindblom has extended the Closure class and created a custom SuperClosure class that supports both serialization and reflection. If you implement your closure using this class, you will be able to cache it. Here’s a revision of the previous script that demonstrates:

<?php
// include SuperClosure class
include 'SuperClosure.class.php';

// check if closure exists in cache
// if yes, retrieve and use
// if not, define and add to cache
if (!apc_exists('area')) {
  // simple closure
  // calculates area given length and width
  $area = new SuperClosure(
    function($length, $width) {
      return $length * $width;
    }
  );
  apc_store('area', $area);
  echo 'Added closure to cache.';
} else {
  $func = apc_fetch('area');
  echo 'Retrieved closure from cache. ';
  echo 'The area of a 6x5 polygon is: ' . $func(6,5);
}
?>

Here’s what the output looks like:

Note that these examples use apc_store() instead of apc_add(). In most cases, you can use these two functions interchangeably. The primary difference lies in how they behave when you attempt to store a value using an identifier that already exists in the cache: apc_add() will return false, while apc_store() will overwrite the previous value with the new value.

You can download the SuperClosure class definition from Jeremy Lindblom’s Github account.

Utility Belt

The APC extension also comes with a few other methods of note. For example, there’s theapc_inc() and apc_dec() methods, which can be used to increment or decrement cached values respectively:

<?php
// store a value
apc_store('a', 20);

// increment and decrement stored value
apc_inc('a');         // 21
apc_inc('a');         // 22
apc_inc('a');         // 23
apc_dec('a');         // 22

// retrieve final value
echo apc_fetch('a');  // 22
?>

The apc_bin_dump() method dumps the current contents of the cache in binary form, while theapc_bin_load() method loads a binary dump into the cache. Consider the following example, which illustrates:
<?php
// clear cache
apc_clear_cache();
apc_clear_cache('user');

// store some values
apc_store('a', 20001);
apc_store('b', 7494);

// dump cache in binary form
$dump = apc_bin_dump();

// clear cache
apc_clear_cache();
apc_clear_cache('user');

// try accessing a stored value
if (apc_fetch('a')) {
  echo apc_fetch('a');
} else {
  echo 'Nothing in cache';
}
// reload cache from binary dump
apc_bin_load($dump);

// try accessing a stored value
if (apc_fetch('a')) {
  echo apc_fetch('a');     // 20001
} else {
  echo 'Nothing in cache';
}
?>

The apc_clear_cache() method can be used to clear the opcode cache or the user cache:
<?php
// clear opcode cache
apc_clear_cache();

// clear user cache
apc_clear_cache('user');
?>

The apc_cache_info() method presents information on current cache status and memory allocation:
<?php
print_r(apc_cache_info());
?>

Here’s what the output looks like:

Tweet Tweet

With all this background information at hand, let’s try APC with a real-world example. This next script uses APC to cache the result of a Twitter search:

<html>
  <head>
    <style type="text/css">
      div.outer {
      	border-bottom: dashed orange 1px;
      	padding: 4px;
      	clear: both;
      	height: 50px;
      }
      div.img {
        float:left;
        padding-right: 2px;
      }
      span.attrib {
        font-style: italic;
      }
    </style>
  </head>
  <body>
    <h2>Twitter Search</h2>
    <form action="<?php echo htmlentities($_SERVER['PHP_SELF']); ?>" method="post">
    Search term: <input type="text" name="q" />
    <input type="submit" name="submit" />
    </form>
<?php
    // if form submitted
    if (isset($_POST['submit'])):
      // sanitize query terms
      $q = strip_tags($_POST['q']);

      // generate cache id from query term
      $id = md5($q);

      // check if this search already exists in cache
      // use if yes, generate fresh results and add to cache if no
      if (apc_exists($id)) {
        $records = apc_fetch($id);
      } else {
        // search Twitter for query term
        $result = simplexml_load_file("http://search.twitter.com/search.atom?q=$q&lang=en");

        // process Atom feed of search results
        $records = array();
        foreach ($result->entry as $entry) {
          $item['image'] = (string)$entry->link[1]['href'];
          $item['owner'] = (string)$entry->author->name;
          $item['uri'] = (string)$entry->author->uri;
          $item['tweet'] = (string)$entry->content;
          $item['time'] = date('d M Y, h:i', strtotime($entry->published));
          $records[] = $item;
        }

        // cache for 5 minutes
        apc_store($id, $records, 300);
      }    

      // display search results
?>
    <h2>Twitter Search Results for '<?php echo $q; ?>'</h2>
      <?php foreach ($records as $r): ?>
      <div>
        <div><img width=48" height="48" src="<?php echo $r['image']; ?>" /></div>
        <div><?php echo $r['tweet']; ?><br/>
        <span>By <a href="<?php echo $r['uri']; ?>"><?php echo $r['owner']; ?></a>
        on <?php echo $r['time']; ?></span></div>
      </div>
      <?php endforeach; ?>
    <?php endif; ?>
  </body>
</html>

Despite its length, this is actually a very simple script. It begins by creating a search form for the user to enter search terms into. Once this form is submitted, it connects to the Twitter Search API, retrieves an Atom-formatted list of search results matching the search term, process the Atom feed and render the final output as an HTML table. The results of the search are cached for five minutes, so that they can be used for subsequent searches containing the same search terms. Notice that each search query is assigned a unique identifier in the APC cache, by using its MD5 signature as key.

You will realize that there are two levels of caching in this script. First, APC’s opcode cache is automatically caching the compiled bytecode of the script, and using this cached bytecode for subsequent requests instead of recompiling it anew. Second, APC’s user cache is caching the results of each Twitter search, and reusing these results (instead of connecting to Twitter afresh) for subsequent searches containing the same query terms. As a result, subsequent searches for the same term will be served from the cache, leading to a noticeable reduction in load time (try it for yourself and see).

Here’s an example of what the output looks like:

In The Frame

If you’re a fan of the Zend Framework, you’ll be happy to hear that Zend_Cache comes with built-in support for APC, allowing you to begin using it out of the box. To illustrate, consider the following Zend Framework controller, which revises the previous example into a Zend Framework controller:

<?php
class IndexController extends Zend_Controller_Action
{
    public function indexAction()
    {
        // action body
    }

    public function searchAction()
    {
      // initialize cache
      $cache = Zend_Cache::factory( 'Core',
                                    'APC',
                                    array('lifeTime' => 300, 'automatic_serialization' => true));

      // create form and attach to view
      $form = new SearchForm();
      $this->view->form = $form;       

      // validate input
      if ($this->getRequest()->isPost()) {
        if ($form->isValid($this->getRequest()->getPost())) {
          // get sanitized input
          $values = $form->getValues();        

          // calculate MD5 hash
          $id = md5($values['q']);

          // look for records in cache
          if (!$records = $cache->load($id) ){
            // if not present in cache
            // search Twitter for query term
            $result = simplexml_load_file("http://search.twitter.com/search.atom?q=" . $values['q'] . "&lang=en");

            // process Atom feed of search results
            $records = array();
            foreach ($result->entry as $entry) {
              $item['image'] = (string)$entry->link[1]['href'];
              $item['owner'] = (string)$entry->author->name;
              $item['uri'] = (string)$entry->author->uri;
              $item['tweet'] = (string)$entry->content;
              $item['time'] = date('d M Y, h:i', strtotime($entry->published));
              $records[] = $item;
            }

            // save to cache
            $cache->save($records, $id);
          }

          // assign results to view
          $this->view->records = $records;
          $this->view->q = $values['q'];
        }
      }
    }
}

// search form
class SearchForm extends Zend_Form
{
  public function init()
  {
    // initialize form
    $this->setMethod('post');

    // create text input for search term
    $q = new Zend_Form_Element_Text('q');
    $q->setLabel('Search term:')
         ->setOptions(array('size' => '35'))
         ->setRequired(true)
         ->addValidator('NotEmpty', true)
         ->addFilter('HTMLEntities')
         ->addFilter('StringTrim');            

    // create submit button
    $submit = new Zend_Form_Element_Submit('submit');
    $submit->setLabel('Search');

    // attach elements to form
    $this->addElement($q)
         ->addElement($submit);
  }
}

Here, the searchAction() method first sets up the Zend_Cache instance, with the Core frontend and the APC backend. The form object, which extends Zend_Form, is then added to the view, together with all necessary validators and filters, and the view is rendered.

When the user submits the form, control transfers back to the action controller, which checks the input and retrieves the filtered values. It then checks the cache to see if a search result already exists for this search term, and uses it if available; if not, it connects to the Twitter Search API, retrieves a result set, and saves it to the cache for future use. The results are then rendered through the view script. On subsequent searches for the same term, the cached result set will be used, producing a much faster response.

Here’s the code for the view script:

<style type="text/css">
  div.outer {
  	border-bottom: dashed orange 1px;
  	padding: 4px;
  	clear: both;
  	height: 50px;
  }
  div.img {
    float:left;
    padding-right: 2px;
  }
  span.attrib {
    font-style: italic;
  }
</style>
<h2>Twitter Search</h2
<?php echo $this->form; ?>

<?php if ($this->records): ?>
  <h2>Twitter Search Results for '<?php echo $this->q; ?>'</h2>
  <?php foreach ($this->records as $r): ?>
  <div>
    <div><img width=48" height="48" src="<?php echo $r['image']; ?>" /></div>
    <div><?php echo $r['tweet']; ?><br/>
    <span>By <a href="<?php echo $r['uri']; ?>"><?php echo $r['owner']; ?></a>
    on <?php echo $r['time']; ?></span></div>
  </div>
  <?php endforeach; ?>
<?php endif; ?>

And here’s a sample of the output:

The Need For Speed

At this point, there’s only one question left to answer: does APC’s opcode caching really deliver the goods and produce a verifiable increase in performance?

A good way to test this is by benchmarking a PHP script with and without APC, and evaluating the performance differential if any. ApacheBench (ab) is my tool of choice for this test, and my testbed will be the default welcome page of a new Zend Framework project. You can create this by installing the Zend Framework and then using the zf command-line tool to initialize a new, empty project, like this:

shell> zf create project example

Now, turn off APC, by disabling the extension in your php.ini configuration file and restarting the Web server. Then, use ab to benchmark the application welcome page by sending it 1000 requests with a concurrency level of 5, as follows:
shell> ab -n 1000 -c 5 http://example.localhost/default/index/index

On my development system, this produces output like the following:

The main numbers to look at here are the requests per second and the average time per request. The lower the average time per request, the better the performance. Similarly, the greater the number of requests served, the better the performance.

Next, re-enable APC, restart the Web server and try the test again. On my development system, this produces output like the following:

As you can see, enabling APC has resulted in an almost 185% increase in performance, with the server now being able to handle 71 requests per second (up from 25 earlier) and the average time per request coming down to 69 ms (from 194 ms earlier).

The above test was run with APC’s default settings. However, APC comes with a number of configuration settings that you can tweak further to squeeze even better performance from it. Here are some of the important ones:

  • ‘apc.shm_size’ controls the size of the APC memory cache;
  • ‘apc.stat’ controls whether APC checks each script to see if it has been modified and needs to be recompiled and recached;
  • ‘apc.optimization’ determines the degree of optimization to apply;
  • ‘apc.filters’ specifies which files should be cached;
  • ‘apc.write_lock’ places an exclusive lock for caching compiled script bytecode;
  • ‘apc.lazy_functions’ and ‘apc.lazy_classes’ enables lazy loading for functions and classes.

You can read more about these and other configuration directives here.

That’s about all I have for the moment. I hope this tutorial has given you some insight into how APC works, and how you can use it to improve the performance of your PHP applications. Try it out the next time you have a performance optimization problem, and see what you think!

有同事分享了array_merge与+的区别,以前也遇到过,备忘如下:array_merge和+,遇到相同key的elem时,array_merge是覆盖,而+是抛弃后面的值。

<?php

$arr1 = array(‘a’ => 1);
$arr2 = array(‘a’ => 2);

var_dump(array_merge($arr1, $arr2));
var_dump($arr1 + $arr2);

[chengy@jx-uc-rd00.jx.baidu.com test]$ ./test.php
array(1) {
[“a”]=>
int(2)
}
array(1) {
[“a”]=>
int(1)
}