Calculation of service performance grades in a multi-node environment that hosts the services Number:7,437,459 from the United States Patent and Trademark Office (PTO) owispatent

Title: Calculation of service performance grades in a multi-node environment that hosts the services

Abstract: An approach generates performance grades. A performance grade is a value or set of values that indicates the relative work performance offered by a resource, such as a server in a multi-node system, relative to other resources. Performance grades are used to dynamically allocate work within a multi-node computer system. For example, performance grades are used for connection-time balancing, and to direct a database connection request to a multi-node database instance that a performance grade indicates provides better performance. Performance grade deltas are generated that reflect the incremental change in performance grade for additional sessions or work. Performance deltas are used to direct incoming sessions, to more optimally balance workload, and to deal with logon storms where a large number of connect requests arrive in a very short time period.

Patent Number: 7,437,459 Issued on 10/14/2008 to Chidambaran,   et al.

---

Inventors:	Chidambaran; Lakshminarayanan (Sunnyvale, CA), Kantarjiev; Christopher A. (Palo Alto, CA), Stamos; James W. (Saratoga, CA), Povinec; Peter (Redwood City, CA)
Assignee:	Oracle International Corporation (Redwood Shores, CA)
Appl. No.:	10/917,661
Filed:	August 12, 2004

---

Related U.S. Patent Documents

---

	Application Number	Filing Date	Patent Number	Issue Date
	60500096	Sep., 2003
	60500050	Sep., 2003
	60495368	Aug., 2003

---

Current U.S. Class:	709/226
Current International Class:	G06F 15/173 (20060101); G06F 12/00 (20060101)
Field of Search:	709/223,224,229,250 708/105 718/105

---

References Cited [Referenced By]

---

U.S. Patent Documents

2883255	April 1959	Anderson
5774668	June 1998	Choquier et al.
5890167	March 1999	Bridge, Jr. et al.
5918059	June 1999	Tavallaei et al.
5951694	September 1999	Choquier et al.
6035379	March 2000	Raju et al.
6041357	March 2000	Kunzelman et al.
6088728	July 2000	Bellemore et al.
6178529	January 2001	Short et al.
6243751	June 2001	Chatterjee et al.
6327622	December 2001	Jindal et al.
6556659	April 2003	Bowman-Amuah
6587866	July 2003	Modi et al.
6601101	July 2003	Lee et al.
6728748	April 2004	Mangipudi et al.
6816907	November 2004	Mei et al.
7024394	April 2006	Ashour et al.
7058957	June 2006	Nguyen
7174379	February 2007	Agarwal et al.
7178050	February 2007	Fung et al.
7263590	August 2007	Todd et al.
7269157	September 2007	Klinker et al.
7292961	November 2007	Dias et al.
2001/0056493	December 2001	Mineo
2002/0073139	June 2002	Hawkins et al.
2002/0129157	September 2002	Varsano
2002/0161896	October 2002	Wen et al.
2002/0194015	December 2002	Gordon et al.
2003/0005028	January 2003	Dritschler et al.
2003/0007497	January 2003	March et al.
2003/0014523	January 2003	Teloh et al.
2003/0063122	April 2003	Cichowlas et al.
2003/0088671	May 2003	Klinker et al.
2003/0108052	June 2003	Inoue et al.
2003/0135642	July 2003	Benedetto et al.
2003/0208523	November 2003	Gopalan et al.
2004/0024979	February 2004	Kaminsky et al.
2004/0111506	June 2004	Kundu et al.
2004/0117794	June 2004	Kundu
2004/0176996	September 2004	Powers et al.
2004/0268357	December 2004	Joy et al.
2005/0021771	January 2005	Kaehn et al.
2005/0038800	February 2005	Chidambaran et al.
2005/0038833	February 2005	Colrain et al.
2005/0055383	March 2005	Dias et al.
2005/0086242	April 2005	Ngai et al.
2005/0086263	April 2005	Ngai et al.
2005/0165925	July 2005	Dan et al.
2005/0239476	October 2005	Betrabet et al.
2005/0267965	December 2005	Heller
2006/0036617	February 2006	Bastawala et al.
2006/0161816	July 2006	Gula et al.
2007/0226323	September 2007	Halpern

Foreign Patent Documents

	0 942 363		Sep., 1999		EP
	0942363		Sep., 1999		EP
	0 992 909		Apr., 2000		EP
	1 170 662		Jan., 2002		EP
	WO 02/05116		Jan., 2002		WO
	WO 02/07037		Jan., 2002		WO
	WO 02/07037		Jan., 2002		WO
	WO 02/097676		Dec., 2002		WO
	WO 03/014928		Feb., 2003		WO
	WO 03/014928		Feb., 2003		WO
	WO 03/062983		Jul., 2003		WO

Other References

Ying-Dar Lin et al., "Direct Web Switch Routing with State Migration, TCP Masquerade, and Cookie Name Rewriting," Globecom 2003, IEEE Global Telecommunications Conference, Dec. 12003, IEEE, CP010677300, pp. 3663-3667. cited by other . Current Claims of International Application No. PCT/US2004/026570, 4 pages. cited by other . Current Claims of International Application No. PCT/US2004/025805, 8 pages. cited by other . Current Claims of International Application No. PCT/US04/26570, 4 pages. cited by other . Current Claims for International Application No. PCT/US2004/026389, pp. 1-7. cited by other . Current Claims for International Application No. PCT/US2004/026445, pp. 1-5. cited by other . Current Claims, PCT/US2004/026405, 7 pages. cited by other . Chase, Jeffrey S., "Dynamic Virtual Clusters in a Grid Site Manager," Proceedings of the IEEE International Symposium on HPDC-2003, XP010463715, pp. 90-100. cited by other . Claims, PCT/US2004/026405, 7 pages. cited by other . Claims, App. No. 04781256.3, pp. 34-36. cited by other . Ravi Kokku et al., "Half-pipe Anchoring: An Efficient Technique for Multiple Connection Handoff," Proceedings 10.sup.th International Conference on Network Protocols, Nov. 12, 2002, XP010632563, 10 pages. cited by other . Ying-Dar Lin et al., "Direct Web Switch Routing with State Migration, TCP Masquerade, and Cookie Name Rewriting," Globecom 2003, IEEE Global Telecommunications Conference, Dec. 12003, IEEE, CP010677300, pp. 3663-3667. (Dec. 2003). cited by other . International Searching Authority, "Notification of Transmittal of the International Search Report and the Written Opinion of the International Searching Authority, or the Declaration," Nov. 12, 2004, 13 pages. cited by other . Current Claims of International Application No. PCT/US2004/026570, 4 pages. (2004). cited by other . Current Claims of International Application No. PCT/US2004/0025805, 8 pages. (2004). cited by other . International Searching Authority, "Notification of Transmittal of the International Search Report and the Written Opinion of the International Searching Authority, or the Declaration," dated Dec. 3, 2004, 12 pages. cited by other . International Searching Authority, "Notification of Transmittal of the International Search Report and the Written Opinion of the International Searching Authority, or the Declaration, " PCT/US2004/026405, dated Jan. 6, 2006, 13 pages. cited by other . Current Claims, PCT/US2004/026405, 7 pages. (2004). cited by other . "Office Action" received in U.S. Appl. No. 11/057,043 dated Apr. 9, 2008, 19 pages. cited by other . "Office Action" received in related case U.S. Appl. No. 10/918,056, filed Aug. 12, 2004, 9 pages. cited by other . "Office Action" received in related case U.S. Appl. No. 10/917,687, filed Aug. 12, 2004, 9 pages. cited by other . European Patent Office, "Communication Pursuant to Article 96(2) EPC," App. No. 04781256.3, dated Feb. 2, 2007, 4 pages. cited by other . Claims, App. No. 04781256.3, pp. 34-36. (Feb. 10, 2006). cited by other . Kokku, Ravi et al., "Half-Pipe Anchoring: An Efficient Technique for Multiple Connection Handoff," Proceedings of the 10.sup.th International Conference of Network Protocols, Nov. 12, 2002, 10 pages. cited by other . International Preliminary Examining Authority, "Written Opinion of the International Preliminary Examining Authority," Jun. 7, 2005, 4 pages. cited by other . Current Claims of International Application No. PCT/US04/26570, 4 pages. (2004). cited by other . Current Claims for International Application No. PCT/US2004/026389, pp. 1-7. (2004). cited by other . Current Claims for International Application No. PCT/US2004/026445, pp. 1-5. (2004). cited by other . Henry Song, et al., "Browser State Repository Service," Lecture Notes in Computer Science, vol. 2414, 2002, pp. 1-14, XP002904339. cited by other . Eric Skow, et al., "A Security Architecture for Application Session Handoff," 2002, IEEE International Conference Proceedings, Apr. 28-May 2, 2002, vol. 1 of 5, pp. 2058-2063, XP010589848. cited by other . Chase, Jeffrey S., "Dynamic Virtual Clusters in a Grid Site Manager," Proceedings of the IEEE International Symposium on HPDC-2003, XP010463715, pp. 90-100 (2003). cited by other . International Searching Authority, "Notification of the Transmittal of the International Search Report and the Written Opinion of the International Searching Authority, or the Declaration", PCT/US2004/026405, dated Aug. 19, 2005, 5 pages. cited by other . Claims, PCT/US2004/026405, 7 pages. (2004). cited by other.

Primary Examiner: Maung; Zarni
Attorney, Agent or Firm: Hickerman Palermo Truong & Becker

---

Parent Case Text

---

RELATED CASES

The present application claims priority to U.S. Provisional Application No. 60/495,368, Computer Resource Provisioning, filed on Aug. 14, 2003, which is incorporated herein by reference; the present application claims priority to U.S. Provisional Application No. 60/500,096, Service Based Workload Management and Measurement in a Distributed System, filed on Sep. 3, 2003, which is incorporated herein by reference; the present application claims priority to U.S. Provisional Application No. 60/500,050, Automatic And Dynamic Provisioning Of Databases, filed on Sep. 3, 2003, which is incorporated herein by reference.

The present application is related to the following U.S. Applications:

U.S. application Ser. No. 10/718,747, Automatic and Dynamic Provisioning of Databases filed on Nov. 21,2003, which is incorporated herein by reference;

U.S. application Ser. No. 10/917,873, Hierarchical Management of the Dynamic Allocation of Resources in a Multi-Node System, filed by Benny Souder, et al. on the equal day herewith, and incorporated herein by reference;

U.S. application Ser. No. 10/917,953, Transparent Session Migration Across Servers, filed by Sanjay Kaluskar, et al. on the equal day herewith and incorporated herein by reference;

U.S. application Ser. No. 10/918,055, Incremental Run-Time Session Balancing in a Multi-Node System filed by Lakshminarayanan Chidambaran, et al. on the equal day herewith and incorporated herein by reference;

U.S. application Ser. No. 10/918,056, Service Placement for Enforcing Performance and Availability Levels in a Multi-Node System, filed by Lakshminarayanan Chidambaran, et al. on the equal day herewith and incorporated herein by reference;

U.S. application Ser. No. 10/917,687, On Demand Node and Server Instance Allocation and De-Allocation, filed by Lakshminarayanan Chidambaran, et al. on the equal day herewith and incorporated herein by reference;

U.S. application Ser. No. 10/918,054, Recoverable Asynchronous Message Driven Processing in a Multi-Node System, filed by Lakshminarayanan Chidambaran, et al. on the equal day herewith and incorporated herein by reference; and

U.S. application Ser. No. 10/917,715, Managing Workload by Service, filed by Carol Colrain, et al. on the equal day herewith and incorporated herein by reference.

---

Claims

---

What is claimed is:

1. A method used for managing resources in a multi-node system, the method comprising the steps of: generating a plurality of performance grades that each indicate relative performance that can be realized for categories of work performed by a plurality of servers, wherein each server of said plurality of servers is running on a node in said multi-node system, and comprises integrated software components executed by at least one process running on said particular node, wherein each node in said multiple node system is a computing element interconnected to at least another node of said multiple node system, and wherein each server of said plurality of servers is associated with a performance grade of the plurality of performance grades; wherein each performance grade of said plurality of performance grades indicates, for a particular category of work, relative performance that may be realized on a server relative to performance that may be realized on other servers of said plurality of servers; and without allocating a new server to perform a category of said categories of work, adjusting allocation of said plurality of servers between said categories of work based on said performance grades.

2. The method of claim 1, further including generating performance deltas, wherein each performance delta is associated with a certain performance grade of said plurality of performance grades and indicates a change to the certain performance grade caused by adjusting allocation of work to the server associated with the performance grade.

3. The method of claim 2, wherein the step of generating performance deltas includes generating a quotient of: the difference between a revised performance grade for the particular category of work and a previous value for the revised performance grade, divided by the difference of a number of sessions associated with the revised performance grade and a previous value of a number sessions for the particular category of work.

4. The method of claim 2, wherein adjusting allocation of said plurality of servers between said categories of work includes: adjusting allocation of a first server of said plurality of servers for a first category of work; wherein said first server is associated with a first performance grade and a first performance delta; and in response to allocating said first server, changing the first performance grade based on the first performance delta.

5. The method of claim 4, wherein: adjusting allocation of a first server of said plurality of servers for a first category of work includes establishing a new session on said first server; and changing the first performance grade includes changing the first performance grade to reflect the establishment of said new session.

6. The method of claim 4, wherein: adjusting allocation of a first server of said plurality of servers for a first category of work includes causing termination of a certain session on said first server; and changing the first performance grade includes changing the first performance grade to reflect the termination of said certain session.

7. The method of claim 2, wherein: the step of generating a plurality of performance grades is based on a mathematical function; and the step of generating performance deltas is based on one or more derivatives of said mathematical function.

8. The method of claim 7, wherein a derivative of said one or more derivatives is calculated based on performance metrics generated for said categories of work.

9. The method of claim 1, wherein the step of generating a plurality of performance grades is based on one or more of the following: a value representing a proportion of CPU usage, on a certain server of said plurality Of servers, realized for a certain category of work per each request of a subset of requests associated with said certain category of work; and a value representing an amount of time that elapses for said each request.

10. The method of claim 9, wherein the step of generating a plurality of performance grades is based on a mathematical function X/Y, wherein: X represents said proportion of CPU usage; and Y represents said amount of time that elapses for said each request.

11. The method of claim 10, further including generating performance deltas, wherein: each performance delta of said performance deltas indicates a change to a performance grade caused by allocating work to the server associated with the performance grade; and the step of generating performance deltas is based on a derivative of said mathematical function.

12. The method of claim 1, wherein: a first system component performs the step of generating a plurality of performance grades; the steps further include said first system component transmitting to a second system component said plurality of performance grades; and said second system component performs the step of adjusting allocation of said plurality or servers.

13. The method of claim 12, wherein: the steps further include said first system component transmitting to said second system component information about a violation of one or more service-level agreements for said categories of work that occurs on said plurality of servers; and said second system component performs the step of adjusting allocation of said plurality of servers based on said information.

14. The method of claim 12, wherein: the steps further include said first system component transmitting to said second system component certain information about a certain category of work of said categories of work that is hosted on said plurality of servers; and said second system component performs the step of adjusting allocation of said plurality Of servers based on said certain information about a certain category of work.

15. The method of claim 14, wherein said certain information includes information about at least one of the following: that a particular server is no longer allocated to host the certain category of work; and that a particular performance grade has not been generated for the certain category Of work hosted on a particular server.

16. The method of claim 1, wherein: a first system component performs the step of generating a plurality of performance grades, wherein the step of generating a plurality of performance grades includes generating a first version of performance grades of the said plurality of performance grades and a second version of performance grades of said plurality of performance grades; the steps further include generating a plurality of performance grades that each indicate relative performance that can be realized for categories of work performed by a plurality of servers; the steps further include said first system component transmitting to a second system component said plurality of performance grades; said second system component performs the step of adjusting allocation of said plurality of servers; and the steps further include: said second system component receiving said first version of performance grades and storing a copy of said first version of performance grades, subsequent and separate to receiving said first version of performance grades, said second system component receiving said second version of performance grades, said second system component receiving performance deltas, wherein each performance delta is associated with a certain performance grade of said first version of performance grades and indicates a change to the certain performance grade caused by allocating work to the server associated with the performance grade, and in response to adjusting allocation of said plurality of servers before receiving said second version of performance grades, said second system component adjusting said copy of said first version of performance grades based on said performance deltas.

17. A computer-readable storage medium storing instructions, the instructions including instructions which, when executed by one or more processors, cause the one or more processors to perform the steps of: Generating a plurality of performance grades that each indicate relative performance that can be realized for categories of work performed by a plurality of servers, wherein each server of said plurality of servers is running on a node in said multi-node system and comprises integrated software components executed by at least one process running on said particular node, wherein each node in said multiple node system is a computing element interconnected to at least another node of said multiple node system, and wherein each server of said plurality of servers is associated with a performance grade of the plurality of performance grades; wherein each performance grade of said plurality of performance grades indicates, for a particular category of work, relative performance that may be realized on a server relative to performance that may be realized on other servers of said plurality of servers; and without allocating a new server to perform a category of said categories of work, adjusting allocation of said plurality of servers between said categories of work based on said performance grades.

18. The computer-readable storage medium of claim 17 wherein the instructions further include the step of generating performance deltas, wherein each performance delta is associated with a certain performance grade of said plurality of performance grades and indicates a change to the certain performance grade caused by adjusting allocation of work to the server associated with the performance grade.

19. The computer-readable storage medium of claim 18, wherein the step of generating performance deltas includes generating a quotient of: the difference between a revised performance grade for the particular category of work and a previous value for the revised performance grade, divided by the difference of a number of sessions associated with the revised performance grade and a previous value of a number sessions for the particular category of work.

20. The computer-readable storage medium of claim 18, wherein adjusting allocation of said plurality of servers between said categories of work includes: adjusting allocation of a first server of said plurality of servers for a first category of work; wherein said first server is associated with a first performance grade and a first performance delta; and in response to allocating said first server, changing the first performance grade based on the first performance delta.

21. The computer-readable storage medium of claim 20, wherein: adjusting allocation of a first server of said plurality of servers for a first category of work includes establishing a new session on said first server; and changing the first performance grade includes changing the first performance grade to reflect the establishment of said new session.

22. The computer-readable storage medium of claim 20, wherein: adjusting allocation of a first server of said plurality of servers for a first category of work includes causing termination of a certain session on said first server; and changing the first performance grade includes changing the first performance grade to reflect the termination of said certain session.

23. The computer-readable storage medium of claim 18, wherein: the step of generating a plurality of performance grades is based on a mathematical function; and the step of generating performance deltas is based on one or more derivatives of said mathematical function.

24. The computer-readable storage medium of claim 23, wherein a derivative of said one or more derivatives is calculated based on performance metrics generated for said categories of work.

25. The computer-readable storage medium of claim 17, wherein the step of generating a plurality of performance grades is based on one or more of the following: a value representing a proportion of CPU usage, on a certain server of said plurality of servers, realized for a certain category of work per each request of a subset of requests associated with said certain category of work; and a value representing an amount of time that elapses for said each request.

26. The computer-readable storage medium of claim 25, wherein the step of generating a plurality of performance grades is based on a mathematical function X/Y, wherein: X represents said proportion of CPU usage; and Y represents said amount of time that elapses for said each request.

27. The computer-readable storage medium of claim 26, wherein the instructions further include generating performance deltas, wherein: each performance delta of said performance deltas indicates a change to a performance grade caused by allocating work to the server associated with the performance grade; and the step of generating performance deltas is based on a derivative of said mathematical function.

28. The computer-readable storage medium of claim 17, wherein: a first system component performs the step of generating a plurality of performance grades; the steps further include said first system component transmitting to a second system component said plurality of performance grades; and said second system component performs the step of adjusting allocation of said plurality of servers.

29. The computer-readable storage medium of claim 28, wherein: the steps further include said first system component transmitting to said second system component information about a violation of one or more service-level agreements for said categories of work that occurs on said plurality of servers; and said second system component performs the step of adjusting allocation of said plurality of servers based on said information.

30. The computer-readable storage medium of claim 28, wherein: the steps further include said first system component transmitting to said second system component certain information about a certain category of work of said categories of work that is hosted on said plurality of servers; and said second system component performs the step of adjusting allocation of said plurality of servers based on said certain information about a certain category of work.

31. The computer-readable storage medium of claim 30, wherein said certain information includes information about at least one of the following: that a particular server is no longer allocated to host the certain category of work; and that a particular performance grade has not been generated for the certain category of work hosted on a particular server.

32. The computer-readable storage medium of claim 17, wherein: a first system component performs the step of generating a plurality of performance grades, wherein the step of generating a plurality of performance grades includes generating a first version of performance grades of the said plurality of performance grades and a second version of performance grades of said plurality of performance grades; the steps further include generating a plurality of performance grades that each indicate relative performance that can be realized for categories of work performed by a plurality of servers; the steps further include said first system component transmitting to a second system component said plurality of performance grades; said second system component performs the step of adjusting allocation of said plurality of servers; and the steps further include: said second system component receiving said first version of performance grades and storing a copy of said first version of performance grades, subsequent and separate to receiving said first version of performance grades, said second system component receiving said second version of performance grades, said second system component receiving performance deltas, wherein each performance delta is associated with a certain performance grade of said first version of performance grades and indicates a change to the certain performance grade caused by allocating work to the server associated with the performance grade, and in response to adjusting allocation of said plurality of servers before receiving said second version of performance grades, said second system component adjusting said copy of said first version of performance grades based on said performance deltas.

---

Description

---

FIELD OF THE INVENTION

The present invention relates to work load management, and in particular, work load management within a multi-node computer system.

BACKGROUND OF THE INVENTION

Enterprises are looking at ways of reducing costs and increasing efficiencies of their data processing system. A typical enterprise data processing system allocates individual resources for each of the enterprise's applications. Enough resources are acquired for each application to handle the estimated peak load of the application. Each application has different load characteristics; some applications are busy during the day; some others during the night; some reports are run once a week and some others once a month. As a result, there is a lot of resource capacity that is left unutilized. Grid computing enables the utilization or elimination of this unutilized capacity. In fact, grid computing is poised to drastically change the economics of computing.

A grid is a collection of computing elements that provide processing and some degree of shared storage; the resources of a grid are allocated dynamically to meet the computational needs and priorities of its clients. Grid computing can dramatically lower the cost of computing, extend the availability of computing resources, and deliver higher productivity and higher quality. The basic idea of grid computing is the notion of computing as a utility, analogous to the electric power grid or the telephone network. A client of the grid does not care where its data is or where the computation is performed. All a client wants is to have computation done and have the information delivered to the client when it wants.

This is analogous to the way electric utilities work; a customer does not know where the generator is, or how the electric grid is wired. The customer just asks for electricity and gets it. The goal is to make computing a utility--a ubiquitous commodity. Hence it has the name, the grid.

This view of grid computing as a utility is, of course, a client side view. From the server side, or behind the scenes, the grid is about resource allocation, information sharing, and high availability. Resource allocation ensures that all those that need or request resources are getting what they need. Resources are not standing idle while requests are left unserviced. Information sharing makes sure that the information clients and applications need is available where and when it is needed. High availability ensures that all the data and computation must always be there--just as a utility company must always provide electric power.

Grid Computing for Databases

One area of computer technology that can benefit from grid computing is database technology. A grid can support multiple databases and dynamically allocate and reallocate resources as needed to support the current demand for each database. As the demand for a database increases, more resources are allocated for that database, while other resources are deallocated from another database. For example, on an enterprise grid, a database is being serviced by one database server running on one server blade on the grid. The number of users requesting data from the database increases. In response to this increase in the demand for the database, a database server for another database is removed from one server blade and a database server for the database experiencing increased user requests is provisioned to the server blade.

Grid computing for databases can require allocation and management of resources at different levels. At a level corresponding to a single database, the performance and resource availability provided to the users of the database must be monitored and resources of the database allocated between the users to ensure performance and resource availability goals for each of the users are met. Between databases, the allocation of a grid's resources must be managed to ensure that performance and resource availability goals for users of all the databases are met.

A mechanism that can allocate resources at these different levels is described in Hierarchical Management of the Dynamic Allocation of Resources in a Multi-Node System Ser. No. 10/917,873. One measure employed by this mechanism to manage allocation of resources used for a database is referred to as session balancing. Session balancing is used to balance workload between the servers of a multi-node server that manages access to a particular database. Each database server that comprises the multi-node server is referred to herein as a database instance. Session balancing entails balancing a number of sessions between database instances to optimize performance realized by a user and ensures requirements for performance are being met.

Two forms of session balancing that may be used are connection-time session balancing and run-time session balancing. Under run-time session balancing, sessions on one database instance, where workload may be too high, are moved to a target database instance that offers relatively superior performance. Under connection-time balancing, a user requesting a connection to a multi-node database server is connected to a database instance that provides better performance for the user.

In both types of session balancing, a database instance is selected based on whether the database instance can provide relatively superior performance for a user. Therefore, there is a need to measure and determine which server from a set of servers can provide relatively superior performance for a user.

Approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements and in which:

FIG. 1 is a block diagram showing a multi-node computer system on which an embodiment of the present invention may be implemented.

FIG. 2 is a block diagram showing performance metrics generated according to an embodiment of the present invention.

FIG. 3 is a flow chart showing a procedure for generating performance grades and performance deltas according to an embodiment of the present invention.

FIG. 4 is a diagram illustrating a mathematical function used to calculate performance grades and performance deltas according to an embodiment of the present invention.

FIG. 5 is a block diagram showing performance metrics generated according to an embodiment of the present invention.

FIG. 6 is a flow chart showing a procedure for generating performance grades and performance deltas according to an embodiment of the present invention.

FIG. 7 is a block diagram of a computer system that may be used in an embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

A method and apparatus for managing the allocation of resources in a multi-node environment is described. In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be apparent, however, that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the present invention.

Described herein are approaches for generating "performance grades." A performance grade is a value or set of values that indicate the relative work performance offered by a resource, such as a server in a multi-node computer system, relative to other resources. Performance grades are used to dynamically allocate work within a multi-node computer system. For example, performance grades are used for connection-time balancing, to direct a database connection request to a multi-node database instance that a performance grade indicates provides better performance.

FIG. 1 shows a multi-node computer system that may be used to implement an embodiment of the present invention. Referring to FIG. 1, it shows cluster farm 101. A cluster farm is a set of nodes that is organized into groups of nodes, referred to as clusters. Clusters provide some degree of shared storage (e.g. shared access to a set of disk drives) between the nodes in the cluster. Cluster farm 101 includes clusters 110, 170, and 180. Each of the clusters hosts one or more multi-node database servers that provide and manage access to databases. The nodes in a cluster farm may be in the form of computers (e.g. work stations, personal computers) interconnected via a network, and may be part of a grid.

Clusters and Multi-Node Database Servers

Clusters 110, 170, and 180 host one or more multi-node database servers. Cluster 110 hosts a multi-node database server for database 150, the multi-node database server comprising database instances 123, 125, and 127, which are hosted on nodes 122, 124, and 126, respectively. Cluster 110 also hosts a multi-node database server for database 160, the multi-node database server comprising database instances 133, 135, and 137 hosted on nodes 132, 134, and 136, respectively.

A server, such as a database server, is a combination of integrated software components and an allocation of computational resources, such as memory, a node, and processes on the node for executing the integrated software components on a processor, the combination of the software and computational resources being dedicated to performing a particular function on behalf of one or more clients. Among other functions of database management, a database server governs and facilitates access to a particular database, processing requests by clients to access the database.

Resources from multiple nodes in a multi-node computer system can be allocated to running a particular server's software. Each combination of the software and allocation of the resources from a node is a server that is referred to herein as a "server instance" or "instance". Thus, a multi-node server comprises multiple server instances that can run on multiple nodes. Several instances of a multi-node server can even run on the same node. A multi-node database server comprises multiple "database instances", each database instance running on a node, and governing and facilitating access to a particular database. Database instances 123, 125, and 127 are instances of the same multi-node database server.

A service is work of a particular type or category that is hosted for the benefit of one or more clients. The work performed as part of a service includes any use or expenditure of computer resources, including, for example, CPU processing time, storing and accessing data in volatile memory, read and writes from and to persistent storage (i.e. disk space), and use of network or bus bandwidth.

One type of service is a database service. Cluster 110 provides a database service for accessing database 150 and a database service for accessing database 160. In general, a database service is work that is performed by a database server for a client, work that typically includes processing queries that require access to a particular database.

Like any service, a database service may be further categorized. Database services for database 150 are further categorized into the FIN service and PAY service. The FIN service is the database service performed by database instances 123 and 125 for the FIN application. Typically, this service involves accessing database objects on database 150 that store database data for FIN applications. The PAY services are database services performed by database instances 125 and 127 for the PAY application. Typically, this service involves accessing database objects on database 150 that store database data for PAY applications.

Sessions Established for Clients by Listener

In order for a client to interact with a database server on cluster 110, a session is established for the client. A session, such as a database session, is a particular connection established for a client to a server, such as a database instance, through which the client issues a series of requests (requests for execution of database statements). For each database session established on a database instance, session state data is maintained that reflects the current state of a database session. Such information contains, for example, the identity of the client for which the session is established, and temporary variable values generated by processes executing software within the database session.

A client establishes a database session by transmitting a database connection request to cluster 110. A listener, such as listener 190, receives the database connection request. Listener 190 is a process running on cluster 110 that receives client database connection requests and directs them to a database instance within cluster 110. The client connection requests received are associated with a service (e.g. service FIN and PAY). The client request is directed to a database instance hosting the service, where a database session is established for the client. Listener 190 directs the request to the particular database instance and/or node in a way that is transparent to the application.

Listener 190 performs connection-time balancing. When listener 190 receives a connection request for a user associated with a particular service, listener 190 directs the connection request to a database instance that offers relatively superior performance for the service.

Listener 190 may be running on any node within cluster 110. Once the database session is established for the client, the client may issue additional requests, which may be in the form of function or remote procedure invocations, and which include requests to begin execution of a transaction, to execute queries, to perform updates and other types of transaction operations, to commit or otherwise terminate a transaction, and to terminate a database session.

Monitoring Workload

Resources are allocated and re-allocated to meet levels of performance and cardinality constraints on the resources. Levels of performance and resource availability established for a particular service are referred to herein as service-level agreements. Levels of performance and cardinality constraints on resources that apply to a multi-node system in general and not necessarily to a particular service are referred to herein as policies. For example, a service-level agreement for service FIN maybe require as a level of performance that the average transaction time for service FIN be no more than a given threshold, and as an availability requirement that at least two instances host service FIN. A policy may require that the CPU utilization of any node should not exceed 80%.

Policies may also be referred to herein as backend policies because they are used by backend administrators to manage overall system performance and to allocate resources between a set of services when it is deemed there are insufficient resources to meet service-level agreements of all the set of services. For example, a policy assigns a higher priority to a database relative to another database. When there are insufficient resources to meet service-level agreements of services of both databases, the database with the higher priority, and the services that use the database, will be favored when allocating resources.

To meet service-level agreements, a mechanism is needed to monitor and measure workload placed on various resources. These measures of workload are used to determine whether service-level agreements are being met and to adjust the allocation of resources as needed to meet the service-level agreements.

According to an embodiment of the present invention, a workload monitor is hosted on each database instance and generates "performance metrics". Performance metrics is data that indicates the level of performance for one or more resources or services based on performance measures. Approaches for performing these functions are described in Managing Workload by Service Ser. No. 10/917,715. The information generated is accessible by various components within multi-node database server 222 that are responsible for managing the allocation of resources to meet service-level agreements, as shall be described in greater detail later.

A performance metric of a particular type that can be used to gauge a characteristic or condition that indicates a level of performance or workload is referred to herein as a performance measure. A performance measure includes for example, transaction execution time or percent of CPU utilization. In general, service-level agreements that involve levels of performance can be defined by thresholds and criteria that are based on performance measures.

For example, execution time of a transaction is a performance measure. A service-level agreement based on this measure is that a transaction for service FIN should execute within 300 milliseconds. Yet another performance measure is percentage CPU utilization of a node. A backend policy based on this measure is that a node experience no more than 80% utilization.

Performance metrics can indicate the performance of a cluster, the performance of a service running on a cluster, a node in the cluster, or a particular database instance. A performance metric or measure particular to a service is referred to herein as a service performance metric or measure. For example, a service performance measure for service FIN is the transaction time for transactions executed for service FIN.

FIG. 2 is a diagram that shows performance metrics generated by a workload monitor running on database instance 125 according to an embodiment of the present invention. Referring to FIG. 2, performance metrics table 201 reflects the performance realized by services hosted by database instance 125. Performance metrics table 201 contains columns with values that serve as a performance metric for a particular service hosted by database instance 125. Rows 202 and 203 contain performance metrics for service FIN and PAY, respectively. The table includes the columns CPU UTILIZATION, NUMBER OF SESSIONS, CPU PER CALL, and ELAPSED TIME PER CALL. CPU UTILIZATION contains values that specify a percentage of CPU utilization on node 124 used by a service. NUMBER OF SESSIONS contains values that specify a number of database sessions allocated for a service.

The columns CPU PER CALL, and ELAPSED TIME PER CALL are based on work performed to process a call. A call is a request from a client to perform a task, such as executing a database statement. A call can be commenced by invoking an API function of a database server. An example of such an API is the Oracle Call Interface ("OCI"). A client can call the function of the API directly, or via interface components used by a client to interface with a database server. Such interface components include, for example, Oracle Business Components for Java, as described in Oracle Business Components for Java, An Oracle White Paper, by Oracle Corporation.

CPU PER CALL contains values that specify an average percent CPU utilization used by a database instance to process a call made by a user associated with a service. ELAPSED TIME PER CALL contains values that specify an average period of time that elapses to process a call made by a user associated with a service.

Hierarchy Of Directors

A hierarchy of directors, such as that described in Hierarchical Management of the Dynamic Allocation of Resources in a Multi-Node System Ser. No. 10/917,873, is used to dynamically adjust the allocation of resources within cluster farm 101 to meet service-level agreements. Cluster farm 101 includes a database director for each database managed by a database server on cluster farm 101, a cluster director for each cluster within cluster farm 101, and a farm director for cluster farm 101.

A database director, such as database director 152 and 162, dynamically manages and adjusts the allocation of resources of a database between services hosted by the database instances of the database. One measure a database director uses to perform this responsibility is to perform incremental runtime session balancing between the database instances of a database as described in Incremental Run-Time Session Balancing in a Multi-Node System Ser. No. 10/918,055. Incremental runtime session ba