Internal network device dynamic health monitoring Number:7,143,153 from the United States Patent and Trademark Office (PTO) owispatent The present invention provides a method and apparatus for internal network device dynamic health monitoring. To increase network device availability, internal network device evaluations of resource attributes are conducted against threshold expressions and network managers are notified of any threshold events to allow them to address issues before failures occur. Flexibility is added by allowing users to select between various predefined threshold expressions, and flexibility is further increased by allowing users to input new threshold expressions. The user provided threshold expressions are dynamically incorporated into the network device's threshold evaluations while the network device is running. Thus, network managers may change threshold expressions at any time in accordance with their needs. The types of threshold expressions that may be used are practically limitless and may include any operand and multiple variables, including the time of day. In addition, multiple threshold expressions may be cascaded together. Network managers may be notified of threshold events in multiple manners as well. Moreover, the types of resource attributes that may be evaluated is extended by assigning unique identifiers to network device resources and allowing threshold evaluations for any identifiable resource and associated attributes.<H monitoring, Internal, Internal, networ, 7143153.html, Patent, pto, 7143153

Title: Internal network device dynamic health monitoring

Abstract: The present invention provides a method and apparatus for internal network device dynamic health monitoring. To increase network device availability, internal network device evaluations of resource attributes are conducted against threshold expressions and network managers are notified of any threshold events to allow them to address issues before failures occur. Flexibility is added by allowing users to select between various predefined threshold expressions, and flexibility is further increased by allowing users to input new threshold expressions. The user provided threshold expressions are dynamically incorporated into the network device's threshold evaluations while the network device is running. Thus, network managers may change threshold expressions at any time in accordance with their needs. The types of threshold expressions that may be used are practically limitless and may include any operand and multiple variables, including the time of day. In addition, multiple threshold expressions may be cascaded together. Network managers may be notified of threshold events in multiple manners as well. Moreover, the types of resource attributes that may be evaluated is extended by assigning unique identifiers to network device resources and allowing threshold evaluations for any identifiable resource and associated attributes.

Patent Number: 7,143,153 Issued on 11/28/2006 to Black, et al.

Inventors: Black; Darryl (Hollis, NH), Winiewicz; Anne K (Lexington, MA)
Assignee: CIENA Corporation (Linthicum, MD)

Appl. No.: 09/718,224

Filed: November 21, 2000

Current U.S. Class: 709/223 ; 709/220; 709/221; 709/222; 709/224

Current International Class: G06F 15/177 (20060101); G06F 15/173 (20060101)

Field of Search: 709/223,224,232

References Cited [Referenced By]

U.S. Patent Documents


4750136	June 1988	Arpin et al.
4942540	July 1990	Black et al.
5515403	May 1996	Sloan et al.
5638410	June 1997	Kuddes
5726607	March 1998	Brede et al.
5751964	May 1998	Ordanic et al.
5850399	December 1998	Ganmukhi et al.
5903564	May 1999	Ganmukhi et al.
5905730	May 1999	Yang et al.
5926463	July 1999	Ahearn et al.
5953314	September 1999	Ganmukhi et al.
5955946	September 1999	Beheshti et al.
5987513	November 1999	Prithviraj et al.
5991163	November 1999	Marconi et al.
5991297	November 1999	Palnati et al.
5995511	November 1999	Zhou et al.
5999179	December 1999	Kekic et al.
6008805	December 1999	Land et al.
6008995	December 1999	Pusateri et al.
6015300	January 2000	Schmidt, Jr. et al.
6021116	February 2000	Chiussi et al.
6021263	February 2000	Kujoory et al.
6033259	March 2000	Daoud
6041307	March 2000	Ahuja et al.
6044540	April 2000	Fontana
6078595	June 2000	Jones et al.
6085237	July 2000	Durham et al.
6175868	January 2001	Lavian et al.
6182157	January 2001	Schlener et al.
6356282	March 2002	Roytman et al.
6363421	March 2002	Barker et al.
6446136	September 2002	Pohlmann et al.
6456306	September 2002	Chin et al.
6611866	August 2003	Goldman
6647416	November 2003	Lorian et al.
6714977	March 2004	Fowler et al.
2002/0049838	April 2002	Sylor et al.

Foreign Patent Documents


9826611	Jun., 1998	WO
9905826	Feb., 1999	WO
9911095	Mar., 1999	WO
9914876	Mar., 1999	WO
9927688	Jun., 1999	WO
9930530	Jun., 1999	WO
9935577	Jul., 1999	WO

Other References

Microsoft Corporation, "SNMP security properties", Feb. 28, 2000, Web Page, http://www.microsoft.com/windows2000/en/server/help/sag.sub.--snmps- ecurity.htm. cited by examiner .
Cisco Systems, Inc., "Using Threshold Manager", Jan. 7, 1998, Threshold Manager Getting Started Guide, Chapter 2, Retrieved fro http://www.cisco.com/univercd/cc/td/doc/product/rtrmgmt/thresmgr/tmusing.- pdf. cited by examiner .
Cisco Systems, Inc., "Using Threshold Manager", 1997, Web Page, Retreived from http://www.cisco.com/univercd/cc/td/doc/product/rtmgmt/thresmgr/tmus- ing.htm. cited by examiner .
Cisco Systems, Inc., "About This Guide", 1997, Web Page, Retrieved from http://www.cisco.com/univercd/cc/td/doc/product/rtmgmt/thresmgr/about.htm- . cited by examiner .
Cisco Systems, Inc., "Installing Threshold Manager", 1997, Web Page, Retrieved from http://www.cisco.com/univercd/cc/td/doc/product/rtrmgmt/thresmgr/tminstl.- htm. cited by examiner .
"The Abatis Network Services Contractor," Abatis Systems Corporation product literature, 1999. cited by other .
AtiMe-3E Data Sheet, 1-17 (Mar. 8, 2000). cited by other .
Black, D., "Building Switched Networks," pp. 85-267. cited by other .
Black, D., "Managing Switched Local Area Networks A Practical Guide" pp. 324-329. cited by other .
"Configuration," Cisco Systems Inc. webpage, pp. 1-32 (Sep. 20, 1999). cit- ed by other .
Leroux, P., "The New Business Imperative: Achieving Shorter Development Cycles while Improving Product Quality", QNX Software Systems Ltd. webpage, (1999). cited by other .
NavisXtend Accounting Server, Ascend Communications, Inc. product information (1997). cited by other .
NavisXtend Fault Server, Ascend Communications, Inc. product information (1997). cited by other .
NavisXtend Provisioning Server, Ascend Communications, Inc. product information (1997). cited by other .
Network Health LAN/WAN Report Guide, pp. 1-23. cited by other .
"Optimizing Routing Software for Reliable Internet Growth," JUNOS product literature (1998). cited by other .
PMC-Sierra, Inc. website (Mar. 24, 2000). cited by other .
Raddalgoda, M., "Failure-proof Telecommunications Products: Changing Expectations About Networking Reliability with Microkernel RTOS Technology," QNX Software Systems Ltd. webpage, (1999). cited by other .
"Real-time Embedded Database Fault Tolerance on Two Single-board Computers," Polyhedra, Inc. product literature. cited by other .
"Start Here: Basics and Installation of Microsoft Windows NT Workstation," product literature (1998). cited by other .
Syndesis Limited product literature, 1999. cited by other .
"Using Polyhedra for a Wireless Roaming Call Management System," Polyhedra, Inc., (prior to May 20, 2000). cited by other .
Veritas Software Corporation webpage, 2000. cited by other.

Primary Examiner: Lim; Krishna
Assistant Examiner: Strange; Aaron
Attorney, Agent or Firm: Dougherty Clements Bernard; Christopher L. Brown; Tyler S.

Parent Case Text

This application is a continuation-in-part of U.S. Ser. No. 09/711,054, filed Nov. 9, 2000, entitled "Network Device Identity Authentication", still pending.

Claims

The invention claimed is:

1. A method of managing a telecommunications network, comprising: executing at least one application on a network device; receiving a threshold expression from a user through a user interface; and implementing the threshold expression within the network device while the network device is operational, wherein said implementing step comprises: storing the threshold expression in a configuration database, establishing at least one active query between said configuration database and said application so as to send a notification from the configuration database to the application upon occurrence of a change in said threshold expression, and linking the application to a threshold monitoring library (TML), said TML causing the application to retrieve said threshold expression from the configuration database, wherein said step of establishing the at least one active query further comprises: establishing an active query between said configuration database and said TML for providing the TML with notification of a change to said threshold expression.

2. The method of claim 1, wherein receiving a threshold expression comprises: receiving a user selection of an existing threshold expression.

3. The method of claim 1, wherein receiving a threshold expression comprises: receiving a new threshold expression from the user.

4. The method of claim 1, where the threshold expression includes a plurality of cascaded threshold expressions.

5. The method of claim 1, wherein prior to receiving a threshold expression, the method further comprises: receiving a resource selection from the user through the user interface; and displaying a threshold dialog box, wherein the threshold expression is received through the threshold dialog box.

6. The method of claim 5, further comprising: displaying a plurality of existing threshold expressions to the user through the threshold dialog box.

7. The method of claim 5, further comprising: displaying a default threshold expression to the user through the threshold dialog box.

8. A method of managing a telecommunications network, comprising: displaying a threshold dialog box to a user; receiving a threshold expression from the user through the threshold dialog box; and implementing the threshold expression within a network device while the network device is operational, wherein implementing the threshold expression within the network device while the network device is operational comprises: establishing an active query between a configuration database within the network device and at least one application capable of executing a thresholding code, writing data from the threshold dialog box into at least one table in said configuration database, linking said application to a threshold monitoring library (TML), said TML causing the application to retrieve said threshold expression from the configuration database, and updating said thresholding code with the data written into the at least one table in response to an active query notification from said configuration database to said application, wherein said step of establishing the active query further comprises: establishing an active query between said configuration database and said TML for providing the TML with notification of a change to said threshold expression.

9. The method of claim 8, wherein writing data from the threshold dialog box into at least one table in a configuration database within the network device comprises: sending the data from the threshold dialog box through a network management system (NMS) client to an NMS server; and writing the data into the at least one table through the NMS server.

10. A method of managing a telecommunications network, comprising: executing at least one application on a network device; receiving a new threshold expression from a user through a user interface; and implementing the new threshold expression within the network device while the network device is operational, wherein said implementing step comprises: updating a configuration database with the new threshold expression, establishing at least one active query between said configuration database and said application so as to send a notification from the configuration database to the application when the new threshold expression is received, and linking the application to a threshold monitoring library (TML), said TML causing the application to retrieve said new threshold expression from the configuration database wherein said step of establishing the at least one active query further comprises: establishing an active query between said configuration database and said TML for providing the TML with notification of a change to said threshold expression.

11. A method of managing a telecommunications network, comprising: displaying a plurality of existing threshold expressions through a user interface; receiving a user selection of one of the existing threshold expressions; and implementing the selected existing threshold expression within a network device while the network device is operational, wherein said implementing step comprises: storing the threshold expression in a configuration database, establishing at least one active query between said configuration database and said network device so as to send a notification from the configuration database to the network device upon occurrence of a change in said threshold expressions, and linking the application to a threshold monitoring library (TML), said TML causing the application to retrieve said threshold expression from the configuration database, wherein said step of establishing the at least one active query further comprises: establishing an active query between said configuration database and said TML for providing the TML with notification of a change to said threshold expression.

12. A method of managing a telecommunications network, comprising: executing at least one application on a network device; implementing a plurality of cascaded threshold expressions within the network device, wherein said implementing step comprises: storing the cascaded threshold expressions in a configuration database, establishing at least one active query between said configuration database and said application so as to send a notification from the configuration database to the application upon occurrence of a change in said cascaded threshold expressions, and linking the application to a threshold monitoring library (TML), said TML causing the application to retrieve said cascaded threshold expressions from the configuration database, wherein said step of establishing the at least one active query further comprises: establishing an active query between said configuration database and said TML for providing the TML with notification of a change to said cascaded threshold expressions.

13. A method of managing a telecommunications network, comprising: assigning a unique identifier to each of a plurality of resources in a network device; receiving a resource selection from a user through a user interface, wherein receiving the resource selection from the user through the user interface comprises: receiving a resource selection and a resource attribute selection from the user through a threshold dialog box; establishing a threshold evaluation for the selected resource using the unique identifier assigned to the selected resource, wherein establishing the threshold evaluation for the selected resource using the unique identifier assigned to the selected resource comprises: receiving a threshold expression from the user through the threshold dialog box; writing data, including the unique identifier assigned to the selected resource and the threshold expression, from the threshold dialog box into at least one table in a configuration database within the network device; updating thresholding code executing within the network device with the data written into the at least one table; linking said resource to a threshold monitoring library (TML), said TML causing the resource to retrieve said threshold expression from the configuration database, and establishing an active query between said configuration database and said TML for providing the TML with notification of a change to said threshold expression.

14. The method of claim 13, wherein the threshold evaluation is established within the network device and while the network device is operational.

15. The method of claim 13, wherein receiving a threshold expression from the user through the threshold dialog box comprises: receiving a user selection of an existing threshold expression.

16. The method of claim 13, wherein receiving a threshold expression from the user through the threshold dialog box comprises: receiving a new threshold expression from the user.

17. The method of claim 13, wherein writing data from the threshold dialog box into at least one table in a configuration database within the network device comprises: sending the data from the threshold dialog box through a network management system (NMS) client to an NMS server; and writing the data into the at least one table through the NMS server.

18. The method of claim 13, wherein updating thresholding code executing within the network device with the data written into the at least one table comprises: sending an active query notice to each application including the thresholding code and corresponding to the selected resource.

19. A method of managing a telecommunications network, comprising: establishing an active query between at least one record of a configuration database within a network device and an application executing a thresholding code in said device, said record storing at least one threshold expression, wherein said step of establishing the active query comprises linking the application to a threshold monitoring library (TML), said TML causing the application to retrieve said threshold expression from the configuration database, and establishing an active query between said configuration database and said TML for providing the TML with notification of a change to said threshold expression; modifying said threshold expressions while the network device is operational; sending an active query notification from the configuration database to the application indicative of said modification, wherein said application updates the thresholding code in accordance with said modification; detecting a threshold event in the application in accordance with said modified threshold expression; notifying the modified thresholding code of the threshold event; and responding to the threshold event in accordance with an action defined within the modified thresholding code.

20. The method of claim 19, wherein detecting a threshold event in an application within a network device comprises: monitoring a resource attribute continuously; and comparing the resource attribute against a threshold expression provided to the application by the thresholding code.

21. The method of claim 19, wherein responding to the threshold event in accordance with an action defined within the thresholding code comprises: notifying a network manager of the threshold event.

22. The method of claim 19, wherein notifying a network manager of the threshold event comprises: sending a notice to network management system software external to the network device.

23. The method of claim 21, wherein notifying a network manager of the threshold event comprises: sending an electronic mail message to the network manager.

24. The method of claim 21, wherein notifying a network manager of the threshold event comprises: sending a page to the network manager.

25. The method of claim 19, wherein responding to the threshold event in accordance with an action defined within the thresholding code comprises: logging the threshold event.

26. The method of claim 1, further comprising: causing the TML to poll the application to retrieve values of selected attributes of said application defined for monitoring by said threshold expressions.

27. The method of claim 26, further comprising: notifying an SNMP agent if one or more of said retrieved attribute values indicates occurrence of said threshold event.

Description

BACKGROUND

A generally accepted goal for network service providers is five nines availability or 99.999% network up time. In order to achieve this goal service providers must minimize network outages. This requires proactive monitoring of network resources such that issues may be identified and addressed before they cause network outages. Often, proactive monitoring is provided by having the network management system (NMS) software periodically poll each network device in the network and retrieve data corresponding to particular, predetermined resource attributes. By monitoring and evaluating the resource attributes, certain errors may be avoided and network up time may be increased.

To prevent NMS polling from consuming too much network bandwidth, NMS polling is typically limited to, for example, every 15 20 minutes (i.e., polling interval). The longer the polling interval, the less burden NMS polling presents to the-network and each network device. Unfortunately, errors and issues may arise during the polling interval and be undetected by the NMS. Thus, the longer the polling interval, the higher the likelihood that undetected errors may lead to network, outages.

To address the polling interval gap, many network devices include some type of internal network device monitoring, which continuously samples certain resource attributes even during the polling interval. If the values' of resource attributes exceed or fall below expected levels (i.e., thresholds), a notice (e.g., an SNMP trap) is sent to the NMS. For example, an SNMP trap may be sent each time the number of packets received by a network device during a sample period exceeds a predetermined threshold value (i.e., a measure of utilization), and with this information, a network manager may choose to increase the capacity of the network or move a portion of the services handled by that network device to another network device. This is a simple threshold monitoring mechanism. Pushing data from the network devices to the NMS in accordance with thresholds provides increased scalability over NMS polling techniques. Unfortunately, scalability is still an issue since simple threshold monitoring may result in an excessive number of notices being sent to the NMS each time a resource attribute value exceeds or falls below a threshold.

Other network devices include a more advanced monitoring system that implements the Remote Monitoring (RMON) specification. RMON is a set of Simple Network Management Protocol (SNMP) based MIBs (Management Information Bases) that define groups of Ethernet and Token Ring diagnostics. Basically, through RMON, the resource attribute data gathered within the network device is evaluated against a fixed expression including both a rising and a falling threshold. That is, instead of sending a notice each time an attribute value is above or below a threshold, notices are only sent in accordance with the expression after both thresholds have been crossed. RMON is designed to suppress false alarms/limit the amount of reporting and, thus, increase scalability over the simple threshold mechanism.

The sampling frequency, threshold values and the resource attributes checked against those threshold values are often predetermined and fixed in software. Thus, to change a sampling frequency, a threshold value or check a different resource attribute against a threshold, requires a change to and re-release of the software. To provide some flexibility, many current systems allow users to input sampling frequencies and threshold values to override the predetermined values provided in software. A user may, therefore, customize sampling frequencies and threshold values without having to change or re-release the software. Still other systems monitor the predetermined resource attributes for a certain initial period of time and then automatically set the threshold values based on the data gathered during that initial period.

Within current systems, however, the resource attributes that may be checked against threshold values remain fixed in software and to add a different resource attribute requires a software modification and a re-release. In addition, the thresholds remain simple high and/or low values or a fixed expression as in RMON. Thus, network managers have no flexibility in determining when reports representing resource attribute values are made.

SUMMARY

The present invention provides a method and apparatus for internal network device dynamic health monitoring. To increase network device availability, internal network device evaluations of resource attributes are conducted against threshold expressions and network managers are notified of any threshold events to allow them to address issues before failures occur. Flexibility is added by allowing users to select between various predefined threshold expressions, and flexibility is further increased by allowing users to input new threshold expressions. The user provided threshold expressions are dynamically incorporated into the network device's threshold evaluations while the network device is running. Thus, network managers may change threshold expressions at any time in accordance with their needs. The types of threshold expressions that may be used are practically limitless and may include any operand and multiple variables, including the time of day. In addition, multiple threshold expressions may be cascaded together. Network managers may be notified of threshold events in multiple manners as well. Moreover, the types of resource attributes that may be evaluated is extended by assigning unique identifiers to network device resources and allowing threshold evaluations for any identifiable resource and associated attributes.

In one aspect, the present invention provides a method of managing a telecommunications network including receiving a threshold expression from a user through a user interface and implementing the threshold expression within a network device while the network device is operational. Receiving a threshold expression may include receiving a user selection of an existing threshold expression or receiving a new threshold expression from the user. The threshold expression may include a plurality of cascaded threshold expressions. Prior to receiving a threshold expression, the method may further comprise receiving a resource selection from the user through the user interface and displaying a threshold dialog box, where the threshold expression is received through the threshold dialog box. The method may also include displaying a plurality of existing threshold expressions to the user through the threshold dialog box and displaying a default threshold expression to the user through the threshold dialog box. Implementing the threshold expression within a network device while the network device is operational may include writing data from the threshold dialog box into at least one table in a configuration database within the network device and updating thresholding code executing within the network device with the data written into the at least one table. Writing data from the threshold dialog box into at least one table in a configuration database within the network device may include sending the data from the threshold dialog box through a network management system (NMS) client to an NMS server and writing the data into the at least one table through the NMS server. Updating thresholding code executing within the network device with the data written into the at least one table may include sending an active query notice to each application including the thresholding code and corresponding to the selected resource.

In another aspect, the present invention provides a method of managing a telecommunications network including receiving, a new threshold expression from a user through a user interface and implementing the new threshold expression within a network device while the network device is operational.

In yet another aspect, the present invention provides a method of managing a telecommunications network including displaying a plurality of existing threshold expressions through a user interface, receiving a user selection of one of the existing threshold expressions and implementing the selected existing threshold expression within a network device while the network device is operational.

In still another aspect, the present invention provides a method of managing a telecommunications network including implementing a plurality of cascaded threshold expressions within a network device.

In another aspect, the present invention provides a method of managing a telecommunications network including assigning a unique identifier to each of a multiple of resources in a network device, receiving a resource selection from a user through a user interface and establishing a threshold evaluation for the selected resource using the unique identifier assigned to the selected resource. Receiving a resource selection from a user through a user interface may include receiving a resource selection and a resource attribute selection from the user through a threshold dialog box. The threshold evaluation may be established within the network device and while the network device is operational. Establishing a threshold evaluation for the selected resource using the unique identifier assigned to the selected resource may include receiving a threshold expression from the user through the threshold dialog box, writing data, including the unique identifier assigned to the selected resource and the threshold expression, from the threshold dialog box into at least one table in a configuration database within the network device and updating thresholding code executing within the network device with the data written into tile at least one table. Receiving a threshold expression from the user through the threshold dialog box may include receiving a user selection of an existing threshold expression or receiving a new threshold expression from the user. Writing data from the threshold dialog box into at least one table in a configuration database within the network device may include sending the data from the threshold dialog box through a network management system (NMS) client to an NMS server and writing the data into the at least one table through the NMS server. Updating thresholding code executing within the network device with the data written into the at least one table may include sending an active query notice to each application including the thresholding code and corresponding to the selected resource.

In yet another aspect, the present invention provides a method of managing a telecommunications network including detecting a threshold event in an application within a network device, notifying thresholding code of the threshold event and responding to the threshold event in accordance with an action defined within the thresholding code. Detecting a threshold event in an application within a network device may include monitoring a resource attribute continuously and comparing the resource attribute against a threshold expression provided to the application by the thresholding code. Responding to the threshold event in accordance with an action defined within the thresholding code may include notifying a network manager of the threshold event, which may include sending a notice to network management system software external to the network device or sending an electronic mail message or a page to the network manager. Responding to the threshold event in accordance with an action defined within the thresholding code may include logging the threshold event.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a computer system with a distributed processing system;

FIGS. 2a 2b are block and flow diagrams of a distributed network management system;

FIG. 3a is a block diagram of a logical system model;

FIGS. 3b and 3d 3f are flow diagrams depicting a software build process using a logical system model;

FIG. 3c is a flow diagram illustrating a method for allowing applications to view data within a database;

FIG. 3g is a flow diagram depicting a configuration process;

FIG. 3h and 3j are flow diagrams depicting template driven network services provisioning processes;

FIGS. 3i and 3k 3m are screen displays of an OSS client and various templates;

FIGS. 4a 4z, 5a 5z, 6a 6p, 7a 7y, 8a 8e, 9a 9n, 10a 10i, 11a 11k, 11n 11o, 11s and 11x are screen displays of graphical user interfaces;

FIGS. 11L 11m are tables representing data in a configuration database;

FIGS. 11p 11r and 11t 11u are tables representing data in a network management system (NMS) database;

FIG. 11v is a block and flow diagram representing the creation of a user profile logical managed object including one or more groups;

FIG. 11w is a block and flow diagram of a network management system implementing user profiles and groups across multiple databases;

FIGS. 12a and 13a are block and flow diagrams of a computer system incorporating a modular system architecture and illustrating a method for accomplishing hardware inventory and setup;

FIGS. 12b 12c and 14a 14f are tables representing data in a configuration database;

FIG. 13b is a block and flow diagram of a computer system incorporating a modular system architecture and illustrating a method for configuring the computer system using a network management system;

FIGS. 13c and 13d are block and flow diagrams of an accounting subsystem for pushing network device statistics to network management system software;

FIG. 15 is a block and flow diagram of a line card and a method for executing multiple instances of processes;

FIGS. 16a 16b are flow diagrams illustrating a method for assigning logical names for inter-process communications;

FIG. 16c is a block and flow diagram of a computer system incorporating a modular system architecture and illustrating a method for using logical names for inter-process communications;

FIG. 16d is a chart representing a message format;

FIGS. 17 19 are block and flow diagrams of a computer system incorporating a modular system architecture and illustrating methods for making configuration changes;

FIG. 20 is a block and flow diagram of a computer system incorporating a modular system architecture and illustrating a method for distributing logical model changes to users;

FIG. 21 is a block and flow diagram of a computer system incorporating a modular system architecture and illustrating a method for making a process upgrade;

FIG. 22 is a block diagram representing a revision numbering scheme;

FIG. 23 is a block and flow diagram of a computer system incorporating a modular system architecture and illustrating a method for making a device driver upgrade;

FIG. 24 is a block diagram representing processes within separate protected memory blocks;

FIG. 25 is a block and flow diagram of a line card and a method for accomplishing vertical fault isolation;

FIG. 26 is a block and flow diagram of a computer system incorporating a hierarchical and configurable fault management system and illustrating a method for accomplishing fault escalation.

FIG. 27 is a block diagram of an application having multiple sub-processes;

FIG. 28 is a block diagram of a hierarchical fault descriptor;

FIG. 29 is a block and flow diagram of a computer system incorporating a distributed redundancy architecture and illustrating a method for accomplishing distributed software redundancy;

FIG. 30 is a table representing data in a configuration database;

FIGS. 31a 31c, 32a 32c, 33a 33d and 34a 34b are block and flow diagrams of a computer system incorporating a distributed redundancy architecture and illustrating methods for accomplishing distributed redundancy and recovery after a failure;

FIG. 35 is a block diagram of a network device;

FIG. 36 is a block diagram of a portion of a data plane of a network device;

FIG. 37 is a block and flow diagram of a network device incorporating a policy provisioning manager;

FIGS. 38 and 39 are tables representing data in a configuration database;

FIG. 40 is an isometric view of a network device;

FIGS. 41a 41c are front, back and side block diagrams, respectively, of components and modules within the network device of FIG. 40;

FIG. 42 is a block diagram of dual mid-planes;

FIG. 43 is a block diagram of two distributed switch fabrics and a central switch fabric;

FIG. 44 is a block diagram of the interconnections between switch fabric central timing subsystems and switch fabric local timing subsystems;

FIG. 45 is a block diagram of a switch fabric central timing subsystem;

FIG. 46 is a state diagram of master/slave selection for switch fabric central timing subsystems;

FIG. 47 is a block diagram of a switch fabric local timing subsystem;

FIG. 48 is a state diagram of reference signal selection for switch fabric local timing subsystems;

FIG. 49 is a block diagram of the interconnections between external central timing subsystems and external local timing subsystems;

FIG. 50 is a block diagram of an external central timing subsystem;

FIG. 51 is a timing diagram of a first timing reference signal with an embedded second timing signal;

FIG. 52 is a block diagram of an embedded circuit;

FIG. 53 is a block diagram of an extractor circuit;

FIG. 54 is a block diagram of an external local timing subsystem;

FIG. 55 is a block diagram of an external central timing subsystem;

FIG. 56 is a block diagram of a network device connected to test equipment through programmable physical layer test ports;

FIG. 57 is a block and flow diagram of a network device incorporating programmable physical layer test ports;

FIG. 58 is a block diagram of a test path table;

FIG. 59 is a block and flow diagram of a network management system incorporating proxies to improve NMS server scalability;

FIGS. 60a 60n are tables representing data in a configuration database;

FIG. 61a is a block diagram representing a physical managed object;

FIG. 61b is a block diagram representing a proxy;

FIG. 62 is a screen display of a dialog box;

FIG. 63 is a block diagram of a network device connected to an NMS;

FIG. 64 is a table representing data in an NMS database;

FIG. 65 is a block and flow diagram of a threshold management system;

FIGS. 66a 66e are screen displays of a graphical user interface;

FIG. 67 is a screen display of a threshold dialog box; and

FIGS. 68, 69a 69b, 70a 70b and 71 are tables representing data in a configuration database.

DETAILED DESCRIPTION

A modular software architecture solves some of the more common scenarios seen in existing architectures when software is upgraded or new features are deployed. Software modularity involves functionally dividing a software system into individual modules or processes, which are then designed and implemented independently. Inter-process communication (IPC) between the processes is carried out through message passing in accordance with well-defined application programming interfaces (APIs) generated from the same logical system model using the same code generation system. A database process is used to maintain a primary data repository within the computer system/network device, and APIs for the database process are also generated from the same logical system model and using the same code generation system ensuring that all the processes access the same data in the same way. Another database process is used to maintain a secondary data repository external to the computer system/network device; this database receives all of its data by exact database replication from the primary database.

A protected memory feature also helps enforce the separation of modules. Modules are compiled and linked as separate programs, and each program runs in its own protected memory space. In addition, each program is addressed with an abstract communication handle, or logical name. The logical name is location-independent; it can live on any card in the system. The logical name is resolved to a physical card/process during communication. If, for example, a backup process takes over for a failed primary process, it assumes ownership of the logical name and registers its name to allow other processes to re-resolve the logical name to the new physical card/process. Once complete, the processes continue to communicate with the same logical name, unaware of the fact that a switchover just occurred.

Like certain existing architectures, the modular software architecture dynamically loads applications as needed. Beyond prior architectures, however, the modular software architecture removes significant application dependent data from the kernel and minimizes the link between software and hardware. Instead, under the modular software architecture, the applications themselves gather necessary information (i.e., metadata and instance data) from a variety of sources, for example, text files, JAVA class files and database views, which may be provided at run time or through the logical system model.

Metadata facilitates customization of the execution behavior of software processes without modifying the operating system software image. A modular software architecture makes writing applications--especially distributed applications--more difficult, but metadata provides seamless extensibility allowing new software processes to be added and existing software processes to be upgraded or downgraded while the operating system is running. In one embodiment, the kernel includes operating system software, standard system services software and modular system services software. Even portions of the kernel may be hot upgraded under certain circumstances. Examples of metadata include, customization text files used by software device drivers; JAVA class files that are dynamically instantiated using reflection; registration and deregistration protocols that enable the addition and deletion of software services without system disruption; and database view definitions that provide many varied views of the logical system model. Each of these and other examples are described below.

The embodiment described below includes a network computer system with a loosely coupled distributed processing system. It should be understood, however, that the computer system could also be a central processing system or a combination of distributed and central processing and either loosely or tightly coupled. In addition, the computer system described below is a network switch for use in, for example, the Internet, wide area networks (WAN) or local area networks (LAN). It should be understood, however, that the modular software architecture can be implemented on any network device (including routers) or other types of computer systems and is not restricted to a network switch.

A distributed processing system is a collection of independent computers that appear to the user of the system as a single computer. Referring to FIG. 1, computer system 10 includes a centralized processor 12 with a control processor subsystem 14 that executes an instance of the kernel 20 including master control programs and server programs to actively control system operation by performing a major portion of the control functions (e.g., booting and system management) for the system. In addition, computer system 10 includes multiple line cards 16a 16n. Each line card includes a control processor subsystem 18a 18n, which runs an instance of the kernel 22a 22n including slave and client programs as well as line card specific software applications. Each control processor subsystem 14, 18a 18n operates in an autonomous fashion but the software presents computer system 10 to the user as a single computer.

Each control processor subsystem includes a processor integrated circuit (chip) 24, 26a 26n, for example, a Motorola 8260 or an Intel Pentium processor. The control processor subsystem also includes a memory subsystem 28, 30a 30n including a combination of non-volatile or persistent (e.g., PROM and flash memory) and volatile (e.g., SRAM and DRAM) memory components. Computer system 10 also includes an internal communication bus 32 connected to each processor 24, 26a 26n. In one embodiment, the communication bus is a switched Fast Ethernet providing 100 Mb of dedicated bandwidth to each processor allowing the distributed processors to exchange control information at high frequencies. A backup or redundant Ethernet switch may also be connected to each board such that if the primary Ethernet switch fails, the boards can fail-over to the backup Ethernet switch.

In this example, Ethernet 32 provides an out-of-band control path, meaning that control information passes over Ethernet 32 but the network data being switched by computer system 10 passes to and from external network connections 31a 31xx over a separate data path 34. External network control data is passed from the line cards to the central processor over Ethernet 32. This external network control data is also assigned a high priority when passed over the Ethernet to ensure that it is not dropped during periods of heavy traffic on the Ethernet.

In addition, another bus 33 is provided for low level system service operations, including, for example, the detection of newly installed (or removed) hardware, reset and interrupt control and real time clock (RTC) synchronization across the system. In one embodiment, this is an Inter-IC communications (I.sup.2C) bus.

Alternatively, the control and data may be passed over one common path (in-band).

Network/Element Management System (NMS):

Exponential network growth combined with continuously changing network requirements dictates a need for well thought out network management solutions that can grow and adapt quickly. The present invention provides a massively scalable, highly reliable comprehensive network management system, intended to scale up (and down) to meet varied customer needs.

Within a telecommunications network, element management systems (EMSs) are designed to configure and manage a particular type of network device (e.g., switch, router, hybrid switch-router), and network management systems (NMSs) are used to configure and manage multiple heterogeneous and/or homogeneous network devices. Hereinafter, the term "NMS" will be used for both element and network management systems. To configure a network device, the network administrator uses the NMS to provision services. For example, the administrator may connect a cable to a port of a network device and then use the NMS to enable the port. If the network device supports multiple protocols and services, then the administrator uses the NMS to provision these as well. To manage a network device, the NMS interprets data gathered by programs running, on each network device relevant to network configuration, security, accounting, statistics, and fault logging and presents the interpretation of this data to the network administrator. The network administrator may use this data to, for example, determine when to add new hardware and/or services to the network device, to determine when new network devices should be added to the network, and to determine the cause of errors.

Preferably, NMS programs and programs executing on network devices perform in expected ways (i.e., synchronously) and use the same data in the same way. To avoid having to manually synchronize all integration interfaces between the various programs, a logical system model and associated code generation system are used to generate application programming interfaces (APIs)--that is integration interfaces/integration points--for programs running on the network device and programs running within the NMS. In addition, the APIs for the programs managing the data repositories (e.g., database programs) used by the network device and NMS programs are also generated from the same logical system model and associated code generation system to ensure that the programs use the data in the same way. Further, to ensure that the NMS and network device programs for managing and operating the network device use the same data, the programs, including the NMS programs, access a single data repository for configuration information, for example, a configuration database within the network device.

Referring to FIG. 2a, in the present invention, the NMS 60 includes one or more NMS client programs 850a 850n and one or more NMS server programs 851a 851n. The NMS client programs provide interfaces for network administrators. Through the NMS clients, the administrator may configure multiple network devices (e.g., computer system 10, FIG. 1; network device 540. FIG. 35). The NMS clients communicate with the NMS servers to provide the NMS servers with configuration requirements from the administrator. In addition, the NMS server provides the NMS client with network device management information, which the client then makes available to the administrator. "Pushing" data from a server to multiple clients synchronizes the clients with minimal polling. Reduced polling means less management traffic on the network and more device CPU cycles available for other management task. Communication between the NMS client and server is done via Remote Method Invocation (RMI) over Transmission Control Protocol (TCP), a reliable protocol that ensures no data loss.

The NMS client and server relationship prevents the network administrator from directly accessing the network device. Since several network administrators may be managing the network, this mitigates errors that may result if two administrators attempt to configure the same network device at the same time.

The present invention also includes a configuration relational database 42 within each network device and an NMS relational database 61 external to the network device. The configuration database program may be executed by a centralized processor card or a processor on another card (e.g., 12, FIG. 1; 542, FIG. 35) within the network device, and the NMS database program may be executed by a processor within a separate computer system (e.g., 62, FIG. 13b). The NMS server stores data directly in the configuration database via JAVA Database Connectivity (JDBC) over TCP, and using JDBC over TCP, the configuration database, through active queries, automatically replicates any changes to NMS database 61. By using JDBC and a relational database, the NMS server is able to leverage database transactions, database views, database journaling and database backup technologies that help provide unprecedented system availability. Relational database technology also scales well as it has matured over many years. An active query is a mechanism that enables a client to post a blocked SQL query for asynchronous notification by the database when data changes are made after the blocked SQL query was made.

Similarly, any configuration changes made by the network administrator directly through console interface 852 are made to the configuration database and, through active queries, automatically replicated to the NMS database. Maintaining a primary or master repository of data within each network device ensures that the NMS and network device are always synchronized with respect to the state of the configuration. Replicating changes made to the primary database within the network device to any secondary data repositories, for example, NMS database 61, ensures that all secondary data sources are quickly updated and remain in lockstep synchronization.

Instead of automatically replicating changes to the NMS database through active queries, only certain data, as configured by the network administrator, may be replicated. Similarly, instead of immediate replication, the network administrator may configure periodic replication. For example, data from the master embedded database (i.e., the con figuration database) can be uploaded daily or hourly. In addition to the periodic, scheduled uploads, backup may be done anytime at the request of the network administrator.

Referring again to FIG. 2a, for increased availability, the network device may include a backup configuration database 42' maintained by a separate, backup centralized processor card (e.g., 12, FIG. 1; 543, FIG. 35). Any changes to configuration database 42 are replicated to backup configuration database 42'. If the primary centralized processor card experiences a failure or error, the backup centralized processor card may be switched over to become the primary processor and configuration database 42' may be used to keep the network device operational. In addition, any changes to configuration database 42 may be written immediately to flash persistent memory 853 which may also be located on the primary centralized processor card or on another card, and similarly, any changes to backup configuration database 42' may be written immediately to flash persistent memory 853' which may also be located on the backup centralized processor card or another card. These flash-based configuration files protect against loss of data during power failures. In the unlikely event that all copies of the database within the network device are unusable, the data stored in the NMS database may be downloaded to the network device.

Instead of having a single central processor card (e.g., 12, FIG. 1; 543, FIG. 35), the external control functions and the internal control functions may be separated onto different cards as