To understand the concept of a communication protocol,let us assume that A and B need to talk to one another. They want to exchange their ideas. But it turns out that,both,A and B are egoists. They start talking again simultaneously,then pause for breath simultaneously,and then start talking again. Now imagine the confusion and chaos. To avoid it,they must follow a set of rules while talking. For instance,say first A must talk,then he/she must give B a chance to put forward his/her ideas,and so on. This common set of rules would be known as communication protocol for A and B.

A protocol is “a formal description of message formats and the rules that two or more machines must follow to exchange those messages.” We need protocols every time we want to do something on a network printer we need protocols. Every time we want to save our work on disk,we don’t need protocols-unless the disk is on a network file server. Usually multiple protocols will be in use simultaneously. For one thing,computers usually do  several things at once,and often for several people at once. Therefore, most protocols support multitasking. Also,one operation can involve several protocols. For example,consider the NFS(Network File System) protocol. A write to a file is performed with an NFS operation,that uses another protocol to deliver a datagram on an Ethernet,and so on.

HTTP (Hypertext transfer protocol)- 

The hypertext transfer protocol (HTTP) is an application level protocol with the lightness and speed necessary for distributed, collaborative, hypermedia information systems. It is a generic, stateless, object oriented protocol which can be used for many tasks, such as name servers and distributed object management systems, through extension of its request methods (commands). A feature of HTTP is the typing of data being transferred. HTTP has been in use by the world wide web global information intitiative since 1990.

Practical information systems requires more functionality than simple retrieval, including search, front-end update, and annotation. HTTP allows an open-ended set of methods to be used to indicate the purpose of a request. It builds on the disciple of reference provided by the uniform resource identifier as a location (URL) or name (URN), for indicating the resource on which a method is to be applied. Messages are passed to HTTP in a format similar to that used by internet mail and Multipurpose Internet Mail Extensions (MIME). HTTP is also used as a generic protocol for communication between user agents and proxies/gateways to other internet protocols, such as SMTP, NNTP, FTP, gopher and WAIS, allowing basic hypermedia access to resources available from diverse applications and simplifying the implementation of user agents.

The HTTP protocol consists of two fairly ditinct items: the set of requests from browsers to servers and the set of responses going back the other way. Although HTTP was designed for use in the web, it has been intentionally made more general than necessary with an eye to future object oriented applications.

The HTTP has various built in request methods with allow users to read a web page, or to read a web page’s header, or to store a webpage, or to connect two existing resources or to break an existing connection between two resources.

FTP (File transfer protocol)-

One of the original services on the internet was designed to allow for transferring files from one system to another. It goes by the name ftp. Files of any type can be transferred, although you may have to specify whether the file is an ASCII or binary file. They can be transferred to any system on the internetm provided that permissions are set accordingly. FTP offers these advantages:

(i)Its very useful to transfer files from one network in an organization to another.
(ii)Its an effective way to get geographically dispersed group to co-operate on a project.
(iii)Its a potent and popular way to share information over the internet.

FTP isn’t just the name of the protocol; its also the name of a program or command. Issue the command by typing ftp, followed by the address of another site, and press enter.

FTP acts as a client server process. You give the command ftp using a remote address such as the following:

FTP newday.horizon.com

The FTP running on your system is client to an FTP process that acts as server on newday.horizon.com. You issue commands to the ftp process at newday, and it responds appropriately. The protocol is mainly concerned with transfer of file.

The objectives of FTP are-

• To promote sharing of files
• To encourage indirect or implicit use of remote computers
• To shield a user from variations in file storage systems among hosts
• To transfer data reliably and efficiently. FTP, though usable directly by a user at a terminal, is designed mainly for use by programs.

• Internal modems are the modems that are fixed within the computer.
• External modems are the modems that are connected externally to a computer as other peripherals are connected.

RJ-45- RJ-45 is short for registered jack. Its an eight wire connector, which is commonly used to connect computers on Local Area Networks (LAN), especially ethernet.  The RJ-45 connector looks very similar to standard telephone connector (RJ-11) , but its wider than RJ-11, because it houses 8 wires instead of 4. RJ-45 connectors are used to connect computers in Ethernet LANs. Ethernet Card- The computers that are part of Ethernet, have to install a special card called ethernet card.  An ethernet card contains connections for either coaxial or twisted pair cables (or both). If it is designed  for twisted pair, it will have a RJ-45 connection. Some ethernet cards also have AUI connector.  This can be used to attach coaxial, twisted pair, or fiber optics cable to an ethernet card. When this connection is used, there is always an external transceiver attached to the workstation. Hub- A hub is a hardware device used to connect several computers together. A hub that contains multiple independent but connected modules of network and inter-networked equipment. A similar term is concentrator. A concentrator is a device that provides a central connection point for cables from workstations, servers and peripherals. In a star topology, twisted pair wire os run from each workstation to a central concentrator. Basically hubs are multi-slot concentrators into which a number of multi-port cards can be plugged to provide additional access as the network grows in size . Hubs can be either active or passive.  Active hubs electrically amplify the signal as it moves from one connected device to another. Active concentrators are used like repeaters to extend the length of the network. Passive hubs allow the signal to pass from one computer to another without any change. Hubs usually support 8,12, or 24 RJ-45 ports. These are often used in star or star wired ring topology and require specialized software for port management. Hubs are simple devices that interconnect groups of users. Hubs forward any data packets-including email, word processing documents, graphics, print requests- they receive over one port from one workstation to all of their remaining ports. All users connected to a single hub or stack of connected hubs are in the same segment, sharing  the hubs bandwidth or data carrying capacity. As more users are added to a segment, they compete for a finite amount of bandwidth devoted to that segment. To understand how a hub serves a network, imagine a hotel with just one phone line available to all guests. Lets say one guest wants to call another. She picks up her phone and the phone rings in all rooms. All the other guests have to answer the phone and determine whether or not the call is intended for them. Then, as long as the conversation lasts, no one else can use the line. With a few guests, this system is marginally acceptable. However at peak times of the day- say, when everyone returns to his or her room in the evening, it becomes difficult to communicate. The phone line is always busy.

Switch- A switch is a device that is used to segment networks into different subnetworks called subnets or LAN segments. Segmenting the network into smaller subnets, prevents traffic overloading in a network. A switch is responsible for filtering i.e., transforming data in a specific way and for forwarding packets (a piece of message being transmitted) between LAN segments. Switch support any packet  protocol. LANs that are segmented through switches are called Switched LANs. In the case of Ethernet LANs, they are called Switched Ethernet LANs.

HOW A SWITCH FUNCTIONS- To insulate the transmission from the other ports, the switch establishes a temporary connection between the source and destination, and then terminates the connection once the conversation is done. A switch would be like a phone system with private lines in place of the hub’s party line. For instance, Meira Sen at the Maurya Hotel callsIbrahim Soz in another room, and the operator or phone switch connects the two of them on a dedicated line. This allows more conversations at any one time thereby allowing more guests to communicate.

Repeater-

A repeater is a device that amplifies a signal being transmitted on the network. Its used in long network lines, which exceed the maximum rated distance for a single run. Over distance, the cables connecting a network lose the signal transmitted. If the signal degrades too much, it fails to reach the destination. Or, if it does arrive, the degradation of the message makes it useless. Repeaters can be installe along the way to ensure that data packets reach their destination.  Repeaters are of two kinds- amplifier and signal repeater. The first merely amplifies noise. The second type collects the inbound packet and then retransmits the packet as if it were starting from the source station.

Bridge-

A bridge is a device that lets you link two networks together. Bridges are smart enough to know which computers are on which side of the bridge, so they only allow those messages that need to get to the other side to cross the bridge. This improves performance on both sides of the bridge. As a packet arrives at the bridge, the bridge examines the physical destination address of the packet. The bridge then decides whether or not let the packet cross. A bridge is a network device that establishes an intelligent connection between two local networks with the same standard but with different types of cables.

Router-

A device that works like a bridge but can handle different protocols, is known as a router. For example. a router can link ethernet to a mainframe. If the destination is unknown to a router it sends the traffic (bound to unknown destination) to another router (using logical addresses) which knows the destination. A router differs from a bridge in a way that former uses logical addresses and the latter uses physical addresses. A router is a network device that is used to separate different segments in a network to improve performance and reliability. A router works like a bridge, but can handle different protocols.

HOW A ROUTER FUNCTIONS-

Compared to hubs and switches,routers are smarter still. Routers use a more complete packet address to determine which router or workstation should receive each packet next. Based on a network road map called a routing table,routers can help ensure that packets are travelling the most efficient paths to their destinations. If a link between two routers fails,the sending router can determine an alternate route keep traffic moving.

To understand routing,imagine the Maurya Hotel and all the fellow hotels in its chain have trained their operators to be more efficient. When guest Meira Sen at the Maurya Hotel calls guest Ishaan Kapoor at the Shereton Hotel,the operator at the Maurya Hotel knows the best way to patch that call through even in the case when direct linking to Shereton Hotel is not possible for a while .He sends the call to the local operator of the MS hotel,who passes it to the Shereton Hotel. If there’s ever a problem with the switch board at the MS Hotel,the operator at the Maurya can use a alternate route to get the call through-for example,by routing it to another hotel’s switch board,which,in turn,sends the call to the Shereton.

Gateway:

A gateway is a device that connects dissimilar networks. A gateway is a device that connects dissimilar networks. A gateway operates at the highest layer of network abstraction. It expands the functionality of routers by performing data  translation and protocol conversion. It is needed to convert Ethernet traffic from the LAN,to SNA10 (Systems Network Architecture) traffic on the legacy system. It then routes the SNA traffic to the mainframe. When the mainframe answers,the reverse process occurs.

A gateway is actually a node on a network that serves as an entrance to another network. In enterprises,the gateway is the computer that routes the traffic from a workstation to the outside network that is serving web pages. In homes,the gateway is the ISP that connects the user to the Internet.

In enterprises,the gateway node often acts as a proxy server and a firewall. The gateway is also associated with both a router,which uses headers and forwarding tables to determine where packets are sent,and a switch,which provides the actual path for the packet in and out of the gateway.

Nodes (Workstations)-

The term nodes refers to the computers that are attached to a network and are seeking to share the resources of the network. Ofcourse, if there were no nodes (also called workstations), there would be no network at all.

Server-

A computer that facilitates the sharing of data, software, and hardware resources on the network is termed as a Server. On small networks, sometimes all the shareable stuff (like files, data, software etc) is stored on the server. A network can have more than one server also. Each server has a unique name on the network and all users of network identify the server by its unique name.

Servers can be of two types-

• Non-dedicated servers- On small networks, a workstation that can double up as a server is known as non-dedicated server since its not completely dedicated to the cause of serving. Such servers can facilitate the resource sharing among workstations on a proportionally smaller scale. Since one computer works as a workstation as well as a server, its slower and requires more memory. The small networks using such a server are known as PEER-TO-PEER networks.
• Dedicated servers- On bigger network installations, there is a computer reserved for server’s job and its only job is to help workstations access data, software and hardware resources. It doesn’t double up as a workstation and such a server is known as dedicated server. The networks using such a server are known as MASTER-SLAVE networks.

On a network there maybe many servers that allow workstations to share specific resources. For example, there may be a server exclusively for serving files-related requests like storing files, deciding about their access privileges and regulating the amount of space allowed for each user. This server is known as file server. Similarly, there may be printer server and modem server. The printer server takes care of the printing requirements of a number of workstations and the modem server helps a group of network users use a modem to transmit long distance messages.

Network Interface Unit (NIU)-

A network interface unit is an interpreter that helps establish communication between the server and the workstations. A standalone computer lives in its own world and carries out its tasks with its own inbuilt resources. But as soon as it becomes a workstation, it needs an interface to help establish a connection with the shared network because without this, the workstations will not be able to share network resources.

The network-interface-unit is a device that is attached to each of the workstations and the server, and helps the workstation establish the all important connection with the network. Each network-interface-unit that is attached to a workstation has a unique number identifying it which is known as the node address. The NIU is also called Terminal Access Point (TAP). Different manufacturers have different names for the interface. The NIU is also called NIC- Network Interface Card. The NIC manufacturer assigns a unique physical address to each NIC card; this physical address is known as MAC address.

SWITCHING TECHNIQUES-

One major purpose and use of networks is the sharing or transfer of data and information. Do you know how data are transmitted across networks? Well, for this various switching techniques are used.

Different types of switching techniques are employed to provide communication between two computers. These are: circuit switching, message switching and packet switching.

CIRCUIT SWITCHING-

In this technique, first the complete physical connection between two computers is established and then data are transmitted from the source computer to the destination computer. That is, when a computer places a telephone call,the switching equipment withing the telephone system seeks out a physical copper path all the way from sender telephone to the receiver’s telephone. The important property of this switching is to setup an end-to-end path between computer before any data can be sent.

MESSAGE SWITCHING-

In this technique, the source computer sends data or the message to the switching office first, which stores the data in its buffer. It then looks for a free link to another switching office and then sends the data to this office. The process is continued until the data are delivered to the destination computers,. Owing to its working principle, its also know as store and forward. That is, store first, forward later, one jump at a time.

PACKET SWITCHING-

With message switching, there is not limit on the block size, in contrast, packet switching places a tight upper limit on block size. A fixed size of packet which can be transmitted across the network is specified. Another point of its difference from message switching is that packets are stored on the disk in message switching, whereas in packet switching, all the packets of fixed size are stored in the main memory. This improves the performance as the access time (time taken to access a data packet) is reduced, thus, the throughput (measure of performance) of the network is improved.

TRANSMISSION MEDIA- 

By transmission media or communication channels of network, its meant that the connecting cable or connecting media are being talked about. The cables that connect two or more workstations are the communication channels. In LAN’s many different types of media are in use. Copper conductors in the form of twisted pair or co-axial are by far the most common. More recently, very serious consideration has been given to the use of optical fiber technology in LANs. Other media, e.g microwave transmission, infrared, telephone line etc are also used. We can group the communication media in two categories- guided media and unguided media. The guided media includes cables and unguided media includes waves through air, water or vacuum. The basic types of cables, are being discussed below. The unguided media are also being discussed.

Twisted pair cable-

The most common form of wiring in data communication application is the twisted pair cable. As a voice grade medium(VGM), its the basis for most internal office telephone wiring. It consists of two identical wires wrapped together in a double helix.

During the twentieth century, the key technology has been information gathering, processing and distribution. Among other developments, we have seen the installation of worldwide telephone networks, the birth and unexpected growth of computer industry, and the launching of communication satellites.

As we have entered the 21st century, these areas are rapidly converging. The merging of computers and communications had a deep influence on the way computer systems are organized. The old model of a single computer serving all of the organizational needs, is rapidly being replaced by one in which a large number of separate but interconnected computers do  the job. These systems are called computer networks.

What is a network?

A network is an interconnected collection of autonomous computers. Two computers are said to be interconnected if they are capable of exchanging information. Central to this definition is the fact that the computers are autonomous. This means that no computer on the network can start, stop or control another.

Need for networking

Before getting into the technical details of computer networks, we must find out why people are interested in it and what makes it necessary for the survival of any organization in this competitive era.

Network goals

Many organizations already have a substantial number of computers, often located far apart. For example, a company with many offices may have computers at each location to keep track of customer orders, monitor sales, and do the local payroll. Previously, each of these computers may have worked in isolation with others but at some point, management decided to connect them to gather information about entire company. In general we can refer to it as-

• Resource sharing- The aim is to make all programs, data and peripherals available  to anyone on the network, irrespective of the physical locations of the resources and the user.
• Reliability- A file can have copies on two or three different machines, so if one of them is unavailable (hardware crash), the other copies could be used. For military, banking, air reservation and many other applications it is of great importance.
• Cost factor- Personal computers have better price/performance ratio than micro computers. So its better to have PCs one per user, with data stored on one shared file server machine.
• Communication medium- Using a network, its possible for managers, working far apart, to prepare financial  report of the company. The changes at one end can be immediately noticed at another and hence it speeds up the co-operation among them.

Application of networks-

Computer networks have made a major impact on the society as a whole but we will discuss only of the important ones.

1.SHARING- The potential advantage of network is to provide an easy and flexible means of sharing. There are three distinct types of sharing:

(a) Peripherals- These are often expensive. Its impractical for each computer on the network to have both its laser printer and dot-matrix printer. A main frame may have one of each connected to it allowing all users controlled access in a cost effective manner.

(b) User of  a multiuser system can share and exchange information in a number of  ways. Example are- sending electronic mail or having controlled access to the some files or data base.

(c) In a traditional time sharing system, all control is performed centrally; if the processor fails then the entire system fails. In a network system, this need not be the case. The failure of one node should not have a domain effect on the rest. This is called distributed control and is a very lively area of research at present.

2. ACCESS TO REMOTE DATABASE-Another major area of network use is access to remote database. Its easy for the average person sitting at his PC to make reservation for airplanes, trains, hotels and so on anywhere in the world with instant confirmation.

3. COMMUNICATION FACILITIES- A third category of potential widespread network use is as a communication medium. It is possible for everyone, not just people in the computer business, to send and receive electronic mail. This mail is also able to contain digital voice, still pictures and videos.

Using computer network as a sophisticated communication system may reduce the amount of travelling done, thus saving energy. The information revolution is expected to change society as much as the industrial revolution did.

EVOLUTION OF NETWORKING-

Evolution of networking started way back in 1969 by the development of first network called ARPANET, which led to the development of Internet. Lets learn how first network evolved to change society as much as the industrial revolution did.

ARPANET-

The seeds of today’s internet were planted in 1969, when US Department of Defense sponsored a project named ARPANET( Advanced research projects agency network). The goal of this project was to connect computers at different universities ans US defense. Soon engineers, scientists, students, and researchers who were part of the system, began exchanging data and messages on it. The users of this system were also able to play long distance games and socialize with people who shared their interests. ARPANET started with a handful of computers but it expanded rapidly. In mid 80′s, another federal agency, the National Science Foundation, created a new, high capacity network called NSFnet, which was more capable than ARPANET.
NSFnet allowed only the academic research on its network and not any kind of private business on it. So many private companies built their own networks, which were later interconnected along with ARPANET and NSFnet to form Internet.  It was inter networking i.e., the linking of these two and some other network (i.e., the ARPANET, NSFnet and some private networks) that was named internet. The original ARPANET was shut down in 1990, and the government funding for NSFnet discontinued in 1995. But the commercial internet services came into picture, which are still running internet.

THE INTERNET-  

The internet is a worldwide network of computer networks that evolved from the first network ARPAnet (Advanced Research Projects Agency Network). The internet is made up of many networks each run by a different company and interconnected at peering points. Its an interconnection of large and small networks around the globe. The common use of internet standards allows users connected to one network to communicate with users on another network.

The internet is a super-network. It connects many smaller networks together and allows all the computers to exchange information with each other. To accomplish this all the computers on the Internet have to use a common set of rules for communication. Those rules are called protocols, and the internet uses a set of protocols called TCP/IP (Transmission control protocol/ Internet Protocol).

How does the internet work?                            

In internet, most computers are not connected directly to the internet. Rather they are connected to smaller networks, which in turn are connected through gateways to the internet backbone. A gateway is a device that connects dissimilar networks. A backbone is central interconnecting structure that connects one or more networks just like the trunk of a tree or the spine of a human being.

Lets now see how the internet functions:

• At the source computer, the message or the file/document to be sent to another computer is firstly divided  into very small parts called Packets. A packet generally contains some information.
• Each packet is given a number serial wise e.g., 1,2,3.
• All these packets are then sent to the address of destination computer.
• The destination computer receives the packets in random manner. If a packet is garbled or lost, its demanded again.
• The packets are reassembled in the order of their number and the original message/file/document is obtained.

Lets now see what all helps internet in doing so-

Internet functioning-

The reason that the internet works at all is that every computer connected to it uses the same set of rules for communication. The set of rules, as i have written before, is called protocol. The communication protocol used by internet is TCP/IP. The TCP part is responsible dividing the file/message into packets on the source computer. It (TCP) is also responsible for reassembling the received packets at the destination or recipient computer. The IP part is responsible for handling the address of destination computer so that each packet is routed (sent) to its proper destination.

• Bill Inmon’s Paradigm- Data warehouse is one part of the overall business intelligence system. An enterprise has one data warehouse, and data marts source their information from the data warehouse. In the data warehouse, information is stored in 3rd normal form.
• Ralph Kimball’s paradigm-Data warehouse is the conglomerate of all data marts within the enterprise. Information is always stored in the dimensional model.

There is no right or wrong between these two ideas,as they represent different data warehousing philosophies. In reality,the data warehouse in most enterprises are closer to Ralph Kimball’s idea. This is because most data warehouses started out as a departmental effort,and hence they originated as a data mart. Only when more data marts are built later do they evolve into a data warehouse.

 THE DATA WAREHOUSE LIFE CYCLE  :

Any data warehouse project involves the idea of designing a database that would act as Analytical storage system,with the capabilities for data mining ,OLAP(On-line Analytical Processing),data analysis and basic reports. The key factor for the success of the project is dependent on the capabilities of the extent of data mining.

Data warehouse contains data that is consolidated from many operational databases. The size of data warehouse ranges from hundreds of gigabytes to terabytes in size. The steps followed in designing such data warehouses requires a clear cut process which is different from the traditional approach. A typical data warehouse project has the following phases:

 >Requirements and Analysis phase

•             -User requirements of data
•            -Tool identification phase

>Design Phase

Design of database with operational and analytical data

•            -Interface design phase
•            -Reports (and OLAP) design

   >Build Phase

•             -Database build phase
•             -Report build phase

  >Testing Phase

•             -Interface and (or) data testing
•             -Report testing migration

Data warehouses are designed to perform well with aggregate queries running on large amounts of data. The structure of data warehouse is easier for end users to navigate,understand and query against unlike the relational databases primarily designed to handle lots of transactions. Data warehouses enable queries that cut across different segments of a company’s operation, e.g. production data could be compared against inventory data even if they were originally stored in different databases with different structures. Queries that would be complex in very normalized database would be easier to build and maintain in data warehouses, decreasing the workload on transaction systems. Data warehousing is an efficient way to manage and report on data that is from a variety of sources, non uniform and scattered throughout a company. It is an efficient way to manage demand for lots of information from lots of users. It provides the capability to analyze large amounts of historical data for nuggets of wisdom that can provide an organization with competitive advantage.

Understanding of how to leverage information technology for their organization’s success has been in high use in recent times. Successful companies are benefiting from these technologies, what they learned and what the results of their efforts were.

REQUIREMENT FINDING-

The first thing that the project team should engage in is gathering requirements from end users. Because end users are typically not familiar with the data warehousing process or concept, the help of the business sponsor is essential. Requirement gathering can happen as one-to-one meetings or as Joint Application Development (JAD) sessions, where multiple people are talking about the project scope in the same meeting.
The primary goal of this phase is to identify what constitutes as a success for this particular phase of data warehouse project. In particular, end user reporting/analysis requirements are identified and the project team will spend the remaining period of time trying to satisfy these requirements. User requirements for the data warehouse are classified in two categories- (i) Data requirements and (ii) Report requirements. Some of the typical user requirements would be as follows-

• Consistent, accurate and latest data to be available
• Central storage for all the data required
• Eliminate manual entry and manipulation of data by users
• Friendly and advanced reporting and query tool. Eliminate offline reporting
• Eliminate data redundancy
• Seamless integration with the existing operational/transactional systems

Associated with the identification of user requirements is a more concrete definition of other details such as hardware sizing information, training requirements, data source identification, and most importantly, a concrete project plan indicating the finishing date of the warehousing project.

Based on the information gathered above, a disaster recovery plan needs to be developed so that the data warehousing system can recover from accidents that disable the system. Without an effecting backup and restore strategy, the system will only last until the first major disaster, and, as many data warehousing DBAs will attest, this can happen very quickly after after the project goes live. A time frame of 2 to 8 weeks may be given for this purpose. The deliverable are as follows-

• A list of reports/cubes to be delivered to the end users by the end of this current phase
• An updated project plan that clearly identifies resource loads and milestone delivery dates

REPORTING TOOLS- 

Streamlining and storing of data in data warehouse is not sufficient to make it effective, since it demands the users to be more technical in nature. To eliminate this difficulty, the needs of reporting tool are required with capabilities o data analysis (slice, dice, pivot, drill-down etc.,) and data reporting. Some of the issues and requirements around report requirements are- offline-online reporting system and user friendly OLAP tool.

Also there are different other approaches to gather the requirements and match them to obtain technical requirements. This also brings the fact of various tool studies and the approach taken to finalize on reporting tool. Some of the approaches are-

• Enhance Model- Add new data elements to existing data warehouse system; Automate existing interfaces;add new reporting querying tool to existing data warehouse system; and define new report templates.
• Mirror model-Create new data warehouse system;mirror all interface of old system into new system.
• Data source model-Create new data warehouse system; Define interfaces from available data sources for the new system; No interface from old system marked; and identify all report requirements of users in the new system.

Depending on the approach, we are taking the direction of requirements gathering from business users and subsequently to derive the technical requirements can be defined.

DATA MODELLING- 

All the user requirements are documented and they are understood. If there is an existing system of data warehouse, then these requirements are matched to understand the existing technical architecture. All the business requirements of data summarization, reports, data analytic requirements and data mining requirements are segregated into various components.
After data summarization and data mining requirements are gathered, they are used to define the logical database. All the key business concepts are captured and their relationships are built (E-R relationships are most commonly used to capture this view).
There are three levels of data modelling. They are – conceptual, logical and physical. We will now discuss the difference among the three, the order with which each one is created and how to go from one level to the other.

Conceptual Data Model- 

Features of conceptual data model include the following-

• Includes the important entities and the relationships among them.
• No attribute key is specified.
• No primary key is specified.
• At this level, the data modeler attempts to identity the highest level relationships among different entities.

Logical Data Model- 

Features of logical data model include the following

• Includes all entities and relationships among them.
• All attributes for each entry are specified.
• The primary key for each entity specified.
• Foreign keys (keys identifying the relationship between different entities) are specified.
• Normalization occurs at this level.

At this level, the data modeler attempts to describe the data in as much detail as possible, without regard to how they will be physically implemented in the database.

In data warehousing,  it is common for the conceptual data model and the logical data model to be combined into a single step (deliverable). The steps for designing the logical data model are as follows:

• Identify all entities
• Specify primary keys for all entities
• Find the relationships between different entities
• Find all attributes for each entity
• Resolve many-to-many relationships
• Normalization

If there is an existing data warehouse, then all the entities of that database are captured and converted for logical database design. Once this is captured, they are compared with the user data requirement and required modifications are made.
Based on summarized data, the transactional data elements are identified and are appended into the database design. After the entities are identified, the relationships are re-built with the whole database in mind. The data elements that evolve from data mining and reporting requirements need not be worked through logical database design, as these elements will be captured in physical design. The key factors in logical design are to capture the data summarization and related transactional data elements.

Physical Data Model-

Logical database design and report requirements are considered key elements to define physical database design. Once logical design is defined, then each business entity is extracted from logical and would be analysed to define the physical table. The source transactional items are analysed to understand the data that is stored and then the elements in the table are defined in the physical design.

• Transactional tables- The existing transactional database will identity the key field required to capture certain data related to a business entity. These keys are captured and the tables are appropriately designed in the data warehouse. The same process would be repeated for all the transactional business elements captured in logical e.g. all the transactions related to either security transactions or order transactions depending on financial or distribution corporations.
• Summarized/Fact tables- Once the transactional tables and their fields are identified, the primary keys for summarized tables are identified and the data fields captured.
• Normalization- Data is verified for redundancy and the level of data mining required in the normalization process. This process will result in dimension tables for data warehouse apart from normalizing the required tables.
• Referential integrity- Once all the tables are defined, the referential integrity is defined.

Features of physical data model include the following-

• Specification all tables and columns
• Foreign keys are used to identify relationships between tables
• De-normalization  may occur based on user requirements
• Physical consideration may cause the physical data model to be quite different from the logical data model

At this model, the data modeler will specify how the logical data model will be realized in the database schema. The steps for physical data model design are as follows-

• Convert entities into tables
• Convert relationships into foreign keys
• Convert attributes into columns
• Modify the physical data model based on physical constraints/requirements

Database build-

If this is an enhance model, then a document will be prepared, by identifying the source element for each of the element in our data warehouse database. This exercise brings out the rules and exceptions that are required to build the conversion programs. This would be the program specification for the program that would bring the data from source system to target system. Following is the process followed here.

• Identify a source field for all data warehouse fields.
• Define the conversion methodology i.e, the process of Extract-Transformation-Load (ETL)
• ETL process-This is an industry wise standard process that defines the movement of data from source system to the expected target system.
• A transformation program is defined or the extract data is loaded into an extract file (the good practice is to have them in spreadsheet for manual files)
• All the data that is rejected to be loaded would be stored in exceptional tables that are specifically designed for this purpose. Users will analyze the data at a later point and define the strategy to bring the relevant data into the main database. Automated routines can be built to fix them, if a pattern for rejected rows can be found in the build process, then reapply them in the transformation program.

Report Development-

We have various types of reports that need to be designed and developed based on the functional usage of those. Some of them are-

(a) Canned reports
(b) Query reports
(c) Web reports and
(d) OLAP reports

Apart from just designing the reports, security design and implementation plays a key part in development and usage of reports that are designed.

Performance Testing- 

Performance takes high priority after data and report testing. Most of the time, the performance of degradation happens due to long run interfaces holding the system as non-usable. This is a very critical piece of data warehouse, if people in multiple geographical locations with time difference use this. The time the database can be non-usable reduces depending on the spread of users and their work timings. Time is also an important factor in determining the reporting tool and the data presentation. The lesser time the tool takes to show the data, the more acceptable the tool would be.

The criteria of calculations to be done in reporting tool or stored in database would depend on  the performance of reporting tool. This would result in creating of new views in database that would reduce the time on reporting tool.

This criteria of using a load utility or a load application program in interfaces would also depend on the performance of the interface processing.

Operational vs Informational Systems-

Perhaps the most important concept that has come out of the data Warehouse movement is the recognition that there are two fundamentally different types of information systems in all organizations: operational system and informational system.
Operational systems are just what their name implies; they are the systems that help us run the enterprise operation day-to-day. These are the backbone systems of any enterprise, our ‘order entry’, ‘inventory’, ‘manufacturing’. ‘payroll’, and ‘accounting’ systems. Because of their importance to the organization, operational systems were almost always the first parts of the enterprise to be computerized.  Over the years, these operational systems have been extended and re-written, enhanced and maintained to the point that they are completely integrated into the organization. Indeed, most large organizations around the world today wouldn’t operate without their operational systems and the data that these systems maintain.

On the other hand, there are other functions that go within the enterprise that have to do with planning, forecasting and managing the organization. These functions are also critical to the survival of the organization, especially in our current fast paced world. Functions like ‘marketing planning’, ‘engineering planning’ and ‘financial analysis’ also require information systems to support them. But these functions are different from operational ones, and the types of systems and information required are also different. The knowledge based functions are informational systems.

Informational systems have to do with analyzing the data and making decisions, often major decisions about how the enterprise will operate, now and in the future. And not only do informational systems have a different focus from operational ones, they often have a different scope, where operational data needs are normally focused upon a single area, informational data needs often span a number of different areas and need large amounts of large amounts of related operational data.

In the last few years, data warehousing has grown rapidly from a set of related ideas into an architecture for data delivery for enterprise end-user computing.

DATA WAREHOUSE FAILURES-

Common sources of failure while building data warehouse are as follows:

• Missing business drivers
• Use of wrong architecture
• ‘Dirty’ source data
• Top down development
• Neglecting scalability and performance issues

The data warehouse should be built to solve a recognized and well defined business problem. Sometimes these business problems are not identified properly. Some examples of such problems are as follows-

• Customers are moving to competitors
• Management has little insight and control over costs
• Promotions are failing for unknown reasons
• There is a high turnover of goods and high cost of inventory
• The organization has an inadequate understanding of consumer needs

If the business drivers are missing and the business manager decide to build a DW because the others have it, or an IT manager decided to build a DW hoping the business managers will use it, then such a data warehouse  project is likely to fail.

Also many data warehouse projects fail due to selection of an architecture that is incapable to meet business requirements. A desire to build a data warehouse (DW) quickly and cheaply often leads to selection of a wrong architecture. There exist architectures that are generally considered to be wrong:

• ‘Virtual’ data warehouse
• ‘Data Mart in a Box’
• ‘Stovepipe data Marts

We will discuss some of such architecture in the next section.

Also dirty Read is resulted due to missing of data, erroneous or inconsistent data values. Examples of ‘dirty’ data are-

• Multiple attribute values in one field
• One attribute value across two or more fields
• different spellings for the same attribute value
• inconsistent names for legal entities
• incorrect use of codes across records

It is generally observed that up to 20% of fields can contain such dirty data.

Data mart-

A data mart (DM) is a segment of data warehouse that can provide data for reporting and analysis on a section, unit, department or operation in the company, e.g., sales, payroll,production. Data marts are sometimes complete individual data warehouses which are usually smaller than the corporate data warehouse.

OLTP vs OLAP-    

OLTP stands for online transaction processing. It uses normalized tables to quickly record large amounts of transactions while making sure that these updates of data occur in as few places as possible. Consequently OLTP database are designed for recording the daily operations and transactions of a business, E,g, a timecard system that supports a large production environment must record successfully a large number of updates during critical periods like lunch hour, breaks, startup and close of work.

OLAP stands for Online Analytical Processing. It uses database tables (fact and dimension tables ) to enable multidimensional viewing, analysis and querying of large amounts of data e.g., OLAP technology would provide management with fast answers to complex queries on their operational data or enable them to analyze their company’s historical data for trends and patterns.

The first attempt to provide a definition to OLAP was by Dr. Codd, who proposed 12 rules for OLAP. Later, it was discovered that this particular white paper was sponsored by one of the OLAP tool vendors, thus causing it to lose its objectivity. The OLAP report has proposed the FASMI test, Fast Analysis of Shared Multidimensional Information.

For people on the business side, the key feature out of the above list is ‘Multidimensional’ . In other words, the ability to analyze metrics in different dimensions  such as time, geography, gender, product, etc. For example, sales for the company is up,what region is most responsible for this increase? Which store in this region is most responsible for the increase? What particular product category or categories contribute the most to the increase? Answering these types of questions in order means that we are performing an OLAP analysis.

Depending on the underlying technology used, OLAP can be broadly divided into two different camps: MOLAP and ROLAP. Another hybrid OLAP (HOLAP) refers to technologies that combine MOLAP and ROLAP.

MOLAP-        

This is the more traditional way of OLAP analysis. In MOLAP, data is stored in a multidimensional cube. The storage is not in the relational databse, but in proprietary formats.

Advantages-

• Excellent performance- MOLAP cubes are built for fast data retrieval, and is optimal for slicing and dicing operations.
• Can perform complex calculations- All calculations have been pre-generated when the cube is created. Hence, complex calculations are not only doable, but they return quickly.

Disadvantages-

• Limited in the amount of data it can handle- Because all calculations are performed when the cube is built, its not possible to include a large amount of data in the cube itself. This is not to say that the data in the cube cannot be derived from a large amount of data. Indeed, this is possible. But in case, only summary-level information will be included in the cube itself.
• Requires additional investment- Cube technologies are often proprietary and do not already exist in the organization. Therefore, to adopt MOLAP technology, chances are additional investments in human and capital resources are needed.

ROLAP-  

This methodology relies on manipulating the data stored in the relational database to give the appearance of traditional OLAP’s slicing and dicing functionality. In essence each action of slicing and dicing is equivalent to adding a ‘WHERE’ clause in the SQL statement.

Advantages-

• Can handle large amounts of data- The data size limitation of ROLAP technology is the limitation on data size of the underlying relational database. In other words, ROLAP itself places no limitation on data amount.
• Can leverage functionality inherent in the relational database: Often, relational database already comes with a host of functionality. ROLAP technologies, since they sit on top of the relational database, can therefore leverage these functionality.

Disadvantages-

• Performance can be slow:  Because each ROLAP report is essentially a SQL query (or multiple SQL queries) in the relational database,the query time can be long if the underlying data size is large.
• Limited by SQL functionality- Because ROLAP technology mainly relies on generating SQL statements to query the relational database, and SQL statements do not fit all needs (for example, it is difficult to perform complex calculations using SQL), ROLAP technologies are therefore traditionally limited to what SQL can do. ROLAP vendors have mitigated this risk by building into the tool out of the box complex functions as well as ability to allow users to define their own functions.

HOLAP-

Holap technologies attempt to combine the advantages of MOLAP and ROLAP. For summary type information, HOLAP leverages cube technology for faster performance. When detail information is needed, HOLAP can drill through from the cube into the underlying relational data.

A typical OLAP system-

A standard OLAP system contains six common characteristics as give below-

• Client server architecture
• Advanced support to data management
• User interface adopted to the user knowledge and needs
• Multidimensional data structures
• Techniques of multi-dimensional data analysis, and
• Metadata repository

Client/server architecture is a kind of distributed data processing in which the functions of a user program are assigned to atleast two process which communicate with each other. These two processes are called client process and server process. The client process sends a message to the server process asking for some service and the server process responds to client’s request by executing the requested task and sending the answer back to the client process.
The client part of the user program is called ”front end” and the server part is called ”back end”. A typical client-server distribution of tasks can be as follows-

• Front end is responsible for user interface issues and very often ,transaction execution .
• Back end is responsible for data management

Apart from that there are other task distribution as well, e.g., back end is responsible for transaction execution. Therefore lean clients and thick clients exist in such an architecture.

An OLAP system can be implemented by means of the following:

• Traditional relational database server
• Specialized SQL server
• ROLAP server, or
• MOLAP server

Although traditional relational servers are not aimed at efficiently supporting OLAP queries and huge databases of the order of gigabytes, they may be used to accomplish these tasks to some extent.

The objective of a specialized SQL server is to provided an advanced query language and query processing support for OLAP queries over multidimensional structures. SQL is extended with appropriate commands (CUBE, WINDOW, OPTIMIZE,…)). Query processing engine is enhanced to support and utilize new kinds of indices in an intelligent way.

ROLAP servers are built as intermediate servers between a relational back end server a client front end with OLAP tools. Here relational back end stores and manage data. ROLAP server is used to optimize OLAP specific queries for relational back end by-

• Identifying views to be materialized
• Rephrasing user queries to use materialized views, and
• Generating multi-statement SQL for the back end server (e.g., to execute a pivot operation defined by CUBE clause)

MOLAP servers directly support multi-dimensional view of data through a multidimensional storage engine. They use arrays to build hyper cubes and execute multidimensional front end queries directly against hyper cubes. MOLAP is a specialized system that efficiently supports:

• Queries involving aggregate and group by operators,
• Complex boolean functions
• Varies statistical functions
• Time related queries

OLAP systems can use data from operational databases to execute data analysis queries, but very often they poses tools for building their own multidimensional Data Warehouse from operational databases. Also, a separate specialized software can be used for data extracting, filtering, and integration of operational data into data warehouse. Also data analysis (against multidimensional and operational data) is done by OLAP front end components.

DIMENSIONAL MODELLING-

Dimensional modeling is the design concept used by many data warehouse designers to build their data warehouse. Dimensional model is underlying data model used by many of the commercial OLAP products available today in the market. In this model, all data is contained in two types of tables called Fact table and Dimension table-

• Fact Table- In a dimensional model, fact table contains the measurements or metrics or facts of business processes is sales, then a measurement of this business process such as monthly sales number is captured in the fact table. In addition to the measurements, the only other things a fact table contains are foreign keys for the dimension tables.
• Dimension table-In a dimension model, context of the measurements is represented in dimension tables. You can also think of the context of  measurement as the characteristics such as who, what, where, when, how of a measurement (subject). In this business process sales, the characteristics of the monthly sales number measurement can be a location (where), time (when), product sold (what).

FOURTH GENERATION LANGUAGE (4GL)-

The varuiys basic concepts  introduced in SQL can be used in many environments. For example, Microsoft access running on a single user windows application or SQL server running with 100 user connections. One of SQL’s greatest benefits is that it is truly a cross platform language and a cross product language. Because it is also what programmers refer to as high level or fourth generation language (4GL), a large amount of work can be done higher-level language 4GL (4th generation language) in fewer lines of  codes.

The query language-

The term SQL can be confusing. The S stands for structure and L for language, it is fine up to this, but the Q is a little misleading. Q, of course stands for “Query’, which if taken literally would restrict the user only to asking the database questions. But SQL does much more than ask questions. SQL can also create tables, add data, delete data, splice data together, trigger actions based on changes to the database and store the queries within the program or database.

SQL is the de facto standard language used to manipulate and retrieve data from these relational  databases. SQL enables a programmer or database administrator to do the following-

• Modify a database’s structure
• change system security settings
• Add user permissions on databases or tables
• Query a database for information
• Update the contents of a database

The most commonly used statement in SQL is the SELECT statement which can be seen in next section. This select statement retrieves data from the database and returns the data to the user. In addition to the SELECT statement, SQL provides statements for creating new databases, tables, fields, and indexes, as well as statements for inserting and deleting records. ANSI SQL also recommends a core group of data manipulation functions. Many database systems also have tools for ensuring data integrity and enforcing security that enables programmers to stop the execution of a group of commands if a certain condition occurs.

Personal Oracle 7-

Personal oracle 7 is a scaled down version of the full blown oracle 7server product. As the name implies, Personal Oracle 7 allows only single user connections. However, the SQL syntax used on this product is identical to that used on the larger, more expensive versions of Oracle. In addition, the tools used in Personal Oracle 7 have much in common with the oracle 7 product. It demonstrates command line SQL and database management techniques. In command line SQL statements are entered into Oracle’s SQL *Plus tool. This tool then returns data to the screen for the user to see, or it performs the appropriate action on the database. The oracle RDBMS is distributed with a full complement of development tools. It includes a C++ and visual basic language library that can link an application to a personal oracle database. It also comes with graphical tools for database, user and object administration as well as SQL*Loader utility, which is used to import and export to and from oracle.
Personal Oracle7 RDBMS is a good tool to work with for the following reasons-

•  It includes nearly all the tools needed to demonstrate the topics we discussed in this blog post and in the previous post related to RDBMS.
• It is available on virtually every platform in use today and is one of the most popular RDBMS products worldwide
• A 90 chapter trial copy can be downloaded from Oracle Corporation’s web server- www.oracle.com

Open Database connectivity (ODBC)-

ODBC is a functional library designed to provide a common application programming interface (API) to underlying database systems. It communicates with the database through a library driver just as windows communicates with a printer via a printer driver. Depending on the database being used, a networking driver may be required to connect a remote database. The architecture of the ODBC is illustrated in the following representation-

APPLICATION (call ODBC functions)
|
|
DRIVER MANAGER (loads ODBC driver)
|
|
ODBC DRIVER (processes ODBC calls, submits SQL requests, returns results)
|
|
DATA SOURCE (underlying DBMS)

The unique feature of ODBC (as compared to oracle or sybase libraries) is that none of its functions are database vendor specific. For instance, you can use the same code to perform queries against a Microsoft Access table or an Informix database with little or no modifications. Once again, it should be noted that most vendors add some proprietary extension to the SQL standard, such as Microsoft’s and Sybase’s Transact SQL and oracle’s PL/SQL.

ODBC has developed into a standard adopted into many products, including Visual Basic, Visual C++, FoxPro, Borland Delphi, and PowerBuilder. As always, application developers need to weigh the benefit of using the emerging ODBC standard, which enables you to design code without regard for a specific database, versus the speed gained by using a database specific function library. In other words, using ODBC will be more portable but slower than using Oracle7 or Sybase libraries.

 SQL in Application Programming-

SQL was originally made an ANSI standard in 1986. The ANSI 1989 standard (often called SQL 89) defines three types of interfacing to SQL within an application program:

• Module Language- Uses procedures within programs. These procedures can be called by the application program that can return values to the program via parameter passing.
• Embedded SQL- Uses SQL statements embedded with actual program code. This method often requires the use of pre-compiler to process SQL statements. The standard defines statements for pascal, FORTRAN, COBOL, and PL/1
• Direct invocation- left up to the implementer.

Before the concept of dynamic SQL evolved, embedded SQL was the most popular way of using SQL within a program. Embedded SQL, which is still used, uses static SQL, meaning that SQL statements are compiled into the application and cannot be changed at runtime. The principle is much the same as a compiler versus an interpreter. The performance for this type of SQL is good; however, it is not flexible and cannot always meet the needs of today’s changing business environments. Dynamic SQL is discussed shortly.

The ANSI 1992 standard (SQL-92) extended the language and became an international standard. It defines three levels of SQL compliance: entry, intermediate and full. The new features introduced include the following-

• Connection to databases
• Scroll-able cursors
• Dynamic SQL
• Outer Joins

Dynamic SQL allows construction of SQL statement at runtime. Although the performance for this type of SQL is not so good as that of embedded SQL, it provides the application developer and user a great degree of  flexibility. A call-level interface, such as ODBC or Sybase’s DB-Library is an example of dynamic SQL.

Call level interfces should not be a new concept to application programmers. When using ODBC, for instancem you simply fill a variable with your SQL statement and call the function to send the SQL statement to the database. Errors or results can be returned to the program through the use of other function call designed for those purposes. Results are returned through a process known as the binding of variables.

Database Management System                                                         

A database is mechanized, formally defined, centrally controlled collection of data in an organisation. The data records are physically organised and stored so as to promote shareability, availability, ability to evolve, and integrity, which are some of the important objectives. The database approach is made operational by a database management system on DBMS, a software system which peforms the functions of defining, creating,  revising and controlling the database. It provides facilities for retrieving data generating reports, revising data definitions, updating data and building applications. Many different end users and a variety of application programs can access the database and it is desirable to have an organisational function to exercise control over the database. This function is known as database administration. The person performing the function is a database administrator.

Three Schema Approach

Data consists of symbols which represent people, objects, events or concepts. Simply stated data are now facts. A data model is an abstract representation of data. It defines the way data items are organised and related. The model can use a variety of representation. Examples are: graphs, tables and mathematical formulas. The complexity of the organisation is organised  and simplified by mental methods. The mental model describes the data requirements by some type of formal,logical representation. This logical, user oriented description of requirements is mapped into or related to a physical storage model that describes the structure of data for storage. Therefore, there are two major classes of data model: logical and physical data models. The term model is often used as a synonym for view,schema, or diagram. The three major schema related to logical and physical data models are-

• External schema or user view is the way the user of the data defines the data and data relationships. Also called as sub-schema, since it is one part or one view of the larger conceptual schema.
• Conceptual schema is the overall logical view of the database.
• Internal schema is the physical storage data model i.e. the way the data is physically organised in storage.

Each user of the database is concerned only with a small portion of the database. Each user is interested in only a part of the entities in the database, only part of the attributes of those entities and certain relationships between entities. The external schema is the user’s schema or user’s view of the data being used.

The conceptual schema is the logical view of the entire database. It represents as closely as possible the real entities and their relationships. It contains integrity rules and authorization rules, but it does not contain information about how the data items are stored. The internal schema describes how the database is organised for physical storage and access. The internal schema includes information on ordering of records, block sizes, storage indexes, use of pointers and access strategies being used.

A mapping is a translation from one schema to another. In order for a user to access data, the user view of the data as reflected in the external schema must be translated into the overall conceptual schema. This is accomplished through external/conceptual mapping. In the same way, the conceptual/indexed mapping translates logical descriptions of data in the conceptual schema to physical locations and access paths in the internal schema. This is known as three schema approach.

Database

The database is used to store information useful to an organisation. To represent this information modelling is used. The components used in modelling are limited to the objects of interest to the organization and the relationships among these objects. One example of objects in any organisation is its personnel and one relationship and exists within this category of objects is that of suprevisor to employees. The database can be defined as a high level hierarchy of data in the form of byte as per the definitions given below:

• Byte

- A byte is an arbitrary set of eight bits that represent a character. It is the smallest addressable unit in today’s computer.

• Data item(element)- One or more bytes are combined into a data item to describe an attribute of an object. For example, if the object is a student, one attribute may be roll number, name, age and address. A data item is sometimes referred to as a field. A field is actually a physical space on tape on disk, whereas a data item is the data stored in the field.
• Record- The data representation in storage of each instance of an entity is commonly termed as a record. There is a record key which uniquely identifies a record.
• File- A collection of related records is a file. The site of a file is limited by the size of the memory or the storage medium. For example, the student file will contain maybe 100 records that is for 100 number of students.
• Database- The highest level in he hierarchy is the database. It is a set of interrelated files for real-time progressing. It contains the necessary data for problem solving and can be used by several users accessing data concurrently.
• The database maintains certain objectives as mentioned below-
• Availability

- Data should be available for use by applications (both current and future) and by queries.

• Shareability- Data items prepared by one application are available to all applications on queries.  No data items are owned by an application.
• Ability to evolve- The database can evolve as application usage as query needs evolve.
• Data independence- The users of the database establish their view of the data and its structure without regard to the actual physical storage of the data. There are two distinct levels of data independence.
• Data integrity- The database establishes a uniform high level of accuracy and consistency. Validation rules are applied by database management system.
• Data security- This is needed to protect the data from unauthorized access and from accidental damage or destruction.
• Reduced Data redundancy- Data redundancy is storage of the same piece of data in more than one place in the computer system. This should be reduced.
• Data relatability- It is the ability to establish logical relationships between different types of records, usually in different files. For example,price and earning data might be kept in different records in a system, the price data must be updated more frequently than earning. But if a query involves a relationship such as price divided by earnings, both records must be retrieved to create the desired information.
• Standardization- It refers to the need for common definitions of data items in terms of both the precise definition of a data item name and its storage format in the database. Most database management systems provide a data dictionary facility in this regard.
• Reduce personnel time and increase productivity- In any organisation, 80% of programming budget and man hour are utilized in maintaining the conventional file system. This is reduced by database.

Moreover simple requests for data handled via query language helps in more easy data retrieval which eventually results in positive effects on the organisation and hence increase in productivity.

Components of Database Management System

A database system involves four major components:

• Data
• Hardware
• Software
• User

1.Data- The database can be though of as a unification of several otherwise distinct files, with any redundancy among those files at least partly eliminated.

2.Hardware- It consists of the secondary storage volumes mostly magnetic disks, which is used to hold the stored data, together with the associated I/O devices, device controller, and I/O channels. The hardware processor and associated main memory are used to support the execution of database system software.

3.Software- Between the physical database layer and the users of the system is a layer of software generally known as DBMS. All requests for access to the database are handled by the DBMS. The other software includes utilities, application, and application development tools, design aids, report writers and the transaction manager.

4.Users- There are three broad classes of users-

1. The application programmer responsible for writing database application programs in some high level programming languages. Such programs access the database by issuing the appropriate request typically on SQL statement to the DBMS.
2. The user interacts with the system through online workstation or terminals.
3. The database administrator or DBA is solely responsible for all the activities going on in a database system. The main functions performed by the DBA or his department falls into the following categories:

• Data definition, Standardization and Liaison
• Design of data base organisation and techniques performance measurement and improvement software and hardware selection.
• Security control.
• Daily operational control.
• Consultant to programmers, analysts and users.

The biggest and most essential of these is the first achieving agreement on common data definitions, representations, and structures. All data-item types are named and a standard schema format is agreed upon for them. With some data-base management systems data item synonyms are permitted so that when same data item has been named or represented differently in different subsystems these different names and representations maybe preserved and equated. Details of all data items and their definitions are documented,preferably by means of an automated data dictionary. A cross reference list stating which programs use which data items, should be maintained. When any changes are proposed, the cross reference list will show who or what programs are affected. The combining of data items into named groups (data aggregates, records, segments, and relations) is agreed upon and the conceptual schema definition is created. The DBA and his staff make any necessary changes to the schema when the data are changed or new data are added. A separate person under the control of DBA  known as Database Definition Analyst may look after these jobs. Similarly, the design functions in database administration should therefore be handled by a different man who can be termed as Database Design Analyst.

The day to day operations can be handled by the Data Operations Supervisor.

Security  administration is a separate function from data base administration on many systems in which security is a major concern. The day-to-day security operations may not be the concern of the database administrator. But the overall security responsibility of the database may be looked after a separate person known as Security Officer.

A language called the data definition language (DDL) is used by DBA and by database designer to define schemas. The DBMS will have a DDL compiler whose function is to process the DDL statements. In order to identify description of the schema constructs and to store the schema descriptions in DBMS catalog. Once the database schemas are compiled and the database is populated with data, users much have some means to manipulate the database. Typical manipulation includes retrieval, insertion, deletion and modification of data. The DBMS provides the data manipulation language (DML) for those purposes. A data control language (DCL) is used for all the security and control procedures.

File system versus DBMS-

The earlier file systems of sequential, indexed and relative file organization can be discussed a little bit here in this section. There are the organisations which show the way the data are stored. A comparative study can also be made here. The sequential file organisation can have the following ten points of discussion-

• Th records are physically stored in the sequence in which they are created.
• The file maybe created on any device e.g, sequential like tape or random device like disk.
• Only sequential access is permitted giving records in their physical order on the file volume.
• No extra storage is needed other than required for storing file records.
• Updating of files possible only for disk files. A new updated file must be created on other devices.
• Addition at end in EXTEND file function. For addition in between or deletion, a new file must be created.
• Organization is well suited for files with high record activity, where percentage of transacted record to the total record is very high. Batching of transaction is actually done to activate this.
• The average record processing speed is fast.
• The file cannot be accessed by keys of the records.
• Blocking of the records is usually employed.

The indexed file comparison can be discussed with the following ten comparative points-

• The records are stored in ascending sequence of primary keys .
• The file will be created only on direct access storage devices like disk.
• This permits both random and sequence access mode. In sequential access records are given in ascending order of primary key.
• Additional space is required to store indexes.
• Updating is permitted.
• Additions and deletions may be made.
• Suitable for low activity files due to random access. Sequential access may be preferred in high activity application.
• The speed is comparatively slower.
• The file maybe accessed by one or more keys.
• Blocking is not permitted (may depend on system)

The relative file organisation will similarly be based on the following ten points-

• The position of record is decided by hashing function.
• The file will be created only on direct access storage devices like disk.
• Permits both access modes. In sequential access, it may be necessary to check if a position is empty.
• A file is generally allocated more resource positions than necessary to get a better performance from hashing procedures.
• Updating is permitted.
• Addition possible if space is available. Deletions are allowed.
• Well suited for low activity files. Applicable for high activity files also.
• Speeds range in between those of sequential and indexed files.
• The hashing procedures of algorithms, which organize the records are usually based on a single key.
• Blocking is not permitted.

However, the database has got a different objective mainly centered on standardization, reduced data redundancy, data security and data integrity. The clear advantages of database over the conventional file system can be specified as described below.:

• In conventional file management system same data is duplicated in more than one application. The redundancy in data storage, if not consistent , is harmful as it projects conflicting picture about facts. But in DBMS redundancy of data is overcome.
• A file only contains data values. The descriptions of file contents is available in program e.g, file section of COBOL program. Even this description is lost once the program is compiled in conventional file system.
• The design of new programs to obtain new reports required by management is very difficult even when we know that all necessary data are already available in files. This maybe due to incompatible representation of data e.g, a data in one file is decimal, but in another its binary.
• It is not possible to write general purpose querying programs for adhoc on-line information needs in conventional file systems. But in DBMS, adhoc queries are possible.
• In conventional system, any change in file characteristics changes the record descriptions. For example, if we change a file’s organisation from sequential to indexed, it is necessary to modify the program and compile it again. As the business application requirements are constantly changing, we have to maintain existing programs, which discourages growth and evolution and requires considerable investment in maintenance of old programs.
• Security of file system is not secured in conventional file management system. But in DBMS security is of high respect as the access paths are only open to part pf the database.

• The area of study and work is very interesting and challenging
• There is scope for innovating
• The pay scale is very high when compared with other fields of work

Another added incentive is that, the field of engineering has many sub-streams to choose from. Depending on the interests of the students, they can choose their branch. For example, if someone has a strong fundamental knowledge of C programming from high school years, then he/she can opt for Computer Engineering. As the branch specializes in dealing with computers and their functioning, the student will find it very appropriate to study computer science.  All these factors are responsible for the rise in the number of students enrolling in various engineering course like diplomas, bachelor and masters degrees.

Keeping all that apart, sometimes it also becomes difficult for a student to choose which branch to study and in most cases, parents of the students decide which branch will be most appropriate for their wards. Students, who have just passed out of school or colleges are asked to make a vital decision about their careers and they often get confused under the circumstances. Indian school education system has many lacuna, one of them is generalizing the topics taught in school. In most of the schools, a student has to study each and every subject, irrespective of his/her interest in it. In place of that, an education system should be devised, in which students, keeping in mind which careers they want to pursue in the future, are allowed to choose what they want to study. That will remove the confusion out of their minds and they will be sure and confident about choosing their career trend.

The core branches in engineering are-

• Mechanical Engineering
• Civil Engineering
• Electrical Engineering

These three branches are the oldest branches of engineering. Other branches like Computer Engineering, Instrumentation Engineering, Electronics and telecommunications engineering etc are relatively new branches as they bloomed in the last century. It is always advisable to choose a core branch because they are evergreen branches. There is also a scope of finding work for a mechanical, civil or electrical engineer. The current time might be very good for computer and telecom engineers, but there is no guarantee that the future will be bliss too. You might feel i’m stereotyping things, but this is the hard fact. Now, talking about the other branches like CSE, IT, ECE and EEE (i hope you understand the abbreviations), they have their importance too. No one can deny the role of computers, electronic gadgets and mobile phones in shaping the modern day world. These things are infact, the face of the modern face and if you are passionate about these subjects then no one can stop you from joining these branches. But before choosing any branch, one has to think about the prospect of finding a job in the future. If the demand of the branch is high, then the engineer is very likely to remain employed every time, otherwise there is a risk of loosing the job. For example, due to the recent economic crisis that America, Britain and many other countries experienced, many software professionals lost their jobs. You will never want to end up like that and make it sure that you don’t, you need to choose your branch very wisely and carefully.

The scope of finding jobs in the future is not the one and only aspect that should affect your decision while choosing an engineering branch. Another important thing that should be weighed before making a decision is your interest and passion for the branch. If you are crazy about studying mechanical engineering then you should only for mechanical engineering. If your parents force you to join computer engineering, then you have to persuade them that mechanical engineering is best for you and nothing else. If you keep on trying, then your parents might have to give up, because in the end of the day it is your career which is at stake. And passion and interest alone cannot drive your charge-You need to have strong fundamental knowledge about the subject too. This make the subsequent study of those topics in the 4 year bachelors course much easier and hassle free.

So, keep in mind all these point before choosing your branch and then no one can stop you from becoming a successful engineer.

the new toy of Amazon has created lot of buzz being the most worthy competitor. Check it out below

The exclusive features and advantages amazon kindle fire has over ipad or other company electronic pads are listed below
All the free movies and tv shows from instant video, not only movies but 17 million songs are all accessible through Kindle fire for a very reasonable price. As amazon kindle is mostly meant for eBook’s and reading books amazon kindle fire also has access to all the Kindle books (by the way who don’t know about kindle eBook’s they are much cheaper than paper books).
Not much is known yet by Amazon kindle fire has the Cloud-Accelerated Web Browser called amazon silk. The amazon kindle fire is smaller than ipad which could be beneficial for portability but may not be all purposes that ipad can be used for. For app lovers kindle fire may be little less from itunes apps count but android apps market is growing pretty aggressively and most of the have to have apps are available in both itunes and android market.

Amazon kindle fire has Fast, Powerful Dual-Core Processor but anywhere close to the power of ipad2 processed and the screen is a brilliant color touchscreen but may be apple lover might not agree with it. But people with lover budget but who want get onto the ipad bandwagon Amazon kindle fire is the best option and all other competitors(except Apple) are far away from Kindle fire. Hurry up and book your piece as it is first come first serve according to amazon.

Please feel free to let us know what you think?

Sub-modules are located elsewhere in the program and the superior module, whenever necessary, make a reference to subordinate module and call for its execution. This activity on the part of the superior module is known as calling, and this module is referred as calling module and sub module referred to as ‘called module’.

The sub-modules maybe  sub-programs such as functions or procedures. For example, if a program need initial and boundary conditions, sub-routines are used to set them. Then if someone else wants to compute a different solution using that program , only these sub-routines need to be changed. This is a lot easier than having to read through a program line by line, trying to figure out what each line is supposed to do and whether it needs to be changed. The values passed to the sub-routine are arguements and the variables defined in the sub-routine are parameters. Further, arguments show exactly what information a sub-program is using. That makes it easier to figure out whether it needs to be changed when we are modifying our program. Forgetting to change all occurrences of a variable is a very common source of errors.

Sub-programs make it simpler to figure out how the program operates. If the boundary conditions are implemented using a sub-routine, the program can be searched for this subroutine to find all places where the boundary conditions are used. This might include some unexpected places, such as in the output, or in performing a numerical check on the overall accuracy of the program.

Sub-programs reduce the likelihood of bugs. Because sub-programs can use local variables, there is less chance that the code in the sub-routine interferes with that of the program itself, or with that in other sub-programs. The smaller size of the individual modules also makes it easier to understand the global effects of changing a variable.

Modular approach provides following advantages-

1.COMPLEXITY: Probably the most important reason to produce a module is to reduce the complexity of the problem. By sub dividing the problem into a number of modules,we can hide the information not needed outside the module, and then only concentrate on the specifics of the module inside. If this is still complex, then sub-modules should be created.

2. DUPLICATION: Avoiding duplicate code is a reason for using modules. The basic fact that the code is in one place means it is easier to maintain, save space, and reduce the complexity and hence reliability of a program.

3. DEBUGGING: By isolating areas, debugging becomes easier and reliable because we are working with smaller section of thenprogram. Things like hardware dependencies, operating system dependencies, input/output dependencies, laws and business rules, and complex data structures and complex logic are some of the ideal candidates.

4. PERFORMANCE: We can also use modules to create improved performance.  We can optimize or recode in a limited amount of modules (that are ‘hot-spots’) rather than trying to improve the efficiency of the whole code.

5. HIDING DATA STRUCTURES: The hiding of data structures, and their use, in modules, form a good use for modules. Complex data structures tend to be very tangled and difficult to understand things and their physical use doesn’t really translate very well to the problem domain. Therefore the best way to avoid complexity when using these structures is to make modules that do the work, and then call these.

6. HIDING GLOBAL DATA: Global data can also be hidden in modules. That means we can monitor access to the data if necessary, and change the structure of the data without changing the whole program. Quite often putting access to the data in a module means we can find that the data can become local or module level.

7. RE-USABILITY: Modules can be used in other programs without rewriting rewriting or retesting. This reusability is a very important characteristic of modules.

8. READABILITY: Programs are more readable because of the modular structure. A readable module is worth a lot, simply because someone might need to understand it to upgrade or remove a bug. We should never underestimate readability.

9. PORTABILITY: By isolating non-portable sections of code in modules, we can make it easier to move the program to other machines. Such sections include hardware dependencies and operating system dependencies. There are two levels of porting-

• Porting to a machine with the same processor- Porting to a machine having the same processor is a matter of re-coding the OS differences.
• Porting to a machine with a different processor- Porting to a machine having a different processor involves re-coding the hardware and OS differences.

10. DEVELOPMENT WORK: Several programmers can work on different modules of a large program relatively independent of one another. Each module can also be tested separately from the rest of the program.

TO READ MORE ABOUT COMPUTER ENGINEERING, VISIT- Engineering Community