10 Keys to Implementing Business Intelligence Projects on a Shoestring

Price is often considered the greatest factor influencing the decision to implement a Business Intelligence (BI) system.Don’t despair. You can implement BI on a shoestring budget. However, to do so requires strict discipline and adherence to a philosophy that may be contrary to the advice of huge (read: expensive) global consulting firms. Following our shoestring philosophy will not only make success possible but likely; and at a cost well below typical expectations.At times everyone is willing to tell you what not to do, or to describe the pitfalls to avoid in your BI implementation. Experts may even be willing to talk in vague generalities about the things you should do as long as they don’t give away too many “trade secrets.” Here are 10 specific keys to successfully implement your BI project on a shoestring budget:1 Stick to a Well-Established MethodologySelecting a vendor that utilizes a clearly defined, repeatable process for system development will impact the successful achievement (or implementation) of the BI project. The system integration or software development vendor you select should have a methodology that clearly defines the processes necessary to produce quality, usable software that meets your business needs and goals and is competitively priced.Continuous improvement should be the mantra of a vendor with a well developed, clearly defined methodology. You can be a little more confident in a vendor’s commitment to methodology if you see an emphasis on continually improving, cleaning up, clarifying, and solidifying their processes.Carefully evaluate a potential vendor’s processes and methodology to determine if it is a) repeatable, b) well defined, and c) focused on continual improvement. If you select a vendor that has little or no process or methodology in place you run the risk of paying for it in slipped schedules, cost overruns, scope creep and outright failure – all of which are costly to both your project and your career.2 Clearly Define Requirements Up FrontIn his book Quality Without Tears, Philip Crosby notes that the cost of rework is the “price of nonconformity (PONC)” because “it is what management chooses to pay for not ensuring work is right the first time.” Lofty words from a giant in the quality revolution. In dollars and cents rework typically costs double (or more) the actual price of doing it right the first time. It only makes sense from an intuitive perspective, since you paid to do it wrong, and then pay again (at least once) to do it over. There is an entire industry that has sprung up around unscrupulous vendors who offer low, fixed price project estimates to win your business. These vendors then inundate their customer with unnecessary change orders at $5k and $10k per order, or more. But let’s save that discussion for a later point.In a well-established methodology (see point 1 above) requirements are a valuable asset to be managed with great care. In fact, “The Capability Maturity Model: Guidelines for Improving the Software Process” by the Software Engineering Institute says, “the purpose of requirements management is to establish a common understanding between the customer and the software project of the customer’s requirements that will be addressed by the software project.” Quite obviously this also entails setting and maintaining agreed upon requirements for the project and clearly defined expectations for all parties involved.Expect to pay for an initial evaluation of your requirements. This is only due diligence and fees paid for a scoping or requirements gathering phase should be applied to the cost of the project if you and the vendor agree to move forward.3 Go With a Fixed Price VendorWe all have budget constraints. We all feel the stress of trying to second guess “time and material” vendors in an attempt to spend what is actually necessary to do the job. We all hope to find that 95th percentile vendor who will be above the board and honest in time and material billing.You can eliminate a lot of the guess work by going with a fixed price vendor. A fixed price vendor offers you a predictable expenditure, one that can be budgeted ahead of time or that can be planned for well in advance with few if any surprises. With a fixed price vendor you also eliminate the adversarial nature of the relationship that may be fostered by the constant vigilance of billing statements and project reports that try to prove their worth.There are, however, a few points of attention to be noted when working with a fixed price vendor. First and foremost, beware the needless change order. The unscrupulous fixed price vendor will agree to a low price to win the work and then will continue to issue change orders that run the price up. Here are a few ways to insulate against this most unprincipled supplier. Offer to engage the vendor for a requirements gathering or project scoping effort up front. This provides several things including well established requirements that minimize changes, clearly defined scope to give the vendor an opportunity to develop a fair price proposal, reduced risk, and time for your team to discover issues and roadblocks that require more attention on your part.The second noteworthy point when working with a fixed price vendor is to find one that can point to specific, successful, similar projects. A similar project does not necessarily mean one in the same specific industry (or even within the same sector – federal vs. state/local vs. commercial). It’s important to look for a vendor who has successfully implemented a BI of comparable or larger size and complexity. Size is typically a measure of the number of users and queries and volume of data. Things begin to become more complex when you consider things like global footprint, 24/7 user activity (meaning that there is essentially no down period in which to perform routine system tasks), complexity of the ETL (extract, transform and load) needs, number of data sources or data feeds, and other factors.4 Insist On Rack-Grade HardwareMoore’s Law (named after Gordon Moore, co-founder of Intel) states that the computing power of processors roughly doubles annually. Hardware usually represents a significant percentage of expenditures for any BI project, and the wise use of hardware dollars can be a challenging decision for any organization. Considering Moore’s Law in your decision can simplify your choices. Rely on the high value, low cost of typical rack-grade hardware, and upgrade every 2-3 years to improved machines. Furthermore, the recent development of blade servers allows for multiple servers to be packaged in a single, inexpensive, rack mountable unit that takes full advantage of scalable architecture (more in point 5), providing for rapid growth and increased business line adoption – building today for tomorrow’s expansion. Server vendors offer blade servers in a variety of configurations and price points to match your needs.A similar axiom to consider is Kryder’s Law – named after Mark Kryder, whose observations about the increases in hard disk storage are astonishing. Since the development of the disk drive in 1956 storage per square inch has soared more than 50 million fold. This has resulted in ever-increasing storage capability at ever-decreasing cost, which indicates a clear direction toward low cost storage.A well known small systems developer was asked by a federal customer to build a transitional system that would only be used until the global vendor could get a suitable system built on large, expensive, proprietary hardware and software. After several years the small vendor’s low cost, high value system continued to grow and expand to service thousands of users using rack grade equipment and other principles outlined in this article. By the way, the customer’s original plan to build the large costly system was scrapped and the small vendor’s “shoestring” system became the “official” source for information in its functional domain – a role it continues to fill to this day.5 Start Small and Scale Out as NeededTraditionally, projects have purchased large, muscular database servers – TeraData, Oracle or DB2, which is fine if you need that kind of horse power. But to build on a shoestring we have to stick to the rack grade equipment outlined above. The key is scalability – how well a solution to a problem works if the problem increases in scope or scale. If you build a superlative system that meets your needs today but a year from now is not able to meet the increased demands, it lacks scalability.Over time business needs change, challenges get more complex, businesses grow, employees increase in numbers, reporting needs expand, analysis needs intensify, and so on. You must build today with tomorrow in mind. Taking into consideration the use of rack-grade equipment (see point 4 above), a scalable architecture makes more sense than ever. The key to scalability is to focus initially on the area of “greatest pain” and work outward from there. Focus on a small section of your business, a single line of business, or an area or function that has high reporting or analysis needs. In time you can add other areas of your business or additional whole lines of business. This approach allows your initial hardware requirements to be smaller, and lets you prove the value of the system. This approach opens the door for gradual growth and expansion of the system until it is an enterprise-wide tool valued and used by virtually the entire organization. This modular approach allows for expansion by scaling horizontally, adding additional servers to accommodate the additional load of integrating other business areas.An additional benefit of this “scale out” approach is that scaling typically produces redundancy. Redundancy equates to a less disruptive migration or transition of users to a new service as you grow and scale the system.The example cited in point 4 began with a 2-server system supporting 200-300 users. The same system “scaled out” to a global system that now supports 15,000 users worldwide processing 600k ad hoc queries a month – with 85% of all queries answered in 10 seconds or less. The original architecture never had to be redesigned since it was built with scale-out or horizontal growth in mind from the outset. Making scalability a design feature from the start eliminated the need for proprietary “band-aid” fixes. During your initial requirements development establish scalability as an uncompromising requirement.6 Select a High Value, Feature Rich User InterfaceThere are two basic categories of BI user interface toolsets available today: those that are free and those that cost money. Both categories have pros and cons, and each has its own set of features and benefits.The high end (expensive) tools have a place and I recommend them for customers who genuinely need them and have the budget in place to include them in their project.In a shoestring approach, however, the price tag on most BI tools is prohibitive. In these instances i recommend one of the free or nearly free BI solutions. Microsoft® bundles their Reporting Services and Analysis Services with SQL Server. SQL Server carries the reputation for lacking the vigor necessary to be competitive as a “serious” data warehouse product. This is a misconception that is commonly proliferate by those who either don’t understand the strength and value of SQL Server or who are in a position to profit from its competitors. In the example used in previous points the global system servicing 15,000 users is built on Microsoft® SQL Server.On the other hand, you may opt to go with a proprietary solution developed by your selected vendor. Several vendors offer powerful, feature-rich, fully-integrated OLAP tools, dashboards, ad hoc reporting, managed reporting and more, specifically designed to support large data warehouses and data marts at no additional cost to their customers. Simply ask your vendor what they offer – and ensure you check references to validate the value of the tool offered.There are also excellent tools out there like The Pentaho BI Project that provides enterprise-class reporting, analysis, dashboard, data mining and workflow capabilities that help organizations operate more efficiently and effectively, all from an Open Source project.In short, there are options for selecting a high-value, feature-rich BI toolset that won’t break the bank.7 Focus on System Improvement from the OutsetJust as you must focus on scalability from the beginning, it’s as important to focus on system improvement. System improvement means enhancements to performance tuning, changes and clarifications to business rules, adoption of technological advancements, etc. This is your chance to take a cue from your vendor. When we discussed methodologies above we mentioned that the best of these emphasizes continual improvement and refinement of processes. You should strive to implement a BI project that does just that: continually improves itself. Things that are important to you today may not be in six months. Queries that your analysts rely on today may be meaningless next year. Techniques like saving and analyzing historical query statistics help you to understand your data and how your customers need to use it. This knowledge is invaluable in helping you tune your system for peak performance. Build extensive internal analysis into your system and then develop the processes to use it faithfully to evaluate your customers’ usage of the system.Tracking and analyzing usage allows you to benchmark and quantify your performance improvements, to recognize trends and recommend enhancements and improvements to your customers and users. Perhaps most important, this analysis gives you a great story tell as you sell your newly built capability to more users. According to a recent Gartner study, of the 50% of BI initiatives that were deemed a failure the number one reason was lack of use by its intended audience. Begin now determining how you can avoid becoming part of this statistic by planning to have a finely tuned solution that continues to become more valuable to its user base over time. Improvement in performance and capability increases the number of users that rely on the solution, enhancing your system’s inherent and perceived value.8 Leverage Internal Functional ExpertiseWe cannot say enough about leveraging your own internal functional experts. There are many advantages, but the primary rewards are cost control, retention of internal expertise, and maintaining intimate contact with the user community.The cost savings are obvious: external consultants cost serious money. Other less obvious costs include paying outsiders to become familiar with your processes only to take that knowledge and leverage it against you with your competition (non-disclosure agreements notwithstanding). The proper vendor should minimize impact to your staff. Also, the professional growth opportunity this offers your team members should motivate them to participate eagerly.In addition, if you “farm out” the expertise in any given area over the long term you will eventually discover that you’ve lost any organic ability to address that need. It’s a common practice in some industries for consultants to gradually replace the organic assets and human capital with contract personnel, with the intent to eventually own all of the expertise in that area. This effectively guarantees a continuing revenue stream while you – the customer – have run low on options for regaining that organic expertise. Outsourcing ancillary, non-core and non-critical services and functionality is a great business move – do what you’re really good at and outsource the rest is a common business approach today. However, care should be taken when considering releasing your grasp on those areas that are key to your business or represent a critical functional domain.Finally, it is crucial that your internal, organic functional experts maintain close contact with the end users. The individual or group with the closest contact and interaction with end users will have the greatest influence with them as well. Should you completely outsource that functional expertise, you may lose your ability to interact on an essential level with your primary customer – the users. Never give up your ability to influence your customer.9 Ensure You Implement the Proper SolutionIf your business works fine, but just needs the reporting and analysis capability that BI offers, then why let an expensive global consulting firm convince you to re-engineer the basic foundational processes that are key to the success you’ve been enjoying? Many times, a business process re-engineering (BPR) effort stems from a consultant recommending a particular product or system that fits technically and functionally but does not fit well within the work flow of the business. Rather than select a different product or system, an enterprise may be persuaded to change its core business processes so they align with the process flow of the system. In some instances this is a valid approach and ultimately improves the performance of the business, increasing its productivity and value. However, this is rarely an easy undertaking and involves broad, sweeping, expensive changes in the business culture at its core philosophical level. One reason vendors may push for BPR is that they are paid on a time and materials (T&M) basis. It is in their interest to extend the length and/or scope of the project (see point 3 above).If your business really needs a BPR engagement, then a BI implementation is not going to resolve your issues. If, on the other hand, you need to simplify and enhance your operational reporting and analysis, or have data on disparate systems that needs to be presented homogeneously, then you may actually need a BI solution, not a BPR effort.Ensure you’re fixing the real problem and not being sold an expensive effort that will drive up your cost of doing business while not addressing your simple reporting and analysis needs.10 Drive Down Price PerformanceWhat are the primary factors that influence price performance? There are many technical factors but a small handful hold the most sway. Database systems inherently have built-in aspects such as indexed views, materialized views, partitioning, and clustering that impact their performance. While these and other technical points do greatly impact the performance of the system and the subsequent cost of that performance, the way in which they are designed and implemented is the primary determining factor. The most critical factor to driving down the price your performance is the selected vendor. This may seem obvious on the surface but probably for the wrong reason. I don’t advocate selecting the vendor who quotes the lowest price, but the vendor who offers the greatest value.Let me relate a short story about BBQ ribs and Southern culture. I’m located near Birmingham, Alabama, and have been told that we have more rib joints per capita than any other US city (no, I have no data to back this up). One of these rib joints of some renown is Dreamland Barbeque. Dreamland began as a shack by the side of the railroad tracks in rural Tuscaloosa County about an hour west of Birmingham. For years they had a sign that read, “We don’t serve: baked beans, cole slaw, mashed potatoes, potato salad, green beans…” and went on and on naming the things they did not serve. What’s the point? Dreamland served BBQ ribs, period. And did it better than anyone else. Since they specialized they were (and still are, by the way) very, very good at it. They became adept at doing it for a reasonable price. If you want ribs, go to Dreamland. They offer a great, value-priced product and do it very, very well.If you need a Business Intelligence implementation make a list of vendors that meet all of the other qualifications we’ve outlined here for a vendor. Then go through your list and look for the one that specializes in business intelligence. Find the “Dreamland Barbeque” of data. Find that vendor that is small enough to be agile, large enough to be sound, experienced enough to handle the complexity of your implementation, and focused enough to learn your data intimately. Pick a vendor that does business intelligence and does it extremely well.Following these 10 key steps does not guarantee success. Ignoring them, however, will almost certainly guarantee failure.