- SYSTEM INTEGRATION
Why many integration projects fail and 10 tips for a positive outcome
How system integration becomes a success
– Part 1/2
System Integration is a complicated, multi-layered undertaking.
Despite this, it is often underestimated as a key success factor in most IT projects. The reasons why integration projects experience significant implementation challenges or fail outright are manifold. The good news, however, is that any organization considering a System Integration may avail themselves of the following experience and wisdom from IKOR AG. Please read IKOR’s Top 10 System Integration Tips.
Please read IKOR’s Top 10 System Integration Tips.
Determine all critical and relevant system information
In addition to the new (core) system being introduced, an integration project must - in most cases - take into account many other existing / legacy systems within a company's application landscape. Each of these systems have their own unique and often very different characteristics. There is a real risk in terms of cost and timing when the characteristics of the systems involved and their role within the development planning are not clearly defined or transparent. For example, in-house developments, standard products or so-called customized assets, often "speak" different languages (such as Java, Cobol or PL/I) in different formats and dialects (such as UTF-8 or EBCDIC).
The operation of these systems takes place across a set of numerous, different and diverse platforms, including host systems with different operating systems and transaction monitors. Servers use different operating systems (such as Linux or Windows). The same applies to cloud environments with different technology stacks. Different Operators are typically responsible for system development and operation. However, it is precisely this variable (Operator expertise) that has a significant impact on how well an integration project actually progresses - in terms of the Operator’s own IT infrastructure, data center services (internal / external), cloud platforms and mixed forms.
Question and understand the initial situation carefully:
- Lifecycle: Do you need or want to operate the system for longer? If so, for how long? How long can you continue to operate it? For example, will components become unsustainable in the foreseeable future - or has this already happened?
- Maintainability: Is the system lifespan easily extended or changed?
- Skills: Do you have trained, skilled staff to maintain the current system? Over what period of time will the existing skills still be available? Or can these employees only take on maintenance work?
- Resources: Do you have the budget and qualified people to sustain the existing / legacy systems? Last but not least: What is the demographic situation like? Over what period of time will the existing skills still be available? How long can fluctuation be compensated for?
Admittedly, for most companies, the initial IT situation is complex. It is important to establish a valid overview of the application landscape. The easiest way to do this is to monitor the systems through your Enterprise Architecture Management (EAM) application. This should not only allow you to evaluate your starting point, but also answer the question of which systems match the target architecture of your development plan. Or at least one of the intermediate stages along the way.
This enables you to assess your initial situation and also answer the question of which existing systems correspond to the new project’s target architecture. On the one hand, this leads to the fact that the manifold questions that are needed to plan an integration project are often only rudimentarily considered. However, the problems that arise from this in the course of the project often do not result in new planning. The insufficient justification: It was "only" a little more difficult than expected. Unfortunately, a supposedly pragmatic approach is usually not enough.
Don't focus on the new system alone
Typically, companies conducting system integrations initially focus on the (newly) introduced system: decision-makers have made implementation decisions and hope for benefits from the investment. Integrating a new system into the current application landscape can produce new interactions and unexpected effects to the overall landscape. Map out the key dependencies and system interactions in advance for both the new system and the key linked systems within the application landscape.
Concentrate integration expertise in one place
Centralize the integration logic in one place, between the integrated system and the system(s) to be integrated rather than separately. This avoids creating multiple dependencies between systems that can create issues in analysis, development, testing, release and environment management. Uncoordinated, individual integrations often translate into delays and short-term problems. Longer term, it can create increased error rates, operational problems and a difficulty to sustain. That's certainly not what loose coupling is all about. This is because multiple dependencies quickly arise in analysis, development, testing, and release and environment management. As a result, delays and further problems are virtually pre-programmed.
An interdisciplinary, cross system integration team can drive a unified integration architecture and deliver tangible benefits: On the one hand, the uncontrolled, individual implementation of integrations leads to short-term problems. On the other hand, it also causes serious disadvantages in the long term: in particular with increased error rates, operational problems and a lack of maintainability. This is due not least to the head monopolies that have arisen from individual implementations. It is therefore better to centralize the integration logic in one place.
Only an interdisciplinary and cross-system integration team can drive a unified integration architecture. An interdisciplinary, cross system integration team can drive a unified integration architecture and deliver tangible benefits:
- Sensible Operation
- Continuous improvement of the interface landscape (Part 2: "Integration never stops")
- Sustainability: Retain expertise as resources scale down at the end of a project. If a uniform integration architecture is missing, problems arise during testing and stabilization because not all project experts are available on an ad hoc basis.
Conclusion: Integration needs a place so that these problems do not arise in the first place. Instead, the integration project aims at a powerful, reliable and maintainable integration architecture. Then the same rules apply to all integrations - and these are thus also understandable for everyone.
Ensure a high level of integration mindset in your project
System Integration is multi-layered and complex. Avoid the misconception that SI technical challenges are handled similarly in different systems. This is not the reality. Instead, systems often carry the same content redundantly but sometimes with widely differing meanings.
Success requires detailed analysis and specific implementation requirements. As a result, challenging analysis and implementation requirements arise, because: Integration is multi-layered and complex.
Secure the starting conditions for the integration project
Companies often underestimate the complex requirements for successful integration. The prerequisite system environments may not be sufficiently defined or prioritized at the project start. A new system is created in a new environment. Technical options to connect legacy / existing systems to the new system are not available. The environments effectively “stand” next to each other. However, it is the integration capability of the environments that creates the basis for the systems to be integrated as well.
To address this, all test environments of the new (core) system should: And even if they do, companies often prioritize them low. After all, they are usually primarily concerned with the new (core) system and not with its integration. Companies often underestimate that very complex requirements have to be implemented during integration. For this, at least three test levels must support end-to-end testing of integrations.
Be integrated with the test environments of any peripheral systems. Be implemented with consistent test data and suitable release statuses. Ensure the current peripheral systems can be tested in parallel and developed further. Ensure both developer and integrator have their own integrated test environments.
Acknowledge the complexity and impact of cross-cutting issues
In addition to the actual functional integrations, there are several other areas of consideration such as security, monitoring, logging, user and rights management. Timing and planning are required here: Many fundamental questions must be clarified at an early stage. Otherwise, the individual integration implementations are often threatened with far-reaching changes (much too) late. If you leave it up to each individual integration to implement overarching aspects, you will end up with a multitude of individual solutions - and thus no uniform integration architecture worthy of the name.
Understanding and including these peripheral activities in the project planning at an early stage is a time and resource saver. Consider the risks of uncoordinated individual integration implementations and the ensuing delays that will arise during testing, error analysis and troubleshooting. Clarifying these issues is therefore anything but simple, because changes here can even lead to central risks.
Take an agile approach to test integrations early on
An integration project is particularly suited for Agile methodology. The important thing here is to develop executable integration solutions in concrete implementation steps at an early stage. Mocking is a testing method that allows developers to isolate the behavior of an object / program by simulating the behavior of real objects. The benefit is that changes on one side or the other do not constantly hinder the development of the integration solutions. This greatly simplifies release management. Overall, this means considerably more net time is available for testing. Finally, the compatibility of the systems involved (or, alternatively, their mocks) can be better guaranteed during testing.
Budget enough time for Agile timeboxes ("sprints")
During integration development, many different elements can cause delays - such as technical queries and infrastructure problems. These are precisely the aspects that delay project progress time and again. Two-week sprints are often too tightly scheduled and don’t provide the right balance of discovery / development time. On the other hand, three-week sprints usually produce better results. Organizational, "non-agile" overhead ("waterfall mix-in") should be minimized and obstacles removed for the team.
Thoroughly Test the Integration
Comprehensive integration is critical to ensuring that production is unaffected by potential errors arising from inadequate test coverage and timing. Integration errors can only be detected if corresponding test cases are also performed. If these test cases are only formulated from the point of view of the functional systems, many errors are only discovered very late. If these test cases are only formulated from the point of view of the business systems, many errors are only discovered very late. This creates a significant production risk, because: The test coverage for the integration solutions is often set far too low. This is especially true for load and performance tests. In order to analyze the infrastructure limits in a targeted manner, an isolated integration test is absolutely essential for these tests.
Think earlier about later
Once you have completed the implementation of a new (core) system, it moves into regular maintenance operations and potentially further development. But what happens with the results and organizational experience derived from these integration projects?
Ideally, these key learnings should also be reused and further developed for use in other contexts. Integrations, by their very nature, have a completely independent character and are not exclusively tied to their original project. In times of increasing service orientation, they introduce new potential value in the application landscape. Key project learnings are often not part of the introduction and project handover planning of integration service providers. But the value of this experience to clients demands new approaches to solutions and targeted, coordinated further development.
Outlook for the second episode
We outline exactly what this means in the second part of this series - "Integration Never Stops - Challenges and Approaches". The focus here is on potentials, solution approaches and further developments when already integrated systems have to transform themselves into generally available integration services.
"New, usable potential requires new approaches to solutions and further development"