- SYSTEM INTEGRATION
Integration Never Stops - Challenges and Solutions
How system integration becomes a success – Part 2/2
Short development times, modular architectures and improved usability for customer and employee front-ends
Many integration solutions are initially created on a project basis. Unfortunately, after project completion, many of the IT solutions do not make the leap from project to enterprise-wide integration service. But if you want to leverage new business potential, that's exactly what you should aim for - time for a concept change.
The development of new interfaces within the scope of a new core project makes sense in principle. However, a development organization can hardly manage development that is detached from the project: The interdependencies in conception, implementation and joint testing are simply too great for this.
– Fast Lane
In this article you will learn:
- What challenges to overcome to evolve project solutions into general-purpose integration services
- 5 recommendations on how to coordinate integration solutions to your advantage
- What opportunities current integration solutions offer compared to Enterprise Service Bus (ESB)
- Why coordinated test and release management is critical to success
If the development effort exceeds the capability of internal development
Additionally, the effort required to develop appropriate solutions usually overtaxes the capabilities of an internal development department. After all, this department has responsibility to take care of daily operations and further development of existing solutions.
After project completion, the situation changes: solution development is finished for the time being. This means that fully integrated systems are now available. Other integrations can also be developed on their basis. However, if they are to be used in other contexts - especially new projects - they should avoid further, individual project solutions. Project solutions must be developed evolutionarily into generally usable integration services. This sounds simple - but it is usually not. There are a number of reasons for this, with very different characters. Here are two main reasons for this:
Requirements emerge - over time
Each integration solution is developed to meet the customer’s requirements. However, these requirements originate from the project in which the solution was developed. More general requirements that ensure universal usability are usually not taken into account here. This is usually a good thing and follows the motto "Do not implement requirements that have not been set!”. Wanting too much in the first development step significantly increases the project risk. And this often does not have the desired success in terms of content. Requirements must always cover a concrete need.
Example: If you are introducing a new inventory management system, you will also need integration with an output management system. It is very likely that other systems will also need integration later and will then have different and/or additional requirements. This is completely normal. What is needed is an analysis and further development of the existing integration service. The goal is to gradually achieve general usability - just as the principle of agile software development also envisages.
Test environments are finite resources
The development and testing of integration solutions is anything but trivial from an infrastructure perspective. You need different development and test environments. These in turn must be integrated with the environments of the core systems or peripheral systems to be connected.
In practice, major problems often arise here, especially in environments for older conversion systems. They cannot be provided for every project to the desired extent. Test environments must therefore often allow joint use by several projects (especially maintenance projects). It is imperative to keep these multiple test requirements organized. - Test releases to be tested of the most diverse solutions from different projects nor in the test data required for this.
Why is this "all of a sudden" such a pressing issue?
Technology advances, changes in customer needs, and the emerging access to new business models is driving this Industry wide urgency. Over the years, system landscapes have grown but often with monolithic systems. These were integrated in a natural and usually homogeneous way.
However, the trend towards purchased solutions now means that many systems from different vendors and architectures require integration solutions. Over the years, a system landscape with often very monolithic systems has grown. These were integrated in a natural and generally homogeneous way. However, the trend towards purchasing solutions now means that many systems from different vendors and architectures require integration solutions. However, they need to integrate them not only with each other, but also with their existing application landscapes.
IT organizations are transforming into system integrators
For the IT organization, this results in a change to your core competence from solution developer to system integrator. The new core systems and peripheral systems are provided by different software vendors or even as SaaS solutions. These "only" have to be adapted, configured and then integrated with all other systems - including legacy systems.
Until now, the focus of integration has been on large, monolithic systems. That is already changing. Systems are becoming small-scale and specialized - microservices offers are becoming more common. The demands of new business models also require a new view here: companies no longer have to deal only with their own application landscape. Cross-company integration is also becoming increasingly important.
As a result, companies have to adjust to completely new technology and customer requirements. In the past, the focus was on implementing business requirements within the company's own infrastructure. This led to the creation of technically outstanding systems. However, the maintenance and further development of these systems became a challenge that could hardly be met due to the increasing number and complexity of requirements.
Those who opt for (purchase) systems free themselves from this predicament. However, what is needed now is a strong understanding of how the new systems work. Companies need this understanding in order to address these new tasks; ensuring that a large number of very different systems or even external services from a wide range of providers work together smoothly, automatically, in heterogeneous infrastructures. Always better. Permanently.
Integration never stops
How well companies succeed in doing this determines what business value is created. It is no longer sufficient to pursue stand-alone system integration. That approach creates a complicated overall solution. Correspondingly, high maintenance costs - due to a lack of maintainability and flexibility – also result. This would also no longer be manageable from an organizational point of view, since a large number of individual solutions would lead to system interaction dependencies. To counter these risks, companies must bundle and coordinate integration tasks, both technically and organizationally:
Integration needs a place: How to coordinate integration solutions in a bundled way
To address these risks, the best approach involves bundling and coordinating integration tasks, both technically and organizationally. IKOR’s recommendations:
1 Organize integration infrastructure centrally
Establish a line organization for the central management of all resources required for the integrations. This unit is the internal service provider for all projects and systems with integration requirements. Resources include development and test environments, integration testing and acceptance, and appropriate production environments. IT organizations must also ensure that automated processes are in place to supply these environments via CI/CD pipelines. The goal is to integrate these environments with all relevant surrounding systems (from development to production). Additionally, it is important to centrally coordinate the use of these ecosystems for all projects and systems, in coordination with the test and release management of these projects.
2 Establish a standardized (cloud) infrastructure for integration solutions
New projects should develop their integration solutions directly in a standardized infrastructure from the outset. A cloud infrastructure is ideal here as it quickly connects new projects with the necessary resources IT organizations that build or use cloud infrastructure become more agile. For example, if a problem occurs, an environment can be restored in the shortest possible time. Conflicts between projects can be avoided, associated staff capacities (for the provision of environments) can be conserved and unnecessary prioritization discussions can be avoided.
3 Establish a standardized integration architecture
What applies to the design of landscapes also has significance for the architectural design of software systems. Application landscapes and their integration solutions must be developed with sustainability in mind. IT organizations need integration solutions that mediate between the systems to be integrated - autonomously. Enterprise Service Buses (ESBs) have been used in the past for precisely this purpose. Integrationslogik – und damit das Wissen aus anderen Systemen – gehört dort nicht hinein. Tight coupling and head monopolies would be the result.
Instead, IT organizations need integration solutions that mediate between the systems to be integrated - autonomously. Enterprise Service Buses (ESBs) have emerged and been used in the past for precisely this purpose. However, even ESBs have their limits. Therefore, you need to evaluate for your organization whether these "integration monoliths" with a central infrastructure, the high dependencies on the solution and the solution provider, and a principally limited flexibility are the right approach for you.
Current integration solutions offer advantages over Enterprise Service Bus
Modern integration solutions provide significantly more scope for ESB development. They make companies independent of a single central integration system.
IKOR therefore recommends developing integration solutions as microservices. They are independent and Cloud Native. Cloud Native. Open-source solutions such as the integration framework Apache Camel and Spring Boot are viable tools. IKOR’s uses these solutions with their System Integration Platform - SIP.
4 Establish management for integration solutions
The Building Blocks of integration architecture management include: Management and Support for the Integration Platform. For this purpose, it may make sense to set up a hybrid organization. Management and Support for the Integration Platform is about providing central support for all integration development. This includes standard solutions, frameworks, tooling, continuous integration and delivery (CI/CD), and a corresponding consulting offering.
The management of Individual Integration Solutions has a completely different character. Here, the focus is on concrete specialized solutions such as the integration of an output management system. The responsibility for maintenance and further development can be distributed to integration teams specialized in content. These teams are then called upon when a new project has integration requirements for the topic in question. And they are also responsible for ensuring that uniform integration services are created over time. Central integration platforms such as an ESB, on the other hand, also lead to central responsibility for the individual integrations in organizational terms. In combination with the resulting head monopolies, this creates a single point of failure - both technically and organizationally.
"When designing, always think of preserving"
Hansjörg Spielmann, mountain guide and hotelier
5 Coordinate test and release management
Testing in shared environments can cause significant dislocation in the affected projects. However, it is not practical to have only one test and release management for all projects. Therefore, ensure that test and release management for all projects is interconnected and coordinated. This can, but does not have to be done centrally. For example, two projects introduce two different new systems. These must be integrated with the same peripheral system. During testing, it can easily happen that the first project (temporarily) has to be tested with a different release of the peripheral system than the other project. If each project could access a different test environment of the surrounding system, this would not be a problem. But especially with old host systems this is often not the case. It is the same with test data where both projects have different test data constellations. The tested systems must not destroy each other's test data during testing.
Review of the first part
We outline exactly what this means in the second part of this series - "Integration Never Stops - Challenges and Approaches". The focus here is on potentials, solution approaches and further developments when already integrated systems have to transform themselves into generally available integration services.
"New, usable potential requires new approaches to solutions and further development"