Notes of an IT Architect

** The network layer ensures the operation of WEB-interfaces in the user's browsers, transmitting data. The upper layer of applications is needed by users who are comfortable using the graphical interface and so that any routine actions are performed for them. The network layer provides communication for server applications such as WEB servers and DBMS.

** The hardware layer is represented by runners. These devices can be hardware-based with varying degrees of versatility. For example, a load balancer can be purely hardware, it can be hardware with changeable firmware, it can be software executable on a general purpose computer of the x86 type, it can be launched both directly and in a virtual machine, and in a container – all these are implementation details.

** The storage layer is made up of storage devices. These devices can be specialized devices such as IBM DataPower or regular RAID with a control module. The data in it completely describes the state and is the result of work, and the previous layers are only needed to change and provide convenient access to users.

If necessary, other layers can be implemented, for example:

* information security layer implemented by the firewall;

* a layer of basic containers;

* layer of local fault tolerance (HA) using the example of the Kubernetes layer;

* containerization layer;

* virtualization layer;

* a layer of resilience to failures implemented by load balancers on different DataCenter.

In any case, the number of layers is standardized, those that differ are indicated, so that each layer belongs to a specific info-structural department and an operation department.

Let's pay more attention to the integration architecture, since this is the most critical layer for the architecture. In this layer, connections can be presented both in a graphical form (in the arrows on the diagrams between systems), and in a tabular form – in the form of a description of the supplier, consumer and contract (the supplier's obligations to the contractor). The arrows point from the supplier to the consumer, that is, in the direction of the integration flow, while the service modules are not indicated. Depending on whether the parameters are functional or non-functional, they will be described either by API or SLA. Also, depending on whether the connections are inside or from outside to inside and outside. The first type is more visual, and the second allows you to give more detailed characteristics.

The system itself can be integrated in different ways, such as:

* direct integration (communication via API point-to-point, advantages: minimum overhead, disadvantages: two-way revision of systems is required, complexity of change management, complexity of scaling, no reuse);

* using gateways (communication through the API of an integration layer, such as a queue with a firewall, advantages: minimum overhead, unified API, disadvantages: complexity of change management, complexity of scaling, no reuse);

* Enterprise data bus or enterprise service bus (ESB) provides asynchronous umbrella integration based on the principles of event and service approach (SOA, service-oriented architecture). The corporate data bus is able to flexibly route messages from one service to another. (advantages: unification, reusability due to SOA, replaceability of services due to SOA, disadvantages: an expensive solution in many applications, delivery time from tens of milliseconds);

* Service Mesh, like ESB, is umbrella, but applications do not need to integrate with it, since applications running in a containerized environment immediately receive integration. (microservices, advantage: minimum overhead, not noticeable for application developers);

* Integration file gateways and point-to-point file transfer (file overload). Point-to-point file transfer is the same point-to-point transfer, but it allows you to transfer large data in exchange for the transmission speed (advantage: it is possible to transfer very large amounts of data, high delivery guarantee, weak connectivity of integrated systems, greater control, broadcast mode, disadvantages: transmission speed, possibility of desynchronization, high security requirements). Communication protocols are CIFS (Common Internet File System), NFS (Network File System extends the local file system) and S3 (Simple Storage Service provides access to object storage such as Minio and Ceph) and transfer protocols HTTPS (HTTP + SSL), SFTP (SSH + SSL) and FTPS (FTP + SSL). From the point of view, records can be divided into block (disk) and object (writing to the key-value database: Bucket).

Systems can communicate in various ways:

* integration request (normal synchronous request-response),

* remote procedure call (RPC),

* sending a command to the queue (from the supplier to the consumer directly through the event queue),

* publish-subscribe, Push-Sub (sending events to a common queue, from which groups of events are retrieved from the system in advance, undefined by the provider),

* packet data transmission,

* transferring files to storage,

* streaming data.

The interactions themselves should be described, and preferably unified. To describe functional or non-functional parameters (response time, availability, message size, bandwidth) of interactions between the supplier and the consumer, a Contract is used, which the supplier undertakes to fulfill. Functional parameters are described using the Application Programming Interface (API). Service APIs can be divided into groups based on message format (DTO, JSON, XML, binary) or protocol (HTTP, REST, SOAP). It is important to specify in the API contract: ID, name, version, purpose, template, specification of input and output parameters. The API itself will contain methods (encouraging the consumer to take data, change, allocate, etc.). Parameters passed in a method are described by a method specification, for example, using OpenAPI or Swagger. For many languages, and primarily for the Java language, you can automatically generate a specification for OpenAPI using Swagger by Javadoc annotations (by special comments) in the code. The specification will be displayed in both text formats (JSON and YAML) and graphical. For ease of design, you can use the Swagger Editor (https://swagger.io/tools/swagger-editor/). This helps to organize automated contract testing. In general, the presence of the specification helps to organize API-first development, when the API specification is first written, and already the application code is developed for it, implementing it.

To bind an API to the system providing it, the endpoint concept is used. Endpoint is described either by an address with a port, or MMT, hiding them. APIs are described by a machine-readable description (declaration), for example Java interface and WSDL document.

The API life cycle is close in stages to the software life cycle and the following cycle stages can be outlined: requirements gathering, interface design, implementation, testing, operation, and decommissioning stages: decommissioning warning (marked as obsolete to avoid new users), decommissioning (tracking the decrease of existing users), disposal (closing access to the API and removing it), and the stage of creating a new version. The new version of the API can be compatible and incompatible with the old one, for this you can differentiate them by semantic versioning into incompatible with the previous version, compatible with the previous version and bug fixes. By semantic versioning, the version can be written as "major.minor.patch", for example 1.9.0 – > 1.10.0 – > 1.11.0.

The solution integration layer is described in the conceptual architecture. The conceptual framework only indicates integration. It is intended for coordination with stakeholders: architects of the integrated solution, corporate architect responsible for the landscape, product owners, owner of the variable landscape, security service, resource provision service (infrastructure department) for its quick approval. The content is indicated in the target architecture (detailed architecture) of the solution. She participates in the early stages of creating a service and in the acceptance tests of the solution within the framework of architectural control. The conceptual architecture is the source of artifacts for implementing and developing a detailed solution architecture. We can distinguish the following stages of creating a conceptual architecture: planning, creation, development of an incremental draft (sketch), approval, revision. In order for the diagrams to be understood by both the developer and the owner of the product, a short description (Architecture Decision Records, ADR) is needed: the goal, the essence of the solution, rejected options, rationale, consequences, affected systems, a link to the diagram and the originator's contacts.

The corporate data bus is usually used to connect individual systems. For connections within the system – point-to-point network connection and their control using integration gateways. Integration gateways can and can be used between systems, but often, these systems are not protocol compatible and require their transformation, which is provided by the corporate data bus, but not provided by the integration gateway. And the use of a corporate data bus to connect the components of one system is expensive, since the asynchronous protocol will be loaded by other systems due to the fact that the message queue that implements it will be filled with messages from other systems and subsequent messages will wait for their sending, which causes delays in data transmission… For the exchange of large amounts of data between systems and within systems – distributed information storage.

With the layers we figured out, now, imagine the architecture as a layer cake. We can cut off a piece from it and narrow it down in more detail, but the number of layers in all pieces of this pie will be the same. By analogy, we can divide the corporate architecture into chunks. These pieces can be separate systems that are directly used on the business layer, or it can be a group of these applications developed by some department or department. The pie itself is the corporate architecture, and the piece is the solution (service) architecture. Enterprise architecture appears when we need to add new pieces to the pie, while the pieces are the architecture of the solution, and the rules according to which are made, for example, the number of layers – corporate architecture standards, cakes and cream – providing technologies for corporate architecture.

Enterprise Architecture

Corporate architecture is the architecture of the entire IT landscape of a company. A corporate architect is guided by the principles of creating architecture, a kind of constitution. The principles are described in the third section of TOGAF 9.2 under clause 20. They govern which requirements, for example, the principle of customer focus or lean manufacturing will meet. It is important that all participants (stakeholders) agree to adhere to the same principles. The principles are categorized according to their applicability to the architecture layers: business, data, application, and technology.

In practice, corporate architecture develops in three directions: unification of technologies, development of conceptual architectures, unification of the landscape. A landscape is understood not so much as an AS map, but as a set of unified solutions on the basis of which a business system is built, and with which it interacts, usually with infrastructure systems (logging, monitoring). To build the business system itself, unified solutions and technologies are used. To unify solutions, old solutions are adapted or new solutions are created, the customer of which is the department of corporate architects. To unify technologies, working groups of corporate architects are created – researchers who test the capabilities of existing solutions, analogs and make decisions either on the distribution of existing ones, or on the implementation of new ones. On the basis of research, architects – researchers create standards that describe the boundary possibilities of their applicability and regulate their use. According to GOST 1.1-2002. Standard: "a normative document that is developed on the basis of consensus, adopted by a recognized body at the appropriate level and establishes rules, general principles and characteristics for general and repeated use concerning various activities or their results, and which is aimed at achieving an optimal degree of harmonization in a certain area". It is important to emphasize that the standard fixes the agreements already found and that the stakeholders are interested in its implementation. If this is not the case, then the standard will not be met. Depending on the activities, the corporate architect may have different customers: the management of the organization, the regulatory departments (security, support, and others), the development teams of platform solutions, the architects of the development teams (to create a conceptual architecture of their future service). To maintain standards compliance, their requirements need to be validated through automated means, such as embedded in development tools and DevOps, or running validations at runtime. For this, it is necessary that the requirements of the standard are not only formalized, but also machine-readable. Not only the implementation, but also the architecture itself can be checked automatically. At the same time, checking the architecture should be carried out especially carefully, since its change over time is especially expensive afterwards. The principles of checking the corporate architecture and the solution architecture are different, since the corporate architecture is not just the sum of the designs of all solutions – it has different tasks.

When building a business system, it will be opened from existing unified solutions, new functionality, and already existing technologies and infrastructure systems will be used. For example, unification and standardization of a set of programming languages. Here, the criteria are, among other things, the economic indicators of the development cost (the speed of development in a certain language and the cost of the developers themselves with the necessary qualifications in this language), guarantees for support (the availability of a sufficient number of free personnel and resume), the complexity of support, outsourcing and others.

Enterprose Architect participates in the service development process at least in two stages – checking the developed conceptual architecture of the service and checking the compliance of the detailed architecture of the developed service with it and the standards for acceptance tests. In practice, he makes the conceptual architecture of the service himself and adds it to the service map himself, so he has the necessary experience and knowledge of all standards. In practice, Enterprose Architect assists the service architect in developing the detailed architecture of the service in its drafting and conceptual architecture in accordance with the vision of the service architect. In fact, the conceptual architecture is the architecture for integrating a service with other services to bring it into the service landscape, while the detailed architecture is the implementation of the service in accordance with the expectations of stakeholders (customers, controlling departments). It is the implementation that must comply with the restrictions imposed by Enterprise Architect on the implementation, for example, to unify technologies. If there is no collaboration, then Enterprise Architect becomes a regulatory body that blocks the deployment of the service with critical remarks or sets a technical debt with a deadline for elimination.

An enterprise architect, when developing a conceptual architecture, negotiates with the service architects with whom his service needs to integrate and finds compromise solutions, as well as other stakeholders, such as security personnel. Having agreed, he brings to a higher level of agreement, to confirm the consensus. An enterprise architect needs knowledge of services and standards, communication skills and knowledge of integration architect, such as an enterprise service bus and the ability to quickly create an architecture with insufficient information in a tightly constrained time frame.

A more lenient auditing-based standards review process can be applied, which can often be boiled down to checklists or automatic compliance checks. In practice, the service architect can go through such checklists under the guidance of Enterprise Architect for correct interpretation at the very beginning and then go through as necessary to correct compliance. Modern systems provide their setting in the form of configuration files, which allows automating the check against a list of such systems and their configuration files. Most of them work in asynchronous mode, that is, the states of the corresponding systems are asynchronously brought to the state described in these configuration files – IaaC. In most cases, the state to which the system is reduced can be obtained in a declarative form in either JSON or YML format. These formats are compatible and mutually convertible. There is an OPA (Open Policy Agent) project by the CNCF (Cloud Native Computing Foundation) consortium that allows you to create rules in a declarative form to check that they match JSON configurations. The check is carried out in a synchronous and asynchronous manner. For most systems, the application of changes is possible in the form of IaaC, that is, as sending new configuration files to the systems, on the basis of which the state of the systems is brought into conformity with them. In fact, the differences between them apply. This means that you do not need to check the current state of the system, but you can check the poisoned configuration files themselves. As a result, we have a configuration file for validating changes in a declarative format – GaaS (governance as a code). Let's look at a few layers to check:

* Layer of virtual machines (OpenStack, VMWare vSphere JSON Template);

* Network layer (VMWare NSX);

* Server configuration layer (Ansible AWX);

* Service configuration layer (Hashicorp Terraform / AWS CloudFormation);

* Layer for configuring service containers (Kubernetes / OpenShift);

* Layer of traffic routing between services (Istio, Envoy);

* Application libraries layer (NPM for JavaScript, Maven for Java, Composer for PHP).

The interaction between services (traffic routing) is described by the Istio and Envoy configuration files (more fine-tuning), which are submitted to Kubernetes and are Kubernetes configuration files. OpenShift provides a Kubernetes extension, but its config files are Kubernetes native too. Kubernetes itself is configured using YML or JSON configs transmitted asynchronously. For example, Kubernetes configuration files fully describe the state of containers (kubectl get deployment – o yaml), allowed inbound and outbound traffic from the service (kubectl get NetworkPolicy – o YAML), service accounts (kubectl get ca – o yaml), encryption between services when applied Istio (kubectl get Destination Rule – o yaml) and so on.

Many cloud providers that provide APIs for their service management clouds either have their own IaaC configurations on top of them, such as AWS CloudFormation, or integrate with the Terraform abstraction for which you can develop your model. The configurations themselves are described in a declarative form, but in their own configuration language. But, you can get the state of the reduced system state in JSON format:

terraform init

terraform plan – out tfplan.binary

terraform show – json tfplan.binary> tfplan.json

Ansible AWX is built on top of Ansible and accepts its configuration files. The configuration files themselves are written in YAML format. Ansible converts the state of servers from the list (in the inventory file) to the one described in the configuration file. Allocation of servers in OpenStack, to some extent, can also be described in YML format. At the level in VMWare NSX describes intersegment communication in configuration files in the same YML format as others. If we talk about the library layer, then many builders install and install packages according to the configuration files, so NPM in NodeJS works with JSON configuration package.JSON, Composer in PHP also works with JSON configuration composer.JSON. Conda in Python uses the conda.YAML configuration file in YAML format, which is unambiguously converted to JSON. The exception is Maven in Java, which stores XML configuration in the pom.xml file, but, as practice shows, it is not difficult to convert pom.xml to valid JSON format using Python / NodeJS.

Solution architect

The solution architect (Solution architect, Software architect), for example, a service or system, is responsible for the detailed design of the architecture of the developed solution and its API. As part of the solution, he defines the detailed design of the solution, manages the dependencies and technical debt of the solution. His work depends on the enterprise architecture (standards), the architecture of the area in which his solution belongs, and the architecture of the platform he uses. His work is judged by:

* quality and speed of development of a detailed architecture of the service,

* reuse of developed components,

* dynamics of closing technical debt.

It connects at the earliest stages of service approval by management, so that it can influence decisions regarding this service before they are made. Moreover, these are not technical decisions, but managerial ones, for example, timeframes and budget. If these decisions are made without it, then the architect will not have to decide the choice of the stack and the type of architecture, but choose what and to what extent he can implement. The architect also oversees the implementation of architectural solutions and architectural control when accepting the service. Usually, the service architect accompanies the service itself and at the reception, protects the architecture and carries out the acceptance of the work before the customer at the acceptance tests.

A service architect is a role, while by position he can be a developer, a database engineer, and a business analyst and director. Usually, when a direction is being formed, this role is played by the forming director. Later, when individual services are known, he connects either technical specialists with high communication skills, or a business analyst with an assistant in the form of leading technical specialists of this service.

Most often, this role is assumed by a technical specialist, since it is practically impossible for non-technical specialists to build up a technical base so that they can descend from the upper layers of abstraction to the lowest and see technical problems and bottlenecks. It is the ability to switch between different levels of abstraction and work effectively at them that is the key skill of an architect. But technicians use their communication skills to attract technicians where they lack expertise, which technicians are not used to doing. This is going away because the technicians do not have enough resources to cover everything themselves. Out of habit, when they had one task in their professional area 1, they plunged into it to solve it, in the role of an architect, they cannot become specialists of all technical profiles and do everything themselves, there are not so many resources for this and from them this and not required – the team and the specialists involved in the project will implement it. The downside is that technical specialists, by virtue of the ability to delve into the problems of each component, can take this into account, but this is possible on small projects, on larger projects it is solved by involving experts and corporate architects practitioners for auditing, following standards, which is familiar to managers… Another difference is the understanding of business requirements, in particular in the financial area, timing and integrations. If technical specialists, as a rule, do not have problems with integrations, then non-technical ones, on the contrary, shifting them to others, they may find themselves with the fact that it is difficult to put everything together. On the other hand, if you have strict financial and timing requirements in custom development, the project can fail, since the customer can refuse it and still demand a forfeit, and due to weak communication skills – at the end of the project. It is also observed when an architect defends a project from directors, especially from a financial one, when the architect cannot justify the choice of the technology he likes, but expensive, when in this situation there are no obvious objective grounds, for example, Java instead of PHP, Oracle instead of MySQL, micro-services instead of a monolith, a self-written solution instead of a CMS for a small online store.