Artificial intelligence. Freefall
- -
- 100%
- +
And we can see how leaders are building up computing power, building new data centers and hurriedly solving energy supply and cooling issues for these monsters. At the same time, improving the quality of the model by a conditional 2% requires an increase in computing power by an order of magnitude.
Now a practical example to the question of maintenance and maintenance due to degradation. Tut will also be noticeable about the effect of people. Any AI, especially at an early stage, will learn based on feedback from people (their satisfaction, initial requests and tasks). For example, the same ChatGPT4 uses user requests to retrain its model in order to give more relevant answers and at the same time reduce the load on the “brain”. And at the end of 2023, there were articles that the AI model has become “more lazy”. The chatbot either refuses to answer questions, interrupts the conversation, or simply responds with excerpts from search engines and other sites. And by mid-2024, this has already become the norm, when the model simply cites excerpts from Wikipedia.
One possible reason for this is the simplification of the user requests themselves (they are becoming more primitive). After all, LLMs do not invent anything new, these models try to understand what you want them to say and adapt to it (in other words, they also form stereotypes). It also looks for the maximum efficiency of the labor-result bundle, “disabling” unnecessary neural connections. This is called function maximization. Just math and statistics.
Moreover, this problem will be typical not only for LLM.
As a result, to prevent the AI from becoming degraded, you will have to load it with complex research, while limiting its load to primitive tasks. And once it is released into the open world, the ratio of tasks will be in favor of simple and primitive user requests or solving applied problems.
Remember yourself. Do you really need to evolve to survive and reproduce? Or what is the correlation between intellectual and routine tasks in your work? What level of math problems do you solve in this job? Do you need integrals and probability theory, or just math up to 9th grade?
The second factor is the amount of data and hallucinations.
Yes, we can increase the current models by XXXX times. But the same ChatGPT5 prototype already lacks training data in 2024. They gave him everything, they had. And with a modern AI that will navigate uncertainty, there simply won’t be enough data at the current level of technology development. You need to collect metadata about user behavior, think about how to circumvent copyright and ethical restrictions, and collect user consent.
In addition, using the current LLMs as an example, we can see another trend. The more “omniscient” a model is, the more inaccuracies, errors, abstractions, and hallucinations it has. At the same time, if you take a basic model and give it a specific subject area as knowledge, then the quality of its responses increases: they are more objective, she fantasizes less (hallucinates) and makes fewer mistakes.
The third factor is vulnerability and costs.
As we discussed above, we will need to create a data-center worth a trillion US dollars. And its power consumption will exceed all current electricity generation in the United States. This means, that the creation of an energy infrastructure with a whole complex of nuclear power plants will also be required. Yes, windmills and solar panels can’t solve this problem.
Now let’s add that the AI model will be tied to its “base”, and then one successful cyber-attack on the energy infrastructure will de-energize the entire “brain”.
And why should such an AI be tied to the center, why can’t it be distributed?
First, distributed computing still loses performance and efficiency. These are heterogeneous computing capacities that are also loaded with other tasks and processes. In addition, a distributed network cannot guarantee the operation of computing power all the time. Something turns on, something turns off. The available power will be unstable.
Secondly, it is a vulnerability to attacks on communication channels and the same distributed infrastructure. Imagine that suddenly 10% of the neurons in your brain just turned off (blocking communication channels or just turned off due to an attack), and the rest are working half-heartedly (interference, etc.). As a result, we again have the risk of a strong AI that forgets who it is, where it is, for something, then just thinks for a long time.
And if everything comes to the point that a strong AI will need a mobile (mobile) body to interact with the world, then this will be even more difficult to implement. After all, how to provide all this with energy and cool it? Where do I get data processing power? Plus, you also need to add machine vision and image recognition, as well as processing other sensors (temperature, hearing, etc.). This is huge computing power and the need for cooling and energy.
That is, it will be a limited AI with a permanent wireless connection to the main center. And this is again a vulnerability. Modern communication channels give a higher speed, but this affects the reduction of range and penetration, vulnerability to electronic warfare. In other words, we get an increase in the load on the communication infrastructure and an increase in risks.
Here, of course, you can object. For example, the fact that you can take a pre-trained model and make it local. In much the same way as I suggest deploying local AI models with “additional training” in the subject area. Yes, in this form, all this can work on the same server. But such an AI will be very limited, it will be “stupid” in conditions of uncertainty, and it will still need energy and a connection to the data transmission network. That is, this story is not about the creation of human-like super-beings.
All this leads to questions about the economic feasibility of investing in this area. Especially considering two key trends in the development of generative AI:
– creating cheap and simple local models for solving specialized tasks;
– create AI orchestrators that will decompose a request into several local tasks and then redistribute it between different local models.
Thus, weak models with narrow specialization will remain more free and easier to create. At the same time, they will be able to solve our tasks. And as a result, we have a simpler and cheaper solution to work tasks than creating a strong AI.
Of course, we leave out neuromorphic and quantum systems, but we will discuss this topic little later. And, of course, there may be mistakes in my individual figures and arguments, but in general I am convinced that strong AI is not a matter of the near future.
In summary, strong AI has several fundamental problems.
– Exponential growth in the complexity of developing and countering degradation and complex models.
– Lack of data for training.
– Cost of creation and operation.
– Attachment to data centers and demanding computing resources.
– Low efficiency of current models compared to the human brain.
It is overcoming these problems that will determine the further vector of development of the entire technology: either a strong AI will still appear, or we will move into the plane of development of weak AI and AI orchestrators, which will coordinate the work of dozens of weak models.
But now strong AI does not fit in with ESG in any way, environmentalists like it commercial. Its creation is possible only within the framework of strategic and national projects financed by the state. And here is one of the interesting facts in this direction: the former head of the US National Security Agency (until 2023a), retired General, Paul Nakasone joined the board of directors of Open AI in 2024. The official version is for organizing Chat GPT security.
I also recommend reading the document titled “Situational Awareness: The Coming Decade”. Its author is Leopold Aschenbrenner, a former Open AI employee from the Super alignment team. The document is available by QR-code and hyperlink.
SITUATIONAL AWARENESS
A shortened analysis of this document is also available using the QR-code and hyperlink below.
Analysis of the AGI document by Leopold Aschenbrenner, former Open AI employee
To simplify it completely, the author’s key theses are:
– By 2027, strong AI (AGI) will become a reality.
I disagree with this statement. My arguments are given above, plus some of the theses below and risk descriptions from the authors say the same. But again, what is meant by the term AGI? I have already given my own definition, but there is no single term.
– AGI is now a key geopolitical resource. We forget about nuclear weapons; this is the past. Each country will strive to get AGI first, as in its time an atomic bomb.
The thesis is controversial. Yes, this is a great resource. But it seems to me that its value is overestimated, especially given the complexity of its creation and the mandatory future errors in its work.
– Creating an AGI will require a single computing cluster worth a trillion US dollars. Microsoft is already building one for Open AI.
Computing power also requires spending on people and solving fundamental problems.
– This cluster will consume more electricity than the entire US generation.
We discussed this thesis above. More than a trillion dollars is also invested in electricity generation, and there are also risks.
– AGI funding will come from tech giants – already Nvidia, Microsoft, Amazon, and Google are allocating $100 billion a quarter to AI alone.
I believe that government funding and, consequently, intervention is essential.
– By 2030, annual investment in AI will reach $8 trillion.
Excellent observation. Now the question arises, is this economically justified?
Despite all the optimism of Leopold Aschenbrenner regarding the timing of the creation of AGI, he himself notes a number of limitations:
– Lack of computing power for conducting experiments.
– Fundamental limitations associated with algorithmic progress
– Ideas are becoming more complex, so it is likely that AI researchers (AI agents who will conduct research for people) will only maintain the current rate of progress, and not increase it significantly. However, Aschenbrenner believes that these obstacles can slow down, but not stop, the growth of AI systems ' intelligence.
Chapter 3. What can weak AI do and general trends?
Weak AI in applied tasks
As you, probably already, understood, I am a proponent of using what is available. Perhaps this is my experience in crisis management, and whether it’s just an erroneous opinion. But still, where can the current weak AI based on machine learning be applied?
The most relevant areas for applying AI with machine learning are:
– forecasting and preparing recommendations for decisions taken;
– analysis of complex data without clear relationships, including for forecasting and decision-making;
– process optimization;
– image recognition, including images and voice recordings;
– automating the execution of individual tasks, including through content generation.
The new direction, which is at its peak of popularity in 2023—2024, is image recognition, including images and voice recordings, and content generation. This is where the bulk of AI developers and most of these services come from.
At the same time, the combination of AI + IoT (Internet of Things) deserves special attention:
– AI receives pure big data, which does not contain human errors, for training and finding relationships.
– The effectiveness of IoT increases, as it becomes possible to create predictive analytics and early detection of deviations.
Key trends
– Machine learning is moving towards an increasingly low entry threshold.
One of the tasks that developers are currently solving, is to simplify the creation of AI models to the level of site designers, where special knowledge and skills are not needed for basic application. The creation of neural networks and data science is already developing according to the “service as a service” model, for example, DSaaS – Data Science as a Service.
You can start learning about machine learning with AUTO ML, its free version, or DSaaS with initial audit, consulting, and data markup. You can even get data markup for free. All this reduces the entry threshold.
– Creating neural networks that need less and less data for training.
A few years ago, to fake your voice, it was necessary to provide a neural network with one or two hours of recording your speech. About two years ago, this indicator dropped to a few minutes. Well, in 2023, Microsoft introduced a neural network that takes just three seconds to fake.
Plus, there are tools, that you can use to change your voice even in online mode.
– Create support and decision-making systems, including industry-specific ones.
Industry-specific neural networks will be created, and the direction of recommendation networks, so-called “digital advisors” or solutions of the class “support and decision-making systems (DSS) for various business tasks” will be increasingly developed.
Practical example
We will look at this case again and again, as it is my personal pain and the product I am working on.
There is a problem in project management – 70% of projects are either problematic or fail:
– the average excess of planned deadlines is observed in 60% of projects, and the average excess is 80% of the original deadline;
– 57% of projects exceed their budgets, while the average excess is 60% of the initial budget;
– failure to meet the success criteria – in 40% of projects.
At the same time, project management already takes up to 50% of managers ' time, and by 2030 this figure will reach 60%. Although at the beginning of the 20th century, this figure was 5%. The world is becoming more and more volatile, and the number of projects is growing. Even sales are becoming more and more “project-based”, that is, complex and individual.
And what does such project management statistics lead to?
– Reputational losses.
– Penalties.
– Reduced marginality.
– Limiting business growth.
The most common and critical errors are:
– unclear formulation of project goals, results, and boundaries;
– insufficiently developed project implementation strategy and plan;
– inadequate organizational structure of project management;
– an imbalance in the interests of project participants;
– ineffective communication within the project and with external organizations.
How do people solve this problem? Either they don’t do anything and suffer, or they go to school and use task trackers.
However, both approaches have their pros and cons. For example, classical training provides an opportunity to ask questions and practice various situations during live communication with the teacher. At the same time, it is expensive and usually does not imply further support after the end of the course. Task trackers, on the other hand, are always at hand, but they do not adapt to a specific project and company culture, do not contribute to the development of competencies, but on the contrary, are designed to monitor work.
As a result, after analyzing my experience, I came up with the idea of a digital advisor – artificial intelligence and predictive recommendations “what to do, when and how” in 10 minutes for any project and organization. Project management becomes available to any manager conditionally for a couple of thousand rubles a month.
The AI model includes a project management methodology and sets of ready-made recommendations. The AI will prepare sets of recommendations and gradually learn itself, finding new patterns, and not be tied to the opinion of the creator and the one who will train the model at the first stages.
Chapter 4. Generative AI
What is generative artificial intelligence?
Earlier, we reviewed the key areas for applying AI:
– forecasting and decisions-making;
– analysis of complex data without clear relationships, including for forecasting purposes.
– process optimization;
– image recognition, including images and voice recordings.
– content generation.
The areas of AI that are currently at the peak of popularity, are image recognition (audio, video, numbers) and content generation based on them: audio, text, code, video, images, and so on. Generative AI also includes digital Expert Advisors.
Generative AI Challenges
As of mid-2014, the direction of generative AI cannot be called successful. For example, in 2022, Open AI suffered a loss of $540 million due to the development of ChatGPT. And for further development and creation of a strong AI, about $ 100 billion more will be needed. This amount was announced by the head of Open AI himself. The same unfavorable forecast for 2024 is also given by the American company CCS Insight.
For reference: the operating cost of Open AI is $ 700,000 per day to maintain the chat bot ChatGPT.
The general trend is supported by Alexey Vodyasov, Technical Director of SEQ: “AI does not achieve the same marketing results that we talked about earlier. Their use is limited by the training model, and the cost and volume of data for training is growing. In general, the hype and boom are inevitably followed by a decline in interest. AI will come out of the limelight as quickly as it entered, and this is just the normal course of the process. Perhaps not everyone will survive the downturn, but AI is really a” toy for the rich”, and it will remain so in the near future.” And we agree with Alexey, after the hype at the beginning of 2023, there was a lull by the fall.
Adding to the picture is an investigation by the Wall Street Journal, according to which the majority of IT giants have not yet learned how to make money on the capabilities of generative AI. Microsoft, Google, Adobe and other companies that invest in artificial intelligence, are looking for ways to make money on their products. Here are some examples:
– Google plans to increase the subscription price for AI-enabled software;
– Adobe sets limits on the number of requests to services with AI during the month.
– Microsoft wants to charge business customers an additional $30 per month for the ability to create presentations using a neural network.
Well, and the icing on the cake-calculations by David Cahn, an analyst at Sequoia Capital, showing that AI companies will have to earn about $600 billion a year to offset the costs of their AI infrastructure, including data centers. The only, one who now makes good money on AI, is the developer of Nvidia accelerators.
More information about the article can be found in the QR-code and hyperlink below.
AI industry needs to earn $600 billion per year to pay for massive hardware spend – fears of an AI bubble intensify in wake of Sequoia report
Computing power is one of the main expenses when working with Gen AI: the larger the server requests, the larger the infrastructure and electricity bills. Only suppliers of hardware and electricity benefit. So, Nvidia in August, 2023 earned about $5 billion thanks to sales of the accelerators for AI A100 and H100 only to the Chinese IT sector.
This can be seen in two examples on practice.
First is Zoom tries to reduce costs by using a simpler chatbot developed in-house and requiring less computing power compared to the latest version of ChatGPT.
Second is the most well-known AI developers (Microsoft, Google, Apple, Mistral, Anthropic, and Cohere) began to focus on creating compact AI models, as they are cheaper and more cost-effective.
Larger models, such, as Open AI’s GPT-4, which has more than 1 trillion parameters and is estimated to cost more than $ 100 million to build, do not have a radical advantage over simpler solutions in applications. Compact models are trained on narrower data sets and can cost less than $ 10 million, while using less than 10 billion parameters, but solve targeted problems.
For example, Microsoft introduced a family of small models called Phi. According to CEO Satya Nadella, the model’s solutions are 100 times smaller than the free version ChatGPT, of ChatGPT, but they handle many tasks almost as efficiently. Yusuf Mehdi, Microsoft’s chief commercial officer, said the company quickly realized that operating large AI models is more expensive than initially thought. So, Microsoft started looking for more cost-effective solutions.
Apple also plans to use such models to run AI directly on smartphones, which should increase the speed and security. At the same time, resource consumption on smartphones will be minimal.
Experts themselves believe that for many tasks, for example, summarizing documents or creating images, large models may generally be redundant. Ilya Polosukhin, one of the authors of Google’s seminal 2017 article on artificial intelligence, figuratively compared using large models for simple tasks to going to the grocery store on a tank. “Quadrillions of operations should not be required to calculate 2 +2,” he stressed.
But let’s look at everything in order, why did this happen and what restrictions threaten AI, and most importantly, what will happen next? Sunset of generative AI with another AI winter or transformation?
AI limitations that lead to problems
Earlier, I gave the “basic” problems of AI. Now, let’s dig a little deeper into the specifics of generative AI.
– Companies ' concerns about their data
Any business strives to protect its corporate data and tries to exclude it by any means. This leads to two problems.
First, companies prohibit the use of online tools that are located outside the perimeter of a secure network, while any request to an online bot is an appeal to the outside world. There are many questions about how data is stored, protected, and used.
Secondly, it limits the development of any AI at all. All companies from suppliers want IT solutions with AI-recommendations from trained models, which, for example, will predict equipment failure. But not everyone shares their data. It turns out a vicious circle.
However, here we must make a reservation. Some guys have already learned how to place Chat GPT level 3 – 3.5 language models inside the company outline. But these models still need to be trained, they are not ready-made solutions. And internal security services will find the risks and be against it.
– Complexity and high cost of development and subsequent maintenance
Developing any” general” generative AI is a huge expense-tens of millions of dollars. In addition, the am needs a lot of data, a lot of data. Neural networks still have low efficiency. Where 10 examples are enough for a person, an artificial neural network needs thousands, or even hundreds of thousands of examples. Although yes, it can find such relationships, and process such data arrays that a person never dreamed of.
But back to the topic. It is precisely because of the data restriction that ChatGPT also thinks “better” if you communicate with it in English, and not in Russian. After all, the English-speaking segment of the Internet is much larger than ours.
Add to this the cost of electricity, engineers, maintenance, repair and modernization of equipment, and get the same $ 700,000 per day just for the maintenance of Chat GPT. How many companies can spend such amounts with unclear prospects for monetization (but more on this below)?
Yes, you can reduce costs if you develop a model and then remove all unnecessary things, but then it will be a very highly specialized AI.