Bsrgvty

I listened to a talk by panel consisted of two influential Chinese scientists: Wang Gang and Yu Kai and others.

When being asked about the biggest bottleneck of the development of artificial intelligence in the near future (3 to 5 years), Yu Kai, who has a background in the hardware industry, said that hardware would be the essential problem and we should pay most of our attention to that. He gave us two examples:

1. In the early development of the computer, we compare our machines by its chips;


2. Artificial intelligence which is very popular these years would be almost impossible if not empowered by Nvidia's GPU.

The fundamental algorithms existed already in the 1980s and 1990s, but artificial intelligence went through 3 AI winters and was not empirical until we can train models with GPU boosted mega servers.

Then Dr. Wang commented to his opinions that we should also develop software systems because we cannot build an automatic car even if we have combined all GPUs and computation in the world together.

Then, as usual, my mind wandered off and I started thinking that what if those who can operate supercomputers in the 1980s and 1990s utilized the then-existing neural network algorithms and train them with tons of scientific data? Some people at that time can obviously attempt to build AI systems we are building now. But why did AI become a hot topic and become empirical until decades later? Is it only a matter of hardware, software and data?

There is a lot of factors for the boom of AI industry. What many people miss though is the boom has mostly been in the Machine Learning part of AI. This can be attributed to various simple reasons along with their comparisons during earlier times:

• 
  Mathematics: The maths behind ML algorithms are pretty simple and known for a long time (whether it would work or not was not known though). During earlier times it was not possible to implement algorithms which require high precision of numbers, to be calculated on a chip, in an acceptable amount of time. One of the main arithmetic operations division of numbers still takes a lot of cycles in modern processors. Older processors were magnitude times slower than modern processors (more than 100x), this bottleneck made it impossible to train sophisticated models on contemporary processors.


• 
  Precision: Precision in calculations is an important factor in ML algorithms. 32 bit precision in processor was made in the 80's and was probably commercially available in the late 90's (x86), but it was still hella slow than current processors. This resulted in scientists improvising on the precision part and the most basic Perceptron Learning Algorithm invented in the 1960's to train a classifier uses only $1$'s and $0$'s, so basically a binary classifier. It was run on special computers. Although, it is interesting to note that we have come a full circle and Google is now using TPU's with 8-16 bit accuracy to implement ML models with great success.


• 
  Parallelization : The concept of parallelization of matrix operations is nothing new. It was only when we started to see Deep Learning as just a set of matrix operations we realized that it can be easily parallelized on massively parallel GPU's, still if your ML algorithm is not inherently parallel then it hardly matters whether you use CPU or GPU (e.g. RNN's).


• 
  Data: Probably the biggest cause in the ML boom. The Internet has provided opportunities to collect huge amounts of data from users and also make it available to interested parties. Since an ML algorithm is just a function approximator based on data, therefore data is the single most important thing in a ML algorithm. The more the data the better the performance of your model.


• 
  Cost: The cost of training a ML model has gone down significantly. So using a Supercomputer to train a model might be fine, but was it worth it? Super computers unlike normal PC's are tremendously resource hungry in terms of cooling, space, etc. A recent article on MIT Technology Review points out the carbon footprint of training a Deep Learning model (sub-branch of ML). It is quite a good indicator why it would have been infeasible to train on Supercomputers in earlier times (considering modern processors consume much lesser power and gives higher speeds).
  Although, I am not sure but I think earlier supercomputers were specialised in "parallel+very high precision computing" (required for weather, astronomy, military applications, etc) and the "very high precison part" is overkill in Machine Learning scenario.

Another important aspect is nowadays everyone has access to powerful computers. Thus, anyone can build new ML models, re-train pre-existing models, modify models, etc. This was quite not possible during earlier times,

All this factors has led to a huge surge in interest in ML and has caused the boom we are seeing today. Also check out this question on how we are moving beyond digital processors.

GPUs were ideal for AI boom becouse

• They hit the right time

AI has been researched for a LONG time. Almost half a century. However, that was all exploration of how would algorithms work and look. When NV saw that the AI is about to go mainstream, they looked at their GPUs and realized that the huge parellel processing power, with relative ease of programing, is ideal for the era that is to be. Many other people realized that too.

• GPUs are sort of general purpose accelerators

GPGPU is a concept of using GPU parallel processing for general tasks. You can accelerate graphics, or make your algorithm utalize 1000s of cores available on GPU. That makes GPU awesome target for all kinds of use cases including AI. Given that they are already available and are not too hard to program, its ideal choice for accelerating AI algorithms.

Sorry, but Artificial Intelligence wasn't invented yet. In the FIRST Lego League, the robots aren't able to drive on a simple line, in the DARPA robotics challenge the humanoid robots struggle to open the valve, and the Tesla Autopilot isn't recommended for real traffic situations. The only situation in which deeplearning works is on the powerpoint slides in which the accuracy to detect a cat is 100%, but in reality the normal image search engines doesn't find anything.

Let us go a step backward: what kind of AI application is available today? Right, nothing. The only control system which is available in reality is the normal refrigerator which holds the temperature at 5 degree, but this has nothing to do with machine learning but with a thermostat.

The reason why Deeplearning is available everywhere is not because it's a powerful technology for detecting images, but because it's part of the curriculum to teach humans. Deeplearning means, that the human should learn something about statistics, python programming and edge detection algorithm. Not computers will become smarter but students.

Books about the subject

Even if Deeplearning itself isn't a very powerful technique to control robots, the amount and the quality of books about the subject is great. Since the year 2010 lots of mainstream publications were published which helped to introduce Artificial Intelligence into a larger audience. All of them have something with GPU supported neural networks in the title and they are explaining very well what image recognition, motion planning and speech recognition is. Even if the readers decides not using machine learning at all but realize a robot project with the conventional paradigm he will profit from reading the newly created tutorials.