The Architecture of Neural Networks

#Machine Learning
#High Tech
July 29, 2017 4 min read

A lot of success in deep neural networks and Deep Learning lays in the meticulous design of the neural network architecture software development. Below you can see the top-1 one crop precision in proportion to the number of operations needed for one forward pass in numerous popular neural network architectures.

Big data science picture

In 1994, the first complex neural network, LeNet 5 was created and this launched Deep Learning exploration. LeNet 5 architecture was rudimentary, especially the fact that image details were spread out across the entire image and convolutions with learnable frameworks are a good way to pick out feature at multiple locations with little parameters.

From 1998-2010, neural networks were in the early development stages. Few people noticed their increasing power yet lots of researchers made progress in this area. More and more data became available due to the rise of cell-phone cameras as well as low-cost digital cameras. Computing power was also gaining steam since CPUs became faster and the GPUs became a general-purpose computing tool. These trends contributed to the slow progress of neural networks and made the tasks they accomplished more interesting.


A deeper and much wider version of LeNet was created by Alex Krizhevsky in 2012 which won the ImageNet competition by a wide margin. It expanded the insights of LeNet into a significantly large neural network which can be used to learn much more complex objects and hierarchies. The innovations were the use of rectified linear units as non-linearities, overlapping max pooling and using GPUs NVIDIA GTX 580 to reduce training time.

During that time, GPUs provided much larger numbers of cores than CPUs as well as 10 times faster training time which led to the use of larger datasets and grander images.

The success of this project started a revolution. Complex neural networks became a staple of Deep Learning which became known as “big neural networks which can accomplish useful things.”

Big data science


This network had a great yet simple idea of using 1X1 complexities to enable more combinational power to the layers of complexity layers.

NiN architecture uses MLP layers after each convolution to make a better combination of features before the next layer. You may think that the 1X1 convolutions conflict with the original ideas of LeNet, but in fact, they help combine convolution features, something that is not possible by simply stacking convolutional layers. This is not the same as using raw pixels to insert into the next layer. 1X1 convolutions spatially incorporate features across feature maps so, in fact, they do not use many parameters shared across all pixels.

The strength of MLP can significantly increase the effectiveness of individual complexities features by combining them into even more complex groups. This is the very same idea used later on by ResNet and Inception. It incorporated an average pooling layer as a component of the last classifier, a practice that will become standard later on.

GoogLeNet and Inception

Christian Szegdy form Google, started a quest to find a method which reduces the computational demand of deep neural networks thus creating the GoogLeNet, the first of its kind inception architecture.

In the Fall of 2014, when Christian Szegedy created GoogLeNet deep learning models became very useful for categorizing the content of images and video frames. Even the most ardent skeptics conceded that Deep Learning and neural nets are here to stay. Since these techniques were very useful to internet giants such as Google, they became interested in the efficient and wide allocation of architectures on their servers.

Christian Szegedy’s goal was to reduce the computational demand of deep neural nets, yet at the same time retaining state-of-the-art performance, or at the very least keeping the computational cost at the current level. With this goal in mind, he and his team created the Inception module:

Big data science picture

At first, it may seem that this is a parallel combination of 1X1, 3X3 and 5X5 convolution filters, but upon closer examination, we see that by using 1X1 convolution blocks he reduces the number of features before the parallel blocks, something that we refer to today as “bottleneck.”

The “bottleneck layer deserves a section of its own, but GoogLeNet uses a classifier that has a low number of operations compared to AlexNet and VGG. It uses median pooling plus a softmax classifier similar to NiN and a stem without inception modules as part of the first layers.

Bottleneck Layer

The bottleneck layer of Inception was inspired by NiN and reduced the number of features and operations in all layers thereby keeping the inference time low. The number of features was reduced by nearly 4 times before passing the data to the expensive convolution modules. This led to significant savings in computational cost and ensured the success of this architecture.

Let’s take a look at this more closely. Imagine you have 252 features coming in and 252 coming out and the inception layer performs 3×3 convolutions. We end up with 250X250 X 3X3 convolutions that need to be performed. This is more than the entire computational budget we have to run this layer in .5 milli-seconds on a Google Server. We can reduce the number of features that must be convolved to 63 or 252/4. First, we perform 252-> 63 1X1 convolutions and then 63 convolutions on each branch and an additional 1X1 convolution from 63->252 features back once more. In total, we get about 70,000 operations instead of almost 600,000 we had before.

Even though we are doing fewer operations, we retain generality in this layer. The bottleneck layers perform remarkably on the ImageNet dataset. The reason it was so successful is that the input features are correlated and the repetition can be removed by combining them as needed with the 1×1 convolutions. Then, they can be expanded again into a meaningful combination for the next layer after convolution with a smaller number of features.

As for the future, we believe that creating neural networks is critically important to the progress of the Deep Learning field. We could speculate into why we have to invest so much time in the creations of architectures instead of using data to tell what to use and how to combine modules. Although this would be very helpful, it remains a work in progress. Also, it is important to note that so far, we have talked about architectures for computer vision. Neural networks architectures have been created in other areas as well and it is interesting to study how these architectures evolved for all the other tasks.

The future is sure to bring many wonderful innovations and software development solutions in the world of neural network architectures. These innovations will also have practical applications and making all of our devices smarter and easier to use.

4.49 5.0
21 Reviews
October 22, 2021
Svetlana Rumyantseva, Project Manager: “I want to see happiness around. And it is usually in small things”
Who did you dream of becoming as a child? I dreamed of becoming a lawyer and defending people’s rights. The most challenging question for me has always been whether I could defend the guilty. I haven’t come up with an answer, even after graduation. Back in the day, I wanted to be a lawyer to uphold justice. How important is family support in becoming an individual and a professional? Were you supported in choosing a career path? Support is critical. Its impact is difficult to overestimate. I realized this, probably at the age of 27. Since I wanted to be a lawyer, I went to study at the law academy. At the end of the fifth year, I realized that I would not be a lawyer. Politics and public administration seemed more interesting to me. My family didn’t stop me; it was me who thought, “Why did I study at all?”. Then I continued my studies, and after graduate school, I did a degree in a completely different specialty — political science. Obviously, I am not a political scientist now. However, during my postgraduate studies, I began to study management not only in the state but also in the project type, which I really liked. Now I want to thank my family, who did not reproach me about the time spent, telling me something like: “You’ve wasted ten years, both specialties are no longer needed, and you are in the middle of nowhere.” Therefore, support is vital. And even if support is not active but passive — being able not to interfere with the development of others. What qualities do you think are most important in your profession? It’s sociability and everything that no one reads in the classic “proper” resume. Stress resistance, purposefulness — these are the qualities. On the one hand, this is a pun, but in fact, a lot is built on those features. Also, I would point out multitasking and the ability to react quickly to circumstances. If something is already out of control, you must first respond to the problem, and only then be afraid that something may not work out. I adore myself for being optimistic. I vote for giggling once more than falling into depression, which no one needs. Practice shows that it is difficult to get angry, despair, and give up when someone smiles and supports you. So is this your superpower that helps you, the team, and inspires other people? Maybe. It’s always good to have someone around who can say with a smile, “It’s no big deal. We’ll figure it out together.” If I can be this person for my surroundings, that’s great. What does your typical workday look like? Sometimes it seems that all working days are the same, but it is nothing like that in reality. Every day is about communicating with numerous people, solving problems, questions, planning for the future, and monitoring what went wrong. Of course, all this is mixed with letters, reports, processing and writing documentation, maintaining contact with all stakeholders. In other words, each of my days is very different and very similar to the previous ones at the same time. What challenges do you encounter in your daily work? As a rule, the most difficult is the challenge of the communication type: to agree, encourage, persuade, approve. Negotiation issues are the most difficult, as they often lie in different areas, for example, the team or the customers. It is imperative to understand everyone and reach a win-win condition. Generally speaking, what is the most difficult thing in your work? Having time and choosing a quality solution in specific conditions. The main task of the manager and me, in particular, is to create the most productive working conditions so that everything is done under the agreement. Sometimes it is necessary to do it before a specific time, on a certain date, or to ease the team’s workload, giving up everything and paying attention to a particular issue. In addition, I have to call everyone, discuss everything, help, and so on. Therefore, the most difficult thing is to squeeze in 24 hours. What helps you in this? It’s the priority system: first, we do the important thing, then comes another important thing, but the one that can wait a bit. Often, tasks that directly affect customers are automatically prioritized. Likewise, the same pertains to controversial issues within the team. I try to plan the day in a way that I deal with work blockages by noon and then do the rest. Therefore, prioritization is our everything. And what is most interesting in your work? Each challenge is exciting in its own way. The most remarkable people constantly surround me; thus, communication is the most complex and exciting. Daily contact with the team and stakeholders always encourages self-development because I learn a lot from them. I am driven by challenges, the process of solving them, and reaching the result. And how do you feel when you see the result? This is the feeling of Zen. We move toward it every day, solving some problems along the way. I rejoice when I see the results of my work and manage to help the team and clients; when the team flaunts its achievements, I see the clients’ positive emotions caused by us. Every little victory is very inspiring. How do you assess the cohesion of your team? How does this affect the result and quality of the task performed? We have a very friendly team, and I think that’s crucial. There are managers supporting work processes in the team purely, but I favor the atmosphere in the team that is family-like and friendly. When you work in cohesion, everyone has a common goal, vision, and one desired result. Unity means you understand each other right away. Everyone knows each other’s thoughts, who does what, what one enjoys and wants. And this motivates me to get things done, help instead of blocking or setting someone up. I like that we have such an atmosphere throughout the gaming division, regardless of the project. Is there a practice in your team where everyone can freely speak their minds? Sure, but certain filtering must be considered, of course — turning information into positive. If it is critical feedback, then it should be constructive. Everyone is mindful of politeness and diplomacy. We focus not only on the result and teamwork but also on personal growth, so everyone is happy to receive constructive feedback on the work. In your opinion, how will technology change your profession in the future? Hopefully, for the better. Developing new games, in any case, will require constant communication and creativity, so it would be nice to see technology help engage the team. Lack of communication can be particularly acute in the distributed and remote teams. I hope that in the future, thanks to technology, we will see: An opportunity to invest less time in the development of new content and have more time for learning, self-development, and family; Technologies becoming lighter and more intuitive for use by both developers and non-technicians; Remote mode getting more smooth and convenient. I totally would like to see positive changes in the remote mode, not only maintaining but also improving the team’s engagement regardless of location. I’d be interested in seeing what it would look like in 5-10 years. But I am positive that communication will remain in the foreground. How should one start learning your profession? I would advise being curious and trying to stay like that all the time. It is essential to read, listen, communicate with people in this field as much as possible. This is how theory is developed as a basis for the practical use of skills. For example, I started my journey as a project coordinator in website development, and in my free time, I practiced making prototypes and mockups. First, I recreated popular services and then came up with my own schemes. Therefore, there are three pillars: curiosity, theory, and practice. What are you studying now, and what would you like to learn in the future? Today I learn everything from my team. Since I am not a technical person, I have an open field of new knowledge and skills in front of me, and I am happy to learn how everything works. In addition, project tasks encourage me to deal with a variety of tools. I have a lot of interesting work ahead that is related to the marketing and analytics of the product we are working on. I have to learn a lot of new things. But this is awesome. In fact, there are a million areas where you can (and in a certain period of time — should) develop. That is, the novelty does not scare you? It is very unreasonable to think that one can know everything. On the contrary, it is essential to be clear about how little you know because that is what motivates you to develop and constantly grow your value. Is there a work/life balance and clear boundaries between work and life outside it? I have no clear borderline between work and life. I can even say that I’ve never had one. Furthermore, I often think about work at home, on weekends, and during vacation. It isn’t easy to let it go because I also plan everything at home. Work and private life always intersect, and so far, I can’t say that it bothers me very much. I may want to draw the line someday, but now it seems difficult to do so for a managerial position. Sometimes a reaction to external or internal issues is required at a very unexpected moment. Do you have new hobbies/hobbies/activities since joining Innovecs? Probably not. However, I assume that the Innovecs’ environment encouraged me to move from dream to action. Thanks partly to my colleagues, I decided to buy a motorcycle. We took trips with our Moto Community, and it was incredible. It seems that if it were not for this motivation from the Community, I would not have dared to buy a motorcycle this moto-season. Innovecs is great no matter what you do and what you would like to do. A variety of Communities constantly is a call to action. You can’t be inert. What inspires you? I’m looking for inspiration in the results. They motivate me to move on to new achievements. I can’t stand still, and I feel bad if nothing happens for half a day. In general, I am very grateful to my colleagues because if I seem to give up at a certain point, the team comes with results and achievements and invigorates me. It’s a kind of perpetual motion machine. What motivates you to stay at Innovecs? People and growth. At Innovecs, our gaming division is hugely passionate. I want to come here, and I don’t want to leave. Which of Innovecs’ corporate culture do you like the most? Innovecs culture is based on people. Our SPECIFIC DNA perfectly reflects a focus on the people we are looking for in teams. Working at Innovecs, I feel that the team, colleagues, and I are treated as individuals with their own interests in the first place, but not as a resource that can be replaced (although business is business). For me, this attitude is important, because we spend most of our time at work. What makes you a happy person? A little of this and that, and everything that does not kill. I love life. I have great friends, a work environment, a family, and I have a hobby — all this inspires and makes me happy. What do you dream about today? Right now, I dream of going to the ocean for the New Year’s celebration. In general, I dream of always being able to fulfill my desires. And I want to see happiness around. And it is usually in small things.  
November 28, 2019
Is Microservices Architecture Only About the Benefits?
Microservices architecture is not a new thing in software development. However, today this approach to building applications is actively gaining traction. In 2018, Camunda conducted research and surveyed 354 companies to determine the value of microservices adoption for fulfilling digital transformation initiatives. Around 63% of respondents mentioned that the transition to microservices helped them improve user experience, employee efficiency, and save on development and infrastructure tools. The respondents also identified top reasons why they had switched to microservices: Additional reasons for switching to the microservices Today, developing a monolithic program comes with various challenges. For example, the more complicated the composition of the monolith is, the more difficult it is for developers to properly maintain it. Let’s take a look at a scenario where there are many teams working on the same monolithic program. If team B fails to deliver high-quality code, it blocks the other teams from pushing their error-free code to the environment. So, all teams wait till team B fixes the code and tries to deploy it again. All teams are dependant on each other, which results in constant delays in delivering features. Additionally, adopting new technologies while having a monolith application can be a very slow process. For example, if you want to apply a new framework, you must rewrite all the code, which can also become a huge challenge and lead to crashes. To minimize these risks, software companies have started actively using microservices application structure. However, this “monolith → microservices” transition may come with its own challenges. So, in this blog post, we are going to cover the following topics: The peculiarities of microservices architecture and some examples Benefits of microservices and drawbacks that can occur along with these benefits Successful cases of switching to microservices structure
Article, Big Data & Highload, Software Development