An overview of the Elastic products past, present and future by Mélissa Benali-Richard

jsoon
  • By: jsoon
  • In: Uncategorized
  • Posted: février 18, 2018

Girls in Tech Paris is delighted to be a partner of the Elastic tour as they make their way to Paris and Munich.

Our generous partner provided Girls in Tech members with a free opportunity to be educated about Elastic on their sold-out tour.  Thank you Elastic!!

Here’s an account of the Paris conference through the eyes of Mélissa Benali-Richard:

 An overview of the Elastic products past, present and future by Mélissa Benali-Richard

This 30rd January 2018, I got the opportunity to attend the Elastic{On} Tour conference in the superb Pavillon Dauphine, at Paris. The goal of this conference was to present the products developed by Elastic, their current state and the improvements to come.

The speakers working for Elastic talked about the technical aspects of the products and also some external speakers shared their experience with the Elastic products, highlighting their key forces but also their limits. They spoke a lot about Elasticsearch that is the first product created by Elastic and that is at the heart of the Elastic stack.

Why Elasticsearch? A bit of history

A few years ago, the current CEO of Elastic, Shay Banon, was disappointed regarding  search technologies. He wanted to build a tool that could be more efficient and that could support many use cases, with the availability of getting data from multiple sources. Some tools existed such as Lucene but it was not really easy to use. Thus, Shay Banon thought about adding a layer on top of it that would make search better and easier. That is how Elasticsearch came to life.

Open Source

The Elastic team thought it would be great to have Elasticsearch as an Open Source tool. This way, people could contribute and the product would be able to evolve faster. That is exactly what happened. A lot of other open source projects came around Elasticsearch and allowed it to grow fast.

Elastic Ecosystem

Over the years, the Elastic ecosystem became richer and richer. Logstash, which allows us to collect and enrich data to feed Elasticsearch was first implemented as an independent project. Then, Elastic integrated it as a supported product. Kibana, a visualization tool, was then created and the three products – Elasticsearch, Logstash and Kibana – are now known as the ELK stack when used together. Today, there are many connectors and plugins that allows us to plug Elasticsearch over multiple data source types and to extend the integrated features of the stack.

Elastic philosophy

The Elastic team really wants to provide the user with a pleasant and smooth experience. The motto of the conference was “Simple things should be simple”.
The accessibility is a key point for the Elastic products and especially Kibana. It should be easy to get started with the product.

Improvements and new features

The goal of the Elastic products is to accomplish complex things and to produce valuable results in a simple way. To that end, a lot of technical improvements have been made, useful and highly demanded features have been added and the user experience have been enhanced.

Among the technical improvements we can mention a better resilience and a better performance along with a better management of the disk usage.

Also a new feature called “Modules” was presented. Given a data source type, Kibana now proposes default Dashboards, default alerting, etc… All of this is pre-configured to save us time and can be further adapted to be closer to our needs.

Finally, efforts are also made to be attractive for less technical people with “Canvas” that allows us to create nice, pretty slides updated dynamically that can be shared to non-technical people for example.

Security

Another important issue at Elastic is to guarantee the security when the product is used. Thus, the component “Security” from the paid version offers different tools to ensure the security through Access and Authorization layers.

Machine Learning

According to me, the Machine Learning component is the most exciting feature in Kibana. It offers a way to automate and boost the anomalies detection for time series data. It can be used, for example, to spot when a process of data exfiltration is happening on your website or when a bruteforce attack is on going.

Elastic allows us to create Machine Learning jobs, using a really advanced algorithm, that scans the past data, build a model and spot the outliers with a certain level of confidence. This tool is really powerful and they have also added the possibility of building models using multiple metrics to correlate different factors.

Another great feature of these jobs is that they can spot what they call “influencers”. Let’s say you are tracking the IPs connecting to your website and the number of connections. The number of connections becomes abnormally high. In the context of data scraping for example, Kibana could spot and point at the IP(s) creating the anomaly. These IPs are the “influencers”

Alerts can be plugged to this job to be noticed when an anomaly is detected. And finally, Kibana allows us to build a model on actual data to create forecasts for the future.

These Machine Learning features are very powerful and thanks to them, Elastic gives us a great way to add intelligence to the monitoring process.

Elastic Enterprise Cloud

When you start a project using Elastic products, the questions arises: “Where should I install my stack?”.
First, you decide to use Elasticsearch, Kibana and maybe Logstash – your project is small. You may try it in local or on remote virtual machines to create a cluster. When it proves itself, more and more data are added and it starts growing – you need more resources. At a point, other teams may be interested to add their own data. You now have a single cluster supporting multiple apps with sometimes really different needs. This situation is not sustainable, you need to split your cluster into several ones but it is not that easy to manage multiple clusters… Here comes Elastic Cloud Enterprise (ECE) to the rescue!

ECE allows us to operate easily multiple cluster. It simplifies the clusters management by making centralized and easy the version upgrades, the scaling, the nodes recoveries, the backups, the logging and monitoring.

How Elasticsearch can be used in production

Renault’s experience illustrates that Elasticsearch can be used in multiple different use cases.
Indeed, at Renault it is used for cases as wide as Marketing Intelligence, KPIs and Operational Monitoring and Incidentology.
To handle this cases, Renault has two clusters, one dedicated to Business and one dedicated to logs. Also, they use Elastic Cloud Enterprise to facilitate the management of the clusters.

Amadeus uses the Elastic products for Functional Monitoring. Elastic allows Amadeus to reach the level of reliability, flexibility, security they need thanks to the paid version. The industrialization of the processes lead to reusable components that can be used in many departments in Amadeus.

These two examples shows us that the Elastic stack can work great in production and leverage useful and valuable insights.

Conclusion

It is likely that Elastic will interest more and more companies, the different products allow to have a very complete stack, from data collection to data visualization, intelligent monitoring and alerting. Conscious of their drawbacks, the Elastic teams are permanently improving the different tools to provide more and more powerful and quality products. The Elastic stack has definitely a bright future.

 

Mélissa Benali-Richard for Girls in Tech

 

 

 

 

 

Laisser un commentaire

Votre adresse de messagerie ne sera pas publiée. Les champs obligatoires sont indiqués avec *