Table of Contents

1. Challenge

   Analyze the Filebeat Data

   Visualize

   1. In the Default space, create a filebeat-* index pattern with the @timestamp field set as the time filter field.
   2. Create a saved search from the filebeat-* index pattern called Failed SSH Authentications that only shows events where the field system.auth.ssh.event has a value of failed. [NOTE: If Failed is not an option, you can choose to filter by any field present in your data]. Configure the search to display the following columns:
      • source.ip
      • source.geo.continent_name
      • source.geo.country_iso_code
      • source.geo.city_name
      • source.as.organization.name
      • user.name
   3. Create a metric visualization from the filebeat-* index pattern called Failed SSH Attempts that displays a count of events where the field system.auth.ssh.event has a value of failed labelled as Failed Attempts. [NOTE: If Failed is not an option, you can choose to filter by any field present in your data]
   4. Create a tag cloud visualization from the filebeat-* index pattern called Top Failed SSH Users that shows the top 25 values of user.name based on the count of events where the field system.auth.ssh.event has a value of failed. Configure the tag cloud to not show labels.
   5. Create a map from the filebeat-* index pattern called Failed SSH Authentication Geography with the following layers:
      • Default Road map layer.
      • EMS Boundaries layer for World Countries called Countries.
        • Display the name field in the tooltip
        • Add a term join between the World Countries field ISO 3166-1 alpha-2 code and the filebeat-* field source.geo.country_iso_code field that performs a count of events labelled as "Failed Attempts" where the field system.auth.ssh.event has a value of failed.
        • Set the fill color to use the red color schema baed off the number of "Failed Attempts".
      • Documents (vector) layer on the source.geo.location field called Failed Attempts.
        • Display the fields source.as.organization.name, source.geo.city_name, source.ip, and user.name in the tooltip.
        • Add a filter to only plot events where the field system.auth.ssh.event has a value of failed.
        • Set the symbol to marker.
        • Set the fill color to use a different color for each source.ip.
        • Set the symbol size to 10.
   6. Create a dashboard called Failed SSH Authentication Attempts that includes the saved objects Failed SSH Authentications, Failed SSH Attempts, Top Failed SSH Users, and Failed SSH Authentication Geography.

   Analyze

   1. How many failed SSH authentication attempts have there been in the last 15 minutes?
   2. From what country (source.geo.country_iso_code) has there been the most failed SSH authentication attempts?
   3. For the country (source.geo.country_iso_code) with the most failed authentication attempts, what was the most attempted username (user.name)?
   4. What organization (source.as.organization.name) made the most recent failed SSH authentication attempt and from what IP address (source.ip)?
2. Challenge

   Analyze the Metricbeat Data

   Visualize

   1. In the Default space, create a metricbeat-* index pattern with the @timestamp field set as the time filter field.
   2. Create a TSVB time series visualization from the metricbeat-* index pattern called CPU Usage Over Time.
      • Create a green-colored series called User that displays the average value of the field system.cpu.user.pct formatted as a percent value and visualized as a stacked and stepped line chart with a fill of 1, line width of 0, and point size of 0.
      • Create a blue-colored series called System that displays the average value of the field system.cpu.system.pct formatted as a percent value and visualized as a stacked and stepped line chart with a fill of 1, line width of 0, and point size of 0.
      • Create a yellow-colored series called Steal that displays the average value of the field system.cpu.steal.pct formatted as a percent value and visualized as a stacked and stepped line chart with a fill of 1, line width of 0, and point size of 0.
      • Create a red-colored series called IO Wait that displays the average value of the field system.cpu.iowait.pct formatted as a percent value and visualized as a stacked and stepped line chart with a fill of 1, line width of 0, and point size of 0.
      • Create a black-colored series called Total that displays the average value of the field system.cpu.total.pct formatted as a percent value and visualized as an invisible unstacked line chart by configuring a fill of 0, line width of 0, and point size of 0.
      • Configure the time interval to be greater than or equal to 10 seconds.
   3. Create a TSVB metric visualization from the metricbeat-* index pattern called System Load.
      • Create a series called Load that displays the average value of the field system.load.1 for the latest time interval.
      • Create a series called Overall that displays the average value of the field system.load.1 but for the entire time range.
      • Configure the time interval to be greater than or equal to 10 seconds.
      • Configure the text color to turn green if the value is less than 0.75.
      • Configure the text color to turn yellow if the value is greater than or equal to 0.75.
      • Configure the text color to turn red if the value is greater than or equal to 1.
   4. Create a TSVB top n visualization from the metricbeat-* index pattern called Top Users by CPU.
      • Create a series called CPU to display the average value of the field system.process.cpu.total.pct formatted as a percent value and grouped by the top 5 terms of the field user.name ordered by the average CPU in descending order.
      • Configure the time interval to be greater than or equal to 10 seconds.
      • Configure the bar color to turn green if the value is less than 0.5.
      • Configure the bar color to turn yellow if the value is greater than or equal to 0.5.
      • Configure the bar color to turn red if the value is greater than or equal to 0.9.
   5. Create a TSVB gauge visualization from the metricbeat-* index pattern called Memory Usage Meter.
      • Create a series called Memory to display the average value of the field system.memory.used.pct formatted as a percent value.
      • Configure the time interval to be greater than or equal to 10 seconds.
      • Set the max value for the gauge to 1.
      • Configure the gauge color to turn green if the value is less than 0.75.
      • Configure the gauge color to turn yellow if the value is greater than or equal to 0.75.
      • Configure the gauge color to turn red if the value is greater than or equal to 0.95.
   6. Create a TSVB markdown visualization from the metricbeat-* index pattern called Disk Usage.
      • Create a series called Total with a variable name of total_bytes that computes the average value of the field system.filesystem.total formatted as a bytes number.
      • Create a series called Free with a variable name of free_bytes that computes the average value of the field system.filesystem.free formatted as a bytes number.
      • Create a series called Used with a variable name of used_bytes that computes the average value of the field system.filesystem.used.bytes formatted as a bytes number.
      • Create a series called Used Percent with a variable name of used_percent that computes the average value of the field system.filesystem.used.pct formatted as a percent.
      • Configure the time interval to be greater than or equal to 1 minute.
      • Enter the following markdown to render the visualization:

      You are using **{{ used.used_bytes.last.formatted }} ({{ used_percent.used_percent.last.formatted }})** of disk space out of **{{ total.total_bytes.last.formatted }}** leaving **{{ free.free_bytes.last.formatted }}** of free disk space left.
      

   7. Create a TSVB table visualization from the metricbeat-* index pattern called Top Processes.
      • Group by the top 10 of the process.name field labelled as Process.
      • Create a series called CPU that computes the average value of the field system.process.cpu.total.pct formatted as a percentage with trend arrows enabled.
      • Create a series called Memory that computes the average value of the field system.process.memory.rss.bytes formatted as a bytes number with trend arrows enabled.
   8. Create a dashboard called System Telemetry that includes the saved objects CPU Usage Over Time, System Load, Top Users by CPU, Memory Usage Meter, Disk Usage, and Top Processes.

   Analyze

   1. What user is the top CPU consumer?
   2. How much CPU steal is the system currently experiencing?
   3. What percentage of disk space is currently being used?
   4. Is the current load average greater than or less than the overall average?
   5. How much memory is the filebeat process (process.name) using, and is it trending up or down?
   6. At what time in the last 15 minutes did our system experience the highest total CPU usage, and what was the total percent used?
3. Challenge

   Analyze the eCommerce Data

   Visualize

   1. Create a new space called eCommerce with only the following Kibana features enabled:
      • Discover
      • Visualize
      • Dashboard
      • Advanced Settings
      • Index Pattern Management
      • Saved Objects Management
      • Machine Learning
      • Maps
   2. In the eCommerce space, create a ecommerce index pattern with the order_date field set as the time filter field.
   3. Configure the products.price field in the ecommerce index pattern to display as a comma separated two decimal number with a leading dollar sign.
   4. Define a single-metric machine learning job from the ecommerce index pattern called sales.
      • Use the full ecommerce data for the time range.
      • Analyze the sum of products.price.
      • Use a 1h bucket span.
      • Start the job to run real time.
   5. Create a stacked bar chart with Kibana Lens from the ecommerce index pattern called Sales by Category Per Day.
      • Show the sum of the products.price field over the order_date with 1 day time intervals.
      • Break down the bar chart for each order_date by the top 10 values of the category.keyword field ordered by the sum of products.price in descending order.
   6. Create a pie chart from the ecommerce index pattern called Top Products.
      • Calculate the sum of products.quantity.
      • Split the chart with a terms aggregation on the customer_gender field.
      • Split the slices with a top 10 terms aggregation on the products.product_name.keyword field.
      • Hide the legend and configure the visualization to show labels.
   7. Create a line visualization from the ecommerce index pattern called Sales Over Time.
      • Calculate the sum of products.price.
        • Label it as Sales.
        • Use a smooth line.
        • Don't show dots.
        • Color the line green.
      • Calculate the moving average of Sales.
        • Label it as Moving Average.
        • Use a smooth line.
        • Don't show dots.
        • Color the line orange.
      • Configure the x-axis as a date histogram of the order_date field with a auto interval and label it as Order Date.
      • Configure the legend to show at the top of the visualization.
   8. Create a metric visualization from the ecommerce index pattern called Sales Metrics.
      • Show the count of documents labelled as Orders.
      • Show the sum of products.quantity labelled as Items Sold.
      • Show the unique count of products.product_id labelled as Unique Products.
      • Show the unique count of products.category.keyword labelled as Product Categories.
      • Show the unique count of products.manufacturer.keyword labelled as Manufacturers.
      • Show the unique count of customer_id labelled as Customers.
      • Show the sum of products.price labelled as Sales.
   9. Create a data table visualization from the ecommerce index pattern called Orders.
      • Show the sum of products.quantity labelled as Items Purchased.
      • Show the sum of products.price labelled as Amount Spent.
      • Split the rows on the top 100 of order_id labelled as Order ID and ordered alphabetically in descending order.
      • Split the rows on order_date labelled as Order Date.
      • Split the rows on customer_id labelled as Customer ID.
      • Split the rows on customer_full_name.keyword labelled as Customer Name.
      • Split the rows on geoip.country_iso_code labelled as Country.
      • Split the rows on geoip.city_name labelled as City.
      • Configure the table to show 25 results per page.
   10. Create a dashboard called Sales that has a "Last 7 Days" time range and includes the saved objects Sales by Category Per Day, Top Products, Sales Over Time, Orders, and Sales Metrics.

   Analyze

   1. What is the most sold product (product.product_name.keyword) purchased by men (customer_gender is MALE) and how many men purchased that product in the last 7 days?
   2. What product category (products.category.keyword) had the most sales (products.price) 3 days ago?
   3. How many unique products with a price (products.price) of $100 or more were sold in the last 7 days?
   4. How many orders have been made so far today?
   5. How many manufacturers (products.manufacturer.keyword) make products priced (products.price) less than $19.99 and what is the top product in this price range for men (customer_gender is MALE)?
   6. When was the worst sales (sum of products.price) anomaly and how much higher or lower was the actual value versus the typical value?
4. Challenge

   Analyze the Flights Data

   Visualize

   1. Create a new space called Flights with only the following Kibana features enabled:
      • Discover
      • Visualize
      • Dashboard
      • Advanced Settings
      • Index Pattern Management
      • Saved Objects Management
      • Machine Learning
      • Maps
   2. In the Flights space, create a flights index pattern with the timestamp field set as the time filter field.
   3. Configure the AvgTicketPrice field in the flights index pattern to display as a comma separated two decimal number with a leading dollar sign.
   4. Create a scripted field for the flights index pattern called AvgTicketPricePerMile.
      • Configure the field to be a numeric data type.
      • Divide the AvgTicketPrice by DistanceMiles to get the average ticket price per mile but only if DistanceMiles is greater than 0.
      • Format the field as a comma separated two decimal number with a leading dollar sign.
   5. Define a multi-metric machine learning job from the flights index pattern called flights.
      • Use the full flights data for the time range.
      • Analyze the count of flights (each document is a flight).
      • Analyze the average of AvgTicketPrice.
      • Analyze the sum of FlightDelayMin.
      • Split the analysis on the Carrier field.
      • Set Carrier and FlightDelayType as influencers.
      • Use a 1h bucket span.
      • Start the job to run real time.
   6. Create a gauge visualization from the flights index pattern called Price Per Mile Per Carrier.
      • Show the average of AvgTicketPricePerMile labelled as Ticket Price Per Mile.
      • Split the gauge on the Carrier field and label it Carrier.
      • Set the gauge type to circle.
      • Configure 3 ranges with 0.5 point increments (0-0.5, 0.5-1, 1-1.5) and use the "Green to Red" color schema.
      • Hide both the legend and scale.
   7. Create an unstacked horizontal bar visualization from the flights index pattern called Delayed Flights by Delay Type Per Carrier.
      • Show the count of flights labelled as Flights.
      • Split the x-axis on the Carrier field labelled as Carrier.
      • Split the series on the top 10 value of the FlightDelayType field labelled as Delay Type and exclude the value No Delay.
      • Hide axis lines and labels for the y-axis.
      • Configure the legend to display in the top of the visualization.
      • Order the carriers by the sum of buckets.
      • Show value labels on the chart.
   8. Create a vertical bar visualization from the flights index pattern called Ticket Price Rate of Change Over Time.
      • Show the derivative of the average of AvgTicketPrice labelled as Change in Ticket Price.
      • Split the x-axis with a date histogram and auto interval but drop partial buckets.
      • Hide the legend.
   9. Create a controls visualization from the flights index pattern called Flight Controls.
      • Add an option list for the OriginCityName field labelled as Origin with multiselect and dynamic options enabled.
      • Add an option list for the DestCityName field labelled as Destination with multiselect and dynamic options enabled and with a parent field to Origin.
      • Add a range slider for the AvgTicketPrice field labelled as Ticket Price with a step size of 1 and 0 decimal places.
      • Configure the controls visualization to update Kibana filters on each change.
      • Configure the controls visualization to use the time filter when determining control options.
   10. Create a map from the flights index pattern called Flight Geography with the following layers:
       • Default Road map layer.
       • EMS Boundaries layer for World Countries called Countries.
         • Configure the tooltip to display the country name.
         • Add a term join for the World Countries field ISO 3166-1 alpha-2 code and the flights field OriginCountry that performs a count of events labelled as Outgoing Flights.
         • Add a term join for the World Countries field ISO 3166-1 alpha-2 code and the flights field DestCountry that performs a count of events labelled as Incoming Flights.
         • Color the regions based on the value of Incoming Flights with the blue color schema.
         • Configure the border color to be solid black with a line width of 2.
       • Documents (vector) layer for the field OriginLocation called Origin Airports.
         • Configure the tooltip to display the OriginAirportID.
         • Enable top hits per entity for the OriginAirportID field with 1 document per entity.
         • Configure the symbol to be a yellow airport icon with a symbol size of 10.
       • Documents (vector) layer for the field DestLocation called Destination Airports.
         • Configure the tooltip to display the DestAirportID.
         • Enable top hits per entity for the DestAirportID field with 1 document per entity.
         • Configure the symbol to be a orange airport icon with a symbol size of 10.
       • Point to point layer for the source field OriginLocation and destination field DestLocation called Flights.
         • Add an aggregation for the average of FlightDelayMin labelled as Delayed Minutes.
         • Color the lines by the value of Delayed Minutes with the green to red color schema.
         • Set the line width to 2.
   11. Create a dashboard called Flights that has a "Today" time range and includes the saved objects Price Per Mile Per Carrier, Delayed Flights by Delay Type Per Carrier, Ticket Price Rate of Change Over Time, Flight Controls, and Flight Geography.

   Analyze

   1. At what time today was the greatest increase in ticket prices (AvgTicketPrice) and by how much did it increase?
   2. What is the ticket price per mile (AvgTicketPricePerMile) for the carrier (Carrier) "ES-Air"?
   3. For the carrier (Carrier) "Kibana Airlines", what is the most common delay type (FlightDelayType) and how many flights are delayed so far today with said type and carrier?
   4. How many incoming and outgoing flights were there for the country of "Canada" yesterday where the ticket price (AvgTicketPrice) was less than or equal to $500?
   5. Which carrier is the most anomalous?
   6. For the carrier "JetBeats", when was the worst anomaly for the sum of FlightDelayMin, what was the FlightDelayType, and how much higher or lower was the actual value versus the typical value?
5. Challenge

   Analyze the Logs Data

   Visualize

   1. Create a new space called Logs with only the following Kibana features enabled:
      • Discover
      • Visualize
      • Dashboard
      • Advanced Settings
      • Index Pattern Management
      • Saved Objects Management
      • Machine Learning
      • Maps
   2. In the Logs space, create a logs index pattern with the @timestamp field set as the time filter field.
   3. Configure the bytes field in the logs index pattern to display as a human readable bytes number.
   4. Define a population machine learning job for the logs index pattern called clients.
      • Use the full logs data for the time range.
      • Use the clientip field as the population field.
      • Analyze the high count of requests (each document is a request).
      • Analyze the high sum of bytes.
      • Use a 15m bucket span.
      • Set clientip as an influencer.
      • Start the job to run real time.
   5. In the Logs space, create a .ml-anomalies-shared index pattern with the timestamp field set as the time filter field.
   6. Create an area visualization from the logs index pattern called Requests by Response Code Over Time.
      • Show the count of events labelled as Requests.
      • Split the series by the top 5 of response.keyword in ascending order by Requests.
      • Split the x-axis with a date histogram and an automatic time interval.
      • Color the response.keyword values such that 503 is red, 404 is yellow, and 200 is green.
      • Configure the visualization to stack each split with a stepped line mode.
      • Configure the legend to display at the top of the visualization.
   7. Create a TSVB time series visualization from the logs index pattern called Bytes Over Time.
      • Create a series called Bytes that calculates the sum of the bytes field displayed as a human readable bytes number and color the series blue.
      • Configure the visualization to hide the legend.
      • Add an annotation to plot all anomalies with a red flag icon where the machine learning job_id is clients and the function is high_sum. The clientip and the anomaly's record_score should both be displayed in the annotation's tooltip.
   8. Create a markdown visualization called Contacts with the following markdown text:

      # Contacts
      * For **visualization requests**, contact the Data Engineering team at <data@company.com>.
      * For **troubleshooting help**, contact the System Reliability Engineering team at <sre@company.com>.
      * For **incident reporting**, contact the Network Operations Center at <noc@company.com>.
      

   9. Create a map from the logs index pattern called Client Geography with the following layers:
      • Default Road map layer.
      • Grid layer for the geo.coordinates field displaying as grid rectangles called Clients.
        • Add an aggregation called Clients that calculates the unique count of the clientip field.
        • Set the grid resolution to "finest".
        • Configure the grid fill color based of the value of Clients using the green to red color schema.
   10. Create a saved search from the logs index pattern called Requests with the following columns:
       • clientip
       • url
       • response
       • bytes
   11. Create a dashboard called Web Requests that has a "Last 7 Days" time range and includes the saved objects Requests by Response Code Over Time, Bytes Over Time, Contacts, Client Geography, and Requests.

   Analyze

   1. In the last 7 days, when did we experience the most server errors (response.keyword is 503) and how many were there?
   2. For clients (clientip) using either osx or ios operating systems, what was the highest amount of requested bytes (bytes) yesterday and were there any anomalous clients around that time?
   3. Who was the most recent client (clientip) to download an RPM (extension.keyword is rpm) file and what file did they download?
   4. What is the most anomalous clientip?
   5. When was the worst anomaly for the high count of requests, what was the clientip, and how much higher or lower was the actual value versus the typical value?
   6. When was the worst anomaly for the high sum of bytes, what was the clientip, and how much higher or lower was the actual value versus the typical value?