We're Moving!
Together with the release of FlowViewer version 4.0 we are moving to SourceForge
FlowViewer Version 4.0 is a Major Upgrade, including ...
|
* Requires SiLK version 3.0 or above. Version 3.0 is available to US Federal Agencies and Universities. It is planned for release to the general public in the near future.
| |
|
FlowViewer now works with both flow-tools and SiLK at the same time making the transition to version 4 an easy one. Processing is included to carry over any filters, saved reports, or FlowTrackings established with an earlier version of FlowViewer with minor user intervention. Having a management tool that simultaneously supports older and newer netflow capture and analysis also allows network organizations to migrate to the new protocol on a schedule that is optimal for them.
FlowViewer is an umbrella set of three tools: FlowViewer, FlowGrapher, and FlowTracker. These tools provide an easy web-based user interface for selecting, viewing, graphing, and now tracking netFlow data stored using Mark Fullmer's flow-tools (new fork) software, or, now with version 4.0, Carnegie-Mellon's NetSA group's SiLK. The user is able to filter data (inclusion or exclusion) by device, IP address range, port, router interface, autonomous system (AS), specified time interval, protocols, TOS field, TCP flags, and now Exporter, and Next Hop. Many of the flow-tools reports are configured as drop-down selections. Users are also able to save reports and graphs for later viewing, as well as track filtered data over the long run. FlowViewer makes flow data analysis and tracking quick and easy. The user must install and configure either flow-tools, or SiLK. Users already running FlowViewer with flow-tools may opt to install SiLK in parallel to handle IPFIX exporters while leaving older exporters in place. FlowGrapher will require the installation of Lincoln Stein's GD, and Martien Verbruggen's (now Benjamin Warfield's) GD::Graph packages for Perl. FlowTracker will require installation of Tobi Oetiker's RRDtool package. A FlowViewer configuration file is quickly configured and FlowViewer is almost immediately put into operation. Many thanks to FlowViewer v1-v3 users who have contributed ideas, with a special thanks to Wojciech Kozicki whose suggestion doubled the FlowViewer processing speed, and Mark Foster, Ed Ravin and others for contributions to v3. Many thanks to Carles Kishimoto, Eric Lautenschlaeger, and Sean Cardus for their ideas and code contributions to v3.3. Thanks to Dario La Guardia for pointing out a graphing problem that turned out to be a rounding error in FlowGrapher. Credit to Peter Hoffswell for the idea of linking the tools. Mike Stowe has provided valuable help with a v4.0 beta. Contact Joe Loiacono if you have problems with download, installation, configuration, or operation. Or, check the FAQ |
|
Requirements
1. Web Server w/ CGI |
Screen Shots
FlowViewer Main Screen
|
Download FlowViewer
FlowViewer_4.0.tar |
FlowViewer v4.0
# README (this file) FlowViewer V4.0 Date: 05/08/2012
#
# FlowViewer is a set of three tools (FlowViewer, FlowGrapher,
# FlowTracker) that create text reports, graph reports, and
# long-term tracking reports from flow-tools and SiLK captured
# and stored netflow data. FlowViewer can run with both flow-tools
# and SiLK simultaneously. Flow-tools can handle up to v7, SilK
# can handle v5, v9, and IPFIX.
#
# Quick Upgrade
#
# 1. Untar the package into a new cgi-bin subdirectory
# 2. Configure FlowViewer_Configuration.pm variables to your environment
# and create all necessary directories with proper permissions
# 3. Replace old logos with new buttons (will be done automatically)
# 4. Copy FlowViewer.css, FlowViewer.pdf to $reports_directory
# 5. Configure FlowViewer_Configuration.pm to point to existing
# FlowTracker_Filter and FlowTracker_RRDtool directories
# 6. Configure FlowViewer_Configuration.pm to point to existing
# FlowTracker html directory (or copy all existings into a new one)
# 7. Stop old Flowtracker_Collector and FlowTracker_Grapher
# 8. Start new Flowtracker_Collector and FlowTracker_Grapher
# 9. Copy NamedInterfaces_Devices, names file, user logo to new directory
# 10. Run convert_pre40_filters against existing filters (ie FlowViewer_SavedFilters)
# 11. Use included 'User Relay' scripts if desired (recommended - see below)
#
# Quick Install
#
# 1. Untar into cgi-bin subdirectory
#
# For netflow v5 and older (option):
#
# 2. Download, install, configure flow-tools
#
# For IPFIX (e.g., v9 - also handles v5):
#
# 3. Download, install, configure SiLK and libfixbuf
#
# For sflow (translate into v5):
#
# 4. Download, install, configure Inmon's sflowtool
#
# For FlowViewer
#
# 5. Configure FlowViewer_Configuration.pm variables as necessary
# 6. Create all necessary directories with proper permissions
# 7. Copy FlowViewer.css, FlowViewer.pdf to $reports_directory
# 8. Point browser to FV.cgi
#
# For FlowGrapher
#
# 9. Install gd (C), GD (Perl), GD::Graph (Perl)
# 10. Configure FlowViewer_Configuration.pm variables as necessary
# 11. Point browser to FV.cgi
#
# For FlowTracker
#
# 12. Install RRDtool (at least version 1.4)
# 13. Create FlowTracker_Filter and FlowTracker_RRDtool directories
# 14. Configure FlowViewer_Configuration.pm variables as necessary
# 15. Start FlowTracker_Collector, FlowTracker_Grapher in background
# 16. Point browser to FV.cgi
#
# For all FlowViewer tools
#
# 16. Review all FlowViewer directories and files for proper permissions
#
# Version 4.0 Release Notes
#
# Version 4.0 is a major upgrade that enables FlowViewer to handle IPFIX netflow
# data (i.e., v9, etc.) The User Interface has been completely redone and now
# features a Dashboard. Aside from the new collector interface and User interface,
# version 4.0 introduces some new capabilities:
#
# 1. FlowViewer report sorting by column header
# 2. Dashboard of thumbnail versions of selected FlowTracker graphs
# 3. Ability to 'recreate' FlowTrackings, starting at a time specified in the past
#
# The distribution manifest has changed significantly.
#
# Preserved Scripts, Files, and Tools:
#
# FlowViewer.cgi Modified for new user interface.
# FlowViewer_Main.cgi Modified for new user interface.
# FlowViewer_Relay.cgi No change.
# FlowViewer_Save.cgi Significant modification.
# FlowGrapher.cgi Modified for new user interface.
# FlowGrapher_Main.cgi Modified for new interface.
# FlowGrapher_Colors No change.
# FlowGrapher_Relay.cgi No change.
# FlowGrapher_Sort.cgi Significant modification.
# FlowTracker.cgi Modified for new user interface.
# FlowTracker_Collector Modified to process stored SiLK data.
# FlowTracker_Grapher Modified to update Thumbnails.
# FlowTracker_Group Modified for new user interface.
# FlowTracker_Dumper Modified for new user interface.
# FlowTracker_Relay.cgi No change.
# FlowViewer_CleanASCache No change.
# FlowViewer_CleanFiles Minor changes.
# FlowViewer_CleanHostCache No change.
# FlowViewer_Configuration.pm Modifications for SiLK and user interface.
# FlowViewer_Utilities.pm Removed filter output processing.
# NamedInterfaces_Devices No change.
# NamedInterfaces_Exporters No change.
# flowcapture_restart No change.
# flow-capture-table.conf No change.
# flowtracker_restart No change.
# performance_check Parses FlowTracker logs and reports performance
# rsync_flows Rsync all of raw flow data to backup host
# rsync_trackings Rsync all of Tracking data to backup host
#
# New Scripts, Files, and Tools
#
# FlowViewer_Replay.cgi Presents saved FlowViewer reports.
# FlowViewer_SaveManage.cgi Manages saved reports.
# FlowViewer_Sort.cgi Sorts FlowViewer reports.
# FlowViewer_UI.cgi Utilities for creating user interface.
# FlowGrapher_Replay.cgi Presents saved FlowGrapher reports.
# FlowTracker_Dashboard.cgi Manages the Dashboard contents.
# FlowTracker_Display.cgi Presents a FlowTracking.
# FlowTracker_DisplayPublic.cgi Presents a FlowTracking from Public list.
# FlowTracker_Management.cgi Manages FlowTrackings (e.g., remove, etc.)
# FlowTracker_Recreate Background process for recreating FlowTrackings.
# FlowTracker_Thumbnail Invoked to create a Thumbnail FlowTracking.
# FlowViewer.css FlowViewer cascading style sheet.
# FV_button.png New button link to FlowViewer from front page.
# FG_button.png New button link to FlowGrapher from front page.
# FT_button.png New button link to FlowTracker from front page.
# convert_pre40_filters Converts old saved filters (pre version 4.0).
# flowtracker_archive_restore Restores archived FlowTrackings if they go astray
# flowtracker_grapher_nonlazy Forces a re-graphing of all FlowTracker graphs
# resize_rrdtools Extends RRDtools created prior to 3-Year capability
#
# Removed Scripts and Files
#
# FlowViewer_SavedFilters File kept saved filters
#
# General Notes:
#
# This is a major upgrade of FlowViewer. The upgrade preserves this
# open-source option for netflow analysis in the age of IPFIX. The user
# is urged to read through the User's Guide for a better understanding
# of installation and configuration.
#
# Those who upgrade can preserve all previous filters and reports easily.
# Saved reports are automatically available in the new version. The only
# manual change requires users to run the 'convert_pre40_filters' script
# from the command line to move saved filters into the new format. Example:
#
# host>convert_pre40_filters /var/www/cgi-bin/FlowViewer_3.4/FlowViewer_SavedFilters
#
# With Respect to SiLK: The SiLK tool suite, developed by the NetSA group
# at Carnegie Mellon, is excellent software with equally excellent documentation.
# Version 3.0 of SiLK together with libfixbuf v1.1.0 are their entree into
# IPFIX/v9 netflow capture and analysis supporting IPv6. Initially they have
# chosen to limit the number of IPFIX Information Elements (IE) that the SiLK
# software will process. They have chosen a set that matches what flow-tools has
# provided with the addition of IPv6 data, but sadly with the exception of automomous
# system (AS) elements. I have requested that they add the AS Elements, but we'll see.
# They have mentioned a future overhaul (beyond v3.0) to handle the entire IE space
# through user configuration. As of Summer, 2012, SiLK v3.0 is not fully through
# the process required to make the software open-source to the general public but
# they are proceeding with getting the approval. It is currently freely available
# to US Federal agencies and Universities.
#
# The FlowViewer_Configuration.pm file has changed:
#
# New parameters (configurable):
#
# $dashboard_directory = "/var/www/html/FlowViewer_Dashboard";
# $dashboard_short = "/FlowViewer_Dashboard";
# $silk_data_directory = "/data/flows";
# $silk_bin_directory = "/usr/local/bin";
# $sensor_config_directory = "/etc";
# @ipfix_devices = ("Router_v9_1","Router_v9_2","Test_6509_v9");
# $sip_prefix_length = "16";
# $dip_prefix_length = "16";
# $silk_all_only = "N";
# $left_title = "Any Title You Like";
# $left_title_link = "http://abc.com/";
# $right_title = "Any Second Title You Like";
# $right_title_link = "http://abc.com/";
# $recreate_cat_length = 1*(60*60); # Time length of concatenated file
# $thumbnail_width = 250; # probably should leave this alone
# $thumbnail_height = 80; # probably should leave this alone
# $filename_color = "#CF7C29";
# $dig_forward = "/usr/bin/dig +time=1 +tries=1 ";
# $default_identifier = "DNS"; # Use "IP" for IP addresses; "DNS" to resolve to names
#
# Removed Parameters
#
# $bg_color = "#FFFFFF";
# $text_color = "#000000";
# $link_color = "#000000";
# $vlink_color = "#BF294D";
# $trackings_title = "Your Company Name";
# $user_logo = "Generic_Logo.jpg";
# $user_hyperlink = "http://www.yourcompany.com/";
#
# With respect to the "Relay" scripts, many of you may already have resolved this
# issue by setting up a generic 'FlowViewer' directory and simply re-linking it to
# the new version's directory. I've been told this is proper :-). It certainly makes
# good sense. Otherwise the "Relay" approach is best explained below in Version 3.4
# Release Notes.
#
|
About Version 3.4Version 3.4 Release Notes It's been awhile, so version 3.4 will fix a myriad of little problems which I mostly can't remember. The primary new capabilities include: 1. In most cases, the user may now switch the device without losing entered filter criteria 2. The different tool logos now provide a link to the Saved Reports page 3. Users can now provide a meaningful name for saved FlowViewer and FlowGrapher reports 4. Fixes to an end-of-year problem have resulted in a 8% speed up of FlowGrapher in general 5. Users can select to limit FlowGrapher stats to no-zero data points, if desired 6. Fixed problems with sorting 7. Corrected the graphing by 'flows' (was graphing 'flags' :-) 8. Can now provide up to 20 source or destination IP address/address ranges 9. Can now exclude specified IP addresses from a larger included address range Modifications have been made to FlowGrapher_Main.cgi since the original 3.4 distribution to fix a problem caused by the new speed-up processing. The speed-up was not accounting for Daylight Savings considerations. New Scripts and Files: FlowGrapherM.png New logo link points to Saved reports web page FlowGrapherS.png Revised logo link permits naming of Saved Reports FlowViewerM.png New logo link points to Saved reports web page FlowViewerS.png Revised logo link permits naming of Saved Reports FlowTrackerM.png New logo link points to Saved reports web page flowcapture_restart Renamed flowcap script for restarting flow-captures flowtracker_restart New script for re-starting FlowTracker_Collector General Notes: Remember to copy into the new directory (e.g., /usr/lib/cgi-bin/FlowViewer_3.4) user logos, names file, as_names, NamedInterfaces_Devices, NamedInterface_Exporters, FlowViewer_SavedFilters, etc., from the old cgi-bin directory. The simplest way to transition to the new version is to leave all FlowViewer_Configuration.pm settings alone except: $reports_directory = "/var/www/FlowViewer_3.4"; $reports_short = "/FlowViewer_3.4"; $graphs_directory = "/var/www/FlowGrapher_3.4"; $graphs_short = "/FlowGrapher_3.4"; $tracker_directory = "/var/www/FlowTracker_3.4"; $tracker_short = "/FlowTracker_3.4"; $old_tracker_directory = "/var/www/FlowTracker_3.3.1"; $cgi_bin_directory = "/usr/lib/cgi-bin/FlowViewer_3.4"; $cgi_bin_short = "/cgi-bin/FlowViewer_3.4"; $work_directory = "/usr/lib/cgi-bin/FlowViewer_3.4/Flow_Working"; $work_short = "/cgi-bin/FlowViewer_3.4/Flow_Working"; $names_directory = "/usr/lib/cgi-bin/FlowViewer_3.4"; $log_directory = "/usr/lib/cgi-bin/FlowViewer_3.4" The following can remain the same (or else copy the contents to the new directory): $save_directory = "/var/www/FlowViewer_Saves"; $save_short = "/FlowViewer_Saves"; $filter_directory = "/usr/lib/cgi-bin/FlowTracker_Files/FlowTracker_Filters"; $rrdtool_directory = "/usr/lib/cgi-bin/FlowTracker_Files/FlowTracker_RRDtool"; If this is an upgrade for you (e.g., from v3.3.1) I recommend using the FlowViewer_Relay.cgi, FlowGrapher_Relay.cgi, and the FlowTracker_Relay.cgi scripts to alert users to the new version with links and a reminder to change their bookmarks. In each of the Relay scripts tailor the following line to your environment (i.e., point to the new FlowViewer_Configuration.pm file): require "/usr/lib/cgi-bin/FlowViewer_3.4/FlowViewer_Configuration.pm"; ... then, copy the following new Relay files overtop of the related files in your old cgi-bin directory (e.g., FlowViewer_3.3.1): cp .../FlowViewer_3.4/FlowViewer_Relay.cgi .../FlowViewer_3.3.1/FlowViewer.cgi cp .../FlowViewer_3.4/FlowGrapher_Relay.cgi .../FlowViewer_3.3.1/FlowGrapher.cgi cp .../FlowViewer_3.4/FlowTracker_Relay.cgi .../FlowViewer_3.3.1/FlowTracker.cgi Now, when users go to their book-marked FlowViewer web pages, they will be directed to the new ones. FlowTracker_Relay.cgi is particularly important if this is an upgrade; it copies over archived FlowTrackings which would be a bit tedious to copy by hand. The rsync_flows and rsync_trackings scripts are useful for easily backing up all raw netflow data and FlowTracker state information (Filters and RRDtool databases.) The FlowViewer_CleanFiles script is useful for deleting aging files that are not necessary anymore. I run it out of 'cron' once a day. The performance_check script can be used from the command line to keep track of how well your implementation is performing. I run it against my Flowtracker_Collector.log file to see how things are going. Here at the NASA Earth Observing System network I have over 200 FlowTrackings and they complete in an average of 44 seconds. FlowTracker_Collector runs every five minutes and I watch for runs that take longer than five minutes. Even in those situations, however, FlowTracker_Collector seems to continue on with no real visible effects.About Version 3.3 Version 3.3.1 fixes some problems related to using the new Exporter capability. A FlowGrapher problem with sorting host names has been fixed. A problem with FlowGrapher graphing across the midnight boundary is fixed. The FlowViewer_Relay.cgi optional scripts have been made easier to set up. A problem with FlowTracker and the use of $no_devices_or_exporters has been fixed. FlowViewer_Cleanup scripts have been cleaned up. A number of fixes have been added: exporter problems, FlowTracker at month boundaries, very long FlowTracker_Collector and ...Grapher runs freezing on Debian platforms (Debian Perl handles 'sleep' differently than Red Hat.) Version 3.3 introduces a number of new useful features including:
Version 3.2 introduces FlowTracker groups. These groups merge copies of
multiple previously defined individual trackings onto one graph. Improvements
have been made to the speed of FlowTracker_Collector. All FlowViewer, FlowGrapher,
and FlowTracker logos now have embedded links for quicker navigation between the
tools. A new Autonomous System (AS) resolving capability has been added. A couple
of fixes since the original v3.2 release: Version 3.1 includes statistcal information (MAX, MIN, 95th PCT, AVG) with FlowGraphs, introduces the ability to archive and restart trackings, now permits queries longer than 30 days, permits a range of port numbers (e.g. ports 1024:1048) on queries, and fixed a number of small issues. Version 3.0 introduces FlowTracker, a tool that permits users to track flow_data subsets, defined by specified filter criteria, over long time periods This version also modifies the interaction of the scripts so that the create_FlowViewer _webpage and create_FlowGrapher_webpage scripts are eliminated. Users will now point their browsers to FlowViewer.cgi, FlowGgrapher.cgi, and FlowTracker.cgi scripts instead. These scripts will pre-load the data input pages with the current time. About Version 2 Version 2.3 is a major rework of FlowGrapher to increase it's processing speed. A 10-fold increase in speed was accomplished by eliminating calls to Time::Local:timelocal for each flow record processed. Computations of flow_length and determining of appropriate buckets are now in-line as much as possible. Version 2.2 introduces the flow_select option on the input page. This new parameter allows the user to specify which flows should be included in the specified time period. It may be the case that large, long flows may not end prior to the specified end time. These flows would not be counted previously as a flow was included only if it's end-time was within the time period. The flow_select option now permits the user to specify the type of flows he would like to see (e.g., "Any part of flow is within time period", "Start-time of flow within time period", etc.) This is more important over shorter time periods. The default value is "Any part of flow is within time period". Version 2.1 fixes a concatenation processing time "improvement" which turned out to be necessary after all, and end-of-year processing for FlowGrapher. Version 2 of FlowViewer introduces FlowGrapher, has halved the prosessing time, and has introduced several new features. The changes include:
|