# README (this file) FlowViewer V3.4  Date: 03/17/2011  Updated: 08/17/2011
#
# FlowViewwer is a set of three tools (FlowViewer, FlowGrapher,
# FlowTracker) that create text reports, graph reports, and 
# long-term tracking reports from flow-tools captured and
# stored netflow data.
#
# Quick Upgrade
#
#   1. Untar the package into a new cgi-bin subdirectory
#   2. Configure FlowViewer_Configuration.pm variables as necessary
#   3. Replace old logos with new logos (will be done automatically)
#   4. Configure FlowViewer_Configuration.pm to point to existing 
#      FlowTracker_Filter and FlowTracker_RRDtool directories
#   5. Stop old Flowtracker_Collector and FlowTracker_Grapher
#   6. Start new Flowtracker_Collector and FlowTracker_Grapher
#   7. Copy NamedInterfaces_Devices, names file, user logo to new directory
#   8. Use included 'User Relay' scripts if desired (recommended - see below)
#
# Quick Install
#
#   1. Untar into cgi-bin subdirectory
#
#   For FlowViewer
#
#   2. Configure FlowViewer_Configuration.pm variables as necessary
#   3. Point browser to FlowViewer.cgi
#
#   For FlowGrapher
#
#   4. Install gd (C), GD (Perl), GD::Graph (Perl)
#   5. Configure FlowViewer_Configuration.pm variables as necessary
#   6. Point browser to FlowGrapher.cgi
#
#   For FlowTracker
#
#   7. Install RRDtool (at least version 1.2.12)
#   8. Create FlowTracker_Filter and FlowTracker_RRDtool directories
#   9. Start FlowTracker_Collector, FlowTracker_Grapher in background
#   10. Configure FlowViewer_Configuration.pm variables as necessary
#   11. Point browser to FlowTracker.cgi
#
#   For all FlowViewer tools
#  
#   12. Copy files (e.g., names, nameInterface_Devices, etc.) to new directory
#   13. Review all FlowViewer directories and files for proper permissions
#
# Version 3.4 Release Notes
#
#  Update - 8/17/2011 - Fixed FlowViewer bug when requesting time periods just shy
#  of midnight. This had already been fixed in FlowGrapher. Modifications were
#  made to FlowViewer_Main.cgi.
#
#  Update - 5/20/2011 - Modifications have been made to FlowGrapher_Main.cgi to fix a
#  problem caused by the new speed-up processing. The speed-up was not accounting for
#  Daylight Savings considerations.
#
#  It's been awhile, so version 3.4 will fix a myriad of little problems which
#  I mostly can't remember. The primary new capabilities include:
#
#   1. In most cases, the user may now switch the device without losing entered filter criteria
#   2. The different tool logos now provide a link to the Saved Reports page
#   3. Users can now provide a meaningful name for saved FlowViewer and FlowGrapher reports
#   4. Fixes to an end-of-year problem have resulted in a 8% speed up of FlowGrapher in general
#   5. Users can select to limit FlowGrapher stats to no-zero data points, if desired
#   6. Fixed problems with sorting
#   7. Corrected the graphing by 'flows' (was graphing 'flags' :-)
#   8. Can now provide up to 20 source or destination IP address/address ranges
#   9. Can now exclude specified IP addresses from a larger included address range
#
#  New Scripts and Files:
#
#   FlowGrapherM.png            New logo link points to Saved reports web page
#   FlowGrapherS.png            Revised logo link permits naming of Saved Reports
#   FlowViewerM.png             New logo link points to Saved reports web page
#   FlowViewerS.png             Revised logo link permits naming of Saved Reports
#   FlowTrackerM.png            New logo link points to Saved reports web page
#   flowcapture_restart         Renamed flowcap script for restarting flow-captures
#   flowtracker_restart         New script for re-starting FlowTracker_Collector
#
#  General Notes:
#
#   Remember to copy into the new directory (e.g., /usr/lib/cgi-bin/FlowViewer_3.4) user
#   logos, names file, as_names, NamedInterfaces_Devices, NamedInterface_Exporters,
#   FlowViewer_SavedFilters, etc., from the old cgi-bin directory.
#
#   The simplest way to transition to the new version is to leave all 
#   FlowViewer_Configuration.pm settings alone except:
#
#    $reports_directory       = "/var/www/FlowViewer_3.4"; 
#    $reports_short           = "/FlowViewer_3.4"; 
#    $graphs_directory        = "/var/www/FlowGrapher_3.4"; 
#    $graphs_short            = "/FlowGrapher_3.4"; 
#    $tracker_directory       = "/var/www/FlowTracker_3.4"; 
#    $tracker_short           = "/FlowTracker_3.4"; 
#    $old_tracker_directory   = "/var/www/FlowTracker_3.3.1"; 
#    $cgi_bin_directory       = "/usr/lib/cgi-bin/FlowViewer_3.4"; 
#    $cgi_bin_short           = "/cgi-bin/FlowViewer_3.4"; 
#    $work_directory          = "/usr/lib/cgi-bin/FlowViewer_3.4/Flow_Working"; 
#    $work_short              = "/cgi-bin/FlowViewer_3.4/Flow_Working";
#    $names_directory         = "/usr/lib/cgi-bin/FlowViewer_3.4"; 
#    $log_directory           = "/usr/lib/cgi-bin/FlowViewer_3.4"
#
#   The following can remain the same (or else copy the contents to the new directory):
#
#    $save_directory          = "/var/www/FlowViewer_Saves"; 
#    $save_short              = "/FlowViewer_Saves"; 
#    $filter_directory        = "/usr/lib/cgi-bin/FlowTracker_Files/FlowTracker_Filters";   
#    $rrdtool_directory       = "/usr/lib/cgi-bin/FlowTracker_Files/FlowTracker_RRDtool";
#
#   If this is an upgrade for you (e.g., from v3.3.1) I recommend using the FlowViewer_Relay.cgi,
#   FlowGrapher_Relay.cgi, and the FlowTracker_Relay.cgi scripts to alert users to the new
#   version with links and a reminder to change their bookmarks.  In each of the Relay scripts
#   tailor the following line to your environment (point to the new FlowViewer_Configuration.pm
#   file):
#
#    require "/usr/lib/cgi-bin/FlowViewer_3.4/FlowViewer_Configuration.pm";
#
#   ... then, in your old cgi-bin directory (e.g., FlowViewer_3.3.1), copy the following:
#
#    cp FlowViewer_Relay.cgi FlowViewer.cgi
#    cp FlowGrapher_Relay.cgi FlowGrapher.cgi
#    cp FlowTracker_Relay.cgi FlowTracker.cgi
#   
#   Now, when users go to their book-marked FlowViewer web page, they will be directed
#   to the new one. FlowTracker_Relay.cgi is particularly important if this is an upgrade;
#   it copies over archived FlowTrackings which would be a bit tedious to copy by hand.
#
#   The rsync_flows and rsync_trackings scripts are useful for easily backing up all
#   raw netflow data and FlowTracker state information (Filters and RRDtool
#   databases.) The FlowViewer_CleanFiles script is useful for deleting aging files
#   that are not necessary anymore. I run it out of 'cron' once a day.
#
#   The performance_check script can be used from the command line to keep track 
#   of how well your implementation is performing. I run it against my
#   Flowtracker_Collector.log file to see how things are going. Here at the
#   NASA Earth Observing System network I have over 200 FlowTrackings and
#   they complete in an average of 44 seconds. FlowTracker_Collector runs every
#   five minutes and I watch for runs that take longer than five minutes.
#   Even in those situations, however, FlowTracker_Collector seems to continue
#   on with no real visible effects.
#
# Version 3.3 Release Notes
#
# ### Version 3.3.1 fixes a FlowTracker_Collector bug when using exporters
#     Also fixes problem for users without devices at all. If you are not
#     using any devices (or exporters) you will now have to set:
#     $no_devices_or_exporters = "Y"; Fixes FlowGrapher sorting of host
#     names. This version fixes the problem of links to Trackings embedded
#     in Group graphs not lining up properly. Fixes problem with 
#     FlowTracker_Grapher not printing out named interfaces. Fixes 
#     FlowGrapher graph and output to now have exporter name. Fixes 
#     problem with end-of-month graphs (missing days_in_month.)
#
# New Capabilities
#
#   1. Some devices will now have 'named interfaces' (thanks C. Kishimoto)
#   2. The user can now save filters of interest and recall them later
#   3. Data can now be analyzed by Exporter ID (in addition to device name)
#   4. Users can now set thresholds on FlowTrackings, and be alerted
#   5. Users can now sort FlowGrapher output based on column type
#   6. FlowViewer now provides Pie Charts
#   7. Capability added to apply a Sampling Multiplier to output
#   8. FlowTrackings now have a '3 year' graph
#   9. The user can now generate text listings of FlowTracker output
#  10. Filtering on next-hop has been added
#  11. Logging has been made more flexible (e.g., less data)
#  12. Preserve latest three notations (was keeping first three)
#  13. Can now specify and display time-zones
#  14. A hook has been provided for a User Logo with link out of FlowViewer
#  15. New file cleanup scripts have been added
#  16. Unit Conversion capability has been added (thanks C. Kishimoto)
#  17. Can now graph Flows, Packets as well as Octets (thanks E. Lautenschlaeger)
#  18. Improved AS name resolution (thanks S. Cardus)
#  19. New saved_directory for storing saved Reports and Graphs.
#
# New Scripts and Files:
#
#  FlowGrapher_Sort.cgi        Sorts FlowGrapher Detail Lines by column
#  FlowTracker_Dumper.cgi      Invoked by link in Trackings, prints text values
#  FlowViewer_Save.png         New logo with links for saving filters, reports
#  Flowgrapher_Save.png        New logo with links for saving filters, reports
#  FlowViewer_CleanASCache     Tool used to remove obsolete AS name resolutions
#  FlowViewer_CleanFiles       Tool used to remove old intermediate files
#  FlowViewer_CleanHostCache   Tool used to remove obsolete host name resolutions
#  FlowViewer_Relay.cgi        Optional for pointing users to new version (see Notes)
#  FlowGrapher_Relay.cgi       Optional for pointing users to new version (see Notes)
#  FlowTracker_Relay.cgi       Optional for pointing users to new version (see Notes)
#  flowcap                     Optional start-up script for flow-tools and FlowTracker
#
#  NamedInterfaces_Devices     Holds interface names for SNMP indexes for devices
#  NamedInterfaces_Exporters   Holds interface names for SNMP indexes for exporters
#  FlowViewer_SavedFilters     Created during processing to hold saved filters
#
# Notes:
#
#  Many thanks to Carles Kishimoto, Eric Lautenschlaeger, and Sean Cardus for 
#  their ideas and code contributions. Thanks to Dario La Guardia for pointing out
#  a graphing problem that turned out to be a rounding error in FlowGrapher. Credit
#  to Peter Hoffswell for the idea of linking the tools.
#
#  There are no new software dependencies with FlowViewer version 3.3.1, however
#  Named Interfaces now requires Javascript in the browser to operate.
#
#  If you are having trouble with creating Tracking Groups, you may have a problem
#  with the installation of RRDs.pm. This needs to be placed in a library that Perl
#  includes in it's @INC array. For a fix, see the FlowViewer FAQ on the web site.
#
#  Using the 'Relay' scripts (these are optional)
#
#   If you have other users and you would like to point them to the new version,
#   copy the included 'Relay' scripts over the old FlowViewer.cgi, FlowGrapher.cgi,
#   and FlowTracker.cgi scripts in the last version's directory.
#
#   For example:
#
#    In the old directory /htp/cgi-bin/FlowViewer_3.2:
#
#    mv FlowViewer_Relay.cgi  FlowViewer.cgi
#    mv FlowGrapher_Relay.cgi FlowGrapher.cgi
#    mv FlowTracker_Relay.cgi FlowTracker.cgi
#   
#   Then, when the user goes to the old FlowViewer, he will be provided a link to
#   the new FlowViewer, and asked to change his bookmarks.
#
#
#  Setting up crontab file for cleaning FlowViewer files:
#      
#   min   hr  dom  moy  dow    command
#
#     5   0    *    *    *     /usr/lib/cgi-bin/FlowViewer_3.4/FlowViewer_CleanFiles
#                                > /usr/lib/cgi-bin/FlowViewer_3.4/cleanup.log 
#                                2 >> /usr/lib/cgi-bin/FlowViewer_3.4/cleanup.log
#
#  The file cleanup is controlled by parameters in FlowViewer_Configuration.pm:
#
#   $remove_workfiles_time   = 86400;     
#   $remove_graphfiles_time  = 7*86400; 
#   $remove_reportfiles_time = 7*86400;
#
#  Remember, whichever crontab account this is started from must have adequate 
#  permissions to remove files created by the web process owner (e.g., apache.)
#
# Documentation
#
# The FlowViewer User's Guide is available on the FlowViewer Website:
#
#   http://ensight.eos.nasa.gov/FlowViewer
#
# Dependencies
#
# - FlowGrapher requires the Perl GD and GD:Graph packages
#   gd package. Thomas Boutrell's graphics package written in 'C'
#   GD package: http://search.cpan.org/~lds/GD-2.30/
#   GD::Graph package: http://search.cpan.org/~mverb/GDGraph-1.43/
# - FlowViewer.cgi requires the GDBM or NDBM capability in Perl
# - FlowTracker requires RRDtool (at least version 1.2.12)
#   RRDtool: http://oss.oetiker.ch/rrdtool
#
# Contents
#
# FlowViewer_Configuration.pm
#
# This file contains parameters that configure and control the 
# FlowViewer, FlowGrapher, and FlowTracker environments. This package 
# should remain in the same directory that the CGI scripts are in.
#
# FlowViewer_Utilities.pm
#
# This file contains processing used by multiple programs (e.g., to
# create the Report Parameters output for each tool, and other utilities
# (e.g., 'epoch_to_date' which converts between typical date formats 
# and 'seconds since 1972') that are invoked by other scripts. This 
# package should be placed in the same directory as the CGI scripts.
#
# FlowViewer.cgi
#
# This script produces the web page which provides the user the form
# for entering analysis selection criteria for FlowViewer. Version 3.0
# reorganized the processing. FlowViewer.cgi is now the old
# create_FlowViewer_webpage. This change permits the input date and time
# to be updated with each invocation.
#
# FlowViewer_Main.cgi
#
# This script responds when the user completes the selection criteria
# form and submits the 'Generate Report' command. The script creates a
# flow-tools filter file based on the selection criteria. Based on the
# input time period, the script concatenates the relevant flow-tools
# data files for the selected device. The location of the flow-tools
# raw data files is specified via the 'flow_data_directory' parameter.
# The script then invokes the selected statistics/print report flow-tools
# program and reformats the output into HTML. An option is available in
# FlowViewer_Configuration to have this script use the NDBM capability
# (for caching resolved host names) instead of the default GDBM
# capability for users whose Perl distribution does not have GDBM.
#
# FlowGrapher.cgi
#
# This script produces the web page which provides the user the form
# for entering analysis selection criteria for FlowGrapher. Version 3.0
# reorganized the processing. FlowGrapher.cgi is now the old
# create_FlowGrapher. This change permits the input date and time
# to be updated with each invocation.
#
# FlowGrapher_Main.cgi
#
# This script responds when the user completes the FlowGrapher selection
# criteria form and submits the 'Generate Graph' command. The script
# creates intermediate processing files exactly like FlowViewer above.
# The script then parses intermediate output, fills time buckets, and
# generates a graphic image. Textual output accompanies the graph. An
# option is available in FlowViewer_Configuration to have this script use
# the NDBM capability (for caching resolved host names) instead of the
# default GDBM capability for users whose Perl distribution does not have
# GDBM.
#
# FlowGrapher_Sort.cgi
#
# This script is invoked when the user clicks on a column header for the
# Detail Lines of a FlowGrapher report. The textual data on the page is
# sorted and re-presented.
#
# FlowGrapher_Colors
#
# This file contains a translation between textual color names and their
# RGB value counterparts.
#
# FlowTracker.cgi
#
# This script produces the web page which provides the user the form
# for entering analysis selection criteria for FlowTracker. The script
# also provides the user with the ability to review, revise, or remove
# existing trackings. FlowTracker was new in version 3.0. 
#
# FlowTracker_Main.cgi
#
# This script responds when the user completes the FlowTracker selection
# criteria form and submits the 'Establish Tracking' command. The script
# responds to the users desire to create, remove, or revise a tracking.
#
# FlowTracker_Group.cgi
#
# This script controls the building of groups from existing Individual
# FlowTrackings. The user has the ability with FlowTracker v3.2 to create
# 'groups' from pre-defined Individual trackings. A Group Tracking has no
# RRD database associated with it, but simply creates a multifaceted graph
# from several existing trackings. The Group 'merges' the Individual graphs
# onto a single graph.
#
# FlowTracker_Dumper.cgi
#
# This script is invoked when the user clicks on a link within the
# FlowTracking graph labeled '[List values]'. The script dumps the 
# RRDtool contents onto a web page.
#
# FlowTracker_Collector
#
# The script is started once by the user and placed in the 'background'.
# The script will execute and then sleep for the duration of a five minute
# period, essentially running every five minutes. For each existing tracking, 
# the script applies the associated filter to the flow data and extracts the
# amount that occured during a 5-minute window approximately 30 miuntes
# earlier. This is to permit long-running flows to have been exported and 
# available to the collector. The script then divides the total bits by 
# 300 seconds to get an average bits-per-second rate during the period.
# The data point is then provided to RRDtool for storage. The script
# should be started out of the cgi-bin directory.
#
# FlowTracker_Grapher
# 
# The script is started once by the user and placed in the 'background'.
# The script will execute and then sleep for the duration of a five minute
# period, essentially running every five minutes. The script runs the
# RRDtool graph function for each existing tracking. Daily, weekly,
# monthly, and yearly graphs are updated with the latest information. The
# script creates an html page for each tracking that includes the filter
# parameters and the four graphs. The script also creates an overall web
# page ($tracker_webpage) that provides links to all active tracking pages.
# The script should be started out of the cgi-bin directory.
#
# FlowViewerM.png
#
# The FlowViewerM logo with links. Leave this file in the 'cgi-bin_directory', 
# the FlowViewerM.cgi script will place a copy of the image in 
# 'html_directory'. This image contains mapped links to FlowGrapher and
# FlowTracker such that those input pages are pre-loaded with the filter
# criteria from FlowViewer.
#
# FlowViewerS.png
#
# The FlowViewerS logo with links. Leave this file in the 
# 'cgi-bin_directory', the FlowViewerS.cgi script will place a copy of the 
# image in 'reports_directory'. This image contains mapped links to the other
# tools as well as links for saving the filter used or the report generated.
#
# FlowGrapherM.png
#
# The FlowGrapherM logo with links. Leave this file in the 'cgi-bin_directory', 
# the FlowGrapherM.cgi script will place a copy of the image in 
# 'graphs_directory'. This image contains mapped links to FlowViewer and
# FlowTracker such that those input pages are pre-loaded with the filter
# criteria from FlowGrapher.
#
# FlowGrapherS.png
#
# The FlowGrapherS logo with links. Leave this file in the 
# 'cgi-bin_directory', the FlowGrapherS.cgi script will place a copy of the 
# image in 'graphs_directory'. This image contains mapped links to the other
# tools as well as links for saving the filter used or the report generated.
#
# FlowTrackerM.png
#
# The FlowTracker logo with links. Leave this file in the 'cgi-bin_directory', 
# the FlowTrackerM.cgi script will place a copy of the image in 
# 'tracker_directory'. This image contains mapped links to FlowViewer and
# FlowGrapher such that those input pages are pre-loaded with the filter
# criteria from FlowTracker.
#
# FlowViewer_Save.cgi
#
# This script moves temporary save files into a permanent residence
# as defined by either the 'reports_directory' or 'graphs_directory'
# environment variables.
#
# FlowViewer_CleanFiles
# 
#  A utility for cleaning out temporary files that have been left
#  over from debugging (e.g. $debug_files = 'Y'). Files older than
#  the following configurable parameters are removed:
#  
#   $remove_workfiles_time   = 86400; 
#   $remove_graphfiles_time  = 7*86400; 
#   $remove_reportfiles_time = 7*86400
#
#  See above for crontab settings for running this automatically.
#
# FlowViewer_CleanASCache
# 
# A utility for cleaning out from the AS resolving cache ($as_file) a 
# resolved AS name that is no longer valid.
#
# FlowViewer_CleanHostCache
# 
# A utility for cleaning out from the DNS resolving cache ($names_file) 
# a resolved host name that is no longer valid.
#
# FlowViewer_Relay.cgi, FlowGrapher_Relay.cgi, FlowTracker_Relay.cgi
#
# Short scripts that refer users from version 3.3.1 to version 3.4. This
# keeps you from having to notify users to go to a different web site.
#
# flowcapture_restart
#
# A shell script used for starting up and restarting flow-captures. Tailor 
# this for your environment.
#
# flowtracker_restart
#
# A shell script used for starting up and restarting FlowTracking_Collector
# and FlowTracker_Grapher. Tailor this for your environment.
#
# Generic_Logo.jpg
#
# This image is to be replaced by your own image that can point back to
# anywhere (e.g., your overarching NMS system.)
#
# NamedInterfaces_Devices
# 
# This file is used for SNMP index to named interface translation. This file 
# provides translation when you are saving data by individual devices. Examples
# are provided.
#
# NamedInterfaces_Exporters
# 
# This file is used for SNMP index to named interface translation. This file 
# provides translation when you are saving data into a single directory but
# for (possibly) multiple devices differentiated by EXPORTER_ID. Examples are
# provided.
#
# Configuration parameters
#
# The FlowViewer, FlowGrapher, and FlowTracker scripts all use parameters
# in the FlowViewer_Configuration.pm file to control the environment that
# they run in. Here is a brief explanation of some of the relevant 
# parameters:
#
# $ENV(PATH) - modify as appropriate for your installation
# $FlowViewer_server - IP address of server hosting this software
# $FlowViewer_service - Either HTTP (port 80) or HTTPS (port 443)
# $reports_directory - Directory to hold saved FlowViewer reports
# $reports_short - Reports directory beginning from web server default
# $graphs_directory - Directory to hold saved FlowGrapher reports
# $graphs_short - Graphs directory beginning from web server default
# $tracker_directory - Directory to hold FlowTracker trackings
# $tracker_short - Tracker directory beginning from web server default
# $filter_directory - Directory in which to keep FlowTracker filter files
# $rrdtool_directory - Directory in which to keep FlowTracker RRDtool files
# $cgi_bin_directory - Directory which holds cgi scripts
# $cgi_bin_short - cgi-bin directory beginning from web server default
# $flow_data_directory - Directory that holds all flow-tools data files
# $exporter_directory - Directory where netflow stored for multiple exporters
# $flow_bin_directory - Directory where all flow-tools reside
# $rrdtool_bin_directory - Location of RRDtool programs
# $work_directory - Directory to store intermediate files 
# $names_directory - Directory to save permanent 'names' file
# $flow_capture_interval - Interval beyond end point to capture all flows
# $flow_file_length - Length (in seconds) of each of your flow files 
# $devices - List of device names exporting netflow (see #4 below)
# $no_devices_or_exporters - Set this to "Y" if you have no devices and no exporters
# $N - Used to control directory organization (see #5 below)
# $dig - Location of DNS utility 'dig' (set to nslookup if required)
# $actives_webpage - Name of HTML file which will list your Trackings
# $trackings_title - Title for HTML page which lists Trackings
# $user_logo - Filename of image used for your logo
# $user_hyperlink - Link associated with $user_logo
# $use_even_hours - Will start default time periods at the top of the hour
# $use_NDBM - Some Perls don't have GDBM (default), but do have NDBM
# $start_offset - Offset from current time for beginning pre-loaded time period
# $end_offset - Offset from current time for end of pre-loaded time period
# $flow_capture_interval - Minutes beyond end period for collecting all flows
# $flow_file_length - Size in minutes of each flow-tools flow file (default = 15)
# $labels_in_titles - Whether to print FlowTracker title in the graph itself
# $debug_files - If Yes, will not remove intermediate files
# $collection_offset - Seconds into past to begin collection period
# $collection_period - Period to examine for FlowTracker (keep at 5 minutes!)
# $use_existing_concats - If Yes, will re-use concatenations (much faster)
# $rrd_dir_perms - (And others) UNIX directory or file permissions
#
# The rest of this file contains basic parameters such as colors, etc. Each
# parameter is dicussed in more detail in the User's Guide.
#
# Additional Considerations
#
# 1. Directory permissions for the subdirectories created for the
# 'htdocs', 'work', 'names', 'cgi-bin' (e.g., FlowTracker_Filter, 
# FlowTracker_RRDtool) directories must permit the owner of the web
# server process (e.g., apache) to write into these directories.
# The directories may have been created by a different user. Version 3.0
# introduced the use of $dir_perms. There are several of these included
# in FlowViewer_Configuration.pm. These are the permissions that the
# scripts will set your various FlowViewer files and directories to.
# They default to '0777' which permits the open interaction between the
# web server process owner and the FlowTracker background process owner.
# You may want to adjust these permissions differently according
# to your security policies, and whether you use the same or different
# accounts for the web and background processes.
#
# 2. FlowViewer and FlowGrapher offer the ability to save interesting
# reports. To do this, the scripts save a temporary copy of every report
# in advance of the user electing to save it permanently. These 
# intermediate files will accumulate in the 'work' directory specified
# in the FlowViewer_Configuration file. These files could be removed 
# daily via a cron script to prevent unecessary use of disk space. When
# the user elects to save a report, it is copied into either the 
# 'reports_directory', or the 'graphs_directory' depending on which 
# function he is running. See discussion of FlowViewer_CleanFiles above.
#
# 3. FlowViewer and FlowGrapher offer the ability to resolve NetFlow IP
# addresses into their host names on the fly. This process is speeded
# up by caching names into a 'names' file which resides in the directory
# specified by the 'names_directory' parameter. This parameter defaults
# to /tmp, but this may not be the best directory for you since it will
# disappear with a reboot. As you are building up your 'names' file
# with early runs, you will notice the speed increase dramatically
# as the 'names' file is used more. The process of resolving names is
# the primary reason for slower overall FlowViewer performance. You
# should preferably use the GDBM array database which is fastest.
# However, not all Perl distributions support GDBM but most do support
# NDBM. The '$use_NDBM' flag in FlowViewer_Configuration.pm will 
# cause the FlowViewer_Main and FlowGrapher_Main scripts to use NDBM.
#
# 4. The FlowViewer and FlowGrapher reporting features use a flow-tools
# data directory layout that has a particular device at the top. A
# typical flow-tools directory looks like:
#
# /flows/router_1/2005/2005-07/2005-07-04
#
# The device name (router_1) is obtained from an array called 'devices'
# in the FlowViewer_Configuration.pm file. Populate this array with your
# device names. If your flow-data file structure does not include a
# device name, for example you are collecting only from one device, set
# the @devices array to empty (i.e., @devices = ("");) On the web page
# you can ignore the Devices pulldown selection.
#
# As of version 3.3, users may now apply FlowViewer to directories that 
# collect from multiple sources, differentiated by EXPORTER_ID. If you
# are using this method (i.e., all flow-captures going into a single
# directory), simply set the $exporter_directory parameter to the
# directory that is set up to store the flow-data files.
#
# 5. Different organizations store captured netflow data differently
# according to the 'N" setting on the flow-capture statement. However,
# there is a bug in the flow-tools documentation such that the default
# value is truly '3' and not '0' as indicated. I have set $N = 3 to
# reflect the more common setting. The directory structure associated 
# with $N = 3 is shown below:
#
# /flows/router_1/2005/2005-07/2005-07-04
#
# If you are not seeing output, please check this setting.
#
# 6. Version 3.2 introduces Groups. Intermediate RRDtool databases are 
# created on the fly in order to create a temporary FlowTracker graph that
# shows the user how the final graph will look. It uses the Perl RRDs.pm
# RRDtool module to speed this up. Make sure your RRDtool distribution
# has a compatible RRDs.pm module.
#
# Change Log
# 
# Version 3.4 - March 17, 2011
# 
# See Version 3.4 Release Notes above
#
# Version 3.3
# 
# See Version 3.3 Release Notes above
#
# Version 3.2
# 
# Version 3.2 introduces Group trackings which are simply a tracking graph
# made up from the merging of several predefined Individual trackings onto
# one graph. There are no permanent RRDtool databases associated with a Group.
# In the construction of a group however, temporary RRDtool databases are 
# created to simulate how the Group will eventually look. The new script 
# Flowtracker_Group uses RRDs.pm (comes with RRDtool) to generate these 
# transient databases quickly. Please ensure that your RRDs.pm module is 
# compatible with your RRDtool distribution (this should normally be the 
# case - but if you see "ERR: can't handle RRD file version 0003" in 
# DEBUG_GRAPHER, you'll need to upgrade your RRDs.pm.
#
# This version also includes a 'speed-up' for FlowTracker_Collector which
# now concatenates once for each device. This is controlled by 
# $use_existing_concats, which defaults to "Y".
#
# Version 3.2 will continue to work happily along with earlier version
# exiting Filter and RRDtool files. There are no new FlowViewer_Configuration
# parameters of consequence. There are now two types of trackings; Individual
# and Group. The FlowTracker input screen will default to Individual which is
# the same as the existing trackings.
#
# Woj Kozicki has contributed an Autonomous System (AS) resolving capability 
# and it is included in v 3.2.
#
# New FlowViewer, FlowGrapher, and FlowTracker logos have been developed for 
# version 3.2. These new logos provide embedded links to the other tools so that
# the user can switch between them easily and retain input parameters.
#
# Version 3.1
# 
# 1. Added MIN, MAX, AVG, 95th PCT to FlowGrapher
# 2. Added ability to 'archive' trackings
# 3. Added ability to enter port ranges separated by a colon (:)
# 4. Can now use any mask length for networks (1 - 32)
# 5. Added RRDgraph 'lazy-mode' option to speed up graphing
# 6. Upgraded FlowViewer/Grapher ability to go back more than 30 days
# 7. Added the ability to configure file permissions
# 8. Improved speed of FlowGrapher for larger values of 'detail lines'
# 9. Fixed $rrdtool_bin-directory variable name
# 10. Added ability to retain intermediate files for debugging
# 11. Sorted list of Active Trackings
# 12. Fixed bug where non-zero 'cutoof lines' would supress some reports
# 13. Fixed FlowViewer rate output to calculate average from all flows
# 14. Fixed FlowTracker_Collector log output (to collect_period_average)
#
# Version 3.0
# 
# 1. Major new addition of FlowTracker
# 2. Reorganized scripts so that the date and time fields are updated
#    with each invocation
# 3. Moved common code (e.g., filter creation) to FlowViewer_Utilities.pm
# 4. Improved Report Parameters output formatting
# 5. Provided host names capability for FlowGrapher (thanks Mark Foster)
# 6. Introduced debug and logging capabilities
# 7. Merged GDBM/NDBM into a single script (thanks Ed Ravin)
# 
# Version 2.3
#
# 1. Modified FlowGrapher record processing to not call 'timelocal' for
#    epoch times. Other speed improvements. Result: up to 10 times faster.
# 2. FlowGrapher error leaving spikes is fixed (thanks Mark Foster)
# 3. Bug with concatenation when $N=0 fixed (thanks Dave Faught)
#
# Version 2.2
#
# 1. Added flow_select parameter to control which flows are considered
#    with respect to the specified time period
# 2. Removed Easterm Time (ET) notation. All times are system local
#
# Version 2.1
#
# 1. Fixed concatenation. Needs to start one flow file length before start time
# 2. Fixed end-of-year problem in FlowGrapher
# 3. Small problem for time requests that end just before midnight
#
# Version 2.0
#
# 1. Used pipe (|) instead of re-reading intermediate files (thanks Woj Kozicki!)
# 2. Introduced configurable variable $N to specify flow-directory nesting levels
# 3. Reduced default value of configurable variable $flow_capture_interval to 1800
# 4. Created FlowViewer_NDBM.cgi for users whose Perl does not have GDBM
# 5. Created configurable 'work_directory' separate from cgi_bin_directory
# 6. Sped up concatenation for requests that cross day boundaries
# 7. Added filter fields: Protocol, TOS Field, TCP Flags
# 8. Added some more syntax checking
# 9. Added FlowGrapher capability (requires GD for Perl)
#
# Version 1.0 (Original)
#
#
# Vital Assistance
#
# Special thanks to those FlowViewer users who provided feedback and valuable
# suggestions, including Sejin Ahn, Mark Boolootian, Bogdan Ghita, Woj Kozicki,
# Ed Arvin, Alex Shepherd, Mike Smith, Scott Wingfield, Vali Magdalinoiu,
# Eric Lautenschlaeger, Sean Cardus, Carles Kishimoto, Shigeki Taniguchi, Dave
# Faught, and Dario La Guardia. Big thanks to fellow toiler in the NASA vineyard
# Mark Foster for some detailed testing, excellent suggestions, and code to go
# along with it. Thanks from all of us to NASA whose unending pursuit of 
# innovation in all fields, has resulted in this toolset.
# 
#
# Bugs, recommendations
#
# If you need help installing, have a question, discover a bug, or have a 
# recommendation, please send an email to:
#
# Joe Loiacono
# jloiacon@csc.com
#
# FlowViewwer is being developed at NASA by a contractor in the employ of the United 
# States Federal Government in the course of his official duties. Pursuant to Title 
# 17, Section 105 of the United States Code, this software is not subject to copyright
# protection and is in the public domain. FlowViewer is an experimental system. NASA 
# assumes no responsibility whatsoever for its use by other parties, and makes no 
# guarantees, expressed or implied, about its quality, reliability, or any other 
# characteristic.