=head1 NAME

App::Netsync - network/database synchronization utility/library

=begin HTML

<a href="https://travis-ci.org/dmtucker/netsync"><img src="https://img.shields.io/travis/dmtucker/netsync.svg" alt="Build Status"></a>&nbsp;<a href="https://metacpan.org/pod/App::Netsync"><img src="https://img.shields.io/cpan/v/App-Netsync.svg" alt="Latest Release"></a>

=end HTML

=head1 DESCRIPTION

netsync is a network synchronization tool that:

 - maps network interfaces to their respective (potentially stacked) devices
 - gathers interface-specific information from an asset management database
 - sends the information it gathers to each device
 - provides ways of producing useful information about the network

I<Note: All communication with network nodes is done using SNMP.>

B<Overview>

In a typical run, execution begins by parsing a configuration file (-c).
Next, Netsync searches for active network devices at each node.
Nodes may be specified by filename, DNS (-D), or the keyboard (STDIN).
Furthermore, nodes may be required to match a hostname pattern (-m).
Gathered information may be used to identify each device in a database.
A database may be specified using Database (-d) or DB in the configuration file.
Identified devices may then be updated (-u) with interface-specific information.

See F<doc/algorithm.svg> for corresponding visual guidance
(F</usr/share/netsync/algorithm.svg> after installation).

=head2 0 Installation

 $ cpan App::Netsync

or

 $ perl Makefile.PL
 $ make
 $ make test
 $ make install

B<Modules>

=over 4

=item App::Netsync::Configurator

configuration file support

=item App::Netsync::Network

methods for manipulating network structures

=item App::Netsync::Scribe

I/O framework

=item App::Netsync::SNMP

SNMP framework

=back

=head2 1 Invocation

After installation, netsync may be invoked by name:

 $ netsync

If installation is skipped, netsync may be manually invoked as follows:

 $ perl -I lib script/netsync

B<Options>

=over 4

=item -h --help

Help. Print usage and options.

I<Note: Help and Version print information and exit.
        Normal execution is skipped when either of these options are used.>

=item -V

Version. Print build information.

I<Note: Help and Version print information and exit.
        Normal execution is skipped when either of these options are used.>

=item -v

Verbose. Print everything.

I<Note: Quiet mode overrides Verbose mode.>

=item -q

Quiet. Print nothing.

=item -c .ini

Config. Specify a configuration file.

default: F</etc/netsync/netsync.ini>

=item -p #

Probe. There are 2 Probe levels.

If the Probe option is used, resources will be created to aid in future runs.

I<Note: Each level corresponds to a stage of execution and is accumulative.>

=over 5

=item Probe Level 1

Probe the network for active nodes. See the Probe1Cache setting.

=item Probe Level 2

Probe the database for those nodes. See the Probe2Cache setting.

=back

=item -D

DNS. Use your network's domain name system to retrieve a list of nodes.

I<Note: If neither DNS (-D) nor Nodes are specified,
        input will be read from standard input (STDIN).
        If both are present, Nodes is used.>

=item -m pattern

Match. Only discover nodes with hostnames matching the given pattern.

=item -d .csv

Database. Specify an RFC4180-compliant database file.

=item -a

Automatch. Enable interface auto-matching.

I<Note: Automatch will be helpful if the database manages interfaces numerically.
        It aligns numerical ports (e.g. 23) in the database
        with ifNames (e.g. ethernet23 or Gi1/0/23) on the device.>

=item -u

Update. Send interface-specific information to network nodes.

=back

B<Parameters>

=over 4

=item nodes

Nodes. Specify an RFC1035-compliant list of network nodes.

I<Note: If neither DNS (-D) nor Nodes are specified,
        input will be read from standard input (STDIN).
        If both are present, Nodes is used.>

=back

=head2 2 Configuration

A configuration file may be specified using the Config (-c) option.
Otherwise, a generic configuration file (F</etc/netsync/netsync.ini>) is used,
but it will most likely require additional information before running properly.
Namely, the following settings must be provided for a typical installation:

=over 3

=item DNS

B<Required>

I<Note: These settings are unnecessary if only the Nodes parameter is used.>

=over 4

=item domain

network domain name (e.g. example.com)

=back

B<Optional>

See Net::DNS for more options.

=item SNMP

B<Required>

I<Note: SNMP settings are unnecessary if a default SNMP environment is used;
        however, this is HIGHLY insecure and discouraged.>

=over 4

=item Version

=over 5

=item SNMPv3 (recommended)

=over 6

=item SecLevel

the level of security to deploy

B<Security Levels>

=over 7

=item noAuthNoPriv (default)

This causes SNMPv3 to mimic v2!

=item authNoPriv

=item authPriv

=back

=item SecName

username

default: initial

=item AuthPass

the access (authentication) key

=item PrivPass

the privacy (encryption) key

=back

=item SNMPv2

=over 6

=item Community

the SNMP community to address

default: public

=back

=back

=back

B<Optional>

See SNMP::Session for more options.

=item DB

B<Optional>

See DBI for more options.

=item Netsync

B<Synopsis>

                           Table
 ---------------------------------------------------------
 |  DeviceField  |  InterfaceField  |  InfoFields...     |
 ---------------------------------------------------------         =============
 |   (serial)    |     (ifName)     |(interface-specific)|  --->   || SyncOID ||
 |                          ...                          |         =============
 ---------------------------------------------------------              (device)

I<Note: If the Update (-u) option is used,
        netsync will overwrite the device with the corresponding InfoFields.>

B<Required>

=over 4

=item DBMS

the type of database (i.e. a Perl DBD, e.g. Pg, mysql, Oracle, etc.).

=item Server

the database network location

=item Port

the database transport location

=item Database

the name of the database to use

=item Username

the name of a user that has access to the database

=item Password

the authentication key of the user

=back

I<Note: Settings above are unnecessary if only the Database (-d) option is used.>

=over 4

=item MIBdir

the location of required MIBs

default: F</usr/share/netsync/mib/>

B<Required MIBs>

=over 5

=item standard

=over 6

=item IF-MIB

=item ENTITY-MIB

=back

=item Cisco

=over 6

=item CISCO-STACK-MIB

=back

=item Brocade

=over 6

=item FOUNDRY-SN-AGENT-MIB

=item  FOUNDRY-SN-SWITCH-GROUP-MIB

=back

=item HP

=over 6

=item SEMI-MIB

=back

=back

=back

=over 4

=item Table

the name of the table in the database that contains the following fields

=item DeviceField

the field that provides a unique ID for each device (typically a serial number)

=item InterfaceField

the field that holds interface names retrieved from the IF-MIB (ifName) via SNMP

=item InfoFields

a comma-separated list of fields containing interface-specific information

=back

B<Optional>

=over 4

=item SyncOID

the MIB location to store interface-specific information via SNMP

=back

See netsync.pl and Netsync.pm for more information.

=back

=head2 3 Structures

Netsync builds an internal view of the network hardware whenever it is used.
Each node is associated with its IP address and device(s).
Each device is associated with its serial number and interface(s).
Each interface is associated with its ifName and interface-specific information.

The resulting data structure could be described as a list of trees.

 |-> node (IP)
 |-> node (IP)
 |-> node (IP)
 |                              -interface (ifName)
 |                             /
 |             -device (serial)--interface (ifName)
 |            /                \
 |-V node (IP)                  -interface (ifName)
 |            \
 |             -device (serial)--interface (ifName)
 |                             \
 |                              -interface (ifName)
 |-> node (IP)
 |-> node (IP)
 |                              -interface (ifName)
 |                             /
 |-V node (IP)--device (serial)--interface (ifName)
 |                             \
 |                              -interface (ifName)
 |-> node (IP)
 |-> node (IP)
 |-> node (IP)
 |-> node (IP)
 ...

B<States>

=over 3

=item inactive

unreachable or unresponsive on the network

=item active

reachable and responsive on the network

=item unrecognized

active but not processable

=item recognized

active and processable

=item unidentified

recognized but not in the database

=item identified

recognized and in the database

=back

=head2 4 Discovery

The first task netsync has is to find all relevant nodes on the network.
Relevant nodes are specified one of three ways:

=over 3

=item DNS (-D) option

A zone transfer is filtered by Netsync,
and a list of A or AAAA records with valid IPv4 or IPv6 addresses is produced.

 $ netsync -D

=item Nodes parameter

Nodes is a path to a file containing an RFC1035-compliant list of nodes.

 $ netsync -Dqp1
 $ netsync /var/cache/netsync/active.txt

=item STDIN (e.g. pipe or keyboard)

If neither -D nor Nodes are present, a node list is pulled from standard input.
This allows pipelining with other utilities for extended functionality.

 $ dig axfr example.com | grep hub123 | netsync

Z<>

 $ cat superset.txt | grep hub[0-9]+ | netsync

=back

I<Note: Match (-m) may be used with any of the methods above.>

 $ netsync -Dm "(sw|hub)[0-9]+"

Z<>

 $ netsync -Dqp1
 $ netsync -m "(sw|hub)[0-9]+" /var/cache/netsync/active.txt

Z<>

 $ dig axfr example.com | netsync -m "(sw|hub)[0-9]+"

Z<>

 www.example.com            <-- no match (www)
 hub123.example.com         <-- match (hub123)
 sw1234.example.com         <-- match (sw1234)
 ...

B<About RFC1035>

RFC1035 (3.2.1) specifies a format for resource records served by name servers.
Thus, for compatibility reasons, this application is RFC1035-compliant.

 name             ttl class type address
 www.example.com. 300 IN    A    93.184.216.119
 www.example.com. 300 IN    AAAA 2606:2800:220:6d:26bf:1447:1097:aa7
 ...

=head2 5 Recognition

netsync must attempt to contact each specified node to see if it is active.
Any node that netsync attempts to contact is logged in NodeLog.
If a node is active, netsync will try to retrieve its device serial number(s).
If multiple devices are discovered,
netsync will try to map each interface to its device (serial).
If no devices are discovered (or mapping is impossible),
The node is considered unrecognized and skipped.

I<Note: Only ASCII serials are supported.>

B<Supported Vendors>

=over 3

=item Brocade (stacks)

=item Cisco (stacks)

=item HP

=back

B<Interface Mapping>

=over 3

=item 1 Extract interfaces.

B<Standard>

=over 4

=item 1 ifType (1.3.6.1.2.1.2.2.1.3)

B<Excluded ifTypes>

=over 5

=item 1 other

=item 24 softwareLoopback

=item 53 propVirtual

=back

=item 2 ifDescr (1.3.6.1.2.1.2.2.1.2)

I<Note: ifName (1.3.6.1.2.1.31.1.1.1) may be an alternate.>

=back

=item 2 Extract serials.

B<Standard>

=over 4

=item 1 entPhysicalClass (1.3.6.1.2.1.47.1.1.1.1.5)

B<Included entPhysicalClasses>

=over 5

=item 3 chassis

=back

=item 2 entPhysicalSerialNum (1.3.6.1.2.1.47.1.1.1.1.11)

=back

B<Proprietary>

=over 4

=item Brocade

 snChasUnitSerNum : 1.3.6.1.4.1.1991.1.1.1.4.1.1.2

I<Note: snChasSerNum (1.3.6.1.4.1.1991.1.1.1.1.2) may be an alternate;
        however, it does NOT support stacks.>

=item Cisco

 moduleSerialNumber : 1.3.6.1.4.1.9.5.1.3.1.1.3

I<Note: moduleSerialNumberString (1.3.6.1.4.1.9.5.1.3.1.1.26) may be an alternate.>

=item HP

 hpHttpMgSerialNumber : 1.3.6.1.4.1.11.2.36.1.1.2.9

=back

=item 3 Map interfaces to serials.

B<Standard>

I<Note: There is no known standard for achieving this.>

B<Proprietary>

=over 4

=item Cisco

=over 5

=item 1 portIfIndex

1.3.6.1.4.1.9.5.1.4.1.1.11

=item 2 portModuleIndex

1.3.6.1.4.1.9.5.1.4.1.1.1

=item 3 moduleSerialNumber

1.3.6.1.4.1.9.5.1.3.1.1.3

I<Note: moduleSerialNumberString (1.3.6.1.4.1.9.5.1.3.1.1.26) may be an alternate.>

=back

=item Brocade

=over 5

=item 1 snSwPortIfIndex

1.3.6.1.4.1.1991.1.1.3.3.1.1.38

=item 2 snSwPortDescr

1.3.6.1.4.1.1991.1.1.3.3.1.1.39

I<Note: Unit/Module/Interface (U/M/I) is assumed to map Unit to module IID.>

=item 3 snChasUnitSerNum

1.3.6.1.4.1.1991.1.1.1.4.1.1.2

=back

=back

=back

=head2 6 Probe Level 1

Probe Level 1 is specified using -p1 and updates Probe1Cache.

During Probe Level 1, only the discovery stage is executed.
It caches discovered nodes in Probe1Cache.
This cache is created in RFC1035-complaint format.
It may subsequently be passed as the Nodes parameter to skip inactive nodes.

 $ netsync -Dm "sw[^.]+|hub[0-9]+" -p1
 > configuring (using /etc/netsync/netsync.ini)...
 > discovering (using DBMS)...  480 nodes (10 skipped), 500 devices
 $ netsync /var/cache/netsync/active.txt
 > configuring (using /etc/netsync/netsync.ini)...
 > discovering (using /var/cache/netsync/active.txt)...  480 nodes, 500 devices
 > identifying (using DBMS)...  498 synchronized (4 conflicts)

I<Note: Since the goal of Probe Level 1 is to save time later,
        unrecognized nodes will not be included in Probe1Cache.>

=head2 7 Identification

Once netsync has a view of the network's hardware,
it requires a database to find information specific to each interface.

I<Note: Since netsync treats the database as read-only,
        the table and fields must already be present in the database.>

The database may be provided one of two ways:

=over 3

=item DBMS

This must be preconfigured in the configuration file and on the DBMS.

=item Database (-d)

An RFC4180-compliant database file may be specified.

=back

netsync identifies the network interface referenced in each database entry.
A valid entry must not have missing DeviceField or InterfaceField information.
Valid rows are synchronized with the network while invalid rows are skipped.
Entries with unknown (not found) devices are skipped.
Entries are then checked for conflicts.

B<About RFC4180>

RFC4180 specifies a simple format (CSV) for database files.
This format is widely supported making it useful for importing and exporting.
Thus, for compatibility, this application is RFC4180-compliant.

 DeviceField,InterfaceField,InfoField
 A1B2C3D4E5F6,ethernet1/1/1,PoE Enabled
 A1B2C3D4E5F6,ethernet1/2/1,Uplink
 ...

=head2 8 Synchronization

All conflicts are logged in ConflictLog and skipped.

B<Conflicts>

=over 3

=item Mismatch

This occurs when an InterfaceField value can't be found on an identified device.

=item Duplicate

This occurs when more than 1 entry for an interface exists in the database.

=item Unidentified

This occurs when hardware is found on the network but not in the database.

=back

=head2 9 Probe Level 2

Probe level 2 is specified using -p2 and updates Probe1Cache and Probe2Cache.

During Probe Level 2, only the discovery and identification stages are executed.
It caches identified nodes in Probe2Cache.
This cache is created in RFC4180-complaint format.
It may be passed as Nodes in subsequent runs to skip synchronization.

 $ netsync -Dm "host[0-9]+" -a -p2
 > configuring (using /etc/netsync/netsync.ini)... done
 > discovering (using DNS)...  480 nodes (10 skipped), 500 devices
 > identifying (using DBMS)...  498 synchronized (4 conflicts)
 > Do you want to resolve conflicts now? [y/n] n
 $ netsync -d /var/cache/netsync/synced.csv /var/cache/netsync/active.txt
 > configuring (using /etc/netsync/netsync.ini)... done
 > discovering (using /var/cache/netsync/active.txt)...  480 nodes, 500 devices
 > identifying (using /var/cache/netsync/synced.csv)...  500 synchronized

I<Note: All unidentified hardware will be present in Probe2Cache.
        This is so the output of Probe Level 2 can serve as a network snapshot.>

=head2 10 Updating

All modifications made to any device are logged in UpdateLog.

If probing is not used and the Update (-u) option is specified,
netsync attempts to actualize its internally synchronized network using SNMP.
It pushes gathered interface-specific information to the devices on the network.
This information is stored in the device at the specified SyncOID,
and is overwritten anytime netsync updates it.

 $ netsync -Dau

=head1 EXAMPLES

 $ netsync -Dm "sw[^.]+|hub[0-9]+" -au
 > configuring (using /etc/netsync/netsync.ini)...
 > discovering (using DNS)...  780 nodes (50 skipped), 800 devices (10 stacks)
 > identifying (using DBMS)...  670 synchronized (4 conflicts)
 > updating... 670 successful

Z<>

 $ dig axfr domain.tld | egrep ^(sw[^.]+|hub[0-9]+) | netsync -a
 > configuring (using /etc/netsync/netsync.ini)...
 > discovering (using STDIN)...  780 nodes (50 skipped), 800 devices
 > identifying (using DBMS)...  670 synchronized (4 conflicts)

Z<>

 $ netsync -Dm "sw[^.]+|hub[0-9]+" -p1
 > configuring (using /etc/netsync/netsync.ini)...
 > discovering (using DBMS)...  780 nodes (50 inactive), 800 devices
 $ netsync /var/cache/netsync/active.txt
 > configuring (using /etc/netsync/netsync.ini)...
 > discovering (using /var/cache/netsync/active.txt)...  780 nodes, 800 devices
 > identifying (using DBMS)...  670 synchronized (4 conflicts)

Z<>

 $ netsync -Dm "sw[^.]+|hub[0-9]+" -a -p2
 > configuring (using /etc/netsync/netsync.ini)...
 > discovering (using DNS)...  780 nodes (50 skipped), 800 devices
 > identifying (using DBMS)...  670 synchronized (4 conflicts)
 $ netsync -d /var/cache/netsync/synced.csv /var/cache/netsync/active.txt
 > configuring (using /etc/netsync/netsync.ini)...
 > discovering (using /var/cache/netsync/active.txt)...  780 nodes, 800 devices
 > identifying (using /var/cache/netsync/synced.csv)...  800 synchronized

=head1 AUTHOR

David Tucker, C<< <dmtucker at ucsc.edu> >>

=head1 BUGS

Please report any bugs or feature requests to C<bug-netsync at rt.cpan.org>, or through
the web interface at L<http://rt.cpan.org/NoAuth/ReportBug.html?Queue=App-Netsync>.  I will be notified, and then you'll
automatically be notified of progress on your bug as I make changes.

=head1 SUPPORT

You can find documentation for this module with the perldoc command.

 perldoc App::Netsync

You can also look for information at:

=over 4

=item * RT: CPAN's request tracker (report bugs here)

L<http://rt.cpan.org/NoAuth/Bugs.html?Dist=App-Netsync>

=item * AnnoCPAN: Annotated CPAN documentation

L<http://annocpan.org/dist/App-Netsync>

=item * CPAN Ratings

L<http://cpanratings.perl.org/d/App-Netsync>

=item * Search CPAN

L<http://search.cpan.org/dist/App-Netsync/>

=back

=head1 LICENSE

Copyright 2013 David Tucker.

This program is free software; you can redistribute it and/or modify it
under the terms of either: the GNU General Public License as published
by the Free Software Foundation; or the Artistic License.

See L<http://dev.perl.org/licenses/> for more information.

=cut