Data collection

This section describes how RA collects data.  If you are using RA on a cluster, please read the section "Using RA on a cluster" which explains how data collection works on a cluster.

Logicals you need
Data collection requires the four logicals: RA_DAT, RA_COM, RA_EXE, and RA_CDU be defined.  For more information on RA logicals including how to define them, refer to the section "Logicals you need to use RA".

Description of data collection
Since the data collection procedure collects information on each user's disk space as well as resource usage, it is most accurate if run every night. A system manager familiar with DCL could modify these procedures to run less often if desired; ensure that the ACC_SUMMARY job extracts the appropriate number of days of information if you choose not to run each night. Only disk usage would be affected by collecting data less often. Average disk usage is calculated by taking the total disk usage for a UIC and dividing by the number of entries for that UIC, therefore, running less often may produce a less accurate average but will still be fairly close if your users do not have large fluctuations in disk usage.  Once the data collection procedure is set up, it will normally run without further interaction.  However, if the system goes down, the batch jobs may have to be re-submitted.  To re-submit the data collection procedures, run the command procedure RA_COM:AUTOSUBMIT.COM or, if you have customized the procedures, resubmit them manually.

Here is a detailed description of the data collection process:

1)   ACC_DAY_UPDATE_nodename.COM runs before midnight to collect disk information. If the accuracy of disk storage charges is not critical this procedure can be modified to run less often. This procedure:

1.1) creates a disk usage file using the OpenVMS utility ANALYZE/DISK for each disk name in the RA_DAT:DISKNAMES.nodename file.

1.2) summarizes all the files by UIC to create a single disk usage record containing the total blocks allocated for each UIC for that day.

1.3) writes records containing the disk information to the OpenVMS accounting file ACCOUNTNG.DAT

ACC_DAY_UPDATE_nodename.COM must end before midnight so the records have the correct date.
2)    ACC_SUMMARY_nodename.COM runs after midnight.  This procedure:

2.1) extracts the accounting records for the PREVIOUS day from the OpenVMS accounting file ACCOUNTNG.DAT into a temporary file called nodename_ACCOUNTNG.TMP.  This includes the disk records generated by ACC_DAY_UPDATE_nodename.COM.

2.2) appends the records in nodename_ACCOUNTNG.TMP to RA's month-to-date file.  The file name is RA_DAT:nodename_ACCOUNTNG.month where "month" is a three character abbreviation for month (e.g. "JAN")

2.3) reads the nodename_ACCOUNTNG.TMP, summarizes the records by username and creates a summary file RA_DAT:nodename_ACCOUNTNG.SAV

2.4) submits DAILY_REPORTS.COM to create reports which are run every day.  You can use a text editor to modify this procedure to generate different reports or to run them using the month-to-date instead of the single day's data.

If these jobs abort, they run a procedure, TROUBLE.COM, which submits a job RA_ABORT which sends a message to the operator console every hour notifying them that there is trouble with data collection.  The message starts with the text "RA_ABORT" followed by a description of the error that caused the job to abort.

You should check the batch logs each morning to verify they completed successfully.  The logs are in SYS$LOGIN of the username under which the data collection procedures were submitted.

If ACC_DAY_UPDATE_nodename.COM fails before the final data-combining stage, you will not have any disk accounting records for that day.

If ACC_SUMMARY_nodename.COM fails before appending the prior day's accounting records to the RA accounting file, you will not have any accounting information for that day.

 Changing or re-starting data collection
The command procedure, AUTOSUBMIT.COM, submits the data collection batch procedures to start data collection.  This procedure is run during installation and may have to be run again if you change the data collection procedures or the system goes down.  Once the data collection procedures are started, they automatically re-submit themselves each night.

AUTOSUBMIT.COM asks you for the following information:

Deciding whether to collect disk information
Enter YES if RA is running on a stand alone system or just a single node of a cluster.  If RA is running on multiple nodes of a cluster, only one node will usually be collecting disk information.

Specifying disk drives
 AUTOSUBMIT asks which node to perform disk data collection on.  The disk names are stored in the file RA_DAT:DISKNAMES.nodename.  This file can be edited by a text editor, or recreated by running CREATE_DISK_FILE.COM.  RA can collect information on up to 200 disk drives. CREATE_DISK_FILE also prompts for a multiplication factor, default 1. This multiplication factor is useful for charging more for disk storage on expensive devices or for use with shadow sets where you would enter the physical device for 1 disk and multiply that by the number of disks in the set.

Picking a data collection time
During RA installation, AUTOSUBMIT calculates a submit time based on the number of disks on which data is collected.  Since ACC_DAY_UPDATE_nodename.COM must be finished before midnight, be sure to allow adequate time.  If RA is collecting disk information on this node, AUTOSUBMIT.COM will suggest a time for submitting ACC_DAY_UPDATE_nodename.COM.  This time is calculated based on the number of disks in DISKNAMES.nodename.  Usually the submit time is between 9:00 p.m. and 11:00 p.m.  This is usually a time of low activity so that the data collection process won't impact other processes.  ACC_DAY_UPDATE_nodename.COM must be finished by midnight.  You should ensure there is ample time for the procedure to complete.

Specifying a batch queue
You should not submit RA to a batch queue that runs at a lower priority or may be executing other jobs at the time RA is set to run.  If you do, RA may not be done by midnight.

Using the RA account
AUTOSUBMIT.COM will ask if you created the RA account.  If you did create the RA account, AUTOSUBMIT will submit the jobs under the RA username.  Otherwise they will be submitted under your current username.  Since ACC_SUMMARY_nodename.COM accesses the node's accounting file, ensure you submit the job to a queue that runs on the required node.

Collecting data manually
If data collection fails and you miss a day's data, use the following procedure to recover all the system accounting information; since ACC_DAY_UPDATE hasn’t run the data will not include any disk information:

1. Rename the existing OpenVMS accounting file as ACCOUTNG.TMP.
2. Use the /SINCE/BINARY qualifiers to extract the missing day's records from ACCOUNTNG.SAV into the file ACCOUNTNG.DAT.  Refer to the OpenVMS Accounting Utility manual for more information.
3. Run ACC_SUMMARY_nodename.COM to add the records to the RA month-to-date file.
4. Delete ACCOUNTNG.DAT.
5. Rename ACCOUNTNG.TMP as ACCOUNTNG.DAT.

Data collection's impact on the system
  Although data collection can be run while there are other users on the system, it is best to run it when there is no activity since ACC_DAY_UPDATE_nodename.COM uses the ANALYZE/DISK/USAGE (OpenVMS VERIFY) utility which causes a sizable load on the drive for a few minutes.
Data collection and BACKUP
If you run BACKUP at night, be sure that the run time does not conflict with the data collection run.  No errors will occur, but data collection will take MUCH longer if BACKUP is also running.  This also applies for any other site specific jobs that run during the night.  RA data collection is best performed on a lightly loaded system to ensure that the data collection is done by midnight.

Handling volume sets
RA works on volume sets.  Only the root volume device name should be specified in the RA_DAT:DISKNAMES.DAT file.  The root volume contains all the necessary information for the entire volume set.  If a device that is a member of a volume set is entered into this file (other than the root volume), it will be noted in the collection log file and the device will be bypassed.

Handling shadow sets
To collect disk usage on a shadow set you should specify the first volume of the shadow set for the disk and then, if desired, use the number of drives in the shadow set for the multiplication factor.

How invalid dates are handled
If the OpenVMS accounting file contains records with a date in the future, the OpenVMS accounting utility will use the resource values.  RA will not include the values from such records in its reports.  Normally this situation is quite unusual.  It could occur if you change the system date to a time in the past while accounting has been running.

RA and VDM
 With Version 6 of RA it is now possible to collect data for both RA and VDM in a single pass. Cohort V2 users will find that this decreases the data collection overhead by nearly 50%. To set this up a new command procedure has been included with RA called COMBINED_AUTOSUBMIT.COM. If VDM and RA data collections are running normally simply run this command procedure to submit SAIGA_DISK_COLLECTION_node.COM and remove the individual data collections from the queues. We do suggest having data collections working normally before making this change.