Vermont for Linux with V

Abstract

An interpretive, declarative computer language is described and its grammar is defined. The language's applicability to helping system administrators with package configuration, resource monitoring, and security compliance is demonstrated with examples. The first implementation of the language, running on Linux Fedora 16, with a basic set of resource and validation types is documented.

The V language

The V language is a domain-specific language for capturing the operational values of a host computer's resources and for checking their validity against user-defined rules. The problem domain addressed by V is computer administration. In particular, V is well suited with helping an administrator in monitoring a host's configuration, performance, and vulnerabilities.

V is nullipotent, that is, its statements can be executed any number of times to produce the same result. The language is used only to observe the state of the host, never to alter its state.

V is a declarative language: it specifies what to do, but doesn't prescribe how to do it. Unlike imperative languages, V does not contain conditional branching statements or looping constructs. V is not a functional language and does not have subroutines, but V does have the concept of dependencies, which can be used to define the order of execution.

V operates on objects, which describe host resource variables and user-defined rules. Host resource variables are described with resource objects, and user-defined rules are described with validation objects. Both types of objects use the same object definition syntax. Both types of objects belong to namespaces, where namespaces define variable and rule defaults for the objects in their scope.

V is a dynamically typed language where the default type for all variables is a string. Whole numbers and real numbers are implicitly recognized and used—as integers and floating point values—in comparison operations.

V is an interpreted language, rather than a compiled language, and thus requires no explicit pre-processing step. Language statements are inspected during the parsing stage and validated against the language's grammar and syntax.

In practice, V is used in a manner similar to some scripting languages (like Bash and Perl), to solve problems with a well-defined set of inputs. It is not used as an application development language, and it is not typically well suited to solving problems with arbitrary user inputs.

V is modular in design, and it's interpreter can discover and use plugin modules, written by third-parties, without rebuilding the interpreter.

V is well suited to helping system administrators with monitoring computer systems using a declarative, interpretive, rule-based language that safely works without side effects.

Problem domain

Computer system administration entails the configuration, maintenance, monitoring, and troubleshooting of hardware and software systems. The principal objectives of the system administrator are to ensure the correct functioning of each of the system's parts; to ensure the availability of system resources to authorized users when they are needed; and to safeguard the information assets that are stored in, or pass through, the system. Special computer administration tools exist to support each of these objectives, with many of these tools providing detailed and precise inspection of system resources.

A system administrator's ability to meet the principal objectives are dependent upon his working knowledge of the system's configuration and the availability of diagnostic tools. Working knowledge is the critical dependency addressed by the V language; in this context, working knowledge is the amalgamation of the administrator's training and experience, plus any formal component or package documentation that is available, plus, very often, the anecdotal experience of others working with similar systems.

The problem that V addresses, is the system administrator's limited capacity to retain and recall his working knowledge of the system's correct configuration. Each of us is limited in what we can remember, how long we can remember it, and how accurate our memory is. And this last point about accuracy is critical, because accuracy is not optional: configuration syntax is both exacting and unforgiving. Configuration omissions can cause sub-systems to become completely inoperable; and configuration options that are not missing, but are simply wrong, can create latent problems that only manifest themselves much later. Even worse, in some cases, configuration mistakes can cause systems to work as expected—but for the wrong reasons, adding false information to the system administrator's working knowledge.

V addresses these problems of retention and recall through the specification of rules. This specification, in it's simplist form, is akin to the type of specification provided when someone creates system documentation manuals, or when someone regularly records information in a system log. But the specification of rules using V provides extra benefits that manuals and logs don't have. This is because V is executable, and it can evaluate the true state of the system against those rules, and in the process can identify resources that are not in compliance. And it can do this repeatedly, on demand, or at regular intervals. System administrators who use V as part of their routine work flow—specifying new rules as packages are installed, and changing existing rules to match new working knowledge as it is acquired—can effectively deal with the problem of retention and recall.

In addition to its ability to evaluate rules against the active state of the system, V is well suited for use with software development tools, such as "diff" tools and source code repositories. This means that the administrator can readily see the history of changes made to the system.

Because V is a file-based source code language, administrators can copy rulebooks from one host to another, evaluate those rules on the new host, and compare the evaluation results of the two systems. This use of the language could lend itself well to use in community based technical support forums, where the exchange of rules would allow the two parties to compare expected versus actual results.

V also addresses the problem of technology transfer, where technology transfer is the problem of how to prepare for the orderly transition from one administrator to the next. If an organization embraces the use of V and specifies rules for its systems, it will, over time, build a collection of working knowledge. And most importantly, this collection of working knowledge will reside, not in its administrators, but rather in readily transferable source code files. The all too common situation where a departing administrator is asked to "do a brain dump" on his way out the door, would in some measure be reduced to simply passing along a collection of V language files.

The span of a computer system's deployment is often referred to as its lifetime. And this is an apt description because systems grow in ways that are seemingly unpredictable, and almost organic in nature. A system's body is carefully grafted and pruned by an administrator when components and packages are installed, upgraded or removed; but each of these operations can lead to problems that the system administrator needs to resolve, and some of the final resolutions to these problems are obtained only after hard-fought battles. V provides a way to capture this hard-fought knowledge and to apply it to future situations. When applied over a system's lifetime, V becomes an excellent language for ensuring the health of the system.

Usage scenarios

V can effectively be used in scenarios involving system configuration, system availability, system performance and system security.

Installing, upgrading, or removing system components—system configuration for short—is a disruptive process. Taking a system from one stable state to a new stable state, is the goal of system configuration. Between these two stable states is the messy arena of testing, tuning, monitoring, problem discovery, and diagnostic troubleshooting. V is a good tool for working in this arena, providing the system administrator with repeatable results that can reliably be used to compare before and after differences.

Systems that have been running in a proven, stable state, are not immune to the effects of stochastic inputs, such as higher than average volumes of data, out of range data, hardware failures, or broken links to other dependent systems. These stochastic inputs may cause the system to become unstable or unavailable. V can be used to determine whether or not a system's individual components are available: are networks accessible, can mail be sent, are filesystems available, are daemons running?

Over time, systems are asked to do more with the same set of resources. This increased work load can cause system throughput to decrease and users to be subject to unacceptable delays. System performance monitoring can be accomplished with V's threshold trip-point evaluators. Using V statements, a system administrator defines the resource to be monitored, and specifies the maximum or minimum acceptable value for that resource. Rules can be written that monitor CPU, memory, filesystem, network bandwidth, and other hardware utilization. Also, V language rules can be written for software packages that have monitoring API's, such as database servers, web servers, DNS servers and mail servers.

Systems are provisioned and deployed for an intended usage and for an intended audience. When unethical users place excessive or illegitimate demands on a system, the intended audience can be prevented from using the system as designed. Also, when criminal users steal data or intellectual property from a system, corporate responsibility rules mandate that legal authorities and affected third parties be promptly notified. V has provisions for examining system log files, for identifying policy violations, and for reporting breaches. Reports from V can be used by a system administrator to implement tighter restrictions on resource usage.

The Vermont for Linux implementation of V can be set to run at periodic intervals using the Linux cron facility. Rulebooks can be written in the V language, where each rulebook contains distinct sets of resource objects and validation objects. Cron can be set to evaluate each rulebook at a different frequency. For example, a security rulebook might be run every fifteen minutes, a performance monitoring rulebook might be run every hour, a system availability rulebook might be run once a day, and a system configuration rulebook might be run only on demand, such as just before and just after a planned upgrade.

No side effects

The V language evaluates the state of system resources and verifies that the state is within specified bounds. Both the evaluation step and the verification step, can be thought of as read-only operations, and as such should not effect the state of the system. V language statements can thus be safely executed without disrupting the normal operation of the system.

This guarantee, that the language have no side effects, can be enforced by executing V with restrictive usage rights. For example, on systems that have Security Enhanced Linux, auditing rules could be established where a user could have super-user access to read all resources, but no access to write to any resource.

In the Vermont for Linux implementation of the language, this guarantee is made without regard to any special user access rights. Any user may safely execute V language statements without side effects, because all parts of the system have been created with this rule specifically in mind.

This guarantee can be broken however, by users who write plugin modules. Plugin modules are interpreted and executed at runtime without restriction. For this reason, community members who obtain third-party plugins, should be prudent in their selection of plugins from untrustworthy sources.

Declarative language

V is a declarative language, which means that it does not have expressions that describe how to do something. There are no conditional, if-then-else branches, there are no subroutines, and there are no for-while-do loops.

The language processes instructions without regard to the order of statements in the source files. Processing follows a fixed set of steps, proceeding from parsing, to evaluation, to validation; although the processing of objects within these fixed steps may be explicitly ordered using V's dependency grammar.

For those who are unfamiliar with the distinction between declarative and imperative, consider the CSS language, which is used for specifying the typeface, color, and layout of HTML documents: V, just like CSS, is a declarative language.

Because the language has no conditional flow statements, it is not Turing complete, that is, it can't be used to express any arbitrary computational problem, although in practice, this is less of a limitation than it may at first seem, because the language does provide for the general purpose evaluation and validation of expressions.

Object definition

V processes objects that are placed in one or more source files, which contain statements that define objects. These objects are defined using a regular grammar.

Two types of objects can be defined with this grammar: those that describe host resources that are to be evaluated, such as CPU, memory, filesystems, or networks; and those that express rules that are to be validated.

The process of resource evaluation is the process of instantaneously capturing the metrics of a resource. The process of rule validation is the process of comparing these instantaneous resource metrics to threshold values.

Objects are defined using statements that follow a regular set of syntactic rules. Briefly these include:

• One statement can occur per line, and each statement is fully expressed in that single line. There are no multi-line statements, and there is no line continuation symbol.
• Comments can occur on any line, and the start of a comment signals the end of the statement; there are no embedded comments.
• There are two broad types of statements: namespace-object scoping statements, and keyword-value definition statements.
• Namespace and object names may be any length and may be composed of any Unicode character except these [ ] : . # = |
• Keyword names may be composed only of the 63 characters in the set a-z A-Z 0-9 _, that is, the alphanumerics plus the underscore.
• Scoping statements use hard brackets [] to surround the namespace-object name, and a colon : to separate the namespace from the object.
• Definition statements use an equals sign = to assign values to a keyword.
• Keywords that have scalar values (single values) are expressed using the simple assignment symbol =.
• Keywords that have sets of values (arrays) are expressed with concatenation, using either a pipe | or a piped assignment symbol |=.
• Keyword values may be referenced through variables, where the variable name is composed of a namespace, object, and keyword, prefixed with a dollar sign $.
• Single quotes and double quotes are not specially interpreted and have no syntactic effect.
• String values begin with the first non-whitespace character and end with the last non-whitespace character; values cannot be expressed with leading or trailing spaces.

Here is a sample of what the V language looks like.

[Sample]
type = glob
path = /var/vermont/sandbox
permissions = 640     # owners R/W, group R/O, public none
					

In this sample, the statements define an object called Sample that verifies that all files in the /var/vermont/sandbox directory have filesystem permissions of 640 (where the owner has both read and write access, the group has read-only access, and the general public has no access).

Note that this is not just a code snippet, it is the complete definition of an object, and is fully executable by the Vermont for Linux implementation.

Language versus implementation

The V language is distinct from the Vermont for Linux runtime implementation. V is a specification of grammar, syntax, and semantics. In contrast, Vermont for Linux (or its acronym "VFL"), is a particular implementation of the language, with limitations that match the operating system it runs on. As a language specification, V could be implemented on other operating systems and V language source code files could be interoperable between those implementations.

Clearly there are large differences in operating systems, and likewise there are large differences in the administration of those different systems. Some, but not all, of these differences can be masked through the language. For example, a keyword such as path, which is used to specify a fully qualified directory name, can equally apply to nearly all operating systems, with such minor differences as the use of forward and backward slashes being gracefully handled by the implementation. On the other hand, keywords such as selinux_type and selinux_user are only applicable to Linux operating systems, such as Fedora, that have Security Enhanced Linux installed.

The V language specification is not affected by these differences, but any specific implementation of the language is. Vermont for Linux, for example, treats the presence of unknown keywords as if they were typing mistakes, and issues a warning message to the console. This treatment can be overridden using the V language vocabulary keyword, thus allowing the system administrator to suppress these warning messages. Other implementations may choose to make their own decision on how to treat such differences.

Architecture

A set of resource objects and validation rules specified in the V language can be written with one or more source code files. By convention these filenames use the "" filename extension. Source code files placed in a common directory are collectively referred to as a rulebook.

The VFL runtime is the executable environment that reads the rulebook, evaluates resources, validates rules, and sends output to the console and the log file. The VFL runtime consists of a parser, an object composer, a dependency tree builder, a resource evaluator, a variable substitution processor, a rule validator, and a trip-point emitter.

Note however, that the four advanced features—object composer, dependency tree builder, resource evaluator, and variable substitution processor—do not need to be used in all scenarios. The VFL runtime can be a powerful system administrator tool even when without these advanced features. They are an integral part of the architecture and processing flow, and are introduced here to enable the reader to conceptually understand the full capabilities of the runtime and language; nevertheless, a beginner's understanding of these advanced features can be deferred until later.

The parser reads each line of a source code file, and builds an in-memory representation of the objects it specifies. The parser is responsible for distinguishing between valid and invalid syntax. It should gracefully recover from syntax errors and continue parsing with the next valid statement.

The object composer is responsible for merging base objects with scoping objects. It is called by the runtime system after all source code files have been parsed.

The dependency tree builder prepares the rulebook for future use by ordering the base objects according to any dependencies that the system administrator specifies in the source code files.

The resource evaluator obtains snapshot data for all specified resource objects, as defined by the system administrator using the V language. Snapshot data is stored in-memory, in the resource collection, for use in subsequent processing steps.

The variable substitution processor scans the collection of composed-and-evaluated objects looking for dollar-sign prefixed variables, replacing these variable names with the snapshot data obtained by the resource evaluator.

The rule validator scans all composed-evaluated-substituted objects, in dependency order, validating each object rule for compliance. If any rule does not comply, it trips the object and sends the object and the non-complying rule to the trip-point emitter.

The trip-point emitter determines what to do with objects that have failed validation. The emitter shunts the output to either the console's standard output or to the VFL log file. The emitter also gathers diagnostic information to include in the output message.

Collectively the seven components of the VFL runtime operate as a whole, and—under normal operation—only produce output via the trip-point emitter. Nevertheless, the VFL runtime can be instructed to operate in verbose mode to allow the system administrator to selectively examine the in-memory structures of each component.

Processing flow

The VFL runtime operates on a rulebook, where a rulebook is the set of all V language source code files located in a common directory. The location of this directory is specified on the command line using the -d or --directory argument. The VFL runtime is executed using the vv command.

$ vv -d /var/vermont/etc/conf.samples
					

A rulebook may optionally contain a Vermont for Linux configuration file, which is itself written using the V language (albeit a restrictive subset). The configuration file specifies how the VFL trip-point evaluator should behave. By convention, the configuration file is called config, but may be explicitly named using the -c or --config command line argument.

The runtime operates on each V language file that it finds in the rulebook directory (in no particular order), sending each file to the parser to start the process. The runtime may be instructed to operate on a portion of the rulebook, rather than the entire rulebook, by specifying a subset of files using the -f or --files command line argument.

Parser

The parser operates within the context of object scopes. An object scope is the set of all statements that follow an object name and precede the next object name. Object names are specified by surrounding a string label with hard brackets. In this sample source code file, there are two base objects, [A] and [B], each with three statements in their scope.

[A]
type = file
path = /etc/sysconfig/network
permissions = 640
[B]
type = file
path = /etc/sysconfig/rsyslog
permissions = 640                    
					

Object composer

The object composer is called by the runtime system immediately after the parser has finished reading all source code files in the rulebook.

In the previous example, two objects were specified with simple, unqualified names, [A] and [B]. Objects can also be defined with compound, qualified names—where the two components of the name are separated with a colon—as in [N:A], where N is the scoping name, and A is the base name.

A scoping object is one whose scoping name and base name are the same. Scoping objects are used when specifying statements that are common to two or more objects. In the following sample source code file, there is one scoping object N, specified with the [N:N] object name, and two base objects A and B specified with the [N:A] and [N:B] object names. This example is equivalent to the previous example.

[N:N]
type = file
permissions = 640
[N:A]
path = /etc/sysconfig/network
[N:B]
path = /etc/sysconfig/rsyslog                    
					

The object composer merges the statements of the scoping object together with the statements of the base object. The merged objects are called composed objects. Composed objects are used by the dependency tree builder, resource evaluator, variable substitution processor, rule validator, and trip-point emitter.

Dependency tree builder

The dependency tree builder reorders the rulebook's base objects according to any dependencies specified in the source code files. By default, the ordering of objects is indeterminate, and will vary according to how the host's filesystem responds to a request to retrieve a glob of files. When one object must be evaluated before another object, the dependency keyword can be used to instruct the dependency tree builder to explicitly reorder the future use of those objects.

In this example, A is explicitly stated to be dependent upon B, so the order of evaluation will be altered such that B is evaluated before A. Note also that in this example, C is independent, and could be evaluated before, during, or after A and B.

[N:N]
type = file
permissions = 640
[A]
path = /etc/sysconfig/network
dependency = B
[B]
path = /etc/sysconfig/rsyslog
[C]
path = /etc/sysconfig/i18n                    
					

Resource evaluator

The resource evaluator captures a snapshot of the host computer's dynamic attributes. The resource evaluator iterates over each composed object—in the order determined by the dependency tree builder—capturing point-in-time measures of those dynamic attributes.

As a frame of reference, the reader should understand that composed objects may be one of two generic types: resource objects or validation objects. The resource evaluator skips over the validation objects (which are described later), working only with resource objects.

Resource objects specify which component of the host computer to inspect, and which attribute of the component to evaluate. Components may be any part of the host that has measurable attributes, such as CPU, memory, filesystems, network interfaces, etc. Attributes may be any numeric or textual value that provides information about the operational status of the component, such as cpu load, memory utilization, filesystem availability, or network throughput.

In this example, a resource is defined which captures four virtual memory attributes, and places a snapshot of those values in the four variables: vms, vmu, vmf, vmp.

[R]
type = resource
resource = memory
vms = virtual_memory_size
vmu = virtual_memory_used
vmf = virtual_memory_free
vmp = virtual_memory_percent
					

Resource modules may be defined, by plug-in developers, to capture a variety of hardware, operating system, and software package values. When plug-in developers create a custom resource module, they publish a module vocabulary which is the formal definition of keywords that can be used by system administrators when writing rulebooks.

In the previous example, the four names (virtual_memory_size, virtual_memory_used, virtual_memory_free, virtual_memory_percent) used in the right-hand side of the statements, are keywords that are part of the vocabulary published by the memory resource module.

When the resource evaluator has completed its work, the internal representation of the object [R] might look something like this:

. . .
vms = 4128764
vmu = 596
vmf = 4128168
vmp = 99
. . .                    
					

With this evaluation complete, the variable substitution processor can be started.

Variable substitution processor

The variable substitution processor scans the collection of composed-and-evaluated objects looking for dollar-sign prefixed variables.

In this step, objects are again processed in dependency order, but without distinction as to whether the objects are resource objects or validation objects. (Conceptually, validation objects can be thought of as having a meaning that is distinct from resource objects, but formally, resource objects are themselves validation objects, albeit simple ones.)

For each dollar-sign prefixed variable name encountered by the substitution processor, the variable name is replaced with the value obtained from the just-evaluated resource collection. In this example, the [R] object variable vmp is referenced by the [S] object using the syntax .vmp.

[R]
type = resource
resource = memory
vmp = virtual_memory_percent
[S]
type = assertion
assert = $R.vmp GT 15  # Free virtual memory should be > 15%