FAST ESP Operations Guide

FAST Enterprise Search Platform 

version:5.3 

Operations Guide 

Document Number: ESP1025, Document Revision: B, December 03, 2009

Copyright 

Copyright © 1997-2009 by Fast Search & Transfer ASA (“FAST”). Some portions may be copyrighted 

by FAST’s licensors. All rights reserved.The documentation is protected by the copyright laws of Norway, 

the United States, and other countries and international treaties. No copyright notices may be removed 

from the documentation. No part of this document may be reproduced, modified, copied, stored in a 

retrieval system, or transmitted in any form or any means, electronic or mechanical, including 

photocopying and recording, for any purpose other than the purchaser’s use, without the written 

permission of FAST. Information in this documentation is subject to change without notice.The software 

described in this document is furnished under a license agreement and may be used only in accordance 

with the terms of the agreement. 

Trademarks 

FAST ESP, the FAST logos, FAST Personal Search, FAST mSearch, FAST InStream, FAST AdVisor, 

FAST Marketrac, FAST ProPublish, FAST Sentimeter, FAST Scope Search, FAST Live Analytics, FAST 

Contextual Insight, FAST Dynamic Merchandising, FAST SDA, FAST MetaWeb, FAST InPerspective, 

NXT, FAST Unity, FAST Radar, RetrievalWare, AdMomentum, and all other FAST product names 

contained herein are either registered trademarks or trademarks of Fast Search & Transfer ASA in 

Norway, the United States and/or other countries. All rights reserved. This documentation is published 

in the United States and/or other countries. 

Sun, Sun Microsystems, the Sun Logo, all SPARC trademarks, Java, and Solaris are trademarks or 

registered trademarks of Sun Microsystems, Inc. in the United States and other countries. 

Netscape is a registered trademark of Netscape Communications Corporation in the United States and 

other countries. 

Microsoft, Windows, Visual Basic, and Internet Explorer are either registered trademarks or trademarks 

of Microsoft Corporation in the United States and/or other countries. 

Red Hat is a registered trademark of Red Hat, Inc. 

UNIX is a registered trademark of The Open Group in the United States and other countries. 

Linux is the registered trademark of Linus Torvalds in the U.S. and other countries. 

AIX and IBM Classes for Unicode are registered trademarks or trademarks of International Business 

Machines Corporation in the United States, other countries, or both. 

HP and the names of HP products referenced herein are either registered trademarks or service marks, 

or trademarks or service marks, of Hewlett-Packard Company in the United States and/or other countries. 

Remedy is a registered trademark, and Magic is a trademark, of BMC Software, Inc. in the United States 

and/or other countries. 

XML Parser is a trademark of The Apache Software Foundation. 

All other company, product, and service names are the property of their respective holders and may be 

registered trademarks or trademarks in the United States and/or other countries. 

Restricted Rights Legend 

The documentation and accompanying software are provided to the U.S. government in a transaction 

subject to the Federal Acquisition Regulations with Restricted Rights. Use, duplication, or disclosure of 

the documentation and software by the government is subject to restrictions as set forth in FAR 52.227-19 

Commercial Computer Software-Restricted Rights (June 1987).

Contact Us 

Web Site 

Please visit us at: http://www.fastsearch.com/ 

Contacting FAST 

FAST 

Cutler Lake Corporate Center 

117 Kendrick Street, Suite 100 

Needham, MA 02492 USA 

Tel: +1 (781) 304-2400 (8:30am - 5:30pm EST) 

Fax: +1 (781) 304-2410 

Technical Support and Licensing Procedures 

Technical support for customers with active FAST Maintenance and Support agreements, e-mail: 

fasttech@microsoft.com 

For obtaining FAST licenses or software, contact your FAST Account Manager or e-mail: 

fastcsrv@microsoft.com 

For evaluations, contact your FAST Sales Representative or FAST Sales Engineer. 

Product Training 

E-mail: fastuniv@microsoft.com 

To access the FAST University Learning Portal, go to: http://www.fastuniversity.com/ 

Sales 

E-mail: sales@fastsearch.com

Contents 

Preface..................................................................................................ii 

Copyright..................................................................................................................................ii 

Contact Us...............................................................................................................................iii 

Chapter 1: Starting, stopping, suspending, and resuming............11 

Starting...................................................................................................................................12 

Setting environment variables on UNIX.......................................................................12 

Setting environment variables on Windows.................................................................12 

Starting FAST ESP on a UNIX node...........................................................................12 

Starting FAST ESP on a Windows node......................................................................12 

Starting FAST ESP with fault tolerant indexers...........................................................13 

Stopping.................................................................................................................................13 

Stopping FAST ESP via sequential shutdown.............................................................13 

Stopping FAST ESP via simple shutdown (UNIX).......................................................15 

Stopping FAST ESP via simple shutdown (Windows).................................................15 

Starting and stopping modules via nctrl.................................................................................15 

Single node scenario...................................................................................................16 

Multi-node scenario.....................................................................................................16 

Chapter 2: Backing up and restoring...............................................17 

Determining backup and restore needs.................................................................................18 

Configuration...............................................................................................................18 

Content........................................................................................................................19 

Checklists: Backing up and restoring specific types of nodes................................................20 

Single node system backup and restore.....................................................................20 

Administration node backup and restore in a multi-node system................................21 

QRServer node backup and restore............................................................................22 

Processing server node backup and restore...............................................................22 

Backing up and restoring configuration data..........................................................................23 

System configuration backup and restore...................................................................23 

Resource service backup and restore.........................................................................26 

Backing up and restoring dictionaries..........................................................................26 

Admin server backup and restore................................................................................27 

Backing up and restoring content...........................................................................................28 

Preparation for backup................................................................................................28 

Backing up and restoring content connectors.............................................................29 

Index data backup and restore....................................................................................29 

5


6 

Chapter 3: Applying runtime deployment changes........................33 

Installing nodes from the install profile...................................................................................34 

Reinstalling a node via the install profile.....................................................................34 

Adding new nodes via the install profile .....................................................................34 

Increasing indexing and search capacity...............................................................................35 

Adding new search or index nodes via the install profile.............................................36 

Using row and column IDs for search engine nodes...................................................37 

Adding a new dedicated search row............................................................................38 

Adding an indexer column...........................................................................................39 

Adding an indexer column (for archive indexing).........................................................42 

Adding an indexer row.................................................................................................44 

Adding a top-level dispatcher.................................................................................................46 

searchrc-dispatch.xml attributes..................................................................................48 

Adding a new QRServer.........................................................................................................48 

Increasing document feed and processing capacity..............................................................50 

Adding a crawler on an existing node via FAST ESP administrator interface..............50 

Adding a crawler node via install profile......................................................................50 

Adding a processor server on an existing node via FAST ESP administrator interface.51 

Adding a document processor node via install profile.................................................51 

Adding a content distributor....................................................................................................52 

Adding an indexing dispatcher...............................................................................................53 

Multi-node license file considerations.....................................................................................54 

Increasing indexing and search capacity with independent search nodes.............................54 

Moving RTS.................................................................................................................55 

Moving a QRServer.....................................................................................................56 

Moving an RTS top dispatcher....................................................................................57 

Chapter 4: Running FAST ESP in standalone mode.......................59 

About standalone mode.........................................................................................................60 

Creating standalone versions of processes...........................................................................60 

Creating a standalone QRServer................................................................................60 

Creating a standalone search process........................................................................61 

Creating a standalone top-level dispatcher.................................................................62 

Switching search to standalone mode....................................................................................63 

Restoring normal operations from standalone search...........................................................65 

Chapter 5: Working with an Indexing Fault Tolerant Configuration.67 

Starting Indexers in a Fault Tolerant Configuration.................................................................68 

Stopping Indexers in a Fault Tolerant Configuration...............................................................68 

Enabling Indexing Fault Tolerance..........................................................................................69 

Disabling Indexing Fault Tolerance.........................................................................................70 

Synchronizing Indexer Rows..................................................................................................72

Upgrading Indexer Binaries in a Fault Tolerant Configuration ...............................................73 

Backing up Indexing in a Fault Tolerant Configuration............................................................73 

Indexing Fault Tolerance Configuration Parameters...............................................................74 

Chapter 6: Changing the host name of a FAST ESP node.............75 

Changing the host name in a UNIX system...........................................................................76 

Changing the host name in a Windows system......................................................................77 

Chapter 7: Changing the admin user password.............................79 

Changing the admin user password.......................................................................................80 

Chapter 8: Granting system access to non-privileged users........81 

Granting system access to non-privileged users....................................................................82 

Chapter 9: Uploading a new index profile from command-line.....83 

Uploading a new index profile using bliss...............................................................................84 

Cold update of index profile using dedicated search row.......................................................84 

Chapter 10: SNMP Agent...................................................................87 

About the SNMP Agent..........................................................................................................88 

Starting the SNMP Agent.......................................................................................................88 

SNMP MIB..............................................................................................................................88 

SNMP MIB variables..............................................................................................................93 

SNMP MIB variables - Imports....................................................................................93 

SNMP MIB variables - Type definitions........................................................................93 

SNMP MIB variables - Managed variables..................................................................93 

SNMP MIB variables - Events/Traps............................................................................99 

SNMP MIB variables - Groups....................................................................................99 

SNMP MIB variables - Compliance...........................................................................102 

Chapter 11: Monitoring tool............................................................103 

About the monitoring tool.....................................................................................................104 

Monitoring tool navigation bar selections.............................................................................104 

Associating clusters with the monitoring tool.............................................................106 

Integration with FAST ESP........................................................................................106 

Administration navigation bar selections..............................................................................106 

Adding the graphing engine.................................................................................................107 

Creating and configuring cluster views.................................................................................108 

Create cluster configuration screen options..............................................................109 

Creating a custom logger based on query count.................................................................110 

Creating a custom logger based on standard alerts............................................................112 

7


8 

Creating a schedule...................................................................................................113 

Managing global parameters................................................................................................113 

Global configuration screen selections......................................................................114 

XML data..............................................................................................................................115 

Alerting e-mail configuration information..............................................................................115 

Monitoring ESP in a redundant data center setup................................................................117 

Configuring the ESP redundant data center monitoring tool................................................117 

Rule options for rule sets in monitor.xml....................................................................118 

Chapter 12: Deleting documents from an index...........................121 

Deleting a single document from the index (option 1)..........................................................122 

Deleting a list of documents in one operation (option 2)......................................................122 

Deleting documents from an index (option 3).......................................................................123 

Chapter 13: Real-time search tuning..............................................125 

About real-time search tuning..............................................................................................126 

Real-time search tuning configuration file............................................................................126 

Optimizing for query performance........................................................................................126 

Reducing memory footprint..................................................................................................127 

Reducing index latency........................................................................................................129 

Chapter 14: Configuring logging....................................................131 

About configuring logging.....................................................................................................132 

Log levels.............................................................................................................................133 

Log destinations...................................................................................................................133 

filedestination.............................................................................................................133 

netdestination............................................................................................................134 

stdoutdestination.......................................................................................................134 

Chapter 15: nctrl tool.......................................................................135 

nctrl tool................................................................................................................................136 

Chapter 16: psctrl and doclog tools...............................................139 

psctrl tool..............................................................................................................................140 

doclog tool............................................................................................................................141 

Chapter 17: collection-admin tool..................................................143 

About collection-admin tool..................................................................................................144 

collection-admin tool usage..................................................................................................144 

Add or create collection........................................................................................................144 

Modify collection...................................................................................................................145

View collection......................................................................................................................145 

List collections......................................................................................................................146 

Delete collection...................................................................................................................146 

View pipeline stages.............................................................................................................146 

List all pipelines....................................................................................................................147 

List all modules.....................................................................................................................147 

List all datasources...............................................................................................................148 

List all clusters......................................................................................................................148 

Clear collection.....................................................................................................................148 

Global options for collection-admin......................................................................................148 

Chapter 18: indexeradmin, indexerinfo and searchadmin tools...151 

indexeradmin tool.................................................................................................................152 

indexerinfo tool.....................................................................................................................154 

searchadmin tool..................................................................................................................156 

Chapter 19: boostbt tool.................................................................159 

boostbt tool...........................................................................................................................160 

boostbt tool command line options.......................................................................................160 

Commands with arguments..................................................................................................161 

Output commands................................................................................................................161 

Administrative commands....................................................................................................161 

boostbt examples.................................................................................................................162 

Chapter 20: rc tool...........................................................................165 

rc tool....................................................................................................................................166 

Chapter 21:exportsearchprofile, importsearchprofile and view-admin tools.167 

exportsearchprofile tool........................................................................................................168 

importsearchprofile tool........................................................................................................169 

view-admin tool....................................................................................................................170 

view-admin operations...............................................................................................171 

Chapter 22: qrserver-admin tool....................................................179 

qrserver-admin tool..............................................................................................................180 

qrserver-admin tool usage....................................................................................................180 

Register operation.....................................................................................................181 

Remove operation.....................................................................................................182 

Offline operation........................................................................................................182 

Standalone operation................................................................................................182 

Down operation.........................................................................................................183 

Up operation..............................................................................................................183 

9


10 

List operation.............................................................................................................183 

Status operation........................................................................................................183 

Find operation............................................................................................................184

Chapter 

1 

Starting, stopping, suspending, and resuming 

Topics: 

• Starting 

• Stopping 

• Starting and stopping modules via 

nctrl 

This section decribes how to start, stop, suspend, and resume FAST ESP or 

individual components of FAST ESP. It also describes how to set environment 

variables to support FAST ESP.


12 

Starting 

This section describes how to set environment variables and how to start FAST ESP. 

Setting environment variables on UNIX 

To set up FAST ESP environment variables, do one of the following: 

Note: For information about which variables FAST ESP uses, consult the appropriate script. 

1. If you are using a bash or sh related shell, add the following command to the .bashrc or .profile file: 

cd /bin 

source ./setupenv.sh 

2. If you are using a csh related shell, add the following command to the .cshrc file: 

cd /bin 

source ./setupenv.csh 

Due to a limitation of the csh shell (not tcsh), if the setupenv.csh is sourced correctly from a csh, the 

following will be printed after successful sourcing: No file for $0. Note: This message can be 

disregarded, it does not mean that something is wrong. 

Setting environment variables on Windows 

The FAST ESP installer modifies the PATH variable and some other variables to support FAST ESP. You 

do not need to modify any variables on Windows before starting FAST ESP. 

Note: If you want to control FAST ESP from the command line, you must start a new command shell 

(cmd.exe) after installing FAST ESP. Command shells running before installation will not have the correct 

environment variables set. 

Starting FAST ESP on a UNIX node 

To start FAST ESP on UNIX: 

Note: On UNIX, FAST ESP is not configured to start automatically after a system reboot. If this is 

required, consult the documentation for your operating system for how to enable it. 

1. Ensure that environment variables are set properly. Refer to Setting environment variables on UNIX. 

2. Enter the following command: 

$FASTSEARCH/bin/nctrl start 

All the processes currently marked as active are started. 

3. In a multi-node installation, repeat the previous steps on all nodes, starting with the FAST ESP 

administration node. 

Starting FAST ESP on a Windows node 

FAST ESP runs as a service by default on Windows systems. 

To start FAST ESP: 

1. Perform one of the following: 

• From the taskbar, select: Start > Program > FAST ESP > FAST ESP - Start

• Use the following command: 

net start FastESPService 

• From the taskbar, select: Start > Control Panel > Administrative Tools > Services. 

Start FAST ESP. 

In the Properties dialog of the FAST ESP service you can choose whether or not this service should 

be started after system boot. 

2. In a multi-node installation, repeat the previous step on all the nodes, starting with the FAST ESP 


Starting FAST ESP with fault tolerant indexers 

In an installation with fault tolerance, the master indexer should be started first. 

If one of the backup indexers is started before the master indexer, it will try to become the master. This will 

cause substantial delays in indexing, as the new master will have to regenerate its index structure from fixml 

before it can start indexing new content. 

1. Start the row with the master indexer. 

2. Wait until the master indexers are up and have assumed the "master" role. 

Open the FAST ESP administrator interface Matching Engines page.Wait, refreshing the page periodically, 

until the columns appear. For each of the columns, open its control panel and wait until "Column role" is 

reported as "Master". 

3. Start the backup indexer rows. 

Stopping 

This section describes how to stop FAST ESP; procedures include suspending and resuming components. 

Stopping FAST ESP via sequential shutdown 

A sequential shutdown suspends or stops components individually and in a specific sequence before completing 

a shutdown. This is necessary for all components to keep the same view of the content after a restart and to 

maintain consistent behavior and data before and after the shutdown. 

Run this shutdown procedure on all nodes in your system, as the command-line tool nctrl that is used for 

shutdown works on a per node basis only. 

1. Suspend the crawler or file traverser on any node. 

2. Suspend all active connectors on any node. 

3. Wait for ongoing document processing to finish. 

4. Shut down all FAST ESP processes involved in document processing; this includes all nodes that are 

running document processors or content distributors. 

5. Shut down all QRServers on any node. 

6. Suspend indexing on all indexer nodes. Note that a suspended index will remain suspended after a 

stop/start sequence, so the indexer must be explicitly resumed. 

7. Shut down indexer and search nodes on any nodes. 

If the system has been configured with fault tolerance, the master indexer row must be shut down last. 

Refer to Stopping FAST ESP with fault tolerant indexers. 

8. Perform simple shutdown on all nodes to shut down remaining components. 


13


14 

Suspending the crawler 

Refer to the Crawler Guide for proper procedures. 

Suspending the file traverser 

The file traverser does not have an explicit suspending or resuming operation. Either a run must continue to 

completion, or if you choose to interrupt a file traverse being executed, you need to re-process the entire run 

after system restart. 

Suspending the FAST connectors 

If there are FAST connectors running within your FAST ESP installation, refer to the respective FAST connector 

documentation for proper suspend procedures. 

Suspending document processing 

After the connectors have been stopped, there may still be documents in the pipeline within the system. If 

this is the case, then suspend document processing. However, if all connectors in use keep track of the status 

of individual updates, and are able to re-process incomplete documents after restart, then you do not need 

to suspend document processing. 

1. To ensure that no more content is being processed, suspend document processing. Select the suspend 

symbol for all document processors on the System Management screen in the FAST ESP administrator 

interface. 

2. The safest strategy for a connector is to re-process all documents for which the final callback has not been 

received when the system comes back up. 

Suspending indexing on an indexer node 

Suspending indexing is useful to ensure a consistent and known state of the index on disk before backup, or 

during shutdown and restart of whole or parts of the system. 

1. Depending on your need, use one of the following indexeradmin commands: 

• To suspend processing, the persistence to disk, and the indexing of documents on all indexer nodes, 

run the following command: 

indexeradmin -a suspend 

The corresponding command for resuming indexing is 

indexeradmin -a resume 

• To stop only the indexing of documents, run the following command: 

indexeradmin -a suspendindexing 

The corresponding command for resuming indexing is 

indexeradmin -a resumeindexing 

Refer to indexeradmin tool for usage information. 

Stopping FAST ESP with fault tolerant indexers 

In an installation with fault tolerance, the master indexer should be stopped last. 

If the master indexer is stopped before backup indexers, one of the backup indexers will try to become the 

master. This will cause substantial delays, as the new master will regenerate its index structure from fixml. 

1. Stop the backup indexer rows. 

2. Wait until the backup indexers have shut down. 

Open the FAST ESP administrator interface Matching Engines page.Wait, refreshing the page periodically, 

until the backup indexers have disappeared from the display.

3. Stop the master indexer rows. 

Stopping FAST ESP via simple shutdown (UNIX) 

A simple shutdown stops a node completely without first suspending or stopping individual components. This 

procedure does not ensure consistent behavior or data after a restart of the system. It is usually helpful only 

in a testing environment where consistency is not required, or in a system where documents are regularly 

reprocessed and will eventually be re-entered and reprocessed at a later time. In such cases, the possibility 

of a few documents not being available for a limited period of time may not be as critical. 

Run this procedure on all nodes in your system, as the command-line tool nctrl that is used for shutdown 

works on a per node basis only. 

1. Ensure that environment variables are set properly. Refer to Setting environment variables on UNIX. 

2. Enter the following command: 

$FASTSEARCH/bin/nctrl stop 

All the processes currently marked as active are stopped. 

3. In a multi-node installation, repeat the previous step on all the nodes, finishing with the FAST ESP 


Stopping FAST ESP via simple shutdown (Windows) 

A simple shutdown stops a node completely without first suspending or stopping individual components. This 

procedure does not ensure consistent behavior or data after a restart of the system. It is usually helpful only 

in a testing environment where consistency is not required, or in a system where documents are regularly 

reprocessed and will eventually be re-entered and reprocessed at a later time. In such cases, the possibility 

of a few documents not being available for a limited period of time may not be as critical. 

Run this procedure on all nodes in your system: 

1. Perform one of the following: 

• From the taskbar, select: Start > Program > FAST ESP > FAST ESP - Stop 

• Use the following command: 

net stop FastESPService 

• From the taskbar, select: Start > Control Panel > Administrative Tools > Services. 

Stop FAST ESP. 

2. In a multi-node installation, repeat the previous step on all the nodes, finishing with the FAST ESP 


Starting and stopping modules via nctrl 

You can control modules from the FAST ESP administrator interface or by executing binaries associated with 

the modules.The binaries are located in $FASTSEARCH/bin directory and most of them require a proper setup 

of the environment. This section describes how to do this through the node controller nctrl command. 

Use $FASTSEARCH/bin/nctrl to start and stop running modules. 

Refer to nctrl tool for usage information. 


15


16 

Single node scenario 

Since all processes are running on a single node, a single nctrl start or nctrl stop command can be issued 

from the command-line to start or stop the system.You can view the order in which processes start by issuing 

the nctrl start command. 

Multi-node scenario 

The administration node should be the last node to go down and the first node to come up. This is because 

it runs several critical services including the license manager and configuration server. Be sure to bring the 

administration node online first in a multi-node environment. 

If you have a multi-node environment, each node should be started up one node at a time after the 

administration node is up and running. To start a multi-node system: 

1. Start the administration node (primary node). 

2. Start the indexer nodes. 

3. Start the secondary nodes. 

4. Wait until indexes are initialized. Go to Collection Overview->Edit Collection in the FAST ESP 

administrator interface and make sure that a green box with text similar to this is visible: Search is available, 

x public documents found where x is the expected number of documents in this collection. 

5. If processes do not start, make sure your environment is correct: 

• Check the environment variables and paths 

• Check the configuration files, access rights 

• Check for port conflicts

Chapter 

2 

Backing up and restoring 

Topics: 

• Determining backup and restore 

needs 

• Checklists: Backing up and 

restoring specific types of nodes 

• Backing up and restoring 

configuration data 

• Backing up and restoring content 

This section decribes how to backup and restore FAST ESP system components.


18 

Determining backup and restore needs 

Decisions on what parts of a FAST ESP system to backup are closely related to decisions made during 

deployment. Consider all servers in the FAST ESP system and whether they have configuration needs.There 

are two types of information that you need to backup and restore: configuration and content. Configuration 

information is changed by explicit action of an administrator and is stable once the system is operational. 

Content changes all the time as long as content feeding and indexing is active. 

Some steps in the following procedures are optional; you may choose to omit the step, for example to make 

a tradeoff between the amount of data to back up and the time to make a recovery. Such steps are marked 

"(optional)". 

Other steps do not apply in all circumstances, but if the conditions are met, the step must be performed. 

These steps are marked "(conditional)". 

Configuration 

This section describes the different types of FAST ESP configuration data including how the system is 

deployed, how data is handled in the index, and how incoming documents and queries are processed. 

Configuration of FAST ESP can be divided into the following subcategories: 

• Install configuration (one per FAST ESP system): 

The basic layout of the system, distribution of tasks and mapping to nodes, are created by the FAST ESP 

installer, and saved in $FASTSEARCH/InstallProfile.xml. Upon reinstallation of a system or installation 

of a copy of a system, select advanced mode and supply the saved configuration file kept from the earlier 

install. If desired, it is possible to run the installer to create a new index profile without starting the installation. 

Refer to Adding new nodes via the install profile for detailed procedures. 

• Index profile (one per cluster) 

The index profile is the top-level interface as to how documents are represented and stored on disk, and 

to what features the installation makes available. Details of the index profile features are documented in 

the Configuration Guide. 

• Collection specific configuration 

Collection specific configuration information is in use in several components: 

• Certain data sources maintain their own collection specific information. Most of the configuration 

information associated with the crawler is in the form of a collection specific configuration, such as 

what URLs to crawl or document types to accept.You can also export/import configuration per collection; 

refer to Exporting and importing collection specific crawler configuration in the Crawler Guide. For other 

connectors, consult the relevant connector guide for details. 

• The configuration server maintains a list of all existing collections in $FASTSEARCH/etc/CSConfig.xml. 

This file is updated by the configuration server upon exit. Refer to System configuration backup and 

restore. 

• Document processing configuration 

Document processing configuration consists of the configuration of pipelines, processing stages, and 

dictionaries. Pipeline and processing stage configuration is maintained by the configuration server. If 

custom dictionaries have been made, these must also be backed up, in addition to any custom document 

processor stages. 

• Query and result processing configuration: 

The QRServer configuration is similar to the document processing configuration.The query transformation 

and result processing pipeline configuration, as well as the configuration for the individual query transformers 

and result processors are maintained by the configuration server and admin server. There may be custom

dictionaries and other resources, as well as custom transformers and processors (shared libraries/DLLs) 

which must be backed up. 

• Admin server: 

The admin server maintains FAST Home and SBC, including search profiles, views, boosts, administrative 

users and their rights assignments. Dictionaries are also maintained in the admin server. This information 

is stored in the storage service database and in the resource service. 

The FAST ESP installation contains numerous configuration files which are not normally are customized, or 

are derived from other master configuration files. These files are not explicitly covered by the backup and 

restore procedures. 

The backup and restore procedures specified in this chapter are intended to cover all the configuration changes 

that can be carried out through the FAST ESP administrator interface or by using the command-line tools 

described in the Configuration Guide. By accessing the files directly, you can modify or add configuration 

files which are not covered, and thus, there may be a need for site-specific additions and adaptations to the 

backup and restore procedures. It is recommended to test site-specific backup and restore procedures to 

ensure there are no omissions. 

Content 

Content includes the different representations of the submitted and indexed data within FAST ESP. 

Content includes: 


• Content related repositories within FAST ESP. From a backup/restore perspective, the following content 

related repositories must be considered: 

• Data Sources. Refer to Backing up and restoring content connectors. 

• Index Data. Refer to Index data backup and restore. 

• Document Processing related repositories. If authority ranking based on anchor text is in use for WEB 

content (by default in the SiteSearch and NewsSearch pipelines), the data source state (e.g. Enterprise 

Crawler) should be taken backup of. This is because it is not advised to attempt a backup of the 

WebAnalyzer data store.The reason for this is that restoring such a backup will result in the WebAnalyzer 

data being out of sync with the data source (e.g. Enterprise Crawler). If the WebAnalyzer needs to be 

restored, the data should be re-fed instead. Such a re-feed can be performed without having to re-index. 

• Content being passed through the system. At the time of a backup, content processing should be 

suspended. 

Which parts of the content flow to backup is mainly related to how long it will take to restore a FAST ESP 

system to the state it was before a system failure. 

• A full re-feed may be acceptable if search down-time equals the time it takes to re-feed and re-index, and 

if the original data is available for re-feeding. This is the simplest scenario. In this case you only need to 

back up the system configuration. 

• In some cases, a full re-feed takes a much longer time than re-indexing, or the original data is not readily 

available. For such cases it may be acceptable to re-index but not re-feed. Content fed into FAST ESP 

using the content APIs or connectors will first pass through document processing, which normally does 

not involve any storing of content to disk. Then documents will be fed to the indexer, which will store it to 

disk using the FAST internal indexer XML format named fixml. By backing up the fixml data only it is 

possible to recover in reasonable time, where each indexer will perform a full re-index from the fixml. 

• High-availability cases may consider backing up the entire index data. 

Only the data that needs to be available for indexing and result processing/presentation needs to be 

represented in the fixml; FAST ESP does not store the original content. (Exceptions are the crawler and other 

similar connectors. This is only in order to be able to re-feed original content due to document processing 

changes, not in order to be able to retrieve the original document by a search application.) 

19


20 

Checklists: Backing up and restoring specific types of nodes 

This section provides checklists of the procedures to do backup and full restore of some specific types of 

nodes in common FAST ESP deployments, in order to recover from anything up to the full loss of the node. 

While some steps may be independent and interchangeable, others are dependent on being completed in 

the proper sequence. It is strongly recommended to follow the exact sequences specified. 

Important: The backup and restore procedures and some other procedures in this guide assume that 

there exists an up-to-date InstallProfile.xml describing the full FAST ESP installation, such that 

each and every node can be reinstalled from scratch using this install profile. 

Single node system backup and restore 

Procedures in this section include single node system backup and restore. 

Backing up a single node system 

1. Back up the index profile that was last uploaded. 

2. Prepare for backup. 

Refer to Preparation for backup. 

3. Back up the system configuration. 

Refer to Backing up system configuration. 

4. (Optional) Back up dictionaries. 

Refer to Backing up compiled dictionaries. 

5. Back up the resource service. 

Refer to Backing up the resource service. 

6. Back up the admin server. 

Refer to Backing up the admin server. 

7. (Conditional) Back up content connectors. 

Refer to Backing up and restoring content connectors. 

8. Back up index data. 

Refer to Backing up index data. 

9. Resume indexing. 

Restoring a single node system 

Complete this procedure if you want to restore a single node system. 

1. Verify that the install profile to be used in next step references the index profile file to be restored. 


2. Reinstall the node from the install profile. 

Refer to Reinstalling a node via the install profile. The node must be installed with the same hostname. 

3. Restore the system configuration. 

Refer to Restoring system configuration. 

4. Restore the dictionaries. 

Refer to Restoring dictionaries.

5. Start FAST ESP, then suspend indexing. 

6. Restore the resource service. 

Refer to Restoring the resource service. 

7. Restore the admin server. 

Refer to Restoring the admin server. 

8. (Conditional) Restore content connectors. 

Refer to Backing up and restoring content connectors. 

9. Restore data. 

Refer to Basic search and index recovery. 


Administration node backup and restore in a multi-node system 

Procedures in this section include administration node backup and restore. These procedures assume that 

there is a single administration node, where the configuration server, resource service and admin server are 

deployed. 

Backing up the administration node 

Perform the following steps on the administration node. 

1. Back up the index profile that was last uploaded. 

2. Prepare for backup. 


3. Back up the system configuration. 


4. Back up the resource service. 

Refer to Backing up the resource service. 

5. (Optional) Back up dictionaries. 

Refer to Backing up compiled dictionaries. 

6. Back up the admin server: 

Refer to Backing up admin server. 


Restoring the administration node 

Complete this procedure if you want to restore the administration node. 

1. Verify that the install profile to be used in next step references the index profile file to be restored. 


2. Reinstall the administration node (single node only) from the install profile. 


3. Restore the system configuration on the administration node. 

Refer to Restoring system configuration. 

4. Restore the resource service. 


5. Start the administration node. 


21


22 

6. Restore admin server. 


7. Restore the dictionaries on the administration node. 

Refer to Restoring dictionaries. 

QRServer node backup and restore 

Procedures in this section include QRServer backup and restore. 

Backing up the QRServer node 

By default, the QRServer has no local state that needs to be backed up. QRServer configuration is maintained 

in the configuration server and admin server (on the administration node). Note that the QRServer is frequently 

co-located with other services. 

1. (Conditional) Back up dictionaries which are locally installed in $FASTSEARCH/resources/dictionaries. 

These dictionaries cannot be regenerated and/or redeployed from the admin server). 

Refer to Backing up and restoring dictionaries. 

2. (Conditional, if using the Security Access Module, SAM) Back up 

$FASTSEARCH/etc/FASTSecurityPipelineConfig.xml and $FASTSEARCH/etc/FDSSM_client.pem. 

Note: FDSSM_client.pem contains a private key necessary for the secure operation of SAM and must be 

kept secret. 

3. (Conditional) Back up any other services, such as indexer or search, on the same node. 

Restoring the QRServer node 

Complete this procedure if you want to restore the QRServer. 

1. Reinstall the QRServer node (single node only) from the install profile. 

Refer to Reinstalling nodes via the install profile. The node must be installed with the same hostname. 

2. (Conditional) Restore dictionaries that were backed up from $FASTSEARCH/resources/dictionaries. 

3. (Conditional) Redeploy dictionaries that are downloaded from admin server. 


4. (Conditional) Restore $FASTSEARCH/etc/FASTSecurityPipelineConfig.xml and 

$FASTSEARCH/etc/FDSSM_client.pem. 

5. (Conditional) Restore any other services on same node. 

6. Clear the cached QRServer configuration. To do so, remove $FASTSEARCH/var/qrserver/*. 

7. Start the node. 

8. Redeploy all views on the administration node. 

view-admin -m deploy -a 

Processing server node backup and restore 

Procedures in this section include processing server backup and restore. 

Backing up the processing server node 

By default, the processing server has no local state that needs to be backed up. Processing server configuration 

is maintained in the configuration server (on the administration node). 

1. (Conditional) Back up dictionaries which are locally installed in $FASTSEARCH/resources/dictionaries. 

These dictionaries cannot be regenerated and/or redeployed from the admin server). 

Refer to Backing up and restoring dictionaries.

2. (Conditional, if using the Security Access Module, SAM) Back up 

$FASTSEARCH/etc/FASTSecurityPipelineConfig.xml and $FASTSEARCH/etc/FDSSM_client.pem. 

Note: FDSSM_client.pem contains a private key necessary for the secure operation of SAM and must be 

kept secret. 

3. (Conditional) Back up any other services, such as indexer or search, on the same node. 

Restoring the processing server node 

Complete this procedure if you want to restore the processing server. 

1. Reinstall the processing server node (single node only) from the install profile. 


2. (Conditional) Restore dictionaries that were backed up from $FASTSEARCH/resources/dictionaries. 

3. (Conditional) Redeploy dictionaries that are downloaded from admin server. 


4. (Conditional) Restore $FASTSEARCH/etc/FASTSecurityPipelineConfig.xml and 

$FASTSEARCH/etc/FDSSM_client.pem. 

5. (Conditional) Restore any other services on same node. 

6. Start the node. 

Backing up and restoring configuration data 

This section describes the procedures for backing up and restoring configuration data. Configuration data is 

relatively stable and does not change as content is fed into FAST ESP unless an administrator explicitly 

changes the configuration. The configuration data that needs to be backed up depends on the extent of 

customization of the FAST ESP installation. 

A typical FAST ESP installation includes a single administration node that hosts the configuration server, 

resource service and admin server. All master configuration data physically resides on the administration 

node and can be backed up from that node. The backup procedures in this section assume this typical 

installation. The restore procedures assume master configuration data must be restored from the backup to 

the administration node. Certain derived configuration data is physically copied to the nodes where they are 

used and is covered separately in the procedures. 

The procedures do not cover cases where the administrative services are distributed over multiple nodes, or 

with redundant administration nodes. 

System configuration backup and restore 

This section covers the most fundamental system configuration, which consists of the install profile, the index 

profile and the configuration server data. 

This configuration determines the format of the index and the document processing necessary to correctly 

populate the index fields. It also determines how the system components (including index rows and columns) 

are deployed on physical nodes. 

When restoring the system, always start with restoring this configuration to a consistent state. 


Both the document processing and QRServer configuration are covered by these procedures, except 

dictionaries and search views, which are deployed from the admin server. Refer to Admin server backup and 

restore and Backing up and restoring dictionaries. 

23


24 

Backing up system configuration 

This procedure backs up the fundamental system configuration that is needed for reinstalling any node in the 

system, as well as restoring the administration node with index, document processing and query and result 

processing configuration. 

1. Back up the following files: 

• The install profile ($FASTSEARCH/InstallProfile.xml). This file is identical on all nodes. 

• The license file (fastsearch.lic) that was specified when FAST ESP was installed. 

• (Conditional, if customized after installation.) The node controller configuration 

($FASTSEARCH/etc/NodeConf.xml) from all nodes. 

• The index profile that was last uploaded. (No fixed file name, frequently located in the directory 

$FASTSEARCH/index-profiles.) 

• (Conditional) If the index profile uses custom rank models, back them up from 

$FASTSEARCH/etc/resources/relevancy. 

These files seldom change. 

2. To prepare for restore, modify the InstallProfile.xml to reference the index profile and rank models: 

a) In the element, replace the line 

 

with 

 

 

b) If you have custom rank models, add a rank-model-filename property for each: 

 

 

. . . 

After any modifications to the install profile, it is recommended to copy the modified file to 

$FASTSEARCH/InstallProfile.xml on all nodes in the installation. In that case, the index profile and 

rank model files referenced from the install profile must also be copied. 

3. The following files/directories are maintained by the configuration server. Before backing them up, restart 

the configuration server to ensure all modifications have been flushed to disk. 

• $FASTSEARCH/etc/CSConfig.xml 

• $FASTSEARCH/etc/CSUniCfg.xml 

• $FASTSEARCH/etc/PipelineConfig.xml 

• $FASTSEARCH/etc/DefaultCrawlerConfig.xml 

• $FASTSEARCH/etc/config_data/RTSearch/webcluster/rtsearchrc.xml 

• $FASTSEARCH/etc/config_data/QRServer/webcluster/etc/qrserver/* 

• $FASTSEARCH/etc/config_data/QRServer/webcluster/preload 

(If there are other clusters than "webcluster" in the system, modify/repeat the last three entries accordingly.) 

Some of these files may be present on other nodes (i.e. nodes without a configuration server). These are 

default versions of the files left by the installer. They are not used and do not need to be backed up, as 

the components using them get them from the configuration server.

Instead of backing up individual files, it is also possible back up the entire configuration server database 

in $FASTSEARCH/etc/config_data. 

Restoring system configuration 

Use this procedure when the administration node has crashed and needs to be reconstructed. 

1. If failure was caused by a hardware problem, resolve the hardware problem. 

Note: If the host ID as seen by the license manager has changed, it is necessary to get a replacement 

for the license file. Refer to the Licensing chapter in the Installation Guide. 

2. Reinstall the node from the install profile, using the install profile that is modified to reference the index 

profile, as described in Backing up system configuration. 

Refer to Reinstalling a node via the install profile. Do not start the node. 

3. Restore the files maintained by the configuration server. 

Refer to the file list in Backing up system configuration. 

If restoring the entire $FASTSEARCH/etc/config_data directory tree instead of individual files, ensure this 

is done before starting FAST ESP with the configuration server on this node. If performed later, the restore 

might overwrite configuration changes initiated by other components. 

4. Leave node in stopped state for further restore operations. 

Some components may cache configserver-provided configurations locally. If restoring a backup that rolls 

back the system configuration to an earlier state, it may be necessary to clear those caches. 

• processing server: $FASTSEARCH/var/procserver 

• QRServer: $FASTSEARCH/var/qrserver 

Restoring the index profile 

The procedures here assume that the index profile is restored by the FAST ESP installer, by making the 

install profile reference the actual index profile as described above. 

This avoids the detour when the installer installs a default index profile, which is subsequently replaced by 

the actual index profile. If the index profile instead is restored by uploading via the FAST ESP administrator 

interface or by using bliss, as described in Uploading a new index profile using bliss, be aware that this often 

will be a cold index update (since the restored index profile is compared to the default one). This will cause 

existing indexes to be cleared. 

Backing up index configuration 

Indexer and search columns fetch most of their configuration from the configuration server, but depend on 

two local "bootstrap" configuration files for information such as where to find the configuration server. If the 

indexer or search node is to be restored by other means than by using the installer, it may be necessary to 

back up and restore these files manually. 

The configuration files are: 

$FASTSEARCH/etc/searchrc-1.xml 

$FASTSEARCH/etc/rtsplatformrc.xml 


If more than one search column is running on a node, there will be corresponding files searchrc-2.xml, 

searchrc-3.xml... for each of the additional search columns. 

These files are produced by the installer, and it will not be necessary to restore them if the indexer or search 

node is reinstalled using the Reinstalling a node via the install profile procedure. 

25


26 

Resource service backup and restore 

The resource service is a centralized repository for versioned "resources", i.e. files with arbitrary (and frequently 

quite large) content. The resource service state is updated by adding new resource versions and pruning 

away outdated versions.The procedures describe full backups of the resource service database, but it should 

be simple to adapt them to an incremental scheme since a resource version never changes. 

The clients of the resource service reference resources by (version number, resource name). It is important 

to maintain referential integrity: If you restore an older backup, clients may retain references to newer versions 

of resources that do not exist in the backup. In the worst case, such errors might pass undetected because 

the version numbers are re-used as new resources are added. Therefore, backing up and restoring the 

resource service should only be done in conjunction with backup and restore of its clients, first and foremost 

the admin server. For clients that cache resources locally (such as the QRServer), the caches may have be 

purged. 

Backing up the resource service 

Complete this procedure if you want to back up the resource service. 

1. Stop the resource service. 

2. Back up the directory: $FASTSEARCH/data/resourceservice 

3. Start the resource service. 

Restoring the resource service 

Complete this procedure if you want to restore the resource service. 

1. Stop the resource service. 

2. Restore the directory from backup: $FASTSEARCH/data/resourceservice 

3. Start the resource service. 

Backing up and restoring dictionaries 

The (editable) master dictionaries are maintained by the admin server. Before use, the dictionaries are 

compiled to a binary format (automata, .aut or .ftaut files) which supports efficient lookup. Backup and 

recovery for master dictionaries is covered by the admin server backup and restore procedures. In addition, 

compiled dictionaries may have to be backed up, if source master dictionaries are not available, or to speed 

recovery. 

Compiled dictionaries can be reconstructed from the master dictionaries as long as their type is supported 

by esp4jtool-dictman (spell check, lemmatization, synonym, and phrase dictionaries are supported) and 

corresponding master dictionaries are available. Note that such reconstruction may take time to complete, 

so it is recommended to backup the compiled versions as well. 

Deployment of compiled dictionaries is done in one of two ways: 

• Query-side dictionaries managed via SBC (view-specific synonym directories) are deployed via the resource 

service and covered via resource service backup and restore. 

• Other dictionaries (managed by esp4jtool-dictman) are placed in 

$FASTSEARCH/adminserver/webapps/adminserver/downloads/dictionaries on the admin server 

node and manually deployed to $FASTSEARCH/resources/dictionaries on the nodes with QRServers 

or processing servers. 

Some dictionaries are only available in the non-editable compiled form. These dictionaries are installed by 

the FAST ESP installer directly into $FASTSEARCH/resources/dictionaries on each node. Since these 

dictionaries do not change, there is normally no need to back them up, but be aware that customization could 

have installed additional or modified automata in these locations.

If dictionaries are manually customized and deployed according to the procedures in Configuring Linguistics 

Processing in the Configuration Guide, FAST ESP does not keep a record of which dictionaries need 

recompilation and where they are deployed. It is recommended to make scripts to perform a full recompilation 

and redeployment of all these dictionaries. Then use these scripts in steps 3 through 5 of the Restoring 

dictionaries procedure. 

Backing up compiled dictionaries 

This procedure is optional and applies if it is not feasible to regenerate compiled dictionaries after a restore. 

1. Ensure that no dictionary compilation jobs are in progress. 

2. (Optional) On the node with the admin server: Back up the content of 

$FASTSEARCH/adminserver/webapps/adminserver/downloads/dictionaries 

3. (Optional) On processing server or QRServer nodes: Back up selected automata from 

$FASTSEARCH/resources/dictionaries. 

Restoring dictionaries 

After the source dictionaries are restored in the admin server, compiled dictionaries must be regenerated. 

Any compiled dictionaries that were backed up must be restored, and the compiled dictionaries are redeployed 

to their installed locations on the processing server and QRServer nodes. 

Except for the next to last two steps, all steps are performed on the node with the admin server. 

1. (Optional) Restore the resource service. 


2. (Optional) Restore the admin server. 


3. (Optional) Restore backed up compiled dictionaries to 

$FASTSEARCH/adminserver/webapps/adminserver/downloads/dictionaries. 

4. (Conditional) Regenerate compiled dictionaries from master dictionaries: 

a) Start esp4jtool-dictman. 

b) For each dictionary to regenerate, enter the command compile. 

You receive a job number . 

c) Use jobstatus to verify that compilation has finished. 

Status changes to "ok". 

5. On the processing server and QRServer nodes, redeploy the compiled dictionaries to 

$FASTSEARCH/resources/dictionaries. 

Use esp4jtool-dictman with the download command. 

6. (Optional) On the processing server and QRServer nodes, restore any dictionaries that were backed up 

from $FASTSEARCH/resources/dictionaries. 

7. Redeploy views from the admin server. 



Admin server backup and restore 

The admin server maintains FAST Home and SBC, i.e. search profiles, views, boosts, administrative users 

and their rights assignments, as well as editable master dictionaries. 

Before backing up or restoring the admin server, all users should have finished their FAST Home or SBC 

sessions, or at least save their unfinished tasks. In particular: Dictionaries marked as "open" in running 

esp4jtool-dictman clients must be saved to make changes permanent; after saving all changes the 

esp4jtool-dictman should be closed. Refer to Configuring linguistics processing in the Configuration Guide. 

27


28 

The admin server database references resources in the resource service database. Therefore, the admin 

server and the resource service should be backed up and restored together. Refer to Resource service backup 

and restore. 

Backing up the admin server 

Complete this procedure if you want to back up the admin server. Use the backupadminserver utility to back 

up the admin server databases. 

1. Run the following command to produce a snapshot of all the admin server databases into the specified 

directory (for example, backupdir): 

cobra $FASTSEARCH/esp4j/bin/backupadminserver.py -m backup -d backupdir 

backupadminserver suspends the admin server while completing the backup. 

2. (Conditional) Back up $FASTSEARCH/etc/esp4j.properties. 

This file contains the admin user password and should be kept secret. 

Restoring the admin server 

Complete this procedure if you want to restore the admin server. 

1. Run the following command to restore the database snapshot produced by backupadminserver in a 

specified directory (for example, backupdir): 

cobra $FASTSEARCH/esp4j/bin/backupadminserver.py -m restore -d backupdir 

FAST ESP should be running. The admin server will temporarily be stopped and restarted once the 

snapshot is restored. 

2. (Conditional) Restore $FASTSEARCH/etc/esp4j.properties. 

This file contains the admin user password and should be kept secret. Remove all access permissions to 

it for any other user but the FAST ESP service user. 

3. (Conditional) If the admin server password has changed, restore $FASTSEARCH/etc/esp4j.properties 

on all nodes running admin server clients. 

Refer to Changing the admin user password. 

Backing up and restoring content 

This section describes the procedures for backing up and restoring content. New and changed content flows 

continuously into the system, and a content backup is a snapshot of this content flow. 

Preparation for backup 

Before performing a full backup you should suspend content feeding and indexing in order to ensure a 

consistent state in your FAST ESP system. Refer to Consistency of index data 

1. Before backup or restore: 

a) Stop crawler, file traverser and connector processes. 

Refer to the respective sections under Stopping FAST ESP via sequential shutdown. 

b) Give the document processing time to finish outstanding batches submitted by the connector(s). 

c) Suspend indexing on all working indexing nodes using the indexeradmin suspend command. 


2. After backup or restore:

a) Resume indexing. 


b) Restart crawler, file traverser and connectors. 

It is mandatory to suspend indexing. 

Stopping content feeding is not strictly necessary, as the backup will get a consistent snapshot of the index 

even if content feeding is left running. But as long as indexing is suspended, content feeding and document 

processing will be blocked. In most cases, this will cause problems such as timeouts, and it is strongly 

recommended to suspend content feeding as well. 

Also consider that connectors may have states that must be consistent with the index, so must be backed 

up together with the index in a quiescent state. Refer to Backing up and restoring content connectors. 

Backing up and restoring content connectors 

Some FAST connectors store information related to content flow. This information has to be backed up along 

with content in other parts of the system. 

The content connectors will operate in one of the following modes: 

• A simple batch retrieve connector will retrieve all or a subset of documents from a content repository and 

submit the documents unconditionally to FAST ESP. Such a simple connector will not involve any content 

store within the connector, so there is no need for backing up content. 

The file traverser is used in this way. 

• Certain connectors use a database to store change control information, for instance represented as a 

hash value for the document. By using this hash database it is possible to determine whether or not a 

retrieved document is changed, so only changed documents are forwarded to FAST ESP. 

Backing up the change control database is optional, and will only avoid the need for re-submitting all 

documents to FAST ESP after a recovery. The need for such a backup depends on how long it will take 

to re-fetch all content from the content source. 

• Some connectors subscribe to changed information that is available from the content source, and only 

retrieve new and changed documents. In this case the changed information does not need to be stored 

in the connector, and there will normally not be any need for backing up content information. 

• Some connectors (in particular, the crawler) can store original documents fetched from the content sources 

in a private store, in order to support re-feed of documents upon document processing changes. This 

includes pipeline changes and new anchor texts detected. In this case, the connector's document store 

should be backed up. 

Be aware of consistency requirements between the state maintained by the connector and the state of the 

index.The conservative approach is to stop both feeding and indexing, as described in Preparation for backup, 

before backing up both the full index and the connector state. 

Refer to the connector documentation for specific backup requirements and procedures. 


Index data backup and restore 

Search results are generated from the index data. Index data consists of two parts, the "fixml store" containing 

the processed documents and the index, where the actual search is taking place. The index can always be 

regenerated from the fixml, but since this process can be time-consuming, there is a tradeoff of backup volume 

versus recovery time when determining whether to back up the full index. 

Content fed into FAST ESP using the content APIs or a connector will first pass through document processing, 

which normally does not involve any storing of content to disk.Then documents will be fed to the index, which 

will store them to disk using FAST fixml. What parts of the original document that are maintained in the fixml 

format is largely dependent on the index profile and the document processing configuration. 

29


30 

By default, data is located in $FASTSEARCH/data . If FAST ESP was installed with a different location for the 

data directory, then use that location. (If in doubt, check the entries for indexDir and fixmlDir in 

$FASTSEARCH/etc/rtsplatformrc.xml . The alternate data location applies to the data/data_fixml , 

data/data_index and data/ftStorage directories. Other services place data in other subdirectories of 

$FASTSEARCH/data , and do not use the alternate data location.) 

The fixml representation of content is stored in $FASTSEARCH/data/data_fixml . 

The $FASTSEARCH/data/data_index directory contains the generated index used by the search engine.This 

index can be regenerated completely from the data in the $FASTSEARCH/data/data_fixml directory. Normally, 

backup of data_index is not necessary, but there could be cases where the system parameters are such that 

the time to recover is shorter when the generated index can be restored. Whether or not this is the case 

depends on the size of the resulting index, characteristics of the system such as available network bandwidth, 

backup medium and I/O performance. 

In some cases, rebuilding the index from fixml is faster than restoring it from backup. Another possible 

advantage of re-indexing (dependent on the data, and the pattern of which it was fed into the system originally) 

is that a re-constructed index may be slightly smaller and better distributed, since an older index structure 

may have accumulated disk space overhead due to fragmentation over time. 

The index is distributed over rows and columns on a per document basis. Each column contains a distinct 

partition of the index. The full index consists of the union of data from each column. All rows within a column 

contain identical data. Within a column, one or more of the row nodes are indexer nodes, the remaining rows 

are pure search nodes which do not build indexes but get their data from an indexer node. To back up the 

entire index, back up one indexer node within each column. 

Consistency of index data 

To eliminate potential index consistency problems, the conservative procedure is to perform backup and 

restore in a state when all content feeding and indexing is suspended. All columns of the index are backed 

up at the same time. Less conservative procedures cause less feeding downtime, if the potential inconsistencies 

can be tolerated or avoided by other means. 

1. Stop content feeding and suspend indexing. 

2. Back up or restore all indexer nodes in the same session. 

3. Suspend indexing until all nodes are finished. 

This conservative procedure is the basis for all the procedures described in Backing up and restoring content. 

This causes fairly long downtimes for content feeding (and for search, in the case of restore), and may not 

be acceptable in all cases. It is possible to perform backup and restore node by node if the potential 

inconsistencies this may cause are acceptable to the application. When designing such custom backup and 

restore procedures, use the following guidelines: 

• As long as indexing is suspended on the node being backed up or restored, the index is internally consistent. 

The only issue is whether it is consistent with the indexes on other nodes and with the feeding application. 

• Unless custom routing is implemented, FAST ESP does not move a document from one column to another. 

Thus a document will not disappear or be duplicated if a column is restored from a backup reflecting an 

earlier system state than another column. 

• Do not restore a single indexer node independently of other indexers in the same column. Otherwise, a 

query will have nondeterministic behavior depending on which row it happens to be directed to. Moreover, 

the difference will persist indefinitely (if there are some documents that are not updated by feeding). Refer 

to Index recovery with redundant indexers. 

• Restoring an earlier backup of the index inevitably causes updates (add, delete, modify) to be lost, even 

though they have been acknowledged as successful to the connector or feeding application.The connector 

may have a state database which must be consistent with the index: In that case, that database should 

be restored from a backup consistent with the index backup, or there might be a way to force a re-feed to 

synchronize the index and connector state. Refer to Backing up and restoring content connectors.

• Two or more updates may have been fed in the same batch, with the expectation that all become visible 

to search at the same time. But if routed to different columns and one of the columns is restored from 

backup, one or more of the updates may be lost while the others still are visible. 

Backing up index data 

Index data is backed up with indexing suspended, by making a snapshot copy of the fixml and index data 

directories on the indexer node. 

Backup of indexer nodes in a FAST ESP system is performed with the following procedure applied to each 

of the nodes to back up. To back up an entire indexing structure, steps 2 - 4 of this procedure must be applied 

to one of the nodes in each column in the system. 

1. Suspend content feeding and indexing. 


2. (Required) Back up the entire contents of the $FASTSEARCH/data/data_fixml directory structure. 

3. (Conditional) In a fault-tolerant (high-availability) configuration, back up 

$FASTSEARCH/data/ftStorage/indexer_checkpoint.txt 

4. (Optional) Back up the entire contents of the $FASTSEARCH/data/data_index directory structure. 

5. Resume content feeding and indexing. 


Basic search and index recovery 

This procedure describes how to recover a single index or search node from an available backup. 

This scenario is applicable in a situation where a complete search and index column needs to be reconstructed 

after a fatal hardware crash. For mission critical deployments, where multiple indexers are configured per 

column, this scenario would only be a last resort in the case of multiple, simultaneous node failures. 

1. If failure was caused by a hardware problem, resolve the hardware problem. 

2. Re-install the node from the install profile. Do not start the services on the recovered node. 

Refer to Reinstalling a node via the install profile. 

3. Restore the latest index data backup on the re-installed node, by restoring the directory: 

$FASTSEARCH/data/data_fixml 

4. Remove $FASTSEARCH/data/data_fixml/hashFlushed 

5. (Conditional) In a fault-tolerant configuration, restore $FASTSEARCH/ftStorage/indexer_checkpoint.txt 

6. (Optional) Restore the actual index directory: $FASTSEARCH/data/data_index 

7. (Conditional) Restore from backup any other services that were running on this node. 

8. Start the necessary FAST ESP services on the recovered node. 

9. Refresh the FAST ESP administrator interface System Overview page. The new node should appear on 

the screen. 

10. Resume indexing. Restart content feeding. 

11. If the node also was running a QRServer, redeploy views. 

On the administration node, perform the command: 


12. As documents may have been lost from the time of the last backup, it may be necessary to perform a full 

re-feed or re-crawl from the content source. 

Refer to the Crawler Guide or other relevant connector guides. 


31


32 

Index recovery with redundant indexers 

This procedure describes how to recover from a search and index node failure in a system with multiple 

indexer nodes in each column. One of the nodes in a column needs to be reconstructed after a fatal hardware 

crash, and one or more backup nodes exist within the column (multiple rows). To handle a single node error 

it is not required to have a full backup of the node. 

Back up the InstallProfile.xml file. The install profile is automatically generated during FAST ESP 

installation. 

You should also perform a regular backup of one row node within each column regularly, in case of a fatal 

hardware error involving more than one node. 

1. If the system is configured with fault-tolerance, and provided the recovery does not take too long, the 

recovery can be made without suspending content feeding. The other nodes in the column continue to 

perform indexing, and the recovered node catches up with them when it comes up. In that case, the 

procedure is: 

a) Re-install the failed node as described in steps 1 and 2 in Basic search and index recovery. 

b) Select another (intact) node in the column and suspend indexing on that node only: 

indexeradmin suspend 

c) Make a backup of the index on that node, as described in steps 2 to 4 in Backing up index data. 

d) Resume indexing on that node: 

indexeradmin resume 

e) Recover the failed node as described in steps 3 to 10 in Basic search and index recovery, using the 

backup from step c. 

f) When the indexer on the recovered node is started, it should re-do the operations the other indexers 

in the column have performed since the backup was made. Verify that by examining the log: In the 

FAST ESP administrator interface, go to Matching Engines, select the recovered node and go to 

Status Details. When Active sessions becomes 0, the re-do process has finished. Check the logs 

for error or warning messages from the indexer. 

2. If steps c to e above take too long time (exceeding the fault tolerance window), some operations will be 

lost and cannot be re-done, and the log will show error messages to that effect. In that case, you could 

try to reduce the feeding load (in effect, extending the fault tolerance window in time) and retry the 

procedure. Alternately, fall back to the safe procedure, which is to suspend content feeding during recovery. 

This procedure also applies to a system without fault tolerance. 

a) Re-install the failed node as described in steps 1 to 2 in Basic search and index recovery. 

b) Suspend content feeding and indexing. Refer to Preparation for backup. 

c) Make a backup of the index of one of the other (intact) nodes in the column, as described in steps 2 

to 4 in Backing up index data. Do not resume indexing and feeding yet. 

d) Recover the failed node as described in steps 3 to 10 in Basic search and index recovery, using the 

backup from step c. 

e) Resume indexing and content feeding.

Chapter 

3 

Applying runtime deployment changes 

Topics: 

• Installing nodes from the install 

profile 

• Increasing indexing and search 

capacity 

• Adding a top-level dispatcher 

• Adding a new QRServer 

• Increasing document feed and 

processing capacity 

• Adding a content distributor 

• Adding an indexing dispatcher 

• Multi-node license file 

considerations 

• Increasing indexing and search 

capacity with independent search 

nodes 

This section provides procedures for applying deployment changes in a running 

FAST ESP system.


34 

Installing nodes from the install profile 

The InstallProfile.xml file describes the configuration of the FAST ESP system as created by the installer, 

and is first defined when installing FAST ESP. It can be used to perform a full re-install of the entire system, 

as well as for configuration purposes when adding a single node to to the system, or to re-install a single 

node in a multi-node installation. 

Refer to the Installation Guide, sections Standard Installation and Silent Installation for detailed procedures. 

Reinstalling a node via the install profile 

This section describes how to reinstall a single node in a multi-node installation.This may become necessary 

in the event of a hardware failure or if the machine is to be upgraded. 

1. Keep a copy of the original InstallProfile.xml file. This file can be found in the root directory of the FAST 

ESP installation on every node. 

2. Start the installer on the node that is to be reinstalled. 

3. Select advanced mode. 

4. Select the install profile. 

5. Select install on local node only and run local tests only. 

6. Complete installation. 

Note that it may be necessary to restore any custom configuration and data on the reinstalled node. Refer 

to the Backing up and restoring for more information. 

Adding new nodes via the install profile 

This section describes how to modify the install profile and use the installer in advanced mode to add new 

nodes to the system. 

Refer to the FAST ESP Installation Guide for more information on the InstallProfile.xml file format and 

how to complete the installation. 

This procedure is a substep of several other procedures. Some steps are specialized to add specific FAST 

ESP services to the new node. The point of specialization is the step marked "(Conditional)" in the following 

procedure. 

1. Modify the InstallProfile.xml to include the new nodes. 

This can be done in two alternate ways: 

• Edit the install profile using the installer: 

1. Run the installer on one of the existing nodes, and enter the multi node installer GUI. 

2. Select Installation Overview -> Install profile, and open the install profile of the existing system. 

The install profile is located in the root directory of the FAST ESP installation. 

3. Add the new node(s) (hosts) to the install profile: Select Hosts and then Add host. See also Adding 

Hosts in the Installation Guide. 

4. (Conditional) Select Services and assign the services to run on the new node. 

5. Save the edited configuration as a new install profile ( Installation Overview -> Install profile), 

and exit the installer. 

• Edit the install profile in a text or XML editor: 

1. Open the install profile on one of the existing nodes. 

2. Copy one of the sections in the list. Give each new host a unique id, set 

the correct hostname and check whether any of the other attributes need to be changed.

Example: 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

3. (Conditional) Edit or create service-specific entries and assign the services to run on the new node(s) 

by setting their host-ref attribute. 

2. Perform the following steps for each of the new nodes. 

a) Copy the modified install profile to the new node. 

b) Start the installer on the new node. 

c) Select advanced mode. 

d) Select the edited install profile. 

e) Select install on local node only and run local tests only. 

f) Complete installation. 

You may still have to change the configuration of the other nodes to integrate the new node into the installation. 

How to do this depends on what components are running on the new node. Refer to the applicable sections 

in this chapter for more information. 

After modifying the install profile, it is recommended to copy the modified install profile to 

$FASTSEARCH/InstallProfile.xml on all nodes in the installation, to ensure they all are equal and use the 

current version to correctly describe the system. 

Increasing indexing and search capacity 


Indexing and search capacity and fault tolerance is increased by adding new indexer or search nodes. 

• To increase search capacity or search fault tolerance, add a new row of search nodes. Refer to Adding 

a new dedicated search row. (You may also have to add a new QRServer. Refer to Adding a new 

QRServer.) 

• To increase indexing capacity (feed rate or document volume), add a new column. Refer to Adding an 

indexer column. Usually, indexing capacity has to be increased in step with the capacity on the input side; 

refer to Increasing document feed and processing capacity. 

• To increase indexing fault tolerance, add a new indexer row (or add indexers to an existing search row). 

Refer to Adding a new indexer row. 

If you are changing from a single node to a multi-row or multi-column system, you have to add a top-level 

dispatcher. Refer to Adding a top-level dispatcher. 

35


36 

Adding new search or index nodes via the install profile 

This chapter describes how to edit the InstallProfile.xml file to add new search and indexer nodes. 

The procedure applies both when adding a new column or adding a new row. Refer to applicable sections in 

this chapter for instructions on how to integrate the new node in the installation. 

1. Add the new index or search nodes. Refer to Adding new nodes via the install profile for the general 

procedure to add nodes. 

2. Add the new nodes in the section. Copy an existing entry and 

change the “id” and “hostname”. 

 

 

 

 

 

 

 

 

 

 

a) If you add a combined search/index node the properties “search” and “index” should be set to “true”. 

b) If you add a dedicated indexer node the property “search” should be set to “false”. 

c) If you add a dedicated search node the property “index” should be set to “false”. 

If you add a column make sure that the new column has the same number of search nodes and indexers 

as the existing columns, e.g. both “search” and “index” need to be set to “true” for the same number of 

nodes. 

3. Add the new nodes in the section. 

a) If you are adding a new column, copy an existing entry and change the id 

and id-ref: 

 

 

 

 

 

 

 

 

 

 

 

 

 

a) If you are adding a new row, copy an existing entry and change the id-ref: 

 

 

 

 

 

 

 

 

 

 

 

4. Save the InstallProfile.xml file. 

5. Complete the installation of the new nodes, using the edited install profile.

Using row and column IDs for search engine nodes 

The search nodes are organized in rows and columns. This applies to rows having an indexer, not dedicated 

search rows. Throughout this guide several configuration files refer to row IDs and column IDs. 

• The IDs are numbered from 0-n, where 'n+1' is number of columns/rows. 

• The first column installed will have ID 0, the next will have ID 1, etc. 

• The first row within a column will have ID 0, the next will have ID 1, etc. 

The $FASTSEARCH/etc/CSConfig.xml file defines the node configuration and is located on the Configuration 

Server node. The row/column IDs are used as follows: 

 

 

 

 

 

 

 

 

partition_id = Column ID 

rowid = Row ID 

node01.yourdomain.com is the first row within column 0. 

node02.yourdomain.com and node03.yourdomain.com are the additional rows within column 0. 

CSConfig.xml attributes 

The CSConfig.xml file defines the node configuration and is located on the Configuration Server node. 

$FASTSEARCH/etc/CSConfig.xml Attributes 

Attribute 

host 

port 

mode 

partition_id 

ft_mode 

docapiport 

rowid 

Description 

Fully qualified host name for the new node. 

Port number used for this search node. Set to same value as for existing entry. 

Set to same value as for existing entry. 

Set partition_id to one more than the largest partition_id already in the file. 

Set to same value as for existing entry. 

Currently unused, keep setting at 15500. 

The second row (first entry) is rowid=“1”. If two or more rows are already defined, 

set the rowid to the next available value (“2” if this is the third row defined for the column). 

$FASTSEARCH/etc/CSConfig.xml Attributes 

Attribute 

host 

port 

docapiport 

rowid 

Description 

Fully qualified host name for the new node. 


Port number used for this search node. Normally this should be set to the same value as the 

corresponding attribute in the entry (or existing entry). 

Port number to the internal search node document API. Normally this should be set to the same 

value as the corresponding attribute in the entry (or existing entry). 

The second row (first entry) is rowid=“1”. If two or more rows are already defined, set 

the rowid to the next available value (“2” if this is the third row defined for the column). 

37


38 

CSConfig.xml file 

The row/column IDs are used as follows: 

 

 

 

 

 

 

 

 

where 

• partition_id = Column ID 

• rowid = Row ID 

• node01.yourdomain.com is the first row within column 0 

• node02.yourdomain.com and node03.yourdomain.com are the additional rows within 

column 0 

Adding a new dedicated search row 

This procedure describes how to add new dedicated search row nodes to the system. 

When adding a new search row, you need to set up one extra search node per column in the installation. 

If you wish to utilize existing nodes currently not running search, you will need to replace step 2 in the procedure 

with editing the $FASTSEARCH/etc/NodeConf.xml file to add "search-1" to the list of active components, 

and run $FASTSEARCH/bin/nctrl reloadcfg to activate it. 

To add a new search row, complete the following procedure: 

1. If the current system is configured with only one search node, you need to configure a top-level dispatcher. 

2. Install the new node(s) needed for the new search row. 

Refer to Adding new search or index nodes via the install profile. 

3. Check the $FASTSEARCH/etc/searchrc-1.xml configuration file to verify that the following parameters 

on the new node are correct: 

• Indexpath 

• Hostname and port 

• Clustername 

• Columnid 

• Index copy setup. Set copyindex to "false" or "on_demand" if this node may also contain an indexer 

at some point. Disable index copying if the index is shared through a SAN or NAS. Disable 

socketcopyindex if multicasting is used. 

If this is an existing node that has not been running search, $FASTSEARCH/etc/searchrc-1.xml may be 

missing. In that case, copy the file from another search node in the system and modify the above-mentioned 

parameters. 

4. Ensure that the license file is configured properly on the node(s). Refer to Multi-node license file 

considerations. 

5. Start the new search node(s). 

When the search nodes are started, they will automatically contact the indexers running on the column, 

and request a copy of the latest index set if necessary. Search will start on the extra search row once the

index set is up to date. The top-level dispatcher will automatically be reconfigured to add the extra search 

row. 

6. To verify that the search rows have been correctly added, select Matching Engines. Select the applicable 

indexer node host name from the Host column. Check to make sure the new search controller is listed in 

the Search Controllers section. 

Note that this process may take some time to complete since the new search row will have to copy the 

index. The Status of the search row is dependent on how you have configured the new search row. If the 

Status is ok, then the search row is actively up and running and has completed the copying process. If 

the Status is down, then the search row is still functioning properly, but it has not yet received a current 

version of the index set. 

Upon completion, the screen information will be similar to the following: 


7. To verify that the node is recognized, the new node should be listed under System Management in the 

FAST ESP administrator interface and registered with the Config Server. 

Adding an indexer column 

If the number of documents served is increasing beyond recommended sizing you need to increase the 

content volume capacity by adding new indexer columns. This procedure can also be used for performance 

optimization in larger installations. It is recommended that you contact FAST Solution Services before deploying 

such a configuration. 

To ensure a balanced system, new columns should be similar to existing columns, both in terms of hardware, 

and in terms of roles.When a new indexer column has been added, search nodes matching the search nodes 

on the existing columns must also be added. If indexer fault-tolerance is used, the new column must have 

the same number of indexers as the other columns. Refer to Adding a new dedicated search row . 

This procedure allows the system to serve searches uninterruptedly while the deployment changes are 

applied. However, the base procedure does not ensure data consistency for a short period of time, when the 

new column gets available and some of the old search columns still serve searches from their old indexes. 

This may lead to some searches resulting in duplicates. Once all old nodes are done with the resetindex 

operation and have switched to the new index, data consistency is restored. 

If data consistency at any given time is critical for your implementation, there are two options: 

39


40 

• If you have one or more dedicated search rows, you can set search standalone using these rows while 

the index is reorganized by this procedure. 

• Stop all search processes while the indexers are reindexing and restart the search processes when the 

new index is available. 

In the following procedure, "old indexers" refer to all indexers that are present in the system before the 

expansion; "new indexers" refer to newly added indexer columns. To add a new indexer column, complete 

the following procedure: 

1. Install the search and index nodes for the new column. Refer to Adding new search or index nodes via 

the install profile. Start the new nodes. 

The search and indexer processes will remain idle until the new column has been configured in the 

configuration server. 

2. (Conditional) If the system does not have a top-level dispatcher, that is it has a single search process), 

then add one. Refer to Adding a top-level dispatcher . 

3. Instantiate a document processor stage called MigrationNoChanges from the default stage Migration 

with the ConfigFile parameter set to Migration/MigrationConfig-NoChanges.xml 

4. Insert the stage MigrationNoChanges in the pipeline(s) you are using for the collections that you are about 

to refeed. The stage should be inserted anywhere before the RTSOutput stage 

The MigrationNoChanges stage makes sure that all FiXML data fed by the fixmlfeeder bypasses all 

the other stages in the pipeline, so there will be no extra processing if you have a complex pipeline. 

5. Stop content feeding for the installation. Refer to the respective FAST connector documentation. 

6. (Optional) Set search standalone. 

Refer to Switching search to standalone mode . 

7. Suspend indexing for all indexers. 

On one node, for example, the admin node: 


8. Stop all indexing dispatchers. 

On all nodes with indexing dispatchers: 

$FASTSEARCH/bin/nctrl stop indexingdispatcher 

9. Configure the new column in the configuration server. 

Perform the following steps on the admin node: 

a) Stop the configuration server. 

$FASTSEARCH/bin/nctrl stop configserver 

b) Edit the $FASTSEARCH/etc/CSConfig.xml file. Add a new column entry in the 

section by copying the existing entry. Give the new column a partition_id one higher than 

the previous highest ID. 

The following example shows a cluster entry where a second column is added. The added column 

entry is highlighted: 

 

 

 

 

 

 

 

 

c) Start the configuration server. 

$FASTSEARCH/bin/nctrl start configserver 

10. (Re)start the new indexer(s). After verifying that the indexers have started and generated an empty index, 

suspend indexing. 

On all new indexer nodes: 

$FASTSEARCH/bin/nctrl restart indexer 

After indexer(s) have started, on one node: 


11. (Re)start the new search processes. 

On all nodes in the new column: 

$FASTSEARCH/bin/nctrl restart search-1 

12. (Conditional) If not in standalone mode, restart the top-level dispatcher process. 

On the node with the top-level dispatcher: 

$FASTSEARCH/bin/nctrl restart topfdispatch 

If search was set standalone according to Switching search to standalone mode , 

topfdispatch-standalone is running instead of topfdispatch , and there is no need to restart it. 

13. Stop all indexers (nctrl stop indexer). At this point, no indexers should be running in the system. 

To verify, select System Management in the FAST ESP administrator interface. In the System Monitor 

section, the indexer ( RI ) and indexingdispatcher ( Ind ) will show a stopped status. The display will be 

similar to the following shown for node02.boston.fast.com: 

14. Move the data/data_fixml folder on all the old indexers to data/data_fixml_old . 

15. Start all indexers (old and new). 

On all indexer nodes: 

$FASTSEARCH/bin/nctrl start indexer 

16. Start the indexingdispatcher(s). 

On the node(s) with an indexing dispatcher: 

$FASTSEARCH/bin/nctrl start indexingdispatcher 


At this point, the index reconfiguration is completed. All indexers should be up and running, but in the 

suspended state, with 0 documents. Search is served from the old indexes. 

With indexing is suspended, the indexers will collect the FiXML documents that go into each index partition, 

but not generate the index itself. Firstly, this reduces the load on the servers, since there is no point in 

generating new indexes until this procedure is completed. Secondly, the indexes that the search processes 

are serving are left untouched. 

41


42 

17. Restart content feeding by starting all connectors. 

You may want to postpone this step until the very end to make it easier to verify that the refeeding of the 

old FiXML is completed successfully. However, this is not necessary. 

18. Refeed the old documents using the fixmlfeeder tool. On each old indexer node, run the following 

command to refeed the documents: 

fixmlfeeder -i $FASTSEARCH/data/data_fixml_old 

Refer to Migrating FiXML data in the Migration Guide . 

If you need assistance in completing this step, contact FAST Technical Support. 

You can safely feed new data into the collection at the same time that you are re-feeding the old FiXML 

19. After verifying that the content has been successfully fed into the installation, and is distributed across all 

indexer columns, delete the data/data_fixml_old directory. 

At this point, the data/data_fixml directories of the indexers contain the desired new index state, but 

data/data_index contains the old index and is still used to serve search. 

20. (Optional) If the temporary inconsistencies while switching to the new index cannot be tolerated, stop 

search now. 

21. Reset the index on all indexer nodes. 

On one node, e.g. the admin node: 

indexeradmin -a resetindex 

Until all indexers have completed this step and the search nodes have switched to the new indexes, search 

results may be inconsistent. 

22. (Conditional) If search was previously stopped, restart search. 

23. (Conditional) If search was set standalone, restore search to normal operations. 

Refer to Restoring normal operations from standalone search . 

Adding an indexer column (for archive indexing) 

Follow this procedure to add a new indexer column in a system implementing archive indexing. 

Refer to the procedure describing how to add an indexer column in a regular system for more context 

information. 

1. Install the search and index nodes for the new column. Refer to Adding new search or index nodes via 

the install profile. Start the new nodes. 

The search and indexer processes will remain idle until the new column has been configured in the 


2. (Conditional) If the system does not have a top-level dispatcher, that is it has a single search process), 

then add one. Refer to Adding a top-level dispatcher . 

3. Stop content feeding for the installation. Refer to the respective FAST connector documentation. 

4. Stop all indexing dispatchers. 

On all nodes with indexing dispatchers: 

$FASTSEARCH/bin/nctrl stop indexingdispatcher 

5. Configure the new column in the configuration server. 

Perform the following steps on the admin node: 

a) Stop the configuration server. 


b) Edit the $FASTSEARCH/etc/CSConfig.xml file.

Add a new column entry in the section by copying the existing entry. 

Give the new column a partition_id one higher than the previous highest ID. 

The following example shows a cluster entry where a second column is added. 

 

 

 

 

 

 

 

 

 

c) Start the configuration server. 


6. (Re)start the new indexer(s) nad verify that the indexers have started and generated an empty index. 

On all new indexer nodes: 

$FASTSEARCH/bin/nctrl restart indexer 

7. (Re)start the new search processes. 

On all nodes in the new column: 


8. (Conditional) If not in standalone mode, restart the top-level dispatcher process. 

On the node with the top-level dispatcher: 

$FASTSEARCH/bin/nctrl restart topfdispatch 

If search was set standalone according to Switching search to standalone mode , 

topfdispatch-standalone is running instead of topfdispatch , and there is no need to restart it. 

9. Start the indexingdispatcher(s). 

On the node(s) with an indexing dispatcher: 

$FASTSEARCH/bin/nctrl start indexingdispatcher 

At this point, the index reconfiguration is completed. All indexers should be up and running. Search is 

served from the old indexes. 

With indexing is suspended, the indexers will collect the FiXML documents that go into each index partition, 

but not generate the index itself. Firstly, this reduces the load on the servers, since there is no point in 

generating new indexes until this procedure is completed. Secondly, the indexes that the search processes 

are serving are left untouched. 

10. Restart content feeding by starting all connectors. 


43


44 

Adding an indexer row 

To increase indexer fault tolerance, one or more backup indexer rows can be added (by converting dedicated 

search rows to combined indexer/search rows, or by installing new dedicated indexer rows). This permits 

feeding and indexing to continue in case of indexer node failures. 

The procedure describes how to add a backup indexer row operating in fault tolerance mode. The load on a 

backup indexer is much less than on a master indexer, so it is common to use a shared search/index row as 

the backup indexer row. In that case, take an existing dedicated search row and convert it to a combined 

search/index row. Otherwise, install new nodes for a dedicated backup indexer row. 

In this procedure, "old" indexers refer to the indexer instances that were configured in the system before the 

start of the procedure, while "new" indexers are the indexer instances added in the procedure. 

1. (Optional) If making a new dedicated indexer row, install new indexer nodes, as described in Adding new 

search or index nodes via the install profile . Don't start the nodes yet. 

2. Set all search processes to standalone mode. 

See Configuring search nodes to standalone mode , repeating the procedure for all nodes with search 

processes. 

After this step, all search processes are in standalone mode. This means they continue running from the 

existing index, but detached from the indexers and the configuration server, and they will not be disturbed 

by the configuration changes throughout the rest of the procedure.You can verify that the search processes 

are standalone by visiting each of the indexers in the Matching engines page of the admin GUI and see 

that the "Search controllers" section at the bottom of the Real Time Search Engine status page is empty. 

3. Stop content feeding. 

Refer to connector documentation. See also Preparation for backup . 

4. Suspend indexing: 


5. Stop all (old) indexers: On all nodes in the existing indexer row(s), perform: 

$FASTSEARCH/bin/nctrl stop indexer 

At this point, no indexer processes (whether new or existing) should be running. You can verify this from 

the System Management page of the admin GUI: All indicators in the column labelled "RI" should be 

yellow. 

6. On the admin node: update the cluster configuration data. 

a) Stop the configuration server by running the following command: 


b) Edit the $FASTSEARCH/etc/CSConfig.xml file. Add a new row in the section by adding 

a entry in each entry. Enable fault-tolerance mode for each column by setting 

ft_mode ="1". If two or more rows are already defined you can copy the existing entry and 

only modify the rowid attribute. 

The following example shows a cluster entry where one backup row is added. The added/changed 

data is highlighted: 

 

 

 

 

 

 

 

c) Start the configuration server by running the following command: 


7. For each indexer column, copy the $FASTSEARCH/data/data_fixml folder from the old indexer node to 

the new indexer node. 

ESP may have be configured with an alternate location for the data_fixml directory. In that case, use 

the alternate location instead of the default one. See Index data backup and restore . 

8. Start all the old indexers: 

$FASTSEARCH/bin/nctrl start indexer 

Make sure all the old indexers are up before proceeding. They should come up in the role "Master" with 

0 backups. The new indexers must not be started before the old indexers have successfully started. 

9. On each of the nodes with new indexers, edit $FASTSEARCH/etc/NodeConf.xml to start the indexer. Add 

an entry for "indexer" to the section. Then use the node controller reloadcfg to reload the 

changed NodeConf.xml . 

NodeConf.xml example: 

 

indexer 

search-1 

. . . 

After editing NodeConf.xml : 

$FASTSEARCH/bin/nctrl reloadcfg 

If you have installed new indexer nodes by modifying the install profile, the NodeConf.xml changes have 

already been done by the installer, and this step can be omitted. Instead, just start the new indexer nodes: 

$FASTSEARCH/bin/nctrl start 

At the completion of this step, all the new indexers should be running. In the admin GUI, verify that all 

indexers are visible in the Matching engines and that the old indexer in each column is designated as 

"master" and the new as "backup".Wait for the new indexers to complete initial indexing before proceeding 

to the next step. 

10. Resume indexing: 


11. Resume content feeding. 

12. Disable standalone mode for all search processes. In $FASTSEARCH/etc/searchrc-1.xml , remove the 

standalone parameter that was set in the first step. In addition, for all search processes on the same 

node as an indexer, set copyindex to "on_demand". 

To minimize search downtime, restart one search row at a time. 

Example searchrc-1.xml after editing: 

 

 


45


46 

 

Then restart search-1 using: 


Adding a top-level dispatcher 

Top-level dispatchers must be present in any system that has or is to have more than one search node. The 

installer will automatically configure a top-level dispatcher when installing a multiple search node system. If 

you want to upgrade a single search node system to a multiple search node system, you must add a top-level 

dispatcher. 

A top-level dispatcher also works with only a single node, but adds a small amount of overhead to the system. 

Consequently, you may add a top-level dispatcher even if you only have a single node system to prepare for 

future extensions of the system. 

The top-level dispatcher typically runs on the same node as the QRServer. 

If you have multiple QRServer nodes (QR), you need to add a top-level dispatcher (TLD) to each them as 

shown in the figure. 

Figure 1: Top-Level dispatcher in single and multi-node configurations

1. In the $FASTSEARCH/etc folder, create a file called searchrc-dispatch.xml. If there are existing 

top-level dispatchers in the installation, this file can be copied from one of them, and edited. If not, it should 

look like this: 

 

 

 

2. Enable the top-level dispatcher in the $FASTSEARCH/etc/NodeConf.xml file for the top-level dispatcher 

node. 

a) Add the top-level dispatcher process configuration. 

A configuration for the top-level dispatcher may already be included in NodeConf.xml. It should be 

configured as follows: 

 

 

 

fsearchctrl 

--root=$FASTSEARCH --configfile=searchrc-dispatch.xml -ORBendPointNo 

Listen giop:tcp::$PORT -ORBendPointNoPublish giop:tcp::$PORT 

 

5 

 

 

5 

kill 

 

topfdispatch.scrap 

 

Replace with the fully qualified hostname of the local host. 

b) Add the top-level dispatcher to the list of active components in the node. 

In the section: 

 

... 

 

add the following line: 

topfdispatch 

The top-level dispatcher should come before the QRServer in the list. 

3. Load the new node controller configuration: 


4. Start the new top-level dispatcher: 

$FASTSEARCH/bin/nctrl start topfdispatch 

5. Edit the $FASTSEARCH/etc/qrserver/webcluster.spec file to reflect the new top-level dispatch host: 

Replace the existing port number, which is the port number of the single search node, by the port number 

of the new top-level dispatcher. This port number is calculated based on the following formula: 

+ 2150 


with representing the baseport number you indicated during the initial system 

installation. By default, this number is 13000. 

47


48 

If you specified a different cluster name during installation, the name of the file that must be changed uses 

this cluster name. 

6. Restart the QRServer by running the following command: 

$FASTSEARCH/bin/nctrl restart qrserver 

7. To verify that the setup is correct, select System Management in the FAST ESP administrator interface. 

The new node should have a Top-level Dispatcher (RTS) present in the System Monitor area of the 

screen. 

Now any query is directed to the top-level dispatcher, which will automatically adapt to search node layout 

changes. 

searchrc-dispatch.xml attributes 

searchrc-dispatch.xml Attributes 

searchrc-dispatch.xml 

Attribute 

hostname 

baseport 

clustername 

subsystem 

mode 

debuglog 

usestrictbind 

Default 

15150 

webcluster 

indexing 

dispatch 

false 

false 

Adding a new QRServer 

Description 

The fully qualified hostname of the top-level dispatcher node. 

The number of the baseport for this component (not the FAST ESP 

installation baseport). Determine the component baseport number using 

the following formula: 

+ 2150 

Note that the default baseport number of the FAST ESP installation is 

13000. 

The name of the cluster the top-level dispatcher is to be assigned to. 

Name of subsystem. 

The role of the component (search node or dispatcher). 

For internal debugging only. 

Use the strictbind for all server sockets for the dispatcher. 

The following procedure describes how to add a new QRServer on an existing node within your installation, 

such as a newly installed second search row node. 

If you want to add a QRServer on a dedicated machine, refer to Adding new nodes via the install profile for 

information on how to add a node to your installation. 

1. Copy the contents of the $FASTSEARCH/etc/qrserver directory from one of the existing QRServer nodes 

to the same location on the new node. For example: 

scp -r $FASTSEARCH/etc/qrserver @qrserver2.example.com:$FASTSEARCH/etc

2. The $FASTSEARCH/etc/NodeConf.xml file usually contains a process section for the QRServer (), but it may not be complete.The file should be on every new node.The parameters 

subsection has the following format: 

-n [NAME] -R fds/resourceservice/0 -d [CLUSTER] -P $PORT 

-C $FASTSEARCH -c qrserver/qrserverrc -L file:stdout -ORBendPointNoListen giop:tcp: 

[LOCALHOST] :$PORT3 -ORBendPointNoPublish giop:tcp::$PORT3 

where: 

[NAME] is a symbolic, unique name of the QRServer 

[CLUSTER] is the search cluster name, usually webcluster 

[LOCALHOST] is the fully qualified hostname of the host running the new QRServer 

For example: 

 

 

qrserver 

-n qrserver2 -R fds/resourceservice/0 -d webcluster –P $PORT -C 

$FASTSEARCH 

-c qrserver/qrserverrc -L file:stdout -ORBendPointNoListen 

giop:tcp:qrserver2.example.com:$PORT3 

-ORBendPointNoPublish giop:tcp::$PORT3 

 

 

qrserver.scrap 

 

3. If you are adding a QRServer to an existing node, edit the $FASTSEARCH/etc/NodeConf.xml file to add 

the QRServer to the list of active components on the node. In the section: 

 

... 

 

add the following line: 

qrserver 

There is no required order for adding this component into the list. 

4. After editing the $FASTSEARCH/etc/NodeConf.xml file, run: 


5. Add a top-level dispatcher every time you add a QRServer, on the same node. It is possible to run multiple 

QRServers using a single top-level dispatcher, but this is not recommended. Refer to Adding a top-level 

dispatcher. 

6. Start the QRServer: 

$FASTSEARCH/bin/nctrl start qrserver 

7. Add the new QRServer to the sfe.qrservers property of 

$FASTSEARCH/adminserver/webapps/sfe/WEB-INF/classes/sfe.properties on the administration 

node. This is a comma-separated list if there are multiple QRServers. Each QRServer is specified in the 

form 

[HOST]:[PORT] 

where 

[HOST] is the hostname of the QRServer 

[PORT] is the http port. 

8. Restart the admin server and wait until it is fully up and running: 

$FASTSEARCH/bin/nctrl restart adminserver 


49


50 

9. Using the qrserver-admin command, register the new QRServer with the admin server. 

qrserver-admin -m add -n fds/searchservice/qrserver2 -o qrserver2.example.com -p 15100 

-c webcluster 

Note that qrserver2 used in this example must be the same as [NAME] in step 2. 

10. To verify that the setup is correct, select System Management in the FAST ESP administrator interface. 

The new node should have a QRServer (QR) and Top-level Dispatcher (RTS) present in the System 

Monitor area of the screen. 

The new QRServer will only automatically receive queries if the load balancing option is used in the Search 

API, or if it is added to a load-balancer with the existing query servers. 

Increasing document feed and processing capacity 

In order to increase content feed and document processing capacity you may need to perform one of the 

following tasks. 

Each crawler instance will crawl a set of web pages, as determined by its collection and sub collection 

configuration (start URIs and include/exclude filters). If possible, it is recommended to split up the total set 

of web pages into disjoint collections in such a way that each of them can be handled by a single node crawler 

instance. 

There are two ways to add a single node crawler: 

• add a crawler on a existing node, using the FAST ESP administrator interface 

• install a new crawler node by modifying the InstallProfile.xml file 

In large-scale installations, the set of web pages reachable from one start URI may be too large for a single 

node crawler, or there could be several start URIs which can not be separated in distinct collections because 

their reachable sets partially overlap. In these cases, it is necessary to have a single crawler instance distributed 

over multiple nodes.To install and configure a multi-node crawler, see Large Scale XML Crawler Configuration 

in the Enterprise Crawler Guide. 

Document processing pipelines are run in processor servers. Each processor server instance is single-threaded. 

For maximum CPU utilization, in particular with multiple CPU or multiple kernel hosts, it is recommended to 

run multiple processor server instances on each server node. There are two ways to add processor servers: 

• add a processor server instance on existing node, using the FAST ESP administrator interface 

• install new a processor server node by modifying the InstallProfile.xml file 

Adding a crawler on an existing node via FAST ESP administrator interface 

This procedure adds a new single node crawler instance on an existing node, using the FAST ESP administrator 

interface. 

The original installation can be a single node installation or a multiple node installation. It is only possible to 

have a single crawler instance on each node. 

1. Verify that the administration node is up and running. 

2. Select System Management from the FAST ESP administrator interface and click the name of the host 

where you want to add the crawler. 

3. Select Add Crawler. 

Adding a crawler node via install profile 

This procedure installs a new node with a new single node crawler.

1. Add the new document processor node. Refer to Adding new nodes via the install profile for the general 

procedure to add a node. 

2. Specialize the add new node procedure by assigning a new crawler service to the added node. 

• If using the installer GUI, assign and configure a "Crawler" service on the Services panel. 

• If editing the install profile directly, add a new entry in the section of 

the install profile, referencing the new node. 

For example: 

 

 

 

 

 

 

This example assumes that you have set up a host with id1 as identifier in the 

section of the InstallProfile.xml, and crawler2 is a unique name 

identifying the crawler. 

3. Complete the installation of the new node, using the edited install profile. 

Adding a processor server on an existing node via FAST ESP administrator interface 

This procedure adds a new processor server instance on an existing node, using the the FAST ESP 

administrator interface. 

The procedure is the same regardless of whether: 

• the FAST ESP installation is single or multiple node, and 

• there is already a processor server on the node where the new processor server instance is to be added. 

1. Verify that the administration node is up and running. 

2. Select System Management from the FAST ESP administrator interface and click the name of the host 

where you want to add the processor server instance. 

3. Select Add Doc. Processor. 

Adding a document processor node via install profile 

This procedure installs a new document processor node. 

1. Add the new document processor node. Refer to Adding new nodes via the install profile for the general 

procedure to add a node. 

2. Specialize the add new node procedure by assigning the document processor service to the added node. 

• If using the installer GUI, assign the "Document Processor" service on the Services panel. 

• If editing the install profile directly, add a new entry in the 

section of the install profile, referencing the new node. 

For example: 

 

 

 

 


This example assumes that one has set up a host with id1 as identifier in the section of 

the InstallProfile.xml, and dp2 is a unique name identifying the document processor. 

51


52 

Instead of adding the new document processor server by editing the install profile before installation, it is 

also possible to add a new node without a processor server and add the processor server to it afterwards, 

using the procedure Adding a processor server on an existing node via FAST ESP administrator interface. 

3. Complete the installation of the new node, using the edited install profile. 

Adding a content distributor 

For load balancing or improved fault tolerance in content feeding, content distributors can be added. 

1. (Optional) Install a new node for the content distributor. Refer to Adding new nodes via the install profile. 

Before installing, edit the install profile: Add a entry to the 

section. The new entry must have a unique ID and reference the added 

node. One of the content distributors should have the master property set to "true"; for the others is should 

be "false". For example, assuming the new node was labeled "node1": 

 

 

 

 

 

 

 

 

2. (Conditional) If adding a content distributor on an existing node, edit $FASTSEARCH/etc/NodeConf.xml 

to start the process: Add an entry for the content distributor in the section. Edit the process 

command line parameters of the process: Check the entries for the other content distributor(s) 

in the system. Select a unique ID (--id) for the new content distributor and set the --next-subsystem 

parameter to the same as for the other content distributors (usually 

"esp/clusters/%CLUSTER%/indexing/dispatcher"). 

NodeConf.xml example (edited or added text in bold): 

. . . 

 

contentdistributor 

indexer 

. . . 

 

 

contentdistributor 

 

--next-subsystem=esp/clusters/%CLUSTER%/indexing/dispatcher 

--id=1 

--hostname=node02.yourdomain.com 

... 

 

. . . 

After editing NodeConf.xml: 

If the node is running, tell the node controller to reload its configuration: 


If the node is not already running, start it: 

$FASTSEARCH/bin/nctrl start

If a new content distributor was added by the installer, the NodeConf.xml will already have been properly 

set up on the new node, and this step can be omitted. 

3. Add the new content distributor to $FASTSEARCH/etc/contentdistributor.cfg. Copy the modified file 

to all nodes in the system (not just those running content distributors). 

The crawler and file traverser locate the content distributors by reading this (local) file. If the content 

distributor was added by the installer, contentdistributor.cfg will have been updated on the new node 

only: Copy this file to the other nodes. 

contentdistributor.cfg example: 

node01.yourdomain.com;16100 

node02.yourdomain.com;16100 

4. Restart crawler(s) and file traverser(s), to make the new content distributor configuration take effect. 

5. Reconfigure and restart other content feeding clients. Refer to the documentation for the connectors in 

use to determine how to configure an additional content distributor. 

6. (Optional) To verify that failover to the new content distributor is working, shut down the old content 

distributor and check that content processing continues. (Existing sessions to the content distributor that 

were shut down will fail, but new sessions should be established to the new one.) 

Adding an indexing dispatcher 

To improve fault tolerance, extra indexing dispatchers can be added. 

The new indexing dispatcher can be added to an existing node, or a new node with an indexing dispatcher 

can be added using the installer. 

1. (Optional) Install a new node. Refer to Adding new nodes via the install profile. Start the node. 

Before installing, edit the install profile: 

Add an entry to the (top-level) section. The new 

entry must have a unique ID and reference the added node. 

Also, assign the new indexing dispatcher to the search cluster: Add an entry to 

the section of the cluster, referencing the ID of the new indexing dispatcher. 

Example, assuming the new node was labeled "node1": 

. . . 

 

 

 

 

. . . 

 

. . . 

 

 

 

 

 

. . . 


2. (Conditional) If adding an indexing dispatcher on an existing node, edit $FASTSEARCH/etc/NodeConf.xml 

to start the process: Add an entry for the indexing dispatcher to the section. 

53


54 

NodeConf.xml example (edited or added text in bold): 

. . . 

 

indexingdispatcher 

indexer 

. . . 

After editing NodeConf.xml: 


If a new indexing dispatcher node was added by the installer, the NodeConf.xml will already have been 

properly set up on the new node, and this step can be omitted. 

3. (Optional) To verify that failover to the new indexing dispatcher works, stop the old indexing dispatcher 

and check that indexing continues. (Existing content feeding sessions should recover.) 

Multi-node license file considerations 

In a distributed configuration the license manager uses the master license file located on the host where the 

license manager is running. It is this master license file that includes the detailed feature and capacity license 

entries. 

The license manager host is normally the same as the host where the configuration server is running. Each 

of the other nodes in the system must also have a license file, but this is only used in order to tell the processes 

where to find the license manager. 

On all nodes other than the administration node, the license file only needs the two first lines in the license 

file: 

SERVER ANY 

USE_SERVER 

The license file is located on each node at: $FASTSEARCH/etc/fastsearch.lic 

If another non-administration node is available, copy the license file from this node. Otherwise the license 

file from the administration node can also be used, although the actual FEATURE lines in the license file are 

not relevant in this case (always verified by the license manager on the administration node). 

Also ensure that the environment variables are properly set. The LM_LICENSE_FILE environment variable 

must point to the installed license file, $FASTSEARCH/etc/fastsearch.lic. 

Increasing indexing and search capacity with independent search nodes 

Indexing and search capacity issues may arise if a system is configured to run search and indexing on the 

same node. This can be alleviated by setting up independent search nodes. This section describes how to 

move existing services to different nodes. 

Important: The backup and restore procedures and some other procedures in this guide assume that 

there exists an up-to-date InstallProfile.xml describing the full FAST ESP installation, such that 

each and every node can be reinstalled from scratch using this install profile. The procedures in this 

section move services by modifying configuration files directly, without involving the install profile. You 

should modify your saved InstallProfile.xml to reflect these changes: In the entries for moved search, 

QR server or topfdispatch processes, update the host-ref attribute with the new location for the process.

Moving RTS 

The following procedure describes how to move RTS from an existing node to a new node. 

1. Copy the contents of rtsplatformcache.xml, rtsplatformrc.xml, and searchrc-1.xml from 

$FASTSEARCH/etc on one of the existing RTS nodes to the same directory on the new node. For example: 

scp $FASTSEARCH/etc/searchrc-1.xml @new.server.com:/$FASTSEARCH/etc 

2. Edit the rtsplatformrc.xml file so that the hostname is that of the new node. For example: 

 

 

 

3. Edit the searchrc-1.xml file so that the hostname is identical to that of new node. Change the copyindex 

field to on_demand. For example: 

 

 

 

4. On the existing search node, remove search-1 from in $FASTSEARCH/etc/NodeConf.xml. 

5. Stop all indexers. 

6. On the existing search node, stop search-1: 

$FASTSEARCH/bin/nctrl stop search-1 

7. On the new search node, add search-1 to the in $FASTSEARCH/etc/NodeConf.xml. 

search-1 


8. Add the RTS process description. For example: 


 

 

fsearchctrl 

--root=$FASTSEARCH 

--configfile=searchrc-1.xml 

-ORBendPointNoListen giop:tcp:NEW.SERVER.COM:$PORT 

-ORBendPointNoPublish giop:tcp::$PORT 

 

 

5 

 

 

55


56 

120 

kill 

 

searchctrl/search-1.scrap 

 

9. On the new node, run: 


10. On new node, run: 

$FASTSEARCH/bin/nctrl start search-1 

11. Start all indexers. 

12. Verify that the index starts copying data to new search node. Check the partition status under the Matching 

Engines tab of the FAST ESP Administrator Interface to make sure the partition status is copying. 

Moving a QRServer 

The following procedure describes how to move a QRServer from an existing node to a new node. 

1. Copy the contents of $FASTSEARCH/etc/qrserver from the existing QRServer node to the same location 

on the new node. For example: 

scp -r $FASTSEARCH/etc/qrserver @qrserver2.example.com:$FASTSEARCH/etc 

2. On the new node, add the following line to in $FASTSEARCH/etc/NodeConf.xml: 

qrserver 


3. Add the QRServer process description. For example: 

 

 

qrserver 

-n qrserver2 -R fds/resourceservice/0 -d webcluster -P $PORT 

-C 

$FASTSEARCH -c qrserver/qrserverrc -L file:stdout 

-ORBendPointNoListen giop:tcp:qrserver2.example.com:$PORT3 

-ORBendPointNoPublish giop:tcp::$PORT3 

 

 

 

qrserver.scrap 

 

4. On the existing QRServer node, stop qrserver: 

$FASTSEARCH/bin/nctrl stop qrserver 

5. Remove the existing QRServer: 

$FASTSEARCH/bin/qrserver-admin -m remove -n fds/searchservice/qrservername -o 

qrserver.example.com 

If necessary, use $FASTSEARCH/bin/qrserver-admin -m list to find the ns name and address. 

6. On the new QRServer node, run: 


$FASTSEARCH/bin/nctrl start qrserver

7. On the administration node, add the new QRServer to sfe.properties and remove the existing QRServer 

from same file: 

vim $FASTSEARCH/adminserver/webapps/sfe/WEB-INF/classes/sfe.properties 

8. On the administration node, edit the file $FASTSEARCH/etc/esp4j.properties so that the hostname and 

port number of the new QRServers are correct. 

# SFE properties 

# 

sfe.qrservers=NEW.HOSTNAME1.COM:PORT,NEW.HOSTNAME2.COM:PORT 

sfe.relative.resource.path= 

The sfe.properties file is a comma-separated list if there are multiple QRServers. 

9. Restart the admin server, either from the Administrator Interface or on administration node using: 

$FASTSEARCH/bin/nctrl restart adminserver 

10. Register the new QRServer with the admin server: 

$FASTSEARCH/bin/qrserver-admin -m add -n fds/searchservice/qrservername -p 25100 -c 

webcluster -o qrserver.example.com 

11. On the administration node, stop the configserver: 

$FASTSEARCH/bin/ncrtl stop configserver 

12. On the administration node, edit the $FASTSEARCH/etc/CSConfig.xml file with the new locations of the 

QRServers. 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

13. On the administration node, start the configserver: 

$FASTSEARCH/bin/ncrtl start configserver 


14. Verify that the QRServer is up and running. Check for error or warning messages in the Logs tab of the 

FAST ESP Administrator Interface to make sure search is available under the Search View tab. 

Moving an RTS top dispatcher 

The following procedure describes how to move the RTS top dispatcher (searchctrl) from an existing node 

to a new node. 

1. Copy the contents of searchrc-dispatch.xml from $FASTSEARCH/etc on one of the existing nodes to 

the same directory on the new node. For example: 

scp $FASTSEARCH/etc/searchrc-dispatch.xml @new.server.com:/$FASTSEARCH/etc 

57


58 

2. Edit the searchrc-dispatch.xml file so that the hostname is that of the new node. For example: 

 

 

 

3. On the existing node, stop topfdispatch-searchctrl: 

$FASTSEARCH/bin/nctrl stop topfdispatch-searchctrl 

4. On the existing node, remove topfdispatch-searchctrl from in 

$FASTSEARCH/etc/NodeConf.xml. 

5. On the new node, add topfdispatch-searchctrl to the 

in$FASTSEARCH/etc/NodeConf.xml. 

topfdispatch-searchctrl 


6. Add the RTS Top Dispatcher process information. For example: 

 

 

fsearchctrl 

--root=$FASTSEARCH 

--configfile=searchrc-dispatch.xml 

-ORBendPointNoListen giop:tcp:NEW.SERVER.COM:$PORT 


 

 

5 

 

 

5 

kill 

 

topfdispatch-searchctrl.scrap 

 

7. On the new node, run: 


8. On new node, run: 

$FASTSEARCH/bin/nctrl start topfdispatch-searchctrl 

9. Verify that the RTS top dispatcher is up and running. Check for error or warning messages in the Logs 

tab of the FAST ESP Administrator Interface to make sure search is available under the Search View 

tab.

Chapter 

4 

Running FAST ESP in standalone mode 

Topics: 

• About standalone mode 

• Creating standalone versions of 

processes 

• Switching search to standalone 

mode 

• Restoring normal operations from 

standalone search 

This section describes procedures for working in standalone mode.


60 

About standalone mode 

In standalone mode, search and dispatch nodes operate independently of the configuration server and the 

indexers. Standalone mode is used to permit search to operate from a subset of the nodes, holding an old 

snapshot of the index, while other nodes are upgraded or reconfigured. Standalone mode can also be used 

in other cases which require manual configuration of search. 

Standalone mode applies to the following ESP components: 

• Search nodes, which are "locked" to a particular snapshot of the index. By deregistering from the 

configuration server and indexers, a standalone search node neither receives index updates, nor is it 

affected by cluster configuration changes. 

• (Top-level) dispatcher nodes, when standalone, are also isolated from configuration changes and can be 

manually configured. 

Search nodes by themselves can be set standalone, to get control of when they are updated with new indexes 

or to copy new indexes to them manually. 

To have operational search while performing system upgrades or testing, a subset of the search nodes (one 

or more rows) are selected and set standalone. Then the top-level dispatchers are also set standalone and 

manually configured to dispatch search to those search nodes only. 

To implement custom load balancing schemes, create multiple QRServers (e.g. one per row) with an external 

load balancer to distribute load between them. Each QRServer has a distinct standalone top-level dispatcher 

which is manually configured to dispatch search to a subset of the search nodes (e.g. one row). 

Creating standalone versions of processes 

This section describes how to add standalone versions of processes. 

Creating a standalone QRServer 

This procedure adds a standalone QRServer process as an alternative for an existing process, to permit 

switching between the original and the standalone version. 

Before carrying out this procedure: On the admin-node, ensure that the following is set (adding or changing 

values): 

httpdisable = 1 

allowpartialresults = 1 

datasearch.autoreconfigure.level = 0 

in $FASTSEARCH/etc/config_data/QRServer/webcluster/etc/qrserver/qrserverrc and 

$FASTSEARCH/etc/bliss-templates/QRServerRCWriter/qrserverrc.tmpl 

Note that changing the autoconfigure level will require the QRServers to be restarted manually whenever the 

index-profile has been changed, also during hot-update. 

Repeat this procedure on each node with a QRServer in the system. (QRServer and the associated top-level 

dispatcher usually run on the same node, so you may want to carry out the NodeConf.xml changes described 

here at the same time as those in Creating a standalone top-level dispatcher.) 

1. Add a process entry for the standalone QRServer qrserver-standalone in 

$FASTSEARCH/etc/NodeConf.xml: Copy the entry for the original QRServer process and add a nonexistent 

host:port combination as the configuration server address (-D dummy:1000) in its section. 

 

 

qrserver

-D dummy:1000 -n query1_15100 -R esp/admin/resourceservice/0 

-d webcluster -P $PORT -C $FASTSEARCH -c qrserver/qrserverrc -L file:stdout 

-ORBendPointNoListen 

giop:tcp:query1.search.net:$PORT3 -ORBendPointNoPublish giop:tcp::$PORT3 

 

 

qrserver-standalone.scrap 

 

2. Add the qrserver-standalone process in the section in the top of NodeConf.xml: 

... 

 

... 

qrserver 

qrserver-standalone 

... 

 

3. To avoid automatic startup of the qrserver-standalone process when reloading the configuration, set 

it to stopped. 

nctrl stop qrserver-standalone 

To avoid accidentally starting the standalone version of the process, you might in addition comment out 

its entry in the section when not in active use: 

4. Reload the node controller configuration. 

nctrl reloadcfg 


After completion of this procedure, there are two defined processes qrserver and qrserver-standalone 

with equivalent configurations, except the latter operates in standalone mode. To switch between normal and 

standalone mode, it is sufficient to stop the running one and start the alternate one. (The processes cannot 

run at the same time.) 

Creating a standalone search process 

This procedure adds a standalone search process as an alternative for an existing process, to permit switching 

between the original and the standalone version. 

Repeat this procedure on each search node in the system. There is normally at most one search controller 

process per node, which is named "search-1", and with a master configuration file 

$FASTSEARCH/etc/searchrc-1.xml. If there are multiple search controller processes per node, the process 

name and the corresponding configuration file name will have suffixes "-2", "-3" etc. Repeat this procedure 

for each search controller process, as required. 

1. In $FASTSEARCH/etc/NodeConf.xml, make a copy of the search-1 process entry, name it 

search-1-standalone and add the options --no-indexer --no-fds to its parameters section. 

 

 

fsearchctrl 

--no-indexer --no-fds --root=$FASTSEARCH --configfile=searchrc-1.xml 

-ORBendPointNoListen giop:tcp:search00.search.net:$PORT -ORBendPointNoPublish 

giop:tcp::$PORT 

 

 

 

 

120 

kill 

 

search-1-standalone.scrap 

 

50 

61


62 

 

 

2. Add the search-1-standalone process in the section in the top of NodeConf.xml: 

... 

 

... 

search-1 

search-1-standalone 

... 

 

3. To avoid automatic startup of the search-1-standalone process when reloading the configuration, set 

it to stopped. 

nctrl stop search-1-standalone 





After completion of this procedure, there are two defined processes search-1 and search-1-standalone 

with equivalent configurations, except the latter operates in standalone mode. To switch between normal and 

standalone mode, it is sufficient to stop the running one and start the alternate one. (The processes cannot 

run at the same time.) 

Creating a standalone top-level dispatcher 

This procedure adds a standalone top-level dispatcher process as an alternative for an existing process, to 

permit switching between the original and the standalone version. 

Repeat this procedure on each node with a top-level dispatcher in the system. (QRServer and the associated 

top-level dispatcher usually run on the same node, so you may want to carry out the NodeConf.xml changes 

described here at the same time as those in Creating a standalone QRServer.) 

1. When running on standard configuration, topfdispatch will automatically distribute queries to all running 

search nodes. To be able to control which search nodes are in production, add an explicit engines map 

configuration file to the parameters of the topfdispatch process in $FASTSEARCH/etc/NodeConf.xml 

using the --enginefile=$FASTSEARCH/etc/enginesrc-tld option. 

 

 

fsearchctrl 

--root=$FASTSEARCH --enginefile=$FASTSEARCH/etc/enginesrc-tld 

--configfile=searchrc-dispatch.xml -ORBendPointNoListen giop:tcp:query1.search.net:$PORT 


 

5 

 

topfdispatch.scrap 

 

2. Create the engines map configuration file $FASTSEARCH/etc/enginesrc-tld listing the search node the 

current query server should query. The file can be based on the file 

$FASTSEARCH/var/searchctrl/etc/enginesrc-tld_0 which is auto-generated when running a standard 

installation. 

Example, from a system with 2 columns, 1 row: 

dataset 0 

partitions 2

efcost 1 

firstpart 0 

rowbits 2 

engine search0-0.search.net:15700 0 0 1 


3. To avoid deployment of new configuration during index-profile update, we also need a standalone version 

of the topfdispatch process. Add a process entry for topfdispatch-standalone process to 

NodeConf.xml by copying the entry for the original topfdispatch and adding the options --no-indexer 

--no-fds in the section. 

 

 

fsearchctrl 

--root=$FASTSEARCH --enginefile=$FASTSEARCH/etc/enginesrc-tld 

--configfile= 

searchrc-dispatch.xml -ORBendPointNoListen giop:tcp:query1.search.net:$PORT 

-ORBendPointNoPublish 

giop:tcp::$PORT --no-indexer --no-fds 

 

5 

 

topfdispatch-standalone.scrap 

 

4. Add the topfdispatch-standalone process in the section in the top of NodeConf.xml: 

... 

 

... 

topfdispatch 

topfdispatch-standalone 

... 

 

5. To avoid automatic startup of the topfdispatch-standalone process when reloading the configuration, 

set it to stopped. 

nctrl stop topfdispatch-standalone 





After completion of this procedure, there are two defined processes topfdispatch and 

topfdispatch-standalone with equivalent configurations, except the latter operates in standalone mode. 

To switch between normal and standalone mode, it is sufficient to stop the running one and start the alternate 

one. (The processes cannot run at the same time.) 

The search nodes this top-level dispatcher will distribute queries to is controlled by editing the engines map 

file $FASTSEARCH/etc/enginesrc-tld. (After editing, restart the topfdispatch or topfdispatch-standalone 

process. Alternately, perform an HTTP GET on the URL 

http://server:15151/admin?command=reload_partmap, where server is the node where the top-level 

dispatcher is running. The port number 15151 assumes the default ESP base port of 13000.) 

Switching search to standalone mode 


Switches all processes involved in search to standalone mode, to allow search to operate continuously from 

an existing index, unaffected by index updates and configuration changes. 

63


64 

This procedure assumes the system has one or more dedicated search rows which can be made standalone 

in order to continue to serve search during index reconfiguration. Determine which search rows are used to 

serve each QRServer, in order to balance the load over the search rows in standalone mode. 

Before carrying out this procedure the first time, create the standalone process entries for all QRServers, 

top-level dispatchers and search processes, refer to Creating a standalone QRServer, Creating a standalone 

top-level dispatcher and Creating a standalone search process. 

1. To avoid deployment of new views during index-profile update, the QRServers needs to be put into 

standalone mode with regards to the admin server. This is done with the qrserver-admin command. 

a) List existing QRServer, using the qrserver-admin -m list command. 

[search@admin ~]$ qrserver-admin -m list 

# ns name address cluster state 

- ------------------------------ -------------------------------- ---------- ----- 

1 fds/searchservice/query1_15100 query1.search.net:15100 webcluster up 

2 fds/searchservice/query2_15100 query2.search.net:15100 webcluster up 

b) Set all QRServer into standalone mode, using the qrserver-admin -m standalone command. 

This means that views will not be deployed to the QRServer. 

[search@admin ~]$ qrserver-admin -m standalone -n fds/searchservice/query1_15100 

Taking qrserver fds/searchservice/query1_15100 (query1.search.net:15100) standalone. 

10.435 [--------------------------------------] 100% Successfully redeployed all 

views. 

[search@tsiadm1.bos3 ~]$ qrserver-admin -m standalone -n 

fds/searchservice/query2_15100 

Taking qrserver fds/searchservice/query2_15100 (query2.search.net:15100) standalone. 

10.396 [--------------------------------------] 100% Successfully redeployed all 

views. 

2. Switch each pair of QRServer/top-level dispatcher to standalone mode. 

The QRServer and associated top-level dispatcher usually run on the same node. Perform this step on 

each node with a QRServer and top-level dispatcher. 

a) Start the standalone version of the QRServer process. 

nctrl stop qrserver 

nctrl start qrserver-standalone 

b) Point the top-level dispatcher to the dedicated search row(s), by editing 

$FASTSEARCH/etc/enginesrc-tld. 

There should be an "engine" line for each search process this dispatcher is going to use. The format 

of the line is 

engine host:port column row refcost 

For instance: 

dataset 0 

partitions 2 

refcost 1 

firstpart 0 

rowbits 2 



c) Start the standalone version of the top-level dispatcher process. 

nctrl stop topfdispatch 

nctrl start topfdispatch-standalone 

d) Make sure the QRServer and top-level dispatcher are up before going on to the next node.

This can be done by checking http:// node :15151/status (where node is the current node). 

3. Set all search nodes standalone. 

On each search node (in the dedicated search rows), perform 

nctrl stop search-1 

nctrl start search-1-standalone 

Restoring normal operations from standalone search 

This procedure restores normal operation after search has been in standalone mode, phasing in a new or 

reorganized index and optionally a new configuration. 

This procedure is applied in a state when search is in standalone mode (refer to Switching search to standalone 

mode), being served from dedicated, standalone search rows. Indexing has been reconfigured, and the index 

row(s) have completed generating an updated index, which is not yet visible on the search row(s). By taking 

the search rows out of standalone mode one by one, they will be updated with the new configuration and the 

new index, without interruption of overall search availability or inconsistencies in search results. 

Each iteration of the procedure takes one standalone QRServer, switches it over from standalone search 

rows to updated rows and restores it to normal operation from standalone mode.This is freeing up standalone 

search rows, which are no longer used by any QRServer. The rows can then be restored to normal operation, 

updated with the new index and used to restore another QRServer on the next iteration. 

1. Select one of the remaining standalone QRServers to restore to normal operation. 

This QRServer will be unavailable for search until this iteration of the procedure has completed. It is 

important to complete the procedure on one QRServer before starting on the next, ensuring that search 

is up at all times. 

a) Stop topfdispatch-standalone: 

nctrl stop topfdispatch-standalone 

The QRServer will then close port 15100 for incoming queries. 

b) Point topfdispatch to one of the updated rows, for instance the index row, by editing 

$FASTSEARCH/etc/enginesrc-tld. 

Example: 

dataset 0 

partitions 2 

refcost 1 

firstpart 0 

rowbits 2 



If the cluster was reconfigured with a different number of columns, remember to set partitions to the 

new number of columns. 

If restoring several QRServers, ensure load is distributed over the updated rows: Select a different row 

in this step for each iteration, or perform the final step to redistribute load on each iteration. 

c) Stop the standalone QRServer and start the normal QRServer: 

nctrl stop qrserver-standalone 

nctrl start qrserver 

d) Bring the QRServer back online: 

qrserver-admin -m up -n fds/searchservice/query1_15100 


This step causes the admin server to push updated configuration (in particular, views) to the QRServer. 

65


66 

e) Start the normal top-level dispatcher: 

nctrl start topfdispatch 

This is the step that actually causes the QRServer to start operating. It is important to perform it after 

the two previous steps, otherwise the QRServer might start with the wrong configuration. 

f) Make sure the QRServer and top-level dispatcher are up before going on to the next node. 

This can be done by checking http://node:15151/status (where node is the current node). 

2. Bring unused search row(s) back to normal operation: Stop the standalone search process and start the 

normal process on the search nodes in the rows that were used by the QRServer just updated: 

nctrl stop search-1-standalone 

nctrl start search-1 

The search nodes are now running in normal operation, and an updated index set and configuration will 

be copied to the nodes as soon as possible. 

3. Distribute load on updated rows. 

For all topfdispatch processes restored to normal mode, edit $FASTSEARCH/etc/enginesrc-tld to 

search all updated rows. Example: 

dataset 0 

partitions 2 

refcost 1 

firstpart 0 

rowbits 2 





To make topfdispatch reload the configuration file, a HTTP GET can be done on 

http://node:15151/admin?command=reload_partmap. Alternatively restart the topfdispatch process. 

If required, perform this step on each iteration of the procedure, redistributing load temporarily over the 

current set of updated rows. Otherwise, postpone it until the last QRServer has been restored and all 

search rows are updated. At the end of the last iteration, each enginesrc-tld should be restored to the 

state before the system was taken to standalone mode.

Chapter 

5 

Working with an Indexing Fault Tolerant Configuration 

Topics: 

• Starting Indexers in a Fault 

Tolerant Configuration 

• Stopping Indexers in a Fault 


• Enabling Indexing Fault 

Tolerance 

• Disabling Indexing Fault 

Tolerance 

• Synchronizing Indexer Rows 

• Upgrading Indexer Binaries in a 

Fault Tolerant Configuration 

• Backing up Indexing in a Fault 


• Indexing Fault Tolerance 

Configuration Parameters 

This section describes how to operate an indexing fault tolerant ESP configuration.


68 

Starting Indexers in a Fault Tolerant Configuration 

Follow this procedure to avoid rebuilding the index when starting the indexers in a fault tolerant configuration. 

Indexer fault tolerance is implemented by having multiple indexers per column. One indexer in each column 

is designated the master indexer, all other indexers are backup indexers. In the event that the master indexer 

becomes unavailable, one of the backup indexers takes over the role of master indexer and rebuilds the 

index. 

Note: 

Rebuilding the index can be a time-consuming process and so it is generally undesirable for a backup 

indexer to take over the role of master indexer unless the original master indexer truly has a problem. 

To avoid rebuilding the index you must be careful about the order in which you start the indexers in an 

indexing fault tolerant ESP cluster. 

As a rule of thumb when stopping ESP, you should always stop the master indexer last so that no backup 

indexer tries to take over. Likewise when starting ESP, you should start the indexer that was previously 

the master indexer first. Then the indexer will continue to use its existing index as opposed to a previous 

backup indexer starting first and having to perform a 'reset index' operation. 

1. Run nctrl start indexer on the preferred master indexer. 

The preferred master indexer is the indexer that was previously the master indexer when ESP was last 

shutdown. 

2. Wait for the master indexer to become available. 

You can determine whether an indexer is available by monitoring its status page at http://:/control.html and waiting for the Column role field to display an appropriate value. 

In the case of a master indexer, it is available when this field shows Master … . A backup indexer is 

available when the Column role field shows Backup … . 

3. Start the backup indexers by running nctrl start indexer on each of the hosting nodes. 

Once a backup indexer is started, it will begin heartbeart monitoring of the master indexer. 

Important: Always validate that the indexer has started successfully before proceeding with any 

operation that depends on it. The simplest way to do this is to check that the status page for the 

indexer is available. 

Stopping Indexers in a Fault Tolerant Configuration 

Follow this procedure to avoid rebuilding the index when stopping the indexers in a fault tolerant configuration. 

Indexer fault tolerance is implemented by having multiple indexers per column. One indexer in each column 

is designated the master indexer, all other indexers are backup indexers. In the event that the master indexer 

becomes unavailable, one of the backup indexers takes over the role of master indexer and rebuilds the 

index. Rebuilding the index can be a time-consuming process, it is generally undesirable for a backup indexer 

to take over the role of master indexer unless the original master indexer truly has a problem. 

Note: 

To avoid rebuilding the index you must be careful about the order in which you stop the indexers in an 

indexing fault tolerant ESP cluster.

Generally when stopping ESP, you should suspend the indexer heartbeats first so that no backup indexer 

tries to take over if the master indexer stops first. Alternatively, you can stop all the backup indexers first 

followed by the master indexer. 

1. Run indexeradmin -a suspendheartbeat on any indexer node to suspend the heartbeat of all indexers, 

preventing the backup indexers from detecting that the master indexer is down. 

If you only want to stop indexers in a single column, use the –column option to stop the heartbeat for that 

column's indexers. 

2. Run nctrl stop indexer on each indexer node. 

The order of shutdown is irrelevant once you have suspended the heartbeat. 

Important: Always validate that the indexer processes have stopped before continuing with any 

further operations. 

Enabling Indexing Fault Tolerance 

Follow this procedure to reconfigure an existing ESP cluster that consists of one indexing row and one or 

more dedicated search rows, to one that provides indexing fault tolerance with multiple indexers per column. 

Before you begin, stop feeding content to the system and wait for the indexing of any recently fed content to 

complete. Then stop all the indexers in the system. 

Important: If you want the search service to remain available during this process, configure the search 

nodes to run in standalone mode while the upgrade takes place. 

1. Add new indexers to the configuration. 

a) Stop the configuration server by running nctrl stop configserver on the administration node. 

b) Edit $FASTSEARCH/etc/CSConfig.xml on the administration node. 

For each column, add the backup entry and change the ft_mode attribute from 0 to 1 . For example: 

 

 

 

c) Run nctrl start configserver to restart the configuration server. 

2. Copy the data_fixml folder from the old indexers to the new ones. 

3. Update the configuration on each new indexer node. 

a) Edit the etc/NodeConf.xml file, and add indexer to the startorder section. 

b) Run nctrl reloadcfg to reload the configuration. 

4. Start the indexers. 


a) Start the original indexers by running nctrl start indexer on the hosting nodes. 

b) After making sure that the original indexers are up, start all the new indexers. 

69


70 

Note: Check the admin GUI and make sure that all the indexers are visible, and that they show up 

as backups of the old indexers. If any of the new indexers are stuck doing initial indexing, wait for 

them to complete. 

5. Restart the search processes on the search rows by running nctrl start search-1 on the hosting 

nodes. 

Note: Remember to disable standalone mode as this will cause search downtime unless there are 

multiple search rows, in which case they should be restarted one at a time. 

Disabling Indexing Fault Tolerance 

Follow this procedure to convert an existing ESP multi-row cluster with indexing fault tolerance, to a single 

indexer row configuration without fault tolerance. 

Before you begin, stop feeding content to the system and wait for the indexing of any recently fed content to 

complete.Then stop all the indexers in the system. Refer to Stopping Indexers in a Fault Tolerant Configuration 

for more details. 

Important: Delete $FASTSEARCH/data/ftStorage and $FASTSEARCH/data/data_index/indexValid.* 

on every backup indexer node that is being removed. 

Note: The procedure described below reutilizes the backup indexer nodes as search nodes. 

1. Reconfigure the Config server on the administration node. 

a) Run nctrl stop configserver to stop the Config Server. 

b) Edit $FASTSEARCH/etc/CSConfig.xml . 

Towards the bottom of this file you will find the element that contains a list of 

search clusters. Each cluster contains a element for each of its columns. For each column 

there will be a column host and one or more backup hosts. Remove or comment out every backup 

host entry, and disable fault tolerance by setting the column host’s ft_mode attribute to 0 . 

 

 

 

 

 

c) Run nctrl start configserver to restart the Config Server. 

Note: Use the Admin GUI to review the ESP logs for any Config Server errors or warnings that 

may have occurred when the Config Server was restarted. 

2. Re-configure the node controllers on every node where the indexer is being removed, to ensure that the 

system no longer will try to start the indexers, and reload the configuration file.

a) Edit the $FASTSEARCH/etc/NodeConf.xml configuration file by locating and removing (or comment 

out) the element. 

b) Run nctrl reloadcfg to reload the configuration file. 

3. Re-configure the new search-only nodes to make them copy the index from their column indexer rather 

than using the index created by a local indexer. 

a) Edit the $FASTSEARCH/etc/searchrc-1.xml file on each node. 

Set the copyindex and socketcopyindex parameters to true . For example: 

 

 

 

b) Run nctrl restart search-1 to restart the search service. 

Note: Verify that no errors or warnings related to search were reported during the restart. 

4. Disable indexing fault tolerance by editing the 

$FASTSEARCH/etc/config_data/RTSearch//rtsearchrc.xml file on the administration 

node. 

a) Locate the element and set the value of its indexCopier parameter to socket . 

For example: 

 

qualifiedHostName="" 

useStrictBind="false" 

nameserviceHost="example.com" 

nameservicePort="16099" 

basePort="15000" 

webServerBaseDir="$RTROOT/www/rtsearch" 

multicastIP="239.255.column.1" 

multicastSpeed="15M" 

multicastDataPort="560" 

indexCopier="socket" 

b) Locate the element further down in the file, and disable fault tolerance by setting the value of the 

enabled parameter to false 

For example: 

 

enabled="false" 

storagePath="$RTROOT/data/ftStorage" 

idleHeartbeatTime="120" 

maxStorageSizeMB="1000" 


Note: If the element contains an attribute called passiveMode , remove the attribute. 

71


72 

5. Start each column’s indexer by running nctrl start indexer on the hosting nodes. 

6. Start the search processes on the search rows by running nctrl start search-1 on the hosting nodes. 

Important: Verify that each column’s search controllers are all registered with the column’s indexer: 

Access the indexer control page at http://:/control.html) and ensure 

that all search controllers (in the column) are registered with the given indexer. 

Synchronizing Indexer Rows 

Use the following procedure to recover from discrepancies between indexer rows. 

Symptoms of fault-tolerant indexers being out of sync include: 

• The number of documents differ between indexers in the same column. 

• The document counts in the index are out of sync with the numbers reported by the indexer. 

• The indexer log errors indicating that it is unable to automatically recover row synchronization. 

The total number of documents for each indexer in a given column should always be the same.These numbers 

can be determined either from looking at the matching engine status page for each indexer, or using ESP’s 

indexerinfo command line utility. 

Note: During content feeding, the document count on the backup indexers may lag slightly; this is normal. 

When feeding stops, the numbers should stabilize. If an indexer is doing recovery of operations, this will 

be shown in bold on the matching engine page. Allow this to complete before comparing the document 

counts. 

1. Stop feeding content to the system and wait for any recently fed content to be indexed. 

Note: It is possible to perform this procedure while feeding, but FAST does not recommend this 

approach unless there is no way you can stop feeding. 

2. For each column that contains an out-of-sync indexer: Stop a synchronized indexer as well as the indexer 

that must be re-synchronized. 

Note: Always remember to stop backup indexers before stopping the master indexer. 

3. Copy the state of the synchronized indexer to the out-of-sync indexer. 

a) Copy $FASTSEARCH/data/ftStorage/processed_checkpoint.txt from the synchronized indexer 

to the out-of-sync indexer. 

b) Copy the contents of the data_fixml folder from the synchronized indexer to the out-of-sync indexer. 

Tip: The Unix rsync utility provides an efficient way to do this: Run rsync -a --delete 

/export2/fast/data/data_fixml :/export2/fast/data 

on the the synchronized indexer host. 

4. Restart all the indexers that were stopped. 

Note: Remember to start the master indexer(s) first, to avoid a backup indexer taking over and 

rebuilding the index. 

After completing this procedure, all the indexers start with the same fixml and the same indexer fault tolerance 

checkpoints.

Upgrading Indexer Binaries in a Fault Tolerant Configuration 

Follow this procedure to upgrade indexers on an indexing fault tolerant ESP cluster with a new set of binaries. 

1. Suspend content feeding and wait until all recently fed content has been indexed. 

2. Make sure all indexer rows are in sync. 

Refer to Synchronizing Indexer Rows for a description of asynchronous indexer symptoms. 

3. Stop all indexers. 

Refer to Stopping Indexers in a Fault Tolerant Configuration for more details. 

4. Delete the data/ftStorage folder on each indexer node on all indexer rows. 

5. Upgrade the indexer binaries by copying the new binaries into the $FASTSEARCH/bin folder. 

6. Start the indexer rows. 

Refer to Starting Indexers in a Fault Tolerant Configuration for more details. 

Backing up Indexing in a Fault Tolerant Configuration 

Follow this procedure to back up indexing and the fault tolerance checkpoints file. 

The procedure for backing up an ESP cluster with fault-tolerant indexers is the same as for a single (indexer) 

row cluster, with one exception: In addition to backing up (and restoring) the data_fixml , and possibly the 

data_index folders, you must also back up the processed checkpoints file. 

If there are multiple indexers, it is not strictly necessary to stop feeding while doing the backup. It is possible 

to stop/suspend only one indexer, and continue feeding to the other indexers while doing the backup. Provided 

that the backup procedure is fast enough, the suspended indexer will then automatically re-synchronize once 

it is resumed after backup. 

Note: This procedure will temporarily reduce the fault-tolerance level of the installation, since there is 

one less row active as long as the backup is being done. For example, if your installation consists of a 

master indexer and a single backup indexer (per row), you will effectively have no indexing fault tolerance 

while the backup indexer is suspended. 

1. Prevent cleanup of the ftstorage folder by running indexeradmin -a suspendheartbeat . 

The ftStorage folder is cleaned up every 2 hours, or as set by the storageCleanupInterval parameter. 

This means that as long the backup procedure completes in less than 2 hours, there will be no risk of any 

operations having been cleaned out from the fault tolerance storage. 

2. Identify which indexer row you want to backup 

3. Stop the backup indexers by running nctrl stop indexer on each of the nodes in the given row. 

4. Backup the data_fixml folder and (optionally) the data_index folder. 

5. Backup the $FASTSEARCH/data/ftStorage/processed_checkpoint.txt file. 


By including the processed checkpoint file in the backup and restore, you ensure that the restored indexer 

does not start recovery of all operations in the column. 

6. Restart the indexers on the given row by running nctrl start indexer on each of the row’s nodes. 

73


74 

Indexing Fault Tolerance Configuration Parameters 

Use the following configuration parameters to define the behavior and characteristics of the indexing service 

when failures occur. 

Parameter 

maxNumFailures 

maxStorageSizeMB 

idleHeartbeatTime 

Description 

The maxNumFailures configuration parameter determines how the indexing service within a 

column acts when failures occur. 

Set this parameter's value to 1 to make both feeding and indexing proceed when one of the 

indexer nodes in the column either fails or is not synchronized with the other indexer nodes in 

the column. If the parameter is set to 0 , any failure will block feeding and indexing until the 

failure has been corrected. 

Note: Setting the parameter value to 1 does not prevent the node from re-synchronizing, 

nor does it allow inconsistent search results, but it does mean that all indexing rows may 

not be synchronized at all times. 

Default value: 1 

The maxStorageSizeMB parameter determines the amount of fault-tolerance storage and thus 

the downtime an indexer node can sustain while still being able to automatically re-synchronize 

with the other indexer nodes in the column. 

For example, if the parameter is set to 1024 (i.e 1GB), and the rate of content fed to the column 

is 100MB per hour, there is a window of approximately 10 hours of downtime before a restarted 

indexer node will be unable to synchronize due to the fact that the oldest operations still being 

required have been deleted. If this happens, the indexer node must be synchronized by doing 

a full copy of the data_fixml folder from one of the other indexer nodes. 

Note: If the feed rate varies, the size of the window in terms of time will also vary. 

The idleHeartbeatTime parameter determines how fast feeding and indexing will resume 

after a failure, and is one of the factors deciding how fast a backup indexer node will take over 

if the master indexer node in a column fails. 

There are other parameters influencing the fail-over time, the most important being the CORBA 

timeout setting, clientCallTimeOutPeriod , specified by the 

$FASTSEARCH/etc/omniorb.cfg file. 

Note: When tuning these parameters, keep in mind that shorter timeouts and more frequent 

pings of the other parts of the system imply increased overall load and network traffic.There 

is also a larger chance of small network interruptions causing a re-shuffling of the roles in 

the system which is time consuming. This is the case if a backup indexer takes over as the 

master indexer and performs a 'reset index' operation on a large quantity of data. FAST 

does not normally recommend changing the default values for these settings, but special 

circumstances may warrant it. 

Tip: For more information on the indexing configuration parameters related to fault tolerance, refer the 

rtsearchrc.xml configuration file documentation.

Chapter 

6 

Changing the host name of a FAST ESP node 

Topics: 

• Changing the host name in a 

UNIX system 

• Changing the host name in a 

Windows system 

This section describes how to change the host name of a FAST ESP node. This 

may be necessary due to changing network setup, or if the installation is moved 

between physical machines.


76 

Changing the host name in a UNIX system 

To change the host name used in FAST ESP on a UNIX system, complete the following procedure: 

It is recommended that you make a backup copy of all files before changing them. 

All procedures are based on the assumption that both the old and the new host names are fully qualified host 

names. 

It is not recommended to change the host name of a running installation unless absolutely necessary. 

1. Stop FAST ESP using a sequential shutdown. Refer to Stopping FAST ESP via sequential shutdown. 

2. To get a list of all configuration files that contain the old host name, use the following command: 

grep -s -l -r $FASTSEARCH/etc/* $FASTSEARCH/ 

adminserver/* $FASTSEARCH/esp4j/* $FASTSEARCH/components/* 

where is the old host name. 

3. Replace the old host name with the new one in each of the files. 

The following command can be used to automatically replace the host name in all configuration files: 

for f in `grep -s -l -r $FASTSEARCH/etc/ 

* $FASTSEARCH/adminserver/* $FASTSEARCH/esp4j/* $FASTSEARCH/components/ 

*`; do sed -i.bak -e 's///g' $f; done 

Note that this command will create a backup, with the suffix .bak, of all modified files. 

4. Delete the contents of the following directories: 

$FASTSEARCH/var/etc 

$FASTSEARCH/var/log/omniorb 

5. Repeat steps 2 through 4 on all other nodes in your system. 

6. Start FAST ESP. 

On all nodes run: 

nctrl start 

7. On the administration node, stop the admin server and enter the PostgreSQL database: 

nctrl stop adminserver 

cd $FASTSEARCH/rdbms/bin 

./psql --username fast –p 16070 vespa 

Execute the following SQL command: 

update SearchDispatcher set host="" where host=""; 

update Link set url="http://"":16000/admin/" where url= 

"http://"":16000/admin/"; 

update Link set url="http://"":16000/clarity/" where url= 

"http://"":16000/clarity/"; 

exit; 

Note that the contents of the Link table may be different. Inspect the contents of this table to ensure that 

no URLs are using the old host name. 

8. Restart the admin server: 

nctrl start adminserver 

9. Redeploy all views: 

view-admin -m deploy -a

Changing the host name in a Windows system 

To change the host name used in FAST ESP on a Windows system, complete the following procedure: 

It is recommended that you make a backup copy of all files before changing them. 

All procedures are based on the assumption that both the old and the new host names are fully qualified host 

names. 

It is not recommended to change the host name of a running installation unless absolutely necessary. 

1. Stop FAST ESP using a sequential shutdown. Refer to Stopping FAST ESP via sequential shutdown. 

2. Perform a search in all FAST ESP directories for files that contain the old host name. Do not use the 

Windows file explorer search feature as it may not find all files (for example .cfg files). From a command 

prompt, do the following: 

cd %FASTSEARCH% 

findstr /S /I /M /C:"" /D:%FASTSEARCH%\etc;%FASTSEARCH%\adminserver; 

%FASTSEARCH%\esp4j;%FASTSEARCH%\components *.* 

where is the old host name. 

3. Replace the old host name with the new one in each of the files, except for any .log files. 

4. Delete the contents of the following directories: 

%FASTSEARCH%\var\etc 

%FASTSEARCH%\var\log\omniorb 

5. Repeat steps 2 through 4 on all other nodes in your system. 

6. Start FAST ESP. 

On all nodes run: 

nctrl start 

7. On the administration node, stop the admin server and enter the PostgreSQL database: 

nctrl stop adminserver 

cd %FASTSEARCH%\rdbms\bin 

psql --username fast --host= --port=16070 vespa 

Execute the following SQL command: 

update SearchDispatcher set host="" where host=""; 

update Link set url="http://"":16000/admin/" where url= 

"http://"":16000/admin/"; 

update Link set url="http://"":16000/clarity/" where url= 

"http://"":16000/clarity/"; 

exit; 

Note that the contents of the Link table may be different. Inspect the contents of this table to ensure that 

no URLs are using the old host name. 

8. Restart the admin server 

nctrl start adminserver 

9. Redeploy all views 


Changing the host name of a FAST ESP node 

77

Chapter 

7 

Changing the admin user password 

Topics: 

• Changing the admin user 

password 

This section describes how to change the administrator password of the admin 

server, used by the FAST Home, SBC and Admin GUI web applications.


80 

Changing the admin user password 

This section describes how to change the administrator password of the admin server, used by the FAST 

Home, SBC and Admin GUI web applications. 

1. Change the password on the server side through a browser in the User Administration page of the Fast 

Home web application. 

2. Change the password on the client side so that the client applications are still able to function. 

There are a number of internal modules that are using the services in the admin server. For these modules 

to be able to access the services after changing the password, you need to update the password in a 

common properties file for these modules: 

a) Update the $FASTSEARCH/etc/esp4j.properties file. Change the line: 

esp.adminserver.password= 

to 

esp.adminserver.password= 

where is the new password. 

b) Restart the logtransformer process for the change to take effect. 

nctrl restart logtransformer 

As a security measure, it is a good idea to remove read and write access to the file 

$FASTSEARCH/etc/esp4j.properties for everyone except the user running FAST ESP.

Chapter 

8 

Granting system access to non-privileged users 

Topics: 

• Granting system access to 

non-privileged users 

This section describes how to allow a user to run FAST ESP without administrative 

privileges, on Windows XP/ 2000/ 2003.


82 

Granting system access to non-privileged users 

This section describes how to allow a user to run FAST ESP without administrative privileges, on Windows 

XP/ 2000/ 2003. 

Before you can grant access to your system, the installation process must have completed successfully. 

1. Assign Log on as a service privileges to the user by selecting Start > Control Panel > Administrative 

Tools > Local Security Policy > Local Policies > User Rights Assignment > Log on as a Service > 

Add User or Group and adding the user. 

2. Configure the FAST ESP and the FAST ESP Web Server services to run as this user: 

a) Select Control Panel > Administrative Tools > Services 

b) Right-click the service in question, and select the Properties menu entry. 

c) Open the Log On tab and select This Account . 

d) Provide the username and password for the user in question. 

3. Grant the user the right to administer these services. Follow the instructions relevant to your environment 

in the following knowledge base article: http://support.microsoft.com/kb/288129/en-us/ 

4. Assign Full Control permissions to the user for the registry keys listed as follows. Open the registry editor 

by selecting Start > Run and type regedit. 

a) Click the key to which you want to assign permissions. 

b) Select Permissions from the Edit menu. 

c) Select the Allow checkbox for the Full Control entry. 

These are the keys the user must have full control over: 

• HKEY_LOCAL_MACHINE\SOFTWARE\FAST Search & Transfer 

• HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Eventlog 

• HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\FASTESPService 

• HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ FASTESPWebServer 

Important: Do not edit your registry unless it is absolutely necessary. If there is an error in your 

registry, your computer may not function properly. 

5. Assign Full Control permissions for the installation directory to the user: 

a) Right-click your installation directory. 

b) Select the Properties menu entry and open the Security tab. 

c) Select the Allow checkbox for the Full Control entry.

Chapter 

9 

Uploading a new index profile from command-line 

Topics: 

• Uploading a new index profile 

using bliss 

• Cold update of index profile using 

dedicated search row 

This section describes how to upload a new index profile from command-line. 

Content feeding must be suspended while an index profile update is in progress. 

Stop crawler, file traverser and any other connectors.


84 

Uploading a new index profile using bliss 

The $FASTSEARCH/bin/bliss program is used to update the index profile from the command line. 

To complete this procedure, all ESP components must be running, including admin server and QRServer(s). 

If one or more of multiple QRServers are down, this can be handled by telling the admin server that the 

specific QRServer should not be touched during the index profile update. This can be done by issuing the 

command qrserver-admin -m down -n fds/searchservice/node1_15100. To push the changed 

configuration files to this QRServer after the changes have been made, issue the command qrserver-admin 

-m up -n fds/searchservice/node1_15100 to notify the admin server that this QRServer is back up. This 

will deploy all views to that QRServer. Refer to qrserver-admin tool for more information. 

Perform this procedure on the administration node: 

1. Copy the custom index-profile file into the $FASTSEARCH/index-profile folder. 

2. Open a command prompt and make sure the environment variables are set correctly. On Windows, the 

environment variables have already been set. 

On UNIX: 

cd $FASTSEARCH/bin 

. setupenv.sh 

3. Stop feed. 

Stop any crawlers or scheduled jobs which will feed content. 

4. To validate the new index profile, go to the folder $FASTSEARCH/index-profiles and run the following 

bliss command: 

$FASTSEARCH/bin/bliss -V / 

If errors are reported, correct them and re-run the validation. 

5. To import the new index profile, run the following bliss command: 

$FASTSEARCH/bin/bliss -k 

-C / 

If the update requires a cold update (because the new index profile was not compatible with the existing 

one), you need to use the-i option. Before running bliss again with the -i option, you must delete all the 

data in the cluster you are updating. Refer to indexeradmin tool for usage information. 

IMPORTANT: Using this procedure (purging data plus bliss with the -i option) will cause all your indexed 

data to be lost. 

6. Restart feed. 

Cold update of index profile using dedicated search row 

This procedure describes how to perform a cold update of an index profile without search down time. 

The procedure requires a deployment with one indexer row and one dedicated search row. Search is set in 

standalone mode while the index profile update is taking place. To prepare for standalone mode, some 

configuration changes must be done. See the prerequisites of Switching search to standalone mode. 

A cold update of the index profile clears the existing indexes, and it is necessary to refeed all data from the 

original content sources. 

1. Switch search to standalone mode. 

See Switching search to standalone mode.

2. Back up the views. 

On the admin node, run: 

view-admin -m export 

which will create one XML file per view defined in your current system. 

3. Stop feed. 

Stop any crawlers or scheduled jobs which will feed content. 

4. Backup FiXML. 

On all the nodes in the indexer row: 

• Stop the indexers; nctrl stop indexer 

• Move away old data; mv $FASTSEARCH/data/data_fixml $FASTSEARCH/data/data_fixml_backup 

• start indexers; nctrl start indexer 

Alternatively, if you don't need the backup, purge the indexers' data: 

indexeradmin -a purgedata 

5. Run bliss. 

Run bliss from the admin-node: 

bliss -i -C new_index_profile.xml 

Bliss needs the index profile DTD index-profile-5.0.1.dtd in the current directory. It is recommended 

to run bliss with $FASTSEARCH/index-profiles as the current directory. 

6. Suspend indexing. 


7. Refeed all data using the original data source. 

8. When all data have been fed, run a reset index: 

indexeradmin -a resetindex 

9. Put system back in normal operations. 

Use the procedure in Restoring normal operations from standalone search. 

Uploading a new index profile from command-line 

85

Chapter 

10 

SNMP Agent 

Topics: 

• About the SNMP Agent 

• Starting the SNMP Agent 

• SNMP MIB 

• SNMP MIB variables 

This section discusses the SNMP Agent (Simple Network Management Protocol) 

that monitors FAST ESP components and service status.


88 

About the SNMP Agent 

FAST ESP includes an SNMP Agent (Simple Network Management Protocol) that monitors FAST ESP 

components and service status. This tool enables FAST ESP to be monitored from management consoles 

such as IBM Tivoli and HP OpenView , and supports FAST ESP status read including component status, 

indexing status, document processing status and query status. 

The SNMP Agent supports SNMPv1 and SNMPv2c. SNMPv3 is currently not supported. The SNMP MIB 

(Management Information Base) is compliant with SMIv2 (Storage Management Initiative).The FAST Enterprise 

OID is 4200. 

The SNMP Agent uses UDP (User Datagram Protocol) port baseport + 3001 (port 16001 in a default 

installation). 

Starting the SNMP Agent 

The SNMP Agent is distributed as a part of FAST ESP but is not activated. Complete the following to activate 

SNMP on all nodes that will run the SNMP Agent. 

1. Go to FAST ESP configuration directory: 

% cd $FASTSEARCH/etc 

2. Edit the NodeConf.xml file: 

Add the snmpd proc line in the startorder: 

 

. 

. 

. 

nctrl 

clarity_webcluster 

snmpd 

 

3. Reload the node controller: 

% $FASTSEARCH/bin/nctrl reloadcfg 

4. Start the SNMP Agent: 

% nctrl start snmpd 

5. Verify that the SNMP Agent is running: 

SNMP MIB 

% nctrl sysstatus 

The following is a hierarchical tree of the supported Management Information Base (MIB) for the SNMP Agent. 

Mib files are located at $FASTSEARCH/etc/mibs. 

+-fastRoot(4200) 

+--fastReg(1) 

| +--fastModules(1) 

| +--fastGlobalReqModule(1) 

| +--fastESPCapModule(4) 

| +--fastESPMibModule(7)

SNMP Agent 

| 

+--fastGeneric(2) 

+--fastProducts(3) 

| | 

| +--fastESP(3) 

| | | 

| | +--fastESPApplication(1) 

| | | | 

| | | +-- -R-- Counter fastESPApplicationNumQueries(1) 

| | | +-- -R-- Counter fastESPApplicationNumDocuments(2) 

| | | +-- -R-- Counter fastESPApplicationNumUpdates(3) 

| | | +-- -R-- Counter fastESPApplicationNumComponents(4) 

| | | 

| | +--fastESPNode(2) 

| | | | 

| | | +--fastESPnctrl(1) 

| | | | | 

| | | | +--fastESPComponentTable(1) 

| | | | | 

| | | | +--fastESPComponentEntry(1) 

| | | | | Index: fastESPComponentEntryIndex 

| | | | | 

| | | | +-- ---- Unsigned fastESPComponentEntryIndex(1) 

| | | | +-- -R-- String fastESPComponentEntryName(2) 

| | | | +-- -R-- String fastESPComponentEntryDescription(3) 

| | | | +-- -R-- EnumVal fastESPComponentEntryOperationalState(4) 

| | | | | Textual Convention: FastESPComponentState 

| | | | | Values: dead(1), autosuspend(2), usersuspend(3), 

| | | | | running(4), stopping(5), restarting(6) 

| | | | +-- -RW- EnumVal fastESPComponentEntryAdministrativeState(5) 

| | | | | Textual Convention: FastESPComponentState 

| | | | | Values: dead(1), autosuspend(2), usersuspend(3), 

| | | | | running(4), stopping(5), restarting(6) 

| | | | +-- ---- Unsigned fastESPComponentEntryPID(6) 

| | | | 

| | | +--fastESPindexer(2) 

| | | | | 

| | | | +--fastESPIndexerTable(1) 

| | | | | | 

| | | | | +--fastESPIndexerEntry(1) 

| | | | | | Index: fastESPIndexerEntryIndex 

| | | | | | 

| | | | | +-- ---- Unsigned fastESPIndexerEntryIndex(1) 

| | | | | +-- -R-- Gauge fastESPIndexerEntryAPIQueueSize(2) 

| | | | | +-- -R-- Unsigned fastESPIndexerEntryLastIndexSwitch(3) 

| | | | | +-- -R-- Counter fastESPIndexerEntryAPIOperations(4) 

| | | | | +-- -R-- Counter fastESPIndexerEntryFailedOperations(5) 

| | | | | +-- -R-- Counter fastESPIndexerEntryPartialUpdate(6) 

| | | | | +-- -R-- Counter fastESPIndexerEntryRemoveOperations(7) 

| | | | | +-- -R-- Counter fastESPIndexerEntryStatusUpdates(8) 

| | | | | +-- -R-- Counter fastESPIndexerEntryRemoveCollectionOperations(9) 

| | | | | +-- -R-- Counter fastESPIndexerEntryUpdateOperations(10) 

| | | | | +-- -R-- Gauge fastESPIndexerEntryTotalNumDocuments(11) 

| | | | | +-- -R-- Gauge fastESPIndexerEntryTotalSize(12) 

| | | | | +-- -R-- EnumVal fastESPIndexerEntryOperationalResetIndex(13) 

| | | | | | Textual Convention: BooleanValue 

| | | | | | Values: false(0), true(1) 

| | | | | +-- -R-- EnumVal fastESPIndexerEntryOperationalIndexingSuspended(14) 



| | | | | +-- -R-- EnumVal fastESPIndexerEntryDiskLow(15) 



| | | | | +-- -RW- EnumVal fastESPIndexerEntryAdministrativeResetIndex(16) 



| | | | | +-- -RW- EnumVal 

fastESPIndexerEntryAdministrativeIndexingSuspended(17) 

89


90 

| | | | | Textual Convention: BooleanValue 

| | | | | Values: false(0), true(1) 

| | | | | 

| | | | +--fastESPPartitionTable(2) 

| | | | | 

| | | | +--fastESPPartitionEntry(1) 

| | | | | Index: fastESPPartitionEntryIndexerIndex, fastESPPartitionEntryIndex 

| | | | | 

| | | | +-- ---- Unsigned fastESPPartitionEntryIndexerIndex(1) 

| | | | +-- ---- Unsigned fastESPPartitionEntryIndex(2) 

| | | | +-- -R-- Gauge fastESPPartitionEntryNumDocuments(3) 

| | | | +-- -R-- Counter fastESPPartitionEntryTotalIndexingRuns(4) 

| | | | +-- -R-- Gauge fastESPPartitionEntryVectorMemoryIndexing(5) 

| | | | +-- -R-- Gauge fastESPPartitionEntryVectorMemory(6) 

| | | | +-- -R-- String fastESPPartitionEntryIndexedPerSecond(7) 

| | | | Textual Convention: FloatValue 

| | | | 

| | | +--fastESPprocserver(3) 

| | | | | 

| | | | +--fastESPStagesTable(1) 

| | | | | | 

| | | | | +--fastESPStagesEntry(1) 

| | | | | | Index: fastESPStagesEntryIndex 

| | | | | | 

| | | | | +-- ---- Unsigned fastESPStagesEntryIndex(1) 

| | | | | +-- -R-- String fastESPStagesEntryName(2) 

| | | | | +-- -R-- String fastESPStagesEntrySystemTime(3) 

| | | | | | Textual Convention: FloatValue 

| | | | | +-- -R-- String fastESPStagesEntryUserTime(4) 


| | | | | +-- -R-- String fastESPStagesEntryRealTime(5) 


| | | | | +-- -R-- Gauge fastESPStagesEntryPageSwaps(6) 

| | | | | +-- -R-- Gauge fastESPStagesEntryMemUsage(7) 

| | | | | +-- -R-- Gauge fastESPStagesEntryResidentMem(8) 

| | | | | +-- -R-- Gauge fastESPStagesEntryVirtualMem(9) 

| | | | | +-- -R-- Gauge fastESPStagesEntryNumOK(10) 

| | | | | +-- -R-- Gauge fastESPStagesEntryNumError(11) 

| | | | | 

| | | | +--fastESPPipelinesTable(2) 

| | | | | | 

| | | | | +--fastESPPipelinesEntry(1) 

| | | | | | Index: fastESPPipelinesEntryIndex 

| | | | | | 

| | | | | +-- ---- Unsigned fastESPPipelinesEntryIndex(1) 

| | | | | +-- -R-- String fastESPPipelinesEntryName(2) 

| | | | | +-- -R-- String fastESPPipelinesEntrySystemTime(3) 


| | | | | +-- -R-- String fastESPPipelinesEntryUserTime(4) 


| | | | | +-- -R-- String fastESPPipelinesEntryRealTime(5) 


| | | | | +-- -R-- Gauge fastESPPipelinesEntryPageSwaps(6) 

| | | | | +-- -R-- Gauge fastESPPipelinesEntryMemUsage(7) 

| | | | | +-- -R-- Gauge fastESPPipelinesEntryResidentMem(8) 

| | | | | +-- -R-- Gauge fastESPPipelinesEntryVirtualMem(9) 

| | | | | +-- -R-- Gauge fastESPPipelinesEntryNumOK(10) 

| | | | | +-- -R-- Gauge fastESPPipelinesEntryNumError(11) 

| | | | | 

| | | | +--fastESPDocumentProcessorTable(3) 

| | | | | 

| | | | +--fastESPDocumentProcessorEntry(1) 

| | | | | Index: fastESPDocumentProcessorEntryIndex 

| | | | | 

| | | | +-- ---- Unsigned fastESPDocumentProcessorEntryIndex(1) 

| | | | +-- -R-- EnumVal fastESPDocumentProcessorEntryQueueFull(2) 

| | | | Textual Convention: BooleanValue 

| | | | Values: false(0), true(1) 

| | | | 

| | | +--fastESPindexingdispatcher(4)

| | | | | 

| | | | +--fastESPClusterTable(1) 

| | | | | | 

| | | | | +--fastESPClusterEntry(1) 

| | | | | | Index: fastESPClusterEntryIndex 

| | | | | | 

| | | | | +-- ---- Unsigned fastESPClusterEntryIndex(1) 

| | | | | +-- -R-- String fastESPClusterEntryName(2) 

| | | | | +-- -R-- Gauge fastESPClusterEntryNumIndexingDispatchers(3) 

| | | | | 

| | | | +--fastESPCollectionTable(2) 

| | | | | | 

| | | | | +--fastESPCollectionEntry(1) 

| | | | | | Index: fastESPCollectionEntryIndex 

| | | | | | 

| | | | | +-- ---- Unsigned fastESPCollectionEntryIndex(1) 

| | | | | +-- -R-- Unsigned fastESPCollectionEntryClusterIndex(2) 

| | | | | +-- -R-- Gauge fastESPCollectionEntryNumDocuments(3) 

| | | | | +-- -R-- String fastESPCollectionEntryName(4) 

| | | | | 

| | | | +--fastESPColumnTable(3) 

| | | | | 

| | | | +--fastESPColumnEntry(1) 

| | | | | Index: fastESPColumnEntryIndex 

| | | | | 

| | | | +-- ---- Unsigned fastESPColumnEntryIndex(1) 

| | | | +-- -R-- Unsigned fastESPColumnEntryClusterIndex(2) 

| | | | +-- -R-- Gauge fastESPColumnEntryColumnNumber(3) 

| | | | +-- -R-- Gauge fastESPColumnEntryAPIQueueSize(4) 

| | | | 

| | | +--fastESPresourceservice(5) 

| | | | 

| | | +--fastESPrelbench(6) 

| | | | 

| | | +--fastESPanchorserver(7) 

| | | | 

| | | +--fastESPcontentdistributor(8) 

| | | | | 

| | | | +--fastESPCDTable(1) 

| | | | | 

| | | | +--fastESPCDEntry(1) 

| | | | | Index: fastESPCDEntryIndex 

| | | | | 

| | | | +-- ---- Unsigned fastESPCDEntryIndex(1) 

| | | | +-- -R-- Gauge fastESPCDEntryNumDocprocs(2) 

| | | | +-- -R-- Gauge fastESPCDEntryDocprocsBusy(3) 

| | | | +-- -R-- Gauge fastESPCDEntryDocprocsInvalid(4) 

| | | | +-- -R-- Gauge fastESPCDEntryDocprocsUnused(5) 

| | | | +-- -R-- Gauge fastESPCDEntryOperations(6) 

| | | | +-- -R-- String fastESPCDEntryMode(7) 

| | | | +-- -R-- Gauge fastESPCDEntryCallbacks(8) 

| | | | 

| | | +--fastESPlogtransformer(9) 

| | | | 

| | | +--fastESPcompletionserver(10) 

| | | | 

| | | +--fastESPqrserver(11) 

| | | | | 

| | | | +--fastESPQRServerTable(1) 

| | | | | 

| | | | +--fastESPQRServerEntry(1) 

| | | | | Index: fastESPQRServerEntryIndex 

| | | | | 

| | | | +-- ---- Unsigned fastESPQRServerEntryIndex(1) 

| | | | +-- -R-- Counter fastESPQRServerEntryNumFailedQueriesSystem(2) 

| | | | +-- -R-- Counter fastESPQRServerEntryNumFailedQueriesTotal(3) 

| | | | +-- -R-- Counter fastESPQRServerEntryNumFailedQueriesUser(4) 

| | | | +-- -R-- Counter fastESPQRServerEntryNumQueries(5) 

| | | | +-- -R-- Counter fastESPQRServerEntryNumRequests(6) 

| | | | 

| | | +--fastESPsearch(12) 

SNMP Agent 

91


92 

| | | | 

| | | | 

| | | | 

| | | +--fastESPdeploymentmanager(14) 

| | | 

| | +--fastESPEvents(3) 

| | | | 

| | | +--fastESPTrap(0) 

| | | | 

| | | +--fastESPIndexerStateChange(1) 

| | | +--fastESPIndexerDiskLow(2) 

| | | +--fastESPIndexerResetIndex(3) 

| | | +--fastESPAgentStarted(4) 

| | | +--fastESPComponentStateEvent(5) 

| | | +--fastESPApplicationNumComponentsChange(6) 

| | | 

| | | 

| | +--fastESPConfs(4) 

| | | 

| | +--fastESPGroups(1) 

| | | | 

| | | +--fastESPapplicationGroup(1) 

| | | +--fastESPnctrlGroup(2) 

| | | +--fastESPindexerGroup(3) 

| | | +--fastESPprocserverGroup(4) 

| | | +--fastESPindexingdispatcherGroup(5) 

| | | +--fastESPcontentdistributorGroup(6) 

| | | +--fastESPqrserverGroup(7) 

| | | | 

| | | +--fastESPindexerEventGroup(8) 

| | | | 

| | | +--fastESPagentEventGroup(9) 

| | | | 

| | | +--fastESPnctrlEventGroup(10) 

| | | | 

| | | +--fastESPApplicationEventGroup(11) 

| | | 

| | +--fastESPCompl(2) 

| | | 

| | +--fastESPStandardComplianceV1(2) 

| | 

| | 

| +--fastCrawler(4) 

| | | 

| | +-- -R-- OctetString fastCrawlerVersion(1) 

| | 

| | 

| +--fastClarity(5) 

| | 

| | 

| +--fastFeedingProxy(6) 

| | 

| +--fastFeedingProxyFeeder(1) 

| | | 

| | +-- -R-- Counter fastFeedingProxyFeederRecovery(1) 

| | +-- -R-- Counter fastFeedingProxyFeederInSync(2) 

| | +-- -R-- Counter fastFeedingProxyFeederSumDocuments(3) 

| | +-- -R-- Counter fastFeedingProxyFeederErrDocuments(4) 

| | +-- -R-- Counter fastFeedingProxyFeederDocumentsToBeFed(5) 

| | 

| +--fastFeedingProxyReceiver(2) 

| | 

| +-- -R-- OctetString fastFeedingProxyReceiverDiskLow(1) 

| +-- -R-- Counter fastFeedingProxyReceiverSumDocuments(2) 

| +-- -R-- Counter fastFeedingProxyReceiverErrDocuments(3) 

| 

| 

+--fastCaps(4) 

| +--fastESPAgentV10(1) 

| 

+--fastReqs(5)

+--fastExpr(6) 

SNMP MIB variables 

The section describes the components in the hierarchical tree of the Management Information Base (MIB) 

for the SNMP Agent. 

SNMP MIB variables - Imports 

This table describes... 

FAST-TC:: 

FAST-TC:: 

FAST-GLOBAL-REG:: 

FAST-GLOBAL-REG:: 

SNMPv2-TC:: 

SNMPv2-SMI:: 

SNMPv2-SMI:: 

SNMPv2-SMI:: 

SNMPv2-SMI:: 

SNMPv2-SMI:: 

SNMPv2-SMI:: 

SNMPv2-CONF:: 

SNMPv2-CONF:: 

SNMPv2-CONF:: 

SNMP MIB variables - Type definitions 


FastESPNctrlComponentState 

SNMP MIB variables - Managed variables 


BooleanValue 

FloatValue 

fastModules 

fastProducts 

TEXTUAL-CONVENTION 

MODULE-IDENTITY 

OBJECT-TYPE 

NOTIFICATION-TYPE 

Counter32 

Gauge32 

Unsigned32 

OBJECT-GROUP 

NOTIFICATION-GROUP 

MODULE-COMPLIANCE 

The state of a FAST ESP component. 

SNMP Agent 

93


94 

Variable 

fastESPMibModule::= 

fastESP::= 

fastESPApplication::= 

fastESPApplicationNumQueries::= 

fastESPApplicationNumDocuments::= 

fastESPApplicationNumUpdates::= 

fastESPApplicationNumComponents::= 

fastESPNode::= 

fastESPNctrl::= 

fastESPCompl::= 

fastESPNctrlComponentTable::= 

fastESPNctrlComponentEntryIndex::= 

fastESPNctrlComponentEntryName::= 

fastESPNctrlComponentEntryDescription::= 

fastESPNctrlComponentEntryOperationalState::= 

fastESPNctrlComponentEntryAdministrativeState::= 

fastESPNctrlComponentEntryPID::= 

fastESPIndexer::= 

fastESPIndexerTable::= 

fastESPIndexerEntryIndex::= 

fastESPIndexerEntryAPIQueueSize::= 

fastESPIndexerEntryLastIndexSwitch::= 

Data type 

NODE 

NODE 

NODE 

Counter32 

Counter32 

Counter32 

Gauge32 

NODE 

NODE 

NODE 

TABLE 

Unsigned32 

OctetString 

OctetString 



Unsigned32 

NODE 

TABLE 

Unsigned32 

Gauge32 

Unsigned32 

Description 

FAST ESP system wide query count. 

FAST ESP system-wide aggregation of total 

number of documents. 

FAST ESP system-wide aggregation of update 

operations. 

Number of components registered in naming 

services. 

A table of all components running in the system 

and their state according to the node controller. 

Index in table. 

Short name of component. 

Long name of component. 

State of the component; taken from the 

nodecontroller state. 

Wanted state of component. 

PID of component. 

Table of all indexers on a node. 

Row index. 

API queue size, number of batches. 

Last index switch in seconds since ???

Variable 

fastESPIndexerEntryAPIOperations::= 

fastESPIndexerEntryFailedOperations::= 

fastESPIndexerEntryPartialUpdate::= 

fastESPIndexerEntryRemoveOperations::= 

fastESPIndexerEntryStatusUpdates::= 

fastESPIndexerEntryRemoveCollectionOperations::= 

fastESPIndexerEntryUpdateOperations::= 

fastESPIndexerEntryTotalNumDocuments::= 

fastESPIndexerEntryTotalSize::= 

fastESPIndexerEntryOperationalResetIndex::= 

fastESPIndexerEntryOperationalIndexingSuspended::= 

fastESPIndexerEntryDiskLow::= 

fastESPIndexerEntryAdministrativeResetIndex::= 

fastESPIndexerEntryAdministrativeIndexingSuspended::= 

fastESPPartitionTable::= 

fastESPPartitionEntryIndexerIndex::= 

fastESPPartitionEntryIndex::= 

fastESPPartitionEntryNumDocuments::= 

fastESPPartitionEntryTotalIndexingRuns::= 

fastESPPartitionEntryVectorMemoryIndexing::= 

fastESPPartitionEntryVectorMemory::= 

fastESPPartitionEntryIndexedPerSecond::= 

fastESPprocserver::= 

fastESPStagesTable::= 

Data type 

Counter32 

Counter32 

Counter32 

Counter32 

Counter32 

Counter32 

Counter32 

Gauge32 

Gauge32 

BooleanValue 

BooleanValue 

BooleanValue 

BooleanValue 

BooleanValue 

TABLE 

Unsigned32 

Unsigned32 

Gauge32 

Counter32 

Gauge32 

Gauge32 

FloatValue 

NODE 

TABLE 

Description 

Number of API operations. 

Number of failed API operations. 

Number of API partial update operations. 

Number of API remove operations. 

Number of API status update operations. 

Number of API remove collection operations. 

Number of API update operations. 

Total number of documents in indexer. 

Total size of documents in MBytes. 

Is the indexer performing a reset index operation. 

Indexing suspended state. 

Indexer disk space low. 

Do a reset index operation. 

Suspend indexing. 

A table of all partitions located on the node. 

Row index. 

Index in partition table. 

Number of documents in partition. 

Total number of indexing runs in partition. 

Size of vector memory during indexing. 

Size of vector memory needed to run search. 

Number of documents indexed per second. 

Table of all stages. 

SNMP Agent 

95


96 

Variable 

fastESPStagesEntryIndex::= 

fastESPStagesEntryName::= 

fastESPStagesEntrySystemTime::= 

fastESPStagesEntryUserTime::= 

fastESPStagesEntryRealTime::= 

fastESPStagesEntryPageSwaps::= 

fastESPStagesEntryMemUsage::= 

fastESPStagesEntryResidentMem::= 

fastESPStagesEntryVirtualMem::= 

fastESPStagesEntryNumOK::= 

fastESPStagesEntryNumError::= 

fastESPPipelinesTable::= 

fastESPPipelinesEntryIndex::= 

fastESPPipelinesEntryName::= 

fastESPPipelinesEntrySystemTime::= 

fastESPPipelinesEntryUserTime::= 

fastESPPipelinesEntryRealTime::= 

fastESPPipelinesEntryPageSwaps::= 

fastESPPipelinesEntryMemUsage::= 

fastESPPipelinesEntryResidentMem::= 

fastESPPipelinesEntryVirtualMem::= 

fastESPPipelinesEntryNumOK::= 

fastESPPipelinesEntryNumError::= 

fastESPDocumentProcessorTable::= 

Data type 

Unsigned32 

OctetString 

FloatValue 

FloatValue 

FloatValue 

Counter32 

Gauge32 

Gauge32 

Gauge32 

Counter32 

Counter32 

TABLE 

Unsigned32 

OctetString 

FloatValue 

FloatValue 

FloatValue 

Counter32 

Gauge32 

Gauge32 

Gauge32 

Counter32 

Counter32 

TABLE 

Description 

Row index. 

Name of stage. 

System CPU time. 

User CPU time. 

Real time. 

Number of page swaps. 

Memory usage. 

Resident memory. 

Virtual memory. 

Number of correctly processed documents. 

Number of errors in document processing. 

Table of pipelines. 

Row index in pipeline table. 

Pipeline name. 

System time used. 

User time used. 

Real time used. 

Number of page swaps. 

Memory usage. 

Resident memory. 

Virtual memory. 

Number of correctly processed documents. 

Number of incorrectly processed documents. 

Table of document processors.

Variable 

fastESPDocumentProcessorEntryIndex::= 

fastESPDocumentProcessorEntryQueueFull::= 

fastESPIndexingDispatcher::= 

fastESPClusterTable::= 

fastESPClusterEntryIndex::= 

fastESPClusterEntryName::= 

fastESPClusterEntryNumIndexingDispatchers::= 

fastESPCollectionTable::= 

fastESPCollectionEntryIndex::= 

fastESPCollectionEntryClusterIndex::= 

fastESPCollectionEntryNumDocuments::= 

fastESPCollectionEntryName::= 

fastESPColumnTable::= 

fastESPColumnEntryIndex::= 

fastESPColumnEntryClusterIndex::= 

fastESPColumnEntryColumnNumber::= 

fastESPColumnEntryAPIQueueSize::= 

fastESPResourceService::= 

fastESPRelbench::= 

fastESPAnchorServer::= 

fastESPContentDistributor::= 

fastESPCDTable::= 

fastESPCDEntryIndex::= 

fastESPCDEntryNumDocprocs::= 

Data type 

Unsigned32 

BooleanValue 

NODE 

TABLE 

Unsigned32 

OctetString 

Gauge32 

TABLE 

Unsigned32 

Unsigned32 

Gauge32 

OctetString 

TABLE 

Unsigned32 

Unsigned32 

Gauge32 

Gauge32 

NODE 

NODE 

NODE 

NODE 

TABLE 

Unsigned32 

Gauge32 

Description 

Row index in document processor table. 

Document processor queue full. 

Table of clusters. 

Row index in cluster table. 

Cluster name. 

Number of indexing dispatchers available. 

Table of collections. 

Row index in collection table. 

Second index, referencing the cluster table entry. 

Number of documents in collection. 

Name of collection. 

Table of columns. 

Row index in column table. 

Second index, referencing the cluster table entry. 

Column number. 

API queue size in column. 

Content distributor table. 

Row in the content distributor table. 

Number of document processors. 

SNMP Agent 

97


98 

Variable 

fastESPCDEntryDocprocsBusy::= 

fastESPCDEntryDocprocsInvalid::= 

fastESPCDEntryDocprocsUnused::= 

fastESPCDEntryOperations::= 

fastESPCDEntryMode::= 

fastESPCDEntryCallbacks::= 

fastESPLogtransformer::= 

fastESPCompletionserver::= 

fastESPQRserver::= 

fastESPQRServerTable::= 

fastESPQRServerEntryIndex::= 

fastESPQRServerEntryNumFailedQueriesSystem::= 

fastESPQRServerEntryNumFailedQueriesTotal::= 

fastESPQRServerEntryNumFailedQueriesUser::= 

fastESPQRServerEntryNumQueries::= 

fastESPQRServerEntryNumRequests::= 

fastESPsearch::= 

fastESPdeploymentmanager::= 

fastESPEvents::= 

fastESPTrap::= 

fastESPConfs::= 

fastESPGroups::= 

fastESPCompl::= 

Data type 

Gauge32 

Gauge32 

Gauge32 

Gauge32 

OctetString 

Gauge32 

NODE 

NODE 

NODE 

TABLE 

Unsigned32 

Counter32 

Counter32 

Counter32 

Counter32 

Counter32 

NODE 

NODE 

NODE 

NODE 

NODE 

NODE 

NODE 

Description 

Number of document processors busy. 

Number of document processors in invalid state. 

Number of free document processors. 

Number of operations. 

The mode of the content distributor. 

Number of callbacks. 

QRServer table. 

Row index in the QRServer table. 

Number of failed queries (system). 

Total number of failed queries. 

Number of failed queries (user). 

Number of queries. 

Number of requests.

FeedingProxy variables 

Variable 

fastFeedingProxyFeederRecovery 

fastFeedingProxyFeederInSync 

fastFeedingProxyFeederSumDocuments 

fastFeedingProxyFeederErrDocuments 

fastFeedingProxyFeederDocumentsToBeFed 

fastFeedingProxyReceiverDiskLow 

fastFeedingProxyReceiverSumDocuments 

fastFeedingProxyReceiverErrDocuments 

SNMP MIB variables - Events/Traps 


Event/Trap 

fastESPIndexerStateChange:=fastESPIndexerEntryOperationalIndexingSuspended 

fastESPIndexerDiskLow::=fastESPIndexerEntryDiskLow 

fastESPIndexerResetIndex:=fastESPIndexerEntryOperationalResetIndex 

fastESPAgentStarted::= 

fastESPNctrlComponentStateEvent::= 

fastESPApplicationNumComponentsChange::= 

SNMP MIB variables - Groups 


Group 

fastESPApplicationGroup 

Data type 

Counter32 

Counter32 

Counter64 

Counter64 

Counter32 

OctetString 

Counter64 

Counter64 

Description 

Description 

Feeding proxy recovery status 

Feeding proxy in_Sync variable 

Sum of documents fed to the Feeder 

Sum of fed documents with errors to the feeder 

number of documents to be fed to the feeder from 

receiver 

Runs out of disk estimate variable 

Sum of documents received to the receiver 

Sum of received documents with errors to the 

reader 

Sent on state change of indexer, suspended/running. 

Sent on disk low condition in indexer. 

Sent on indexer performing reset-index. 

Trap triggered on SNMP Agent start. 

Trap triggered on component state change. 

Number of registered components in naming service changed. 

Objects 

Group of application objects: 

• FAST-ESP-MIB::fastESPApplicationNumQueries 

• FAST-ESP-MIB::fastESPApplicationNumDocuments 

• FAST-ESP-MIB::fastESPApplicationNumUpdates 

SNMP Agent 

99


100 

Group 

fastESPNctrlGroup 

fastESPIndexerGroup 

fastESPProcserverGrou 

Objects 

• FAST-ESP-MIB::fastESPApplicationNumComponents 

Group of nodecontroller objects: 

• FAST-ESP-MIB::fastESPNctrlComponentEntryName 

• FAST-ESP-MIB::fastESPNctrlComponentEntryOperationalState 

• FAST-ESP-MIB::fastESPNctrlComponentEntryDescription 

• FAST-ESP-MIB::fastESPNctrlComponentEntryPID 

• FAST-ESP-MIB::fastESPNctrlComponentEntryAdministrativeState 

Group of indexer objects: 

• FAST-ESP-MIB::fastESPIndexerEntryAPIQueueSize 

• FAST-ESP-MIB::fastESPIndexerEntryLastIndexSwitch 

• FAST-ESP-MIB::fastESPIndexerEntryAPIOperations 

• FAST-ESP-MIB::fastESPIndexerEntryFailedOperations 

• FAST-ESP-MIB::fastESPIndexerEntryPartialUpdate 

• FAST-ESP-MIB::fastESPIndexerEntryRemoveOperations 

• FAST-ESP-MIB::fastESPIndexerEntryStatusUpdates 

• FAST-ESP-MIB::fastESPIndexerEntryRemoveCollectionOperations 

• FAST-ESP-MIB::fastESPIndexerEntryUpdateOperations 

• FAST-ESP-MIB::fastESPIndexerEntryTotalNumDocuments 

• FAST-ESP-MIB::fastESPIndexerEntryTotalSize 

• FAST-ESP-MIB::fastESPIndexerEntryOperationalResetIndex 

• FAST-ESP-MIB::fastESPIndexerEntryOperationalIndexingSuspended 

• FAST-ESP-MIB::fastESPIndexerEntryDiskLow 

• FAST-ESP-MIB::fastESPIndexerEntryAdministrativeResetIndex 

• FAST-ESP-MIB::fastESPIndexerEntryAdministrativeIndexingSuspended 

• FAST-ESP-MIB::fastESPPartitionEntryNumDocuments 

• FAST-ESP-MIB::fastESPPartitionEntryTotalIndexingRuns 

• FAST-ESP-MIB::fastESPPartitionEntryVectorMemoryIndexing 

• FAST-ESP-MIB::fastESPPartitionEntryVectorMemory 

• FAST-ESP-MIB::fastESPPartitionEntryIndexedPerSecond 

Group of procserver objects: 

• FAST-ESP-MIB::fastESPStagesEntryName 

• FAST-ESP-MIB::fastESPStagesEntrySystemTime 

• FAST-ESP-MIB::fastESPStagesEntryUserTime 

• FAST-ESP-MIB::fastESPStagesEntryRealTime 

• FAST-ESP-MIB::fastESPStagesEntryPageSwaps 

• FAST-ESP-MIB::fastESPStagesEntryMemUsage 

• FAST-ESP-MIB::fastESPStagesEntryNumOK 

• FAST-ESP-MIB::fastESPStagesEntryNumError 

• FAST-ESP-MIB::fastESPPipelinesEntryName 

• FAST-ESP-MIB::fastESPPipelinesEntrySystemTime 

• FAST-ESP-MIB::fastESPPipelinesEntryUserTime 

• FAST-ESP-MIB::fastESPPipelinesEntryRealTime

Group 

fastESPIndexingDispatcherGroup 

fastESPContentDistributorGroup 

fastESPQRserverGroup 

fastESPIndexerEventGroup 

fastESPAgentEventGroup 

Objects 

• FAST-ESP-MIB::fastESPPipelinesEntryPageSwaps 

• FAST-ESP-MIB::fastESPPipelinesEntryMemUsage 

• FAST-ESP-MIB::fastESPPipelinesEntryNumOK 

• FAST-ESP-MIB::fastESPPipelinesEntryNumError 

• FAST-ESP-MIB::fastESPDocumentProcessorEntryQueueFull 

• FAST-ESP-MIB::fastESPStagesEntryResidentMem 

• FAST-ESP-MIB::fastESPStagesEntryVirtualMem 

• FAST-ESP-MIB::fastESPPipelinesEntryResidentMem 

• FAST-ESP-MIB::fastESPPipelinesEntryVirtualMem 

Group of indexing dispatcher objects: 

• FAST-ESP-MIB::fastESPCollectionEntryNumDocuments 

• FAST-ESP-MIB::fastESPCollectionEntryName 

• FAST-ESP-MIB::fastESPClusterEntryName 

• FAST-ESP-MIB::fastESPClusterEntryNumIndexingDispatchers 

• FAST-ESP-MIB::fastESPColumnEntryClusterIndex 

• FAST-ESP-MIB::fastESPColumnEntryColumnNumber 

• FAST-ESP-MIB::fastESPColumnEntryAPIQueueSize 

• FAST-ESP-MIB::fastESPCollectionEntryClusterIndex 

Group of content distributor objects: 

• FAST-ESP-MIB::fastESPCDEntryNumDocprocs 

• FAST-ESP-MIB::fastESPCDEntryDocprocsBusy 

• FAST-ESP-MIB::fastESPCDEntryDocprocsInvalid 

• FAST-ESP-MIB::fastESPCDEntryDocprocsUnused 

• FAST-ESP-MIB::fastESPCDEntryOperations 

• FAST-ESP-MIB::fastESPCDEntryCallbacks 

• FAST-ESP-MIB::fastESPCDEntryMode 

Group of query server objects: 

• FAST-ESP-MIB::fastESPQRServerEntryNumFailedQueriesSystem 

• FAST-ESP-MIB::fastESPQRServerEntryNumFailedQueriesTotal 

• FAST-ESP-MIB::fastESPQRServerEntryNumFailedQueriesUser 

• FAST-ESP-MIB::fastESPQRServerEntryNumQueries 

• FAST-ESP-MIB::fastESPQRServerEntryNumRequests 

Group of indexer events: 

• FAST-ESP-MIB::fastESPIndexerStateChange 

• FAST-ESP-MIB::fastESPIndexerDiskLow 

• FAST-ESP-MIB::fastESPIndexerResetIndex 

Agent event group: 

• FAST-ESP-MIB::fastESPAgentStarted 

SNMP Agent 

101


102 

Group 

fastESPNctrlEventGroup 

fastESPApplicationEventGroup 

SNMP MIB variables - Compliance 


Compliance 

fastESPStandardComplianceV1 

Objects 

Nodecontroller event group: 

• FAST-ESP-MIB::fastESPNctrlComponentStateEvent 

Application event group: 

• FAST-ESP-MIB::fastESPApplicationNumComponentsChange 

Group 

Compliance for a FAST ESP agent: 

• FAST-ESP-MIB::fastESPApplicationGroup 

• FAST-ESP-MIB::fastESPNctrlGroup 

• FAST-ESP-MIB::fastESPIndexerGroup 

• FAST-ESP-MIB::fastESPProcserverGroup 

• FAST-ESP-MIB::fastESPIndexingDispatcherGroup 

• FAST-ESP-MIB::fastESPContentDistributorGroup 

• FAST-ESP-MIB::fastESPQRserverGroup 

• FAST-ESP-MIB::fastESPIndexerEventGroup 

• FAST-ESP-MIB::fastESPNctrlEventGroup 

• FAST-ESP-MIB::fastESPApplicationEventGroup 

• FAST-ESP-MIB::fastESPAgentEventGroup

Chapter 

11 

Monitoring tool 

Topics: 

• About the monitoring tool 

• Monitoring tool navigation bar 

selections 

• Administration navigation bar 

selections 

• Adding the graphing engine 

• Creating and configuring cluster 

views 

• Creating a custom logger based 

on query count 

• Creating a custom logger based 

on standard alerts 

• Managing global parameters 

• XML data 

• Alerting e-mail configuration 

information 

• Monitoring ESP in a redundant 

data center setup 

• Configuring the ESP redundant 

data center monitoring tool 

This section describes the Monitoring Tool.


104 

About the monitoring tool 

The Monitoring Tool oversees one or more FAST systems; it unifies the status of each individual node in one 

or more deployed search solution system and renders results by way of a web interface as a set of graphs, 

tables, and views. 

The Monitoring Tool enables you to preconfigure and save multiple monitoring regimens. You can create 

these regimens or schedules by associating them to one or more plugins. 

The Monitoring Tool is included with the FAST ESP distribution; no installation is required. The Monitoring 

Tool typically resides on the same machine where the FAST ESP administrator interface is running at: 

http://:+3000/clarity 

The Monitoring Tool is fully integrated with the node controller (nctrl) to start up and shutdown with the other 

FAST ESP processes. 

Monitoring tool navigation bar selections 

After creating the new cluster, the Monitoring Tool main screen is displayed: 

Selection 

Administration 

Indexing 

Overview 

Content 

Overview 

Description 

This selection administers the Monitoring Tool. Start and stop monitoring, change the monitoring setup, 

and start monitoring of other FAST ESP clusters. Manage monitoring schedules, and view the logs from 

the Monitoring Tool to troubleshoot problems with monitoring. 

Refer to the Administration navigation bar selections table. 

Also referred to as Search Engine Overview. 

This selection gives a top down view of search and indexing performance/activity. It allows you to view 

the details about the search engines in the system including information about how many documents 

there are indexed on the different indexer nodes, average qps and response times on the search nodes, 

and how the documents are distributed internally on the indexers. 

The first table on the screen lists the QRServers and top-level dispatchers along with statistics and links 

to graphs. 

The next table (if multi-node setup) is a list of search/index information aggregated by row. 

The final table lists all search/index columns, the hosts in them, indexing activity, and various other 

statistics. 

Click on the extended view link for additional columns of information. 

This selection allows you to view content processing activity including crawler activity, if used. Look at 

statistics for all pipelines in use by the system, such as average throughput, number of errors, and so 

on. Statistics for individual document processors is also available. 

The pipeline table shows aggregated information for all processing activity in that pipeline.

Selection 

Cache 

Overview 

Alert Overview 

Graph Viewer 

Description 

The stage table shows aggregated statistics for each document processing stage.This table is highlighted 

such that docprocs working the most are more highlighted (this is a gradient). 

If graphing is enabled, the docs/sec graph will appear next to the stage table showing the current 

hourly throughput for each pipeline. 

The docproc table lists the same basic information as the pipeline table, except information is 

aggregated per docproc instead of pipeline. 

extended view: all docprocs are visible 

simple view: (default view) docprocs are aggregated per host 

If crawlers are available and collections are being crawled, there will be two additional crawler tables 

showing crawler statistics. 

If the cachestatus logger is in the schedule, this selection will show cache usage statistics for each 

fsearch in the system. 

This view can be filtered by row and column. 

This selection shows an overview of alert status history for the cluster. 

By default, you are presented with any outstanding alerts in the system. The Select drop-downs allow 

filtering as follows: 

• The first drop-down allows you to select an alert group (this will either be a host in the cluster, or 

an abstract group name such as DocumentProcessing, etc.). 

• The second drop-down filters what type of information you see. Possible choices are: 

Errors Only: (default) displays current errors Show All: displays all metrics being tracked Suspended 

Only: displays metrics that have been suspended - If a metric is suspended, no E-mail will be generated 

if it goes into an error/warning state - Metrics can be suspended using the "pause" icon, they can the 

be resumed using the "play" icon - The other icon will allow sending an acknowledgement E-mail 

message for the alert Exclude Services: filters out any rows that track services (polling service for 

statistics) 

Status history includes: 

Status: current state of a metric Actions: Suspend Alerting and Send Acknowledgement available 

Metric: name of the metric being tracked Value: current value for the metric % Time OK, Warning, 

Critical/Error: the percentage of time the metric was in a particular state 

When a metric enters an alert state, the following happens: 

• If the metric is suspended, it is ignored 

• If the metric is not suspended, the mailing list configuration will be filtered to determine where to 

send the E-mail. If this list is non-empty, an E-mail is generated and sent. Regardless of if an E-mail 

is sent, the contents of the message is logged to $FASTSEARCH/var/log/pymail/-/// 

When a metric recovers, the same happens, except this time a report is sent stating the metric has 

recovered. 

If the Monitoring Tool is configured to use graphing, the graph viewer allows you to compare multiple 

graphs from the system, to monitor developments over time. 

Graph views can be saved and recalled later via the save session link. 


Current graphs being viewed are cached via session/cookies; if you leave the page and then return, 

you are presented with the same graphs. 

105


106 

Associating clusters with the monitoring tool 

When you install the Monitoring Tool on a node where FAST ESP is running, the Monitoring Tool uses the 

configuration server to discover relevant configuration information for each cluster. Additional clusters can 

be added from the Cluster Management screen of the Administration navigation bar. 

Integration with FAST ESP 

The Monitoring Tool shares the services of the PHP-enabled web server that serves web pages of the FAST 

ESP administrator interface. It shares the look and feel of the FAST ESP administrator interface. For example: 

When you start up the Monitoring Tool for the first time, it integrates itself into the current FAST ESP node 

configuration (NodeConf.xml). 

Administration navigation bar selections 

This section describes the navigation bar selections if you select Administration from the the Monitoring 

Tool navigation bar. 

Selection 

Clarity 

Configuration 

Cluster 

Management 

Schedule 

Management 

Description 

Also referred to as Monitoring Tool Configuration. 

This selection allows you to configure global configuration settings (mainly front end related). Global 

settings alter how the monitoring tool operates, and how to navigate in the tool. 

Default values are used most often, but it is possible to change settings such as the port number 

used by the node controller. The node controller port is figured by using baseport + 3015. The 

default assumes you are running a standard installation where the node controller is running on the 

same host as the FAST ESP administrator interface (baseport = 13000). 

Default: 16015 

A cluster is a single installation of FAST ESP. 

The Monitoring Tool can be configured to monitor multiple installations from a single machine. 

A schedule is a sequential grouping of loggers.This process is similar to how a pipeline is a sequential 

grouping of document processors. 

It is recommended that the following loggers are in all schedules: 

• SearchStatus 

• ContentStatus 

• StandardGraphs

Selection 

Logger 

Management 

Monitor View 

Logs 

Description 

Refer to Logger Management for logger descriptions. 

This selection allows you to view configured loggers, as well as create custom versions. 

A logger is a customizable plugin component that logs collected data to disk, creates graphs, and 

tracks metrics for alerting purposes. 

The Monitoring Tool by default fetches a large amount of FAST ESP information. Most of this 

information is visible on the Indexing Overview and Content Overview.This information is available 

to all loggers. Cluster layout information is also available to loggers. 

If you want to customize any loggers, contact FAST Solution Services. 

Standard Loggers are: 

Adding the graphing engine 

CacheStatus: This drives the cache overview page. (It is suggested to use this during cache tuning, 

and disable once done as it results in polling every fsearch process in the system every .) 

ContentStatus: This drives the Content Overview page. 

Dispatcher Status: This logs some logline data for top-level dispatchers. 

DocApiStatus: This logs and graphs information for the document API. 

QueryCount: This tracks the query count for a specific query, and optionally sends an alert of the 

count drops below . (This logger should only be used with a newly configured values.) 

SearchStatus: This drives the Indexing Overview page. 

StandardAlerts:This tracks some standard alert values. Alert ranges can be specified in the 

parameter of this logger. This value refers to a configuration file in 

$FASTSEARCH/etc/clarity/config_data/. If this logger is used, search time/latency alerts/warnings 

will be triggered if values go outside of the cluster configured ranges. 

StandardGraphs: This logger creates the majority of graphs available to the Monitoring Tool by 

default. 

This selection returns you to the Monitoring Tool main screen. 

This selection provides a quick and easy view into the Monitoring Tool system logs. This view is 

similar to the FAST ESP Logs page, except it allows you to easily target log files for the Monitoring 

Tool running against a specific cluster. If you encounter monitoring issues, check this page to see 

if the Monitoring Tool itself is throwing errors. 

Advanced statistical graphing can be enabled in the Monitoring Tool with a third party package named rrdtool 

that you can download from the Internet. 

rrdtool (version required is 1.0.48 or 1.0.49) 

For Windows, visual .NET is required in order to enable graphing support. 

1. Open your browser to the rrdtool download screen. 

http://oss.oetiker.ch/rrdtool/download.en.html 

2. Download rrdtool source, and build it. 

3. Copy rrdtool.exe (Windows) or rrdtool (UNIX) binary into $FASTSEARCH/bin. 

4. On the Cluster Management screen, click the Edit Global Configuration link. 

5. Select Administration on the Monitoring Tool navigation bar. 

6. For the Enable Graph View feature: 


107


108 

a) Select Clarity Configuration on the Administration navigation bar 

b) Select Yes from the Enable Graph View drop-down menu 

c) Click Submit to apply your changes 

7. Restart the Monitoring Tool. 

Creating and configuring cluster views 

This section describes how to create and configure a cluster view. Note that you can also manage the 

Monitoring Tool cluster views by way of the clarity command-line utility. For details, go to $FASTSEARCH on 

the FAST ESP administrative node system, and run the following command: clarity -h 

1. Select Administration on the Monitoring Tool navigation bar. The Cluster Management screen is 

displayed: 

2. Select Create New Cluster on the Cluster Management screen. The Create Cluster Configuration 

screen is displayed: 

3. From the Create Cluster Configuration screen, enter values or accept the defaults for the fields.

4. Upon completion, click submit. Monitoring starts automatically if the Monitoring Tool is connected to the 


5. Restart the Monitoring Tool for modifications to the configuration to take effect. 

Create cluster configuration screen options 

Clarity options 

Clarity options 

Cluster Name 

Interface Port 

Log Time (0-60) 

Full Refresh Time 

Logging Schedule 

Mailing List Prefix 

Concurrent Sockets 

Collection Timeout 

Default 

datasearch 

16098 

0 

5 

Generic 

clarity 

10 

10 

Description 

Datasearch configuration options 


Datasearch Cluster 

Config Server Host 

Config Server Port 

A symbolic name to give the cluster. 

The port the Monitoring Tool runs on. 

Value is figured using + 3098. 

Use the default unless running multiple clusters from one machine. 

This value provides the second of the minute that the Monitoring Tool polls for 

information (if running multiple clusters, it is suggested you stagger this). 

Example: If monitoring two large installations from one machine, log time for 

first installation should be 0, and log time for second should be 30. 

Values must be between 0 and 59. 

If you provide a value of 0, log entry times appear as XX:XX:00. 

If you provide a value of 30, entry log times appear as XX:XX:30. 

This value provides how often in minutes the Monitoring Tool performs a full 

poll of information. 

This is relevant if CacheStatus logger is used. 

This value describes what group of loggers to use. 

This value provides the prefix appended to the distribution list for all mailings. 

Configuration of E-mail is via a configuration file that allows creating mailing 

lists that can do regular expression matching against distribution lists to 

determine if E-mail should be sent. 

Default 

This value is the number of concurrent socket connections that the Monitoring 

Tool has open for fetching information. 

For large clusters, the default might be 100 or more. 

This value provides the number of seconds the Monitoring Tool attempts to 

fetch data before timing out a collection. 

webcluster 

localhost 

16005 

Description 

This value describes the logical cluster in datasearch. 

The configuration server for the installation. 

The port the configuration server is running on. 


109


110 


Dispatcher Levels 

Minimum Datasearch Port 

Maximum Datasearch Port 

Default 

2 

13000 

17000 

Basic alerting configuration options 

Basic alerting configuration 

options 

Service Timeout 

Search Time Warning Level 

Search Time Alert Level 

Search Latency Warning Level 

Search Latency Alert Level 

Default 

5 

2 

5 

2 

5 

Description 

Description 

This value is the number of tiers of dispatchers in the system. 

A single node installation will have 1 tier, a typical multi-node 

installation will have 2 tiers. 

The baseport of the cluster. 

The baseport + 4000. 

Creating a custom logger based on query count 

The Monitoring Tool will only register services with itself if the 

port of the service is in the minimum and maximum port range. 

This value is the number of minutes that a service must be unresponsive 

before issuing a service down alert. 

If the StandardAlerts logger is used, warnings will be issued if average search 

time for search nodes enters these bounds. These bounds will be visible on 

the front end/graphs via a gradient in the avg-search-time cell on indexing 

overview. 

If the StandardAlerts logger is used, alerts will be issued if average search 

time for search nodes enters these bounds. These bounds will be visible on 

the front end/graphs via a gradient in the avg-search-time cell on indexing 

overview. 

If the StandardAlerts logger is used, warnings will be issued if average search 

time for search nodes enters these bounds. Graphs do not mark any of these 

values 

If the StandardAlerts logger is used, alerts will be issued if average search 

time for search nodes enters these bounds. Graphs do not mark any of these 

values. 

A logger can create graphs, log text data, create XML files describing current state of system (with SearchStatus 

and ContentStatus), and track metrics for alerting purposes (resulting in E-mails being generated). 

Loggers can define configuration parameters either through the Monitoring Tool web interface (used by 

QueryCount logger) or through an XML file referenced through the Monitoring Tool web interface (used by 

the StandardAlerts logger). 

Creating custom loggers is simple when you base your custom logger on an existing logger and define 

configuration values relevant to your search solution's custom deployment scheme.To create a custom logger 

based on Query Count: 

1. Select Administration on the Monitoring Tool navigation bar. The Cluster Management screen is 

displayed:

2. Select Logger Management on the navigation bar and the following screen is displayed: 

3. On the Logger Management screen, click the + action icon to add a duplicate Query Count logger. The 

Create Custom Logger screen appears: 

4. Provide the following information in the Create Custom Logger screen to create a logger based on 

methods used by the Query Count Logger: 

• Logger Name - The value you supply for Logger Name must be unique and limited to the following 

characters: letters, numbers, dashes, and underscores (no whitespace). Invalid logger names: databases, 

index. 

• Logger Description - Optional text field for holding a user-defined description. 

• Alert (int) - The alert value is either a positive integer as some threshold value or -1 to disable. If the 

query count drops below this value (if greater than 0), then an alert will be triggered. 

• Query (str) - The query value is a query string that will be put into the query attribute and sent to the 

QRServer. Example: The query Query : meta.collection:collecname will return a count of the number 

of documents in the collection . 

5. Click Submit to apply your changes. 


111


112 

Creating a custom logger based on standard alerts 

A logger can create graphs, log text data, create XML files describing current state of system (with SearchStatus 

and ContentStatus), and track metrics for alerting purposes (resulting in E-mails being generated). 

Loggers can define configuration parameters either through the Monitoring Tool web interface (used by 

QueryCount logger) or through an XML file referenced through the Monitoring Tool web interface (used by 

the StandardAlerts logger). 

The following sections describe how to create a custom logger based on StandardAlerts. The first subsection 

describes how to add the custom logger into the Monitoring Tool; the second subsection describes how to 

create a schedule. 

Adding the Custom File into the Logger 

1. Select Administration on the Monitoring Tool navigation bar and the Cluster Management screen is 

displayed. 

2. Select Logger Management on the navigation bar and the Logger Management screen is displayed. 

3. On the Logger Management screen, click the + action icon to add a duplicate StandardAlerts logger. 

The Create Custom Logger screen appears: 

4. Using an editor, create your own custom logger. You can use the default StandardAlerts.xml file to make 

modifications to default settings. The modified file must be located at: 

$FASTSEARCH/etc/clarity/config_data/ 

5. In the Config text box, fill in the name of the configuration file that contains your logger custom settings. 

6. Click submit to apply your changes. 

7. Complete the Creating a Schedule section. 

Default StandardAlerts.xml file 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Creating a schedule 

In order for a logger to be used for monitoring, it must be associated with a schedule. 

To associate a logger with a schedule: 


displayed. 

2. Select Schedule Management on the navigation bar and the Schedule Management screen is displayed: 

3. Select Create New Schedule and the Create New Schedule screen is displayed: 

4. Create a Schedule based on logger properties configured for the Generic Schedule. 

5. From the Available Loggers drop-down list, select the logger you created. Click the -> add button to add 

the logger to the Loggers In Schedule list. 

6. If necessary, use the up or down arrows to move a highlighted logger up or down in the schedule. Use 

the trashcan icon to remove a highlighted logger from the schedule. 

7. Click submit to apply your changes. 

Managing global parameters 

You can access the Monitoring Tool application parameters and system parameters that apply to all tool 

processes, monitoring all nodes, and all clusters. 

To edit the parameters: 


113


114 


displayed: 

2. Select Clarity Configuration on the Administration navigation bar and the Global Configuration screen 

is displayed: 

3. For certain global parameters, you must restart a cluster's particular Monitoring Tool process in order for 

new values to take effect. 

Global configuration screen selections 

Screen field 

Enable Graph View 

Enable Journal View 

Keystroke Navigatio 

Default 

Yes 

No 

No 

Description 

Use this option to enable graphing views within the Monitoring Tool. 

You must install rrdtool before you enable graph views. Refer to Adding the 

graphing engine. 

Use this option to enable journal view within the Monitoring Tool. 

Enable this option to use single keystrokes to navigate quickly over the search 

and content pages. The following table lists the relevant keystrokes. 

Quick Keys: 

A - jumps to alert overview 

C - jumps to content status page 

F - jumps to cache status page 

G - jumps to graph view page 

S - jumps to search status page

Screen field 

Enable Authentication 

Remote Links 

Clarity Theme 

PostgreSQL Port 

Node Controller Port 

Monitor View Refresh 

Time 

XML data 

Default 

Yes 

Passthrough 

Datasearch 5 

16070 

16015 

60 

Description 

Use this option to turn off and on the FAST ESP administrator interface 

authentication: 

A No value allows viewing and administering the Monitoring Tool without 

authentication.This is an advanced value; contact FAST Solution Services before 

turning off this option. 

Possible values: 

Redirect: If set, then clicking on hostnames or other icons in the Monitoring Tool 

(the link to FAST ESP services) will redirect directly to that page. Use this setting 

only if the hosts are directly available on the network running the web browser 

(or proxies are set up to achieve the same affect). 

Passthrough: If set, then FAST ESP service status pages will be passed through 

the Monitoring Tool interface (without images). 

This value specifies the color theme for the Monitoring Tool interface. Available 

themes are located in $FASTSEARCH/admin/www/clarity/themes. 

Example: Datasearch Blue 

This value represents the logical port number of the Storage Server. This is 

required for Journal view which uses PostgreSQL. 

Change this value only if you are using base ports other than the default for your 

FAST ESP installation. 

Note: Baseport is typically 13000. Check your installprofile.xml to ensure you 

are using the default port. 

This value determines the period that the Monitoring Tool uses for fetching monitor 

data. Data appearing in the Monitor View has been fetched within this period. 

Typically, you make this value larger for large clusters. An aggressive refresh 

rate can stress the HTTPD server that is receiving the requests. 

Example: 30 

Views that support XML-view contain a link on the main Overview screens. Click the link to 

obtain the XML file that drives the page being viewed. 

Alerting e-mail configuration information 

The Monitoring Tool uses the configuration file: 

$FASTSEARCH/etc/config_data/Notification/Notification.xml 

This configuration file must exist on the machine running the Monitoring Tool. 

Sample Notification.xml configuration file 

 

 


115


116 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Sending e-mails example 

To send all E-mail messages generated by any Monitoring Tool instance to user1: 

 

 

 

 

 

 

Monitoring two clusters example 

Mailing list prefix for the first cluster is: one; Mailing list prefix the second cluster is: two. 

The following file sends E-mails for one to one group, and two to another group: 

 

 

 

 

 

 

 

 

 

Defining reference lists example 

To define reference lists that are groups of people, and then include those lists in a regular 

list: 

 

 

 

 

... 

 

 

 

... 

 

 

 

 

The tag can be used inside and elements to include s. 

To include a in a , the included must come first. 

The supports attribute lists what features are supported for a user: html (can receive html 

E-mail), plain (can receive plain text E-mail), and attach (can receive attachments). 

By default, the Monitoring Tool will only send plain text E-mails; however, customizations 

can be made to E-mail any generic message in a logger that may contain attachments or 

html mail; contact FAST Solution Services for assistance. 

Monitoring ESP in a redundant data center setup 

One strategy for improving fault tolerance is to replicate a data center. The content of two ESP installations 

in separate data centers must then kept in sync.The benefits of such a setup is that a service can be provided 

to users even if faults or accidents disable one of the data centers. 

The feeding proxy (see the ESP Feeding Proxy Guide) is used to keep two data centers in sync. A feeding 

proxy is set up in front of each data center, and content will be delivered to both data centers. 

In some redundant data center setups, search consistency is also important. Consistency in this context 

means that a document which is introduced in search results must never disappear - even if a different data 

center evaluates the same query later. 

To provide consistency in redundant data centers, a monitoring component is introduced in ESP. This 

component will provide service level status for search and feeding. In cooperation with a custom or third-party 

clustering solution, this component will indicate (1) when a data center is down (and therefore when queries 

should be directed to the other data center), and (2) if a secondary data center is not in sync with a failing 

primary data center (search queries can then not be served until all remaining content is indexed). 

Configuring the ESP redundant data center monitoring tool 

The redundant data center monitoring tool is disabled by default and must be specifically configured for the 

needs of a particular setup. 

As the configurability of this tool is large, it is possible to use the tool for more than monitoring consistent 

redundant data centers, but such usage is not supported by FAST. 

1. Copy the directory $FASTSEARCH/adminserver/espmonitor/espmonitor to 

$FASTSEARCH/adminserver/webapps. 

2. Copy $FASTSEARCH/adminserver/espmonitor/monitor.xml into $FASTSEARCH/etc/monitor.xml. 

3. Modify the monitoring configuration. 

In monitor.xml, rule sets are defined. A rule set is a set of pre-programmed rules, which monitors the 

state of a service in ESP. There are two rule sets pre-defined in the example monitor.xml provided in 

ESP - "search" and "feeding". See Rule options for rule sets in monitor.xml and the comments in the 

provided monitor.xml file for documentation on the different rules you can add in a rule set. 

4. Configure the HTTP interface. 


117


118 

You will integrate with the monitoring tool via an HTTP REST web interface. This interface is enabled in 

the tag in monitor.xml. 

The attributes for the " " tag are: 

• enabled - true/false. This attribute must be set to true to enable the http interface. 

• username - A username for HTTP authentication. If the username is blank, HTTP authentication is not 

used. 

• password - A password used in HTTP authentication. 

5. Restart the adminserver: nctrl restart adminserver 

The monitor logs to $FASTSEARCH/var/log/adminserver/monitor/espmonitor.log. 

6. Query and integrate the monitor into your clustering solution. 

If the http interface is enabled, you can query the status of a rule set on 

http://:baseport+3089/espmonitor/rulesets/ 

Example: http://esp.example.com:16089/espmonitor/rulesets/search 

If the rule set passes successfully, an HTTP query on this URL will result in an HTML page with the status 

code 200. If the rule set fails, the status code of the returned HTML page is 503. 

Rule options for rule sets in monitor.xml 

The following rule options can be added in the rule sets which are defined in monitor.xml. See also the 

comments in the provided monitor.xml file. 

Option 

CountQrServers 

CountProcServers 

LowOnDisk 

ProcessRule 

FeedingProxyInSync 

Description 

This rule will fail if there are less than the given QRServers running in the system. 

Example: 

 

This rule will fail if there are less than the given processing servers running in the system. 

Example: 

 

Fails if one or several indexers are soon to go out of disk space. 

Example: 

 

Checks that a list of processes are running on a ESP node. 

Example: 

 

 

 

This rule fails if the indexed content in this data center is not up to date with the primary data 

center. 

Option 

IndexingSuspended 

SearchQuery 

FeedingProxyDiskShortage 

FeedingProxyErrorQueue 

FeedingProxyFeedingQueues 

IndexAlwaysGrows 

Description 

This rule fails if indexing is suspended on one or more node. 

Example: 

 

This rule performs a search for all documents in a collection or view and fails if the result set 

count is not in the range given by the attributes minResultSet and maxResultSet. 

Attributes: 

• minResultSet - The minimum document count in the result set. 

• maxResultSet - The maximum document count in the result set. 

• qrserverHost - The hostname of the QRServer to use. If not provided, the query will be 

performed on all QRServers. 

• qrserverPort - The port of the QRServer to use. Must be provided if the QRServer hostname 

is given. 

• collection - The collection to search in. 

• view - The view to search in, can not be set if collection is set. 

Example: 

 

This rule fails if the feeding proxy expects to run out of disk within the given time frame in days. 

Example: 

 

 

Fails if the size error-batch queue the the feeding proxy is larger than the attribute 

maxErrorBatches. 

Example: 

 

 

This rule fails if the size of the queue of not yet indexed document batches is larger than the 

attribute maxOutstandingBatches. 

Example: 

 

 

Fails if the document count of the index is less than the previous recording of the document 

count. 

Example: 


 

119

Chapter 

12 

Deleting documents from an index 

Topics: 

• Deleting a single document from 

the index (option 1) 

• Deleting a list of documents in 

one operation (option 2) 

• Deleting documents from an 

index (option 3) 

This section describes possible ways to delete documents from an index.


122 

Deleting a single document from the index (option 1) 

This procedure is suitable for deleting a single document from the index. 

Note: If either option 1 or option 2 is used for a collection configured with a crawler, the crawler might 

later re-add the document you removed if the document has changed. Option 3 should be used instead 

in this case. 

1. Make sure the environment variables are set correctly. On Windows, the environment variables have 

already been set. 

On UNIX: 


. setupenv.sh 

2. Use the command-line utility called docpush located in the $FASTSEARCH/bin directory. For example: 


./docpush -c -d -U http://www.xyz.com/ 

The document ID you use when deleting is the same you used when adding the document. A search result 

will contain a field called "contentid" that contains this ID 

Deleting a list of documents in one operation (option 2) 

This procedure is suitable for deleting a list of documents in one operation. 

Note: If either option 1 or option 2 is used for a collection configured with a crawler, the crawler might 

later re-add the document you removed if the document has changed. Option 3 should be used instead 

in this case. 

1. Make sure the environment variables are set correctly. On Windows, the environment variables have 

already been set. 

On UNIX: 


. setupenv.sh 

2. Create a text file containing all the internal document IDs of the documents you want to delete, one line 

per document ID. The internal document ID can be found by performing a search. 

The result field 'internalid' will contain the internal ID for the document. An example of an internal ID is 

c29904ed59ba5bbbd6c84a08a672e1af_generic. 

If you use the default search front-end (Search view from the FAST ESP administrator interface), select 

Advanced Controls and enable Show All Fields before searching. 

3. Use the indexeradmin command to submit the delete operation: 

$FASTSEARCH/bin/indexeradmin rdocs 

 

For example: 

indexeradmin rdocs /home/fast/docids.txt generic 1 

Refer to the indexeradmin tool chapter.

Deleting documents from an index (option 3) 

This procedure should be used when deleting documents that were added by the FAST Enterprise Crawler 

Refer to the Crawler Guide for detailed information on the crawleradmin and postprocess commands. 

1. Change the crawler configuration. Add any URIs or hosts you wish to remove to Exclude URIs or Host 

Excludes, respectively.This is to prevent the crawler from re-adding the documents after you have deleted 

them 

2. Apply the configuration changes. Do one of the following: 

• Click Apply in the administrator interface, if you changed the crawler configuration in the GUI. 

• Run crawleradmin -f if you changed the crawler configuration in 

an XML file. 

3. Stop the crawler by issuing the command: 

nctrl stop crawler 

4. Verify that all crawler processes have stopped by issuing the command: 

ps -auxwww | grep crawl 

5. Run the following command: 

postprocess -x -X -I master -R 

6. Start the crawler by issuing the command: 

nctrl start crawler 

Deleting documents from an index 

123

Chapter 

13 

Real-time search tuning 

Topics: 

• About real-time search tuning 

• Real-time search tuning 

configuration file 

• Optimizing for query performance 

• Reducing memory footprint 

• Reducing index latency 

This section provides information on how to tune RTS.


126 

About real-time search tuning 

The default configuration for the search engine represents a balance between indexing latency, search 

performance and resource usage (disk/memory). Certain applications require the search installation to be 

optimized in one of these directions. 

Real-time search tuning configuration file 

The rtsearchrc.xml configuration file provides many low-level configuration parameters for tuning search 

engines. Each search engine cluster in the system has its own configuration file. A search engine cluster 

consists of a set of search rows/columns that share the same index profile. The default search engine cluster 

is named webcluster. It should also be noted that some of the real-time search tuning can be made while the 

search engine is up an running through the indexeradmin and searchadmin tools. 

The rtsearchrc.xml file is located on the server node where the configuration server is running, in the following 

directory: 

$FASTSEARCH/etc/config_data/RTSearch//rtsearchrc.xml 

where is the name of the search engine cluster. 

Caution! This configuration file contains many low-level configuration parameters that should normally not 

be modified. Do not change any other parameters than those described in this chapter without consulting 

FAST Solution Services. 

If you change this configuration file, it is necessary to restart all the search nodes within your installation. 

If you have more extensive tuning needs than described here, contact FAST Solution Services. 

Optimizing for query performance 

Parameter 

docsDistributionPst 

docsDistributionMax 

docsDistributionMaxMB 

Recommended value 

"100,100,90,80" 

"" 

"-1" 

Description 

Refer to the detailed description in the Reducing index latency 

section. The indicated recommended value will imply slightly 

reduced indexing latency, but may imply a slight query performance 

improvement for large indices (million documents or more). 

Refer to docsDistributionPst in Reducing memory footprint. 

If set, this configures a maximum number of documents for each 

partition. The configuraiton is given as a comma-separated string, 

use "-" to signify no limit. For example, the value "-,1000,1000" will 

limit partitions 1 and 2 to at most 1000 documents each. When the 

limit is reached for all partitions, no more documents will be 

indexed. 

If set, this will limit the amount of documents in a single partition, 

based on the memory used by the indexing and search processes. 

The indexer measures how much memory is used per document 

during indexing and search, on average. This value is then used 

to calculate a maximum number of documents per partition, based 

on the configured max MB limit set by this parameter. For example, 

setting this parameter to 1000, will limit each partition to max 1000

Parameter 

fsearchCachePstDist 

indexActivationDelay 

partitionCachePstDist 

blacklistInterval 

maxBlacklistInterval 

Recommended value 

Description 

MB. If search uses 1000 bytes per document, this will limit the 

number of documents to 1 million. When the limit is reached for 

all partitions, no more documents will be indexed. 

"1,13,27,13,1,13,7,5,5,5,5,5" 

Use to adjust each cache size that reside within fsearch. The 12 

numbers adjust the following caches: BitVectorCacheSize, 

BooloccCacheSize, DictCacheSize, DocumentInfoCacheSize, 

FilteroccCacheSize, IntoccCacheSize, 

IntRangeBitVectorCacheSize, PhraseCacheSize, resultcachesize, 

resultattributescachesize, PosoccCacheSize and 

PhraseIdxCacheSize. 

"0" 

"" 

"20" 

"600" 

Reducing memory footprint 

Parameter 

docsDistributionPst 

docsDistributionMax 

Recommended 

value 

Description 

This parameter is used to introduce an artificial delay before 

activating the search process for a newly created index on a search 

node. This delay is done during the startup of the new fsearch 

process, thus providing a means to ensure that this startup stage 

takes the same time on all search nodes. When this parameter is 

zero, no such delay is used. Care should be taken to ensure that 

the delay is longer than the time it usually takes to start fsearch. 

A warning will be logged when a non-zero delay is shorter than 

the time it takes for fsearch to start up. 

This parameter can be used to adjust the percent of 

maxMBCacheSize to dedicate to each partition. Enter integer 

numbers in a comma separated string. The numbers represent 

the percentage per partition, starting with the smallest index on 

the left. 

The interval (in seconds) to check the blacklist dirty bit and rebuild 

the blacklist index, for one partition at a time. The minimum value 

is 5 seconds. 

The maximum number of seconds to wait between a complete 

rebuild of the blacklist index, irrespective of the blacklist dirty bit. 

"100,100,90,80" 

Refer to the detailed description in the Reducing index latency section. This is 

mainly related to applications with large amount of deep navigators. In such 

applications the memory usage will have a non-linear growth related to index 

size. Hence, two more evenly sized partitions will consume less memory than 

one large partition. 

"" 


The indicated recommended value will imply slightly reduced indexing latency. 

If set, this configures a maximum number of documents for each partition. The 

configuraiton is given as a comma-separated string, use "-" to signify no limit. 

For example, the value "-,1000,1000" will limit partitions 1 and 2 to at most 

1000 documents each. When the limit is reached for all partitions, no more 

documents will be indexed. 

127


128 

Parameter 

docsDistributionMaxMB 

lowMemoryIndexing 

indexerSwapDetect 

maxQueueSize 

maxMBCacheSize 

maxStaticPartitions 

minPartitions 

maxDynamicPartitions 

safetyMemoryPool 

docIndexType 

Recommended 

value 

"-1" 

"true" 

"-a N -t 1000" 

"20971520" 

"10" 

"1" "1" "1" 

"1 MB" 

"filehash" 

Description 

If set, this will limit the amount of documents in a single partition, based on the 

memory used by the indexing and search processes. The indexer measures 

how much memory is used per document during indexing and search, on 

average.This value is then used to calculate a maximum number of documents 

per partition, based on the configured max MB limit set by this parameter. For 

example, setting this parameter to 1000, will limit each partition to max 1000 

MB. If search uses 1000 bytes per document, this will limit the number of 

documents to 1 million. When the limit is reached for all partitions, no more 

documents will be indexed. 

If set to true, the indexing sub processes will not be allowed to use large buffer 

sizes. Typically, it should use around 15 MB of memory, instead of 70 MB. This 

will have some impact on indexing performance. 

The -a option specifies the fixed memory allocation for the indexer. 

"-a N" where (N*1000) indicates the maximum number of documents that may 

be handled per partition. The standard value is 10000, but it may be reduced 

to save memory footprint if you know that you will not reach more than `n' 

documents within this column (more precisely within the largest partition). 

Example: To obtain support for up to 2 million documents within the largest 

partition: 

indexerSwapDetect="-a 2000 -t 1000" 

Specify the input queue size in bytes for the internal Search Engine API 

operations. 

Note that lower value will increase the risk for congestion. Congestion will be 

reported via the Content API Callback mechanism. 

Default: 268435456 (250 MB) 

Maximum megabytes of memory to allow for search engine caching. The 

indicated value will reduce the cache from 100 to 10 MB. This will have impact 

on search performance. 

This change will reduce the number of incremental indexing partitions to 1.This 

will have certain impact on latency, and does only have an effect on memory 

footprint in smaller data volume installations. 

Note that this change implies that for new/updated documents to become 

searchable all the previous content will need to be re-indexed. Additionally, 

sending in even 1 document will trigger a re-index of all the documents. Hence, 

if considering this option it is recommended to use manual indexing 

(manualIndexLargerPart="1") or use suspend/resume indexing to prevent the 

indexer from starting to index new data before all changes are ready for 

indexing. 

Memory pool that will be freed if we run out of memory. Used to notify users 

and do a controlled shutdown of the Indexer. 

Which type of document index to use. Memory-based document index will store 

all document ids with accompanying information in memory (in a hash). filehash 

is the default, and is an internally developed disk-based hash. gigabase 

docindex is not longer supported.

Reducing index latency 

Parameter 

latency 

indexActivationDelay 

numberPartitions 

Recommended 

Value 

"normal" 

"0" 

"3" 

Description 

This parameter indicates how important latency is for the particular 

installation. "low" latency results in an overall increase in the system 

resources used to index content, so the system load will be higher. 

Valid values: "low", "normal", or "high" 

This parameter introduces an artificial delay before activating the 

search process for a newly created index on a search node. This 

delay is done during the startup of the new fsearch process, thus 

providing a means to ensure that this startup stage takes the same 

time on all search nodes.When this parameter is zero, no such delay 

is used. Be sure that the delay is longer than the time needed to start 

fsearch. A warning is logged if a non-zero delay is shorter than the 

time it takes for fsearch to start up. 

This parameter lists the number of partitions to distribute data across. 

Trade-off between freshness, disk size, and QPS. The maximum 

number of partitions a node can handle is 8. 

docsDistributionPst "100,100,100,100" 

This parameter controls the partition-based indexing scheme. In this 

scheme, a given percentage of all remaining documents is allocated 

to each partition. The largest partition receives a given percentage 

of all the documents in this search engine instance, while the second 

largest partition calculates its percentage based on the remaining 

documents after the largest partition has taken its share. 

normalizeDictionary 

indexerPriority 

normalizeMode 

"true" 

"high" 

"periodic" 


The recommended value for highest freshness is "100,100,100,100". 

This causes the search engine to perform aggressive incremental 

indexing, to try to move documents to the larger index partitions as 

fast as possible. This ensures that the smallest partition is kept as 

small as possible, and improves latency. 

This parameter disables normalization of the Index term dictionaries 

between partitions.This parameter provides faster indexing, but may 

have substantial impact on dynamic ranking. 

Set this parameter to "true" if you are using dynamic ranking. If you 

only need static ranking and/or sorting on field values, set it to false 

(default). 

This parameter increases the operating system priority for the indexer 

process. This may in some cases imply improved latency, but the 

effect depends on deployment scenario. 

Normalization mode. When set to "periodic", new normalized 

dictionaries will be generated at periodic intervals, outside of the 

normal indexing cycle. This sacrifices correct ranking (it will still be 

consistent) for no impact on indexing latency. If set to "synchronous", 

a new normalized dictionary will be generated every time a new index 

set is activated. This ensures correct ranking scores, but will impact 

index latency. 

129

Chapter 

14 

Configuring logging 

Topics: 

• About configuring logging 

• Log levels 

• Log destinations 

This section describes how to configure log levels in FAST ESP.


132 

About configuring logging 

The FAST ESP loggers can be configured to direct messages to various destinations, depending on the log 

level of the messages or the programs generating them. 

The logging configuration file $FASTSEARCH/etc/LoggerConfig.xml controls how much logging output is 

produced and where. The default LoggerConfig.xml defines a few standard log destinations: 

• "server": The system log, which is collected by the log server on the admin node and viewable through 

the FAST ESP administrator interface. 

• "file": Local log files in $FASTSEARCH/var/log, with log files rotated (archived) based on a size threshold. 

• "dailyfile": Local log files as above, but rotated once each night. 

• "stdout": The standard output of the process. 

The section directs the messages generated by the FAST ESP components to particular log 

destinations. The entry specifies log destinations and log level thresholds that 

apply to all programs. Other entries apply to the named "program" only, and those settings override those in 

the default entry. 

For example: 

 

 

 

 

 

This specifies that the "fsearch" program logs to the named destinations "server", "file" and "stdout". Only 

warning, error and critical messages are logged to the log server, while the two other destinations receive all 

messages regardless of level. 

To use daily log rotation instead of size-based rotation for the local log files, replace with . 

To control the amount of space consumed by logs, adjust the log rotation settings in the 

entries. 

Note: If logs are rotated by size (the "file" destination), there is an absolute bound on the size consumed 

by the log. If logs are rotated daily, individual log files could grow arbitrarily large. 

To use alternate settings for some programs, create new log destinations with the new settings. For example, 

to keep fewer archived log generations for programs generating high volumes of log messages: 

. . . 

 

$FASTSEARCH/var/debuglog 

1 

500000000 

 

. . . 

 

 

 

 

. . . 

This will log info, warning, error, and critical level messages to "dailyfile" logger, ensuring that (with the default 

settings) a 10 day record of the operation of the component is kept. Debug and verbose level messages are 

logged to an alternate log which is limited in size.

Log levels 

Log levels used in FAST ESP. 

The following table lists log levels (CRITICAL being the most severe) with a brief description. 

Log level 

CRITICAL 

ERROR 

WARNING 

INFO 

VERBOSE 

DEBUG 

Log destinations 

Description 

Critical is only used for persistent fault conditions that require immediate operator attendance. 

Critical messages indicate that a component fails to provide service or misses the required service 

level. 

Error is used for persistent fault conditions that the operator should look at within a few hours. 

Errors are faults that may have an impact on system performance or service level, but do not stop 

the component from providing service. 

Warning is used for persistent fault conditions that indicate problems with the system installation 

that the operator should know about.Typical usage of Warning is to inform a system operator about 

a system condition that over time may degrade system performance or cause the system to fail, 

or that indicates a faulty configuration. 

Info is used for logging significant and stable state changes in components and services, for instance 

start, stop or reconfiguration of components, resources or services. 

Verbose is used for logging low level information for component and services that is needed for 

tracing and analyzing service and component execution. Verbose gives high level information on 

state of processing logic, sessions and jobs as well as content data being processed. 

Debug is used for logging component or service level information that has value for developer and 

tech-support for analyzing a faulty system, understanding fault conditions or abnormal system 

behavior. 

Debug is normally disabled in a production system. 

Log destinations and options available in FAST ESP. 

filedestination 

Stores log messages into local files. 

Parameter 

directory 

backups 

rotatesize 

rotatetime 

Description 

The base directory where the log files are stored. For each named "program" producing 

log messages, there is a distinct subdirectory holding its log file. Log file names have the 

form program/program.log. 

The number of archive copies of the log file to save when log files are rotated. 

The size in bytes that the log file will grow to before it is rotated. 

The time of day when the log file will be rotated. 

Configuring logging 

133


134 

The default "dailyfile" log destination looks like this: 

 

$FASTSEARCH/var/log 

9 

0100 

 

This specifies that logs are created in $FASTSEARCH/var/log, that 9 old generations are 

kept and the logs are rotated at 0100 each night. 

netdestination 

Forwards log messages to a remote log server. 

Parameter 

hostname 

port 

timeout 

retrycount 

retryperiod 

For example: 

Description 

The name of the host where the log server is running. 

The port number where the log server is listening for connection. 

Timeout (seconds) when connecting to log server. 

Number of times to retry if attempt to connect fails. 

Delay (seconds) before retrying to connect. 

 

admin.example.domain.com 

16010 

10 

10 

20 

 

stdoutdestination 

Sends log messages to the standard output of the process 

This log destination has no configurable parameters. 

For example: 

Chapter 

15 

nctrl tool 

Topics: 

• nctrl tool 

This section describes the node controller (nctrl) tool.


136 

nctrl tool 

Node controller tool.This tool is used to control the nodecontroller that holds the esp processes and to perform 

general operation on the components (stop/start/kill/add/remove/suspend/resume). 

$FASTSEARCH/bin/nctrl [options] [commands] 

nctrl options 

Option 

-h 

-v 

-l 

-C 

nctrl commands 

Command 

start [process 1]...[process n] 

stop [process 1]...[process n] 

kill [process 1]...[process n] 

restart [process 1]...[process n] 

sysstatus 

add [process] 

Description 

Displays help information. 

Displays version information. 

32bit hexadecimal loglevel or one of: 

debug, verbose, info, warning, error 

Specifies config directory. 

Default: $FASTSEARCH/etc 

Description 

Start all or selected processes. 

Note that when the nctrl daemon is running, the start command can also start 

(resume) suspended processes.When the nctrl daemon is not running, nctrl start 

without arguments will not start any processes. 

Stop all (including nctrl daemon) or selected processes. 

Note that when the nctrl daemon is running, the stop command behaves identically 

to the suspend command. The processes will therefore remain stopped 

(suspended) even if the nctrl daemon is restarted. 

Example: To stop search on all indexer and search nodes: 

$FASTSEARCH/bin/nctrl stop search-1 

Kill (SIGKILL/TerminateProcess) selected processes. 

Restart selected processes. 

Display services running on the node including their status. 

Add another instance of type "process". The process must be configured for 

dynamic multiplication.

Command 

remove [process] 

suspend [process 1]...[process n] 

resume [process 1]...[process n] 

reloadcfg 

Description 

nctrl sysstatus output example 

Remove a process from configuration. The process must support dynamic 

multiplication. 

Suspends (stops) the specified processes. The processes must be resumed to 

be restarted. 

Resumes (starts) the previously suspended processes. 

Reloads the configuration for the running daemon from 

$FASTSEARCH/etc/NodeConf.xml. 

The following is a sample of nctrl sysstatus output for a single node installation: 

./bin/nctrl sysstatus 

Connecting to Node Controller at localhost:16015.. 

Issuing 'sysstatus' request to Node Controller.. 

Status for node : test.fast.no 

FASTSEARCH : /usr/local/datasearch 

Disk status : 79% used - 27.22GB free - 97.17GB used - 124.39GB total 

Module Name Process Name PID Status 

---------------------------- -------------------- -------- ------- 

Adminserver adminserver 17193 Running 

Cache Manager cachemanager 17243 Running 

Query Completion Server completionserver 17350 Running 

Config Server configserver 17147 Running 

Content Distributor contentdistributor 17229 Running 

Enterprise Crawler crawler 17652 Running 

FDMWorker fdmworker 17239 Running 

Web Server httpd 17142 Running 

RTS Indexer indexer 17251 Running 

Indexing Dispatcher indexingdispatcher 17391 Running 

Log Server logserver 17146 Running 

Log Transformer logtransformer 17610 Running 

Name Service nameservice 17137 Running 

Node Controller nctrl 17680 Running 

Document Processor procserver_1 20169 Running 

QRServer qrserver 20145 Running 

Resource Service resourceservice 17206 Running 

RTS Search search-1 20095 Running 

Storage Service storageservice 17158 Running 

WaLinkStorerReceiver walinkstorerreceiver 20077 Running 

WALookupDB walookupdb0 20050 Running 

WebAnalyzer webanalyzer 17711 Running 

nctrl tool 

137

Chapter 

16 

psctrl and doclog tools 

Topics: 

• psctrl tool 

• doclog tool 

This section describes the processor controller (psctrl) tool and the document 

log (doclog) tool.


140 

psctrl tool 

Processor controller tool. This tool is used to control the processor servers. 

$FASTSEARCH/bin/psctrl [options] [commands] 

psctrl options 

Option 

-C 

-P 

-v 

-h 

psctrl commands 

Command 

doctrace on 

doctrace off 

debug on 

debug off 

ping 

status 

statistics 

reset 

stop 

memprofile on 

memprofile off 

leakdetect 

flushstate 

Description 

Configuration server host name and port number. 

Default: localhost:16005 

Only execute commands for this processor server. 

Turn on verbose processing. 


Description 

Enable document tracing. 

Disable document tracing. 

Log at debug level. 

Log at status level. 

Ping the processor server to check if it is alive. 

Get the status from the processor server. 

Get the statistics from the processor server. 

Reset the container and reload modules. 

Shutdown processor server. 

Enable memory profiling for processors and pipelines. 

Disable memory profiling. 

A memory status/leakage report is written to stderr. 

Flush internal state in document processors.

doclog tool 

Document log. Get the status and the log of recently processed documents. By default, the logs from all 

processor servers registered with the configuration server are aggregated. 

$FASTSEARCH/bin/doclog [options] [commands] 

doclog options 

Option 

-C 

-P 

doclog commands 

Command 

-l 

-a 

-b 

-e 

-h 

Description 

Description 

Configuration server host name and port number. 

Default: default is loaded from $FASTSEARCH/etc/CSLocation.xml with 

fallback to localhost:16005 

Poll this processor server only. 

List IDs for which a document log is available per server. An ID may appear on several 

servers. 

Show all available document logs. The most recent log per ID is shown. 

Show all available batch logs. 

Show all document logs with errors. 


doclog and psctrl usage example 

To force reprocessing on all collected documents and logging of document processor activity, the crawler is 

stopped and debugging information is logged: 

psctrl status 

psctrl doctrace on 

psctrl debug on 

nctrl stop crawler 

postprocess –I master –R –v > ..\pp.log 

doclog -a 

psctrl and doclog tools 

141

Chapter 

17 

collection-admin tool 

Topics: 

• About collection-admin tool 

• collection-admin tool usage 

• Add or create collection 

• Modify collection 

• View collection 

• List collections 

• Delete collection 

• View pipeline stages 

• List all pipelines 

• List all modules 

• List all datasources 

• List all clusters 

• Clear collection 

• Global options for 

collection-admin 

This section describes the collection-admin tool.


144 

About collection-admin tool 

This administration tool can be used to perform FAST ESP administrative tasks regarding collections from a 

UNIX or Windows command-line. 

collection-admin tool usage 

This tool can be executed on any FAST ESP node. The collection-admin tool reads 

$FASTSEARCH/etc/esp4j.properties to find the location of the Adminserver and the default user/password 

for logging in to the Adminserver. 

The general command syntax for collection-admin is collection-admin 

[option] 

where indicates the requested operation, such as listing collections 

(listcollections). Some operations require options such as providing a collection name or 

pipeline. All available options (preceded with a single ‘-’) are listed for each operation. The 

operation is specified with the -m option. The available operations are listpipelines, 

viewpipeline, listmodules, listclusters, listdatasources, modcollection, addcollection, 

listcollections, viewcollection, and delcollection. 

Commands are case-sensitive. 

collection-admin tool example 

To list all collections: 

collection-admin -m listcollections 

Add or create collection 

addcollection Options 

$FASTSEARCH/esp4j/bin/collection-admin -m addcollection [options] [common-options] 

Option 

-n 

-s 

-p 

Description 

Required. Collection name. 

Associate a data source with the specified collection. 

You would only associate a crawler, not a DB connector or other push API as they do 

not run as a service. The crawler port can be found on the System Overview screen.You 

can also use the ‘listdatasources’ option to list all data source modules. 

Required. Associate the collection with the specified document processing pipeline 

(pipeline name). 

The cluster must always be part of the pipeline name. For example: 

-p "SiteSearch (webcluster)"

Option 

-d 

-c 

Modify collection 

Description 

Create a collection example 

Set an optional description for the collection. 

Required. Associate a search engine cluster with the collection. 

Create a collection named mycollection using the SiteSearch pipeline and the default 

Search Cluster (webcluster): 

bin$ ./collection-admin -m addcollection -n mycollection -p "SiteSearch 

(webcluster)" -d "MyCollection" -c webcluster 

19.418[--------------]100% Successfully deployed collection: mycollection 

modcollection Options 

$FASTSEARCH/esp4j/bin/collection-admin -m modcollection [options] [common-options] 

Option 

-n 

-s 

-p 

-d 

View collection 

viewcollection Options 

Description 


Associate a data source with the specified collection. 

You would only associate a crawler, not a DB connector or other push API as they do not 

run as a service. The crawler port can be found on the System Overview screen.You can 

also use the ‘listdatasources’ option to list all data source modules. 

Associate the collection with the specified document processing pipeline (pipeline name). 


-p "SiteSearch (webcluster)" 

Set an optional description for the collection. 

$FASTSEARCH/esp4j/bin/collection-admin -m viewcollection -n [common-options] 

Option 

-n 

Description 

View collection information example 


bin$ ./collection-admin -m viewcollection -n mycollection 

Name: mycollection 


145


146 

List collections 

Valid: yes 

Description: My Collection 

Cluster: webcluster 

Pipeline: SiteSearch (webcluster) 

Datasource: N/A 

listcollections Options 

$FASTSEARCH/esp4j/bin/collection-admin -m listcollections [-o] [common-options] 

Option 

-o 

Delete collection 

Description 

List all collections example 

Sort collections by name. 

bin$ ./collection-admin -m listcollections 

Collections: 

# Name 

- ------------ 

1 mycollection 

2 anothercollection 

delcollection Options 

$FASTSEARCH/esp4j/bin/collection-admin -m delcollection -n 

Option 

-n 

Delete collection example 

Description 


bin$ ./collection-admin -m delcollection -n mycollection 

72.646 [----------------------------------->]100% Successfully undeployed 

collection: mycollection 

View pipeline stages 

viewpipeline Options 

$FASTSEARCH/esp4j/bin/collection-admin -m viewpipeline -p [options] 

[common-options]

Option 

-p 

-o 

List all pipelines 

listpipeline Options 

Description 

Required. Pipeline name. 


-p "SiteSearch (webcluster)" 

Sort by pipeline name. 

$FASTSEARCH/esp4j/bin/collection-admin -m listpipelines [options] [common-options] 

Option 

-o 

List all modules 

listmodules Options 

Description 

Sort by pipeline name. 

$FASTSEARCH/esp4j/bin/collection-admin -m listmodules [options] [common-options] 

Option 

-a 

-l 

-t 

-o 

Description 

List all modules (default). 

List all module types. 

List only modules of the specified type. 

Sort by module name. 

The following module types are defined in a standard installation: 

• SearchDispatcher - search dispatcher 

• IndexingDispatcher - indexing dispatcher 

• NodeControl - node controller 

• CacheManager - cache manager 

• ResourceService 

• ProcessorServer - processor server module (document processing) 

• FilterAlerter - alert engine 

• ContentDistributor - content distributor module 

• WebAnalyzer - analysis linking 

• DataSource - Data sources (the Enterprise crawler is listed for this module type) 

• DeploymentManager - deployment manager 

• Search Engine - search engine and search dispatcher 

• RelBench 


147


148 

• FilterInterface 

List all datasources 

listdatasources Options 

$FASTSEARCH/esp4j/bin/collection-admin -m listdatasources [options] [common-options] 

Option 

-o 

List all clusters 

listclusters Options 

Description 

Sort by module name. 

$FASTSEARCH/esp4j/bin/collection-admin -m listclusters [options] [common-options] 

Option 

-o 

Clear collection 

Description 

Sort by cluster name. 

The clearcollection option removes all documents in a collection from the index. While documents are being 

removed it will not be possible to submit documents to the content API or content distributor. 

$FASTSEARCH/esp4j/bin/collection-admin -m clearcollection -n 

Option 

-n 

Clear collection example 

Description 


bin$ ./collection-admin -m clearcollection -n mycollection 

65.513 [--------------------------------------] 100% Successfully cleared 

collection mycollection 

Global options for collection-admin 

Use these options when invoking collection-admin. 

Option 

-B 

Description 

Baseport for FAST ESP installation.

Option 

-D 

-h 

-H 

-L 

-O 

-U 

-P 

-v 

Description 

Debug mode, DEBUG level logging to CONSOLE. 

Print the help message. 

Host for FAST ESP Adminserver. 

Verbose mode, INFO level logging to CONSOLE. 

Port for FAST ESP Adminserver. 

Username for FAST ESP Adminserver. 

Password for FAST ESP Adminserver. 

Version of collection-admin program. 


149

Chapter 

18 

indexeradmin, indexerinfo and searchadmin tools 

Topics: 

• indexeradmin tool 

• indexerinfo tool 

• searchadmin tool 

This section describes the indexer configuration and action (indexeradmin) tool, 

the indexer information (indexerinfo) tool, and the searchcontroller configuration 

and action (searchadmin) tool.


152 

indexeradmin tool 

Indexer configuration and action tool. 

$FASTSEARCH/bin/indexeradmin [options] [command options] 

Note: Changes made to the configuration are not persisted to the configuration file, which means that 

they are lost when restarting the indexer. For permanent indexer configuration changes, refer to the real 

time search tuning documentation. 

indexeradmin options 

Option 

-a 

-h or --help 

-v or --version 

--clustername 

--subsystem 

--columnid= 

--row= 

Default 

webcluster 

indexing 

indexeradmin commands 

Command 

blacklistinterval 

cleancollection 

defragfixml 

disabledebug 

disableft 

enabledebug 

enableft 

0 

0 

Description 

Description 

Run commands for all indexers (columns) in cluster. 

Print help. 

Display version information and exit. 

Identifies the name of the current search cluster. 

Identifies the name of the subsystem. 

Identifies the indexer column (column id), starting with 0 (zero). 

The rowid and columnid can be found from the Matching Engines screen 

in the administrator interface. Select the appropriate Search Column 

and select Status Details. The row/column ID is displayed. 

Identifies the indexer row (row id), starting with 0 (zero). 

The number of seconds to wait between sending information about deleted 

documents to the search processes. Low values mean better latency for remove 

operations, but higher system load. 

This will remove all documents of the collection given as argument. 

Compacts the data_fixml folder, but during normal operations, the regular periodic 

cleanup should be sufficient. 

Disables debug logging. 

Disables fault-tolerance for a specific indexer. Can be used to temporarily remove 

an indexer from an indexing column for maintenance. 

Enables debug logging for troubleshooting. 

Re-enables fault-tolerance for an indexer inside an indexing column.

Command 

flushsparequeues 

forcecleanup 

forceftstoragecleanup 

ftmaxstoragesize 

ftmaxfailures 

indexdelay 

makedynamic 

makestatic 

nextindextrigger 

purgedata 

purgeerrors 

rdocs 

 

reprocessft 

resetindex 

resume 

resumedocapi 

resumeheartbeat 

resumeindexing 

reindex 

selfcheck 

Description 

Empties all operation queues from indexer, should normally never be necessary. 

Forces an immediate cleanup cycle (the normal cleanup interval is 3-5AM), which 

flushes the errors&warnings data to disk, and compacts the data_fixml folder. Will 

potentially take a while to complete, and introduce a latency penalty while ongoing. 

Forces a fault tolerant cleanup. Per default, the storage is cleaned with one hour 

intervals. 

Sets the size of the fault-tolerant operation store, which determines how much 

downtime the system can survive. Value in MBs. 

Sets the number of failures from indexer rows that the system will tolerate and still 

proceed. 

The number of seconds to wait after there are un-indexed documents, before starting 

indexing. Low values mean better latency, but more indexing activity, thus more 

load on the machine. 

Opens up a partition for indexing, if it is static. 

Blocks indexing for a partition, and all partitions above it. Partition will never be 

re-indexed. 

Deprecated. Not used in default configuration. 

Deletes all content from the indexer. 

Flushes all document errors and warnings to disk, to var/document_errors.txt and 

var/document_warnings.txt. 

Not used for FAST ESP. 

If fault-tolerance is used, this command forces the indexer to replay all stored 

operations from the disk storage. 

Note: This is a reserved command, only intended for internal testing purposes. 

Re-indexes all content from raw data (fixml), resulting in a distribution of documents 

equal to the distribution percentage specified in the configuration. 

Resumes a suspended indexer. 

Enables content feeding if suspended. 

Resumes the periodic check for a master indexer. 

Resumes indexing on a suspended indexer. 

Reindex the partition given as argument. 

Performs an indexer status and content consistency check. 


153


154 

Command 

setdistpercent 

 

setdoctrigger 

 

setftstoragecleanupinterval 

setsilothreshold 

stop 

suspend 

suspenddocapi 

suspendheartbeat 

suspendindexing 

tracemode {enable,disable} 

indexeradmin commands 

indexerinfo tool 

Bulk loading example 

Description 

Sets the percentage of documents within a partition. 

Set the doc count trigger within for a partition. 

Changes the fault tolerant cleanup interval. Default is 3600 seconds. 

Sets the threshold for the creation of a silo index. Not supported. 

Use nctrl stop indexer instead of this command. 

This command halts the processing, the persistence to disk (causing fixml generation 

also to halt, refer to Content ), and the indexing of documents. 

Additionally, the data/data_fixml folder is prepared for backup. It does not flush the 

document queue. The documents that are not yet persisted by the indexer will stay 

in the queue, and they will be processed when the indexer is resumed. 

Suspends the docapi queue (disallows new content). 

Suspends the periodic check for a master indexer. This command makes the 

shutdown order of indexers in a fault tolerant system irrelevant, as the ability of a 

backup indexer to become a master is disabled. 

Suspends indexing (while fixml generation continues). 

A suspended indexer will not start any new indexing jobs, and will not accept any 

operations. 

The indexer continues to receive and persist documents while indexing is suspended. 

This command is useful when backing up the data/data_index folder and for saving 

system resources with massive document submits. 

Enabling tracemode makes the indexer produce a detailed trace log of all activities. 

Contact FAST Technical Support before using this command. 

Since the indexeradmin commands are asynchronous, nder heavy content load 

indexeradmin commands make take some time before actually taking effect. This can 

sometimes be useful, as in the case of bulk loading: 

indexeradmin ... suspendindexing filetraverser 

... indexeradmin ... resumeindexing 

In this case the indexer is guaranteed not to resume indexing until after the filetraverser 

finishes pushing data. 

Indexer information tool. This tool is used to retrieve information the indexer holds. 

$FASTSEARCH/bin/indexerinfo [options] [command options]

indexerinfo options 

Option 

-h or --help 


-V 

--clustername 

--subsystem 

--columnid 

--row 

Default 

indexerinfo commands 

Command 

doccount [collection] 

apiqueuesize 

reportcontents [collection] 

fixmlfillrate 

listcontrollers 

hasdocument [document] 

webcluster 

indexing 

0 

0 

Description 

Description 

Print help. 

Displays version level of information. 

Provides verbose level of information. 





in the administrator interface. Select the appropriate Search Column and 

select Status Details. The row/column ID is displayed. 


Returns the number of documents. If a collection is specified, return the document count 

for that collection. 

Example: To get the number of indexed documents in a collection, 

./indexerinfo doccount User1Test 

There are 46 docs in the collection 

User1Test. SUCCESS. 

Returns the number of batches in the indexer API queue, i.e. unprocessed batches. 

Example: To get indexer API queue size, 

./indexerinfo apiqueuesize The 

indexer reports 0 items in the api 

queue. 

SUCCESS. 

Returns a list of all documents, either for the indexer as a whole, or for a specific collection. 

Contents are reported as internal ids. 

Example: To dump all internal ids in a collection, 

./indexerinfo reportcontents 

User1Test 

Identifies the amount of valid (active) documents vs. the amount of invalidated 

(deleted/updated) documents. When a document is removed or updated, the old version 

is not removed immediately. Very low fill rates can impact indexing performance. 

Lists all search nodes connected to the indexer. 


Checks if an indexer contains a specific document. Document identifier is the internal _ id. 

155


156 

Command 

activeindexset 

sparequeuesize 

searchadmin tool 

Description 

Reports the current active index set on the indexer . 

Example: To get active index set 

./indexerinfo activeindexset Active 

index set on indexer is 0_0,1_0,2_1 

Returns the number of operations in the spare queue. 

Searchcontroller configuration and action tool.The searchcontroller configuration should be looked at. Changes 

made to the configuration are not persisted to the configuration file, which means that they are lost when 

restarting the searchcontroller. Some commands are asynchronous and use the same operation queue as 

the indexer. 

$FASTSEARCH/bin/searchradmin [options] [command options] 

searchadmin options 

Option 

-a 

-h or --help 


--clustername 

--subsystem 

--columnid 

--row 

Default 

webcluster 

indexing 

searchadmin commands 

Command 

restartfsearches 

setfsearchcachedistribution 

 

0 

0 

Description 

Description 

Run commands for all searchcontrollers (columns) in cluster. 

Print help. 

Display version information and exit. 





in the administrator interface. Select the appropriate Search Column and 

select Status Details. The row/column ID is displayed. 


This will restart the fseaches already running on the active index set. Note: This means 

that there will be some downtime no search. 

This will set the fsearch cache distribution. Used to adjust each cache size that reside 

within fsearch. The 12 (ex: "1,13,27,13,1,13,7,5,5,5,5,5") numbers adjust the following 

caches: BitVectorCacheSize, BooloccCacheSize, DictCacheSize, 

DocumentInfoCacheSize, FilteroccCacheSize, IntoccCacheSize,

Command 

setpartitioncachedistribution 

 

blacklist 

blacklist 

 

makestandalone 

connecttoindexer 

Description 

IntRangeBitVectorCacheSize, PhraseCacheSize, resultcachesize, 

resultattributescachesize, PosoccCacheSize and PhraseIdxCacheSize. 

Example: Set the fsearch cache distribution, 

./searchadmin setfsearchcachedistribution 

5,20,5,20,4,20,6,4,4,6,4,2 

This will set the partition cache distribution. Used to adjust the percent of 

maxMBCacheSize to dedicate to each partition. Enter integer numbers in a comma 

separated string.The numbers represent the percentage per partition, starting with the 

smallest index on the left. 

Example: Set the partition cache distribution, 

./searchadmin setpartitioncachedistribution 1,1,98 

This will blacklist the documents listed in the inputfile. 


This will blacklist the document with docid for a given partition and generation. 

This will be the searchcontroller standalone (will not be connected to the indexer). 

This will make searchcontroller connect to the indexer (this is the opposite of 

makestandalone). 

157

Chapter 

19 

boostbt tool 

Topics: 

• boostbt tool 

• boostbt tool command line 

options 

• Commands with arguments 

• Output commands 

• Administrative commands 

• boostbt examples 

This section describes the boost bulk tool named boostbt.


160 

boostbt tool 

The boost bulk tool, boostbt, allows you to read boost information from XML files in the legacy format, or in 

the FAST ESP format, which is aware of views and search profiles. It also allows you to download (export) 

or convert boosts to the FAST ESP format, and upload the boosts in both formats to the Boosts and Blocks 

Service in FAST ESP, with optional deployment or publishing. Information on views and collections (or search 

profiles) can be read from the input file, or be overridden with command-line options. 

boostbt tool command line options 

boostbt: [options] upload|deploy|publish|convert|validate|download|delete|reset|dtd 

... 

Option 

--espbaseport or -B 

 

--debug or -D 

--help or -E 

--esphost= or -H 

--verbose or -L 

--espport or -O 

--pass or -P 

--user= or -U 

--version or -v 

--collection or -c 

--legacy or -l 

--searchprofile or -s 

 

--type or -t 

--view or -V 

Description 

Specify the baseport for the FAST ESP installation. In a multi node 

installation, this is the base port of the admin node (the node running 

Configserver and Adminserver). 

Print debug output to the console. 

Print the help text. 

Specify the host running the FAST ESP Adminserver. 

Print verbose output to the console. 

Specify the port of the FAST ESP Adminserver. Use this option instead of 

the --espbaseport option if the Adminserver is using a non-standard port. 

The standard port for the Adminserver is +3089. If the FAST ESP 

installation uses 13000 as its base port, --espbaseport=13000 and 

--espport=16089 are the same. 

Specify the password for the FAST ESP Adminserver. Used for 

authentication. 

Specify the username for the FAST ESP Adminserver. Used for 


Print the version number. 

Select collection, overrides any collection information in input. 

Specify that the input is in the legacy format. 

Specify the search profile; overrides any search profile information in input. 

doc (document boosts) or query (query boosts). 

Select view, overrides any view information in input.

Commands with arguments 

Description of command options for boostbt 

Command 

publish 

upload 

deploy 

convert 

validate 

delete 

Argument 

--force or -f 

Output commands 

boostbt output commands 

Command 

convert 

validate 

download 

dtd 

Description 

Administrative commands 

boostbt administrative commands 

Upload boosts and publish them (deploy to both preview and published view). 

Upload boosts into boost database. If the -s (--searchprofile) option is specified, deploy to 

preview view of the specified profile. Otherwise, do not deploy to any view. 

Deploy uploaded boost and blocks to the views, or to the selected views/search profiles 

given by --view or --searchprofile. 

Convert legacy boosts to the new format. Use the --output option to specify the output 

filename. 

Validate input by parsing it, and applying set collection or view/search profile. Useful for 

inspecting what will be shipped to the FAST ESP Adminserver. 

Mark boosts defined in the input file for deletion. Use the deploy command to deploy the 

deletes. 

Deploy all boosts in view, collection or search profile, possibly redeploying any boosts that 

may already be deployed. 

Description 

Prints input in FAST ESP format. Implies --legacy. 

Prints the validated input. 

Download either all boosts, or boosts from the selected search profile, collection or 

view. 

Prints the DTD for the FAST ESP format. 

boostbt tool 

161


162 

Command 

reset 

boostbt examples 

Input file examples 

Description 

Converting legacy files to an output file: 

Undeploys all boosts in the published view of the search profile, and removes them 

from the database. 

boostbt convert ... --output 

Converting to inspect on the console: 

boostbt convert ... 

Importing boosts to database: 

boostbt upload ... 

Validating the input and printing the resulting boosts when they are changed to target a 

different view and collection (for query boosts and document boosts respectively): 

boostbt validate --view --collection ... 

Importing query boosts to a view: 

boostbt upload --type query --view ... 

Importing legacy boosts, setting the search profile for the query boosts and putting the 

document boosts in a different collection: 

boostbt upload --legacy --searchprofile --collection 

... 

Importing only the old query boosts, putting them in a search profile: 

boostbt upload --legacy --searchprofile --type query 

... 

Deploy/Deployment examples 

For all importing examples, deploy can be used instead of upload, or deployment can be 

done afterwards. 

Deploying to a view after uploading query boosts: 

boostbt deploy --view 

Deploying to the preview view after uploading query boosts: 

boostbt deploy --searchprofile 

Publishing examples 

For importing query boosts, the publish command is applicable: 

Publishing boosts in files as-is: 

boostbt publish ... 

Publishing boosts in files to a different search profile: 

boostbt publish --searchprofile ...

Exporting examples 

If download is the new export: 

Exporting query boosts from a search profile, print to a file: 

boostbt download --searchprofile --output 

Exporting document boosts from a collection: 

boostbt download --collection --output 

Exporting all boosts, and print them to the console: 

boostbt download 

Administration examples 

Print the DTD to a file for reference: 

boostbt dtd --output boostsandblocks.dtd 

Or to the console: 

boostbt dtd 

boostbt tool 

163

Chapter 

20 

rc tool 

Topics: 

• rc tool 

This section describes the rc tool.


166 

rc tool 

The rc tool is a command-line utility that works with FAST ESP to display information about runtime resources 

used by various components in the system. This utility is used to diagnose the system in a running FAST 

ESP installation. Use it if content feeding hangs, or logs do not include all the information needed to track 

down the cause of the problem. The rc tool allows you to query components in the system for low-level 

information about the running state. 

In contrast to the FAST ESP monitoring tool, understanding the output of an rc query frequently requires 

knowledge of the component implementation, and is therefore most useful to FAST ESP developers as a 

debugging tool. 

The rc tool can be used with all components in FAST ESP that export information, and the following query 

parameters: components (all, subset), and fields (scopes, allocs, values by type, version). It is possible to 

choose to report only scopes with active threads (typical for process hangs). 

The rc tool can be run against the nameserver(s) in the installation (default) or against explicitly specified 

ones (for remote diagnostics). 

The rc tool is located in $FASTSEARCH/bin/rc. 

How the rc tool works 

• For all middleware interfaces, code must be generated that stores method scope information (number of 

active threads, timing information). 

• C++, Java and Python APIs are used to monitor variables, allocations and setting named values. 

• C++, Java and Python libraries are used in which a middleware servant exports information on method 

scopes, variables and named values. 

• The rc tool calls the relevant middleware servants and present the information to the user. 

Main concepts 

The main concepts displayed by the rc tool are: 

• Scopes, which set various function scope information 

• Allocs, which set the number of allocations of a named resource 

• Values, which set a named resource to a given value 

Usage 

For example invocations and usage information, issue: 

$FASTSEARCH/bin/rc –h

Chapter 

21 

exportsearchprofile, importsearchprofile and view-admin 

tools 

Topics: 

• exportsearchprofile tool 

• importsearchprofile tool 

• view-admin tool 

This section describes the exportsearchprofile, importsearchprofile, and 

view-admin tools.


168 

exportsearchprofile tool 

The exportsearchprofile tool will export all configuration and data (except indexed documents) that are 

associated with a search profile to a file in the zip format. In combination with the importsearchprofile tool, it 

allows you to move all or parts of a search profile configuration from one FAST ESP installation to another. 

It can also be used to take backups of search profiles. 

Usage: 

Note: This tool is not intended for reconfiguring search profiles. To reconfigure search profiles, use the 

Search Business Center or the view-admin, boostbt and esp4jtool-dictman command line tools. 

$FASTSEARCH/bin/exportsearchprofile [options] 

Option 

--espbaseport= or -B 

--debug or -D 

--help or -h 



--espport= or -O 

--pass= or -P 



--ignoreboostsandblocks 

--ignoresynonyms 

--filename= or -f 

--ignoredocumentsummaries 

--includereporting 

--searchprofile= or -s 

 

Description 

Specify the baseport for the FAST ESP installation. In a multi-node 

installation, this is the base port of the administration node (the node 

running config server and admin server). 



Specify the host running the FAST ESP admin server. 


Specify the port of the FAST ESP admin server. Use this option instead 

of the --espbaseport option if the admin server is using a non-standard 

port. 

The standard port for the admin server is +3089. If the FAST 

ESP installation uses 13000 as its base port, --espbaseport=13000 and 


Specify the password for the FAST ESP admin server. Used for 


Specify the username for the FAST ESP admin server. Used for 



Select if you do not want to export boosts and blocks. 

Select if you do not want to export synonyms. 

Specify the output file. 

Select if you do not want to export document summaries. 

Export query statistics data. 

Specify the search profile to export.

Option 

--includeuserdata 

--ignorewatchedqueries 

Description 

Select if you do not want to export users and groups. 

Select if you do not want to export watched queries. 

In most cases, you can omit the --esphost, --espbaseport, --espport, --user, and --pass options. You only 

need to specify these options on the command-line if you want to override the default values. The default 

values are stored in: 

$FASTSEARCH/etc/esp4j.properties 

Exporting a search profile example 

To export a search profile named mysearchprofile to a file named exportedsearchprofile.zip: 

exportsearchprofile -f exportedsearchprofile.zip -s mysearchprofile 

importsearchprofile tool 

The importsearchprofile tool can import configuration and data (except indexed documents) that are associated 

with a search profile from a file. In combination with the exportsearchprofile tool, it allows you to move all or 

parts of a search profile configuration from one FAST ESP installation to another. It can also be used to 

restore backups of search profiles. 

Usage: 

Note: This tool is not intended for reconfiguring search profiles. To reconfigure search profiles, use the 

Search Business Center or the view-admin, boostbt and esp4jtool-dictman command line tools. 

$FASTSEARCH/bin/importsearchprofile [options] 

Option 

--espbaseport= or -B 

--debug or -D 

--help or -h 





Description 


installation, this is the base port of the administration node (the node 

running config server and admin server). 



exportsearchprofile, importsearchprofile and view-admin tools 



Specify the port of the FAST ESP admin server. Use this option instead 

of the --espbaseport option if the admin server is using a non-standard 

port. 






169


170 

Option 



--ignoreboostsandblocks 

--ignoresynonyms 

--filename= or -f 

--onlyinfo or -i 

--overwritegroups 

--ignoredocumentsummaries 

--mergegroups 

--overwritesearchprofile 

--overwriteusers 

--ignorereporting 

--ignoreuserdata 

--mergeusers 

--ignorewatchedqueries 

--overwritesynonyms 

Description 




Select if you do not want to import boosts and blocks. 

Select if you do not want to import synonyms. 

Specify the input file. 

Print the information from the import file. 

Overwrite existing groups. 

Select if you do not want to import document summaries. 

Merge existing groups. 

Overwrite the existing search profile. 

Overwrite existing users. 

Export query statistics data. 

Select not to import users and groups. 

Merge existing users. 

Select not to import watched queries. 

Overwrite synonyms (not merge). 





view-admin tool 

Importing a search profile examples 

To import a search profile from a file named mysearchprofile.zip: 

importsearchprofile -f mysearchprofile.zip 

To import and overwrite a search profile from a file named mysearchprofile.zip: 

importsearchprofile -f mysearchprofile.zip -o 

Use this managing tool to create, edit, delete and deploy views from the command-line.

Search profiles in the Search Business Center use views to realize their features. With view-admin, you can 

access features of the views currently not available through the Search Business Center.You can export the 

views to XML files and edit these manually, or you can create scripts to manipulate views/search profiles 

automatically. 

You can also use the import and export operations to backup and restore views, and to transfer views from 

one system to another. 

view-admin can not be used to export and/or import complete search profiles. Use the importsearchprofile 

and exportsearchprofile tools instead. 

Usage: 

$FASTSEARCH/bin/view-admin [general options] [options] 

where indicates the requested operation, such as creating or deleting a view. The operation is 

always specified with the -m option. 

General options 

--espbaseport= or -B 

 

--debug or -D 

--help or -h 







Description 


installation, this is the base port of the administration node (the node running 

config server and admin server). 





Specify the port of the FAST ESP admin server. Use this option instead of 

the --espbaseport option if the admin server is using a non-standard port. 













view-admin operations 

$FASTSEARCH/bin/view-admin [general options] [operation] [options] 

Name 

Create 

Operation 

-m create 


Description 

Create a view and specify which collection(s) it should search. 

171


172 

Name 

Createmissing 

Delete 

Deploy 

Export 

Import 

Info 

List 

Refresh 

Regenerate 

Status 

Undeploy 

Validate 

Operation 

-m createmissing 

-m delete 

-m deploy 

-m export 

-m import 

-m info 

-m list 

-m refresh 

-m regenerate 

-m status 

-m undeploy 

-m validate 

Description 

Ensure that the system is consistent with respect to default views 

for collections and clusters. 

Delete one or more views. 

Deploy one or more views to make them available for searching. 

Export view specifications to XML files. 

Import view specifications from one or more XML files. 

Show information about one or more views. 

List all views. 

Make changes in qtf-config.xml take effect without redeploying 

any views. 

Delete and then recreate all views. 

Show deployment status for one or more views. 

Undeploy one or more views to make them unavailable for 

searching. The views are not deleted, and can be redeployed. 

Validate the format of one or more view specification XML files. 

Create operation 

Use this operation to create a view and specify which collection(s) it should search. 

view-admin [options] -m create -v -c [-d] 

Operation option 

--view= or -V 

--collection= or -c 

--collections= or -k 

--deploy or -d 

Create a view example 

Description 

Specify the name of the view to create. 

Specify the collection that the view will search in. 

Specify a list collections that the view will search in. 

Deploy the view immediately. 

To create a view myview that uses the collections collection1 and collection2: 

view-admin -m create -V myview -k collection1 collection2 

Createmissing operation 

Use this operation to ensure that the system is consistent with respect to default views for collections and 

clusters: 

• For every collection, there should be a default view with the same name as the collection. 

• For every cluster, there should be a default view named espsystem. 

This operation will create any missing default views, and optionally deploy them. Normally, you should not 

have to use this operation. 

view-admin [options] -m createmissing [-d]



Description 

Deploy the created views. 

Create any missing default views example 

To create any missing default views and deploy them: 

view-admin -m createmissing -d 

Delete operation 

Use this operation to delete one or more views from the Adminserver. The views are deleted permanently. 

To make views unavailable for searching without deleting them, use the undeploy operation. 

view-admin [options] -m delete -V [-x] 



--views= or -w 

--allviews or -a 

--undeploy or -x 

Description 

Specify the name of the view to delete. 

Specify a list of views to delete. 

Delete all views. 

Undeploy and delete operation example 

To undeploy the view myview and then delete it: 

view-admin -m delete -v myview -x 

Undeploy views before deleting them. Without this option views 

that are deployed can not be deleted. 

Deploy operation 

Use this operation to deploy one or more views to make them available for searching. 

view-admin [options] -m deploy -V 





Deploy views example 

To deploy the views myview and otherview: 

Description 

view-admin -m deploy -w myview otherview 

Deploy all existing views. 

Export operation 

Use this operation to export view specifications to XML files. 


Specify the name of the view to deploy. The view must already 

exist in the system. 

Specify a list of views to deploy. The views must already exist in 

the system. 

173


174 

view-admin [options] -m export -V [-f ] 



--views= or -w 

 


--file= or -F 

Export view example 

Description 

Specify the name of the view to export. 

The exported view specification will be stored in a file named .xml 

in the current directory. 

Specify a list of view names to export. 

Each exported view specification will be stored in a file named .xml 


Export all views. 

Each exported view specification will be stored in a file named .xml 


The option can be used to control the names of the saved files. It should specify 

a list of files of the same length as the number of views being exported. 

For example: 

view-admin -m export -w view1 view2 -F v1.xml v2.xml 

will store view1 as v1.xml and view2 as v2.xml. If the list of views to export is 

longer than the list of files, the remaining views will be stored as .xml. 

To export the view myview to a file named myview.xml: 

view-admin -m export -V myview 

Import operation 

Use this operation to import view specifications from one or more XML files. 

view-admin [options] -m import -V [-o] [-d] 




--overwrite or -o 



Description 

Specify the name of the XML file containing the view specification to 

import. 

Specify a list of the XML files containing the view specifications to import. 

Overwrite the view(s) if they already exist. 

If you attempt to import a view that already exists, you will receive an 

error message, unless you specify the --overwrite option. 

Deploy the view(s) immediately after import. 

This option is synonymous with the --views/-w option.

Import view example 

To import the view specified in myview.xml, overwrite it if it already exists, and then deploy 

it: 

view-admin -m import -V myview.xml -o -d 

Info operation 

Use this operation to show information about one or more views. For each view you will see display name, 

created and last modified time, and collections references. 

Note: If you do not specify the -V or -w options, information about all views are shown. 

view-admin [options] -m info -V 





Info views example 

To show information about the view myview: 

view-admin -m info -V myview 

Description 

List operation 

Use this operation to list all views in a FAST ESP installation. 

view-admin [options] -m list 

This operation has no options. 

Specify the name of the view to show information about. 

Specify a list of views to show information about. 

Show information about all views. 

Refresh operation 

Use this operation to update all views with new settings from a configuration file, in most cases qtf-config.xml. 

Note that the changes made to qtf-config.xml will not take effect until you run this command. This will ensure 

that the changes take effect on all QR servers at the same time. 

view-admin [options] -m refresh 

This operation has no options. 

Regenerate operation 

Use this operation to first delete and then recreate all views.The existing information about which collection(s) 

a view uses, is carried over from the deleted views to the new views. 

view-admin [options] -m regenerate [-d] 




Description 

Deploy the regenerated views. 

175


176 

Regenerate views example 

To regenerate and deploy all views: 

view-admin -m regenerate -d 

Status operation 

Use this operation to show the deployment status for a view. 

For each view you can see if the: 

• View is deployed. 

• Deployed version of the view is up to date with the version stored in the Adminserver. A view is not up to 

date if it has been changed after deployment. The view must then be redeployed to be up to date. 

• Last deployment of the view was completed successfully. 

Note: If you do not specify the -V or -w options, deployment status for all views are shown. 

view-admin [options] -m status -V 


--view= or -v 



Show status example 

Description 

To show the deployment status for the view myview: 

view-admin -m status -V myview 

Specify the name of the view to show the deployment status. 

Specify a list of views to show the deployment status. 

Show deployment status for all views. 

Undeploy operation 

Use this operation to undeploy one or more views to make them unavailable for searching. The views are 

not deleted, and can be redeployed. 

view-admin [options] -m undeploy -V 





Undeploy views example 

To undeploy the views myview and otherview: 

view-admin -m undeploy -w myview otherview 

Property Description 

Specify the name of the view to undeploy. 

Specify a list of views to undeploy. 

Undeploy all views. 

Validate operation 

Use this operation to validate the format of one or more view specification XML files. 

view-admin [options] -m validate -V





Validate format view example 

Description 

To validate the format of the view specification in myview.xml: 

view-admin -m validate -V myview.xml 


Specify the name of the XML file containing the view specification 

to validate. 

Specify a list of the XML files containing the view specifications 

to validate. 

This option is synonymous with the --views/-w option. 

177

Chapter 

22 

qrserver-admin tool 

Topics: 

• qrserver-admin tool 

• qrserver-admin tool usage 

This section describes the qrserver-admin tool.


180 


The qrserver-admin tool is used to control how the Adminserver and QRServers work together to deploy 

views. 

The Adminserver maintains an internal list of all QRServers in the system. This list is used when views are 

deployed, to ensure that all QRServers receive the updated views. It also prevents the system from entering 

an inconsistent state if one or more QRServers are down. 

The qrserver-admin tool allows you to control how views are deployed to these QRServers, and enables you 

to register and unregister QRServers. 

In ESP 5.0 the list of QRServers was contained in the $FASTSEARCH/etc/esp4j.properties config file. It 

is no longer necessary to edit this file to add and remove QRServers, and there is neither any need to restart 

the Adminserver when a QRServer has been added or removed. 

qrserver-admin tool usage 

$FASTSEARCH/bin/qrserver-admin [general options] [operation options] 

qrserver-admin takes an operation argument, specified with the -m option. Each operation can in turn be 

modified by additional arguments. 

Name 

Register 

Remove 

Offline 

Standalone 

Down 

Up 

List 

Status 

Find 


Operation 

-m add 

-m remove 

-m offline 

-m standalone 

-m down 

-m up 

-m list 

-m status 

-m find 

--espbaseport= or -B 

 

--debug or -D 

Description 

Description 

Register a QRServer with the adminserver. 

Remove a registered QRServer. 

Notify the Adminserver that a QRServer is in offline mode. 

Notify the Adminserver that a QRServer is in standalone mode. 

Notify the Adminserver that a QRServer is down. 

Notify the Adminserver that a QRServer is working normally. 

List registered QRServers. 

Show status of registered QRServers. 

Find QRServers that aren't registered. 

Specify the baseport for the FAST ESP installation. In a multi node 

installation, this is the base port of the admin node (the node running 

Configserver and Adminserver). 

Print debug output to the console.


--help or -h 







Description 


Specify the host running the FAST ESP Adminserver. 


Specify the port of the FAST ESP Adminserver. Use this option instead of 

the --espbaseport option if the Adminserver is using a non-standard port. 

The standard port for the Adminserver is +3089. If the FAST ESP 

installation uses 13000 as its base port, --espbaseport=13000 and 


Specify the password for the FAST ESP Adminserver. Used for 


Specify the username for the FAST ESP Adminserver. Used for 







Register operation 

Use this option to register a QRServer with the Adminserver. When a QRServer has been added to the 

system, this will ensure that views are correctly deployed to the QRServer. Any active views will be 

automatically deployed to the QRServer when it is added. 

qrserver-admin -m add -n -o -p -c 


--nsname= or -n 

--hostname= or -o 

--port= or -p 

--cluster=or -c 

Register QRServer example 

Description 

The name of the QRServer as registered in the name 

service. 

The host name of the QRServer. 

The HTTP-port of the QRServer. 

The name of the cluster the QRServer belongs to. 

To register a QRServer with address qrnode.example.com:15100 and name service name 

esp/searchservice/qrnode_15100 that has been added to the webcluster cluster: 

qrserver-admin -m add -n esp/searchservice/qrnode_15100 -o 

qrnode.example.com -p 15100 -c webcluster 


181


182 

Remove operation 

Use this option to unregister a QRServer from the Adminserver. When a QRServer has been removed from 

the system, this will ensure that the Adminserver doesn't attempt to deploy views to the QRServer anymore. 

qrserver-admin -m remove -n 



Unregister QRServer example 

Description 


service. 

To register a QRServer with name service name esp/searchservice/qrserver_15100: 

qrserver-admin -m add -n esp/searchservice/qrserver_15100 

Offline operation 

Use this option is used to tell the Adminserver that a QRServer is in "offline" mode. A QRServer is in offline 

mode when it is running and accepting configuration changes, but its HTTP interface is not available to serve 

queries. 

qrserver-admin -m offline -n 



Set QRServer offline example 

Description 


service. 

To tell the Adminserver that a QRServer named esp/searchservice/qrserver_15100 is 

offline: 

qrserver-admin -m offline -n esp/searchservice/qrserver_15100 

Standalone operation 

Use this option is used to tell the Adminserver that a QRServer is in "standalone" mode. A QRServer is in 

standalone mode when it is running and serving queries, but not accepting configuration changes. 

qrserver-admin -m standalone -n 



Set QRServer standalone example 

Description 


service. 


standalone: 

qrserver-admin -m standalone -n esp/searchservice/qrserver_15100

Down operation 

Use this option is used to tell the Adminserver that a QRServer is down. This will cause the Adminserver not 

to deploy views to the QRServer. This option is typically used if a QRServer has been taken down for 

maintenance, but you still want to be able to deploy views to the other QRServers in the system. 

qrserver-admin -m down -n [-s] 



--stop or -s 

Set QRServer down example 

Description 

The name of the QRServer as registered in the name service. 

Stop the QRServer if it is running. 


down: 

qrserver-admin -m down n esp/searchservice/qrserver_15100 

Up operation 

This operation tells the Adminserver that a QRServer is in normal operating mode, i.e. it is taken up from 

either offline, standalone or down mode. All views are immediately deployed to the QRServer when it is taken 

up from either standalone or down mode. 

qrserver-admin -m up -n [-S] 



--start or -S 

Set QRServer up example 

Description 

The name of the QRServer as registered in the name service. 

Start the QRServer if it isn't running. 

To tell the Adminserver that a QRServer named esp/searchservice/qrserver_15100 is up: 

qrserver-admin -m up n esp/searchservice/qrserver_15100 

List operation 

List all QRServers known to the Adminserver 

qrserver-admin -m list 

List QRServers example 

List all known QRServers: 

qrserver-admin -m list 

Status operation 

Show status of QRServers known to the Adminserver. This will show whether the QRServers are running 

and whether their CORBA and HTTP interfaces are responding. 

qrserver-admin -m status 


183


184 

Show QRServer status example 

Show status of all known QRServers: 

qrserver-admin -m status 

Find operation 

Find and list QRServers that are installed but not registered with the Adminserver. A QRServer must be 

registered with the Adminserver for view deployment to work. 

qrserver-admin -m find [-a] 


--add or -a 

Find QRServers example 

Description 

Automatically register any found QRServers that aren't already registered. 

Show all installed QRServers and register those that aren't registered with the Adminserver: 

qrserver-admin -m find -a

FAST ESP Operations Guide

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?