File system importer
Overview
Prerequisites
The tool requires the installation of Java 11 or a later version on the computer.
Installation
To install the synchronization tool, extract the FileSystemImporter.zip archive into a folder.
Usage
The tool is used to synchronize contents coming from Documentum, CSV files, SharePoint or a file system.
It is a JAR file, called on the command line, that takes a properties file as a parameter (-config option). The synchro_mode parameter of this file selects the processing mode: csv, xlsx, filesystem, files or bulk_import_csv.
The tool ships with example files:
- CSV synchronization (synchro-csv.properties).
- file system (synchro-fs.properties).
- Excel (synchro-xlsx.properties).
- files chosen one by one (synchro-files.properties).
- bulk import through CSV (synchro-bulk-import.properties).
For the synchronization of file trees, follow these steps:
- configure the server that receives the synchronization messages (settings.json).
- configure the properties files that specify the files to synchronize (synchro-fs.properties).
Server configuration
Depending on your architecture, you will have to configure the synchronization tool to connect to Any. The settings are done in the settings.json file, inside the config folder of the decompressed archive.
On-premise configuration
If you have an on-premise installation, you must configure the synchronization tool to connect to the Any ActiveMQ server that receives the synchronization messages.
In the settings.json file, edit the mdb-messaging part to set the values based on your environment. A particular parameter is the jms-server-address address, where the synchronization messages are published.
"mdb-messaging": {
"jms_server_address": "tcp://<ActiveMQ Server/Any>:61616",
[...]
Any cloud configuration
If you use Any in Cloud mode, the settings.json file is already configured for your environment. If you have to change the parameters, it is done in the aws-messaging part of the JSON, and more specifically in the sqs part.
Sending through the REST API
Instead of posting the messages to a queue, the tool can send each document to the Any REST API. This mode is enabled as soon as rest_url is set in the properties file; without this URL, the tool uses the message queue configured in settings.json. It applies to the csv, xlsx, filesystem and files modes.
- rest_url: Any URL.
- rest_login: user having the "REST API" role (see the REST API).
- rest_password: password, in clear text or encrypted. An encrypted password requires running the tool with the
-passkey <key>option. Without a key, the password is used as is.
To encrypt a password, run:
java -cp FileSystemImporter-<version>.jar eu.c6.crypto.GetEncryptedPassword -password <password> -passkey <key>
Copy the displayed value into rest_password.
CSV and Excel synchronization
For the CSV or Excel synchronization, proceed as follows:
- configure the server that receives the synchronization messages (settings.json).
- configure the CSV or Excel file to use and the properties to synchronize (synchro-csv.properties or synchro-xlsx.properties).
Synchronization configuration
For this guide, we assume the properties file used by the synchronization tool is synchro-csv.properties (CSV) or synchro-xlsx.properties (Excel).
Parameters:
- message_type: ADD, REPLACE, UPDATE or REMOVE (to be set: no default value).
- source_name: passed to Any and used by the synchronization flow.
- synchro_mode: "csv" for the CSV synchronization, "xlsx" for the Excel synchronization.
- linked_room_id: ID of the room the document is statically linked to.
- application: name of the application the document belongs to.
- input_queue_name: name of the queue to use to send the synchronization message.
- file: path to the CSV or Excel file holding the data to synchronize.
- columns_separator: separator between the data columns (CSV files).
- values_separator: separator between several values in a cell.
- repeating_separator: separator between the repeating values.
- attr_date: list of the columns holding dates, comma-separated.
- attr_boolean: list of the columns holding booleans, comma-separated.
- attr_int: list of the columns holding an integer, comma-separated.
- attr_repeating: list of the columns holding repeated values, comma-separated.
- date_format: date format (e.g. MM/dd/yyyy hh:mm:ss).
- root_folder_path: path of the root folder holding the content to import.
CSV and Excel columns
Here is the list of the available columns, handled automatically by Any:
- object_id: unique external identifier of the object.
- object_name: name of the content.
- root_id: external ID of the parent object (used for versioning).
- version: version number of the object.
- version_tags: version tags (repeating).
- relation_child: external ID of a linked child.
- relation_name: name of the relation.
- relation_versions: versions of the child (repeating).
Note: these relation columns can hold several values, separated by the values_separator.
- file_path: path of the primary content relative to root_folder_path.
- content_type: content type of the primary content.
- format_path: paths of the additional formats, separated by the values_separator.
- format_content_type: content types of the additional formats, separated by the values_separator.
- attachment_path: additional attachment paths, separated by the values_separator.
- attachment_content_type: content types of the attachments, separated by the values_separator.
- component_id: external ID of the package component.
- component_versions: versions of the component (repeating).
Note: these component columns can hold several values, separated by the values_separator.
Every other column you put in the CSV or Excel file is imported as a parameter and used by the synchronization mapping.
Running the synchronization tool
Once the configuration of the tool is finished, run the following command in a terminal:
java -jar FileSystemImporter-<version>.jar -config synchro-csv.properties
or, for an Excel file:
java -jar FileSystemImporter-<version>.jar -config synchro-xlsx.properties
The tool creates a logs folder and puts the run logs inside.
File system synchronization
For the file system synchronization, proceed as follows:
- configure the server that receives the synchronization messages (settings.json).
- configure the properties file that specifies the files to synchronize (synchro-fs.properties).
Synchronization configuration
For this guide, we assume the properties file used by the synchronization tool is synchro-fs.properties.
Parameters:
- source_name: passed to Any and used by the synchronization flow.
- synchro_mode: "filesystem" for the file system synchronization.
Batches
The file system synchronization is done by batches, each batch possibly having different parameters.
Each batch must be defined by its start parameters as filesystem.(batch_number). The batch number can hold digits, letters and an underscore.
Available parameters:
- root: root folder where the import starts.
- mapping: path that represents how the paths of the imported files are used to map the metadata (see the examples).
- include: "glob"-like syntax to define which files are sent (see the examples).
- exclude: "glob"-like syntax to define which files are NOT sent (see the examples).
- room: room the content is statically linked to.
- operation: ADD or REPLACE, to be set for each batch.
Examples:
filesystem.0001.root=./my_root_folder
filesystem.0001.mapping=${country}/${product}/
filesystem.0001.include=**/*.docx;**/*.doc;**/*.pdf;
filesystem.0001.exclude=**/BRAZIL/**;
filesystem.0001.room=my_room
filesystem.0001.operation=REPLACE
In this example, the content of the "my_root_folder" folder is imported.
The name of the first subfolder is mapped to the "country" metadata.
The name of the second subfolder is mapped to the "product" metadata.
The mapping is only done for the values wrapped in ${}. The other values are ignored.
The include rule specifies that only the docx, doc and pdf files are sent.
The exclude rule specifies that any file having a subfolder named BRAZIL is excluded.
For each file, Any also receives the file_name, file_ext, file_folder_(n) (name of each folder of the path) and file_folders (list of these folders) properties, usable by the synchronization mapping. The file path serves as the external object and root ID (sync.object_id and sync.root_id).
Running the synchronization tool
Once the configuration of the tool is finished, run the following command in a terminal:
java -jar FileSystemImporter-<version>.jar -config synchro-fs.properties
The tool creates a logs folder and puts the run logs inside.
Files synchronization
The files mode sends chosen files to Any one by one, with one message per file. The metadata is described in the properties file and is the same for all the files. For this synchronization, follow these steps:
- configure the server that receives the synchronization messages (settings.json).
- configure the properties file (synchro-files.properties).
Synchronization configuration
Parameters:
- synchro_mode: "files" for the files synchronization.
- message_type: operation applied (ADD, REPLACE or UPDATE, see the Sync API). Default: ADD.
- root_id: external root ID (
sync.root_id). By default, the file path. - object_id: external object ID (
sync.object_id). By default, the version. - version: version of the content (default: 0.1).
- source_name: by default
filesystem.<machine>.<user>. - sent_by: sender displayed in Any.
- room: room the files are statically linked to. Without a value, and if the tool is launched without a file as a parameter, a window asks for the room.
- application, input_queue_name: as for the CSV synchronization. Without a queue, the input queue (
input) is used. - create_as_new_object: true so that each version stays an independent document, without rebuilding the version tree.
- ignore_existing: true to always create the document, even if it already exists with the same identifiers.
- create_as_external_content: true to not send the file: Any references the content at its original location.
- create_package: true to create, after the files, a package whose components are all the files sent (latest version).
- task_type, task_recipients[n]: attach a task to the contents (see "Task" in the Sync API).
- properties.(name): metadata sent with each file. A multivalued value is written with indexes (
properties.authors[0],properties.authors[1]).
The wildcards ${FILENAME}, ${ROOT_ID}, ${OBJECT_ID} and ${VERSION} are replaced when each message is created; ${RANDOM_UUID} generates a random identifier in root_id and object_id. The batch number is generated automatically.
Running the synchronization tool
Pass the files to send as parameters:
java -jar FileSystemImporter-<version>.jar -config synchro-files.properties <file1> <file2>
Without a file as a parameter, a file selection window opens. The tool will create a logs folder and put the execution logs inside it.
Bulk import through CSV
The bulk import tool is used when you want to migrate large volumes of existing data coming from another system, to inject them into the Any base. This process was designed to maximize the import speed and requires a prior preparation of the data so it is compatible with the target configuration (aspects and data lists). The tool can only create new documents: it cannot be used to update or delete data. The import is done through messages posted in a queue handled by the Any message server. Warning: to avoid saturating the message server with the binary contents, the processing of the files must be done manually, through a copy of the files by the scripts provided by the tool or through a shared folder.
For more details, please see the "BulkImporter – User Guide" guide.
To configure the bulk import through a CSV, proceed as follows:
- configure the server that receives the synchronization messages (settings.json).
- export your Any configuration into a local folder (e.g. ./config).
- configure the CSV file that handles the bulk import settings (synchro-bulk-import.properties).
Configuration
For this guide, we assume the properties file used by the synchronization tool is synchro-bulk-import.properties.
Parameters:
- source_name: passed to Any and used as information.
- batch_number: name of the batch passed to Any and used as information. If the name is not specified, it is the name of the CSV file followed by the date in milliseconds.
- synchro_mode: "bulk_import_csv" for the bulk import through CSV.
- linked_room_id: ID of the room the document is statically linked to.
- application: name of the application the document belongs to.
- input_queue_name: name of the queue to use to send the synchronization message (default: "bulk_import").
- file: path to the CSV file holding the data to import.
- columns_separator: separator between the data columns.
- repeating_separator: separator between the multiple values.
- values_separator: separator between several values in a cell.
- root_folder_path: path of the root folder holding the contents to import.
- date_format: date format (e.g. MM/dd/yyyy hh:mm:ss).
- config_any: path to the ZIP file of the Any configuration export.
- detect_existing_root: tries to detect the existing roots in Any to attach the new versions to them, based on the synchronization root ID.
- copy_on_consume: true/false, determines the file copy strategy. If "true", the file is copied at the import. In this case, your source files must be reachable by the message server.
- script_folder_path: valid path to the folder where the scripts to copy the source files to the Any file system are created.
- script_any_root_folder_path: root path of the Any files, as configured in the settings.json file of the message server.
CSV columns
Here is the list of the available CSV columns, handled automatically by Any:
- object_id: unique external identifier of the object.
- name: name of the content.
- root_id: external ID of the parent object (used for versioning).
- version: version number of the object.
- version_tags: additional version tags (repeating).
- file_path: path of the primary content relative to root_folder_path.
- content_type: content type of the primary content.
- format_path: paths of the additional formats, separated by the values_separator.
- format_content_type: content types of the additional formats, separated by the values_separator.
- replace_path: replaces the start of the path of the target file with another path.
- (aspect.metadata): metadata to fill in. The metadata must exist in the Any base and in the exported configuration archive. In the case of metadata with a value mapping, you must give the code of the value.
Every aspect.metadata column you put in the CSV file is validated with the available configuration. These values must match the final values and are not transformed.
By default, an Any content is always "latest", unless you have several contents sharing the same "root_id". In this case, only the last one (from top to bottom in the file) is the "latest".
Running the bulk import through synchronization
Once the configuration of the tool is finished, run the following command in a terminal:
java -jar FileSystemImporter-<version>.jar -config synchro-bulk-import.properties
The tool creates a logs folder and puts the run logs inside. Script files to copy the binary files matching the imported contents are created in the scripts folder.
Once the import is finished, you must finalize the processing with the JobBulkImportPostActions job. See the "BulkImporter – User Guide" guide for more details.