Difference between revisions of "Recital Replication Getting Started"
Peterkelly (Talk | contribs) |
Yvonnemilne (Talk | contribs) |
||
| (13 intermediate revisions by 3 users not shown) | |||
| Line 1: | Line 1: | ||
=Recital Replication Summary= | =Recital Replication Summary= | ||
| − | + | Replication in Recital is a Master to Slave(s) configuration, where the updates are performed on one master server and published for slaves to subscribe. | |
| − | + | ||
| − | + | ||
== Master to Slave Replication (MSR) == | == Master to Slave Replication (MSR) == | ||
| Line 12: | Line 10: | ||
* Analytic - live data can be created on the master, while the analysis of the information can take place on the slave without affecting the performance of the master. | * Analytic - live data can be created on the master, while the analysis of the information can take place on the slave without affecting the performance of the master. | ||
* Long-distance data distribution - if a branch office would like to work with a copy of your main data, you can use replication to create a local copy of the data for their use without requiring permanent access to the master. | * Long-distance data distribution - if a branch office would like to work with a copy of your main data, you can use replication to create a local copy of the data for their use without requiring permanent access to the master. | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
= Enabling Replication = | = Enabling Replication = | ||
In order for transactions to be published in the queue table, replication must be set on first. This can be done by adding the ''SET REPLICATION ON'' command into a local or systemwide configuration file. The config.db stored in the ''conf'' directory which is in the root recital installation directory is used for systemwide or you can update the config.db in the local directory. You may also add the command to any 4GL program file. | In order for transactions to be published in the queue table, replication must be set on first. This can be done by adding the ''SET REPLICATION ON'' command into a local or systemwide configuration file. The config.db stored in the ''conf'' directory which is in the root recital installation directory is used for systemwide or you can update the config.db in the local directory. You may also add the command to any 4GL program file. | ||
| − | |||
When replication is turned on all tables already open and all tables opened afterwards are flagged for replication. You may turn replication off to disable replication, but if you want to filter out tables from replication it's better to use the allow and deny access control lists. Index files associated with the tables are also updated. Multiple index tag files are handled automatically. Single index files are sorted by name and then added onto the table name when stored with the transaction. | When replication is turned on all tables already open and all tables opened afterwards are flagged for replication. You may turn replication off to disable replication, but if you want to filter out tables from replication it's better to use the allow and deny access control lists. Index files associated with the tables are also updated. Multiple index tag files are handled automatically. Single index files are sorted by name and then added onto the table name when stored with the transaction. | ||
| − | |||
To turn replication on you must have installed the Replication service on the system already. If this hasn''t been done then a ''You must configure the replication service first with 'dbservice RRS''' error will be returned. | To turn replication on you must have installed the Replication service on the system already. If this hasn''t been done then a ''You must configure the replication service first with 'dbservice RRS''' error will be returned. | ||
| − | |||
The first time a table is flagged for replication all rows must be updated with a sequence number, if the table contains a large number of row this process may take some time. Once it's been the table is flagged so this process will not be done again. All new rows inserted after this point will automatically contain their own unique sequence number. For the life of a table these numbers will never be duplicated. | The first time a table is flagged for replication all rows must be updated with a sequence number, if the table contains a large number of row this process may take some time. Once it's been the table is flagged so this process will not be done again. All new rows inserted after this point will automatically contain their own unique sequence number. For the life of a table these numbers will never be duplicated. | ||
| − | |||
== Access Control Lists == | == Access Control Lists == | ||
| Line 62: | Line 50: | ||
=== replication.allow === | === replication.allow === | ||
This file describes the names of the tables which will be replicated if REPLICATION is SET ON. If this list is not empty then only tables matching will be replicated. | This file describes the names of the tables which will be replicated if REPLICATION is SET ON. If this list is not empty then only tables matching will be replicated. | ||
| − | |||
=== replication.deny === | === replication.deny === | ||
| Line 68: | Line 55: | ||
= Starting Replication Services = | = Starting Replication Services = | ||
| − | |||
| − | |||
== Subscriber == | == Subscriber == | ||
| − | The subscriber service is used by any system that needs to update | + | The subscriber service is used by any system that needs to update its databases from published data. In a Master Slave configuration each slave would start a subscriber service and to connect to the published data on the Master server. |
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
== Recital Database Server == | == Recital Database Server == | ||
| − | + | The Recital Database Server must be installed and running on the system publishing the data. | |
| − | + | ||
== ODBC Data Source == | == ODBC Data Source == | ||
In order for the replication services to connect to the Recital Database Server you must define a ODBC data source in the odbc.ini file. The Data Source name format is ''Recital Replication Service on'' plus the node name. The Driver name is Recital and the Database format is ''ODBC:RECITAL:'' The following options can also be added; | In order for the replication services to connect to the Recital Database Server you must define a ODBC data source in the odbc.ini file. The Data Source name format is ''Recital Replication Service on'' plus the node name. The Driver name is Recital and the Database format is ''ODBC:RECITAL:'' The following options can also be added; | ||
| − | |||
| − | |||
| − | |||
| − | |||
= Starting Master Slave Configuration = | = Starting Master Slave Configuration = | ||
A Master Slave configuration in Recital works with the published data stored in the queue table of the replication database. The Recital engine on the master system (the source of the database changes) writes all updates, inserts, deletes, recalls, packs and zaps into the queue table of the replication database. These transactions are stored in a XML format in the queue table. | A Master Slave configuration in Recital works with the published data stored in the queue table of the replication database. The Recital engine on the master system (the source of the database changes) writes all updates, inserts, deletes, recalls, packs and zaps into the queue table of the replication database. These transactions are stored in a XML format in the queue table. | ||
| − | |||
Slaves are configured to subscribe to the master and to execute the published events on the slave's local databases. The Master is dumb in this scenario. Once replication has been enabled, all statements are published in the replication queue. If required, you can configure the Master to only publish events that apply to particular databases or tables | Slaves are configured to subscribe to the master and to execute the published events on the slave's local databases. The Master is dumb in this scenario. Once replication has been enabled, all statements are published in the replication queue. If required, you can configure the Master to only publish events that apply to particular databases or tables | ||
| − | |||
Each subscribed slave will get a copy of the changes published since the last time it connected. Slaves keep a record of the position within the published queue that they have read and processed. This means that multiple slaves can be connected to the master and executing different parts of the same published data. Because the slaves control this process, individual slaves can be connected and disconnected from the server without affecting the master's operation. Also, because each slave remembers its position within the published queue, it is possible for slaves to be disconnected, reconnect and then 'catch up' by continuing from the recorded position. | Each subscribed slave will get a copy of the changes published since the last time it connected. Slaves keep a record of the position within the published queue that they have read and processed. This means that multiple slaves can be connected to the master and executing different parts of the same published data. Because the slaves control this process, individual slaves can be connected and disconnected from the server without affecting the master's operation. Also, because each slave remembers its position within the published queue, it is possible for slaves to be disconnected, reconnect and then 'catch up' by continuing from the recorded position. | ||
| − | |||
Both the master and each slave must be configured with a unique id. In addition, the slave must be configured with information about the master host name, RTQ database name and position within that file. | Both the master and each slave must be configured with a unique id. In addition, the slave must be configured with information about the master host name, RTQ database name and position within that file. | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
=== Staring Subscriber Service === | === Staring Subscriber Service === | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
The subscriber service will sleep the specified number of seconds between each set of transaction. Then it will connect to the master publisher and retrieve all published data since it last connected. For each transaction that it processes it will attempt to open the table in the required mode. For ZAP and PACK transactions it must open the table exclusively, for all others it will open it shared. If it can't find the table specified via the path or can't open it in the required mode an error will be returned. The Subscriber Service will perform all required locks on tables so existing Recital users can coexist. | The subscriber service will sleep the specified number of seconds between each set of transaction. Then it will connect to the master publisher and retrieve all published data since it last connected. For each transaction that it processes it will attempt to open the table in the required mode. For ZAP and PACK transactions it must open the table exclusively, for all others it will open it shared. If it can't find the table specified via the path or can't open it in the required mode an error will be returned. The Subscriber Service will perform all required locks on tables so existing Recital users can coexist. | ||
| − | == | + | === Starting replication services === |
| − | + | The replication services are administrated with the recitalreplication command. See [[recitalreplication]] for full details. | |
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
Latest revision as of 15:00, 8 June 2010
Contents
Recital Replication Summary
Replication in Recital is a Master to Slave(s) configuration, where the updates are performed on one master server and published for slaves to subscribe.
Master to Slave Replication (MSR)
This type of replication enables data from one Recital database server (called the master) to be published for replication for one or more Recital database servers (slaves) to subscribe. Replication is asynchronous - your replication slaves do not need to be connected permanently to receive updates published on the master. They just need to connect to the publisher when updating the subscription. Which means that updates can occur over long-distance connections and even temporary solutions such as a dial-up service. Depending on the configuration, you can replicate all databases, selected databases and even selected tables within a database.
Target uses for MSR in Recital include:
- Scale-out solutions - spreading the load among multiple slaves to improve performance. In this environment, all writes and updates must take place on the master server. Reads, however, may take place on one or more slaves. This model can improve the performance of writes (since the master is dedicated to updates), while dramatically increasing read speed across an increasing number of slaves.
- Data security - because data is replicated to the slave, and the slave can pause the replication process, it is possible to run backup services on the slave without corrupting the corresponding master data.
- Analytic - live data can be created on the master, while the analysis of the information can take place on the slave without affecting the performance of the master.
- Long-distance data distribution - if a branch office would like to work with a copy of your main data, you can use replication to create a local copy of the data for their use without requiring permanent access to the master.
Enabling Replication
In order for transactions to be published in the queue table, replication must be set on first. This can be done by adding the SET REPLICATION ON command into a local or systemwide configuration file. The config.db stored in the conf directory which is in the root recital installation directory is used for systemwide or you can update the config.db in the local directory. You may also add the command to any 4GL program file.
When replication is turned on all tables already open and all tables opened afterwards are flagged for replication. You may turn replication off to disable replication, but if you want to filter out tables from replication it's better to use the allow and deny access control lists. Index files associated with the tables are also updated. Multiple index tag files are handled automatically. Single index files are sorted by name and then added onto the table name when stored with the transaction.
To turn replication on you must have installed the Replication service on the system already. If this hasnt been done then a You must configure the replication service first with 'dbservice RRS error will be returned.
The first time a table is flagged for replication all rows must be updated with a sequence number, if the table contains a large number of row this process may take some time. Once it's been the table is flagged so this process will not be done again. All new rows inserted after this point will automatically contain their own unique sequence number. For the life of a table these numbers will never be duplicated.
Access Control Lists
If you wish to limit which tables are replicated you may do this with access control lists. Access control lists are defined in two files stored in the conf directory which is in the root recital installation directory. Comment lines can be added to these file by starting the line with # symbol. The access control specification is "directory path" or [database name!]tablename, where an optional database name may be specified. Wild cards may also be specified with ''*'' for all text matches and ''%'' for single character matches. Name expansion from environment variables can also be used.
| Example | Description |
|---|---|
| southwind!* | This would include all tables in the southwind database. |
| recital* | This would include all tables in all databases or directories that start with recital. |
| /tmp/* | This would include all tables in the directory "/tmp/" |
| ${DB_TMPDIR}/* | This would include all tables in the directory expanded from the environment variable DB_TMPDIR |
replication.allow
This file describes the names of the tables which will be replicated if REPLICATION is SET ON. If this list is not empty then only tables matching will be replicated.
replication.deny
This file describes the names of the tables which will NOT be replicated if REPLICATION is SET ON. If this list is not empty then any tables matching will be not be replicated.
Starting Replication Services
Subscriber
The subscriber service is used by any system that needs to update its databases from published data. In a Master Slave configuration each slave would start a subscriber service and to connect to the published data on the Master server.
Recital Database Server
The Recital Database Server must be installed and running on the system publishing the data.
ODBC Data Source
In order for the replication services to connect to the Recital Database Server you must define a ODBC data source in the odbc.ini file. The Data Source name format is Recital Replication Service on plus the node name. The Driver name is Recital and the Database format is ODBC:RECITAL: The following options can also be added;
Starting Master Slave Configuration
A Master Slave configuration in Recital works with the published data stored in the queue table of the replication database. The Recital engine on the master system (the source of the database changes) writes all updates, inserts, deletes, recalls, packs and zaps into the queue table of the replication database. These transactions are stored in a XML format in the queue table.
Slaves are configured to subscribe to the master and to execute the published events on the slave's local databases. The Master is dumb in this scenario. Once replication has been enabled, all statements are published in the replication queue. If required, you can configure the Master to only publish events that apply to particular databases or tables
Each subscribed slave will get a copy of the changes published since the last time it connected. Slaves keep a record of the position within the published queue that they have read and processed. This means that multiple slaves can be connected to the master and executing different parts of the same published data. Because the slaves control this process, individual slaves can be connected and disconnected from the server without affecting the master's operation. Also, because each slave remembers its position within the published queue, it is possible for slaves to be disconnected, reconnect and then 'catch up' by continuing from the recorded position.
Both the master and each slave must be configured with a unique id. In addition, the slave must be configured with information about the master host name, RTQ database name and position within that file.
Staring Subscriber Service
The subscriber service will sleep the specified number of seconds between each set of transaction. Then it will connect to the master publisher and retrieve all published data since it last connected. For each transaction that it processes it will attempt to open the table in the required mode. For ZAP and PACK transactions it must open the table exclusively, for all others it will open it shared. If it can't find the table specified via the path or can't open it in the required mode an error will be returned. The Subscriber Service will perform all required locks on tables so existing Recital users can coexist.
Starting replication services
The replication services are administrated with the recitalreplication command. See recitalreplication for full details.
