Cloning ClientDataSet Cursors

Last winter I got around to writing a book about Delphi's ClientDataSet component. This book was published at the end of March, and is now available worldwide (there's a link at the end of this article).

This post covers an interesting advanced feature of ClientDataSets called cloned cursors. Chapter 11 of my book provides an in-depth discussion of cloning ClientDataSet cursors, but I have refrained from repeating any material from that chapter. Instead, this is a fresh article that I hope adds additional value above and beyond the book's discussion. In addition, this article includes a look of several valuable new features introduced in some of the latest versions of Delphi, including anonymous methods and anonymous threads.

Overview of Cloned ClientDataSet Cursors

Cloning can mean very different things, when it comes to software. In some cases, such as in Delphi's TJSONObject (JavaScript Object Notation Object) and .NET's DataSet, a clone is a complete and separate copy of the original instance.

Cloned ClientDataSet cursors are different. A cloned ClientDataSet cursor is a new reference to an existing ClientDataSet's internal storage.

That internal storage has two parts: Data and Delta. Data is an OLEVariant that holds a ClientDataSet's data, in its current state, which includes any changes that have been made to that data since the data was loaded into the ClientDataSet. Delta is also an OLEVariant, and it contains the change cache. The change cache is a description of just the changes that have been made to the data since it was loaded (so long as the change cache is being maintained, which is what happens when a ClientDataSet's LogChanges property is set to True, the default value).

That both Data and Delta are OLEVariants is important, in that OLEVariants are reference counted data structures. Importantly, when the last reference to an OLEVariant goes out of scope, it is removed from memory automatically, and this behavior plays an important role in ClientDataSet cloned cursors.

Cloning a cursor involves two ClientDataSets, at a minimum, but can include many more than that. One of these ClientDataSets is the source ClientDataSet, and it must be active. When a ClientDataSet is active, it has a reference to a Data and Delta. Even then, Data and/or Delta may be empty, but they exist and can hold data. For example, if they are both empty to begin with, adding a record to the source ClientDataSet will add one record to Data, as well as one to Delta. Data reflects the current state of the ClientDataSet's content with one record, and Delta reflects that one record was added.

The second ClientDataSet in a cloning operation is a ClientDataSet that will receive the cloned cursor, and I will call this ClientDataSet the target. When a source ClientDataSet's cursor is cloned, the target ClientDataSet's Data and Delta properties refer to the Data and Delta properties of the cloned (source) ClientDataSet.

From that point on, and changes to Data (including index creation) and Delta are reflected in both the clone (target) and the original (source). That does not mean, however, that these ClientDataSets will be exactly the same. Although both will refer to the same Data and Delta, there are many other properties that are independent. Specifically, each ClientDataSet referring to a shared Data and Delta will maintain their own values in the AppServer, Filter, Filtered, IndexName (or IndexFieldNames), MasterFields, MasterSource, OnFilterRecord, ProviderName, ReadOnly, RecNo (current record), and RemoteServer properties.

You clone a ClientDataSet's cursor by calling CloneCursor from the target ClientDataSet. This method has the following signature:

procedure CloneCursor(Source :TCustomClientDataSet; 
  Reset: Boolean; KeepSettings: Boolean = False);

You pass at least two parameters to CloneCursor. The first argument is a reference to the source ClientDataSet whose cursor you want to clone.

You use the second and third parameters to indicate whether the clone should adopt view-specific properties of the ClientDataSet it is cloning. If Reset is False and KeepSettings is True, the receiving ClientDataSet will adopt the AppServer, Filter, Filtered, IndexName (or IndexFieldNames), MasterFields, MasterSource, OnFilterRecord, ProviderName, ReadOnly, and RemoteServer properties of the source ClientDataSet. If you pass True in the second parameter (in which case you should pass False, the default, in the third parameter), the values of these properties in the clone will be reset to their default values.

Once a ClientDataSet has cloned another ClientDataSet, the two ClientDataSets will continue to refer to a common Data and Delta, until one of them is closed or clones a different ClientDataSet. Importantly, their properties will remain independent, regardless of the values of the second and third parameters of CloneCursor. From that point on, each can use a different filter, range, current record, and so forth, and therefore may reflect radically different views of the common data store.

There is one exception to the above statement. If you clone a ClientDataSet that is filtered (Filter is True and Filtered is a Boolean expression limiting which records are displayed), and you pass a value of False (no reset) in the second parameter of CloneCursor, the receiving ClientDataSet will also employ the filter and it cannot be removed. Under this situation, you can apply an additional filter to the clone, but you can never get back to a completely unfiltered view. This appears to be a bug. As a result, I recommend that you always pass True in the second parameter and manually apply properties from the source ClientDataSet to the target.

Once a source ClientDataSet is cloned, the target and the source are equal owners of the Data and Delta. Any one of them can post new changes to Data, and can undo changes in Delta.

When one of these ClientDataSets performs one of these actions, the result is immediately apparent in both. Furthermore, closing one of these ClientDataSets has no effect on Data and Delta, unless it is the last ClientDataSet that refers to Data and Delta, in which case Data and Delta will be released.

As implied earlier in this post, cloning is not necessarily about two ClientDataSets. It is perfectly possible to clone a source ClientDataSet, and then use the target as the source of yet another clone, and so on. Once again, all ClientDataSets that point to the same Data and Delta are equal. In other words, a source ClientDataSet has no special privileges with respect to the Data and Delta.

A Simple Example

The sample project SimpleClone demonstrates some basic elements of a cloned cursor. The main form of this application is shown in Figure 1.

Figure 1. The main form of the SimpleClone project

This project contains a ClientDataSet, named ClientDataSet1, which is opened when the project runs and displayed in the upper DBGrid. When you click the button labeled Clone ClientDataSet1, ClientDataSet2 is cloned from ClientDataSet1, and displayed in the lower DBGrid.

The following is the code that appears on the OnClick event handler of the button labeled Clone ClientDataSet1.

procedure TForm1.CloneButtonClick(Sender: TObject);
begin
  if ClientDataSet1.Active then
  begin
    ClientDataSet2.CloneCursor(ClientDataSet1, False, True); 
    CloneButton.Enabled := False;
    CloseButton.Enabled := True;
  end
  else
    raise Exception.Create('ClientDataSet1 not active. Cannot clone');
end;

Once cloning is complete, the views of these ClientDataSets are independent, in that they can have different sort orders and different current records, as shown in the Figure 2.

Figure 2. Two ClientDataSets share Data and Delta

If you now close ClientDataSet1, by clicking the button labeled Close ClientDataSet1, the ClientDataSet2 will remain active, and can continue working with the data by editing it and saving it to a file, or pointing to a properly configured DataSetProvider, after which it could call ApplyUpdates to apply its changes to an underlying database.

A More Complex Example

One technique that I've found particularly useful is to use cloned cursors to permit concurrent access to Data and Delta from two or more threads. So long as Data and Delta are treated as readonly, each thread can read Data (and Delta), perform its task, and then terminate.

Here is an example. You might create a Windows service that exposes an Internet Direct (Indy) TIdTCPServer component. This component can listen for requests from TIdTCPClient components. Each client connection is accepted on a different thread, and each client might request one or more records from a common ClientDataSet. Importantly, each client request is distinct from every other client request.

The thread on the Windows service that receives a client request can clone a cursor onto a globally available ClientDataSet, retrieve the data associated with that request and return it to the client, and then free the clone before terminating. So long as no code needs to make changes to Data or Delta, the ClientDataSet is thread safe.

If changes did need to be made to the shared data, the addition of a synchronization object, such as a TMultiReadExclusiveWriteSynchronizer could be used to create a thread-safe architecture for permitting updates.

I considered creating a somewhat simple example of a Windows service that returned data from a shared ClientDataSet to client applications using Indy socket components. However, doing so would raise a great many issues that I cannot address adequately in this post.

Instead, I decided to demonstrate the essential methods that can be used to retrieve specific subsets of data from a shared ClientDataSet in a multi-threaded environment. Fortunately, this demonstration also makes use of a couple of handy features that have been introduced into Delphi in the last several releases: Anonymous methods (Delphi 2009) and anonymous threads (Delphi XE).

Here's the setup. I created an application that needs to create physical files that contain only a subset of data from a shared ClientDataSet. These physical files are written by worker threads that retrieve a subset of data from the shared ClientDataSet, write that data to a file, and then terminate. These worker threads perform their task concurrently.

The main form of this application is shown Figure 3. Here you can see a button that will initiate the file creation process. Also shown is a listbox that contains the customer numbers for those customer's whose orders records will be written to individual files, one file per customer.

Figure 3. The main form of the CDSCloneToFile project

The code that extracts the data for a particular customer from the shared ClientDataSet is shown in the ReturnRecordsFromCDS method. ReturnRecordsFromCDS is called by a worker thread which passes to this method the customer number whose orders are contained in the shared ClientDataSet named ClientDataSet1.

function TDataModule1.ReturnRecordsFromCDS(CustNo: Integer): TClientDataSet;
var
  CDS: TClientDataSet;
  DSP: TDataSetProvider;
begin
  //Create the Result and a temporary ClientDataSet
  Result := TClientDataSet.Create(nil);
  CDS := TClientDataSet.Create(nil);
  try
    //Create the DataSetProvider and hook it up
    DSP := TDataSetProvider.Create(nil);
    DSP.DataSet := CDS;
    Result.SetProvider(DSP);
    try
      CDS.CloneCursor(ClientDataSet1, True);
      CDS.IndexName := 'CustNoIdx';
      CDS.SetRange([custno], [custno]);
      //The following populates the Result CDS
      //with just the orders for a particular customer
      Result.Open;
    finally
      DSP.Free;
    end;
  finally
    CDS.Free;
  end;
end;

ReturnRecordsFromCDS works by using a DataSetProvider and a ClientDataSet combination to retrieve only a range of records from a cloned cursor. These records, once returned to the thread that called ReturnRecordsFromCDS, are then written to a file using the returned ClientDataSet's SaveToFile method.

But this method also points out some interesting elements of a ClientDataSet on which a range is set. Namely, if you set a range on a ClientDataSet and then call SaveToFile, all of the records of the ClientDataSet are saved, not just those in the range.

To get around this, ReturnRecordsFromCDS creates a new ClientDataSet (Result), and associates it with a DataSetProvider (DSP). This combination is used to populate Result with a set of records from a DataSet (the DataSet to which the DataSetProvider's DataSet property is associated). This DataSet is a ClientDataSet (CDS) that is cloned from ClientDataSet1.

Once cloned, an index is set on CDS (this index is a feature of Data obtained as a result of the cloning process) and then a range is set. After that, calling Open on Result draws only those records from CDS in the range. Result is then returned to the calling method.

The fun stuff, if you will, can be found in the method that calls ReturnRecordsFromCDS. This method, named GenerateXMLFileForCustomer, is passed the customer number whose orders data needs to be written to a file. It performs its work using an anonymous thread. GenerateXMLFileForCustomer is shown in the following code listing.

procedure TForm1.GenerateXMLFileForCustomer(CustNo: Integer);
var
  AnonThread: TThread;
begin
  //Create the anonymous thread to run an anonymous method
  AnonThread := TThread.
     CreateAnonymousThread(
       procedure
       var
            CDS: TClientDataSet;
            FileToWrite: String;
       begin
         CDS := DataModule1.ReturnRecordsFromCDS(CustNo);
         try
           FileToWrite := FAppDir + '\' + IntToStr(CustNo) + '.xml';
           if FileExists(FileToWrite)then
                DeleteFile(FileToWrite);
                CDS.SaveToFile(FileToWrite);
           finally
             //Free the ReturnRecordsFromCDS Result
             CDS.Free;
           end;
       end);
  //Start the anonymous thread
  AnonThread.Start;
end;

An anonymous thread is created by passing a reference to a procedure to the TThread class method named CreateAnonymousThread. CreateAnonymousThread returns a TAnonymousThread (a TThread descendant), which you then execute by calling its Start method. Once the anonymous thread has completed the execution of the procedure, it terminates and frees itself.

In the case of GenerateXMLFileForCustomer, the procedure passed to CreateAnonymousThread is an anonymous method. An anonymous method is an unnamed method whose implementation appears inline in your code.

Importantly, an anonymous method closes over the local variables (including formal parameters) of the method in which it appears, maintaining the context of these values even after the method in which the anonymous method is defined has exited. This is called a closure, and it permits each execution of this anonymous method to maintain the value passed in the CustNo formal parameter.

Each call to GenerateXMLFileForCustomer creates a different worker thread that runs the anonymous method. Each anonymous method calls the ReturnRecordsFromCDS method, which returns a new ClientDataSet that holds the orders for the given customer. After ensuring that the target file does not already exists (it will be deleted if it does), the anonymous methods writes the contents of this ClientDataSet to the appropriately named file, after which it frees the ClientDataSet created by the call to ReturnRecordsFromCDS.

GenerateXMLFileForCustomer is called from the OnClick event handler of the button labeled Create Files for Customers. The following is the code that appears on that event handler:

procedure TForm1.CreateFilesButtonClick(Sender: TObject);
var
  i: Integer;
begin
  for i := 0 to ListBox1.Count - 1 do
    GenerateXMLFileForCustomer(StrToInt(ListBox1.Items[i]));
end;

Turning back our attention to the Windows service/client example I mentioned earlier, GenerateXMLFileForCustomer would be implemented on the OnExecute event handler of the TIdTCPServer component. Since that method executes in a different thread for each client connection, the use of the anonymous method would be unnecessary. All the OnExecute event handler would need to do is to read the customer number passed by the client, execute the ReturnRecordsFromCDS method passing to it this customer number, and then return the XMLData property of the returned ClientDataSet to the client application, after which the returned ClientDataSet would be freed.

Alternatively, the returned ClientDataSet could be returned to the requesting client using a stream. Whether text or binary data is returned to the client, the service might also encrypt and/or compress the data before returning it, in which case the client must know to decrypt and/or decompress the data it receives.

Summary

When you clone a ClientDataSet cursor, you permit an additional ClientDataSet to refer to the Data and Delta properties of an existing ClientDataSet. While all ClientDataSets that share the same cursor share the same Data and Delta, they do not share any other properties, permitting each ClientDataSet to maintain an independent view of the data.

For more information about Delphi in Depth: ClientDataSets, as well as for links for purchasing the book, please visit http://www.JensenDataSystems.com/cdsbook.

Copyright © 2011 Cary Jensen. All Rights Reserved

Author’s Note: This article originally appeared in the SDN Magazine # 109, published on May 5th, 2011.  For information about the Software Development Network, please visit http://www.sdn.nl/.