Monday, October 31, 2011

Cloning ClientDataSet Cursors

Last winter I got around to writing a book about Delphi's ClientDataSet component. This book was published at the end of March, and is now available worldwide (there's a link at the end of this article).

This post covers an interesting advanced feature of ClientDataSets called cloned cursors. Chapter 11 of my book provides an in-depth discussion of cloning ClientDataSet cursors, but I have refrained from repeating any material from that chapter. Instead, this is a fresh article that I hope adds additional value above and beyond the book's discussion. In addition, this article includes a look of several valuable new features introduced in some of the latest versions of Delphi, including anonymous methods and anonymous threads.

Overview of Cloned ClientDataSet Cursors

Cloning can mean very different things, when it comes to software. In some cases, such as in Delphi's TJSONObject (JavaScript Object Notation Object) and .NET's DataSet, a clone is a complete and separate copy of the original instance.

Cloned ClientDataSet cursors are different. A cloned ClientDataSet cursor is a new reference to an existing ClientDataSet's internal storage.

That internal storage has two parts: Data and Delta. Data is an OLEVariant that holds a ClientDataSet's data, in its current state, which includes any changes that have been made to that data since the data was loaded into the ClientDataSet. Delta is also an OLEVariant, and it contains the change cache. The change cache is a description of just the changes that have been made to the data since it was loaded (so long as the change cache is being maintained, which is what happens when a ClientDataSet's LogChanges property is set to True, the default value).

That both Data and Delta are OLEVariants is important, in that OLEVariants are reference counted data structures. Importantly, when the last reference to an OLEVariant goes out of scope, it is removed from memory automatically, and this behavior plays an important role in ClientDataSet cloned cursors.

Cloning a cursor involves two ClientDataSets, at a minimum, but can include many more than that. One of these ClientDataSets is the source ClientDataSet, and it must be active. When a ClientDataSet is active, it has a reference to a Data and Delta. Even then, Data and/or Delta may be empty, but they exist and can hold data. For example, if they are both empty to begin with, adding a record to the source ClientDataSet will add one record to Data, as well as one to Delta. Data reflects the current state of the ClientDataSet's content with one record, and Delta reflects that one record was added.

The second ClientDataSet in a cloning operation is a ClientDataSet that will receive the cloned cursor, and I will call this ClientDataSet the target. When a source ClientDataSet's cursor is cloned, the target ClientDataSet's Data and Delta properties refer to the Data and Delta properties of the cloned (source) ClientDataSet.

From that point on, and changes to Data (including index creation) and Delta are reflected in both the clone (target) and the original (source). That does not mean, however, that these ClientDataSets will be exactly the same. Although both will refer to the same Data and Delta, there are many other properties that are independent. Specifically, each ClientDataSet referring to a shared Data and Delta will maintain their own values in the AppServer, Filter, Filtered, IndexName (or IndexFieldNames), MasterFields, MasterSource, OnFilterRecord, ProviderName, ReadOnly, RecNo (current record), and RemoteServer properties.

You clone a ClientDataSet's cursor by calling CloneCursor from the target ClientDataSet. This method has the following signature:

procedure CloneCursor(Source :TCustomClientDataSet; 
  Reset: Boolean; KeepSettings: Boolean = False);

You pass at least two parameters to CloneCursor. The first argument is a reference to the source ClientDataSet whose cursor you want to clone.

You use the second and third parameters to indicate whether the clone should adopt view-specific properties of the ClientDataSet it is cloning. If Reset is False and KeepSettings is True, the receiving ClientDataSet will adopt the AppServer, Filter, Filtered, IndexName (or IndexFieldNames), MasterFields, MasterSource, OnFilterRecord, ProviderName, ReadOnly, and RemoteServer properties of the source ClientDataSet. If you pass True in the second parameter (in which case you should pass False, the default, in the third parameter), the values of these properties in the clone will be reset to their default values.

Once a ClientDataSet has cloned another ClientDataSet, the two ClientDataSets will continue to refer to a common Data and Delta, until one of them is closed or clones a different ClientDataSet. Importantly, their properties will remain independent, regardless of the values of the second and third parameters of CloneCursor. From that point on, each can use a different filter, range, current record, and so forth, and therefore may reflect radically different views of the common data store.

There is one exception to the above statement. If you clone a ClientDataSet that is filtered (Filter is True and Filtered is a Boolean expression limiting which records are displayed), and you pass a value of False (no reset) in the second parameter of CloneCursor, the receiving ClientDataSet will also employ the filter and it cannot be removed. Under this situation, you can apply an additional filter to the clone, but you can never get back to a completely unfiltered view. This appears to be a bug. As a result, I recommend that you always pass True in the second parameter and manually apply properties from the source ClientDataSet to the target.

Once a source ClientDataSet is cloned, the target and the source are equal owners of the Data and Delta. Any one of them can post new changes to Data, and can undo changes in Delta.

When one of these ClientDataSets performs one of these actions, the result is immediately apparent in both. Furthermore, closing one of these ClientDataSets has no effect on Data and Delta, unless it is the last ClientDataSet that refers to Data and Delta, in which case Data and Delta will be released.

As implied earlier in this post, cloning is not necessarily about two ClientDataSets. It is perfectly possible to clone a source ClientDataSet, and then use the target as the source of yet another clone, and so on. Once again, all ClientDataSets that point to the same Data and Delta are equal. In other words, a source ClientDataSet has no special privileges with respect to the Data and Delta.

A Simple Example

The sample project SimpleClone demonstrates some basic elements of a cloned cursor. The main form of this application is shown in Figure 1.
Figure 1. The main form of the SimpleClone project

This project contains a ClientDataSet, named ClientDataSet1, which is opened when the project runs and displayed in the upper DBGrid. When you click the button labeled Clone ClientDataSet1, ClientDataSet2 is cloned from ClientDataSet1, and displayed in the lower DBGrid.

The following is the code that appears on the OnClick event handler of the button labeled Clone ClientDataSet1.
procedure TForm1.CloneButtonClick(Sender: TObject);
begin
  if ClientDataSet1.Active then
  begin
    ClientDataSet2.CloneCursor(ClientDataSet1, False, True); 
    CloneButton.Enabled := False;
    CloseButton.Enabled := True;
  end
  else
    raise Exception.Create('ClientDataSet1 not active. Cannot clone');
end;
Once cloning is complete, the views of these ClientDataSets are independent, in that they can have different sort orders and different current records, as shown in the Figure 2.


Figure 2. Two ClientDataSets share Data and Delta

If you now close ClientDataSet1, by clicking the button labeled Close ClientDataSet1, the ClientDataSet2 will remain active, and can continue working with the data by editing it and saving it to a file, or pointing to a properly configured DataSetProvider, after which it could call ApplyUpdates to apply its changes to an underlying database.

A More Complex Example

One technique that I've found particularly useful is to use cloned cursors to permit concurrent access to Data and Delta from two or more threads. So long as Data and Delta are treated as readonly, each thread can read Data (and Delta), perform its task, and then terminate.

Here is an example. You might create a Windows service that exposes an Internet Direct (Indy) TIdTCPServer component. This component can listen for requests from TIdTCPClient components. Each client connection is accepted on a different thread, and each client might request one or more records from a common ClientDataSet. Importantly, each client request is distinct from every other client request.

The thread on the Windows service that receives a client request can clone a cursor onto a globally available ClientDataSet, retrieve the data associated with that request and return it to the client, and then free the clone before terminating. So long as no code needs to make changes to Data or Delta, the ClientDataSet is thread safe.

If changes did need to be made to the shared data, the addition of a synchronization object, such as a TMultiReadExclusiveWriteSynchronizer could be used to create a thread-safe architecture for permitting updates.

I considered creating a somewhat simple example of a Windows service that returned data from a shared ClientDataSet to client applications using Indy socket components. However, doing so would raise a great many issues that I cannot address adequately in this post.

Instead, I decided to demonstrate the essential methods that can be used to retrieve specific subsets of data from a shared ClientDataSet in a multi-threaded environment. Fortunately, this demonstration also makes use of a couple of handy features that have been introduced into Delphi in the last several releases: Anonymous methods (Delphi 2009) and anonymous threads (Delphi XE).

Here's the setup. I created an application that needs to create physical files that contain only a subset of data from a shared ClientDataSet. These physical files are written by worker threads that retrieve a subset of data from the shared ClientDataSet, write that data to a file, and then terminate. These worker threads perform their task concurrently.

The main form of this application is shown Figure 3. Here you can see a button that will initiate the file creation process. Also shown is a listbox that contains the customer numbers for those customer's whose orders records will be written to individual files, one file per customer.


Figure 3. The main form of the CDSCloneToFile project

The code that extracts the data for a particular customer from the shared ClientDataSet is shown in the ReturnRecordsFromCDS method. ReturnRecordsFromCDS is called by a worker thread which passes to this method the customer number whose orders are contained in the shared ClientDataSet named ClientDataSet1.

function TDataModule1.ReturnRecordsFromCDS(CustNo: Integer): TClientDataSet;
var
  CDS: TClientDataSet;
  DSP: TDataSetProvider;
begin
  //Create the Result and a temporary ClientDataSet
  Result := TClientDataSet.Create(nil);
  CDS := TClientDataSet.Create(nil);
  try
    //Create the DataSetProvider and hook it up
    DSP := TDataSetProvider.Create(nil);
    DSP.DataSet := CDS;
    Result.SetProvider(DSP);
    try
      CDS.CloneCursor(ClientDataSet1, True);
      CDS.IndexName := 'CustNoIdx';
      CDS.SetRange([custno], [custno]);
      //The following populates the Result CDS
      //with just the orders for a particular customer
      Result.Open;
    finally
      DSP.Free;
    end;
  finally
    CDS.Free;
  end;
end;
ReturnRecordsFromCDS works by using a DataSetProvider and a ClientDataSet combination to retrieve only a range of records from a cloned cursor. These records, once returned to the thread that called ReturnRecordsFromCDS, are then written to a file using the returned ClientDataSet's SaveToFile method.

But this method also points out some interesting elements of a ClientDataSet on which a range is set. Namely, if you set a range on a ClientDataSet and then call SaveToFile, all of the records of the ClientDataSet are saved, not just those in the range.

To get around this, ReturnRecordsFromCDS creates a new ClientDataSet (Result), and associates it with a DataSetProvider (DSP). This combination is used to populate Result with a set of records from a DataSet (the DataSet to which the DataSetProvider's DataSet property is associated). This DataSet is a ClientDataSet (CDS) that is cloned from ClientDataSet1.

Once cloned, an index is set on CDS (this index is a feature of Data obtained as a result of the cloning process) and then a range is set. After that, calling Open on Result draws only those records from CDS in the range. Result is then returned to the calling method.

The fun stuff, if you will, can be found in the method that calls ReturnRecordsFromCDS. This method, named GenerateXMLFileForCustomer, is passed the customer number whose orders data needs to be written to a file. It performs its work using an anonymous thread. GenerateXMLFileForCustomer is shown in the following code listing.

procedure TForm1.GenerateXMLFileForCustomer(CustNo: Integer);
var
  AnonThread: TThread;
begin
  //Create the anonymous thread to run an anonymous method
  AnonThread := TThread.
     CreateAnonymousThread(
       procedure
       var
            CDS: TClientDataSet;
            FileToWrite: String;
       begin
         CDS := DataModule1.ReturnRecordsFromCDS(CustNo);
         try
           FileToWrite := FAppDir + '\' + IntToStr(CustNo) + '.xml';
           if FileExists(FileToWrite)then
                DeleteFile(FileToWrite);
                CDS.SaveToFile(FileToWrite);
           finally
             //Free the ReturnRecordsFromCDS Result
             CDS.Free;
           end;
       end);
  //Start the anonymous thread
  AnonThread.Start;
end;
An anonymous thread is created by passing a reference to a procedure to the TThread class method named CreateAnonymousThread. CreateAnonymousThread returns a TAnonymousThread (a TThread descendant), which you then execute by calling its Start method. Once the anonymous thread has completed the execution of the procedure, it terminates and frees itself.

In the case of GenerateXMLFileForCustomer, the procedure passed to CreateAnonymousThread is an anonymous method. An anonymous method is an unnamed method whose implementation appears inline in your code.

Importantly, an anonymous method closes over the local variables (including formal parameters) of the method in which it appears, maintaining the context of these values even after the method in which the anonymous method is defined has exited. This is called a closure, and it permits each execution of this anonymous method to maintain the value passed in the CustNo formal parameter.

Each call to GenerateXMLFileForCustomer creates a different worker thread that runs the anonymous method. Each anonymous method calls the ReturnRecordsFromCDS method, which returns a new ClientDataSet that holds the orders for the given customer. After ensuring that the target file does not already exists (it will be deleted if it does), the anonymous methods writes the contents of this ClientDataSet to the appropriately named file, after which it frees the ClientDataSet created by the call to ReturnRecordsFromCDS.

GenerateXMLFileForCustomer is called from the OnClick event handler of the button labeled Create Files for Customers. The following is the code that appears on that event handler:

procedure TForm1.CreateFilesButtonClick(Sender: TObject);
var
  i: Integer;
begin
  for i := 0 to ListBox1.Count - 1 do
    GenerateXMLFileForCustomer(StrToInt(ListBox1.Items[i]));
end;

Turning back our attention to the Windows service/client example I mentioned earlier, GenerateXMLFileForCustomer would be implemented on the OnExecute event handler of the TIdTCPServer component. Since that method executes in a different thread for each client connection, the use of the anonymous method would be unnecessary. All the OnExecute event handler would need to do is to read the customer number passed by the client, execute the ReturnRecordsFromCDS method passing to it this customer number, and then return the XMLData property of the returned ClientDataSet to the client application, after which the returned ClientDataSet would be freed.

Alternatively, the returned ClientDataSet could be returned to the requesting client using a stream. Whether text or binary data is returned to the client, the service might also encrypt and/or compress the data before returning it, in which case the client must know to decrypt and/or decompress the data it receives.

Summary

When you clone a ClientDataSet cursor, you permit an additional ClientDataSet to refer to the Data and Delta properties of an existing ClientDataSet. While all ClientDataSets that share the same cursor share the same Data and Delta, they do not share any other properties, permitting each ClientDataSet to maintain an independent view of the data.

For more information about Delphi in Depth: ClientDataSets, as well as for links for purchasing the book, please visit http://www.JensenDataSystems.com/cdsbook.

Copyright © 2011 Cary Jensen. All Rights Reserved

Author’s Note: This article originally appeared in the SDN Magazine # 109, published on May 5th, 2011.  For information about the Software Development Network, please visit http://www.sdn.nl/.

12 comments:

  1. Hi Cary,

    Huge "space" left in Delphi documentation is now filled up by your great "Delphi in Depth : ClientDatasets" book. Thank you for your great work. Foible of Delphi is its lack of technical material. This book is really necessary for those who develop under Delphi. The ClientDatasets Bible !!!
    Thank you.
    Regards,
    Laurent

    ReplyDelete
  2. Hi Cary.

    A couple of days ago I ran into the problem you describe:

    "If you clone a ClientDataSet that is filtered [...] and you pass a value of False (no reset) in the second parameter of CloneCursor, the receiving ClientDataSet will also employ the filter and it cannot be removed [...] This appears to be a bug."

    So I decided to investigate the matter and try to find the cause of this bug. By examining the source code of Midas (curinit.cpp, idxfilt.cpp, dscanexp.cpp and others), I have been lucky enough to get to the bottom of the issue.

    The following code can be added to the overridden CloneCursor method of a class derived from TClientDataSet in order to correct the bug:

    Procedure TClientDataSetB.CloneCursor (Source :TCustomClientDataSet;
    Reset :Boolean; KeepSettings :Boolean = False);
    Var
    I :Integer;
    Begin
    Inherited CloneCursor (Source, Reset, KeepSettings);

    If Reset And Source.Filtered Then
    For I := 1 To Byte (Source.Filter <> '') +
    Byte (Assigned (Source.OnFilterRecord)) Do
    DSCursor.DropFilter (hDSFilter (I));
    End;

    Explanation:

    The IDSCursor.CloneCursor method does not copy the internal filter entries properly. When combining the Filter and OnFilterRecord properties and they are modified, the IDs of the filter entries (internal "ConstrEntry" structs) get a value greater than 2. IDSCursor.CloneCursor recreates these entries by calling the AddFilter and AddFilterCallBack methods, which, in a new cursor, assign filter IDs starting from 1. So, "DropFilter" calls made in the TCustomClientDataSet.CloneCursor method may not have any effect (Source.FExprFilter and Source.FFuncFilter could be invalid IDs for the new cursor).

    I hope this patch is helpful.

    Best regards.

    Al Gonzalez.

    ReplyDelete
    Replies
    1. Al:

      Thank you so much for your input. It's obvious you are on the right track, but the code as posted does not permit the filter to be dropped after a call to CloneCursor where Reset is set to False. Specifically, when Reset is False the code that followed the call to inherited is not executed. Furthermore, if I change the "If Reset" to "If not Reset" the filter is always dropped, which runs counter to the Reset=False parameter.

      In any case, I think you are really close to the solution and I hope that you see this reply and can take a second look at the code. Thank you so much for your time.

      Delete
    2. Hi Cary.

      Thank you so much for read and test my small contribution and for your punctual observation.

      I think the following code will be a more complete and appropriate solution to the problem of the filters.

      Procedure TClientDataSetEx.CloneCursor (Source :TCustomClientDataSet;
      Reset :Boolean; KeepSettings :Boolean = False);
      Var
      FilterFixed :Boolean;
      I :Integer;
      Begin
      Inherited CloneCursor (Source, Reset, KeepSettings);
      FilterFixed := False;

      If Source.Filtered Then
      For I := 1 To Ord (Source.Filter <> '') +
      Ord (Assigned (Source.OnFilterRecord)) Do
      FilterFixed := (DSCursor.DropFilter (hDSFilter (I)) = 0) Or
      FilterFixed;

      If Filtered Then
      Begin
      ActivateFilters; // See TCustomClientDataSet.InternalOpen method
      FilterFixed := True;
      End;

      If FilterFixed Then
      Resync ([]);
      End;

      Note that you must always remove the internal filters, and, if the data set remains/becomes filtered (Reset = False), you must add the internal filters by calling the ActivateFilters method. Furthermore, this latest fixes another bug: When Reset is False and KeepSettings is True, TCustomClientDataSet "forgets" to activate the filters.

      Please check if it is good. I have performed successful tests with different values for the Reset and KeepSettings parameters (in Delphi 7 and XE2). Could I send you my test program to an e-mail?

      In advance, thanks. Best regards.

      Al Gonzalez.

      Delete
    3. Al. you did it! I have tested this code (in XE5) and it solves the problem, whether the filter is applied using the Filter property or the OnFilterRecord event handler.

      Thank you so much for following up with this. I will make sure that someone at Embarcadero takes a look at your code and implements the fix.

      Delete
  3. I have a small problem that I tried to find a solution for a week but I did not solve it, I said, I have a TClientDataSet empty without records and from that point I started to add it new record and Then i edit it several times and Then delete it, all this before ApplyUpdates or MergeChangeLog when I check the UpdateStatus always usInserted satut of the new record.
    And I can not find UpdateStatus usModified, usDeleted for new recode in delta
    My goal is how I can find a way to know the status of historical changes for a new record before applying to the base.
    thank you in advance

    ReplyDelete
    Replies
    1. The record status should be usInserted, as a SQL Update statement is what will be generated to insert the record. If you then delete it before updates are applied, it should be absent from the change cache. The change cache does not keep a log of individual changes to a record over time. For example, if you insert a record, post it to cache, and then update the record, prior to calling ApplyUpdates a call to RevertRecord will remove the entire record from cache. This is how the ClientDataSet is designed.

      Delete
    2. Hi Cary.

      While what you say is correct, Badji can access to the change log (through the StatusFilter property) in order to know the other statuses of a new record. Just that he must take into account the suggestions I made ​​in this QualityCentral's report:
      http://qc.embarcadero.com/wc/qcmain.aspx?d=93987.

      Moreover, and I hope not to seem opportunist, maybe he and you are interested in looking at how I implemented the RecAttrs ("Record Attributes") method in the derived class TghClientDataSet —GH Freebrary—.

      Kind regards.

      Al Gonzalez.

      Delete
  4. Hi all!

    I have a problem in RAD XE 2, I would like to ask some help. I dont see the resulting row in a Dbgrid, if I use this code. Dbgrid shows something changed, but not the data. Cds1.isempty remains true.

    procedure TForm2.btn1Click(Sender: TObject);
    begin
    if cds1.IsEmpty then
    begin
    cds1.CreateDataSet;
    cds1.LogChanges:=false;

    cds2.CloneCursor(cds1,true,false);
    end;

    cds2.Insert;
    cds2.Fields[0].AsString:='tryout';
    cds2.Post;
    end;

    Am I doing sg wrong? Thank You!

    Kind regards.

    eSzeL

    ReplyDelete
    Replies
    1. eSzel: First of all, this is not necessarily a CDS issue, but a CDS-DBGrid issue. After to the call to Post both CDS1 and CDS2 have the new record, it is just not visible in the first DBGrid until you move off that current record in the DBGrid. There is another issue, however. IsEmpty returns true if the ClientDataSet is empty, but you are creating the CDS1 structure if it is empty, which is not correct. You want to create the ClientDataSet if it is not yet created. You should be testing if not CDS1.Active then CDS1.CreateDataSet;. Also, move the call to CloneCursor outside of the test for Active.

      I am assuming that CDS1 has structure and is not associated with a DataSetProvider or a FileName. If CDS1 is associated with a DataSetProvider or FileName, call Open or set Active to True instead of calling CreateDataSet.

      Also, I noticed that if I removed the call setting LogChanges to False on CDS1 everything worked correctly.

      Delete
  5. Hello Cary,

    I got a few problems with cloned datasets. Most i have been able to fix by reassigning IndexFieldNames, MasterSource and MasterFields so they work after the cloning.

    But aggregate fields got me stuck. They work if i turn aggregate off and back on after the cloning, but if i have i group level the aggregates turn blank. I check the indexdef after cloning, still have group level=1 and so does the aggregatefield but... its empty.

    ReplyDelete
  6. Perhaps you can use an FDMemTable instead. Call FDMemTable.CopyDataSet, and include at least the five following flags in the set you pass in the second parameter: [coStructure, coRestart, coAppend, coAggregatesReset, coAggregatesCopy]

    ReplyDelete