Delphi Non-Core Feature Survey: You Can Help

Delphi developers, I want to hear from you. Are you using Delphi's frameworks for unit testing, audits, metrics, and design patterns? If no, why? If yes, to what extent do you use these features? Click here to take the survey

Here is a little background. Since the release of Delphi 2005, there have been a number of interesting support features introduced in Delphi. For example, Delphi 2005 added support for easily creating unit tests. And audits, metrics, and design pattern support was added in Delphi 2006.

Initially Delphi's support for unit testing was available in all versions (Professional, Enterprise, and Architect). By comparison, the support for audits, metrics, and design patterns required Together, and this product was shipped only with the high-end versions of Delphi. And, on top of that, the Together product stayed with Borland when CodeGear was spun off. So what did that mean for these features as Delphi evolved?

I have to admit that I have not used the audits, metrics, and design pattern features of Delphi, though I have noticed the associated menu items in Delphi's menus. So, when I got a request from a client to include discussion of audits and metrics in an upcoming Delphi class that I am going to deliver, it was time to do some research.

What I found was interesting and puzzling. There is not a whole lot of information out there about these features. And, I did discover that not only are these features available in the absence of Together, but are now even included (with limited support) in the Professional sku of Delphi 2010.

I have now committed to writing training material on these topics, and this will also lead to my adapting this material for this blog as well for some magazine articles that I am writing. And this is where your help comes.

I want to hear from you. If you are a Delphi developer, I want to know which of these features you use, and to what extent. If you don't use these features as they ship in the product, do you use third-party tools that provide similar support?

I have created a short, 10 question survey that should take only a couple of minutes to complete. Please help me by filling out this survey. In addition, please ask your Delphi colleagues to help out as well. Click here to take the survey

Copyright (c) 2010 Cary Jensen. All Rights Reserved.

In-Memory DataSets: ClientDataSets and .NET DataTables Part 6: Applying Updates to a Database

In the preceding article in this series I discussed various techniques that you can use to manage the change cache. In this installment I will conclude that discussion by looking at how you can apply the changes held in the change cache to the underlying database from which the data was originally loaded.

Call the ApplyUpdates method of a ClientDataSet to save any changes made to the in-memory data to the underlying database. Specifically, if you edit data obtained through a dataset provider, and then close or free a client dataset without calling ApplyUpdates, any changes stored in the change log are lost.

When ApplyUpdates is called, the contents are sent back to the dataset provider for resolution in the context of a transaction (so long as you pass a non-negative integer as the sole parameter of the ApplyUpdates method. When you pass –1, no transaction is initiated). The dataset provider, in turn, generates the necessary calls to apply the updates to the underlying dataset, applying the changes to the underlying dataset one record at a time.

By default this process is handled by a SQLResolver instance, which is created by the dataset provider. The SQL resolver determines the database that needs to be updated, and then creates the necessary SQL statement to apply the changes based on the contents of the change log. Specifically, one SQL statement will be generated for each change that needs to be applied.

Alternatively, the dataset provider can be configured to use the dataset from which it originally read the data to apply the changes. This approach is only possible when the dataset from which the records were read permits data changes.

For example, if the dataset provider gets its data from a TTable, the dataset provider can edit the TTable directly, inserting, deleting, or posting the changes using the TDataSet interface. Again, these changes are applied one at a time. It should be noted that when the dataset provider resolves the changes through the dataset, the dataset's event handlers, such as BeforeDelete and BeforePost can be used to perform data validation.

By comparison, if the dataset provider gets its data from a dataset that does not permit data editing, the dataset provider cannot resolve the data to the dataset. For example, if the dataset provider gets its data from a SQLDataSet, a dataset that retrieves its data using a unidirectional cursor and which does not permit editing, the dataset provider cannot resolve the changes directly to the dataset.

In these cases, there are two options. Either the default SQLResolver described earlier can be used, or you can write a BeforeUpdateRecord event handler. From within this event handler your code is given a reference to the changes that must be applied, and your code can take any action necessarily to explicitly apply these changes. This approach is the most flexible, although the most difficult to implement.

What kind of SQL (or direct edit) is generated is controlled by the UpdateMode of the DataSetProvider. If set to upWhereAll, a record is updated only if that exact record currently exists in the database. If set to upWhereChanged, the record is updated if a record with the same key fields and same values in the modified fields are found (this is a merge). When set to upWhereKeyOnly, an update is made if a record with the same key is found (last to post wins).

Updating .NET DataSets is similar to updating ClientDataSets. In that case, however, the DbDataAdapter class is the one typically used to apply the updates. DbDataAdapters, such as DataStoreDataAdapter, have four DbCommand properties. The SelectCommand property contains the SQL statement that returns the result set that is inserted into the DataTable when you call the Fill method. The other DbCommand properties, DeleteCommand, InsertCommand, and UpdateCommand, are designed to hold parameterized queries that will get their parameter values from the change log at runtime when you call the DbDataAdapter.Update method.

Some developers write the SQL statements that define the DeleteCommand, InsertCommand, and UpdateCommand objects manually. Doing so give them control over the queries, permitting the queries to be optimized for the underlying database. Other developers use a CommandBuilder to generate these queries. CommandBuilders are easy to use, but do not always generate optimized queries.

To use a CommandBuilder, you call its constructor, passing to it the DbDataAdapter whose SelectCommand command has already been defined. Based on this query, the CommandBuilder generates the DeleteCommand, InsertCommand, and UpdateCommand queries.

The following code shows the configuration of a CommandBuilder:

Connection1 := DataStoreConnection.Create(
  'host=LocalHost;user=sysdba; ' +
  'password=masterkey;database="C:\Users\Public\Documents\' +
  'Delphi Prism\Demos\database\databases\BlackfishSQL\employee"';
Connection1.Open();
//Sql statements are executed by IDbCommand objects
Command1 := DataStoreCommand.Create('SELECT * FROM customers',
                                    Connection1);
//DbDataAdapters are used to populate DataTables and resolve data
DataAdapter1 := SqlDataAdapter.Create(Command1);
//CommandBuilders create DbCommand objects for a DataAdapter's
//DeleteCommand, InsertCommand, and UpdateCommand properties based
//on the DbDataAdapter.SelectCommand IDbCommand property
CommandBuilder1 := DataStoreCommandBuilder.Create(DataAdapter1);
DataSet1 := DataSet.Create;
DataAdapter1.Fill(DataSet1);

The following code shows how the changes made to the DataTable are applied.

var
  DataTable1: DataTable;
begin
  DataTable1 := DataSet1.Tables[0].GetChanges;
  if DataTable1 <> nil then
    DataAdapter1.Update(DataSet1.Tables[0]);

Applying Updates and Persisted Data

Probably one of the most important characteristics of an in-memory dataset's ability to apply its updates to the underlying database is its concurrent ability to persist its data and state to a file, stream, or database. Together, these features permit an in-memory dataset to apply its updates at some future time, regardless of whether it has been persisted or not (and independent of its duration of persistence).

In order for a dataset to be able to apply its updates to a database subsequent to its persistence, the dataset's change log must be intact. Without this information, you lack the data required to determine which changes have been made to the dataset since it was originally populated.

If you are relying on a DataSetProvider (for ClientDataSets) or an DbDataAdapter implementation (for .NET) to apply the updates to the database, the object must be in a state compatible with applying the dataset's updates. For example, a DataSetProvider that you use to apply a previously persisted ClientDataSet's updates to a database must point to a TDataSet whose structure is consistent with the one that was used to originally load the ClientDataSet (unless you are using the DataSetAdapter's BeforeUpdateRecord event handler to programmatically apply the update, in which case it's all up to your code).

In the case of a .NET DataTable, the DbDataAdapter that is used to apply its updates must hold DbCommand instances in its DeleteCommand, InsertCommand, and UpdateCommand properties that contain parameterized queries sufficient to the task of applying those updates. This can be achieved by defining these queries manually, or by having an adequate DbDataAdapter.SelectCommand instance that can be used by a CommandBuilder to construct the necessary queries (of course, you must then use a CommandBuilder to create those delete, insert, and update queries objects). Otherwise, you once again must take matters into your own hands and generate all calls to update the underlying database by programmatically examining the change log and generating the necessary queries.

What If Updates Cannot Be Applied?

There is another issue that is worth mentioning. Specifically, even when you can construct the necessary queries to apply updates contained in a persisted in-memory dataset, those updates may not be possible. Specifically, it is conceivable that between the time that the data was originally loaded into the in-memory dataset and the point in time at which you want to write it back to the underlying database, the corresponding records in the database have been changed.

While many developers worry about this possibility, it is normally less of a concern than it might at first appear. For example, if you delete a record from an in-memory dataset, and then attempt to apply that deletion (at a later time), but find that the record was already deleted by someone else, who cares? It's gone. Mission accomplished (by someone, at least).

Similarly, if you attempt to apply a record insertion, only to find a record with that key already inserted, then the insertion is not necessary. But what, you might ask, if another user inserted a record with the same key as you (which causes your insertion to fail), but the other user's inserted record is different than the one you intended?

This is really an architectural issue, isn't it? Many of today's developers avoid this issue altogether by assigning arbitrary (and more or less meaningless) primary keys to each and every record. For example, many developers use GUIDs (globally unique identifiers ¾ 128-bit values that are guaranteed to be unique), to each record. In these cases, duplicate records are impossible, no matter who inserts the record.

Updates are a bit more problematic, but nonetheless generally workable. In most cases it is once again an architectural issue. There are, in fact, three options. In the first case, an attempt to update a record that you are holding in memory fails if someone else has changed any part of that record (in the underlying database) since the time you originally loaded it into memory. In this situation, your update query should include a test for all original field values in the WHERE clause of the update query.

In reality, failure to update a record that has been subsequently updated by another user, when those updates must be entirely exclusive, is rarely a tragedy.

The second case is to permit two or more users to update a record, so long as neither user changes the key field values, or a common field. This is a fine compromise in many situations. For example, if, since the time that you read your record into memory and the time that you are attempting to apply your update, another user has changed that record, but not one of the fields that your user changed, who cares? Simply include the record's key fields, and your changed fields, in the WHERE clause of the update query, ignoring those non-key fields you did not change. (If you use an arbitrary key field, such as a GUID, it is unreasonable, and should be prohibited, that anyone should ever change the key field value).

The third option, and one that is appropriate for some database applications, is simply that the last user to attempt to write to a record be permitted to do so, overwriting any previous user's applied updates. For example, imagine that five different sensors record the current temperature in a given area. If all sensors are equally correct, the most current update attempt is the most accurate. Who cares about a temperature that was written ten minutes ago? A similar analogy can be made with respect to stock prices. A more current stock price is more accurate. Past updates are old news.

Copyright (c) 2010 Cary Jensen. All Rights Reserved.

In-Memory DataSets: ClientDataSets and .NET DataTables Part 5: Managing the Change Cache

The change cache is the source of much of an in-memory dataset's power. It is through the change cache that you can permit a user to review their changes before committing them, as well as permit you to programmatically work with changes that have been made to the dataset.

Note: In the first article in this series, I called the change cache the change log. A reader responded that what I described was not really a log, since it doesn’t keep track of individual changes, only information about the state of individual records. The term change log originates from the TClientDataSet interface, which includes the LogChanges property and the MergeChangeLog method. I agree that the term change cache more accurately reflects this feature of in-memory datasets, and will use that term from now on.

The following are operations that can be performed on the change cache for any in-memory dataset:

• Detecting changes
• Cancel all changes
• Filter on changes
• Detect field-level changes
• Cancel a single change
• Erase the change cache
• Commit changes in the change cache to the underlying database

The first six of these operations are described in the following sections. Committing changes to an underlying database is a much larger topic, and I will cover it in the next installment of this series.

Detecting Changes

You use the ClientDataSet's ChangeCount property to determine if there have been any changes to the data in the ClientDataSet since if was originally loaded. There are no changes if ChangeCount returns 0 (zero). If there have been one or more changes, ChangeCount returns the total number of records that have been inserted, deleted, or modified (including changes to nested datasets).

Note that ChangeCount only counts changed records. For example, a record that is inserted and then modified counts as only one change.

The following code segment demonstrates the use of ChangeCount.

if ClientDataSet1.State in [dsEdit, dsInsert] then
  ClientDataSet1.Post;
  if ClientDataSet1.ChangeCount > 0 then
    ClientDataSet1.ApplyUpdates(0);

Note that there is a bug in ClientDataSets in the original release with Delphi 6 and 7 with respect to ChangeCount. If you save a ClientDataSet to a file or stream, and then later restore the ClientDataSet, ChangeCount will be zero immediately after you reload the ClientDataSet, even if there are changes in the change cache.

In .NET, you call the DataTable's GetChanges method. If GetChanges returns a nil reference, there are no changes. If there are changes, the DataTable returned by GetChanges contains the inserted, deleted, and modified records.

if DataTable1.GetChanges <> nil then
begin
  //Work with the changes here
  end;

Canceling All Changes

When you cancel all changes, the in-memory dataset reverts to the values it contained when it was originally loaded. You cancel all changes in a ClientDataSet by calling the CancelUpdates method. With a .NET DataTable, you call RejectChanges.

Canceling all changes is an irreversible action.

Filtering On Changes

Both ClientDataSets and .NET DataTables permit you to filter the dataset to display only inserted, deleted, and modified records. With a ClientDataSet, you invoke this operation using the StatusFilter property, which can be set to include the following four flags: usModified, usInserted, usDeleted, and usUnModified.

The following code segment demonstrates the use of StatusFilter:

if ClientDataSet1.State in [dsEdit, dsInsert] then
  ClientDataSet1.Post;
  if ClientDataSet1.ChangeCount > 0 then
  begin
    ClientDataSet1.StatusFilter := [usModified, usInserted, usDeleted];
    //Do something with the changed records
  end;

Just as you must use a DataView to obtain a filtered view of a DataTable, you must use a DataView to filter on Changed records. In this case, you use the RowStateFilter and set it to one of the following eight values of the DataViewRowState enumeration: OriginalRows, CurrentRows, Added, Deleted, ModifiedOriginal, ModifiedCurrent, UnChanged, and None.

The following code demonstrates how to obtain a DataView that contains only the records deleted from a DataTable:

DataView1 := DataView.Create(DataTable1);
  DataView1.RowStateFilter := DataViewRowState.Deleted;
  if DataView1.Count > 0 then
  begin
    // Do something with the deleted records
  end;

Detecting Field-Level Changes

With ClientDataSets, you detect field-level changes by examining the OldValue and NewValue variant properties of your ClientDataSet's TFields. The following code demonstrates how to do this:

if ClientDataSet1.UpdateStatus = usModified then
begin
  if ClientDataSet1.Fields[0].OldValue <> ClientDataSet1.Fields[0].NewValue then
    begin
      //The first field in the current record has been changed
    end;
end;

With .NET DataTables, you must obtain two DataViews for the table, setting the RowStateFilter on one of them to ModifiedOriginal, and the RowStateFilter of the second to ModifiedCurrent. You can them compare the fields of the two DataViews to determine what has changed. This is demonstrated in the following code sample:

DataView1 := DataView.Create(DataTable1);
DataView2 := DataView.Create(DataTable2);
DataView1.RowStateFilter := DataViewRowState.ModifiedOriginal;
DataView2.RowStateFilter := DataViewRowState.ModifiedCurrent;
if DataView1.Item[0].Row[0].ToString <> DataView0.Item[0].Row[0].ToString then
  begin
    //Field 1 of record 1 has been changed.
    //Assumes a string-compatible field
  end;

Canceling a Single Change

ClientDataSets provides you with two mechanisms for canceling a change to a record. You call the UndoLastChange method to revert the record that was last inserted, deleted, or modified to its original state. By comparison, if the ClientDataSet is pointing to a record that was inserted, deleted, or modified, you can call RevertRecord.

The following code demonstrates the use of RevertRecord:

if ClientDataSet1.UpdateStatus in [usInserted, usDeleted, usModified] then
  ClientDataSet1.RevertRecord;

For a .NET DataRow, you call the RejectChanges method to restore an inserted, deleted, or modified DataRow to its original state. The following code demonstrates how to restore all deleted records to a DataTable:

var
  DataView2: DataView;
  i: Integer;
begin
  DataView2 := DataView.Create(DataSet1.Tables[0]);
  DataView2.RowStateFilter := DataViewRowState.Deleted;
  for i := (DataView2.Count -1) downto 0 do
    DataView2.item[i].Row.RejectChanges;

Erasing the Change Cache

Erasing the change cache leaves any changes to the dataset intact, but deletes the record of those changes. Most developers do not want to erase the change cache since it makes it impossible to undo changes and also erases the information necessary to write the changes to the underlying database. About the only time that erasing the change cache makes sense is when you want to make the changes permanent before storing the dataset in a file or stream, and never have to resolve those changes back to some underlying database.

With ClientDataSets, you erase the change cache by calling MergeChangeLog. (Note that you can simply turn off the change cache with a ClientDataSet by setting the ClientDataSet's LogChanges property to a Boolean False before making any changes.)

With .NET DataTables, call AcceptChanges to erase the change cache.

Copyright © 2010 Cary Jensen. All Rights Reserved.

Delphi Developer Days 2010

It’s less than a month away from the first city in the 2010 Delphi Developer Days tour, and I thought this might be a good time to share some history and insight into this event. To begin with, Delphi Developer Days is an intense, two-day Delphi event featuring me and Marco Cantù. Our first city on this tour is Baltimore/Washington DC (May 11-12), continuing on to Chicago (May 14-15), Los Angeles (May 17-18), London, England (May 26-27); and finishing up in Frankfurt, Germany (May 30 – June 1).

Marco and I have known each other for a long time, are good friends, and have been looking for an opportunity to work together. In May of 2008 we talked about the lack of a live Delphi event in the US, and thought that this provided us an opportunity to offer something unique.

We weren’t interested in trying to run a conference similar to the past annual Borland Conferences or DelphiLive!. Instead, we were looking to offer something similar to a traditional training, but with the interaction and networking that a somewhat larger event can provide. As a result, we limit attendance to around 30 attendees in each city. This gives us a chance to interact with each attendee, while maintaining a critical mass for networking. (In addition to our consultant and development services, we’re both seasoned Delphi trainers.)

But there were a number of challenges that we knew we’d have to address in order for the event to be successful. To begin with, Delphi is a mature product. While there are always some newcomers, most Delphi developers have been using it for some time. Furthermore, not everyone is using the latest versions of Delphi. In fact, by some estimates, nearly half of all active Delphi developers are using Delphi 7 or earlier.

Another challenge is that there are a lot of different types of Delphi developers. Many are engaged in traditional client/server database development, while others are building server-side applications that operate over the Internet. How do you offer something that will appeal to such a diverse group of developers?

Honestly, I think the solution we came up with is the right one. To begin with, we embrace the richness of the Delphi community. We include timely topics about the latest versions of Delphi. Even if you are working with an older version, these topics are relevant. Specifically, even if you are not ready to upgrade now, you want to be informed about what the latest versions can do for you and your development efforts. In the long run, unless your applications are all at the end of their lifecycle, you will upgrade sooner or later.

At the same time, we realize that there is always room for fundamentals, topics that apply to many versions of Delphi (and some of these go all the way back to Delphi 1!). As far as types of applications, we understand that we need to have a healthy dose of both Windows client development as well as distributed computing.

Finally, we wanted to deliver this type of content in a way that would be fun for us as well as our attendees, and do so in a way that gives the attendees something of value, something that lasts well beyond the two days we spend together.

All of these considerations went into the format of Delphi Developer Days. There are a total of 12 presentations over the course of two days, four of which Marco and I do together, and eight which we do individually (four sessions where we each present simultaneously in separate rooms). In choosing the topics, we tried to include something for everybody, from “What’s New in Delphi and Delphi Prism,” to “Leveraging ClientDataSets.” From “Internet Delphi Application Technologies Compared,” to “Delphi Development for Windows 7.” We even include a foundation presentation on “Designing Interfaces and Objects.” (For a complete schedule, visit http://www.delphideveloperdays.com/descriptions.html.)

When Marco and I are presenting separately, attendees can attend either talk. But what if you are interested in both topics? While we tried to keep the simultaneous individual session topics from overlapping, we acknowledge that this is an issue. Fortunately, both Marco and I are authors, and have published close to 40 books between us. To put this another way, we know how to write, and like doing it.

Each attendee to Delphi Developer Days receives a course book written by us that includes a paper for each talk. So, if you choose to attend one individual session, you can read the paper for the other session that is going on at the same time. Last year our course book was 500 pages in length (you get our slideshows and code, too, but these papers are detailed papers, covering more information than we have time for in our live sessions). We are currently writing the course book for this year’s event, and I’m guessing that it will be similar in length.

This is a course book that you can’t get anywhere else. While there is occasional overlap with other writings that we’ve done (such as our blogs, books, magazine articles, or other courseware), we do not publish or otherwise make this material available outside of Delphi Developer Days. Sorry, please don’t ask to buy the course book; it’s for attendees only.

As I mentioned above, this is not the first time Marco and I have done this. Delphi Developer Days 2009 was a wonderful success. Marco and I had a blast, and the feedback we received encouraged us to do it again this year, with two additional cities added to the tour.

So how does this year’s tour compare to last year’s? In addition to adding two cities (Los Angeles and Frankfurt, Germany, cities added in response to requests from the Delphi community), we added new topics and brought back some of the popular ones. And in order to make this year’s event relevant for previous attendees, we made sure that when a previous topics was covered during our breakout sessions, it was paired with a new topic, giving previous attendees a choice of something new. And, judging from the number of last year’s attendees who have registered for this year, we succeeded.

I would be remiss if I didn’t also mention our sponsors. Embarcadero Technologies and Sybase are once again our platinum sponsors, and each city will include a short presentation by representatives of one or both of these companies. In addition, we have more than 20 additional sponsors from almost every segment of the Delphi community. Many of these sponsors offer tools that work with Delphi, and many of them have provided copies of their products as door prizes for attendees. These sponsors are among the most active companies in the Delphi community, and we encourage you to visit their sites and learn more about their products. You will find a list of our current sponsors at http://www.delphideveloperdays.com/sponsors.html.

But there is more. On the first night of each US city, either Anders Ohlsson or David I (David Intersimone) will be present to provide a two hour “Embarcadero Delphi Evening,” complete with loads of information about upcoming plans. (Anders will be in Baltimore/Washington DC and Chicago, David I in Los Angeles.) Even if you can’t attend Delphi Developer Days, you can come to these free events. Visit http://edn.embarcadero.com/article/40546 to register (Delphi Developer Days attendees do not need to register. Your registration for Delphi Developer Days takes care of that).

I want to finish up by saying that Marco and I are huge Delphi fans, and we are extremely positive about the future of Delphi and the Delphi Community as a whole. We hope to see you there. It’s going to be a blast.

Learn more at http://www.delphideveloperdays.com/

Copyright © 2010 Cary Jensen. All Rights Reserved.

In-Memory DataSets: ClientDataSets and .NET DataTables Part 4: Navigating, Sorting, and Filtering

In the preceding article in this series I discussed how to populate in-memory datasets. In this continuation, I will look at some of the ways that you can programmatically work with that data, including navigating, sorting, and filtering the data in memory.

Active ClientDataSets are essentially a cursor pointer that refers to the current record in the dataset. To navigate which record your ClientDataSet refers to, you use methods of the ClientDataSet to change the current record. These methods include First, Next, Prior, Last, and MoveBy, to name a few.

The following line of code demonstrates how to navigate to each record of (scan) a ClientDataSet:

ClientDataSet1.Open;
while not ClientDataSet1.EOF do
begin
  //Do something here with the current record
  ClientDataSet1.Next;
end;

DataTables in the .NET framework do not support the concept of a current record. Instead, you use the Rows property to indicate which DataRow of the DataTable you want to work with.

The following code segment demonstrates how to programmatically navigate the DataRows of a DataTable.

var
i: Integer;
DataRow1: DataRow;
begin
  for i := 0 to DataTable1.Rows.Count – 1 do
  begin
    DataRow1 := DataTable1.Rows[i];
    //Do something with the DataRow
  end;

While the DataTable lacks the concept of a current record, Windows Forms applications have a concept that is very similar to current record. Anytime one or more properties are bound to a multivalue binding context (such as a DataTable) in a Windows Forms application, a CurrencyManager is created that can be used to determine to which DataRow the form's controls are synchronized. If you bind data using a BindingSource instance, CurrencyManager property points to the underlying CurrencyManager. Similarly, there are classes, such as BindingSource, that provide features similar to a CurrencyManager.

Note that in .NET Web Forms applications, most multi-value controls automatically iterate through the records of a DataTable or DbDataReader implementing instance when you call the DataBind method. As a result, Web forms do not have currency managers.

Sorting In-Memory Data

ClientDataSets are sorted using ClientDataSet indexes. There are two types of indexes in ClientDataSets, temporary indexes and persistent indexes.

You create a temporary index using the IndexFieldNames property of the ClientDataSet. When you assign a field name (or a comma-separated list of field names) to the IndexFieldNames property, an index is create for that field (or fields) at that moment the property is assigned, and the records of the ClientDataSet are then sorted based on that ascending index. This is demonstrated in the following code segment:

ClientDataSet1.IndexFieldNames := 'State;City';

Persistent indexes, on the other hand, are defined using the IndexDefs property of the ClientDataSet. After defining one or more IndexDef instances, you then set the IndexName property of the ClientDataSet to the name of one of those IndexDef instances.

Indexes created using IndexDefs are more flexible than temporary indexes. While temporary indexes can only be ascending, case-sensitive indexes, your IndexDef definitions can define ascending or descending indexes, as well as case insensitive indexes.

The following line of code demonstrates the creation and use of a persistent index using an IndexDef definition:

ClientDataSet1.IndexDefs.Add('IDIndex', 'State;City',[ixCaseInsensitive]);
ClientDataSet1.IndexName := 'IDIndex';

DataTables cannot be sorted directly. If you want to sort the records in a DataTable, you use a DataView. You sort a DataView using its Sort property, which can hold either the name of a DataColumn to sort the DataView DataRows, or a comma-separated list of DataColumn names.

The following code demonstrates creating a DataView to display a sorted list of a DataTable's DataRows.

var DataView1: DataView;
DataView1.Create(DataTable1);
DataView1.Sort := 'Customer ID';

Rather than creating a new DataView to display sorted DataRows, you can use the DefaultView of a DataTable to refer to the DataView that each DataTable automatically supports. This is shown in the following code segment:

DataTable1.DefaultView.Sort := 'Customer ID';

The alternative is to create a new DataView instance for each sort order you want to use. The following example creates two DataViews for a single DataTable, each with its own sort order:

var
DataView1, DataView2: DataView;
begin
  DataView1 := DataView.Create(DataTable1);
  DataView1.Sort := 'Customer ID';
  DataView2 := DataView.Create(DataTable1);
DataView2.Sort := 'Country;City';

Filtering In-Memory Data

When you filter a ClientDataSet, it behaves as though it contains only those records that meet the filter criteria. For example, if you filter a ClientDataSet for all records where the City field contains the name 'London', that ClientDataSet will act as though it only contains records where the City field value is London.

Note: There is one exception to the behavior of filtered ClientDataSets. Specifically, if you read the XMLData property of a ClientDataSet or call SaveToFile or SaveToStream, the results are as if the ClientDataSet is not filtered. In other words, all records are involved, not just the filtered records. To save only the filtered records, you must point to the filtered ClientDataSet with the DataSet property of a DataSetProvider that is referenced by another ClientDataSet. After opening that other ClientDataSet, it will contain only the filtered records (a copy), which you can then save to a file or stream. These records, however, do not include a change log.

There are two ways to filter a ClientDataSet. One is to use the Filter and Filtered properties. Filter takes a Boolean expression. Records that are available in the filtered view are those for whom the expression evaluates to True. The following code segment demonstrates how to filter a ClientDataSet using the Filter and Filtered properties.

ClientDataSet1.Filter := '[City] = ' + QuotedStr('London');
ClientDataSet1.Filtered := True;

The second way to filter a ClientDataSet is to use the SetRange method. Unlike Filter and Filtered, SetRange requires that you first set an index. Since an index is being used, SetRange is a much faster filter.

The following code demonstrates the use of SetRange.

ClientDataSet1.IndexFieldNames := 'City';
ClientDataSet1.SetRange(['London'],['London']);

You cancel filtering with Filter and Filtered by either setting Filtered to False or setting Filter to an empty string. You cancel filtering using SetRange by calling CancelRange. (For an interesting use of ClientDataSet where Filtered is False but a non-empty value appears in the Filter property, see the FindFirst, FindLast, FindNext, and FindPrior methods).

As is the case with sorting, you do not actually filter a DataTable. Instead, you use a DataView, which supports filtering through its RowFilter property. Filters are more complicated for Delphi developers, since single quotes are used to delimit string literals in the Filter expression. For example, consider the following filter:

DataView1.RowFilter := '[City] = ''London''';

With Delphi Prism, you can use the double quote character, which makes creating filter strings much easier, as shown in the following code.

DataView1.RowFilter := "[City] = 'London'";

You remove a previously set filter on a DataView by setting its RowFilter property to an empty string.

While it’s true that filtering a DataTable requires the use of a DataView, it is not necessary to actually create a new DataView when you want to set a filter. Each DataTable has a default DataView, which controls how the records in the DataTable appear. This DataView is accessed through the DefaultView property of the DataTable. As a result, assuming that the preceding code sample used a DataView created for a DataTable named DataTable1, the following code segment produces the same result:

DataTable1.DefaultView.RowFilter := "[City] = 'London'";

Copyright © 2009-2010 Cary Jensen. All Rights Reserved

Shifting TFields in TDataSets Bound to TDBGrids: A Potential Source of Bugs in Your Code

I've been working with Delphi since the beginning, with particular emphasis on database development. As a result, it's not often that I encounter a fundamental behavior of data-related components that catches me off guard. Well, it happened last month. And what I observed may be the source of an infrequent yet potentially disastrous bug in a large number of Delphi database applications.

Here is what I observed: The TFields in an open TDataSet changed order at runtime. Specifically, a TField that was originally in the zeroeth position (DataSet.Fields[0]) at the time that I created the TDataset, was in different position in the Fields array a short time later. I discovered this behavior when an exception was raised as a result of my attempting to programmatically read the integer value of the TIntegerField that I created in the first (zeroeth) position in my TDataSet. Between the time I created the TDataSet, and the execution of my code, the integer field had moved.

What happened wasn't magic. The TFields didn't change position by themselves, nor did they change based on anything I did in my code. What caused the TFields to physically change position in the TDataSet was that the user had changed the order of the TColumns in a TDBGrid to which a TClientDataSet was attached (through a TDataSource component, of course). The user's ability to change the position of the TColumns in a TDBGrid, by the way, is the default behavior of a TDBGrid.

Besides being interesting (I kind of assumed that once a DataSet was opened, the position of the TFields in the Fields array was pretty much set), this behavior is the potential source of intermittent exceptions, the type that are particularly difficult to track down. It turns out that this behavior, which I've never seen described before, has been around since Delphi 1. (Actually, I observed this effect in Delphi 7, Delphi 2007, and Delphi 2010. However, I understand that the underlying source of this behavior has been around since Delphi 1, though I have not specifically confirmed this.)

I created a very simple Delphi application that demonstrates this effect. It consists of a single form with one TDBGrid, a TDataSource, a TClientDataSet, and a TButton. The TClientDataSet is bound to the TDBGrid through the TDataSource. The OnCreate event handler of this form looks like the following:

procedure TForm1.FormCreate(Sender: TObject);
begin
  with ClientDataSet1.FieldDefs do
  begin
    Clear;
    Add('StartOfWeek', ftDate);
    Add('Label', ftString, 30);
    Add('Count', ftInteger);
    Add('Active', ftBoolean);
  end;
  ClientDataSet1.CreateDataSet;
end;

Button1, which is labeled Show ClientDataSet Structure, contains the following OnClick event handler.

procedure TForm1.Button1Click(Sender: TObject);
var
  sl: TStringList;
  i: Integer;
begin
  sl := TStringList.Create;
  try
    sl.Add('The Structure of ' + ClientDataSet1.Name);
    sl.Add('- - - - - - - - - - - - - - - - - ');
    for i := 0 to ClientDataSet1.FieldCount - 1 do
    sl.Add(ClientDataSet1.Fields[i].FieldName);
    ShowMessage(sl.Text);
  finally
    sl.Free;
  end;
end;

To demonstrate the moving field effect, run this application and click the button labeled Show ClientDataSet Structure. You should see something like that shown in Figure 1.

Figure 1

Next, drag the Columns of the DBGrid to re-arrange the display order of the fields. Click the Show ClientDataSet Structure button once again. This time you will see something similar to that shown in Figure 2.

Figure 2

What is remarkable about this example is that position of the TFields in the TClientDataSet's Fields property changed, such that the field that was in the ClientDataSet.Field[0] position at one point is not necessarily there moments later. And, unfortunately, this is not distinctly a TClientDataSet issue. I performed the same test with BDE-based TTables and ADO-based TADOTables and got the same effect.

It turns out that this behavior has three contributing factors. These are:

• A TDBGrid connects to a DataSet through a DataSource
• The TDBGrid permits the user to move columns at runtime
• The TColumns of the TDBGrid are dynamic, meaning that they are created by the TDBGrid at runtime

If you programmatically refer to the TFields of a DataSet connected to a TDBGrid in which the preceding three conditions exist using a literal indexer, your application can raise an exception, or produce inaccurate results, if the user moves one or more of the TColumns in that TDBGrid. In the following section I will consider several solutions to this problem, as well as share with you the reason for it.

There Are Several Solutions

There are a number of tactics that you can use to eliminating this potential bug from your applications. The first is to define the TColumns of your TDbGrid using persistent TColumns.

Creating persistent TColumns can be done either at design time or runtime. To do it at design time, add the TColumns using the Columns Editor. Display the Columns Editor by right-clicking the TDBGrid and selecting Columns Editor or by clicking the ellipsis in the Columns property of the TDBGrid in Object Inspector. If your DataSet is Active, you can click the Add All Fields button in the Columns Editor toolbar. Otherwise, add one or more TColumns and set their FieldName property in the Property Editor.

To create persistent TColumns at runtime, use the Add or Create methods of the TDBGrid's Columns property. You can then set specific properties of the added or created TColumns.

The second solution, though one that has some negative consequences, it to prevent a user from moving the TColumns of a TDBGrid. This can be done by removing the dgResizeColumn flag from the Options property of the TDBGrid. While this approach is effective, it eliminates a potentially valuable user interface option. Furthermore, removing this flag not only restricts column reordering, it prevents column resizing. (To learn how to limit column reordering without removing the column resizing option, see Zarko Gajic's article How to allow column resize by disable movement (in TDBGrid).

A third solution is to avoid referring to a TDataSet's individual TFields based on a literal indexer to the Fields array property (since this is the essence of the problem). In other words, if you need to refer to the Count TField in the preceding code sample, don't use ClientDataSet1.Fields[2]. So long as you know the name of the TField, you can use something like ClientDataSet1.FieldByName('Count').

There is one rather big drawback to the use of FieldByName, however. Specifically, this method identifies the field by iterating through the Fields property of the TDataSet, looking for a match based on the field name. Since it does this every time you call FieldByName, you should avoid this method in situations where the TField needs to be referenced many times, such as in a loop that navigates a large TDataSet.

If you do need to refer to the field repeatedly (and a large number of times), consider using something like the following code snippet:

var
CountField: TIntegerField;
Sum: Integer;
begin
 Sum := 0;
  CountField := TIntegerField(ClientDataSet1.FieldByName('Count'));
  ClientDataSet1.DisableControls; //assuming we're attached to a DBGrid
  try
    ClientDataSet1.First;
    while not ClientDataSet1.EOF do
    begin
      Sum := Sum + CountField.AsInteger;
      ClientDataSet1.Next;
    end;
  finally
    ClientDataSet1.EnableControls;
  end;
end;

The fourth solution is to use the FieldByNumber method of the TDataSet's Fields property. If you already have code that uses an indexer for the Fields array, and it works reliably, so long as the user does not move the TColumns of the bound TDBGrid, there is another solution. Change your code to use the FieldByNumber.

There are two interesting aspects to the use of FieldByNumber. First, you must qualify its reference with the Fields property of your DataSet. Second, unlike the Fields array, which is zero-based, FieldByNumber takes a one-based parameter to indicate the position of the Field you want to reference.

The following is an updated version of the Button1 event handler shown earlier that uses the FieldByNumber method.

procedure TForm1.Button1Click(Sender: TObject);
var
  sl: TStringList;
  i: Integer;
begin
  sl := TStringList.Create;
  try
    sl.Add('The Structure of ' + ClientDataSet1.Name +
       ' using FieldByNumber');
    sl.Add('- - - - - - - - - - - - - - - - - ');
    for i := 0 to ClientDataSet1.FieldCount - 1 do
    sl.Add(ClientDataSet1.Fields.FieldByNumber(i + 1).FieldName);
    ShowMessage(sl.Text);
  finally
    sl.Free;
  end;
end;

For the sample project, this code produces the following output, regardless of the orientation of the TColumns in the associated TDBGrid. This can be seen in Figure 3.

Figure 3

There is a fifth solution, but this is only available when your TDataSet is a TClientDataSet, like the one in my sample project. In those situations, you can create a clone of the original TClientDataSet, and it will have the original structure. As a result, whichever TField originally appeared in the zeroeth position will still be in that position, regardless of what a user has done to a TDBGrid that displays the TClientDataSet's data.

Note that I am not suggesting that you should reference TFields in a TDataSet using integer literals. Personally, the use of a TField variable that gets initialized through a one-time call to FieldByName is more readable, and is immune to changes in the physical order of a table's structure (though not immune to changes in the names of your fields!).

Wrap Up

There are a couple of final points I want to make. First, the actual structure of the underlying data is not affected. Specifically, if, after changing the order of the TColumns in a TDBGrid, you call the SaveToFile method of a TClientDataSet bound to that TDBGrid, the saved structure is the original (true internal) structure. Similarly, if you assign the Data property of one TClientDataSet to another, the target TClientDataSet also shows the true structure (which is similar to the effect observed when a source TClientDataSet is cloned).

Similarly, changes to the column orders of TDBGrids bound to other tested TDataSets, including TTable and ADOTable, do not affect the structure of the underlying tables. For example, a TTable that displays data from the customer.db sample Paradox table that ships with Delphi does not actually change that table's structure on disk (nor would you expect it to).

The second point is that this is not a bug in either the TDataSet or TDBGrid classes (or TColumn or TField, for that matter). This is how these classes were designed to work. And although this behavior can introduce bugs in your applications, this is because we were not aware of this behavior until now. And, you now know about this behavior, as well as how to prevent it from causing exceptions in your Delphi applications.

The final point comes to us from StackOverflow user Sertac Akyuz, who responded to a question about this behavior that I posted to that Web site. I had inspected the source for both the TDataSet as well as TDbGrid classes, and could not locate where this behavior was originating from. Sertac wrote that the behavior is actually found in the TColumns and TFields classes. Specifically, changing the column position of a dynamic TColumn results in a call to set the corresponding TField's Index property, which affects the position of the TField in its TDataSet's Fields property.

Now that you know that this potential problem exists, under what conditions it can surface, as well as the source of the effect, you should now take a look at your applications to see if you have TDbGrids with dynamic TColumns that the user can move at runtime. If you also refer to the underlying TFields associated with these TColumns using literal indexers to the TDataSet Fields property, you can eliminate potential bugs that will result from your indexer referring to the wrong field at runtime by using one of the solutions I outlined earlier in this article.

Copyright (C) 2010 Cary Jensen. All Rights Reserved

Migrating Existing Delphi Applications to Unicode-enabled Delphi

Over the life of an application it is often necessary or desirable to migrate the application to a newer version of the development environment. By doing so, the application can take advantage of a more modern interface, improvements in performance or memory management, as well as new or improved features.

It comes as no surprise, therefore, that there are many Delphi applications that have been converted, upgraded, and migrated through two or more versions of Delphi.

There is one migration, however, that stands out from the rest. That is the migration of a native code application written in Delphi 2007 or earlier to the latest version. What makes this one so different is that a number of the fundamental data types, types that have been around since the Delphi 1 and Delphi 2 days, have changed. I am talking about String, Char, and PChar, and I am referring to their support for Unicode.

In late 2009, Michael Rozlog, Senior Directory of Delphi Solutions, contacted me to see if I was interested in writing a white paper about migrating Delphi applications to Unicode-enabled Delphi. Ironically, I was in the midst of writing training material about what Unicode support meant for application development. And, I was keenly aware that a Unicode migration white paper, if it was to be effective, needed to rely on much more than just my personal experience. This really was a job for the greater RAD Studio community.

Here was my thinking. Delphi 2009 had been out for almost a year and a half, and Delphi 2010 had shipped three months earlier. As a result, many developers had been through the process of migrating existing applications to the Unicode enabled versions. And, if there was some way of collecting their stories, a clear picture of the migration process would emerge.

The call for stories, code samples, and advice on Unicode migration went out using blogs, tweets, the Embarcadero Developer Network, and some gentle personal prodding of various Delphi experts. And during 6 weeks in October of November I was fortunate enough receive a lot of input. In all, more than 20 contributors provided material, sometimes with a single, valuable code sample, and in other cases, with extensive narratives describing the preparation, process, and techniques used to migrate their applications.

And what a group of contributors this was. A number of them are the innovators behind some of the most popular third-party tools available to Delphi Developers. I also received input from well-known authors, trainers, bloggers, and authorities in the Delphi community. Everyone had something valuable to say, and it all contributed nicely to the paper.

This material is now available in the white paper Delphi Unicode Migration for Mere Mortals: Stories and Advice from the Front Lines. This paper begins with a brief overview of some of the technical aspects of Delphi's Unicode support. It then addresses specific areas of application development that may be affected by the changes to Delphi's default string types.

Throughout the paper you will find direct quotes from the contributors, and in most cases you will also find code samples that reflect the kind of changes that you may have to make to your code as you upgrade existing applications (as well as possible changes to some of the core techniques you are accustom to using). I tried hard to give credit where credit was due, as I strongly feel that a paper of this scope and breadth would be nearly impossible were it not for the generous contributions of the contributors.

To everyone who contributed material, and especially to those brave individuals who agreed to review the paper for technical accuracy, thank you.

If you are preparing to migrate an existing application to Delphi 2010 (or Delphi 2009), or are in the midst of your own migration, you will hopefully find a lot of valuable information in this white paper. I do have one request, however. When you are ready to read the white paper, please download it right before reading. I hope to update this paper sometime in the future with additional material, if it makes sense to do so. In fact, the version of this paper that out there at the time of this writing (January 9th, 2010) includes two corrections that were reported by readers. If you downloaded the PDF prior to January 8th, you have the older version. Get the newer version.

One final note. If you also have stories, advise, or code samples that are not reflected in the current version of the white paper, and would like to have your material considered for a future revision, please do not hesitate to send it to me. I cannot promise that new contributions will actually appear in the paper (there was some nice material that didn't fit into this version), but I would like to consider it.

Send your contributions to cjensen@jensendatasystems.com. Use the subject line "Unicode Migration." I promise to acknowledge every contribution, so if you don't hear from me within a week of sending it, it fell through the cracks. Please resend.

Again, the white paper can be downloaded from Delphi Unicode Migration for Mere Mortals: Stories and Advice from the Front Lines.

Copyright © 2010 Cary Jensen. All Rights Reserved.