No Commit at end of program processing on 2005

Hi All,
I have written some ETL software that allows the user to turn off
autocommit and then set the commit frequency. It is C++ talking via
ODBC.
We are testing on an SQL Server 2005 machine and the behavior we are
seeing is that the rows written after the last commit issued by the
program are not being reflected into the database. These are the
error messages...
It seems like 2005 is not performing a commit when the program
terminates successfully. So rows written between the last commit
issued by the program and program close are not showing up in the
database.
This seems strange as 2000 did issue a commit for these message at the
end of processing...that is...if we used autocommit turned off and
committed every say 10,000 rows...the messages that were written
after the last commmit have always been there...same on oracle and
other databases..
Is there a database level option to make a commit happen at the end of
processing? Or perhaps an option for the MSFT SQL Server ODBC driver
to commit when the connection is closed?
Thanks
Peter
www.peternolan.com
Hi Peter
"Peter Nolan" wrote:

> Hi All,
> I have written some ETL software that allows the user to turn off
> autocommit and then set the commit frequency. It is C++ talking via
> ODBC.
> We are testing on an SQL Server 2005 machine and the behavior we are
> seeing is that the rows written after the last commit issued by the
> program are not being reflected into the database. These are the
> error messages...
> It seems like 2005 is not performing a commit when the program
> terminates successfully. So rows written between the last commit
> issued by the program and program close are not showing up in the
> database.
> This seems strange as 2000 did issue a commit for these message at the
> end of processing...that is...if we used autocommit turned off and
> committed every say 10,000 rows...the messages that were written
> after the last commmit have always been there...same on oracle and
> other databases..
> Is there a database level option to make a commit happen at the end of
> processing? Or perhaps an option for the MSFT SQL Server ODBC driver
> to commit when the connection is closed?
> Thanks
> Peter
> www.peternolan.com
>
I am not sure why you are setting autocommit off!! If you have not
explicitly started a transaction or you are using implicit transaction any
data changed after the transaction (implicitly or explicitly) started will be
rolled back. If you allowed autocommit to be on and you did not explicitly
start the transaction then the data changed will be committed.
I don't think this behaviour has changed, but you it is not good practice to
assume that a feature of the driver is going be a shortcut to handling the
transactions properly.
John

No Commit at end of program processing on 2005

Hi All,
I have written some ETL software that allows the user to turn off
autocommit and then set the commit frequency. It is C++ talking via
ODBC.
We are testing on an SQL Server 2005 machine and the behavior we are
seeing is that the rows written after the last commit issued by the
program are not being reflected into the database. These are the
error messages...
It seems like 2005 is not performing a commit when the program
terminates successfully. So rows written between the last commit
issued by the program and program close are not showing up in the
database.
This seems strange as 2000 did issue a commit for these message at the
end of processing...that is...if we used autocommit turned off and
committed every say 10,000 rows...the messages that were written
after the last commmit have always been there...same on oracle and
other databases..
Is there a database level option to make a commit happen at the end of
processing? Or perhaps an option for the MSFT SQL Server ODBC driver
to commit when the connection is closed?
Thanks
Peter
www.peternolan.comHi Peter
"Peter Nolan" wrote:
> Hi All,
> I have written some ETL software that allows the user to turn off
> autocommit and then set the commit frequency. It is C++ talking via
> ODBC.
> We are testing on an SQL Server 2005 machine and the behavior we are
> seeing is that the rows written after the last commit issued by the
> program are not being reflected into the database. These are the
> error messages...
> It seems like 2005 is not performing a commit when the program
> terminates successfully. So rows written between the last commit
> issued by the program and program close are not showing up in the
> database.
> This seems strange as 2000 did issue a commit for these message at the
> end of processing...that is...if we used autocommit turned off and
> committed every say 10,000 rows...the messages that were written
> after the last commmit have always been there...same on oracle and
> other databases..
> Is there a database level option to make a commit happen at the end of
> processing? Or perhaps an option for the MSFT SQL Server ODBC driver
> to commit when the connection is closed?
> Thanks
> Peter
> www.peternolan.com
>
I am not sure why you are setting autocommit off!! If you have not
explicitly started a transaction or you are using implicit transaction any
data changed after the transaction (implicitly or explicitly) started will be
rolled back. If you allowed autocommit to be on and you did not explicitly
start the transaction then the data changed will be committed.
I don't think this behaviour has changed, but you it is not good practice to
assume that a feature of the driver is going be a shortcut to handling the
transactions properly.
Johnsql

No Commit at end of program processing on 2005

Hi All,
I have written some ETL software that allows the user to turn off
autocommit and then set the commit frequency. It is C++ talking via
ODBC.
We are testing on an SQL Server 2005 machine and the behavior we are
seeing is that the rows written after the last commit issued by the
program are not being reflected into the database. These are the
error messages...
It seems like 2005 is not performing a commit when the program
terminates successfully. So rows written between the last commit
issued by the program and program close are not showing up in the
database.
This seems strange as 2000 did issue a commit for these message at the
end of processing...that is...if we used autocommit turned off and
committed every say 10,000 rows...the messages that were written
after the last commmit have always been there...same on oracle and
other databases..
Is there a database level option to make a commit happen at the end of
processing? Or perhaps an option for the MSFT SQL Server ODBC driver
to commit when the connection is closed?
Thanks
Peter
www.peternolan.comHi Peter
"Peter Nolan" wrote:

> Hi All,
> I have written some ETL software that allows the user to turn off
> autocommit and then set the commit frequency. It is C++ talking via
> ODBC.
> We are testing on an SQL Server 2005 machine and the behavior we are
> seeing is that the rows written after the last commit issued by the
> program are not being reflected into the database. These are the
> error messages...
> It seems like 2005 is not performing a commit when the program
> terminates successfully. So rows written between the last commit
> issued by the program and program close are not showing up in the
> database.
> This seems strange as 2000 did issue a commit for these message at the
> end of processing...that is...if we used autocommit turned off and
> committed every say 10,000 rows...the messages that were written
> after the last commmit have always been there...same on oracle and
> other databases..
> Is there a database level option to make a commit happen at the end of
> processing? Or perhaps an option for the MSFT SQL Server ODBC driver
> to commit when the connection is closed?
> Thanks
> Peter
> www.peternolan.com
>
I am not sure why you are setting autocommit off!! If you have not
explicitly started a transaction or you are using implicit transaction any
data changed after the transaction (implicitly or explicitly) started will b
e
rolled back. If you allowed autocommit to be on and you did not explicitly
start the transaction then the data changed will be committed.
I don't think this behaviour has changed, but you it is not good practice to
assume that a feature of the driver is going be a shortcut to handling the
transactions properly.
John

newbie: processing records sequentially

Hi to all,
I'm a newbie to T-SQL but not to programming. What I want to do is
process every record in a table, and depending on the values in the
record itself, either copy the record to another table or update a
third table, and the delete the original record. Basically, it's a data
preparation area table where the records are imported with no error
check, and then validated against a set of conditions.

THE QUESTION IS: which approach should I follow? Is there a sort of
FORALL loop in T-SQL that I can execute on every record in the table?
Or shall I rely on other tools (VB Scripting, or...)? SQL Server 2000

Thanx to allIn SQL your goal should generally be to avoid processing data a row at
a time. This example may help you:

INSERT INTO AnotherTable (col1, col2, ...)
SELECT col1, col2, ...
FROM YourTable
WHERE ... ? /* unspecified */

UPDATE AnotherTable
SET ... ? /* unspecified */
WHERE ... /* uspecified */

DELETE FROM YourTable
WHERE ...

etc.

Hopefully this gives you the idea. If you need more help, please refer
to the following article which describes the best way to post a fuller
description of your problem:
http://www.aspfaq.com/etiquette.asp?id=5006

--
David Portas
SQL Server MVP
--|||David,
thanks for prompt response. I understand the process is somehow
reversed from a row-by-row cycle. It seems to me somehow inefficient,
though, to run a query against a third table twice in order to UPDATE
the first time and DELETE the second.

Cheers

David Portas wrote:
> In SQL your goal should generally be to avoid processing data a row at
> a time. This example may help you:
> INSERT INTO AnotherTable (col1, col2, ...)
> SELECT col1, col2, ...
> FROM YourTable
> WHERE ... ? /* unspecified */
> UPDATE AnotherTable
> SET ... ? /* unspecified */
> WHERE ... /* uspecified */
> DELETE FROM YourTable
> WHERE ...
> etc.
> Hopefully this gives you the idea. If you need more help, please refer
> to the following article which describes the best way to post a fuller
> description of your problem:
> http://www.aspfaq.com/etiquette.asp?id=5006|||It depends. INSERT, DELETE and UPDATE are separate operations so will
always require separate statements whatever method you use. The main
question is how best to optimise the WHERE selection criteria. You
didn't specify what those criteria are so I can only guess. It is at
least likely that putting the criteria in WHERE clauses, with suitable
indexes, will be much more efficient than the overhead of looping
through each row in a cursor. Understand that SQL is a declarative
language, unlike procedural languages that you are probably more
familiar with.Cursor processing (row by row) is rarely a good idea in
SQL and when manipulating data should usually be a last resort only.

--
David Portas
SQL Server MVP
--|||>> I'm a newbie to T-SQL but not to programming.<<

You missed the point of a declarative, set-oriented language and it
will take you about a year to un-learn your old mental models. I have
a book entitled SQL PROGRAMMING STYLE that deals with this problem in a
few chapters.

>> What I want to do is process every record [sic] in a table, and
depending on the values in the record [sic] itself, either copy the
record [sic] to another table or update a third table, and the delete
the original record [sic] . <<

This is a common problem; rows are not records, tables are not files
and columns are not fields. Totally differrent concepts. I will give
you the long painful details that I post on this topic if you ask.
Have you ever worked with LISP, FP, Prolog or a language that was not
based on procedures? This is a new way of thinking.

>> Basically, it's a data preparation area table where the records
[sic] are imported with no error check, and then validated against a
set of conditions. <<

This is usually done with an ETL tool these days, but you can do this
with a few statements. If you have an implementation of the new SQL-99
MERGE statement, then you can probably do it in one statement. It will
be a very complex statement, tho :)

BEGIN
UPDATE RawData
SET <column edits>;
DELETE GoodStuff
WHERE EXISTS
(SELECT *
FROM RawData
WHERE <<conditions>>;
INSERT INTO GoodStuff
SELECT a, b, c, ..
FROM RawData
WHERE <<conditions>>;
END;

First scrub the raw data, then remove the old data that is going to be
replaced, and finally insert the new clean data.|||moco (makelovenotspamdaldegan@.email.it) writes:
> thanks for prompt response. I understand the process is somehow
> reversed from a row-by-row cycle. It seems to me somehow inefficient,
> though, to run a query against a third table twice in order to UPDATE
> the first time and DELETE the second.

And with a loop there will be many queries against the table.

But as they say, seeing is believing. Go ahead and bench-mark. You will
be amazed...

--
Erland Sommarskog, SQL Server MVP, esquel@.sommarskog.se

Books Online for SQL Server SP3 at
http://www.microsoft.com/sql/techin.../2000/books.asp